Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
| Date | Stars |
|---|---|
| 2026-07-31 | 317 |
| 2026-08-06 | 317 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Firecrawl LLMs.txt Generator A Python script that generates `llms.txt` and `llms-full.txt` files for any website using Firecrawl and OpenAI APIs. ## What is llms.txt? `llms.txt` is a standardized format for making website content more accessible to Large Language Models (LLMs). It provides: - **llms.txt**: A concise index of all pages with titles and descriptions - **llms-full.txt**: Complete content of all pages for comprehensive access ## Features - 🗺️ **Website Mapping**: Automatically discovers all URLs on a website using Firecrawl's map endpoint - 📄 **Content Scraping**: Extracts markdown content from each page - 🤖 **AI Summaries**: Uses OpenAI's GPT-4o-mini to generate concise titles and descriptions - ⚡ **Parallel Processing**: Processes multiple URLs concurrently for faster generation - 🎯 **Configurable Limits**: Set maximum number of URLs to process - 📁 **Flexible Output**: Choose to generate both files or just llms.txt ## Prerequisites - Python 3.7+ - Firecrawl API key ([Get one here](https://firecrawl.dev)) - OpenAI API key ([Get one here](https://platform.openai.com)) ## Installation 1. Clone the repository: ```bash git clone <repository-url> cd <repository-directory> ``` 2. Install dependencies: ```bash pip install -r requirements.txt ``` 3. Set up API keys (choose one method): **Option A: Using .env file (recommended)** ```bash cp env.example .env # Edit .env and add your API keys ``` **Option B: Using environment variables** ```bash export FIRECRAWL_API_KEY="your-firecrawl-api-key" export OPENAI_API_KEY="your-openai-api-key" ``` **Option C: Using command line arguments** (See usage examples below) ## Usage ### Basic Usage Generate llms.txt and llms-full.txt for a website: ```bash python generate-llmstxt.py https://example.com ``` ### With Options ```bash # Limit to 50 URLs python generate-llmstxt.py https://example.com --max-urls 50 # Save to specific directory python generate-llmstxt.py https://example.com --output-dir ./output # Only generate llms.txt (skip full text) python generate-llmstxt.py https://example.com --no-full-text # Enable verbose logging python generate-llmstxt.py https://example.com --verbose # Specify API keys via command line python generate-llmstxt.py https://example.com \ --firecrawl-api-key "fc-..." \ --openai-api-key "sk-..." ``` ### Command Line Options - `url` (required): The website URL to process - `--max-urls`: Maximum number of URLs to process (default: 20) - `--output-dir`: Directory to save output files (default: current directory) - `--firecrawl-api-key`: Firecrawl API key (defaults to .env file or FIRECRAWL_API_KEY env var) - `--openai-api-key`: OpenAI API key (defaults to .env file or OPENAI_API_KEY env var) - `--no-full-text`: Only generate llms.txt, skip llms-full.txt - `--verbose`: Enable verbose logging for debugging ## Output Format ### llms.txt ``` # https://example.com llms.txt - [Page Title](https://example.com/page1): Brief description of the page content here - [Another Page](https://example.com/page2): Another concise description of page content ``` ### llms-full.txt ``` # https://example.com llms-full.txt <|firecrawl-page-1-lllmstxt|> ## Page Title Full markdown content of the page... <|firecrawl-page-2-lllmstxt|> ## Another Page Full markdown content of another page... ``` ## How It Works 1. **Website Mapping**: Uses Firecrawl's `/map` endpoint to discover all URLs on the website 2. **Batch Processing**: Processes URLs in batches of 10 for efficiency 3. **Content Extraction**: Scrapes each URL to extract markdown content 4. **AI Summarization**: For each page, GPT-4o-mini generates: - A 3-4 word title - A 9-10 word description 5. **File Generation**: Creates formatted llms.txt and llms-full.txt files ## Error Handling - Failed URL scrapes are logged and skipped - If no URLs are found, the script exits with an error - API errors are logged with details for debugging - Ra
Excerpt of 5,564 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:b7bbaba4f0e36ec6, llm:Repository name 'create-llmstxt-py' suggests a Python tool to create LLMs .txt (likely prompts or data) but no README or topics provided. Minimal metadata indicates it's related to LLM utilities for prompt/data creation in Python.
