Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants
| Date | Stars |
|---|---|
| 2026-07-31 | 2225 |
| 2026-08-06 | 2231 |
Today
+6 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<h1 align="center">Crawl4AI RAG MCP Server</h1> <p align="center"> <em>Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants</em> </p> A powerful implementation of the [Model Context Protocol (MCP)](https://modelcontextprotocol.io) integrated with [Crawl4AI](https://crawl4ai.com) and [Supabase](https://supabase.com/) for providing AI agents and AI coding assistants with advanced web crawling and RAG capabilities. With this MCP server, you can <b>scrape anything</b> and then <b>use that knowledge anywhere</b> for RAG. The primary goal is to bring this MCP server into [Archon](https://github.com/coleam00/Archon) as I evolve it to be more of a knowledge engine for AI coding assistants to build AI agents. This first version of the Crawl4AI/RAG MCP server will be improved upon greatly soon, especially making it more configurable so you can use different embedding models and run everything locally with Ollama. Consider this GitHub repository a testbed, hence why I haven't been super actively address issues and pull requests yet. I certainly will though as I bring this into Archon V2! ## Overview This MCP server provides tools that enable AI agents to crawl websites, store content in a vector database (Supabase), and perform RAG over the crawled content. It follows the best practices for building MCP servers based on the [Mem0 MCP server template](https://github.com/coleam00/mcp-mem0/) I provided on my channel previously. The server includes several advanced RAG strategies that can be enabled to enhance retrieval quality: - **Contextual Embeddings** for enriched semantic understanding - **Hybrid Search** combining vector and keyword search - **Agentic RAG** for specialized code example extraction - **Reranking** for improved result relevance using cross-encoder models - **Knowledge Graph** for AI hallucination detection and repository code analysis See the [Configuration section](#configuration) below for details on how to enable and configure these strategies. ## Vision The Crawl4AI RAG MCP server is just the beginning. Here's where we're headed: 1. **Integration with Archon**: Building this system directly into [Archon](https://github.com/coleam00/Archon) to create a comprehensive knowledge engine for AI coding assistants to build better AI agents. 2. **Multiple Embedding Models**: Expanding beyond OpenAI to support a variety of embedding models, including the ability to run everything locally with Ollama for complete control and privacy. 3. **Advanced RAG Strategies**: Implementing sophisticated retrieval techniques like contextual retrieval, late chunking, and others to move beyond basic "naive lookups" and significantly enhance the power and precision of the RAG system, especially as it integrates with Archon. 4. **Enhanced Chunking Strategy**: Implementing a Context 7-inspired chunking approach that focuses on examples and creates distinct, semantically meaningful sections for each chunk, improving retrieval precision. 5. **Performance Optimization**: Increasing crawling and indexing speed to make it more realistic to "quickly" index new documentation to then leverage it within the same prompt in an AI coding assistant. ## Features - **Smart URL Detection**: Automatically detects and handles different URL types (regular webpages, sitemaps, text files) - **Recursive Crawling**: Follows internal links to discover content - **Parallel Processing**: Efficiently crawls multiple pages simultaneously - **Content Chunking**: Intelligently splits content by headers and size for better processing - **Vector Search**: Performs RAG over crawled content, optionally filtering by data source for precision - **Source Retrieval**: Retrieve sources available for filtering to guide the RAG process ## Tools The server provides essential web crawling and search tools: ### Core Tools (Always Available) 1. **`crawl_single_page`**: Quickly crawl a single web page and store its content in the vector database 2
Excerpt of 19,456 characters
Read on GitHub
Cole Medin · Dynamous
17
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:9c51ad99deb10641, desc:ai agents