Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Self-hosted search + markdown harvester for AI agents. SearXNG (100+ engines) + FastAPI + trafilatura. Tavily-compatible /search plus /extract with size presets and pagination. One-command Docker Compose.
| Date | Stars |
|---|---|
| 2026-07-24 | 254 |
| 2026-07-25 | 254 |
| 2026-07-28 | 254 |
| 2026-07-30 | 254 |
| 2026-08-06 | 254 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Searcharvester 🌾 **Self-hosted search + extract + deep research for AI agents** > 📖 **Docs:** [English](docs/en/README.md) · [Русский](docs/ru/README.md) · [中文](docs/zh/README.md) Three composable HTTP services in a single `docker compose up`: - **`/search`** — Tavily-compatible search via SearXNG (100+ engines) - **`/extract`** — URL → clean markdown via trafilatura, with size presets and pagination - **`/research`** — deep research agent: give it a question, get back a cited markdown report No API keys, no quotas, fully self-hosted. Pre-built image on GHCR. ## 🚀 Quick start ```bash # 1. Clone git clone [email protected]:vakovalskii/searcharvester.git cd searcharvester # 2. Config cp config.example.yaml config.yaml # Change server.secret_key (32+ chars) # 3. (Optional) LLM credentials for /research — any OpenAI-compatible endpoint cat > .env <<EOF OPENAI_API_KEY=sk-... OPENAI_BASE_URL=https://api.openai.com/v1 EOF # 4. Start — pulls ghcr.io/vakovalskii/searcharvester docker compose up -d # 5. Test search curl -X POST localhost:8000/search -H 'Content-Type: application/json' \ -d '{"query":"bitcoin price","max_results":3}' # 6. Test extract (URL → markdown) curl -X POST localhost:8000/extract -H 'Content-Type: application/json' \ -d '{"url":"https://en.wikipedia.org/wiki/Docker_(software)","size":"m"}' # 7. Test deep research (needs LLM creds from step 3) curl -X POST localhost:8000/research -H 'Content-Type: application/json' \ -d '{"query":"What is trafilatura? One paragraph with source."}' # → {"job_id":"...","status":"queued"} # Poll GET /research/{job_id} until status=completed, grab the report. ``` --- ## 🧱 Three services, one API ### 1️⃣ `POST /search` — Tavily-compatible search Drop-in replacement for the [Tavily](https://tavily.com) API: ```python from tavily import TavilyClient client = TavilyClient(api_key="ignored", base_url="http://localhost:8000") response = client.search(query="...", max_results=5, include_raw_content=True) ``` Request body: ```json { "query": "...", "max_results": 10, "include_raw_content": false, "engines": "google,duckduckgo,brave", "categories": "general" } ``` Response — Tavily schema (see [`docs/en/api.md`](docs/en/api.md)). ### 2️⃣ `POST /extract` — URL → clean markdown Takes a URL, fetches the HTML, runs [trafilatura](https://github.com/adbar/trafilatura) for main-content extraction (strips nav/footer/ads, preserves headings, lists, tables, links), returns ready-to-use markdown. **Size presets for different context windows:** | Size | Chars | Use case | |---|---|---| | `s` | 5 000 | Quick summary, small-context LLMs | | `m` | 10 000 | Default agent reading | | `l` | 25 000 | Deep single-page read | | `f` | full | Paginated by 25 000 — read long docs piece by piece | **Pagination via cache:** ```bash # Get id + page 1 curl -X POST localhost:8000/extract -d '{"url":"...","size":"f"}' # → {"id":"abc123","content":"...","pages":{"current":1,"total":4,"next":"/extract/abc123/2"}} # Next pages — no re-download curl localhost:8000/extract/abc123/2 ``` Cache keyed by `md5(url)[:16]`, TTL 30 minutes. Cold fetch: 1-3 s; cached page: <50 ms. Useful as a standalone service, not just for the agent — plug it into any LLM pipeline that needs clean page text. ### 3️⃣ `POST /research` — deep research agent `{query}` → orchestrator spawns an ephemeral [Hermes Agent](https://github.com/nousresearch/hermes-agent) container with three skills: | Skill | Role | |---|---| | `searcharvester-search` | Tool: calls our `/search` | | `searcharvester-extract` | Tool: calls our `/extract` | | `searcharvester-deep-research` | Methodology (markdown only, no code): plan → gather → gap-check → synthesise → verify | The agent reads the methodology, plans sub-queries, loops search→extract, synthesises a markdown report with `[1][2]` citations, saves it to `/workspace/report.md`. The orchestrator watches for the `REPORT_SAVED:` marker and returns the file to the client. LLM-a
Excerpt of 17,889 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:1e08264dcb06047d, topic:ai-agents, topic:agentic-ai, desc:ai agents