Now your agent can find anything in your computer.
Important
Looking for the original AutoRAG (RAG AutoML / pipeline optimization tool)?
This repository now hosts AutoRAG 2.0, a complete reimagining of AutoRAG as a self-evolving librarian agent. The original Python-based AutoRAG — the RAG AutoML tool for automatically finding an optimal RAG pipeline for your data — now lives in the legacy/ directory of this repository.
The legacy AutoRAG is NOT abandoned. It continues to be maintained (bug fixes, dependency updates, and PyPI releases via pip install AutoRAG) in maintenance mode. Existing users can keep using it exactly as before — see the legacy README for its documentation, and file issues in this repository as usual. New feature development is focused on AutoRAG Agent (2.0).
Search tools dump file paths and matching lines — forcing you to open files, read context, and synthesize answers yourself.
AutoRAG Agent is a self-evolving librarian agent. Built on the Pi agent framework, a single configured model searches across multiple retrieval methods, opens source documents directly via bash to verify ground truth, and curates answers into clean, numbered knowledge units:
You ask: "What changes were made to the cloud infrastructure contract in Q3?"
AutoRAG Agent:
[1] Enterprise Compute Discount — AWS annual discount tier increased to 28% based on commitment volume. (contracts/cloud-2026.pdf:p.4)
[2] Regional Data Residency — Explicit compliance clause pinning customer data storage to the Seoul AWS region. (/slack/legal-ops/chunks/318)
[3] SLA Guarantee — Guaranteed uptime threshold adjusted to 99.95% with penalty credits starting at 15 minutes downtime. (contracts/sla-addendum.md:lines 45-62)
Three principles drive every design decision in AutoRAG Agent:
-
Never migrate your data to search it. Traditional RAG systems force you to upload, ETL, and duplicate your files into a centralized vector database. AutoRAG Agent federates your data in place, querying CLI-native stores (
katok,discrawl,slacrawl,mailcrawl,rclone,qmd) where your data already lives. Results retain opaque, source-native identities (/kakao/...,/slack/...) that preserve local access control and privacy. (See our Competitive Landscape Study on why in-place federation is the durable differentiator). -
Just works — no RAG degree required. No pipeline tuning, no vector-DB maintenance, and no remote embedding API keys. AutoRAG Agent automatically manages MinSync for incremental Change Data Capture (CDC) chunking and provides a local embedding gateway out of the box with zero external telemetry.
-
Fast by design. Rather than coordinating slow multi-agent hierarchies, a single configured model owns the entire retrieval, direct-read, and curation loop. Local CDC chunks (BM25, vector, and hybrid modes) deliver rapid, low-latency turnaround across multi-turn research queries.
AutoRAG Agent orchestrates five integrated subsystems:
- Self-Evolving Memory System (
check_memory): Before querying, the agent consults historical outcomes stored in~/.autorag/memory.jsonto prioritize retrieval methods that have proven successful for similar queries. - Pluggable Multi-Method Retrieval:
- BM25 Lexical Search: Fast keyword ranking via MinSync over parsed markdown mirrors.
- Semantic Vector Search: Dense vector retrieval over CDC chunks via the built-in embedding gateway.
- Hybrid Search: Combines BM25 and vector ranking via Reciprocal Rank Fusion (RRF).
- Jikji Find-First Discovery: Local CLI-backed fast discovery answer packs.
- Datasource Skills: Server-authorized federated retrieval across external applications.
- Result Merger & Scoped Access Gate: Cross-method deduplication, score normalization, and default-deny permission checks.
- Direct Evidence Reading (
bash): The agent directly opens and inspects promising files withcat,grep, orfindto verify facts against ground truth. - Curation & Active Feedback: Structured findings are returned via
emit_autorag_results. When callers provide feedback on which items were useful, AutoRAG records this to optimize future queries.
If you are an AI coding agent or LLM (Claude Code, Cursor, Windsurf, Codex, Senpi, Pi, etc.) tasked with installing, configuring, or querying AutoRAG Agent, use our built-in skills located in the skills/ directory:
| Skill | Directory | When to Use |
|---|---|---|
autorag |
skills/autorag/ |
Querying, searching, comparing, and summarizing documents with an already configured AutoRAG librarian. |
autorag-setup |
skills/autorag-setup/ |
Installing AutoRAG, configuring models, adding document roots/datasources, running health checks, and repairing indexes. |
autorag-lite-setup |
skills/autorag-lite-setup/ |
Initializing and maintaining the model-free AutoRAG Lite lifecycle without an LLM. |
autorag-lite-search |
skills/autorag-lite-search/ |
Performing model-free retrieval, reporting evidence, and recording feedback without an LLM. |
- Install CLI:
command -v autorag >/dev/null || bun install -g @autorag/librarian
- Inspect & Preflight:
autorag status --json # check corpus freshness and index readiness autorag duplicates --json # scan for exact or near-duplicate document families
- Perform Curated Search:
autorag search "your question" --json - Safety Guidelines for Agents:
- Never delete, move, or modify original user documents.
- AutoRAG writes index artifacts strictly into
<workspace>/.autorag/and.jikji/caches. - Never print or leak API keys, tokens, or credential values into stdout or logs.
Need blazing fast local search without configuring an LLM or paying for API tokens? Use AutoRAG Lite.
AutoRAG Lite provides the exact same high-performance indexing, BM25 ranking, and local MinSync vector/hybrid retrieval engine as the full librarian, but runs 100% model-free:
- Zero LLM Token Usage: Run purely local BM25 and vector search offline.
- Agent Integration Ready: Use
autorag lite retrieveinside your own agentic workflows to supply raw context chunks to an external model. - Fast Local CLI: Instant responses directly from your terminal.
# Initialize a model-free workspace
autorag lite init --search-paths ./documents
# Index local files and datasources
autorag lite refresh
# Retrieve ranked document chunks (returns JSON candidates with scores)
autorag lite retrieve "compliance policy exception process" --top-k 5 --json
# Check index status and freshness
autorag lite statusAutoRAG Agent connects to external tools and communication platforms using dedicated datasource skills. Data remains in each tool's native store — AutoRAG does not copy or ingest foreign databases into a centralized store:
| Datasource | Skill Alias | Backend / Driver | Storage & Privacy Model | Search Capabilities |
|---|---|---|---|---|
| Local Documents | local |
Native filesystem & MinSync | Workspace-local parsed mirrors (.autorag/) |
BM25, Semantic, Hybrid |
| KakaoTalk | katok |
katok CLI |
Native KakaoTalk archive; zero direct DB access | BM25, Semantic, Hybrid |
| Discord | discord |
discrawl CLI |
Native SQLite archive; token-free wiretap mode | BM25, Semantic, Hybrid |
whatsapp |
wacrawl CLI |
Local-first incremental archive + FTS5 | Lexical FTS5 | |
| Telegram | telegram |
telecrawl CLI |
Local-first desktop archive + FTS5 | Lexical FTS5 |
| Slack | slack |
slacrawl CLI |
Local workspace/channel/thread archive + FTS5 | Lexical FTS5 |
| Notion | notion |
notcrawl CLI |
Local page/database/block archive + FTS5 | Lexical FTS5 |
| Email Archives | mailcrawl |
mailcrawl CLI |
Local Gmail, IMAP, and Maildir storage | BM25, Semantic, Hybrid |
| Local Mail Export | mail-export |
Built-in .mbox / .eml parser |
Local filesystem mailboxes | Lexical |
| Obsidian Vaults | obsidian |
qmd CLI |
Direct markdown vault indexing | BM25, Semantic |
| GitHub | github |
GitHub REST API | In-memory fetched Issues and Pull Requests | Lexical, Scoped |
| Cloud Drives | cloud-drive |
rclone CLI |
Google Drive (Tier-1), OneDrive, Dropbox, etc. | Incremental Mirror + BM25 |
| Photos & Shots | clawgallery |
clawgallery CLI |
Local screenshot and photo store | Hybrid OCR/Visual Search |
| RSS / News | rss |
Native HTTP Poller | RSS 2.0 & Atom feeds (24h deduplication) | Lexical |
| macOS Spotlight | spotlight |
Native mdfind CLI |
macOS system metadata and content index | System Native |
For configuration syntax and connector details, see docs/datasource-skills.md.
AutoRAG Agent is published as @autorag/librarian (requires Node.js ≥ 24 or Bun):
# Install CLI globally
bun install -g @autorag/librarian
# or with npm:
npm install -g @autorag/librarian
# Add as a TypeScript/JavaScript library
bun add @autorag/librarian- Java 11 or newer: Required for PDF parsing via
@opendataloader/pdf. Check withjava -version. - Rust Toolchain (Optional): Automatically compiles Jikji (
jikji-cli) if installed. - MinSync: Automatically downloaded and installed into
<workspace>/.autorag/binon first run.
# Initialize configuration for your documents folder
autorag init --search-paths ~/Documents/research
# Index documents (parses PDFs/Markdown, builds BM25 and MinSync vectors)
autorag refresh
# Check indexing health
autorag status# Perform a curated search (uses your configured reasoning model)
autorag search "What are our primary Q3 deliverables?"
# Launch the interactive Terminal UI (beta)
autorag tui
# Open the loopback web management dashboard (127.0.0.1)
autorag uiimport { AutoRAGAgent } from "@autorag/librarian";
// Initialize librarian agent
const agent = new AutoRAGAgent({
searchPaths: ["/path/to/documents"],
});
// Run curated search loop
const response = await agent.searchDocuments("Summarize recent compliance updates");
console.log("Answer:", response.answer);
for (const result of response.results) {
console.log(`[${result.number}] ${result.title} (${result.source})`);
console.log(` ${result.summary}`);
}
// Record feedback: Result [1] was useful, [2] was not
agent.recordFeedbackByNumbers(response.sessionId, [1], [2]);| Command | Description |
|---|---|
autorag init |
Initialize ~/.autorag/config.json with search roots and model settings |
autorag refresh |
Refresh parsed mirrors, MinSync CDC chunks, datasources, and Jikji |
autorag search "<query>" |
Run the librarian agent to curate structured answers |
autorag status |
Inspect corpus freshness, indexing status, and vector readiness |
autorag health |
Check model provider authentication, token validity, and API reachability |
autorag tui |
Open the interactive librarian terminal UI |
autorag ui |
Open the local loopback web dashboard to configure datasources visually |
autorag duplicates [DIR] |
Read-only scan for exact and near-duplicate document families with dupey |
autorag lite ... |
Model-free indexing, retrieval, report generation, and status |
autorag feedback <session> |
Record useful / not-useful feedback by item number |
autorag evidence <session> |
Inspect exact underlying document chunks and sources for a past query |
autorag serve |
Start the P2P query server over SimpleX (opt-in) |
autorag p2p ... |
Manage peer trust, query approvals, and sharing policies |
Deep dive into AutoRAG Agent's architecture, security, and integration guides:
- MinSync Setup & Embedding QA: Automatic binary installation, CDC chunking, and EmbeddingGemma verification.
- Local Embedding Runtime & Gateway: AutoRAG-owned local gateway, model prefetching, and zero-egress semantic search.
- Datasource Skills Reference: Full configuration contracts, connection aliases, and connector options.
- Manual QA & Datasource Test Harnesses: Real-world testing guides for Discord, KakaoTalk, Slack, Notion, and email.
- P2P SimpleX Sharing & Path Standard: Decentralized peer query sharing with SimpleX, PII redaction, and approval queues.
- Supply Chain Security & License Audits: Software bill of materials (SBOM) and dependency gate policies.
- Competitive Landscape Study: In-depth analysis of why in-place federation outperforms centralized RAG.
- Legacy AutoRAG (AutoML): Documentation for the legacy Python AutoML pipeline optimizer (
pip install AutoRAG).
AutoRAG Agent is an open-source project built by the community. We welcome contributions, bug reports, datasource connectors, and ideas!
- Contributing: Feel free to open an issue or pull request. Start with CONTRIBUTING.md — development setup, the
make cicheck we ask for, theSigned-off-bytrailer every commit needs, and when to run the live end-to-end lanes. MAINTAINERS lists who reviews which area. - GitHub Contributors: View all contributors on GitHub.
AutoRAG Agent stands on the shoulders of fantastic open-source projects:
- Pi Framework by @earendil-works — The foundational agent framework powering AutoRAG's reasoning loop.
- MinSync — Ultra-fast incremental Change Data Capture (CDC) chunking and local BM25/vector indexing.
- Jikji by NomaDamas — High-performance find-first local document discovery.
- dupey by NomaDamas — Fast duplicate and near-duplicate document family detection.
- Federated CLI Authors: External datasource tools
katok,discrawl,mailcrawl,wacrawl,telecrawl,slacrawl,notcrawl,qmd, andrclone. - OpenDataLoader / Docling — Robust PDF and document parsing runtimes.
When search returns nothing, a datasource disappears from results, refresh hangs, or the gateway will not start, run the autorag-doctor agent skill (skills/autorag-doctor/SKILL.md). Point your coding agent at it and say "check AutoRAG" — it walks the full diagnose-and-repair procedure and ends with a per-datasource status table showing what is indexed and what actually returns hits.
The first three commands cover most of it:
autorag status --json # freshness, per-component state, diagnostics
autorag health --json # model resolution + one live completion probe
autorag gateway status --format json # on-demand embedding runtimeIndexing is not the same thing as searchability, so always confirm retrieval itself — this needs no model:
autorag lite retrieve 'a word that certainly appears' --top-k 3 --json
autorag lite retrieve 'recent topic' --tags discord --top-k 3 --jsonCommon failures and their fix:
| Symptom | Diagnostic code | Fix |
|---|---|---|
| Results are missing recent files | stale-index |
autorag refresh --method parsed,minsync --json |
| Semantic search returns nothing after changing the embedder | embedding-identity-mismatch |
autorag index rebuild --method minsync |
| Gateway will not start, a previous run was killed | lock-conflict |
autorag gateway stop, then retry |
| A datasource is healthy in its own CLI but absent from results | — | add its tag/scope to datasourceAccess |
| A datasource errors during refresh | datasource-index-failed |
run that CLI's own doctor |
| MinSync or Jikji missing | minsync-unavailable, jikji-unavailable |
check the Rust toolchain, re-run refresh |
Native datasource stores stay owned by their CLIs — AutoRAG never rebuilds them. Fix a broken archive with katok doctor, discrawl --json metadata, slacrawl --json doctor, wacrawl --json doctor, telecrawl --json doctor, notcrawl doctor, qmd status, or mailcrawl doctor, then re-run autorag refresh --method datasources --json.
- AutoRAG 2.0 (AutoRAG Agent): Released under the MIT License.
- Legacy Python AutoRAG (
legacy/): Released under the Apache License 2.0. - Production third-party notices and licenses are documented in
NOTICE.
