Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Bring your own agent and build a self-improving agentic system. Automatically mine failures, optimize the agent harness, and gate against regressions.
| Date | Stars |
|---|---|
| 2026-07-31 | 526 |
| 2026-08-02 | 526 |
| 2026-08-06 | 526 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# auto-harness > Give a coding agent a benchmark and an agent file. Let it iterate overnight. It reads failures, improves the system prompt and tools, gates every change against a self-maintained eval suite, and repeats. This repo is a simplified version of our auto-harness agent setup. We demonstrate our system on Tau3 benchmark tasks where the agent's score improves from 0.56 to 0.78 (~40% jump) while mining failures and auto maintaining live evals. If you are curious to learn more, read the full blog here - https://www.neosigma.ai/blog/self-improving-agentic-systems. The loop is defined in `PROGRAM.md`. The coding agent edits `agent/agent.py` to improve the agent and appends findings to `workspace/learnings.md` after each iteration. --- ## Supported Benchmarks | Benchmark | Domain | Tasks | Agent Interface | |-----------|--------|-------|-----------------| | **tau-bench** | Customer service (retail, airline, telecom) | retail: 114, airline: 50, telecom: 114 | Structured tool calls via tau2 | | **Terminal-Bench 2.0** | Real-world terminal tasks (coding, sysadmin, security) | 89 | Bash commands via Harbor containers | | **BIRD-Interact** | Interactive text-to-SQL (multi-turn, CRUD over Postgres) | lite: 300, full: 600 | Google ADK agent against a 3-service environment (user sim, DB env, system agent) | --- ## How it works ``` run benchmark → analyze → improve agent/agent.py → gate → record → update learnings → repeat ``` - **`agent/agent.py`** — the agent being optimized (copied from a benchmark-specific template) - **`agent/templates/`** — starting-point templates for each benchmark (read-only) - **`benchmark.py`** — runs your benchmark, returns per-task rewards - **`gating.py`** — three-step gate: eval suite + full test val_score + suite promotion - **`record.py`** — appends iteration results to `workspace/results.tsv` - **`prepare.py`** — sets up workspace, copies templates, runs baseline - **`program_templates/`** — benchmark-specific PROGRAM.md instructions - **`PROGRAM.md`** — instructions the coding agent follows (copied from template by prepare.py) --- ## Quick start: Terminal-Bench 2.0 **Requirements:** `harbor` CLI, an `OPENAI_API_KEY`, and a coding agent (Claude Code, Codex CLI, or similar). If using a sandboxed `env_provider` (the default), you'll also need its credential: `E2B_API_KEY`, `DAYTONA_API_KEY`, or a Modal token via `modal token new` / `MODAL_TOKEN_ID` + `MODAL_TOKEN_SECRET`. `env_provider: "docker"` needs none of these. ```bash # 1. Clone the repo git clone https://github.com/neosigmaai/auto-harness cd auto-harness # 2. Install harbor uv tool install harbor # 3. Set up environment variables cp .env.example .env # edit .env — set OPENAI_API_KEY, plus your sandbox provider's credential # (E2B_API_KEY, DAYTONA_API_KEY, or MODAL_TOKEN_ID + MODAL_TOKEN_SECRET) — # not needed if you're using env_provider: "docker" # 4. Configure the experiment cp experiment_config.yaml.template experiment_config.yaml # edit experiment_config.yaml — uncomment the terminal-bench section # 5. Initialize workspace + run baseline (runs all 89 tasks, generates train/test split) python prepare.py # 6. Start the optimization loop # Point your coding agent at the repo and prompt: # "Read PROGRAM.md and start the optimization loop." ``` ## Quick start: BIRD-Interact **Requirements:** Docker (for Postgres), Python 3.12+, `git-lfs` (for the HF dataset), an `OPENAI_API_KEY` (or `ANTHROPIC_API_KEY` / `GEMINI_API_KEY` depending on model), and a coding agent. ```bash # 1. Clone this repo git clone https://github.com/neosigmaai/auto-harness cd auto-harness # 2. Set up environment variables cp .env.example .env # edit .env — set OPENAI_API_KEY (or ANTHROPIC_API_KEY) # 3. Configure the experiment cp experiment_config.yaml.template experiment_config.yaml # edit experiment_config.yaml — uncomment the BIRD-INTERACT section # 4. Initialize — prepare.py auto-provisions everything: # - clones BIRD-Interact-ADK into ./
Excerpt of 12,615 characters
Read on GitHub
8
2
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:6a1e7cea33ffd675, desc:agentic