Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Bring your own agent and build a self-improving agentic system. Automatically mine failures, optimize the agent harness, and gate against regressions.
| Date | Stars |
|---|---|
| 2026-08-02 | 526 |
| 2026-08-06 | 526 |
| 2026-08-12 | 530 |
| 2026-08-18 | 532 |
| 2026-08-20 | 533 |
| 2026-08-21 | 535 |
| 2026-08-26 | 536 |
| 2026-08-27 | 535 |
| 2026-08-30 | 534 |
| 2026-09-02 | 534 |
| 2026-09-03 | 536 |
| 2026-09-09 | 536 |
| 2026-09-11 | 536 |
| 2026-09-13 | 537 |
| 2026-09-16 | 537 |
| 2026-09-19 | 535 |
| 2026-09-20 | 537 |
| 2026-09-21 | 538 |
| 2026-09-23 | 539 |
| 2026-09-24 | 539 |
| 2026-09-27 | 542 |
| 2026-09-29 | 543 |
| 2026-10-01 | 544 |
| 2026-10-02 | 545 |
| 2026-10-04 | 546 |
| 2026-10-05 | 545 |
| 2026-10-06 | 546 |
| 2026-10-09 | 547 |
| 2026-10-12 | 550 |
| 2026-10-14 | 551 |
| 2026-10-19 | 552 |
| 2026-10-21 | 553 |
| 2026-10-23 | 554 |
| 2026-10-25 | 554 |
| 2026-10-29 | 555 |
| 2026-10-30 | 554 |
| 2026-11-04 | 554 |
Today
— stars today
This week
— stars this week
This month
+9 stars this month
Momentum
0.0
growth rate 0.00%/day
# auto-harness > Give a coding agent a benchmark and an agent file. Let it iterate overnight. It reads failures, improves the system prompt and tools, gates every change against a self-maintained eval suite, and repeats. This repo is a simplified version of our auto-harness agent setup. We demonstrate our system on Tau3 benchmark tasks where the agent's score improves from 0.56 to 0.78 (~40% jump) while mining failures and auto maintaining live evals. If you are curious to learn more, read the full blog here - https://www.neosigma.ai/blog/self-improving-agentic-systems. The loop is defined in `PROGRAM.md`. The coding agent edits `agent/agent.py` to improve the agent and appends findings to `workspace/learnings.md` after each iteration. --- ## Supported Benchmarks | Benchmark | Domain | Tasks | Agent Interface | |-----------|--------|-------|-----------------| | **tau-bench** | Customer service (retail, airline, telecom) | retail: 114, airline: 50, telecom: 114 | Structured tool calls via tau2 | | **Terminal-Bench 2.0** | Real-world terminal tasks (coding, sysadmin, security) | 89 | Bash commands via Harbor containers | | **BIRD-Interact** | Interactive text-to-SQL (multi-turn, CRUD over Postgres) | lite: 300, full: 600 | Google ADK agent against a 3-service environment (user sim, DB env, system agent) | --- ## How it works ``` run benchmark → analyze → improve agent/agent.py → gate → record → update learnings → repeat ``` - **`agent/agent.py`** — the agent being optimized (copied from a benchmark-specific template) - **`agent/templates/`** — starting-point templates for each benchmark (read-only) - **`benchmark.py`** — runs your benchmark, returns per-task rewards - **`gating.py`** — three-step gate: eval suite + full test val_score + suite promotion - **`record.py`** — appends iteration results to `workspace/results.tsv` - **`prepare.py`** — sets up workspace, copies templates, runs baseline - **`program_templates/`** — benchmark-specific PROGRAM.md instructions - **`PROGRAM.md`** — instructions the coding agent follows (copied from template by prepare.py) --- ## First-run checklist Before running the initialization command in your chosen quick start: - [ ] **Choose one benchmark and install its prerequisites:** [Terminal-Bench 2.0](#quick-start-terminal-bench-20) needs the `harbor` CLI and a configured sandbox provider (or a running Docker daemon for `env_provider: "docker"`); [BIRD-Interact](#quick-start-bird-interact) needs Python 3.12+, Docker, `git-lfs`, and access to the ground-truth data described below; [tau-bench](#quick-start-tau-bench) uses Docker Compose and the image built by `docker compose build`. - [ ] **Configure the experiment:** copy [`experiment_config.yaml.template`](experiment_config.yaml.template) to `experiment_config.yaml`, uncomment only the benchmark section you intend to use, and review its models, dataset/domain, and `max_concurrency`. The template is entirely commented out, so copying it alone does not select a benchmark. - [ ] **Set credentials for the selected models and provider:** use [`.env.example`](.env.example) as a reference. For Terminal-Bench, supply the credential for the selected `env_provider`; Docker needs no sandbox-provider key, but model API credentials are still required. For BIRD-Interact, check both `agent_model` and `user_model`. - [ ] **Make credentials available to the process:** Docker Compose loads `.env` via `env_file`. When running `python prepare.py` directly, export the required variables in your shell first; `prepare.py` does not load `.env` automatically. For a shell-compatible `.env` you have reviewed, Bash/Zsh users can run `set -a; source .env; set +a` from the repository root. - [ ] **Plan for a baseline run:** on a fresh workspace, `python prepare.py` (or `docker compose run autoeval python prepare.py`) performs setup and launches the baseline benchmark. This makes model API calls and may incur sandbox-provider charges. Review your provider budge
Excerpt of 14,748 characters
Read on GitHub
8
2
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:6a1e7cea33ffd675, desc:agentic