Let autoharness run overnight and come back to an optimized agent harness, so your production agents never make mistakes again.
autoharness improves agent harnesses by proposing or applying prompt, config, middleware, and source changes, running evals, and keeping or discarding candidates based on benchmark results.
It is a control plane for an existing harness repo. You point it at a target root and a benchmark command; autoharness manages proposals, iterations, campaigns, and champion state under .autoharness/.
Fastest setup with Codex or Claude:
pipx install "git+https://github.com/kayba-ai/autoharness.git"cdinto your harness repo- open Codex or Claude Code in that repo
- tell the assistant:
Run autoharness guide --assistant codex --print-next-prompt, then use the generated onboarding packet to finish setup.
For Claude Code, swap --assistant codex for --assistant claude.
Else:
pipx install "git+https://github.com/kayba-ai/autoharness.git"
autoharness --helpIf you do not use pipx:
python3 -m pip install --user "git+https://github.com/kayba-ai/autoharness.git"guideinspects a repo, asks a few focused setup questions in a TTY, stays scriptable with flags in non-interactive use, writes a starterautoharness.yamlplus benchmark config, and runs a readiness check.doctorreruns config, generator, and benchmark validation when you want an explicit readiness gate.setupandinitremain available when you want to manage bootstrap explicitly.run-benchmarkexecutes one benchmark directly.generate-proposalpreviews one candidate change without running it.run-iterationoroptimizeexecutes one candidate or a resumable search loop.promoteorpromote-from-comparemoves a winner into champion state.
target root: the harness repo or deployment tree to editbenchmark config: the command or adapter config that scores candidatesworkspace: the long-lived optimization efforttrack: one comparable lane inside a workspacecampaign: a resumable search run over candidate proposals.autoharness/: persisted settings, proposals, records, iterations, and champions
- Adapters:
generic_command,pytest,harbor,tau2_bench,hal,car_bench - Proposal generators:
manual,failure_summary,local_template,local_command,openai_responses,codex_cli,claude_code - Extension model: Python plugins can add generators, preflight checks, and search strategies from
.autoharness/plugins/orAUTOHARNESS_PLUGIN_PATHS
Let autoharness generate a starter project config:
autoharness guideIn a TTY, guide asks a few setup questions. In scripts or CI, use flags like --non-interactive, --benchmark-command, --generator, and --autonomy.
If you want Codex or Claude to help you refine the setup, generate an assistant brief too:
autoharness guide --assistant codex --print-next-prompt
# or
autoharness guide --assistant claude --print-next-promptThis writes autoharness.codex.md or autoharness.claude.md plus a structured autoharness.onboarding.json handoff next to autoharness.yaml, then prints a ready-to-paste assistant prompt. Assistant wrapper prompts live under contrib/agents/.
guide ends with a doctor pass. Run autoharness doctor again later if you want an explicit re-check or a repeated benchmark probe.
On a fresh install, guide prefers a local assistant backend when codex or claude is installed, otherwise uses openai_responses when OpenAI credentials are configured, and falls back to failure_summary only when no model-backed generator is available.
Then run the benchmark directly:
autoharness run-benchmarkIf autoharness.yaml is present, autoharness will auto-bootstrap missing settings and workspace state on this common path. setup and init are still available when you want explicit control.
Generate a proposal against a target harness root:
autoharness generate-proposalIf you switch the project config to openai_responses, export an API key first:
export OPENAI_API_KEY=...Run the outer loop:
autoharness optimize
autoharness reportExample from one tau2 airline benchmark study. Relative deltas are measured against the baseline harness on the same workload. Results depend on the benchmark, harness, and evaluation setup, and some intervention combinations can regress.
- Background campaign workers plus queue and worker-state inspection
- Root-level memory, transfer suggestions, and portfolio scheduling
- Retention policies, pruning, and portable report and bundle exports
- Event logs, inspection commands, and operational reporting surfaces
- Python plugin hooks for generators, preflight checks, and search strategies
Tip
Want deeper analysis or a custom optimization workflow? Kayba offers managed harness optimization and agent-improvement support tailored to your stack.
Star this repo if you find it useful!
Built with ❤️ by Kayba and the open-source community.