Run ARC-AGI tasks against multiple model adapters (OpenAI, Anthropic, Gemini, Fireworks, Grok, OpenRouter, X.AI, custom etc.) with built-in rate limiting, retries, and scoring.
- Clone this repo:
git clone https://github.com/arcprize/arc-agi-benchmarking.git
cd arc-agi-benchmarking- Install (installs all adapters + SDKs):
This project is managed with uv. Install uv if you don't have it, then sync the locked dependencies into a project .venv:
uv syncAll commands below use uv run, which executes against the project's .venv without requiring manual activation. If you prefer pip, you can still pip install . into an environment of your choice and drop the uv run prefix.
- Single-task dry run (no API keys) with the local
random-baselineadapter:
uv run main.py \
--data_dir data/sample/tasks \
--config random-baseline \
--task_id 66e6c45b \
--save_submission_dir submissions/random-single- Run all bundled sample tasks with the random solver:
uv run cli/run_all.py \
--config random-baseline \
--data_dir data/sample/tasks \
--save_submission_dir submissions/random-baseline-sample- Score the outputs you just generated:
uv run src/arc_agi_benchmarking/scoring/scoring.py \
--task_dir data/sample/tasks \
--submission_dir submissions/random-baseline-sample \
--results_dir results/random-baseline-sampleIf using the random solver, expect all the attempts to be incorrect.
If you want to run real models, change the config and add the corresponding API keys (see Data and Config sections below).
Rather than using the sample data in data/sample/tasks/, you can use the real ARC-AGI tasks from the following repositories:
- ARC-AGI-1 (2019):
git clone https://github.com/fchollet/ARC-AGI.git data/arc-agi - ARC-AGI-2 (2025):
git clone https://github.com/arcprize/ARC-AGI-2.git data/arc-agi
--data_dir: Folder containing ARC task.jsonfiles (e.g.,data/sample/tasks).--config: Model config name frommodels.yml. Used by both single-task and batch.--save_submission_dir: Where to write outputs. Use the same flag for single-task and batch (alias:--submissions-rootremains for backward compatibility). Recommended structure:<save_submission_dir>/<config>/<version>/<eval_type>/, ex:submissions/gpt-4o-2024-11-20/v1/public_eval/.--num_attempts: How many attempts per test pair (per task).--retry_attempts: Internal retries within an attempt if the provider call fails.--max-tasks-per-run: Maximum unsubmitted tasks scheduled by each config/dataset child. Existing-submission filtering happens before this cap is applied.--logs-base-dir: Root for per-task application and raw API JSONL logs (default:logs).--enable-metrics: Toggle metrics collection (saved inmetrics_output/).- Multi-config launcher-specific:
--configs: Space-separated model config names to run concurrently.--save_submission_root: Root directory; each child writes to<root>/<config>/<dataset-or-run-name>.--datasets: Space-separatedNAME=PATHdatasets; launches every config against every dataset.NAMEmay be a safe relative path such asv1/public_eval, producing nested output directories.--run_name: Single-dataset run path such asv2/public_eval; mutually exclusive with--datasets.--max-concurrency: Optional hard cap on in-flight ARC tasks per provider across every child process.
- Scoring-specific:
--submission_dir: Where your run wrote outputs--results_dirWhere to write aggregated metrics/results
For runs beyond the Quickstart:
-
Batch (recommended):
uv run cli/run_all.pywith your task list, model config, data dir, submission dir, and attempts/retries. Uses asyncio, best-effort outbound API-request pacing, and tenacity retries; outputs land in--save_submission_dir(e.g.,submissions/<config>/<version>/<eval_type>).run_allhandles one model config per invocation; userun_configs.pyfor multiple configs. Calls visible to the benchmark runner—including model, parsing, retry, and explicit background/batch control calls—consume rate-limit allowance; starting an ARC task does not. Configured rates are operational ballparks, not strict caps: SDK-managed retries, agent SDKs that hide internal model calls, and synchronous work that continues after a task timeout can produce additional provider traffic. -
Multiple configs: use
uv run cli/run_configs.py. It starts onerun_all.pyprocess per config concurrently, gives each process isolated submission/log directories, and prefixes console output with the config name:uv run cli/run_configs.py \ --configs \ gpt-5-2-2025-12-11-thinking-none \ gpt-5-2-2025-12-11-thinking-low \ gpt-5-2-2025-12-11-thinking-medium \ gpt-5-2-2025-12-11-thinking-high \ gpt-5-2-2025-12-11-thinking-xhigh \ --datasets \ v1/public_eval=data/v1/public_eval \ v1/semi_private_eval=data/v1/semi_private_eval \ v2/public_eval=data/v2/public_eval \ v2/semi_private_eval=data/v2/semi_private_eval \ --save_submission_root submissions \ --max-concurrency 8 \ --max-tasks-per-run 10This is the normal full benchmark layout: each config runs against
public_evalandsemi_private_evalfor both v1 and v2. The example starts twenty child runs, schedules at most 10 pending tasks in each child, and writes tosubmissions/<config>/<version>/<eval_type>, such assubmissions/gpt-5-2-2025-12-11-thinking-high/v1/public_eval. Provider rate limits are shared automatically across every child: all twenty OpenAI runs receive one-twentieth of the effective OpenAI rate while retaining its configured period. Configs using different providers are grouped and divided independently. The split controls the average request rate, so small simultaneous bursts can still occur across processes.--max-concurrency 8adds a shared cross-process semaphore that caps each provider at eight in-flight ARC tasks across all child runs.The original single-dataset form remains available with
--data_dir data/v2/public_eval --run_name v2/public_evalinstead of--datasets. -
Single task (debug):
uv run main.pywith a single--config,--task_id, and your data dir/save directory. Use--verbosewhen debug-level output is needed. See the CLI parameters section for flag details.
Raw API logging is automatic and uses the existing structured logging handlers. Raw events are appended to the same git-ignored per-task JSONL files as application logs. For run_configs.py, the default layout is logs/<config>/<dataset>/<task_id>.jsonl; use the existing --logs-base-dir option to move the entire combined log tree.
Each task file is append-only JSONL. Within a run, adapter-visible provider operations are sequential: a request_started event is followed by its request_succeeded or request_failed event. Every event carries a per-process run ID, so repeated invocations can share a task file without ambiguity. Task-level timeouts add a task_timed_out event, and an unmatched request_started event identifies a request interrupted before a terminal event could be recorded.
Application records use the existing level, logger, and message fields, while raw API records use the event field and retain serialized request arguments and provider responses. The provider client and invoked callable are not part of the recorded request payload. No redaction is applied, so logs can contain task data and model reasoning and should be treated as sensitive. SDK-internal retries and internal calls made by agent SDKs are not visible to this recorder; those adapters produce one logical invocation record.
Tests are run based on model configs. Model configs hold the configuration (max output tokens, temperature, pricing etc.) for each test.
Model configs live in src/arc_agi_benchmarking/models.yml. Example:
- name: "gpt-4o-2024-11-20" # config name you reference on the CLI; typically includes the reasoning level for clarity (e.g., "-basic", "-advanced")
model_name: "gpt-4o-2024-11-20" # provider’s actual model id
provider: "openai" # must match an adapter
api_key_env: "OPENAI_API_KEY" # required unless provider is random
max_output_tokens: 4096 # optional; provider-specific
temperature: 0.0 # optional; provider-specific
pricing:
date: "2024-11-20"
input: 5.00 # USD per 1M input tokens
output: 15.00 # USD per 1M output tokens- Standard fields:
name,model_name,provider,pricing(input/outputper 1M tokens,datefor traceability). - API keys: every config except
provider: randommust setapi_key_envto the environment variable containing its API key. Every adapter reads the named variable; adapters do not provide defaults. - Provider kwargs: any extra keys become
kwargsand are passed directly to the SDK (e.g.,temperature,max_output_tokens,stream, etc.). - Rate limits live in
provider_config.yml(rate,periodper provider). - Environment: set provider keys (e.g.,
OPENAI_API_KEY,ANTHROPIC_API_KEY,GEMINI_API_KEY,HUGGING_FACE_API_KEY). Copy.env.exampleto.envand fill in.
-
Add a new model config: add an entry to
models.ymlwith an existing provider; then use--config <name>on the CLI -
If you're adding a new adapter:
- Create
src/arc_agi_benchmarking/adapters/<provider>.pyimplementingProviderAdapter - Export it from
src/arc_agi_benchmarking/adapters/__init__.py - Add a branch in
main.py(and any factories) so the provider name is recognized - Add a config entry in
models.ymlpointing toprovider: "<provider>" - [Optional] Add tests (adapters and parsing) to cover basic flows
- Create
To score a run you'll need 1) your test's submission directory and 2) the source taskset (which contains the solutions)
Score a run:
uv run src/arc_agi_benchmarking/scoring/scoring.py
--task_dir <data_dir>/data/evaluation
--submission_dir submissions/<config>
--results_dir results/<config>- Add new providers/models in
src/arc_agi_benchmarking/adaptersandmodels.yml. - Run tests:
uv run pytest. - Use the bundled sample task + submission for quick scoring checks.