Run ARC-AGI tasks against multiple model adapters (OpenAI, Anthropic, Gemini, Fireworks, Grok, OpenRouter, X.AI, custom etc.) with built-in rate limiting, retries, and scoring.
- Clone this repo:
git clone https://github.com/arcprize/arc-agi-benchmarking.git
cd arc-agi-benchmarking- Install (installs all adapters + SDKs):
This project is managed with uv. Install uv if you don't have it, then sync the locked dependencies into a project .venv:
uv syncAll commands below use uv run, which executes against the project's .venv without requiring manual activation. If you prefer pip, you can still pip install . into an environment of your choice and drop the uv run prefix.
- Single-task dry run (no API keys) with the local
random-baselineadapter:
uv run main.py \
--data_dir data/sample/tasks \
--config random-baseline \
--task_id 66e6c45b \
--save_submission_dir submissions/random-single- Run all bundled sample tasks with the random solver:
uv run cli/run_all.py \
--config random-baseline \
--data_dir data/sample/tasks \
--save_submission_dir submissions/random-baseline-sample- Score the outputs you just generated:
uv run src/arc_agi_benchmarking/scoring/scoring.py \
--task_dir data/sample/tasks \
--submission_dir submissions/random-baseline-sample \
--results_dir results/random-baseline-sampleIf using the random solver, expect all the attempts to be incorrect.
If you want to run real models, change the config and add the corresponding API keys or configure Anthropic WIF (see Data and Config sections below).
Rather than using the sample data in data/sample/tasks/, you can use the real ARC-AGI tasks from the following repositories:
- ARC-AGI-1 (2019):
git clone https://github.com/fchollet/ARC-AGI.git data/arc-agi - ARC-AGI-2 (2025):
git clone https://github.com/arcprize/ARC-AGI-2.git data/arc-agi
--data_dir: Folder containing ARC task.jsonfiles (e.g.,data/sample/tasks).--config: Model config name frommodels.yml. Used by both single-task and batch.--save_submission_dir: Where to write outputs. Use the same flag for single-task and batch (alias:--submissions-rootremains for backward compatibility). Recommended structure:<save_submission_dir>/<config>/<version>/<eval_type>/, ex:submissions/gpt-4o-2024-11-20/v1/public_eval/.--num_attempts: How many attempts per test pair (per task).--retry_attempts: Internal retries within an attempt if the provider call fails.--max-tasks-per-run: Maximum unsubmitted tasks scheduled by each config/dataset child. Existing-submission filtering happens before this cap is applied.--logs-base-dir: Root for per-task application and raw API JSONL logs (default:logs).--enable-metrics: Toggle metrics collection (saved inmetrics_output/).- Multi-config launcher-specific:
--configs: Space-separated model config names to run concurrently.--save_submission_root: Root directory; each child writes to<root>/<config>/<dataset-or-run-name>.--datasets: Space-separatedNAME=PATHdatasets; launches every config against every dataset.NAMEmay be a safe relative path such asv1/public_eval, producing nested output directories.--run_name: Single-dataset run path such asv2/public_eval; mutually exclusive with--datasets.--max-concurrency: Optional hard cap on in-flight ARC tasks per provider across every child process.
- Scoring-specific:
--submission_dir: Where your run wrote outputs--results_dirWhere to write aggregated metrics/results
For runs beyond the Quickstart:
-
Batch (recommended):
uv run cli/run_all.pywith your task list, model config, data dir, submission dir, and attempts/retries. Uses asyncio, best-effort outbound API-request pacing, and tenacity retries; outputs land in--save_submission_dir(e.g.,submissions/<config>/<version>/<eval_type>).run_allhandles one model config per invocation; userun_configs.pyfor multiple configs. Calls visible to the benchmark runner—including model, parsing, retry, and explicit background/batch control calls—consume rate-limit allowance; starting an ARC task does not. Configured rates are operational ballparks, not strict caps: SDK-managed retries, agent SDKs that hide internal model calls, and synchronous work that continues after a task timeout can produce additional provider traffic. -
Multiple configs: use
uv run cli/run_configs.py. It starts onerun_all.pyprocess per config concurrently, gives each process isolated submission/log directories, and prefixes console output with the config name:uv run cli/run_configs.py \ --configs \ gpt-5-2-2025-12-11-thinking-none \ gpt-5-2-2025-12-11-thinking-low \ gpt-5-2-2025-12-11-thinking-medium \ gpt-5-2-2025-12-11-thinking-high \ gpt-5-2-2025-12-11-thinking-xhigh \ --datasets \ v1/public_eval=data/v1/public_eval \ v1/semi_private_eval=data/v1/semi_private_eval \ v2/public_eval=data/v2/public_eval \ v2/semi_private_eval=data/v2/semi_private_eval \ --save_submission_root submissions \ --max-concurrency 8 \ --max-tasks-per-run 10This is the normal full benchmark layout: each config runs against
public_evalandsemi_private_evalfor both v1 and v2. The example starts twenty child runs, schedules at most 10 pending tasks in each child, and writes tosubmissions/<config>/<version>/<eval_type>, such assubmissions/gpt-5-2-2025-12-11-thinking-high/v1/public_eval. Provider rate limits are shared automatically across every child: all twenty OpenAI runs receive one-twentieth of the effective OpenAI rate while retaining its configured period. Configs using different providers are grouped and divided independently. The split controls the average request rate, so small simultaneous bursts can still occur across processes.--max-concurrency 8adds a shared cross-process semaphore that caps each provider at eight in-flight ARC tasks across all child runs.The original single-dataset form remains available with
--data_dir data/v2/public_eval --run_name v2/public_evalinstead of--datasets. -
Single task (debug):
uv run main.pywith a single--config,--task_id, and your data dir/save directory. Use--verbosewhen debug-level output is needed. See the CLI parameters section for flag details.
Raw API logging is automatic and uses the existing structured logging handlers. Raw events are appended to the same git-ignored per-task JSONL files as application logs. For run_configs.py, the default layout is logs/<config>/<dataset>/<task_id>.jsonl; use the existing --logs-base-dir option to move the entire combined log tree.
Each task file is append-only JSONL. Within a run, adapter-visible provider operations are sequential: a request_started event is followed by its request_succeeded or request_failed event. Every event carries a per-process run ID, so repeated invocations can share a task file without ambiguity. Task-level timeouts add a task_timed_out event, and an unmatched request_started event identifies a request interrupted before a terminal event could be recorded.
Application records use the existing level, logger, and message fields, while raw API records use the event field and retain serialized request arguments and provider responses. The provider client and invoked callable are not part of the recorded request payload. No redaction is applied, so logs can contain task data and model reasoning and should be treated as sensitive. SDK-internal retries and internal calls made by agent SDKs are not visible to this recorder; those adapters produce one logical invocation record.
Tests are run based on model configs. Model configs hold the configuration (max output tokens, temperature, pricing etc.) for each test.
Model configs live in src/arc_agi_benchmarking/models.yml. Example:
- name: "gpt-4o-2024-11-20" # config name you reference on the CLI; typically includes the reasoning level for clarity (e.g., "-basic", "-advanced")
model_name: "gpt-4o-2024-11-20" # provider’s actual model id
provider: "openai" # must match an adapter
api_key_env: "OPENAI_API_KEY" # required for API-key authentication
max_output_tokens: 4096 # optional; provider-specific
temperature: 0.0 # optional; provider-specific
pricing:
date: "2024-11-20"
input: 5.00 # USD per 1M input tokens
output: 15.00 # USD per 1M output tokens- Standard fields:
name,model_name,provider,pricing(input/outputper 1M tokens,datefor traceability). - API keys: configs using API-key authentication must set
api_key_envto the environment variable containing their API key. Adapters read the named variable without providing defaults.provider: randomneeds no credentials, andprovider: anthropiccan omitapi_key_envwhen using WIF. - Google configs accept either
provider: googleorprovider: gemini. Both use the Google adapter and normalize internally togemini, sharing its provider limits and timeout settings inprovider_config.yml. - Provider kwargs: any extra keys become
kwargsand are passed directly to the SDK (e.g.,temperature,max_output_tokens,stream, etc.). - Rate limits live in
provider_config.yml(rate,periodper provider). - Environment: set provider keys (e.g.,
OPENAI_API_KEY,ANTHROPIC_API_KEY,GEMINI_API_KEY,HUGGING_FACE_API_KEY). Copy.env.exampleto.envand fill in.
The native Anthropic adapter (provider: anthropic) supports API keys and AWS
Workload Identity Federation (WIF). Existing model configs work with either
method; no WIF options need to be added to the model YAML.
For AWS WIF, set these standard environment variables in the runner environment
or in .env for a local run with AWS credentials:
ANTHROPIC_FEDERATION_RULE_ID=fdrl_...
ANTHROPIC_ORGANIZATION_ID=00000000-0000-0000-0000-000000000000
ANTHROPIC_SERVICE_ACCOUNT_ID=svac_...
ANTHROPIC_WORKSPACE_ID=wrkspc_...All four values are required, including an explicit workspace ID. Setting any
of them selects WIF for every provider: anthropic model in that environment,
even if an API key is also present. Missing, empty, or whitespace-only values
fail validation without falling back to an API key. This applies to regular,
streaming, and batch requests through the native adapter. Other providers,
including OpenRouter and the separate Claude Agent SDK adapter, keep their
existing authentication.
To use legacy API-key authentication, unset all four WIF variables and remove
them from .env if present. Set the variable named by api_key_env in the model
config, normally ANTHROPIC_API_KEY; custom names such as
ANTHROPIC_API_KEY_NO_ZDR remain supported. WIF does not require api_key_env.
The runner follows Anthropic's AWS WIF example:
it uses the workload's AWS credentials to call STS in us-east-1, requesting a
900-second identity token with audience https://api.anthropic.com and signing
algorithm RS256. AWS outbound federation, sts:GetWebIdentityToken permission
on the workload role, and the matching Anthropic issuer, federation rule, and
service-account workspace membership must already be configured.
Preflight checks the environment configuration without contacting AWS or Anthropic. The first API request exchanges an AWS identity token for an Anthropic access token. The SDK caches and refreshes that token, requesting a fresh AWS identity token on each exchange. Authentication failures do not switch the run back to an API key.
-
Add a new model config: add an entry to
models.ymlwith an existing provider; then use--config <name>on the CLI -
If you're adding a new adapter:
- Create
src/arc_agi_benchmarking/adapters/<provider>.pyimplementingProviderAdapter - Export it from
src/arc_agi_benchmarking/adapters/__init__.py - Add a branch in
main.py(and any factories) so the provider name is recognized - Add a config entry in
models.ymlpointing toprovider: "<provider>" - [Optional] Add tests (adapters and parsing) to cover basic flows
- Create
To score a run you'll need 1) your test's submission directory and 2) the source taskset (which contains the solutions)
Score a run:
uv run src/arc_agi_benchmarking/scoring/scoring.py
--task_dir <data_dir>/data/evaluation
--submission_dir submissions/<config>
--results_dir results/<config>- Add new providers/models in
src/arc_agi_benchmarking/adaptersandmodels.yml. - Run tests:
uv run pytest. - Use the bundled sample task + submission for quick scoring checks.