Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Testing baseline LLMs performance across various models
| Date | Stars |
|---|---|
| 2026-07-31 | 356 |
| 2026-08-03 | 356 |
| 2026-08-06 | 356 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Testing systems with ARC-AGI Run ARC-AGI tasks against multiple model adapters (OpenAI, Anthropic, Gemini, Fireworks, Grok, OpenRouter, X.AI, custom etc.) with built-in rate limiting, retries, and scoring. ## Quickstart 0) Clone this repo: ```bash git clone https://github.com/arcprize/arc-agi-benchmarking.git cd arc-agi-benchmarking ``` 1) Install (installs all adapters + SDKs): This project is managed with [uv](https://docs.astral.sh/uv/). Install uv if you don't have it, then sync the locked dependencies into a project `.venv`: ```bash uv sync ``` All commands below use `uv run`, which executes against the project's `.venv` without requiring manual activation. If you prefer pip, you can still `pip install .` into an environment of your choice and drop the `uv run` prefix. 2) Single-task dry run (no API keys) with the local `random-baseline` adapter: ```bash uv run main.py \ --data_dir data/sample/tasks \ --config random-baseline \ --task_id 66e6c45b \ --save_submission_dir submissions/random-single \ --log-level INFO ``` 3) Run all bundled sample tasks with the random solver: ```bash uv run cli/run_all.py \ --config random-baseline \ --data_dir data/sample/tasks \ --save_submission_dir submissions/random-baseline-sample \ --log-level INFO ``` 4) Score the outputs you just generated: ```bash uv run src/arc_agi_benchmarking/scoring/scoring.py \ --task_dir data/sample/tasks \ --submission_dir submissions/random-baseline-sample \ --results_dir results/random-baseline-sample ``` If using the random solver, expect all the attempts to be incorrect. If you want to run real models, change the `config` and add the corresponding API keys (see Data and Config sections below). ## Data Rather than using the sample data in `data/sample/tasks/`, you can use the real ARC-AGI tasks from the following repositories: * ARC-AGI-1 (2019): `git clone https://github.com/fchollet/ARC-AGI.git data/arc-agi` * ARC-AGI-2 (2025): `git clone https://github.com/arcprize/ARC-AGI-2.git data/arc-agi` ## CLI parameters - `--data_dir`: Folder containing ARC task `.json` files (e.g., `data/sample/tasks`). - `--config`: Model config name from `models.yml`. Used by both single-task and batch. - `--save_submission_dir`: Where to write outputs. Use the same flag for single-task and batch (alias: `--submissions-root` remains for backward compatibility). Recommended structure: `<save_submission_dir>/<config>/<version>/<eval_type>/`, ex: `submissions/gpt-4o-2024-11-20/v1/public_eval/`. - `--num_attempts`: How many attempts per test pair (per task). - `--retry_attempts`: Internal retries within an attempt if the provider call fails. - `--max-tasks-per-run`: Maximum unsubmitted tasks scheduled by each config/dataset child. Existing-submission filtering happens before this cap is applied. - `--log-level`: `DEBUG|INFO|WARNING|ERROR|CRITICAL|NONE`. - `--enable-metrics`: Toggle metrics collection (saved in `metrics_output/`). - Multi-config launcher-specific: - `--configs`: Space-separated model config names to run concurrently. - `--save_submission_root`: Root directory; each child writes to `<root>/<config>/<dataset-or-run-name>`. - `--datasets`: Space-separated `NAME=PATH` datasets; launches every config against every dataset. `NAME` may be a safe relative path such as `v1/public_eval`, producing nested output directories. - `--run_name`: Single-dataset run path such as `v2/public_eval`; mutually exclusive with `--datasets`. - `--max-concurrency`: Optional hard cap on in-flight ARC tasks per provider across every child process. - Scoring-specific: - `--submission_dir`: Where your run wrote outputs - `--results_dir` Where to write aggregated metrics/results ## Running models For runs beyond the Quickstart: - Batch (recommended): `uv run cli/run_all.py` with your task list, model config, data dir, submission dir, attempts/retries, and log level. Uses asyncio, provider rate limiting, and tenacity retries; outputs land in
Excerpt of 8,873 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:67a18299dac11f40, llm:description: 'Testing baseline LLMs performance across various models' (repo description)
matched fp:67a18299dac11f40, llm:description: 'Testing baseline LLMs performance across various models' (repo description)
matched fp:67a18299dac11f40, llm:description: 'Testing baseline LLMs performance across various models' (repo description)