Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Benchmarks, leaderboards, LLM-as-judge harnesses, testing and red-teaming.
Signals: benchmark, evaluation, llm-evaluation, leaderboard, llm-eval, red-teaming, testing, model-evaluation
949 results
Open source AI Agent evaluation framework for web tasks 🐒🍌
AdaRubric: Adaptive Dynamic Rubric Evaluator for Agent Trajectories
A generative AI-powered framework for testing virtual agents.
Benchmark for automated failure attributions in agentic systems (🏆 ICML 2025 Spotlight)
MCPMark is a comprehensive, stress-testing MCP benchmark designed to evaluate model and agent capabilities in real-world MCP use.
A large-scale benchmark for evaluating vision-language-action models, embodied agents, and vision-language models
[ICLR 2026] The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution
VisualWebArena is a benchmark for multimodal agents.
Repo-level benchmark for real-world Code Agents: from repo understanding → env setup → incremental dev/bug-fixing → task delivery, with cost-aware α metric.
Evaluation harness for Apodex-1.0 on public deep-research benchmarks.
A multi-player tournament benchmark that tests LLMs in social reasoning, strategy, and deception. Players engage in public and private conversations, form alliances, and vote to eliminate each other
Official github repo for SafetyBench, a comprehensive benchmark to evaluate LLMs' safety. [ACL 2024]
Web-Bench is a benchmark designed to evaluate the performance of LLMs in actual Web development.
BABILong is a benchmark for LLM evaluation using the needle-in-a-haystack approach.
A simple toolkit for benchmarking LLMs on mathematical reasoning tasks. 🧮✨
Python SDK for running evaluations on LLM generated responses
Enterprise RAG Challenge to test accuracy of different LLM-driven assistants
Benchmarking LLMs with Challenging Tasks from Real Users
The lastest paper about detection of LLM-generated text and code
A multi-programming language benchmark for LLMs
Benchmarking LLMs via Uncertainty Quantification
A joint community effort to create one central leaderboard for LLMs.
PromtFuzz is an automated tool that generates high-quality fuzz drivers for libraries via a fuzz loop constructed on mutating LLMs' prompts.
Hallucination Detector is a free and open-source tool that helps you verify the accuracy of your LLM generated content instantly.
24,545 repositories in the index in total.
