Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Benchmarks, leaderboards, LLM-as-judge harnesses, testing and red-teaming.
Signals: benchmark, evaluation, llm-evaluation, leaderboard, llm-eval, red-teaming, testing, model-evaluation
949 results
A modular framework for benchmarking LLMs and agentic strategies on security challenges across HackTheBox, TryHackMe, PortSwigger Labs, Cybench, picoCTF and more.
GPT-Fathom is an open-source and reproducible LLM evaluation suite, benchmarking 10+ leading open-source and closed-source LLMs as well as OpenAI's earlier models on 20+ curated benchmarks under aligned settings.
Codes for our paper "ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate"
Aidan Bench attempts to measure <big_model_smell> in LLMs.
(NeurIPS D&B 2024) STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases
Moonshot - A simple and modular tool to evaluate and red-team any LLM application.
An Analytical Evaluation Board of Multi-turn LLM Agents [NeurIPS 2024 Oral]
[TACL'26] VoiceBench: Benchmarking LLM-Based Voice Assistants
Build, evaluate, understand, and fix LLM-based apps
Run evaluation on LLMs using human-eval benchmark
A comprehensive repository of reasoning tasks for LLMs (and beyond)
Testing baseline LLMs performance across various models
[ICLR 2025 Spotlight] An open-sourced LLM judge for evaluating LLM-generated answers.
A comprehensive set of LLM benchmark scores and provider prices. (deprecated, read more in README)
No description
EvaLearn is a pioneering benchmark designed to evaluate large language models (LLMs) on their learning capability and efficiency in challenging tasks.
[ACL'24 Outstanding] Data and code for L-Eval, a comprehensive long context language models evaluation benchmark
[ACL2024] T-Eval: Evaluating Tool Utilization Capability of Large Language Models Step by Step
Codes and Data for Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
Repo for paper *Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges*
Code for the paper "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples"
Code and data for "Measuring and Narrowing the Compositionality Gap in Language Models"
Code and data for "Lost in the Middle: How Language Models Use Long Contexts"
Code and results accompanying the paper "Refusal in Language Models Is Mediated by a Single Direction".
24,545 repositories in the index in total.
