Top AI Repos β open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Benchmark for automated failure attributions in agentic systems (π ICML 2025 Spotlight)
| Date | Stars |
|---|---|
| 2026-07-31 | 381 |
| 2026-08-06 | 381 |
Today
β stars today
This week
β stars this week
This month
β stars this month
Momentum
0.0
growth rate 0.00%/day
<p align="center">
<img src="assets/title.png" alt="overview" width="550">
</p>
<p align="center">
<a href="https://img.shields.io/badge/License-MIT-blue?style=flat-square">License</a> β’
<a href="https://arxiv.org/pdf/2505.00212">Paper</a> β’
<a href="https://huggingface.co/datasets/Kevin355/Who_and_When">Dataset</a> β’
<a href="https://mp.weixin.qq.com/s/etweZ18bWWVz7thglMdGXA">Synced (ζΊε¨δΉεΏ)</a> β’
<a href="https://mp.weixin.qq.com/s/dIhJj6Z6WmBAobM3Ay1YjQ">AIEra (ζ°ζΊε
)</a> β’
<a href="https://mp.weixin.qq.com/s/6bmoyg2Pho28rYzgjdbFYA">QbitAI (ιεδ½)</a> β’
<a href="https://discord.gg/Z5frZMzQ">Discord</a> β’
<a href="https://ag2ai.github.io/Agents_Failure_Attribution/">Project Page</a>
<a href="hhttps://github.com/ag2ai/Agents_Failure_Attribution/tree/main/awesome-failure-attributions">Awsome Failure Attributions</a>
</p>
> **The first and the best benchmark debugging failures from agentic trajectories.**
## π§ Overview
<div align="center">
<img src="assets/overview.png" alt="overview" width="500">
</div>
This repository provides the implementation of ICML 2025 spotlight paper "Which Agent Causes Task Failures and When?", which introduces the task of automated failure attribution in LLM-based multi-agent systems. Given a failed task, the goal failure attribution is to automatically identify the agent and step responsible for the failure.
Automated failure-attributions offers several key advantages:
- **Reduces manual debugging effort:** Automates the labor-intensive process of inspecting failure logs and tracing errors.
- **Accelerates system development:** Speeds up the iteration cycle by quickly identifying faulty agents and critical mistakes.
- **Enables intermediate feedback for agent self-improvement:** Pinpointing decisive errors provides actionable signals for agentic systems' self-correction or can serve as rewards in reinforcement learning.
## π§ Who&When: #1 Benchmark for MAS **automated failure attribution.**
- **184** annotated failure tasks collected from
- **Algorithm-generated agentic systems** built using [CaptainAgent](https://docs.ag2.ai/latest/docs/use-cases/notebooks/notebooks/agentchat_captainagent/),
- **Hand-crafted systems** such as [Magnetic-One](https://www.microsoft.com/en-us/research/articles/magentic-one-a-generalist-multi-agent-system-for-solving-complex-tasks/).
- **Fine-grained annotations** for each failure, including:
- The failure-responsible agent (who failed),
- The decisive error step (when the critical error occurred),
- A natural language explanation of the failure.
The dataset covers a wide range of realistic multi-agent scenarios based on queries from [GAIA](https://huggingface.co/gaia-benchmark) and [AssistantBench](https://assistantbench.github.io/). It serves as a foundational resource for developing and evaluating methods that aim to automatically pinpoint the causes of failures in complex agentic systems. We follow the following guide to annotate these failure logs. More information could be found in the paper.
> [!IMPORTANT]
> Check out the dataset on [Hugging Face π€](https://huggingface.co/datasets/Kevin355/Who_and_When).
## π‘ Evaluations
### Requirements
To install requirements:
```
pip install -r requirements.txt
```
### Inference
Please ensure that you specify the AutoFA method (--method) in the corresponding sections of the code before executing it.
- Models
We support the following models:
| Model Name | Command-line Argument |
|---------------------------|------------------------|
| GPT-4o | `--model gpt-4o` |
| GPT-4 | `--model gpt4` |
| GPT-4o-mini | `--model gpt4o-mini` |
| Llama-3.1-8B-Instruct | `--model llama-8b` |
| Llama-3.1-70B-Instruct | `--model llama-70b` |
| Qwen2.5-7B-Instruct | `--model qwen-7b` |
| Qwen2.5-72B-Instruct | `--model qwen-72b` |
#### Run
```
python inference.py --methExcerpt of 6,089 characters
Read on GitHubWould you bet a product on this? Bounded 0β100 and slow moving.
matched fp:b7fcd00e4c54f6bd, llm:topics: agent, autogen, llm, multi-agent, multiagent-systems; description: Benchmark for automated failure attributions in agentic systems (ICML 2025 Spotlight).
matched fp:b7fcd00e4c54f6bd, llm:topics: agent, autogen, llm, multi-agent, multiagent-systems; description: Benchmark for automated failure attributions in agentic systems (ICML 2025 Spotlight).