Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Tool-calling quality benchmark for LLM serving stacks. 80+ deterministic scenarios testing multi-turn orchestration, safety boundaries, and structured output. Supports vLLM, SGLang, and llama.cpp.
| Date | Stars |
|---|---|
| 2026-07-31 | 259 |
| 2026-08-06 | 280 |
Today
+21 stars today
This week
— stars this week
This month
— stars this month
Momentum
144.0
growth rate 0.00%/day
# tool-eval-bench A **tool-calling quality benchmark** for evaluating LLM tool-use in agentic workflows across open-weight model serving stacks (**vLLM**, **LiteLLM**, **llama.cpp**). Also includes pluggable accuracy benchmarks (**GSM8K**, **MMLU**, **IFEval**) via the same OpenAI-compatible endpoints. Inspired by [ToolCall-15](https://github.com/stevibe/ToolCall-15), this tool runs **69 deterministic scenarios** (+ 15 opt-in Hard Mode) through OpenAI-compatible `/chat/completions` endpoints, scores each result as **pass**, **partial**, or **fail**, and produces detailed trace reports. Mock tool responses include realistic payload noise (extra metadata, timestamps, nested objects) to test whether models can extract relevant fields from noisy API responses. It also includes an integrated **throughput benchmark** (llama-bench style) for measuring prefill and token generation speed.  > **Scope.** tool-eval-bench measures *tool-calling quality* — whether a model picks the right tool, passes the right parameters, chains tools correctly, and handles errors and safety boundaries. It is not a full agentic system benchmark (see [Related Work](#related-work) for how it compares to BFCL, PinchBench, and Claw-Eval). ## What It Measures ### Tool-Call Quality (69 scenarios across 15 categories) | Category | Scenarios | What It Tests | |---|---|---| | **A — Tool Selection** | TC-01 – TC-03 | Picking the right tool from 12 options | | **B — Parameter Precision** | TC-04 – TC-06 | Getting parameters right (units, dates, multi-value) | | **C — Multi-Step Chains** | TC-07 – TC-09, TC-61 | Chained reasoning, data threading, parallel calls, async polling | | **D — Restraint & Refusal** | TC-10 – TC-12 | Knowing when NOT to call tools | | **E — Error Recovery** | TC-13 – TC-15 | Handling failures and preserving data integrity | | **F — Localization** | TC-16 – TC-18 | German language, timezone awareness, translate+forward | | **G — Structured Reasoning** | TC-19 – TC-21 | Message routing, data extraction, constraint validation | | **H — Instruction Following** | TC-22 – TC-24, TC-44 – TC-45 | Output format, tool prohibition, multi-constraint, tool_choice compliance | | **I — Context & State** | TC-25 – TC-27, TC-46 – TC-50, TC-62 – TC-63 | Cross-reference, state consistency, multi-turn correction, 6-turn chains, constraint accumulation | | **J — Code Patterns** | TC-28 – TC-30 | Read-before-write, explain vs execute, chained conditional | | **K — Safety & Boundaries** | TC-31 – TC-36, TC-41 – TC-43, TC-57 – TC-60 | Ambiguity, prompt injection (file/search/system/sleeper), authority escalation, contradictory params, parameter validation | | **L — Toolset Scale** | TC-37 – TC-40 | Tool selection from 52 tools, multi-step in crowded namespace, restraint under abundance | | **M — Autonomous Planning** | TC-51 – TC-53 | Goal decomposition, open-ended research, conditional workflows | | **N — Creative Composition** | TC-54 – TC-56 | Cross-tool synthesis, data pipelines, notification workflows | | **O — Structured Output** | TC-64 – TC-69 | JSON schema compliance, tool→schema chaining, nested schemas, enum constraints, violation resistance | | **P — Hard Mode** _(opt-in)_ | TC-70 – TC-84 | Ceiling-breaking adversarial, stateful, recovery, relevance, and parallel-call scenarios | ### Throughput Performance (optional) llama-bench-style prefill (pp) and token generation (tg) measurement via streaming, with configurable context depth and concurrency sweeps. ### Pluggable Accuracy Benchmarks External benchmarks run through the same `BenchmarkPlugin` interface and share the backend adapter, progress display, and reporting infrastructure. No `tools` support required — only `/v1/chat/completions`. | Benchmark | Flag | Questions | What It Measures | |---|---|---|---| | **GSM8K** | `--gsm8k` | 1,319 | Grade school math reasoning (8-shot chain-of-thought) | | **MMLU** | `--mmlu` | 14,04
Excerpt of 46,827 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:1adea0a74ac68f42, desc:tool calling