Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Continuous Integration for LLM powered applications
| Date | Stars |
|---|---|
| 2026-07-31 | 262 |
| 2026-08-02 | 262 |
| 2026-08-06 | 262 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# 🏋️♂️ BenchLLM 🏋️♀️
🦾 Continuous Integration for LLM powered applications 🦙🦅🤖
[](https://github.com/v7labs/BenchLLM/stargazers)
[](https://twitter.com/V7Labs)
[](https://discord.gg/x7ExfHb3bG)
[**BenchLLM**](https://benchllm.com/) is a Python-based open-source library that streamlines the testing of Large Language Models (LLMs) and AI-powered applications. It measures the accuracy of your model, agents, or chains by validating responses on any number of tests via LLMs.
BenchLLM is actively used at [V7](https://www.v7labs.com) for improving our LLM applications and is now Open Sourced under MIT License to share with the wider community
## 💡 Get help on [Discord](https://discord.gg/x7ExfHb3bG) or [Tweet at us](https://twitter.com/V7Labs)
<hr/>
Use BenchLLM to:
- Test the responses of your LLM across any number of prompts.
- Continuous integration for chains like [Langchain](https://github.com/hwchase17/langchain), agents like [AutoGPT](https://github.com/Significant-Gravitas/Auto-GPT), or LLM models like [Llama](https://github.com/facebookresearch/llama) or GPT-4.
- Eliminate flaky chains and create confidence in your code.
- Spot inaccurate responses and hallucinations in your application at every version.
<hr/>
> ⚠️ **NOTE:** BenchLLM is in the early stage of development and will be subject to rapid changes.
>
> For bug reporting, feature requests, or contributions, please open an issue or submit a pull request (PR) on our GitHub page.
## 🧪 BenchLLM Testing Methodology
BenchLLM implements a distinct two-step methodology for validating your machine learning models:
1. **Testing**: This stage involves running your code against any number of expected responses and capturing the predictions produced by your model without immediate judgment or comparison.
2. **Evaluation**: The recorded predictions are compared against the expected output using LLMs to verify factual similarity (or optionally manually). Detailed comparison reports, including pass/fail status and other metrics, are generated.
This methodical separation offers a comprehensive view of your model's performance and allows for better control and refinement of each step.
## 🚀 Install
To install BenchLLM we use pip
```
pip install benchllm
```
## 💻 Usage
Start by importing the library and use the @benchllm.test decorator to mark the function you'd like to test:
```python
import benchllm
# Your custom model implementation
def run_my_model(input):
# Your model's logic goes here.
return some_result
@benchllm.test(suite="/path/to/test/suite") # If the tests are in the same directory, just use @benchllm.test.
def invoke_model(input: str):
return run_my_model(input)
```
Next, prepare your tests. These are YAML/JSON files structured as follows:
```yml
input: What's 1+1? Be very terse, only numeric output
expected:
- 2
- 2.0
```
In the above example, the `input` is the query or instruction that your model will process, and `expected` contains the potential responses that your model should return. It's important to note that `input` can be a simple `str` or a more complex nested dictionary; BenchLLM will extract the type of the `input` argument in the Python code and load the `input` field from the YAML file accordingly.
By default, BenchLLM uses OpenAI's GPT-3 model for the `semantic` evaluator. This requires setting the `OPENAI_API_KEY` environment variable. If you do not want to use this default evaluator, you can specify an alternative one (discussed in further detail below):
```bash
export OPENAI_API_KEY='your-api-key'
```
Replace 'your-api-key' with your actual OpenAI API key.
To initiate testing, use the `bench run` command:
```bash
$ bench run
```
By default, the bench run commaExcerpt of 10,018 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:0319b0bd52d04827, topic:llm, topic:gpt