Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Label, clean and enrich text datasets with LLMs.
| Date | Stars |
|---|---|
| 2026-07-31 | 2329 |
| 2026-08-04 | 2329 |
| 2026-08-06 | 2329 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<img alt="Refuel logo" src="https://raw.githubusercontent.com/refuel-ai/autolabel/main/docs/assets/Autolabel_blk_w_background.png">
<h4 align="center">
<a href="https://discord.gg/fweVnRx6CU">Discord</a> |
<a href="https://twitter.com/RefuelAI">Twitter</a> |
<a href="https://www.refuel.ai/">Website</a> |
<a href="https://www.refuel.ai/blog-posts/llm-labeling-technical-report">Benchmark</a>
</h4>
<div align="center" style="width:800px">
[](https://github.com/refuel-ai/autolabel/actions/workflows/black.yaml/badge.svg)   [](https://discord.gg/fweVnRx6CU) [](https://colab.research.google.com/drive/1t-9vNLkyoyySAG_0w3eR98biBOXlMO-E?usp=sharing)
</div>
## ⚡ Quick Install
`pip install refuel-autolabel`
## 📖 Documentation
[https://docs.refuel.ai/](https://docs.refuel.ai/)
## 🏷 What is Autolabel
Access to [large, clean and diverse](https://twitter.com/karpathy/status/1528443124577513472?lang=en) labeled datasets is a critical component for any machine learning effort to be successful. State-of-the-art LLMs like GPT-4 are able to [automatically label data](https://arxiv.org/abs/2303.15056) with [high accuracy](https://arxiv.org/abs/2303.16854), and at a fraction of the cost and time compared to manual labeling.
Autolabel is a Python library to label, clean and enrich text datasets with any Large Language Models (LLM) of your choice.
## 🌟 (New!) Benchmark models on Refuel's Benchmark
Check out our [technical report](https://refuel.ai/blog-posts/announcing-refuel-llm-2) to learn more about the performance of RefuelLLM-v2 on our benchmark. You can replicate the benchmark yourself by following the steps below
```python
cd autolabel/benchmark
curl https://autolabel-benchmarking.s3.us-west-2.amazonaws.com/data.zip -o data.zip
unzip data.zip
python benchmark.py --model $model --base_dir benchmark-results
python results.py --eval_dir benchmark-results
cat results.csv
```
You can benchmark the relevant model by replacing $model with the name of the model needed to be benchmarked. If it is an API hosted model like `gpt-3.5-turbo`, `gpt-4-1106-preview`, `claude-3-opus-20240229`, `gemini-1.5-pro-preview-0409` or some other Autolabel supported model, just write the name of the model. If the model to be benchmarked is a [vLLM supported model](https://docs.vllm.ai/en/latest/models/supported_models.html) then pass the local path or the huggingface path corresponding to the model. This will run the benchmark along with the _same_ prompts for all models.
The `results.csv` will contain a row with every model that was benchmarked as a row. Look at `benchmark/results.csv` for an example.
## 🚀 Getting started
Autolabel provides a simple 3-step process for labeling data:
1. Specify the labeling guidelines and LLM model to use in a JSON config.
2. Dry-run to make sure the final prompt looks good.
3. Kick off a labeling run for your dataset!
Let's imagine we are building an ML model to analyze sentiment analysis of movie review. We have a dataset of movie reviews that we'd like to get labeled first. For this case, here's what the example dataset and configs will look like:
```python
{
"task_name": "MovieSentimentReview",
"task_type": "classification",
"model": {
"provider": "openai",
"name": "gpt-3.5-turbo"
},
"dataset": {
"label_column": "label",
"delimiter": ","
},
"prompt": {
"task_guidelines": "You are an expert at analyzing the sentiment of movie reviews. Your job is to classify the provided movie review into one of the following labels: {labels}",
"labels": [
Excerpt of 8,868 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:99686e45c4b61824, topic:large-language-models, topic:llm
