Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models.
| Date | Stars |
|---|---|
| 2026-07-31 | 2866 |
| 2026-08-03 | 2866 |
| 2026-08-06 | 2873 |
Today
+7 stars today
This week
— stars this week
This month
— stars this month
Momentum
33.0
growth rate 0.00%/day
# Holistic Evaluation of Language Models (HELM)
[comment]: <> (When using the img tag, which allows us to specify size, src has to be a URL.)
<img src="https://github.com/stanford-crfm/helm/raw/v0.5.4/helm-frontend/src/assets/helm-logo.png" alt="HELM logo" width="480"/>
<a href="https://github.com/stanford-crfm/helm">
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/stanford-crfm/helm">
</a>
<a href="https://github.com/stanford-crfm/helm/graphs/contributors">
<img alt="GitHub contributors" src="https://img.shields.io/github/contributors/stanford-crfm/helm">
</a>
<a href="https://github.com/stanford-crfm/helm/actions/workflows/test-python.yml?query=branch%3Amain">
<img alt="GitHub Actions Workflow Status" src="https://img.shields.io/github/actions/workflow/status/stanford-crfm/helm/test-python.yml">
</a>
<a href="https://crfm-helm.readthedocs.io/en/latest/">
<img alt="Documentation Status" src="https://readthedocs.org/projects/helm/badge/?version=latest">
</a>
<a href="https://github.com/stanford-crfm/helm/blob/main/LICENSE">
<img alt="License" src="https://img.shields.io/github/license/stanford-crfm/helm?color=blue" />
</a>
<a href="https://pypi.org/project/crfm-helm/">
<img alt="PyPI" src="https://img.shields.io/pypi/v/crfm-helm?color=blue" />
</a>
_Note: HELM entered maintenance mode on June 1, 2026. For more information, refer to the [Maintenance Mode Policy](https://crfm-helm.readthedocs.io/en/latest/maintenance_mode/)._
**Holistic Evaluation of Language Models (HELM)** is an open source Python framework created by the [Center for Research on Foundation Models (CRFM) at Stanford](https://crfm.stanford.edu/) for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models. This framework includes the following features:
- Datasets and benchmarks in a standardized format (e.g. MMLU-Pro, GPQA, IFEval, WildBench)
- Models from various providers accessible through a unified interface (e.g. OpenAI models, Anthropic Claude, Google Gemini)
- Metrics for measuring various aspects beyond accuracy (e.g. efficiency, bias, toxicity)
- Web UI for inspecting individual prompts and responses
- Web leaderboard for comparing results across models and benchmarks
## Documentation
Please refer to [the documentation on Read the Docs](https://crfm-helm.readthedocs.io/) for instructions on how to install and run HELM.
## Quick Start
<!--quick-start-begin-->
Install the package from PyPI:
```sh
pip install crfm-helm
```
Run the following in your shell:
```sh
# Run benchmark
helm-run --run-entries mmlu:subject=philosophy,model=openai/gpt2 --suite my-suite --max-eval-instances 10
# Summarize benchmark results
helm-summarize --suite my-suite
# Start a web server to display benchmark results
helm-server --suite my-suite
```
Then go to http://localhost:8000/ in your browser.
<!--quick-start-end-->
## Leaderboards
We maintain official leaderboards with results from evaluating recent models on notable benchmarks using this framework. Our current flagship leaderboards are:
- [HELM Capabilities](https://crfm.stanford.edu/helm/capabilities/latest/)
- [HELM Safety](https://crfm.stanford.edu/helm/safety/latest/)
- [Holistic Evaluation of Vision-Language Models (VHELM)](https://crfm.stanford.edu/helm/vhelm/latest/)
We also maintain leaderboards for a diverse range of domains (e.g. medicine, finance) and aspects (e.g. multi-linguality, world knowledge, regulation compliance). Refer to the [HELM website](https://crfm.stanford.edu/helm/) for a full list of leaderboards.
## Papers
The HELM framework was used in the following papers for evaluating models.
- Holistic Evaluation of Language Models - [paper](https://openreview.net/forum?id=iO4LZibEqW), [leaderboard](https://crfm.stanford.edu/helm/classic/latest/)
- Holistic Evaluation of Vision-Language Models (VHELM) - [paper](https://arxiv.org/abs/2410.07112), [leaderboard](htExcerpt of 7,194 characters
Read on GitHub
Tony Lee · Stanford University · United States
1.7k
Yifan Mai
1.5k
Percy Liang · Stanford University · United States
401
Dilara Soylu
330
Rishi Bommasani · Stanford University
250
Dimitris Tsipras
234
187
143
132
90
86
70
70
Farzaan Kaiyom · United States
70
55
51
50
43
42
41
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:2c2711686998f687, desc:reproducible