Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Mistral: A strong, northwesterly wind: Framework for transparent and accessible large-scale language model training, built with Hugging Face 🤗 Transformers.
| Date | Stars |
|---|---|
| 2026-07-31 | 581 |
| 2026-08-03 | 581 |
| 2026-08-06 | 581 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<div align="center"><img src="https://github.com/stanford-crfm/mistral/raw/main/docs/mistral_components.png" height="300px"/></div>
# Mistral
> *Mistral*: A strong and cool northwesterly wind that builds as it moves, bringing good health and clear skies.
[](https://opensource.org/licenses/Apache-2.0)
[](https://github.com/pre-commit/pre-commit)
A framework for transparent and accessible large-scale language model training, built with [Hugging Face 🤗](https://huggingface.co/) . Includes tools
and helpful scripts for incorporating new pre-training datasets, various schemes for single node and distributed training - including on
cloud providers like GCP, and importantly, scripts for evaluation.
Visit our [Read the Docs](https://nlp.stanford.edu/mistral) for the full documentation.
A Propulsion Endeavor 🚀
---
## Quickstart
### Installation
Mistral has been tested with Python 3.8.12, PyTorch 1.11.0 (compiled with CUDA 11.3), CUDA 11.3, NCCL 2.10, Transformers 4.17.0, and DeepSpeed 0.6.0.
The environment can be easily built with the following commands:
```bash
conda create -n mistral python=3.8.12 pytorch=1.11.0 torchdata cudatoolkit=11.3 -c pytorch
conda activate mistral
pip install -r setup/pip-requirements.txt
```
A `.yaml` export of a tested environment is provided at `environments/environment-gpu.yaml`.
Environments and non-Python dependencies can be managed with conda, and Python dependencies can be managed with pip (note: conda was used for the PyTorch install to get the version compiled with CUDA 11.3).
### Training GPT-2 Micro
#### Prerequisites
First, make sure to update `conf/mistral-micro.yaml` with the directories you want to store the Hugging Face
cache and model runs.
```
# Artifacts & Caching
artifacts:
cache_dir: /path/to/artifacts
run_dir: /path/to/runs
```
Next, make sure that `/path/to/mistral` is on your `PYTHONPATH`.
#### Single-node single-GPU training
For single-node single-gpu training, run:
```bash
conda activate mistral
cd mistral
CUDA_VISIBLE_DEVICES=0 python train.py --config conf/mistral-micro.yaml --nnodes 1 --nproc_per_node 1 --training_arguments.fp16 true --training_arguments.per_device_train_batch_size 2 --run_id tutorial-gpt2-micro
```
#### Multi-node multi-GPU training with DeepSpeed
Modify `/job/hostfile` in the following way:
```
<Hostname of first machine> slots=<Number of GPUs>
<Hostname of second machine> slots=<Number of GPUs>
...
<Hostname of the nth machine> slots=<Number of GPUs>
```
Below is an example hostfile where we train on `machine1` and `machine2` with 8 GPUs each:
```
machine1 slots=8
machine2 slots=8
```
To start distributed training, run:
```bash
conda activate mistral
cd mistral
deepspeed --num_gpus 8 --num_nodes 2 --master_addr machine1 train.py --config conf/tutorial-gpt2-micro.yaml --nnodes 2 --nproc_per_node 8 --training_arguments.fp16 true --training_arguments.per_device_train_batch_size 4 --training_arguments.deepspeed conf/deepspeed/z2-small-conf.json --run_id tutorial-gpt2-micro-multi-node
```
Note: You may need to adjust your batch size depending on the capacity of your GPUs.
If you are interested in training a model on Google Cloud, check out our
[Google Cloud + Kubernetes Tutorial](https://nlp.stanford.edu/mistral/tutorials/gcp_plus_kubernetes.html).
### Using the model
Model checkpoints will be stored in the directory specified by the `artifacts.run_dir`. An example checkpoint might be
in `/path/to/runs/tutorial-gpt2-micro/checkpoint-1000`.
Mistral stores model checkpoints in the Hugging Face format, so models can be loaded and used in the same manner as if
one had trained the model with Hugging Face.
For instance, to generate text with 🤗 Transformers:
```python
from transformers import GPT2LMHeadModel, GPT2Tokenizer
tokenizer = GPT2Tokenizer.fromExcerpt of 7,532 characters
Read on GitHub
393
Sidd Karamcheti · Georgia Tech · United States
151
David Hall · @open-athena / @stanford-crfm · United States
83
Tianyi · United States
49
Laurel Orr · United States
33
Tony Lee · Stanford University · United States
23
Karan Goel · United States
12
Avanika Narayan
12
Yifan Mai
2
Aaron Gokaslan · @facebookresearch · United States
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:3084df0ddae4a1dc, llm:description: 'Framework for transparent and accessible large-scale language model training, built with Hugging Face Transformers.'
matched fp:3084df0ddae4a1dc, llm:description: 'Framework for transparent and accessible large-scale language model training, built with Hugging Face Transformers.'
matched fp:3084df0ddae4a1dc, llm:description: 'Framework for transparent and accessible large-scale language model training, built with Hugging Face Transformers.'