Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Pretrained language model with 100B parameters
| Date | Stars |
|---|---|
| 2026-07-31 | 3757 |
| 2026-08-03 | 3757 |
| 2026-08-06 | 3756 |
Today
-1 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# YaLM 100B
**YaLM 100B** is a GPT-like neural network for generating and processing text. It can be used freely by developers and researchers from all over the world.
The model leverages 100 billion parameters. It took 65 days to train the model on a cluster of 800 A100 graphics cards and 1.7 TB of online texts, books, and countless other sources in both English and Russian.
Training details and best practices on acceleration and stabilizations can be found on **[Medium](https://medium.com/p/d1df53d0e9a6)** (English) and **[Habr](https://habr.com/ru/company/yandex/blog/672396/)** (Russian) articles.
We used DeepSpeed to train the model and drew inspiration from Megatron-LM example. However, the code in this repo is not the same code that was used to train the model. Rather it is stock example from DeepSpeed repo with minimal changes needed to infer our model.
## Setup
Make sure to have 200GB of free disk space before downloading weights. The model *(code is based on [microsoft/DeepSpeedExamples/Megatron-LM-v1.1.5-ZeRO3](https://github.com/microsoft/DeepSpeedExamples/tree/068e6561188e9192104e014f70fbe25224b5eb62/Megatron-LM-v1.1.5-ZeRO3))* is supposed to run on multiple GPUs with tensor parallelism. It was tested on 4 (A100 80g) and 8 (V100 32g) GPUs, but is able to work with different configurations with ≈200GB of GPU memory in total which divide weight dimensions correctly (e.g. 16, 64, 128).
### Downloading checkpoint
* Run `bash download/download.sh` to download model weights and vocabulary.
* By default, weights will be downloaded to `./yalm100b_checkpoint/weights/`, and vocabulary will be downloaded to `./yalm100b_checkpoint/vocab/`.
* As another option, you can [clone our HF repo](https://huggingface.co/yandex/yalm-100b/tree/main) and [pull the checkpoint](https://huggingface.co/yandex/yalm-100b/tree/main/yalm100b_checkpoint).
### Docker
* We [published](https://hub.docker.com/r/yandex/yalm-cuda11-ds) image on Docker Hub, it can be pulled with `docker/pull.sh`. It is compatible with A100 and V100.
* Alternatively, you can build docker image from source using `docker/build.sh` (which will just build docker image from `docker/Dockerfile`).
* To run container, use `docker/run.sh` *(volumes, name and other parameters can be changed)*.
## Usage
You can start with the following scripts:
* `examples/generate_interactive.sh`: interactive generation from command line, the simplest way to try the model.
* `examples/generate_conditional_sampling.sh`: conditional generation with sampling strategy. Top-p is used by default, feel free to change temperature or use top-k. Input is jsonlines (example: `examples/example_cond_input.json`), output will be the same jsonlines with generated text field added to each line.
* `examples/generate_conditional_greedy.sh`: same as previous, but generation is greedy. Suitable for solving problems with few-shot.
* `examples/generate_unconditional.sh`: unconditional generation. No input is used, output will be jsonlines.
## License
The model is published under the Apache 2.0 license that permits both research and commercial use, Megatron-LM is licensed under the [Megatron-LM license](megatron_lm/LICENSE).
## Training details
### Dataset composition
Dataset used for the training of YaLM-100B is comprised of the following parts (rough percentages are measured in tokens seen by the model):
* **25%** [The Pile](https://pile.eleuther.ai/) — open English dataset by Eleuther AI team
* **75%** Texts in Russian collected by our team (percentages of the whole dataset are given)
* 49% Russian web pages from Yandex Search index filtered from ~100Tb to ~1Tb by the following heuristics:
1. LSH Deduplication — clusters of similar texts were truncated to just one text each
2. Length filtration — too short or too long texts or texts with too few natural sentences were discarded.
3. Entropy filtration — texts with too high or too low entropy were discarded
4. Domain filtrExcerpt of 5,260 characters
Read on GitHub
7
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:f8d940cffda28a89, llm:Repository description: 'Pretrained language model with 100B parameters' (Yandex YaLM-100B).
matched fp:f8d940cffda28a89, llm:Repository description: 'Pretrained language model with 100B parameters' (Yandex YaLM-100B).