Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A repository for research on medium sized language models.
| Date | Stars |
|---|---|
| 2026-07-31 | 537 |
| 2026-08-04 | 537 |
| 2026-08-22 | 537 |
| 2026-08-26 | 536 |
| 2026-09-20 | 536 |
Today
— stars today
This week
— stars this week
This month
-1 stars this month
Momentum
0.0
growth rate 0.00%/day
# OpenLM
OpenLM is a minimal but performative language modeling (LM) repository, aimed to facilitate research on medium sized LMs. We have verified the performance of OpenLM up to 7B parameters and 256 GPUs.
In contrast with other repositories such as Megatron, we depend only on PyTorch, XFormers, or Triton for our core modeling code.
# Contents
- [Release Notes](#release-notes)
- [Quickstart](#quickstart)
- [Setup](#setup)
- [Process training data](#process-training-data)
- [Run training](#run-training)
- [Evaluate Model](#evaluate-model)
- [Generate Text](#generate-text)
- [Pretrained Models](#pretrained-models)
- [Team and Acknowledgements](#team-and-acknowledgements)
# Release Notes
- 09/26/23: Public release and featured on [Laion Blog](https://laion.ai/blog/open-lm/)
- 08/18/23: Updated README.md
# Quickstart
Here we'll go over a basic example where we start from a fresh install, download and preprocess some training data, and train a model.
## Setup
We require python >=3.9, and a current installation of pyTorch, as well as several other packages. The full list of requirements is contained in `requirements.txt` and can be installed in your python enviornment via
```>>> pip install -r requirements.txt```
Next, to access `open_lm` everywhere in your virtual environment, install it using pip (from within the top level github repo)
```>>> pip install --editable . ```
Some considerations:
- We like [WandB](https://wandb.ai/) and [tensorboard](https://www.tensorflow.org/tensorboard) for logging. We specify how to use these during training below.
## Process Training Data
Next you must specify a collection of tokenized data. For the purposes of this example, we will use a recent dump of english Wikipedia, available on HuggingFace. To download this locally, we've included a script located at [open_lm/datapreprocess/wiki_download.py](open_lm/datapreprocess/wiki_download.py). All you have to do is specify an output directory for where the raw data should be stored:
```
python open_lm/datapreprocess/wiki_download.py --output-dir path/to/raw_data
```
Next we process our training data by running it through a BPE tokenizer and chunk it into chunks of appropriate length. By default we use the tokenizer attached with [GPT-NeoX-20B](https://github.com/EleutherAI/gpt-neox). To do this, use the script `datapreprocess/make_2048.py`:
```
>>> python open_lm/datapreprocess/make_2048.py \
--input-files path_to_raw_data/*.jsonl
--output-dir preproc_data
--num-workers 32
--num-consumers 1
```
Where `input-files` passes all of its (possibly many) arguments through the python `glob` module, allowing for wildcards. Optionally, data can be stored in S3 by setting the environment variables: `S3_BASE`, and passing the flag `--upload-to-s3` to the script. This saves sharded data to the given bucket with prefix of `S3_BASE`. E.g.
```
>>> export S3_BASE=preproc_data-v1/
>>> python open_lm/datapreprocess/make2048.py --upload-to-s3 ... # same arguments as before
```
## Run Training
Tokenized data can now be passed to the main training script, `open_lm/main.py`. Distributed computatation is handled via `torchrun`, and hyperparameters are specified by a variety of keyword arguments. We highlight several of the most important ones here:
- `train-data`: location of the sharded tokenized training data. If locally generated and stored, this will point to a directory containing files like `preproc_data/2048-v1/0/XXXXXXX.tar`. Data are processed using the [webdataset](https://github.com/webdataset/webdataset) package where wildcards are supported like `preproc_data/2048-v1/0/{0000000..0000099}.tar` to select the first 100 .tar files.
- `model`: Which model to use. See the table below to see valid options and parameter sizes for each.
- `train-num-samples`: how many samples to use from the specified training dataset
- `name`: name of this particular training run for logging purposes
- `report-to`: if present, caExcerpt of 19,106 characters
Read on GitHub
Achal Dave · Toyota Research Institute
71
36
34
Mitchell Wortsman · University of Washington
33
30
12
9
7
6
5
4
3
3
HuggingFace · Belgium
2
2
2
2
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:69596caffc243cea, llm:description: 'A repository for research on medium sized language models.' language: Python
matched fp:69596caffc243cea, llm:description: 'A repository for research on medium sized language models.' language: Python
matched fp:69596caffc243cea, llm:description: 'A repository for research on medium sized language models.' language: Python
matched fp:69596caffc243cea, llm:description: 'A repository for research on medium sized language models.' language: Python