Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Pytorch implementation of DoReMi, a method for optimizing the data mixture weights in language modeling datasets
| Date | Stars |
|---|---|
| 2026-07-31 | 357 |
| 2026-08-06 | 357 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# DoReMi🎶: Domain Reweighting with Minimax Optimization [](https://opensource.org/licenses/MIT) [](https://arxiv.org/abs/2305.10429) PyTorch implementation of DoReMi, an algorithm for optimizing data mixtures for language modeling datasets. Modern large language models are trained on many domains (web, books, arXiv, etc.), but how much of each domain to train on is unclear, especially since these models are going to be used for a variety of downstream tasks (no particular target distribution to optimize for). DoReMi tunes the data mixture to be robust to the target distribution using Distributionally Robust Optimization (DRO). DoReMi trains a small proxy model using DRO, which dynamically upweights or downweights domains based on the proxy model's excess loss compared to a pretrained reference model. The reference model provides an estimate of the best loss achievable to avoid being to pessimistic for high entropy / hard domains. The tuned data mixture can then be used to train a much larger model more efficiently. In the paper, a 280M proxy model can improve the training of a 8B parameter model (30x larger), allowing it to achieve the baseline 8B performance 2.6x faster. The graphic below provides an overview of DoReMi. Check out the [paper](https://arxiv.org/abs/2305.10429) for more details. As a black box, this codebase outputs optimized domain weights given a text dataset. Some other useful components: fast, resumable dataloader with domain-level weighted sampling, simple downstream eval harness, and HuggingFace Trainer + FlashAttention2 integration.  ## Getting started To get started, please clone the repo and install it: ``` git clone [email protected]:/sangmichaelxie/doremi.git pip install -e doremi cd doremi && bash scripts/setup_flash.sh ``` The compilation in `scripts/setup_flash.sh` may take a significant amount of time (hours). All code should be run from the outermost `doremi` directory. Before you start, write paths to your cache directories, data directories, etc in a `constants.sh` file in the outer directory of this repo. You can also place any conda or virtualenv activation commands here. Here's an example of the contents of a `constants.sh` file (provided as a file called `sample_constants.sh`): ``` #!/bin/bash CACHE=/path/to/cache DOREMI_DIR=/path/to/this/repo PILE_DIR=/path/to/pile PREPROCESSED_PILE_DIR=/path/to/preprocessed # will be created by scripts/run_preprocess_pile.sh MODEL_OUTPUT_DIR=/path/to/model_output_dir WANDB_API_KEY=key # Weights and Biases key for logging PARTITION=partition # for slurm mkdir -p ${CACHE} mkdir -p ${MODEL_OUTPUT_DIR} source ${DOREMI_DIR}/venv/bin/activate # if you installed doremi in venv ``` Here is how to run the sample script for data preprocessing on The Pile, which separates the Pile data into domains and tokenizes it: ``` bash scripts/run_preprocess_pile.sh ``` Here is a sample script to run 120M baseline, proxy, and main models (all 3 steps in the DoReMi pipeline), tested on one node with 8 A100 GPUs. This is a small version of the Pile experiments in the paper. The script will automatically run perplexity and few-shot evaluation: ``` bash scripts/run_pile.sh ``` These scripts run for 200k steps, following the paper. The DoReMi run outputs domain weights in the `configs` directory with filename `<RUN_NAME>.json`. ## Running DoReMi on your own dataset To run DoReMi on your own dataset, provide preprocessed (tokenized) data in the following format: ``` top_level/ domain_name_1/ files... domain_name_2/ files... ... ``` where each inner directory (e.g., `domain_name_1`) can be loaded via HuggingFace's `load_from_disk` method. If your data is in a different format, you can add a custom data loading function in `doremi/dataloader.py`. You will also need to writ
Excerpt of 8,661 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:6d4479d6878462e6, topic:large-language-models
matched fp:6d4479d6878462e6, topic:nlp