Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
| Date | Stars |
|---|---|
| 2026-07-31 | 3241 |
| 2026-08-01 | 3242 |
| 2026-08-02 | 3243 |
| 2026-08-03 | 3247 |
| 2026-08-04 | 3247 |
| 2026-08-05 | 3250 |
| 2026-08-06 | 3250 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
5.0
growth rate 0.00%/day
# DataTrove
DataTrove is a library to process, filter and deduplicate text data at a very large scale. It provides a set of prebuilt commonly used processing blocks with a framework to easily add custom functionality.
DataTrove processing pipelines are platform-agnostic, running out of the box locally or on a slurm cluster. Its (relatively) low memory usage and multiple step design makes it ideal for large workloads, such as to process an LLM's training data.
Local, remote and other file systems are supported through [fsspec](https://filesystem-spec.readthedocs.io/en/latest/).
## Table of contents
<!-- toc -->
- [Installation](#installation)
- [Quickstart examples](#quickstart-examples)
- [Terminology](#terminology)
- [Pipeline](#pipeline)
* [DataTrove Document](#datatrove-document)
* [Types of pipeline blocks](#types-of-pipeline-blocks)
* [Full pipeline](#full-pipeline)
- [Executors](#executors)
* [LocalPipelineExecutor](#localpipelineexecutor)
* [SlurmPipelineExecutor](#slurmpipelineexecutor)
* [RayPipelineExecutor](#raypipelineexecutor)
* [JobsPipelineExecutor](#jobspipelineexecutor)
- [Logging](#logging)
- [DataFolder / paths](#datafolder--paths)
- [Practical guides](#practical-guides)
* [Reading data](#reading-data)
* [Synthetic data generation](#synthetic-data-generation)
+ [Custom rollouts](#custom-rollouts)
+ [Ready-to-use generation script](#ready-to-use-generation-script)
+ [Advanced configuration](#advanced-configuration)
+ [Progress monitoring](#progress-monitoring)
+ [Benchmarking](#benchmarking)
* [Extracting text](#extracting-text)
* [Filtering data](#filtering-data)
* [Saving data](#saving-data)
* [Deduplicating data](#deduplicating-data)
* [Summary Statistics](#summary-statistics)
* [Custom blocks](#custom-blocks)
+ [Simple data](#simple-data)
+ [Custom function](#custom-function)
+ [Custom block](#custom-block)
- [Contributing](#contributing)
- [Citation](#citation)
<!-- tocstop -->
## Installation
Requires Python 3.10+.
```bash
uv sync
```
Available flavours (combine them by repeating `--extra`, e.g. `uv sync --extra processing --extra s3`):
- `all` installs everything: `uv sync --extra all`
- `io` dependencies to read `warc/arc/wet` files and arrow/parquet/[Optimized-parquet](https://huggingface.co/docs/hub/en/datasets-libraries#optimized-parquet-files) formats: `uv sync --extra io`
- `processing` dependencies for text extraction, filtering and tokenization: `uv sync --extra processing`
- `s3` s3 support: `uv sync --extra s3`
- `cli` for command line tools: `uv sync --extra cli`
- `ray` for distributed compute engine: `uv sync --extra ray`
- `inference` for LLM inference pipelines: `uv sync --extra inference`
- `decont` for decontamination with lighteval: `uv sync --extra decont`
- `multilingual` for multilingual text processing: `uv sync --extra multilingual`
## Quickstart examples
You can check the following [examples](examples):
- [fineweb.py](examples/fineweb.py) full reproduction of the [FineWeb dataset](https://huggingface.co/datasets/HuggingFaceFW/fineweb)
- [process_common_crawl_dump.py](examples/process_common_crawl_dump.py) full pipeline to read commoncrawl warc files, extract their text content, filters and save the resulting data to s3. Runs on slurm
- [tokenize_c4.py](examples/tokenize_c4.py) reads data directly from huggingface's hub to tokenize the english portion of the C4 dataset using the `gpt2` tokenizer
- [estimate_tokens.py](examples/estimate_tokens.py) estimate total token counts for large HF datasets — needed to set the correct `SamplerFilter` rate when creating a random shuffled subsample (e.g. 100B tokens from a multi-trillion-token dataset). Streams a small sample per dataset, converges on the average tokens/doc, and multiplies by the total row count.
- [smol_data.py](examples/smol_data.py) builds ~100B token subsets, 50-30-20 mixtures, and shuffled variants for several large Hugging Face datasets
- [minhash_dExcerpt of 39,525 characters
Read on GitHub
Guilherme Penedo · HuggingFace · France
334
Hynek Kydlíček
72
Joel Niklaus · @huggingface · Switzerland
67
43
@huggingface · Switzerland
21
Thomas Wolf · @huggingface
17
sungjun lee
13
Mario Šaško · Croatia
11
Anton Lozhkov · Hugging Face · France
10
8
Quentin Lhoest · Hugging Face · France
4
Pauline Bailly-Masson
3
Daniel van Strien · Hugging Face · United Kingdom
3
Bram Vanroy · @CCL-KULeuven @instituutnederlandsetaal · Belgium
3
3
2
Ukraine
2
Nelson Liu · @stanfordnlp · United States
2
shizhediao · Thinking Machines Lab · United States
2
Luc Georges · @huggingface · France
2
Would you bet a product on this? Bounded 0–100 and slow moving.
Not classified yet. Classification runs as part of npm run ingest.