Top AI Repos β open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Data processing for and with foundation models! π π π½ β‘οΈ β‘οΈπΈ πΉ π·
| Date | Stars |
|---|---|
| 2026-07-24 | 6775 |
| 2026-07-25 | 6777 |
| 2026-07-28 | 6777 |
| 2026-07-30 | 6777 |
| 2026-07-31 | 6813 |
| 2026-08-06 | 6813 |
Today
β stars today
This week
+36 stars this week
This month
β stars this month
Momentum
71.0
growth rate 0.53%/day
# Data-Juicer: The Data Operating System for the Foundation Model Era
<p align="center">
<a href="https://pypi.org/project/py-data-juicer"><img src="https://img.shields.io/pypi/v/py-data-juicer?logo=pypi&color=026cad" alt="PyPI"></a>
<a href="https://pepy.tech/projects/py-data-juicer"><img src="https://static.pepy.tech/personalized-badge/py-data-juicer?period=total&units=INTERNATIONAL_SYSTEM&left_color=grey&right_color=green&left_text=downloads" alt="Downloads"></a>
<a href="https://hub.docker.com/r/datajuicer/data-juicer"><img src="https://img.shields.io/docker/v/datajuicer/data-juicer?logo=docker&label=Docker&color=498bdf" alt="Docker"></a>
<br>
<a href="https://datajuicer.github.io/data-juicer/"><img src="https://img.shields.io/badge/π_Docs-Website-026cad" alt="Docs"></a>
<a href="https://datajuicer.github.io/data-juicer/en/main/docs/Operators.html"><img src="https://img.shields.io/badge/π§©_Operators-200+-blue" alt="Operators"></a>
<a href="https://github.com/datajuicer/data-juicer-hub"><img src="https://img.shields.io/badge/π³_Recipes-50+-brightgreen" alt="Recipes"></a>
<br>
<a href="https://datajuicer.github.io/data-juicer/zh_CN/main/index_ZH.html"><img src="https://img.shields.io/badge/π¨π³_ζζ‘£-δΈ»ι‘΅-red" alt="Chinese"></a>
<a href="https://arxiv.org/abs/2501.14755"><img src="https://img.shields.io/badge/NeurIPS'25_Spotlight-2.0-B31B1B?logo=arxiv" alt="Paper"></a>
<a href="https://github.com/datajuicer/data-juicer">
<img src="https://img.shields.io/endpoint?style=flat&url=https%3A%2F%2Fgist.githubusercontent.com%2FHYLcool%2Ff856b14416f08f73d05d32fd992a9c29%2Fraw%2Ftotal_cov.json&label=coverage&logo=codecov&color=4c1" alt="Coverage">
</a>
</p>
<p align="center">
<b>Multimodal | Cloud-Native | AI-Ready | Large-Scale </b>
</p>
Data-Juicer (DJ) transforms raw data chaos into AI-ready intelligence. It treats data processing as *composable infrastructure*βproviding modular building blocks to clean, synthesize, and analyze data across the entire AI lifecycle, unlocking latent value in every byte.
Whether you're deduplicating web-scale pre-training corpora, curating agent interaction traces, or preparing domain-specific RAG indices, DJ scales seamlessly from your laptop to thousand-node clustersβno glue code required.
> **Alibaba Cloud PAI** has deeply integrated Data-Juicer into its data processing products. See **[Quickly submit a DataJuicer job](https://www.alibabacloud.com/help/en/pai/user-guide/quickly-submit-a-datajuicer-task)**.
---
## π Quick Start
**Zero-install exploration**:
- [JupyterLab Playground with Tutorials](http://8.138.149.181/)
- [Ask DJ Copilot](https://datajuicer.github.io/data-juicer/en/main/docs_index.html)
**Install & run**:
```bash
uv pip install py-data-juicer
dj-process --config demos/process_simple/process.yaml
```
**Or compose in Python**:
```python
from data_juicer.core.data import NestedDataset
from data_juicer.ops.filter import TextLengthFilter
from data_juicer.ops.mapper import WhitespaceNormalizationMapper
ds = NestedDataset.from_dict({
"text": ["Short", "This passes the filter.", "Text with spaces"]
})
res_ds = ds.process([
TextLengthFilter(min_len=10),
WhitespaceNormalizationMapper()
])
for s in res_ds:
print(s)
```
---
## β¨ Why Data-Juicer?
### 1. Modular & Extensible Architecture
- **200+ operators** spanning text, image, audio, video, and multimodal data
- **Recipe-first**: Reproducible YAML pipelines you can version, share, and fork like code
- **Composable**: Drop in a single operator, chain complex workflows, or orchestrate full pipelines
- **Hot-reload**: Iterate on operators without pipeline restarts
### 2. Full-Spectrum Data Intelligence
- **Foundation Models**: Pre-training, fine-tuning, RL, and evaluation-grade curation
- **Agent Systems**: Clean tool traces, structure context, de-identification, and quality gating
- **RAG & Analytics**: Extraction, normalization, semantic chunking, deduplication, and data profiling
##Excerpt of 19,353 characters
Read on GitHub
Yilun Huang Β· Peking University Β· China
156
53
52
48
Alibaba Β· China
42
Ce Ge (ζη) Β· Bytedance Seed Β· China
35
30
Alibaba, Tongyi Lab Β· China
17
14
13
12
Xuchen Pan Β· Alibaba Group
12
Du Bin Β· Microsoft
11
co63oc
10
10
7
John Giorgi Β· @abridgeai
6
6
chenyushuo
5
4
Would you bet a product on this? Bounded 0β100 and slow moving.
matched fp:d3b9c05b92a75d83, topic:large-language-models, topic:llm, topic:foundation-models
matched fp:d3b9c05b92a75d83, topic:data-pipeline, topic:synthetic-data, readme:chunking