Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
| Date | Stars |
|---|---|
| 2026-07-31 | 255 |
| 2026-08-01 | 255 |
| 2026-08-02 | 256 |
| 2026-08-06 | 256 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# FineWeb 2
FineWeb 2 is the second iteration of the popular 🍷 FineWeb dataset, bringing high quality pretraining data to over 1000 🗣️ languages. For the actual data, please see the HuggingFace repository [https://huggingface.co/datasets/HuggingFaceFW/fineweb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2).
> [!NOTE]
> We have officially released our 📝 **research paper**: https://huggingface.co/papers/2506.20920
## Data Pipeline 🚀
The processing pipeline had to be heavily adapted for a multilingual setting. As each language has its own peculiarities, we **individually tuned each filter**, defining different thresholds and stopwords for each language. 📊
These thresholds and stopwords are available in `/configs/{iso3_lang}_{script}.yml`.
The starting point for our dataset was the non-English data (< 0.65 score in English) we obtained when processing the original FineWeb. 🌐
To this data, we applied the following processing steps:
1. Additional Language Identification and filtering 🔍
2. Deduplication per language 🔄
3. Filtering per language 🧹
4. PII Anonymization and fixes 🎭
### Language Identification 🌍
Performed using [GlotLID](https://github.com/cisnlp/GlotLID), which not only covers a wider variety of languages (2000+ available labels) compared to fasttext176 (used in the original FineWeb), as it also identifies the script used in each document. 📜
For each language, we defined *different minimum language classifier confidence scores* to keep a document.
### Deduplication 🗃️
Unlike in FineWeb, where data was deduplicated per CommonCrawl snapshot, in FineWeb 2, **data is deduplicated per language globally**. However, following our deduplication findings in the original FineWeb, while we remove all except one document from each duplicate cluster, we save the size of this cluster in the kept document's metadata, saved in `minhash_cluster_size`.
This allows us to "re-hydrate" the dataset: by upsampling documents based on their cluster size, we see clear performance improvements for some languages, particularly high resource ones. 📈
We did not extensively explore different upsampling weights, but observed promising results with the following weights:
- documents with no duplicates: 1 time
- documents from a cluster of size N=2 or N=3: document will be N times in the final dataset
- documents from a cluster of size N=4: document will be 3 times in the final dataset
- documents from a cluster of size N>=5 and N<100: document will be 5 times in the final dataset
- documents from a cluster of size N>=100: document will be 8 times in the final dataset
- documents from a cluster of size N>=1000: document will be 1 time in the final dataset (the assumption here is that very large clusters are lower quality)
Example "re-hydration" block:
```python
class Rehydrater(PipelineStep):
def run(self, data: DocumentsPipeline, rank: int = 0, world_size: int = 1) -> DocumentsPipeline:
import bisect
upsampling_weights = {1: 1, 2: 2, 3: 3, 5: 5, 100: 8, 1000: 1}
# Sorted keys
limits = sorted(upsampling_weights.keys())
for doc in data:
upsampling_weight = upsampling_weights[
limits[bisect.bisect_right(limits, doc.metadata["minhash_cluster_size"]) - 1]]
# repeat each document upsampling_weight times
for _ in range(upsampling_weight):
yield doc
```
### Data Filtering 🧹
We mostly keep the original FineWeb set of filters, and do not create new filters targeting individual languages. As such, we had to extensively ablate on different processes of adapting the English filters to all the languages we supported. 🔍
Based on the results of our experiments, we also disabled/changed global values of some specific filters:
- For FineWeb filters, we removed `short_line_thr` and changed `char_dup_ratio` from 0.01 to 0.1.
- Repetition filter: disabled paragraph related filteExcerpt of 6,983 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:6f164a5847a27dac, llm:Repository: huggingface/fineweb-2 — name suggests 'fineweb' related to fine-tuning / web UI; no topics, description, or README provided. Repository by Hugging Face likely relates to fine-tuning or fine-grained web tools for models. Classification chosen based on repository name and org context.