Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality
| Date | Stars |
|---|---|
| 2026-07-31 | 356 |
| 2026-08-01 | 357 |
| 2026-08-06 | 357 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality [](https://physics.allen-zhu.com/part-4-architecture-design/part-4-1) [](https://physics.allen-zhu.com/part-4-architecture-design/part-4-2) [](https://ssrn.com/abstract=5240330) [](https://github.com/facebookresearch/PhysicsLM4) [](https://huggingface.co/collections/facebook/physics-of-language-models-part-42-6883fa5e7218a7369f22a806) **Author**: Zeyuan Allen-Zhu Welcome to this code repository for the *Physics of Language Models* series. This repository provides all the resources required to reproduce results from the series' Part 4, as well as relevant contributions from Parts 1, 3.1, and 3.3. Below, we describe the key components of this release. --- ## 📑Repository Contents ### 🔴 Data Generators: [`data-synthetic-pretrain`](data-synthetic-pretrain/) and [`data-reallife-eval`](data-reallife-eval/) The synthetic pretraining playground includes the **Depo**, **Brevo**, **Capo**, **Mano**, and **Lano** datasets introduced in [*Physics of Language Models: Part 4.1 — Architecture Design and the Magic of Canon Layers*](https://ssrn.com/abstract=5240330). Three are trivial to reimplement; the remaining two are provided here: - [**Lano**](data-synthetic-pretrain/Lano-cfg/) — also featured in [*Part 1: Learning Hierarchical Language Structures*](https://ssrn.com/abstract=5250639). - [**Capo**](data-synthetic-pretrain/Capo-bioS-bioR/) — includes *bioS* and *bioR* generators from [*Part 3.1: Knowledge Storage and Extraction*](https://ssrn.com/abstract=5250633) and [*Part 3.3: Knowledge Capacity Scaling Laws*](https://ssrn.com/abstract=5250617). - [**Depo**](data-synthetic-pretrain/Depo/), [**Brevo**](data-synthetic-pretrain/Brevo/), [**Mano**](data-synthetic-pretrain/Mano/) — includes *Depo1*, *Depo2*, *Brevo1*, *Brevo2* and *Mano* datasets used in [*Part 4.1: Architecture Design and the Magic of Canon Layers*](https://ssrn.com/abstract=5240330). The real-life experiments in [*Part 4.1*](https://ssrn.com/abstract=5240330) also used the following evaluation tasks: - [**multi-hop**](data-reallife-eval/multi-hop/) — a birth-year multi-hop in-context retrieval task, arguably the simplest and most natural real-life multi-hop benchmark. - [**Babilong**](data-reallife-eval/Babilong/) — slightly modified few-shot prompts from the original Babilong evaluation setup. ### 🔴 [`huggingface`](huggingface/) and [`huggingface_linear`](lingua_modified/huggingface/) Huggingface-style models that add Canon layer supports, as highlighted in [*Physics of Language Models: Part 4.1*](https://ssrn.com/abstract=5240330): - **Current models**: - **LlamaCanon**: Includes Canon layers, QK-norm, and partial RoPE support (see [`huggingface`](huggingface/)) - **GLA** (with GLA5 modifications), **GDN** (with GDN2 modifications), and **Mamba2** models (see [`huggingface_linear`](lingua_modified/huggingface/)) ### 🔴 [`lingua_modified`](lingua_modified/) A modified version of [Meta’s Lingua codebase](https://github.com/facebookresearch/lingua) which is optimized for efficient pretraining. Key modifications include: - For Transformer(Llama) training: - Added support for Canon-ABCD layers, QK-norm, z-loss, and partial RoPE for Transformer. - Compatibility with the above Hugging Face `LlamaCanon` model (*bonus*: a `load_from_lingua_state` method for seamless loading of Lingua state_dicts). ```bash cd lingua_modified python -m lingua.stool script=apps.main.train nodes=1 config=apps/main/configs/canon_1B.yaml account=<bla
Excerpt of 10,052 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:ed13c10726dca78c, desc:pretraining