Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Official implementation of "Continuous Autoregressive Language Models"
| Date | Stars |
|---|---|
| 2026-07-31 | 812 |
| 2026-08-05 | 814 |
| 2026-08-06 | 814 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# CALM: Continuous Autoregressive Language Models
[](https://arxiv.org/abs/2510.27688)
[](https://github.com/shaochenze/calm)
[](https://huggingface.co/collections/cccczshao/calm)
[](https://shaochenze.github.io/blog/2025/CALM)
## Overview
<p align="center">
<img src="overview.png" width="700">
<br>
<em></em>
</p>
Modern Large Language Models (LLMs) are constrained by a fundamental bottleneck: they generate text one token at a time. **CALM (Continuous Autoregressive Language Models)** confronts this challenge by introducing a paradigm shift in language modeling. Instead of predicting one discrete token at a time, CALM learns to predict a single **continuous vector** that represents an entire chunk of **K** tokens.
This is achieved through a two-stage process:
1. **A high-fidelity autoencoder** learns to compress K tokens into a single vector and reconstruct them with near-perfect accuracy.
2. **A continuous-domain language model** then performs autoregressive prediction in this vector space.
An in-depth explanation of CALM is available in [this blog](https://shaochenze.github.io/blog/2025/CALM).
### Key Features
* 🚀 **Ultra-Efficient by Design:** Dramatically improves training and inference efficiency by reducing the number of autoregressive steps by a factor of K.
* 💡 **A New Scaling Axis:** Introduces a new scaling dimension for LLMs—**semantic bandwidth (K)**. Instead of just scaling parameters and data, you can now scale the amount of information processed in a single step.
* 🛠️ **A Comprehensive Likelihood-Free Toolkit:** Operating in a continuous domain requires new tools. This repository provides the full suite of algorithms that make CALM possible:
* **A Robust Autoencoder** to learn high-fidelity continuous representations of token chunks.
* **Energy-Based Training**, a principled and likelihood-free method for generative modeling.
* **BrierLM**, a new metric for calibrated, likelihood-free evaluation of language models.
* **Temperature Sampling** for controlled, high-quality text generation using only a black-box sampler.
## Getting Started
1. Clone the Repository
```bash
git clone https://github.com/shaochenze/calm.git
cd calm
```
2. Install Dependencies
```bash
pip install -r requirements.txt
```
3. Prepare the Training Data
Run the following script to download and process the [pile-uncopyrighted](https://huggingface.co/datasets/monology/pile-uncopyrighted) dataset for training.
```bash
bash data/get_data.sh
```
The dataset is large. Please ensure you have at least **2.5TB** of free disk space.
## Training
To replicate the results for CALM with `K=4`, follow these steps. The training process is divided into two main stages: train the autoencoder and then train the CALM language model.
### 1. Train the Autoencoder
First, train the autoencoder on approximately 15B tokens of data. This model learns the mapping between token chunks and their continuous vector representations.
```bash
bash train/train_autoencoder.sh
```
<details>
<summary>Click to see the full training script</summary>
```bash
#!/bin/bash
WORK_PATH=/path/to/the/code
CHECKPOINT_PATH=${WORK_PATH}/checkpoints/autoencoder
TOKENIZER_PATH=${WORK_PATH}/llama3_tokenizer
DATASET_TRAIN=${WORK_PATH}/pile-uncopyrighted/train/00.text.jsonl,${WORK_PATH}/pile-uncopyrighted/train/01.text.jsonl
DATASET_VALID=${WORK_PATH}/data/wikitext_document_level-test.json
torchrun --nnodes 1 --node_rank 0 --nproc_per_node 8 \
-m train.train_autoencoder \
--tokenizer_name $TOKENIZER_PATH \
--config_overrides "latent_size=128,num_encoder_layers=2,num_decoder_layers=2,patch_size=4" \
--train_file $DATASET_TRExcerpt of 10,988 characters
Read on GitHub
12
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:6c748a29361db5ac, llm:Repository description: 'Official implementation of "Continuous Autoregressive Language Models"' (Python).
matched fp:6c748a29361db5ac, llm:Repository description: 'Official implementation of "Continuous Autoregressive Language Models"' (Python).
matched fp:6c748a29361db5ac, llm:Repository description: 'Official implementation of "Continuous Autoregressive Language Models"' (Python).