Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Cuda kernels for leveraging LLM sparsity to improve throughput and decrease the memory requirements during inference and training.
| Date | Stars |
|---|---|
| 2026-07-31 | 255 |
| 2026-08-06 | 255 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<h1 align="center">
<a href="https://github.com/SakanaAI/sparser-faster-llms">
<img src="figures/logo.png" width="300" /></a><br>
<b>Sparser, Faster, Lighter Transformer Language Models</b><br>
</h1>
<p align="center">
📚 <a href="https://arxiv.org/abs/2603.23198">[Paper]</a> |
🤗 <a href="https://huggingface.co/collections/SakanaAI/sparser-faster-lighter-transformers">[Checkpoints]</a>
🐠 <a href="https://pub.sakana.ai/sparser-faster-llms">[Blog]</a>
</p>
This repository contains the reference code for the paper **Sparser, Faster, Lighter Transformer Language Models**. It includes sparse training code and our custom CUDA kernels designed for H100 GPUs for sparse models, leveraging the TwELL packing format.
## Installation
The repository expects a CUDA 12.8+ environment:
```bash
git clone https://github.com/SakanaAI/sparser-faster-llms.git
cd sparser-faster-llms
bash scripts/install.sh
# or uv
# python -m venv .venv
# source .venv/bin/activate
# bash scripts/install.sh --uv
```
## Repo Structure
```text
.
├── accelerate_configs/ # Accelerate + DeepSpeed launch configs
├── benchmark_inference.py # Minimal torch vs TwELL inference benchmark
├── benchmark_base.py # Small benchmark helpers
├── cfgs/ # Hydra configs for model/data/training
├── custom_data/ # Pretraining dataset utilities
├── custom_models/
│ ├── sparse_models.py # Sparse model definitions
│ ├── sparse_testing_utils.py # Sparse -> HF / TwELL conversion helpers
│ ├── twell_modules/ # TwELL CUDA kernels
│ └── hybrid_modules/ # TwELL+Hybrid CUDA kernels
├── energy_utils.py # Optional GPU energy measurement helpers
├── launch.sh # Main multi-GPU training entrypoint
├── load_dataset.py # Dataset loading glue
├── scripts/install.sh # Minimal installation script
├── train.py # Hydra training entrypoint
└── trainers/
└── logging_trainer.py # Trainer used by the public training path
```
## Roadmap
- [x] Sparse model training code
- [x] TwELL inference kernels
- [x] Efficient TwELL+Hybrid training kernels
## Inference Benchmarking
We release pretrained sparse checkpoints on the Hugging Face Hub at:
- `SakanaAI/SparseLM0.5B`
- `SakanaAI/SparseLM1B`
- `SakanaAI/SparseLM1.5B`
- `SakanaAI/SparseLM2B`
You can benchmark our kernels against the Hugging Face PyTorch reference with our benchmarking scripts `benchmark_inference.py`, e.g.:
```bash
python benchmark_inference.py \
--model-path SakanaAI/SparseLM1.5B \
--reps 500 \
--warmup-reps 5 \
--out-csv results/benchmark_inference/SparseLM1.5B.csv
```
We provide two implementations of the TwELL kernels: the default `twell` implementation, and a `twell-flex` variant that is expected to be slightly faster in cases of non-uniform sparsity patterns (differences are still expected to be only less than 0.1% overall). You can enable the `twell-flex` variant with the `--flex-kernels` flag, e.g.:
```bash
python benchmark_inference.py \
--model-path SakanaAI/SparseLM1.5B \
--reps 500 \
--warmup-reps 5 \
--out-csv results/benchmark_inference/SparseLM1.5B-flex.csv
--flex-kernels
```
To also measure GPU energy during the benchmark loop:
```bash
python benchmark_inference.py \
--model-path SakanaAI/SparseLM1.5B \
--reps 500 \
--warmup-reps 5 \
--measure-energy \
--out-csv results/benchmark_inference/SparseLM1.5B_energy.csv
```
You can benchmark your own local sparse models by overriding `--model-path /path/to/local/checkpoint_dir`.
## Training (Torch)
We provide simple functionality using standard PyTorch for Sparse training:
```bash
./launch.sh <num_gpus> <run_cfg> [zero1|offload|offload_optim] [hydra overrides...]
```
We provide premade Hydra configs to obtain sparse models at different sizes, for using Torch kernels:
- `sparsity_gated_0p5b`
- `sparsity_gated_1b`
- `sparsity_gated_1p5b`
- `sparsity_gated_2Excerpt of 4,979 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:ccff207a0f11e0b0, llm:Repository description: "Cuda kernels for leveraging LLM sparsity to improve throughput and decrease the memory requirements during inference and training." Language: Cuda
matched fp:ccff207a0f11e0b0, llm:Repository description: "Cuda kernels for leveraging LLM sparsity to improve throughput and decrease the memory requirements during inference and training." Language: Cuda
matched fp:ccff207a0f11e0b0, llm:Repository description: "Cuda kernels for leveraging LLM sparsity to improve throughput and decrease the memory requirements during inference and training." Language: Cuda
matched fp:ccff207a0f11e0b0, llm:Repository description: "Cuda kernels for leveraging LLM sparsity to improve throughput and decrease the memory requirements during inference and training." Language: Cuda