Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
| Date | Stars |
|---|---|
| 2026-07-31 | 587 |
| 2026-08-06 | 587 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Tessera
A small, from-scratch LLM stack built around one goal: distill a large teacher into a small
student, then serve that student efficiently. Keeping that goal end-to-end means touching most
of the pieces that matter in practice — custom GPU kernels, sharded training, an inference
engine, quantization, and a serving front end — without any of it being a toy.
[](https://github.com/zengxiao-he/tessera/actions/workflows/ci.yml)
It runs and is unit-tested on a laptop (CPU or Apple MPS). The Triton/CUDA kernels are written
for NVIDIA GPUs; on anything else the model transparently falls back to a torch reference, and
the kernels are checked against that reference whenever a GPU is available.
```mermaid
flowchart LR
T[Teacher 40M] -->|distill, FSDP/ZeRO-3| S[Student 6M]
S -->|int8 / AWQ / FP8| ENG[Inference engine]
subgraph ENG[Inference engine]
direction TB
PK[paged KV cache] --- SCH[continuous batching] --- SPEC[speculative decode]
end
K[(Triton + CUDA kernels)] -.-> S & ENG
GW[Rust tokio/axum gateway] -->|PyO3| ENG
client([client]) --> GW
```
## What's in it
Training side:
- Decoder transformer with RMSNorm, RoPE, grouped-query attention and SwiGLU ([`tessera/model`](tessera/model)).
- Knowledge-distillation losses: temperature-scaled KL, optional hard CE, hidden-state matching ([`distill/losses.py`](tessera/distill/losses.py)).
- FSDP/ZeRO-3 written from scratch — flat-parameter sharding with a sharded Adam optimizer. It's checked to be numerically identical to single-process training, in one process and across two gloo ranks ([`distill/fsdp.py`](tessera/distill/fsdp.py)).
- Atomic, sharded checkpoints with resume-from-latest ([`distill/checkpoint.py`](tessera/distill/checkpoint.py)).
Kernels:
- A FlashAttention forward kernel in Triton: online softmax, causal masking, GQA, autotuned tile sizes ([`kernels/triton/flash_attention.py`](tessera/kernels/triton/flash_attention.py)).
- Fused RMSNorm, a fused SwiGLU GEMM, and an int8 weight-only matmul that dequantizes in the K-loop ([`kernels/triton`](tessera/kernels/triton)).
- Raw CUDA C++ versions of RMSNorm and attention for the low-level memory work, plus nvtx ranges and Nsight notes ([`kernels/cuda`](tessera/kernels/cuda)).
Serving:
- Block-paged KV cache with a ref-counted allocator for prefix sharing ([`serve/paged_kv.py`](tessera/serve/paged_kv.py)).
- A continuous-batching scheduler that recomposes the batch every step, with admission control and preemption under memory pressure ([`serve/scheduler.py`](tessera/serve/scheduler.py)).
- Speculative decoding with the standard accept/reject sampling ([`serve/speculative.py`](tessera/serve/speculative.py)).
- Post-training quantization: int8 weight-only, AWQ, and an FP8 (E4M3) path ([`quant`](tessera/quant)).
- A Rust gateway (tokio + axum) that handles HTTP and admission back-pressure and calls into the Python engine over PyO3 ([`tessera-rs`](tessera-rs)).
Extras:
- A JAX/XLA reimplementation of the forward pass, used as an independent parity check against PyTorch ([`jax_ref`](jax_ref)).
- Interpretability helpers: activation hooks, a logit lens, and induction-head detection ([`interp`](tessera/interp)).
- A byte-level BPE tokenizer plus image-patch and log-mel audio front ends for multimodal data ([`data`](tessera/data)).
## Quickstart
```bash
git clone https://github.com/zengxiao-he/tessera && cd tessera
python -m venv .venv && source .venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cpu # or a CUDA build
pip install -e ".[dev]"
pytest -m "not gpu" # CPU tests; the kernel tests skip without a GPU
tessera info # list presets and parameter counts
pythonExcerpt of 6,701 characters
Read on GitHub
29
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:ddf49ffa021bde39, topic:quantization, topic:knowledge-distillation, topic:flash-attention
matched fp:ddf49ffa021bde39, topic:inference-engine, desc:serving engine, desc:continuous batching
matched fp:ddf49ffa021bde39, topic:pytorch, topic:jax