Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Making models smaller and faster: quantization, distillation, pruning, sparsity and kernel-level work.
Signals: quantization, model-compression, pruning, knowledge-distillation, gptq, awq, bitsandbytes, sparsity
184 results
1.58 Bit LLM on Apple Silicon using MLX
Official codebase for "Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling".
Awesome list for LLM pruning.
Cuda kernels for leveraging LLM sparsity to improve throughput and decrease the memory requirements during inference and training.
No description
[ICLR 2025🔥] SVD-LLM & [NAACL 2025🔥] SVD-LLM V2
Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.
W8A8/W4A8 inference + optimized SDPA on Apple Silicon — unlocking unused INT8 TensorOps in M5 for 1.2–1.9× faster LLM prefill, plus FlashInfer-inspired GQA decode attention for up to 1.6× SDPA speedup, built as MLX custom primitives.
Design hardware-friendly model architectures and migrate existing LLMs with minimal performance loss
[MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
Fast Matrix Multiplications for Lookup Table-Quantized LLMs
Code repo for the paper "SpinQuant LLM quantization with learned rotations"
Awesome list for LLM quantization
A vLLM patch + hand‑written SM120 SASS kernels: 2‑bit MoE experts + an FP4 "delta" cache that recovers precision — matching the official (NV)FP4 checkpoint's quality on consumer Blackwell cards
llm-export can export llm model to onnx.
大模型/LLM推理和部署理论与实践
[NeurIPS 2024] BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models
Code repo for the paper "LLM-QAT Data-Free Quantization Aware Training for Large Language Models"
Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
The Truth Is In There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction
Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"
Automated Identification of Redundant Layer Blocks for Pruning in Large Language Models
Code for paper: "QuIP: 2-Bit Quantization of Large Language Models With Guarantees"
[ACL 2025 Main] EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
24,523 repositories in the index in total.
