Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Making models smaller and faster: quantization, distillation, pruning, sparsity and kernel-level work.
Signals: quantization, model-compression, pruning, knowledge-distillation, gptq, awq, bitsandbytes, sparsity
185 results
llm-export can export llm model to onnx.
大模型/LLM推理和部署理论与实践
[NeurIPS 2024] BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models
Code repo for the paper "LLM-QAT Data-Free Quantization Aware Training for Large Language Models"
Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
The Truth Is In There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction
Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"
Automated Identification of Redundant Layer Blocks for Pruning in Large Language Models
Code for paper: "QuIP: 2-Bit Quantization of Large Language Models With Guarantees"
[ACL 2025 Main] EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
Optimization Modeling Using mip Solvers and large language models
a toolkit on knowledge distillation for large language models
CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation
An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).
[ICLR 2026] ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference
[ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
Summary of some awesome work for optimizing LLM inference
From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
Caveman Compression is a semantic compression method for LLM contexts. It removes predictable grammar while preserving the unpredictable, factual content that defines meaning.
Distill your LLM traces into a fast hybrid System 1 models. Open-source ML + LLM inference with calibrated deferral.
[ICLR 2026]QeRL enables RL for 32B LLMs on a single H100 GPU.
[MLSys'25] QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving; [MLSys'25] LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
No description
No description
24,543 repositories in the index in total.
