Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Curated collection of AI inference engineering resources — LLM serving, GPU kernels, quantization, distributed inference, and production deployment. Compiled from the AER Labs community.
| Date | Stars |
|---|---|
| 2026-07-31 | 256 |
| 2026-08-04 | 259 |
| 2026-08-06 | 259 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Learning Guide: AI Inference Engineering ## Purpose A curated collection of resources for engineers working on AI inference systems — covering LLM serving, GPU kernel programming, attention mechanisms, quantization, distributed inference, and production deployment. Compiled from the AER Labs community. ## How to read Recommended reading order: 1. Read "Tier 1" for all topics first (foundational concepts) 2. Read "Tier 2" for all topics (intermediate depth) 3. Read "Tier 3" for all topics (advanced / cutting-edge) ## Table of contents - [1. LLM Inference Fundamentals](#1-llm-inference-fundamentals) - [2. Inference Engines & Serving Systems](#2-inference-engines--serving-systems) - [3. Attention Mechanisms & Memory Optimization](#3-attention-mechanisms--memory-optimization) - [4. Quantization & Model Compression](#4-quantization--model-compression) - [5. CUDA & GPU Kernel Programming](#5-cuda--gpu-kernel-programming) - [6. Structured Output & Guided Decoding](#6-structured-output--guided-decoding) - [7. Distributed & Multi-GPU Inference](#7-distributed--multi-gpu-inference) - [8. Post-Training & Fine-Tuning](#8-post-training--fine-tuning) - [9. Hardware Architecture & Co-Design](#9-hardware-architecture--co-design) - [10. State-Space Models & Alternative Architectures](#10-state-space-models--alternative-architectures) - [11. Compiler & DSL Approaches](#11-compiler--dsl-approaches) - [12. Confidential & Secure Inference](#12-confidential--secure-inference) - [13. AI Agents & LLM Tooling](#13-ai-agents--llm-tooling) - [14. Production Inference at Scale](#14-production-inference-at-scale) - [15. Benchmarking & Profiling](#15-benchmarking--profiling) - [16. Courses & Comprehensive Guides](#16-courses--comprehensive-guides) - [17. Tools & Libraries](#17-tools--libraries) - [18. Reference Collections](#18-reference-collections) --- ## 1. LLM Inference Fundamentals #### Tier 1 - [Transformer Inference Arithmetic](https://kipp.ly/transformer-inference-arithmetic/) - kipply. Breaks down the compute and memory costs of transformer inference, essential for understanding bottlenecks in LLM serving. - [A Visual Guide to Quantization](https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-quantization) - Maarten Grootendorst. Visual walkthrough of quantization techniques for LLMs, covering the core concepts behind memory-efficient inference. - [KV Cache in LLM Inference](https://pub.towardsai.net/kv-cache-in-llm-inference-7b904a2a6982) - Towards AI. Explanation of KV cache mechanics in LLM inference, covering how key-value caching reduces redundant computation during autoregressive generation. - [Top 5 AI Model Optimization Techniques for Faster, Smarter Inference](https://developer.nvidia.com/blog/top-5-ai-model-optimization-techniques-for-faster-smarter-inference/) - Eduardo Alvarez, NVIDIA. Overview of key optimization techniques for improving inference performance and cost as models grow in size and complexity. - [11 Production LLM Serving Engines: vLLM vs TGI vs Ollama](https://medium.com/@techlatest.net/11-production-llm-serving-engines-vllm-vs-tgi-vs-ollama-162874402840) - TechLatest. Comparative survey of 11 production LLM serving engines with trade-off analysis for different deployment scenarios. - [How Fast Can We Perform a Forward Pass?](https://bounded-regret.ghost.io/how-fast-can-we-perform-a-forward-pass/) - Bounded Regret. Analysis of theoretical and practical limits on transformer forward pass speed, complementing kipply's Transformer Inference Arithmetic. - [How Do MoE Models Compare to Dense Models in Inference?](https://epoch.ai/gradient-updates/moe-vs-dense-models-inference) - Epoch AI. Comparison of mixture-of-experts vs dense models focusing on inference costs, efficiency, and decoding dynamics. - [LLM Routing](https://www.liuxunzhuo.com/llm-routing) - Xunzhuo Liu. Overview of LLM routing strategies for directing requests to optimal models based on task characteristics. - [How LLM Inference
Excerpt of 83,963 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:79aee4658145f10b, llm:Description: "Curated collection of AI inference engineering resources — LLM serving, GPU kernels, quantization, distributed inference, and production deployment."
matched fp:79aee4658145f10b, llm:Description: "Curated collection of AI inference engineering resources — LLM serving, GPU kernels, quantization, distributed inference, and production deployment."
matched fp:79aee4658145f10b, llm:Description: "Curated collection of AI inference engineering resources — LLM serving, GPU kernels, quantization, distributed inference, and production deployment."
matched fp:79aee4658145f10b, llm:Description: "Curated collection of AI inference engineering resources — LLM serving, GPU kernels, quantization, distributed inference, and production deployment."