Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A course of learning LLM inference serving on Apple Silicon for systems engineers: build a tiny vLLM + Qwen.
| Date | Stars |
|---|---|
| 2026-07-24 | 4405 |
| 2026-07-25 | 4407 |
| 2026-07-28 | 4421 |
| 2026-07-30 | 4421 |
| 2026-07-31 | 4427 |
| 2026-08-06 | 4427 |
Today
— stars today
This week
+6 stars this week
This month
— stars this month
Momentum
6.0
growth rate 0.14%/day
# tiny-llm [](https://github.com/skyzh/tiny-llm/actions/workflows/main.yml) tiny-llm is a hands-on course for systems engineers who want to understand LLM inference end to end. You can think of it as an LLM-serving counterpart to CMU's [Needle](https://github.com/dlsyscourse/hw1/tree/main/python/needle) project: build the path that loads a Qwen3 model, turns tokens into logits, and generates text. The course begins with array and matrix operations, then introduces kernels and serving machinery as the running model needs them. Keeping the implementation small enough to read end to end makes it possible to connect the equations to memory traffic, kernel occupancy, KV-cache growth, batching, and request scheduling. The course is built on MLX arrays and the MLX extension runtime, without using high-level neural-network layers. When a chapter teaches an operator, your solution implements that operator in Python, C++, or Metal rather than calling the corresponding optimized MLX operation. MLX remains the correctness oracle and performance baseline. ## The Learning Path The course follows a four-week learning path: - **Week 1: From Matmul to Text.** Build a readable Qwen3 model from array operations: attention, RoPE, GQA, RMSNorm, the MLP, sampling, and the autoregressive loop. - **Week 2: A Step Closer to vLLM.** Add a KV cache, establish a synchronized MLX baseline, and let profiles choose the next optimization. The path moves from quantized decode matvec to fused model kernels, tiled prefill, and split-K where the measured Qwen shapes need it. - **Week 3: Build a Mini vLLM.** Introduce continuous batching and chunked admission, then make paged KV the canonical serving layout. Decode attention and FlashAttention learn to read pages directly so the scheduler does not rebuild dense history on every step. - **Week 4: Build a Coding Agent.** Use multi-turn sessions to motivate cache reuse, context compaction, rewind, interruption, and evaluation. ## Why MLX and Qwen3? Apple silicon provides a practical local environment with one shared memory space and direct access to Metal kernels. Students can inspect the complete path on one machine instead of depending on an expensive CUDA GPU setup. Qwen3-4B is large enough to expose real weight-bandwidth, attention, and cache costs, but small enough to iterate on locally. Its grouped-query attention, QK normalization, BF16 activations, and 4-bit weights also keep the exercises close to current model-serving work. ## Start Here The book is published at [skyzh.github.io/tiny-llm](https://skyzh.github.io/tiny-llm/). Begin with the [environment setup](https://skyzh.github.io/tiny-llm/setup.html), or verify an existing checkout with: ```bash pdm install -v pdm run check-installation pdm run test-refsol -- -- -k week_1 ``` The `tiny_llm` package is where students implement the exercises. `tiny_llm_ref` contains the reference solution used by the tests and benchmark appendix. The detailed chapter order and current status live in the [book summary](book/src/SUMMARY.md). ## Roadmap The status columns track whether each chapter's code, tests, and documentation are ready. Week 4 remains a design draft and is not yet part of the rendered daily course. | Week + Chapter | Topic | Code | Test | Doc | |---|---|---|---|---| | 1.1 | Attention | ✅ | ✅ | ✅ | | 1.2 | RoPE | ✅ | ✅ | ✅ | | 1.3 | Grouped Query Attention | ✅ | ✅ | ✅ | | 1.4 | RMSNorm and MLP | ✅ | ✅ | ✅ | | 1.5 | Load the Model | ✅ | ✅ | ✅ | | 1.6 | Generate Responses (aka Decoding) | ✅ | ✅ | ✅ | | 1.7 | Sampling | ✅ | ✅ | ✅ | | 2.1 | KV Cache | ✅ | ✅ | 🚧 | | 2.2 | Benchmark and Profile | 🚧 | 🚧 | 🚧 | | 2.3 | Quantized Matvec | ✅ | ✅ | 🚧 | | 2.4 | Fused Decode Attention | 🚧 | 🚧 | 🚧 | | 2.5 | Fused Model Kernels | 🚧 | 🚧 | 🚧 | | 2.6 | SIMD-Matrix Prefill | ✅ | ✅ | 🚧 | | 2.7 | Split-K Prefill | ✅ | ✅ | 🚧 | | 3.1 | Continuous Batching | ✅ |
Excerpt of 4,906 characters
Read on GitHub
Alex Chi Z · Databricks 🧱 · China
153
48
Eric Zhang · @thinking-machines-lab · United States
5
5
Liu Jinyi · UCAS · China
4
4
2
yangpeng · China
2
you06 · @Airyworks
1
1
1
1
Li0k · RisingWave Labs
1
Jasmine
1
1
Eric Yue
1
Eric Fu · Singapore
1
Mathew Shen · @bytedance @netease @tencent · China
1
Eikasia30
1
Phoenix · China
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:98831755ec16296b, topic:llm, topic:qwen
matched fp:98831755ec16296b, topic:vllm, readme:model serving, readme:kv cache
matched fp:98831755ec16296b, topic:course, desc:course, readme:course