Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
TokenSpeed is a speed-of-light LLM inference engine.
| Date | Stars |
|---|---|
| 2026-07-24 | 1657 |
| 2026-07-25 | 1658 |
| 2026-07-28 | 1724 |
| 2026-07-30 | 1765 |
| 2026-07-31 | 1767 |
| 2026-08-06 | 1767 |
Today
— stars today
This week
+2 stars this week
This month
— stars this month
Momentum
87.0
growth rate 0.11%/day
<p align="center"> <img src="./assets/banner/tokenspeed-banner.png" alt="TokenSpeed: Tokens at the speed of light" width="100%" /> </p> TokenSpeed is a speed-of-light LLM inference engine designed for **agentic workloads**, with TensorRT-LLM-level performance and vLLM-level usability. Our goal is to be the most performant inference engine for production agentic workloads. Core components: - **Modeling layer**: local-SPMD design with a static compiler that generates collective communication from module-boundary placement annotations, so users do not hand-write parallelism logic. - **Scheduler**: C++ control plane and Python execution plane. Request lifecycle, KV cache ownership, and overlap timing are encoded as a finite-state machine, with safe KV resource reuse enforced by the type system at compile time. - **Kernels**: pluggable, layered kernel system with a portable public API and a centralized registry including one of the fastest **MLA** (Multi-head Latent Attention) implementations on Blackwell for agentic workload. - **Entrypoint**: SMG-integrated AsyncLLM for low-overhead CPU-side request handling. ## News - [2026/07] [TML Inkling](https://thinkingmachines.ai/news/introducing-inkling/) at Day 0: FP4 Inference on NVIDIA and [AMD](https://huggingface.co/lightseekorg/Inkling-MXFP4) with [TokenSpeed](https://thinkingmachines.ai/news/introducing-inkling/#inkling-availability). [[blog](https://lightseek.org/blog/tokenspeed-inkling.html)] - [2026/06] Deep dive into the design and optimization of TokenSpeed-Kernel. [[blog](https://pytorch.org/blog/lightseek-tokenspeed-kernel/)] - [2026/05] 🚀 TokenSpeed hits 580 TPS on Qwen3.5-397B-A17B for agentic workloads. [[blog](https://pytorch.org/blog/up-to-580tps-new-speed-record-of-qwen3-5-397b-a17b-on-gpu-for-agentic-workloads-with-tokenspeed/)] - [2026/05] TokenSpeed announced — a speed-of-light LLM inference engine for agentic workloads. [[blog](https://lightseek.org/blog/lightseek-tokenspeed.html)] ## Blogs and Talks For technical blogs, conference talks, and engineering articles from LightSeek Foundation, visit the [LightSeek Blog](https://lightseek.org/blog/). ## Performance Comparison <img src="./assets/perf/tokenspeed-kimi-k2.5-performance.png" alt="TokenSpeed vs. TensorRT-LLM Pareto curves on agentic workload (Kimi K2.5, B200)" width="800" margin="10px"></img> ## Documentation Start here: - [Docs Index](https://lightseek.org/tokenspeed/) - [Getting Started](https://lightseek.org/tokenspeed/guides/getting-started) - [Launching a Server](https://lightseek.org/tokenspeed/guides/launching) - [Model Recipes](https://lightseek.org/tokenspeed/recipes/models) - [Server Parameters](https://lightseek.org/tokenspeed/configuration/server) - [Compatible Parameters](https://lightseek.org/tokenspeed/configuration/compatible-parameters) - [Parallelism](https://lightseek.org/tokenspeed/serving/parallelism)
Excerpt of 2,921 characters
Read on GitHub
178
Yineng Zhang · United States
86
Enwei Zhu · NVIDIA
47
40
Lei Zhang · AMD AI Group · United States
38
30
27
25
17
16
16
12
12
12
11
10
10
9
8
7
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:ef769cbe9417f998, topic:llm, topic:qwen
matched fp:ef769cbe9417f998, topic:vlm