Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Blazing-fast LLM inference in pure Rust. No PyTorch and Python runtime.
| Date | Stars |
|---|---|
| 2026-07-24 | 293 |
| 2026-07-25 | 294 |
| 2026-07-28 | 296 |
| 2026-07-30 | 296 |
| 2026-07-31 | 295 |
| 2026-08-06 | 295 |
Today
— stars today
This week
-1 stars this week
This month
— stars this month
Momentum
35.0
growth rate 0.00%/day
<p align="center"> <img src="logo.svg" alt="xInfer" width="400"><br> <b>Blazing-fast LLM inference in pure Rust.</b> No PyTorch. No Python runtime. Just fast, portable, production-ready inference.<br> <a href="./ReadMe.md">English</a> | <a href="./ReadMe-CN.md">简体中文</a> </p> --- ## ✨ Why xInfer? | | Feature | Details | |---|---|---| | **0️⃣** | Zero Python dependencies | Pure Rust backend — no PyTorch, no CUDA Python bindings | | **⚡** | Fast | Native Flash Attention, FlashInfer, CUDA Graphs, continuous batching, prefix caching, PD disaggregation. Up to **197 tok/s** decode for `30B+` models on consumer GPUs | | **🪶** | Tiny footprint | Core scheduling + attention logic in **< 5 000 lines** of Rust | | **🌍** | Cross-platform | CUDA (Linux/Windows), Metal (macOS). Same binary, same API | | **🏭** | Production-ready | OpenAI/Anthropic-compatible APIs, built-in ChatGPT-style Web UI, MCP tool calling, structured outputs, embedding + tokenizer endpoints, multi-token prediction (MTP) | | **🗜️** | Aggressive KV compression | TurboQuant (`2–4 bit` KV cache) extends context up to **4.3×** with minimal quality loss. Run `30B+` MoE models with **millions of context** on single 24/32 GB GPUs | | **🔥** | V100 + NVFP4 | First-ever NVFP4 + low-bit KV cache on V100 — no hardware FP4 needed, coherent output on legacy GPUs | | **🐍** | Lightweight Python bindings | Optional PyO3 wheel when you need a Python entry point | --- ## 📽️ Demo <video src="pages/assets/demo.mp4" width="600" controls></video> --- ## 🚀 Quick Start ### 📦 Install **Option 1 — Install DEB or Python package** ```bash curl -sSL https://guoqingbao.github.io/xinfer/install.sh | bash ``` **Option 2 — npm** ```bash npm install -g xinfer-ai ``` --- ### ▶️ Run **Using HuggingFace Model ID:** ```bash xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4 --ui-server ``` **Using local model path:** ```bash xinfer --m /home/Qwen3.6-35B-A3B --d 0,1 --ui-server ``` **Python usage:** ```bash # python3 -m xinfer.chat python3 -m xinfer.server --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4 --ui-server ``` **MTP** ```bash xinfer --w /home/Qwen3.6-35B-A3B --d 0,1 --ui-server --mtp 2 ``` > **Tip:** Open `http://IP:8001` for the built-in chat UI, or use `http://IP:8000/v1/` as your API `Base URL`. --- ### 🗜️ KV Cache Compression Add `--kvcache-dtype` to compress KV cache and extend context length: | Flag (`--kvcache-dtype`) | Compression | Quality | GPU Requirement | |---|---|---|---| | _(default)_ | 1× (BF16) | Baseline | All | | `fp8` | **2×** | Near-lossless | SM70+ / Apple M1 | | `turbo8` | **2.6×** | 79–100% throughput | SM70+ / Apple M1| | `turbo4` | **3.7×** | Best balance | SM70+ / Apple M1| | `turbo3` | **4.7×** | Max compression | SM70+ | --- ## 📈 Performance > Tested on **V100-32G**, **A100-40G**, **Hopper-80G** and **RTX 5090** | Model | Format | Size | Decoding Speed (without MTP) | |---|---|---|---| | Ministral-3-3B (**Multimodal**) | ISQ (BF16→Q4K) | 3B | **193.67** tokens/s | | Qwen3-VL-8B-Instruct (**Multimodal**) | Q8_0 | 8B | **112.51** tokens/s | | Llama-3.1-8B | ISQ (BF16→Q4K) | 8B | **133.10** tokens/s | | DeepSeek-R1-0528-Qwen3-8B | Q4_K_M | 8B | **139.25** tokens/s | | GLM-4-9B-0414 | Q4_K_M | 9B | **77.48** tokens/s | | QwQ-32B | Q4_K_M | 32B | **46.02** tokens/s | | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **197.29** tokens/s (**RTX 5090**) | | **Qwen3-30B-A3B** | NVFP4 | **30B (MoE)** | **72.86** tokens/s (**V100, Software FP4**) | | **Qwen3-Next-80B** | AWQ | **80B (MoE)** | **90** tokens/s (**Hopper**) | | **Qwen3.5-27B** (**Multimodal**) | Q4_K_M | **27B (Dense)** | **49.33** tokens/s | | **Qwen3.5-27B/Qwen3.6-27B** | FP8 | **27B (Dense)** | **45** tokens/s (**Hopper**) | | **Qwen3.6-35B-A3B** (**Multimodal**) | FP8 | **35B (MoE)** | **120** tokens/s (**Hopper**) | | **GLM4.7 Flash** | NVFP4 | **30B (MoE)** | **79** tokens/s (**Hopper, Software FP4**) | | **Gemma4-31B** | ISQ (BF16→Q4K) | **31B (Dense)** | **47** tokens/s (**Hopper*
Excerpt of 21,611 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:f9bb16b4dddbdaaa, topic:llm, topic:qwen
matched fp:f9bb16b4dddbdaaa, topic:vllm, readme:kv cache, readme:continuous batching