Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A highly optimized LLM inference acceleration engine for Llama and its variants.
| Date | Stars |
|---|---|
| 2026-07-31 | 908 |
| 2026-08-02 | 908 |
| 2026-08-06 | 908 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# ZhiLight ✨ __ZhiLight__ ✨is a highly optimized LLM inference engine developed by Zhihu and ModelBest Inc. The "Zhi" in its name stands for **Z**hihu. ZhiLight can accelerate the inference of models like Llama and its variants, especially on PCIe-based GPUs. Compared to mainstream open-source inference engines, for example, vllm, it has significant performance advantages. ## News 🔥 * [2025/05] Support DeepSeek-V3/DeepSeek-R1 FP8 block-quantized model. * [2025/02] Support DeepSeek-V3/DeepSeek-R1 AWQ/GPTQ quantized model. * [2025/01] Support [DeepSeek-VL2](https://github.com/deepseek-ai/DeepSeek-VL2) multimodal models: deepseek-vl2-tiny, deepseek-vl2-small, deepseek-vl2. * [2025/01] Support LLaMA3 models: Llama-3.1-8B, Llama-3.1-70B, Llama-3.2-1B, Llama-3.2-3B, Llama-3.3-70B. ## Roadmap see [Roadmap](https://github.com/zhihu/ZhiLight/wiki/Roadmap) ## 🎉🎉 Main Features * Asynchronous OpenAI compatible interface adapted from vllm * Custom defined tensor and unified global memory management * 🔥 Encode and all-reduce overlap, we named "dual streams" * Support Int8-quantized all-reduce to further reduce all-reduce cost. * Host all-reduce based on SIMD instructions * Optimized fused kernels, qkv, residual & layernorm etc. * 🔥 Fused batch attention for decoding based on tensor core instructions * Support TP and PP on one node, TP is recommended * Support dynamic batch * Support flash attention prefill * Support chunked prefill * Support prefix cache * Support Native INT8/SmoothQuant/FP8/AWQ/GPTQ quantization * Support Marlin kernel for GPTQ * Support MoE, DeepseekV2 MoE and DeepseekV2 MLA * Support Llama/Llama2, Mixtral, Qwen2 series and similar models ## 🔧 Basic Usage ```bash # Concurrently compile the wheel package, and turn off the unit test CMAKE_BUILD_PARALLEL_LEVEL=32 TESTING=0 python setup.py bdist_wheel # Compile with ninja backend CMAKE_GENERATER="Ninja" python setup.py bdist_wheel # Install directly cd ./ZhiLight && pip install -e . # Start OpenAI compatible server python -m zhilight.server.openai.entrypoints.api_server [options] ``` ## ✈️ Docker Image ZhiLight only depends on the CUDA runtime, cuBLAS, NCCL, and a few Python packages in requirements.txt. You can use the image below for running or building it. You can also directly refer to docker/Dockerfile. ```bash docker pull ghcr.io/zhihu/zhilight/zhilight:0.4.8-cu124 ``` ## 📈 Performance Notes We conducted performance reviews on various mainstream NVIDIA GPUs with different model sizes and precisions. For dense models ranging from 2B to 110B parameters on PCIe devices, ZhiLight demonstrates significant performance advantages compared to mainstream open-source inference engines. Test Description: - Test purpose is to demonstrate performance, applicable scenarios and limitations - Test metrics include: - QPS: Queries Per Second - TTFT (Time To First Token): First token generation latency - TPOT (Time Per Output Token): Generation latency per output token - Test environments include: - AD102 PCIe : Consumer-grade GPU for experimental research - A800: Data center GPU for production deployment - H20: Data center GPU for production deployment - Test models include: - Large-scale models: DeepSeek-R1, Qwen1.5-110B, Qwen2-72B, LLama-3.1-70B - Medium-scale models: Qwen2.5-14B, Llama-3.1-8B, Minicpm-2B - Compared inference engines include: - vLLM - SGLang - ZhiLight ### DeepSeek-R1 AWQ - **NVIDIA A800*8** | Inference Engine | QPS | TTFT Mean | TTFT P95 | TPOT Mean| TPOT P95| | :---: | :---: | :---: | :----: | :---: | :---: | | vLLM | 0.08 | 1817.7 | 2556.42 | 109.86 | 129.98 | | **ZhiLight** | **0.16**|**1590.96** | **2214.34**|**115.97** |**139.99**| - **NVIDIA H20*8** | Inference Engine | QPS | TTFT Mean | TTFT P95 | TPOT Mean| TPOT P95| | :---: | :---: | :---: | :----: | :---: | :---: | | **ZhiLight** | **0.23**|**2025.15** |
Excerpt of 6,489 characters
Read on GitHub
67
28
3
3
2
maxwell
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:321ff85cc3c9e54f, topic:llm-inference, topic:inference-engine, topic:model-serving
matched fp:321ff85cc3c9e54f, topic:llm, topic:gpt, topic:llama