Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
LvLLM is a special NUMA extension of vllm that makes full use of CPU and memory resources, reduces GPU memory requirements, and features an efficient GPU parallel and NUMA parallel architecture, supporting hybrid inference for MOE large models.
| Date | Stars |
|---|---|
| 2026-07-24 | 388 |
| 2026-07-25 | 388 |
| 2026-07-28 | 388 |
| 2026-07-30 | 388 |
| 2026-08-06 | 388 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
35.0
growth rate 0.00%/day
# LvLLM GPU + NUMA Dual Parallel [[中文]](./README_cn.md) LvLLM is a special extension of vLLM that fully utilizes CPU and GPU computing resources. It features an efficient GPU parallel + NUMA parallel architecture, suitable for MOE model hybrid inference. > **Core Engine:** The actual hybrid inference functionality—including CPU-GPU collaborative computation, NUMA-aware scheduling, expert weight management, and quantization kernel execution—is powered entirely by **[lk_moe](https://pypi.org/project/lk-moe/)**, a highly optimized MOE hybrid inference engine. Within LvLLM (for vLLM) and [Lsglang](https://github.com/guqiong96/Lsglang) (for sglang), each MOE layer can flexibly choose between the original GPU computation path or invoke lk_moe for hybrid inference. For DeepSeek V4, specialized versions are also available: [Lvllmds4](https://github.com/guqiong96/Lvllmds4) (SM120+) and [Lvllmds4-x](https://github.com/guqiong96/Lvllmds4-x) (SM80+). ## System Features - **GPU + NUMA Dual Parallel**: Supports three computing modes: CPU-GPU hybrid decoding, CPU-GPU hybrid prefill, and GPU prefill - **Memory + VRAM Load Balancing**: Total model footprint = VRAM + memory, accommodating model 1+1=2, 100% VRAM utilization <sup>Note 1</sup> - **GPU Prefill Optimization**: GPU prefill runs in parallel with CPU-GPU hybrid decoding, achieving nearly 100% GPU utilization - **NUMA Thread Optimization**: Cross-node communication as low as 3%, L3 cache hit rate above 50%, GPU load can reach 33% to 50% during decoding ## Relationship with vLLM LvLLM uses the latest vLLM source code and has redesigned the MOE model hybrid inference module, maintaining 100% full compatibility with vLLM<sup>Note 1</sup>. Note 1: x86 CPUs with AVX2 or higher instruction set and Nvidia GPU sm75 or higher architecture ## Usage Guide [[中文]](./README_cn.md) - [Version Changes](#version-changes) - [Supported Models](#supported-models) - [Supported Quantization Formats](#supported-quantization-formats) - [Run Command Reference](#run-command-reference) - [Configuration Parameters](#configuration-parameters) - [Installation Steps](#installation-steps) - [Optimization](#optimization) ## Version Changes ```bash 2026-07-17: lvllm-v2.3.6 - add dtype float16 support for SM75 GPU Prefill 2026-07-08: lvllm-v2.3.2 - add ModelOpt W4A16 NVFP4 quantization types support 2026-07-05: lvllm-v2.3.0 - Optimize GPU prefill speed, CPU AVX512 optimization, removed LVLLM_GPU_RESIDENT_MOE_EXPERTS 2026-06-05: lvllm-v2.2.0 - Upgraded lk_moe module, added support for nvfp4, mxfp4 quantization types, added LVLLM_GPU_RESIDENT_MOE_EXPERTS, removed LVLLM_MOE_USE_WEIGHT, LVLLM_MOE_QUANT_ON_GPU 2026-04-06: lvllm-v2.1.0 - Enhanced power saving effect with LK_POWER_SAVING=1, supports FP8+BF16+AWQ4bit hybrid MOE layer inference 2026-03-22: lvllm-v2.0.0 - FP8 MoE models with INT4 expert quantization support layer-wise loading to reduce peak memory usage, LVLLM_ENABLE_MOE_LAYERWISE_LOAD=1 2026-03-19: lvllm-v1.9.10 - Fixed known issues, added support for new moe model types without gate_proj, e.g., NVIDIA-Nemotron-3-Super-120B-A12B-BF16 2026-03-11: lvllm-v1.9.2 - FP8, AWQ4bit models no longer occupy additional memory when GPU Prefill is enabled, FP8 models removed TO_DTYPE runtime type conversion, KEEP does not support GPU Prefill for now 2026-03-05: lvllm-v1.9.0 - Optimized GPU prefill and regular prefill to ensure output quality 2026-03-01: lvllm-v1.8.10 - Fixed known issues, added new model support 2026-02-02: lvllm-v1.7.0 - Added EP parallel support, running minimax-m2.1 model on 8 GPUs requires --enable_expert_parallel 2026-01-26: lvllm-v1.6.1 - fp8 models support FP8 + INT4 inference, supports GPU Prefill acceleration (high memory usage!) 2026-01-25: lvllm-v1.6.0 - fp8 models support GPU Prefill acceleration (high memory usage!) 2026-01-24: lvllm-v1.5.8 - AWQ 4-bit symmetric quantization models support GPU Prefill acceleration 2026-01-21: lvllm-v1.5.7 - Fixed numerical stability issues with MiniMax-M
Excerpt of 13,661 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:e6e315ee5514e407, topic:inference, topic:vllm, readme:inference engine
matched fp:e6e315ee5514e407, topic:gpu
matched fp:e6e315ee5514e407, topic:moe