Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
LvLLM is a special NUMA extension of vllm that makes full use of CPU and memory resources, reduces GPU memory requirements, and features an efficient GPU parallel and NUMA parallel architecture, supporting hybrid inference for MOE large models.
| Date | Stars |
|---|---|
| 2026-07-24 | 388 |
| 2026-07-25 | 388 |
| 2026-07-28 | 388 |
| 2026-07-30 | 388 |
| 2026-08-08 | 430 |
| 2026-08-16 | 438 |
| 2026-08-18 | 439 |
| 2026-08-19 | 440 |
| 2026-08-20 | 442 |
| 2026-08-21 | 444 |
| 2026-08-22 | 445 |
| 2026-08-23 | 447 |
| 2026-08-25 | 448 |
| 2026-08-27 | 450 |
| 2026-08-29 | 451 |
| 2026-08-31 | 452 |
| 2026-09-01 | 453 |
| 2026-09-04 | 452 |
| 2026-09-07 | 453 |
| 2026-09-10 | 452 |
| 2026-09-12 | 452 |
| 2026-09-14 | 452 |
| 2026-09-15 | 453 |
| 2026-09-16 | 455 |
| 2026-09-17 | 456 |
| 2026-09-18 | 457 |
| 2026-09-19 | 458 |
| 2026-09-20 | 460 |
Today
+2 stars today
This week
+8 stars this week
This month
+16 stars this month
Momentum
76.0
growth rate 1.77%/day
# LvLLM
**LvLLM = [vLLM](https://github.com/vllm-project/vllm) + [lk_moe](https://pypi.org/project/lk-moe/)**,
plus **SM80/86/89 adaptation** and **SM120 tuning/fixes** for popular new models
(DeepSeek-V4/V4.1, Qwen3.8, GLM-5.3, …).
- **lk_moe** is the CPU+GPU hybrid, NUMA-aware MoE engine (pip-installable); vLLM provides the GPU
path. LvLLM is lk_moe's integration into vLLM **plus** the low-arch (SM80/86/89) bring-up and
SM120 fixes for these models.
- **Fully optional**: with `LVLLM_MOE_NUMA_ENABLED=0` it behaves exactly like stock vLLM.
---
## Model support
| Model | SM80 | SM86 | SM89 | SM90 | SM100 | SM120 | spec-decode |
|-------|------|------|------|------|-------|-------|-------------|
| DeepSeek-V4.1-Flash | ✅ new | ✅ new | ✅ new | ✅ native | ✅ native | ✅ fixed | ✅ dspark |
| DeepSeek-V4-Flash (0731) | ✅ new | ✅ new | ✅ new | ✅ native | ✅ native | ✅ native | ✅ dspark |
| Qwen3.8-Flash-Next | ✅ new | ✅ new | ✅ new | ✅ native | ✅ native | ✅ new | ✅ MTP |
| GLM-5.3-Flash | ✅ new | ✅ new | ✅ new | ✅ native | ✅ native | ✅ fixed | ✅ MTP |
`native` = upstream vLLM · `new` = support added by this Lvllm release · `fixed` = upstream path
corrected by this Lvllm release. Full per-model hardware tables, benchmarks and CLI:
**[`RELEASE_NOTES.md`](./RELEASE_NOTES.md)**.
### Previously verified (lk_moe hybrid) models
Original MOE models verified on vLLM's Qwen3 / GLM / MiniMax lines:
| Family | Models |
|---|---|
| Qwen3 | 3.6 / 3.5-35B-A3B, 3.5-122B-A10B, 3.5-397B-A17B, Coder-Next, Next-80B-A3B-Instruct, Coder-30B-A3B-Instruct, VL-30B-A3B-Instruct |
| GLM | GLM-5.2-NVFP4 [sm120], GLM-4.7(-Flash) / 4.6V |
| MiniMax | M3 / M2.7 / M2.5 / M2.1 |
| Others | gemma-4-26B-A4B-it, NVIDIA-Nemotron-3-Super-120B-A12B-BF16, Ornith-1.0-35B-FP8, Kimi k2.6 / k2.5 |
Unlisted original MOE models from the Qwen3 / GLM / MiniMax lines are theoretically supported,
pending testing.
---
## Benchmarks
Single request, greedy decode (t/s). Each cell = **plain / spec** t/s; spec = **dspark** for
DeepSeek-V4.x, **MTP** for Qwen3.8. `—` = not applicable. Full hardware tables: [`RELEASE_NOTES.md`](./RELEASE_NOTES.md).
| Model | SM120 TP2 (2× 5060 Ti) | SM86 TP2 (2× 3090) | Mixed TP4 (3090×2 + 5060 Ti×2) |
|-------|-------------------------|--------------------|--------------------------------|
| DeepSeek-V4.1-Flash | 25 / 32–37 | 27 / 26–40 | 22 / 24–34 |
| DeepSeek-V4-Flash (0731) | 28 / up to 44.8 | 30.6–31.1 / 36–43 | 30.5–33.6 / 38–43.5 |
| Qwen3.8-Flash-Next | — | 45 / — | 40.3–40.6 / up to 78 |
| GLM-5.3-Flash | 20.8 / — | 22–23 / — | 22.4 / — |
### Reference hardware
All numbers above were measured on one host:
| Item | Spec |
|---|---|
| CPU | 2× AMD EPYC 7642 48-Core (96c/192t total, NPS4 ⇒ 8 NUMA nodes), `LK_THREADS=48` |
| Host RAM | 1 TiB DDR4-3200 (16-channel, 8 per socket) |
| RAM headroom | DeepSeek-V4.x peak ≈ 590 GB resident (≥ 640 GB advised); Qwen3.8 peak ≈ 335 GB |
---
## Why lk_moe
lk_moe spans a MoE model across **VRAM + system memory** and schedules experts across **CPU + GPU**
with NUMA awareness — reaching ~100% VRAM utilization and overlapping GPU prefill with hybrid decode.
| Role (per MoE layer) | Meaning | Env |
|---|---|---|
| **master switch** | `0` = stock vLLM pure-GPU, `1` = hybrid | `LVLLM_MOE_NUMA_ENABLED` |
| GPU-prefill layer | big batches on GPU, small on CPU | `LVLLM_GPU_PREFILL_MIN_BATCH_SIZE` + `LVLLM_GPU_PREFETCH_WINDOW` |
| GPU-resident layer | weights stay in VRAM | `LVLLM_GPU_RESIDENT_MOE_LAYERS` |
Requires x86 AVX2+ and an NVIDIA GPU (SM75+).
---
## Launch
Ready-made serve scripts (per model × topology) live in **[`commands/`](./commands/)**:
```
commands/
dsv4_0731_serve_tp2_{3090,5060ti}[_dspark].sh dsv4_0731_serve_tp4[_dspark].sh # DeepSeek-V4-Flash 0731
dsv41_serve_tp2_3090_dspark.sh # DeepSeek-V4.1-Flash
qwen38_serve_tp2_{3090,5060ti}[_mtp].sh qwen38_serve_tp4[_mtp].sh # Qwen3.8-Flash-Next
glm53_serve_Excerpt of 6,589 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:e6e315ee5514e407, topic:inference, topic:vllm, readme:inference engine
matched fp:e6e315ee5514e407, topic:gpu
matched fp:e6e315ee5514e407, topic:moe