Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Runtimes and servers that execute model inference at speed and scale.
Signals: llm-inference, inference, inference-engine, model-serving, llm-serving, llama-cpp, vllm, triton-inference-server
410 results
From-scratch PyTorch implementation of Google's TurboQuant (ICLR 2026) for LLM KV cache compression. 5x compression at 3-bit with 99.5% attention fidelity.
MiniLLM is a minimal system for running modern LLMs on consumer-grade GPUs
A throughput-oriented high-performance serving framework for LLMs
Efficient and easy multi-instance LLM serving
Running any GGUF SLMs/LLMs locally, on-device in Android
🤖 TUI for LLMs
The Triton TensorRT-LLM Backend
Running a LLM on the ESP32
Compare compatible open-source local LLM projects by their metrics to assess popularity and activeness.
Build and deploy AI-powered APIs in seconds
a lightweight LLM model inference framework
Official implementation of "Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding"
GraphRAG using Local LLMs - Features robust API and multiple apps for Indexing/Prompt Tuning/Query/Chat/Visualizing/Etc. This is meant to be the ultimate GraphRAG/KG local LLM app.
Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads
Everything I know about running LLMs locally
No description
计图大模型推理库，具有高性能、配置要求低、中文支持好、可移植等特点
A fast inference library for running LLMs locally on modern consumer-class GPUs
Fast inference from large lauguage models via speculative decoding
闻达：一个LLM调用平台。目标为针对特定环境的高效内容生成，同时考虑个人和中小企业的计算资源局限性，以及知识安全和私密性问题
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
A framework for serving and evaluating LLM routers - save LLM costs without compromising quality
Example code and documentation on how to get Stable Diffusion running with ONNX FP16 models on DirectML. Can run accelerated on all DirectML supported cards including AMD and Intel.
WeDLM: The fastest diffusion language model with standard causal attention and native KV cache compatibility, delivering real speedups over vLLM-optimized baselines.
24,535 repositories in the index in total.
