Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Runtimes and servers that execute model inference at speed and scale.
Signals: llm-inference, inference, inference-engine, model-serving, llm-serving, llama-cpp, vllm, triton-inference-server
410 results
LLM-PowerHouse: Unleash LLMs' potential through curated tutorials, best practices, and ready-to-use code for custom training and inferencing.
[ICLR 2024] Efficient Streaming Language Models with Attention Sinks
The easiest & fastest way to run customized and fine-tuned LLMs locally or on the edge
LLM (Large Language Model) FineTuning
Graph Data Science: an abstraction layer in Python for building knowledge graphs, integrated with popular graph libraries – atop Pandas, NetworkX, RAPIDS, RDFlib, pySHACL, PyVis, morph-kgc, pslpython, pyarrow, etc.
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O
[ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
TurboQuant: Near-optimal KV cache quantization for LLM inference (3-bit keys, 2-bit values) with Triton kernels + vLLM integration
[ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
A highly optimized LLM inference acceleration engine for Llama and its variants.
Low-bit LLM inference on CPU/NPU with lookup table
Minimal LLM inference in Rust
High Performance LLM Inference Operator Library
A lightweight, portable pure C99 onnx inference engine for embedded devices with hardware acceleration support.
A lightweight inference engine supporting speculative speculative decoding (SSD).
xDiT: A Scalable Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism
DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm
FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.
High-performance Text-to-Speech server with OpenAI-compatible API, 8 voices, emotion tags, and modern web UI. Optimized for RTX GPUs.
A package for statistically rigorous scientific discovery using machine learning. Implements prediction-powered inference.
Allows you to run machine learning models locally on your ESP32 device.
Represent trained machine learning models as Pyomo optimization formulations
24,523 repositories in the index in total.
