Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Runtimes and servers that execute model inference at speed and scale.
Signals: llm-inference, inference, inference-engine, model-serving, llm-serving, llama-cpp, vllm, triton-inference-server
411 results
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
Fast and accurate AI powered file content types detection
A Datacenter Scale Distributed Inference Serving Framework
ML software (llama.cpp, ComfyUI, Strata, Maya, ...) builds for AMD gfx906 GPUs, e.g. Radeon VII / MI50 / MI60
Distribute and run LLMs with a single file.
Community maintained hardware plugin for vLLM on Huawei Ascend
Silero VAD: pre-trained enterprise-grade Voice Activity Detector
The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
The Swiss Army Knife of Offline AI. Chat, see, speak, and generate images on your phone or Mac — GGUF LLMs, vision, Whisper speech-to-text, Stable Diffusion, tool calling, and local-network servers. Runs on your CPU, GPU, or NPU. No account, no API key, zero data leaves your device.
Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly.
📚 Jupyter notebook tutorials for OpenVINO™
A cosy home for your LLMs.
NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.
Open-source, local-first video editor where creators and AI agents edit the same real timeline.
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
Access to Anthropic's safety-first language model APIs via Go
RDNA-native LLM inference engine in Rust.
Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.
Running any GGUF SLMs/LLMs locally, on-device in Android
llm-d Router: The intelligent entry point for inference requests
Communicate with an LLM provider using a single interface
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
24,545 repositories in the index in total.
