Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Runtimes and servers that execute model inference at speed and scale.
Signals: llm-inference, inference, inference-engine, model-serving, llm-serving, llama-cpp, vllm, triton-inference-server
411 results
LLM Inference benchmark
No description
DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including CUDA, x86 and ARMv9.
Speech-to-Text interface for Emacs using OpenAI's whisper model and whisper.cpp as inference engine.
High-efficiency LLM inference engine in C++/CUDA. Run Llama 70B on RTX 3090.
FlexAttention based, minimal vllm-style inference engine for fast Gemma 2 inference.
a fast cross platform AI inference engine 🤖 using Rust 🦀 and WebGPU 🎮
Inference Engine samples internal development repository. Contains example and template projects for Sentis package use.
~1300 line, minimal, extensible LLM inference engine built from scratch.
Stock inference engine using Spring XD, Apache Geode / GemFire and Spark ML Lib.
LLM inference engine written in .NET
CLI for running large numbers of coding agents in parallel with git worktrees
OpenAlpha_Evolve is an open-source Python framework inspired by the groundbreaking research on autonomous coding agents like DeepMind's AlphaEvolve.
Self-evaluating interview for AI coders
DEEPPOWERS is a Fully Homomorphic Encryption (FHE) framework built for MCP (Model Context Protocol), aiming to provide end-to-end privacy protection and high-efficiency computation for the upstream and downstream ecosystem of the MCP protocol.
A tool for generating function arguments and choosing what function to call with local LLMs
Android 17 local LLM prototype with Jetpack Compose and ONNX Runtime for offline AI inference experiments.
From-scratch PyTorch implementation of Google's TurboQuant (ICLR 2026) for LLM KV cache compression. 5x compression at 3-bit with 99.5% attention fidelity.
MiniLLM is a minimal system for running modern LLMs on consumer-grade GPUs
Dynamic Memory Management for Serving LLMs without PagedAttention
A throughput-oriented high-performance serving framework for LLMs
Efficient and easy multi-instance LLM serving
🤖 TUI for LLMs
The Triton TensorRT-LLM Backend
24,545 repositories in the index in total.
