Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Runtimes and servers that execute model inference at speed and scale.
Signals: llm-inference, inference, inference-engine, model-serving, llm-serving, llama-cpp, vllm, triton-inference-server
410 results
InfiniTensor is a high-performance inference engine tailored for GPUs and AI accelerators. Its design focuses on effective deployment and swift academic validation.
High-efficiency LLM inference engine in C++/CUDA. Run Llama 70B on RTX 3090.
FlexAttention based, minimal vllm-style inference engine for fast Gemma 2 inference.
a fast cross platform AI inference engine 🤖 using Rust 🦀 and WebGPU 🎮
Inference Engine samples internal development repository. Contains example and template projects for Sentis package use.
~950 line, minimal, extensible LLM inference engine built from scratch.
Stock inference engine using Spring XD, Apache Geode / GemFire and Spark ML Lib.
LLM inference engine written in .NET
[ACL 2026] Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
CLI for running large numbers of coding agents in parallel with git worktrees
OpenAlpha_Evolve is an open-source Python framework inspired by the groundbreaking research on autonomous coding agents like DeepMind's AlphaEvolve.
Self-evaluating interview for AI coders
DEEPPOWERS is a Fully Homomorphic Encryption (FHE) framework built for MCP (Model Context Protocol), aiming to provide end-to-end privacy protection and high-efficiency computation for the upstream and downstream ecosystem of the MCP protocol.
A tool for generating function arguments and choosing what function to call with local LLMs
From-scratch PyTorch implementation of Google's TurboQuant (ICLR 2026) for LLM KV cache compression. 5x compression at 3-bit with 99.5% attention fidelity.
MiniLLM is a minimal system for running modern LLMs on consumer-grade GPUs
A throughput-oriented high-performance serving framework for LLMs
Efficient and easy multi-instance LLM serving
Running any GGUF SLMs/LLMs locally, on-device in Android
🤖 TUI for LLMs
Running a LLM on the ESP32
Compare compatible open-source local LLM projects by their metrics to assess popularity and activeness.
Build and deploy AI-powered APIs in seconds
Serverless LLM Serving for Everyone.
24,523 repositories in the index in total.
