Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Runtimes and servers that execute model inference at speed and scale.
Signals: llm-inference, inference, inference-engine, model-serving, llm-serving, llama-cpp, vllm, triton-inference-server
411 results
General plug-and-play inference library for Recursive Language Models (RLMs), supporting various sandboxes.
Turn your images and videos into a personal knowledge base
Rust + CUDA LLM inference engine for Blackwell (Tuned specifically on RTX PRO 6000, RTX 5090, B200): OpenAI-compatible (+converse and ant) serving, per-model X hardware exactness gates. NVFP4/mixed (fp8 hybrid, 4o6, etc - correctness, performance, hardware specific adapted) main quant support.
LLM plugin for models hosted by OpenRouter
NVIDIA AITune is an inference toolkit designed for tuning and deploying Deep Learning models with a focus on NVIDIA GPUs.
No description
LLM speculative inference server for heterogeneous hardware & consumer GPUs
📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉
支持按轴凹总力战, 无缝制造三解, 用于实现蔚蓝档案自动化的程序( Steam已适配 )
A Pure Rust based LLM, VLM, VLA, TTS, OCR Inference Engine, powering by Candle & Rust. Alternate to your llama.cpp but much more simpler and cleaner..
Lightweight, Modular, Kubernetes-native AI serving platform for scalable model serving.
Robust realtime face and facial landmark tracking on CPU with Unity integration
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
Home for "How To Scale Your Model", a short blog-style textbook about scaling LLMs on TPUs
Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust 🦀
A plug-and-play compiler that delivers free-lunch optimizations for both inference and training.
LiteRT and LiteRT-LM sample apps, model recipes, agent skills and utilities.
High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.
QualityScaler - image/video AI upscaler app
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
A simple TUI for serving local LLM models. Pick a model, pick a backend, serve it
Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.
Unified framework for building enterprise RAG pipelines with small, specialized models
List of awesome hosting sorted by minimal plan price
24,545 repositories in the index in total.
