Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Runtimes and servers that execute model inference at speed and scale.
Signals: llm-inference, inference, inference-engine, model-serving, llm-serving, llama-cpp, vllm, triton-inference-server
Compare compatible open-source local LLM projects by their metrics to assess popularity and activeness.
Build and deploy AI-powered APIs in seconds
Serverless LLM Serving for Everyone.
a lightweight LLM model inference framework
Official implementation of "Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding"
GraphRAG using Local LLMs - Features robust API and multiple apps for Indexing/Prompt Tuning/Query/Chat/Visualizing/Etc. This is meant to be the ultimate GraphRAG/KG local LLM app.
Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads
Everything I know about running LLMs locally
Based on Nano-vLLM, a simple replication of vLLM with self-contained paged attention and flash attention implementation
计图大模型推理库，具有高性能、配置要求低、中文支持好、可移植等特点
A fast inference library for running LLMs locally on modern consumer-class GPUs
AI-powered penetration testing assistant using local LLM on linux (Parrot OS)
Fast inference from large lauguage models via speculative decoding
闻达：一个LLM调用平台。目标为针对特定环境的高效内容生成，同时考虑个人和中小企业的计算资源局限性，以及知识安全和私密性问题
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
A framework for serving and evaluating LLM routers - save LLM costs without compromising quality
Example code and documentation on how to get Stable Diffusion running with ONNX FP16 models on DirectML. Can run accelerated on all DirectML supported cards including AMD and Intel.
WeDLM: The fastest diffusion language model with standard causal attention and native KV cache compatibility, delivering real speedups over vLLM-optimized baselines.
LLM.swift is a simple and readable library that allows you to interact with large language models locally with ease for macOS, iOS, watchOS, tvOS, and visionOS.
Analyze the inference of Large Language Models (LLMs). Analyze aspects like computation, storage, transmission, and hardware roofline model in a user-friendly interface.
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
Disaggregated serving system for Large Language Models (LLMs).
LLM-PowerHouse: Unleash LLMs' potential through curated tutorials, best practices, and ready-to-use code for custom training and inferencing.
24,545 repositories in the index in total.
