Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Runtimes and servers that execute model inference at speed and scale.
Signals: llm-inference, inference, inference-engine, model-serving, llm-serving, llama-cpp, vllm, triton-inference-server
410 results
An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more
A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）
FastAPI Skeleton App to serve machine learning models production-ready.
Examples for using ONNX Runtime for machine learning inferencing.
MXNetJS: Javascript Package for Deep Learning in Browser (without server)
⚡VoltaML is a lightweight library to convert and run your ML/DL deep learning models in high performance inference runtimes like TensorRT, TorchScript, ONNX and TVM.
ezkl is an engine for doing inference for deep learning models and other computational graphs in a zk-snark (ZKML). Use it from Python, Javascript, or the command line.
vendor independent TinyML deep learning library, compiler and inference framework microcomputers and micro-controllers
Deep learning inference nodes for ROS / ROS2 with support for NVIDIA Jetson and TensorRT
Open Source Library for GPU-Accelerated Execution of Trained Deep Convolutional Neural Networks on Android
Neural Network proposal for WASI
onnx-go gives the ability to import a pre-trained neural network within Go without being linked to a framework or library.
Deep neural network library and toolkit to do high performace inference on NVIDIA jetson platforms
dabnn is an accelerated binary neural networks inference framework for mobile platform
FeatherCNN is a high performance inference engine for convolutional neural networks.
Real-time neural network inferencing
A higher-level Neural Network library for microcontrollers.
Neural networks module for Redis
Recipes to scale inference-time compute of open models
No description
A cross-platform browser ML framework.
A high-performance HTTP/2-enabled proxy server designed specifically to enable Cursor IDE's Composer to use DeepSeek's and OpenRouter's language models. This proxy translates OpenAI-compatible API requests to DeepSeek/OpenRouter API format, allowing Cursor's Composer and other OpenAI API-compatible tools to seamlessly work with these models.
Provide OpenAI style API for ChatGLM-6B and Chinese Embeddings Model
Ollama for classical ML models. AOT compiler that turns XGBoost, LightGBM, scikit-learn, CatBoost & ONNX models into native C99 inference code. One command to load, one command to serve. 336x faster than Python inference.
24,523 repositories in the index in total.
