Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
| Date | Stars |
|---|---|
| 2026-07-24 | 904 |
| 2026-07-25 | 904 |
| 2026-07-28 | 904 |
| 2026-07-30 | 904 |
| 2026-07-31 | 903 |
| 2026-08-06 | 903 |
Today
— stars today
This week
-1 stars this week
This month
— stars this month
Momentum
5.0
growth rate 0.00%/day
<p align="center">
<img src="https://user-images.githubusercontent.com/38581401/240117836-f06199ba-c80d-413a-9cb4-5adc76316bda.png" width=90% alt="MOSEC" />
</p>
<p align="center">
<a href="https://discord.gg/Jq5vxuH69W">
<img alt="discord invitation link" src="https://img.shields.io/discord/916177932236521533?style=flat&logo=discord&color=blue&cacheSeconds=60">
</a>
<a href="https://pypi.org/project/mosec/">
<img src="https://img.shields.io/pypi/v/mosec?style=flat&logo=python&color=blue&cacheSeconds=60" alt="PyPI version" height="20">
</a>
<a href="https://anaconda.org/conda-forge/mosec">
<img src="https://anaconda.org/conda-forge/mosec/badges/version.svg" alt="conda-forge">
</a>
<a href="https://pypi.org/project/mosec">
<img src="https://img.shields.io/pypi/pyversions/mosec" alt="Python Version" />
</a>
<a href="https://pepy.tech/project/mosec">
<img src="https://static.pepy.tech/badge/mosec/month" alt="PyPi monthly Downloads" height="20">
</a>
</p>
<p align="center">
<i>Model Serving made Efficient in the Cloud.</i>
</p>
## Introduction
<p align="center">
<img src="https://user-images.githubusercontent.com/38581401/234162688-efd74e46-4063-4624-ac32-b197e4d8e56b.png" width=70% alt="MOSEC" />
</p>
Mosec is a high-performance and flexible model serving framework for building ML model-enabled backend and microservices. It bridges the gap between any machine learning models you just trained and the efficient online service API.
- **Highly performant**: web layer and task coordination built with Rust 🦀, which offers blazing speed in addition to efficient CPU utilization powered by async I/O
- **Ease of use**: user interface purely in Python 🐍, by which users can serve their models in an ML framework-agnostic manner using the same code as they do for offline testing
- **Dynamic batching**: aggregate requests from different users for batched inference and distribute results back
- **Pipelined stages**: spawn multiple processes for pipelined stages to handle CPU/GPU/IO mixed workloads
- **Cloud friendly**: designed to run in the cloud, with the model warmup, graceful shutdown, and Prometheus monitoring metrics, easily managed by Kubernetes or any container orchestration systems
- **Do one thing well**: focus on the online serving part, users can pay attention to the model optimization and business logic
## Installation
Mosec requires Python 3.7 or above. Install the latest [PyPI package](https://pypi.org/project/mosec/) for Linux or macOS with:
```shell
pip install -U mosec
# or install with conda
conda install conda-forge::mosec
# or install with pixi
pixi add mosec
```
To build from the source code, install [Rust](https://www.rust-lang.org/) and run the following command:
```shell
make package
```
You will get a mosec wheel file in the `dist` folder.
## Usage
We demonstrate how Mosec can help you easily host a pre-trained stable diffusion model as a service. You need to install [diffusers](https://github.com/huggingface/diffusers) and [transformers](https://github.com/huggingface/transformers) as prerequisites:
```shell
pip install --upgrade diffusers[torch] transformers
```
### Write the server
<details>
<summary>Click me for server codes with explanations.</summary>
Firstly, we import the libraries and set up a basic logger to better observe what happens.
```python
from io import BytesIO
from typing import List
import torch # type: ignore
from diffusers import StableDiffusionPipeline # type: ignore
from mosec import Server, Worker, get_logger
from mosec.mixin import MsgpackMixin
logger = get_logger()
```
Then, we **build an API** for clients to query a text prompt and obtain an image based on the [stable-diffusion-v1-5 model](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) in just 3 steps.
1) Define your service as a class which inherits `mosec.Worker`. Here we also inherit `MsgpackMixin` to employ the [msgpack](https://msgpack.org/index.Excerpt of 17,341 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:e1bc2748c096bebb, topic:deep-learning, topic:pytorch, topic:tensorflow
matched fp:e1bc2748c096bebb, topic:model-serving, topic:llm-serving, desc:model serving
