Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Boosting your Web Services of Deep Learning Applications.
| Date | Stars |
|---|---|
| 2026-07-24 | 1242 |
| 2026-07-25 | 1242 |
| 2026-07-28 | 1242 |
| 2026-07-30 | 1242 |
| 2026-07-31 | 1242 |
| 2026-08-06 | 1242 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<h1 align="center">Service Streamer</h1>
<p align="center">
Boosting your Web Services of Deep Learning Applications.
<a href="./README_zh.md">中文README</a>
</p>
<p align="center">
</p>
<p align="center">
<a href="#what-is-service-streamer-">What is Service Streamer ?</a> •
<a href="#highlights">Highlights</a> •
<a href="#installation">Installation</a> •
<a href="#develop-bert-service-in-5-minutes">Develop BERT Service in 5 Minutes</a> •
<a href="#api">API</a> •
<a href="#benchmark">Benchmark</a> •
<a href="#faq">FAQ</a> •
</p>
<h6 align="center">
<a href="https://travis-ci.org/ShannonAI/service-streamer">
<img src="https://travis-ci.org/ShannonAI/service-streamer.svg?branch=master" alt="Build status">
</a>
• Made by ShannonAI • :globe_with_meridians: <a href="http://www.shannonai.com/">http://www.shannonai.com/</a>
</h6>
<h2 align="center">What is Service Streamer ?</h2>
A mini-batch collects data samples and is usually used in deep learning models. In this way, models can utilize the parallel computing capability of GPUs. However, requests from users for web services are usually discrete. If using conventional loop server or threaded server, GPUs will be idle dealing with one request at a time. And the latency time will be linearly increasing when there are concurrent user requests.
ServiceStreamer is a middleware for web service of machine learning applications. Queue requests from users are sampled into mini-batches. ServiceStreamer can significantly enhance the overall performance of the system by improving GPU utilization.
<h2 align="center">Highlights</h2>
- :hatching_chick: **Easy to use**: Minor changes can speed up the model ten times.
- :zap: **Fast processing speed**: Low latency for online inference of machine learning models.
- :octopus: **Good expandability**: Easy to be applied to multi-GPU scenarios for handling enormous requests.
- :crossed_swords: **Applicability**: Used with any web frameworks and/or deep learning frameworks.
<h2 align="center">Installation</h2>
Install ServiceStream by using `pip`，requires **Python >= 3.5** :
```bash
pip install service_streamer
```
<h2 align="center">Develop BERT Service in 5 Minutes</h2>
We provide a step-by-step tutorial for you to bring BERT online in 5 minutes. The service processes 1400 sentences per second.
``Text Infilling`` is a task in natural language processing: given a sentence with several words randomly removed, the model predicts those words removed through the given context.
``BERT`` has attracted a lot of attention in these two years and it achieves State-Of-The-Art results across many nlp tasks. BERT utilizes "Masked Language Model (MLM)" as one of the pre-training objectives. MLM models randomly mask some of the tokens from the input, and the objective is to predict the original vocabulary id of the masked word based on its context. MLM has similarities with text infilling. It is natural to introduce BERT to text infilling task.
1. First, we define a model for text filling task [bert_model.py](./example/bert_model.py). The `predict` function accepts a batch of sentences and returns predicted position results of the `[MASK]` token.
```python
class TextInfillingModel(object):
...
batch = ["twinkle twinkle [MASK] star.",
"Happy birthday to [MASK].",
'the answer to life, the [MASK], and everything.']
model = TextInfillingModel()
outputs = model.predict(batch)
print(outputs)
# ['little', 'you', 'universe']
```
**Note**: Please download pre-trained BERT model at first.
2. Second, utilize [Flask](https://github.com/pallets/flask) to pack predicting interfaces to Web service. [flask_example.py](./example/flask_example.py)
```python
model = TextInfillingModel()
@app.route("/naive", methods=["POST"])
def naive_predict():
inputs = request.form.getlist("s")
outputs = model.predict(inputs)
Excerpt of 15,359 characters
Read on GitHub
72
26
13
3
2
2
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:16a9e044867f97c2, topic:deep-learning, topic:pytorch, topic:tensorflow