Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
💬 Chatbot web app + HTTP and Websocket endpoints for LLM inference with the Petals client
| Date | Stars |
|---|---|
| 2026-07-24 | 320 |
| 2026-07-25 | 320 |
| 2026-07-28 | 320 |
| 2026-07-30 | 320 |
| 2026-07-31 | 320 |
| 2026-08-06 | 320 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Petals Chat
A chatbot [web app](https://chat.petals.dev) + HTTP and WebSocket endpoints for LLM inference with the [Petals](https://petals.dev) client
## Interactive Chat
<div align="center">
<img src="https://i.imgur.com/QVTzc6u.png" width="600px">
</div>
You can try it out at **https://chat.petals.dev** or run the backend on your server using these commands:
```bash
git clone https://github.com/petals-infra/chat.petals.dev.git
cd chat.petals.dev
pip install -r requirements.txt
flask run --host=0.0.0.0 --port=5000
```
🦙 **Want to serve Llama 2?** Request access to its weights at the ♾️ [Meta AI website](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) and 🤗 [Model Hub](https://huggingface.co/meta-llama/Llama-2-70b-hf), then run `huggingface-cli login` in the terminal before starting the web app. If you don't want Llama 2, just remove the `meta-llama` models from [config.py](https://github.com/petals-infra/chat.petals.dev/blob/main/config.py).
🦄 **Deploying with Gunicorn.** In production, we recommend using gunicorn instead of the Flask dev server:
```bash
gunicorn app:app --bind 0.0.0.0:5000 --worker-class gthread --threads 100 --timeout 1000
```
The chat uses the WebSocket API under the hood.
## APIs
The backend provides two APIs endpoints:
- [WebSocket API](#websocket-api-apiv2generate) (`/api/v2/generate`, recommended)
- [HTTP API](#http-api-apiv1) (`/api/v1/...`)
Please use the WebSocket API when possible - it is much faster, more powerful, and consumes less resources.
If you develop your own web app, you can use our endpoint at `https://chat.petals.dev/api/...` for research and development, then set up your own backend for production using the commands above.
> **Note:** We do not recommend using the endpoint at `https://chat.petals.dev/api/...` in production. It has a limited throughput, and we may pause or stop it any time.
<details>
<summary><b>Endpoint's system requirements</b></summary>
- If you use a CPU-only server, you need enough RAM to fit embeddings for all models (see the table below).
If your CPU supports AVX512, the embeddings will be loaded in 16-bit, otherwise they will be loaded in 32-bit (= 2x more memory).
This is because multiplying 16-bit weights without AVX512 is slow and may introduce a slowdown of 1-2 sec/token.
AVX512 support is available on late Intel Xeon CPUs
(e.g., on [DigitalOcean](https://digitalocean.com) droplets with a dedicated CPU).
- If you use a GPU server, you need enough GPU memory to fit the embeddings for all models.
The embeddings will be loaded in 16-bit.
- You don't have to serve all models. If you don't have enough memory, remove some models in [config.py](config.py).
| Model family | Embeds in 16-bit | Embeds in 32-bit |
| --- | --- | --- |
| Llama 2 (70B, 70B-Chat), Llama-65B, Guanaco-65B | 1.05 GB | 2.1 GB |
| BLOOM-176B, BLOOMZ-176B | 7.19 GB | 14.38 GB |
</details>
## WebSocket API (`/api/v2/generate`)
This API implies that you open a WebSocket connection and exchange JSON-encoded requests and responses.
This may be done from any programming language.
<details>
<summary><b>Example code (Javascript)</b></summary>
This code opens an inference session with the [stabilityai/StableBeluga2](https://huggingface.co/stabilityai/StableBeluga2) model, sends the prompt "A cat sat on",
and samples new tokens until the total length reaches 30 tokens. Sampling is done with [temperature](https://huggingface.co/blog/how-to-generate#sampling) = 0.6 and [top_p](https://huggingface.co/blog/how-to-generate#top-p-nucleus-sampling) = 0.9.
```javascript
const ws = new WebSocket(`wss://chat.petals.dev/api/v2/generate`);
ws.onopen = () => {
const prompt = "A cat sat on";
const maxLength = 30;
ws.send(JSON.stringify({
type: "open_inference_session", model: "stabilityai/StableBeluga2", max_length: maxLength
}));
ws.send(JSON.stringify({
type: "generate", inputs: prompt, max_length: maxLength, do_sample: 1,Excerpt of 9,437 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:63b4d7f224bd2bdf, topic:large-language-models, topic:transformer, topic:gpt