Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Manage scalable open LLM inference endpoints in Slurm clusters
| Date | Stars |
|---|---|
| 2026-07-31 | 291 |
| 2026-08-01 | 291 |
| 2026-08-02 | 291 |
| 2026-08-06 | 291 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<div align="center">
<h1>🐝 llm-swarm</h1>
<p><em>Manage scalable open LLM inference endpoints in Slurm clusters</em></p>
</div>
## Features
- Generate synthetic datasets for pretraining or fine-tuning using either local LLMs or [Inference Endpoints](https://huggingface.co/inference-endpoints/dedicated) on the Hugging Face Hub.
- Integrations with [huggingface/text-generation-inference](https://github.com/huggingface/text-generation-inference) and [vLLM](https://github.com/vllm-project/vllm) to generate text at scale.
## Prerequisites
* A Slurm cluster with Docker support,
* or access to [Inference Endpoints](https://huggingface.co/inference-endpoints/dedicated)
## Install and prepare
```bash
pip install -e .
# or pip install llm_swarm
mkdir -p .cache/
# you can customize the above docker image cache locations and change them in `templates/tgi_h100.template.slurm` and `templates/vllm_h100.template.slurm`
```
## Hello world
```bash
python examples/hello_world.py
python examples/hello_world_vllm.py
```
```python
import asyncio
import pandas as pd
from llm_swarm import LLMSwarm, LLMSwarmConfig
from huggingface_hub import AsyncInferenceClient
from transformers import AutoTokenizer
from tqdm.asyncio import tqdm_asyncio
tasks = [
"What is the capital of France?",
"Who wrote Romeo and Juliet?",
"What is the formula for water?"
]
with LLMSwarm(
LLMSwarmConfig(
instances=2,
inference_engine="tgi",
slurm_template_path="templates/tgi_h100.template.slurm",
load_balancer_template_path="templates/nginx.template.conf",
)
) as llm_swarm:
client = AsyncInferenceClient(model=llm_swarm.endpoint)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
tokenizer.add_special_tokens({"sep_token": "", "cls_token": "", "mask_token": "", "pad_token": "[PAD]"})
async def process_text(task):
prompt = tokenizer.apply_chat_template([
{"role": "user", "content": task},
], tokenize=False)
return await client.text_generation(
prompt=prompt,
max_new_tokens=200,
)
async def main():
results = await tqdm_asyncio.gather(*(process_text(task) for task in tasks))
df = pd.DataFrame({'Task': tasks, 'Completion': results})
print(df)
asyncio.run(main())
```
* [templates/tgi_h100.template.slurm](templates/tgi_h100.template.slurm) is the slurm template for TGI
* [templates/nginx.template.conf](templates/nginx.template.conf) is the nginx template for load balancing
```
(.venv) costa@login-node-1:/fsx/costa/llm-swarm$ python examples/hello_world.py
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
running sbatch --parsable slurm/tgi_1705591874_tgi.slurm
running sbatch --parsable slurm/tgi_1705591874_tgi.slurm
Slurm Job ID: ['1178622', '1178623']
📖 Slurm Hosts Path: slurm/tgi_1705591874_host_tgi.txt
✅ Done! Waiting for 1178622 to be created
✅ Done! Waiting for 1178623 to be created
✅ Done! Waiting for slurm/tgi_1705591874_host_tgi.txt to be created
obtained endpoints ['http://26.0.161.138:46777', 'http://26.0.167.175:44806']
⣽ Waiting for http://26.0.161.138:46777 to be reachable
Connected to http://26.0.161.138:46777
✅ Done! Waiting for http://26.0.161.138:46777 to be reachable
⣯ Waiting for http://26.0.167.175:44806 to be reachable
Connected to http://26.0.167.175:44806
✅ Done! Waiting for http://26.0.167.175:44806 to be reachable
Endpoints running properly: ['http://26.0.161.138:46777', 'http://26.0.167.175:44806']
✅ test generation
✅ test generation
running sudo docker run -p 47495:47495 --network hostExcerpt of 18,951 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:c69445b659349e3b, llm:description: 'Manage scalable open LLM inference endpoints in Slurm clusters' (repository description)
matched fp:c69445b659349e3b, llm:description: 'Manage scalable open LLM inference endpoints in Slurm clusters' (repository description)
matched fp:c69445b659349e3b, llm:description: 'Manage scalable open LLM inference endpoints in Slurm clusters' (repository description)