Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
LLM Inference benchmark
| Date | Stars |
|---|---|
| 2026-07-31 | 438 |
| 2026-08-02 | 437 |
| 2026-08-06 | 437 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# llm-inference-benchmark LLM Inference benchmark ## Inference frameworks | Framework | Producibility**** | Docker Image | API Server | OpenAI API Server | WebUI | Multi Models** | Multi-node | Backends | Embedding Model | | --------- | ------------ | ----------- | ----------------- | ----- | ------------ | ------------ | -------- | --------------- | --------- | | [text-generation-webui](https://github.com/oobabooga/text-generation-webui) | Low | Yes | Yes | Yes | Yes | No | No | Transformers/llama.cpp/ExLlama/ExLlamaV2/AutoGPTQ/AutoAWQ/GPTQ-for-LLaMa/CTransformers | No | | [OpenLLM](https://github.com/bentoml/OpenLLM) | High | Yes | Yes | Yes | No | With [BentoML](https://github.com/bentoml/BentoML) | With [BentoML](https://github.com/bentoml/BentoML) | Transformers(int8,int4,gptq), vLLM(awq/squeezellm), TensorRT | No | | [vLLM](https://github.com/vllm-project/vllm)* | High | Yes | Yes | Yes | No | No | Yes(With [Ray](https://docs.ray.io/en/latest/ray-core/starting-ray.html)) | vLLM | No | | [Xinference](https://github.com/xorbitsai/inference) | High | Yes | Yes | Yes | Yes | Yes | Yes | Transformers/vLLM/TensorRT/GGML | Yes | | [TGI](https://github.com/huggingface/text-generation-inference)*** | Medium | Yes | Yes | No | No | No | No | Transformers/AutoGPTQ/AWQ/EETP/vLLM/ExLlama/ExLlamaV2 | No | | [ScaleLLM](https://github.com/vectorch-ai/ScaleLLM) | Medium | Yes | Yes | Yes | Yes | No | No | Transformers/AutoGPTQ/AWQ/vLLM/ExLlama/ExLlamaV2 | No | | [FastChat](https://github.com/lm-sys/FastChat) | High | Yes | Yes | Yes | Yes | Yes | Yes | Transformers/AutoGPTQ/AWQ/vLLM/ExLlama/ExLlamaV2 | Yes | - *vLLM/TGI can also serve as a backend. - **Multi Models: Capable of loading multiple models simultaneously. - ***TGI does not support chat mode; manual parsing of the prompt is required. ## Inference backends | Backend | Device | Compatibility** | PEFT Adapters* | Quatisation | Batching | Distributed Inference | Streaming | | ------- | ------ | ------------- | ---------- | -------- | ----------- | --------- | ----------- | | [Transformers](https://github.com/huggingface/transformers) | GPU | High | Yes | [bitsandbytes](https://github.com/TimDettmers/bitsandbytes)(int8/int4), [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ)(gptq), AutoAWQ(awq) | Yes | [accelerate](https://huggingface.co/docs/accelerate/index) | Yes | | [vLLM](https://github.com/vllm-project/vllm) | GPU | High | No | awq/squeezellm | Yes | Yes | Yes | | [ExLlamaV2](https://github.com/turboderp/exllamav2) | GPU/CPU | Low | No | GPTQ | Yes | Yes | Yes | | [TensorRT](https://github.com/NVIDIA/TensorRT-LLM) | GPU | Medium | No | [some models](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/precision.md) | Yes | Yes | Yes | | [Candle](https://github.com/huggingface/candle) | GPU/CPU | Low | No | No | Yes | Yes | Yes | | [CTranslate2](https://github.com/OpenNMT/CTranslate2) | GPU | Low | No | Yes | Yes | Yes | Yes | | [TGI](https://github.com/huggingface/text-generation-inference) | GPU | Medium | Yes | awq/eetq/gptq/bitsandbytes | Yes | Yes | Yes | | [llama-cpp](https://github.com/ggerganov/llama.cpp)*** | GPU/CPU | High | No | GGUF/GPTQ | Yes | No | Yes | | [lmdeploy](https://github.com/InternLM/lmdeploy) | GPU | Medium | No | AWQ | Yes | Yes | Yes | [Deepspeed-FastGen](https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen) | GPU | Low | No | No | Yes | Yes | Yes | - *PEFT Adapters: support to load seperate PEFT adapters(mostly lora). - **Compatibility: High: Compatible with most models; Medium: Compatible with some models; Low: Compatible with few models. - ***llama.cpp's Python binding: [llama-cpp-python](https://github.com/abetlen/llama-cpp-python). ## Benchmark Hardware: - GPU: 1x NVIDIA RTX4090 24GB - CPU: Intel Core i9-13900K - Memory: 96GB Software: - VM: WSL2 on Windows 11 - Guest OS: Ubuntu 22.04 - NVIDIA Driver Version: 536.67 - CUDA Version: 12.2 - PyTorch: 2.1.1 Model: - BFloat16: [01-ai/Yi-6B-Chat](https://hug
Excerpt of 6,313 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:a42808491fdfe950, llm:Repository name and description: 'LLM Inference benchmark' — indicates benchmarking inference performance for large language models.
matched fp:a42808491fdfe950, llm:Repository name and description: 'LLM Inference benchmark' — indicates benchmarking inference performance for large language models.
matched fp:a42808491fdfe950, llm:Repository name and description: 'LLM Inference benchmark' — indicates benchmarking inference performance for large language models.