Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A pytorch quantization backend for optimum
| Date | Stars |
|---|---|
| 2026-07-24 | 1048 |
| 2026-07-25 | 1048 |
| 2026-07-28 | 1048 |
| 2026-07-30 | 1048 |
| 2026-07-31 | 1050 |
| 2026-08-01 | 1051 |
| 2026-08-03 | 1051 |
| 2026-08-04 | 1052 |
| 2026-08-06 | 1053 |
Today
+1 stars today
This week
+5 stars this week
This month
— stars this month
Momentum
9.0
growth rate 0.48%/day
# Optimum Quanto
> This project is currently in maintenance mode. We accept pull requests only for minor bug fixes, documentation improvements, and other maintenance tasks. Major new features or breaking changes are unlikely to be merged. For production-ready quantization features or active development, consider alternative projects such as [bitsandbytes](https://github.com/bitsandbytes-foundation/bitsandbytes) or [torchAO](https://github.com/pytorch/ao).
🤗 Optimum Quanto is a pytorch quantization backend for [optimum](https://huggingface.co/docs/optimum/en/index).
It has been designed with versatility and simplicity in mind:
- all features are available in eager mode (works with non-traceable models),
- quantized models can be placed on any device (including CUDA and MPS),
- automatically inserts quantization and dequantization stubs,
- automatically inserts quantized functional operations,
- automatically inserts quantized modules (see below the list of supported modules),
- provides a seamless workflow from a float model to a dynamic to a static quantized model,
- serialization compatible with pytorch `weight_only` and 🤗 `safetensors`,
- accelerated matrix multiplications on CUDA devices (int8-int8, fp16-int4, bf16-int8, bf16-int4),
- supports int2, int4, int8 and float8 weights,
- supports int8 and float8 activations.
Features yet to be implemented:
- dynamic activations smoothing,
- kernels for all mixed matrix multiplications on all devices,
- compatibility with [torch compiler](https://pytorch.org/docs/stable/torch.compiler.html) (aka dynamo).
## Performances
In a nutshell:
- accuracy: models compiled with `int8`/`float8` weights and `float8` activations are very close to the full-precision models,
- latency: whenever optimized kernels are available, the inference of quantized model is comparable with the full-precision models when quantizing only the model weights,
- device memory: approximately divided by float bits / integer bits.
The paragraph below is just an example. Please refer to the `bench` folder for detailed results per use-case of model.
### meta-llama/Meta-Llama-3.1-8B
<div class="row"><center>
<div class="column">
<img src="https://github.com/huggingface/optimum-quanto/blob/main/bench/generation/charts/meta-llama-Meta-Llama-3.1-8B_bf16_Perplexity.png" alt="meta-llama/Meta-Llama-3.1-8B WikiText perplexity">
</div>
</center>
</div>
<div class="row"><center>
<div class="column">
<img src="https://github.com/huggingface/optimum-quanto/blob/main/bench/generation/charts/meta-llama-Meta-Llama-3.1-8B_bf16_Latency__ms_.png" alt="meta-llama/Meta-Llama-3.1-8B Latency">
</div>
</center>
</div>
## Installation
Optimum Quanto is available as a pip package.
```sh
pip install optimum-quanto
```
## Quantization workflow for Hugging Face models
`optimum-quanto` provides helper classes to quantize, save and reload Hugging Face quantized models.
### LLM models
The first step is to quantize the model
```python
from transformers import AutoModelForCausalLM
from optimum.quanto import QuantizedModelForCausalLM, qint4
model = AutoModelForCausalLM.from_pretrained('meta-llama/Meta-Llama-3-8B')
qmodel = QuantizedModelForCausalLM.quantize(model, weights=qint4, exclude='lm_head')
```
Note: the model quantized weights will be frozen. If you want to keep them unfrozen to train them you need to use `optimum.quanto.quantize` directly.
The quantized model can be saved using `save_pretrained`:
```python
qmodel.save_pretrained('./Llama-3-8B-quantized')
```
It can later be reloaded using `from_pretrained`:
```python
from optimum.quanto import QuantizedModelForCausalLM
qmodel = QuantizedModelForCausalLM.from_pretrained('Llama-3-8B-quantized')
```
### Diffusers models
You can quantize any of the submodels inside a diffusers pipeline and seamlessly include them later in another pipeline.
Here we quantize the `transformer` of a `Pixart` pipeline.
```python
from diffusers import PixArtTransformer2DExcerpt of 12,180 characters
Read on GitHub
David Corvoysier · @huggingface
644
Sayak Paul · @huggingface · India
15
Fanli Lin · Intel · China
10
Younes B
9
9
Marc Sun · Hugging Face
8
Dmitry Rogozhkin · Intel
6
5
5
4
3
3
3
Yao Matrix · Intel · China
3
Tolga Cangöz
2
Pauline Bailly-Masson
2
Aryan V S
2
Guillaume LEGENDRE
2
1
Funtowicz Morgan · @huggingface · France
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:3b536ad04d953adf, topic:quantization, desc:quantization, readme:quantization
matched fp:3b536ad04d953adf, topic:pytorch