Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Code for Neurips24 paper: QuaRot, an end-to-end 4-bit inference of large language models.
| Date | Stars |
|---|---|
| 2026-07-31 | 528 |
| 2026-08-05 | 530 |
| 2026-08-06 | 530 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# <img src="img/carrot.png" alt="Your Image" width="40" height="45">QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
This repository contains the code for [**QuaRot**: Outlier-Free 4-Bit Inference in Rotated LLMs](https://arxiv.org/abs/2404.00456).
## Abstract
We introduce QuaRot, a new **Qua**ntization scheme based on **Rot**ations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This *computational invariance* is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4-bits, without any channels identified for retention in higher precision. Our quantized **LLaMa2-70B** model has losses of at most **0.29 WikiText perplexity** and retains **99% of the zero-shot** performance.
## Usage
Compile the QuaRot kernels using the following commands:
```bash
git clone https://github.com/spcl/QuaRot.git
cd QuaRot
pip install -e . # or pip install .
```
For simulation results, check [fake_quant](https://github.com/spcl/QuaRot/tree/main/fake_quant) directory.
### Star History
[](https://star-history.com/#spcl/QuaRot&Date)
## Citation
The full citation is
```
@article{ashkboos2024quarot,
title={QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs},
author={Ashkboos, Saleh and Mohtashami, Amirkeivan and Croci, Maximilian L and Li, Bo and Jaggi, Martin and Alistarh, Dan and Hoefler, Torsten and Hensman, James},
journal={arXiv preprint arXiv:2404.00456},
year={2024}
}
```
Excerpt of 1,902 characters
Read on GitHub
13
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:ba1a70f32bfe1f38, desc:4-bit