Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
| Date | Stars |
|---|---|
| 2026-07-31 | 3600 |
| 2026-08-03 | 3600 |
| 2026-08-06 | 3605 |
Today
+5 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration [[Paper](https://arxiv.org/abs/2306.00978)][[Website](https://hanlab.mit.edu/projects/awq)] **Efficient and accurate** low-bit weight quantization (INT3/4) for LLMs, supporting **instruction-tuned** models and **multi-modal** LMs.  The current release supports: - AWQ search for accurate quantization. - Pre-computed AWQ model zoo for LLMs (Llama-1/2/3, OPT, CodeLlama, StarCoder, Vicuna, VILA, LLaVA; load to generate quantized weights). - Memory-efficient 4-bit Linear in PyTorch. - Efficient CUDA kernel implementation for fast inference (support context and decoding stage). - Examples on 4-bit inference of an instruction-tuned model (Vicuna) and **multi-modal LM** (VILA). - Chunk prefilling for faster prefilling in multi-round Q&A setting. - State-of-the-art prefilling speed of LLMs/VLMs on edge devices: [TinyChat 2.0](./tinychat). **Thanks to AWQ, TinyChat can deliver more efficient responses with LLM/VLM chatbots through 4-bit inference.** * TinyChat with LLaMA-3-8b on RTX 4090 (2.7x faster than FP16):  * TinyChat with LLaMA-3-8b on Jetson Orin (2.9x faster than FP16):  **TinyChat also supports inference with vision language models (e.g., VILA, LLaVA). In the following examples, W4A16 quantized models from VILA family are launched with TinyChat.** * TinyChat with NVILA-8B on RTX 4090 (single-image inputs):  * TinyChat with NVILA-8B on RTX 4090 (multi-image inputs):  <!-- Check out [TinyChat](tinychat), which delievers **30 tokens/second** inference performance (**3.2x faster** than FP16) for the **Llama2** chatbot on the resource-constrained NVIDIA Jetson Orin! --> * TinyChat with video reasoning: https://github.com/user-attachments/assets/b68a7a0d-5175-4030-985b-5ae0ae94f874 **Prompt:** What might be the next step according to the video? **Answer:** The next step in the video could be to place the shaped dough onto a baking sheet and let it rise before baking. **Online demo:** https://vila.hanlab.ai Check out [TinyChat](tinychat), which offers a turn-key solution for **on-device inference** of LLMs and VLMs on **resource-constrained edge platforms**. With TinyChat, it is now possible to efficiently run **large** models on **small** and **low-power** devices even without Internet connection! ## News - [2025/04] 🔥 AWQ now supports DeepSeek-R1-Distilled models. Try our example [here](https://github.com/mit-han-lab/llm-awq/blob/main/scripts/DeepSeek_R1_Distill_example.sh)! - [2025/02] AWQ now supports BF16 precision. See example [here](https://github.com/mit-han-lab/llm-awq/blob/main/scripts/qwen_example.sh). - [2024/10] 🔥⚡ Explore advancements in [TinyChat 2.0](./tinychat), the latest version with significant advancements in prefilling speed of Edge LLMs and VLMs, **1.5-1.7x** faster than the previous version of TinyChat. Please refer to the [README](./tinychat/README.md) and [blog](https://hanlab.mit.edu/blog/tinychat20) for more details. - [2024/05] 🏆 AWQ receives the **Best Paper Award** at **MLSys 2024**. 🎉 - [2024/05] 🔥 The **VILA-1.5** model family which features **video understanding** is now supported in AWQ and TinyChat. Check out out online demo powered by TinyChat [here](https://vila.hanlab.ai). Example is [here](scripts/vila15_example.sh). - [2024/05] 🔥 [AMD](https://community.amd.com/t5/ai/reduce-memory-footprint-and-improve-performance-running-llms-on/ba-p/686157) adopts AWQ to improve LLM serving efficiency. - [2024/04] 🔥 We released AWQ and TinyChat support for The **Llama-
Excerpt of 18,126 characters
Read on GitHub
Shang Yang · MIT · United States
35
Haotian (Ken) Tang · Meta · United States
26
Ji Lin · MIT, EECS
20
16
Younes B
7
Song · MIT, NVIDIA
3
3
2
1
Haofeng Huang · Tsinghua University · China
1
James Hou
1
Ikko Eltociear Ashimine · Japan
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:1e2464ca6b5e3bc1, desc:quantization