Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model
| Date | Stars |
|---|---|
| 2026-07-24 | 1579 |
| 2026-07-25 | 1580 |
| 2026-07-28 | 1580 |
| 2026-07-30 | 1580 |
| 2026-07-31 | 1581 |
| 2026-08-06 | 1581 |
Today
— stars today
This week
+1 stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.06%/day
# rwkv.cpp This is a port of [BlinkDL/RWKV-LM](https://github.com/BlinkDL/RWKV-LM) to [ggerganov/ggml](https://github.com/ggerganov/ggml). Besides the usual **FP32**, it supports **FP16**, **quantized INT4, INT5 and INT8** inference. This project is **focused on CPU**, but cuBLAS is also supported. This project provides [a C library rwkv.h](rwkv.h) and [a convinient Python wrapper](python%2Frwkv_cpp%2Frwkv_cpp_model.py) for it. [RWKV](https://arxiv.org/abs/2305.13048) is a large language model architecture. In contrast to Transformer with `O(n^2)` attention, RWKV requires only state from previous step to calculate logits. This makes RWKV very CPU-friendly on large context lenghts. This project supports RWKV [v4](https://huggingface.co/BlinkDL/rwkv-4-pile-14b), [v5](https://huggingface.co/BlinkDL/rwkv-5-world), [v6](https://huggingface.co/BlinkDL/rwkv-6-world) and the latest [v7](https://huggingface.co/BlinkDL/rwkv-7-world) architectures. Loading LoRA checkpoints in [Blealtan's format](https://github.com/Blealtan/RWKV-LM-LoRA) is supported through [merge_lora_into_ggml.py script](rwkv%2Fmerge_lora_into_ggml.py). <!-- TODO: Update data below --> ## Quality and performance If you use `rwkv.cpp` for anything serious, please [test all available formats for perplexity and latency](rwkv%2Fmeasure_pexplexity.py) on a representative dataset, and decide which trade-off is best for you. Below table is for reference only. Measurements were made on 4C/8T x86 CPU with AVX2, 4 threads. The models are `RWKV v4 Pile 169M`, `RWKV v4 Pile 1.5B`. | Format | Perplexity (169M) | Latency, ms (1.5B) | File size, GB (1.5B) | |-----------|-------------------|--------------------|----------------------| | `Q4_0` | 17.507 | *76* | **1.53** | | `Q4_1` | 17.187 | **72** | 1.68 | | `Q5_0` | 16.194 | 78 | *1.60* | | `Q5_1` | 15.851 | 81 | 1.68 | | `Q8_0` | *15.652* | 89 | 2.13 | | `FP16` | **15.623** | 117 | 2.82 | | `FP32` | **15.623** | 198 | 5.64 | ### With cuBLAS Measurements were made on Intel i7 13700K & NVIDIA 3060 Ti 8 GB. The model is `RWKV-4-Pile-169M`, 12 layers were offloaded to GPU. Latency per token in ms shown. | Format | 1 thread | 2 threads | 4 threads | 8 threads | 24 threads | |--------|----------|-----------|-----------|-----------|------------| | `Q4_0` | 7.9 | 6.2 | 6.9 | 8.6 | 20 | | `Q4_1` | 7.8 | 6.7 | 6.9 | 8.6 | 21 | | `Q5_1` | 8.1 | 6.7 | 6.9 | 9.0 | 22 | | Format | 1 thread | 2 threads | 4 threads | 8 threads | 24 threads | |--------|----------|-----------|-----------|-----------|------------| | `Q4_0` | 59 | 51 | 50 | 54 | 94 | | `Q4_1` | 59 | 51 | 49 | 54 | 94 | | `Q5_1` | 77 | 69 | 67 | 72 | 101 | Note: since cuBLAS is supported only for `ggml_mul_mat()`, we still need to use few CPU resources to execute remaining operations. ### With hipBLAS Measurements were made on CPU AMD Ryzen 9 5900X & GPU AMD Radeon RX 7900 XTX. The model is `RWKV-novel-4-World-7B-20230810-ctx128k`, 32 layers were offloaded to GPU. Latency per token in ms shown. | Format | 1 thread | 2 threads | 4 threads | 8 threads | 24 threads | |--------|----------|-----------|-----------|-----------|------------| | `f16` | 94 | 91 | 94 | 106 | 944 | | `Q4_0` | 83 | 77 | 75 | 110 | 1692 | | `Q4_1` | 85 | 80 | 85 | 93 | 1691 | | `Q5_1` | 83 | 78 | 83 | 90 | 1115 | Note: same as cuBLAS, hipBLAS only supports `ggml_mul_mat()`, we
Excerpt of 9,721 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:489183ab6938019a, topic:llm, topic:language-model
matched fp:489183ab6938019a, topic:quantization, readme:quantized, desc:int8
matched fp:489183ab6938019a, topic:deep-learning
