Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
LLaMa/RWKV onnx models, quantization and testcase
| Date | Stars |
|---|---|
| 2026-07-24 | 368 |
| 2026-07-25 | 368 |
| 2026-07-28 | 368 |
| 2026-07-30 | 368 |
| 2026-08-06 | 368 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# LLaMa/RWKV onnx
## :rocket: Please read [this issue](https://github.com/tpoisonooo/llama.onnx/issues/25) for LLaMa GPU inference
Download onnx models here:
| Model | Precision | Size | URL | Demo |
| :-: | :-: | :-: | :-: | :-: |
| LLaMa-7B | fp32 | 26GB | [huggingface](https://huggingface.co/tpoisonooo/alpaca.onnx/tree/main) | [demo_llama.py](./demo_llama.py) |
| LLaMa-7B | fp16 | 13GB | [huggingface](https://huggingface.co/tpoisonooo/alpaca.onnx/tree/fp16) or [硬件模型库](https://platform.openmmlab.com/deploee) | [demo_llama.py](./demo_llama.py) |
| RWKV-4-palm-430M | fp16 | 920MB | [huggingface](https://huggingface.co/tpoisonooo/rwkv.onnx/tree/RWKV-4-Pile-430M-20220808-8066) or [硬件模型库](https://platform.openmmlab.com/deploee) | [demo_rwkv.py](./demo_rwkv.py) |
## News
05/18 release RWKV-4 onnx models, standalone script and [LLM structure comparison](https://zhuanlan.zhihu.com/p/629821901)
05/09 trt output wrong value until [issue 2928](https://github.com/NVIDIA/TensorRT/issues/2928) solved
04/19 [remove GPTQ zero point guidance](docs/remove-GPTQ-zero-point.md)
04/18 export mixed-precision quant table from [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa/pull/189)
04/11 add 13GB onnx-fp16 models
04/11 add memory pool, support 2GB RAM laptop :star:
04/10 reduce onnx model size to 26GB
04/10 support `temperature` add `topk` logits warp
04/07 add [onnxruntime demo](demo-single.py)
04/05 init project
## Features
* Release LLaMa-7B and RWKV-400M onnx models and their onnxruntime standalone demo
* No `torch` or `transformers` required
* Support memory pool, works on 2GB laptop/PC (very slow :turtle:)
Why do this ?
1. **Visualization**. `graphviz` crashed on LLaMa model. LLM visualization tool must support nest or operator folding feature
2. **Quatization**. LLM often repeat itself, just like [fractal](https://raw.githubusercontent.com/taichi-dev/public_files/master/taichi/fractal_small.gif). For LLaMa quantization, loading part of decoder backbone would be enough (400MB). It could be quantized partially
3. **Embeded device**. Small board IO error occurs when `dd` a big single file
4. **Distributed system**. Inference LLM on many hybrid (FPGA/NPU/GPGPU) devices would be simple
5. **onnx tools**. Device manufacturer has support onnx well, there is no reason to neglect it
## Usage
Here is the graph to call LLaMa (RWKV is similar):
Try LLaMa `onnxruntime` demo, no `torch` required, and the precision has been checked.
```bash
$ python3 -m pip install -r requirements.txt
$ python3 demo_llama.py ${FP16_ONNX_DIR} "bonjour"
..
# If you only have 4GB memory, use `--poolsize`
$ python3 demo_llama.py ${FP16_ONNX_DIR} "bonjour" --poolsize 4
..
Bonjour.
# Try more options
$ python3 demo_llama.py --help
```
Use [demo_rwkv.py](./demo_rwkv.py) to run RWKV:
```bash
$ python3 demo_rwkv.py ${FP16_ONNX_DIR}
```
## Export RWKV onnx
1. git clone [RWKV](https://github.com/BlinkDL/ChatRWKV) and download its models
2. copy [onnx_RWKV_in_150_lines.py](tools/onnx_RWKV_in_150_lines.py) to ChatRWKV
```bash
$ git clone https://github.com/BlinkDL/ChatRWKV --depth=1
$ cp llama.onnx/tools/onnx_RWKV_in_150_lines.py ChatRWKV
$ cd ChatRWKV
$ mkdir models
$ python3 onnx_RWKV_in_150_lines.py
```
Then you would get onnx files.
```bash
$ ls -lah models
..
```
## Export LLaMa onnx
**STEP1 Convert to HF format**
These models converted from [alpaca huggingface](https://github.com/tatsu-lab/stanford_alpaca).
- If you are using [LLaMa](https://github.com/facebookresearch/llama) or [llama.cpp](https://github.com/ggerganov/llama.cpp), convert it to HF format first. Here are steps:
```bash
# install transformers master
$ git clone https://github.com/huggingface/transformers
$ cd transformers && python3 setup.py install
..
$ cd src/transformers
$ python3 src/transformers/models/llama/convert_llama_weights_to_hf.py --input_dir ${LLaMa_PATH} --model_size 7B --output_dir ${HFExcerpt of 5,792 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:7c90d41b0ae0dc88, topic:llm, topic:transformer, topic:llama
matched fp:7c90d41b0ae0dc88, topic:onnx, topic:onnxruntime
matched fp:7c90d41b0ae0dc88, topic:quantization, desc:quantization, readme:quantization