Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Ollama alternative for Rockchip NPU: An efficient solution for running AI and Deep learning models on Rockchip devices with optimized NPU support ( rkllm )
| Date | Stars |
|---|---|
| 2026-07-31 | 581 |
| 2026-08-01 | 581 |
| 2026-08-02 | 581 |
| 2026-08-04 | 581 |
| 2026-08-05 | 582 |
| 2026-08-06 | 581 |
Today
-1 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# RKLLama: LLM Server and Client for Rockchip 3588/3576
### [Version: 0.0.75](#New-Version)
Video demo ( version 0.0.1 ):
[](https://www.youtube.com/watch?v=Kj8U1OGqGPc)
## Branches
- [Without Miniconda](https://github.com/NotPunchnox/rkllama/tree/Without-miniconda): This version runs without Miniconda.
- [Rkllama Docker](https://github.com/NotPunchnox/rkllama/tree/Rkllama-Docker): A fully isolated version running in a Docker container.
- [Support All Models](https://github.com/NotPunchnox/rkllama/tree/Support-All-models): This branch ensures all models are tested before being merged into the main branch.
- [Docker Package](https://github.com/NotPunchnox/rkllama/pkgs/container/rkllama)
## Overview
A server to run and interact with LLM models optimized for Rockchip RK3588(S) and RK3576 platforms. The difference from other software of this type like [Ollama](https://ollama.com) or [Llama.cpp](https://github.com/ggerganov/llama.cpp) is that RKLLama allows models to run on the NPU.
* Version `Lib rkllm-runtime`: V 1.3.0.
* Version `Lib rknn-runtime`: V 2.3.2.
## File Structure
- **`./models`**: contains your rkllm models (wihh their rknn models if multimodal) .
- **`./lib`**: C++ `rkllm` and `rklnn` library used for inference and `fix_freqence_platform`.
- **`./app.py`**: API Rest server.
- **`./client.py`**: Client to interact with the server.
## Supported Python Versions:
- Python 3.9 to 3.12
## Tested Hardware and Environment
- **Hardware**: Orange Pi 5 Pro: (Rockchip RK3588S, NPU 6 TOPS), 16GB RAM.
- **Hardware**: Orange Pi 5 Plus: (Rockchip RK3588S, NPU 6 TOPS), 16GB RAM.
- **Hardware**: Orange Pi 5 Max: (Rockchip RK3588S, NPU 6 TOPS), 16GB RAM.
- **Hardware**: Radxa Rock 4d: (Rockchip RK3576, NPU 6 TOPS), 16GB RAM.
- **OS**: [Ubuntu 24.04 arm64.](https://joshua-riek.github.io/ubuntu-rockchip-download/)
- **OS**: Armbian Linux 6.1.99-vendor-rk35xx (Debian stable bookworm), v25.2.2.
## Main Features
- **Running models on NPU.**
- **Ollama API compatibility** - Support for:
* `/api/chat`
* `/api/generate`
* `/api/ps`
* `/api/tags`
* `/api/embed` (and legacy `/api/embeddings`)
* `/api/version`
* `/api/pull`
- **Partial OpenAI API compatibility** - Support for:
* `/v1/completions`
* `/v1/chat/completions`
* `/v1/embeddings`
* `/v1/images/generations`
* `/v1/audio/speech`
* `/v1/audio/transcriptions`
* `/v1/audio/translations` (to English only for now like OpenAI with Whisper models)
- **Tool/Function Calling** - Complete support for tool calls with multiple LLM formats (Qwen, Llama 3.2+, others).
- **Pull models directly from Huggingface.**
- **Include a API REST with documentation.**
- **Listing available models.**
- **Multiples RKLLM and RKNN models running in memory simultaniusly (parallels executions between distintct models in stream mode, FIFO if non stream)**
- **Dynamic loading and unloading of models:**
* Load the model after new request (if not in memory already)
* Unload when model expires after inactivity (default 30 min)
* Unload the oldest model in memory if new model is required to be loaded and there is not memory available in the server
*
- **Automatically Prompt Cache file saving for each chat sessions of the same llm model:**
* Allow fast response in large context chat sessions when swiching between them for the same model. Usefull if you are using a model in Openclaw (large context chat sessions) and other agents or tools (like OpenWebui) with the same model without affecting performance in any chat session.
* Allow to restore previous chat sessions for fast inference even if the model was unloaded previously from memory
* Prompt cache files are saved for 7 days by default (configurable) and then deleted automatically if not used.
- **Inference requests with streaming and non-streaming modes.**
- **Message history.**
- **Simplified custom model naming** - UsExcerpt of 27,899 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:32d12d615124e874, topic:llm-inference
matched fp:32d12d615124e874, topic:llm
