Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Static suckless single batch CUDA-only qwen3-0.6B mini inference engine
| Date | Stars |
|---|---|
| 2026-07-24 | 556 |
| 2026-07-25 | 556 |
| 2026-07-28 | 556 |
| 2026-07-30 | 556 |
| 2026-07-31 | 556 |
| 2026-08-06 | 556 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# qwen600.cu <p align="center"> <img src="assets/banner.png" width="429" height="139" alt="banner_"> </p> While studying and practicing CUDA & GPGPU, thought why not make an inference engine from scratch ? So, chose [QWEN3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) model, small model than can run smoothly on my `RTX 3050 8GB` VRAM. My intention was (and still) to build educational program to learn about LLMs & transformers while maintaining practice in CUDA programming. I'm introducing static mini inference engine for `QWEN3-0.6B` instruct model in `bf16`, where its benchmarking claims that it's faster than [llama.cpp](https://github.com/ggml-org/llama.cpp) by approximately `8.5%` & `hf with flash-attn` by `292%` in `tokens/sec`, *see benchmarks below*. --- What does `qwen600` include: - single batch inference engine - static-constanted for compile-time optimization - all CUDA C/C++, no python dependencies (except for tokenizer setup) - minimal libraries (cuBLAS, CUB, std IO) - efficient memory pipeline: mmap, single GPU block, async copy - zero-cost pointer-based weight management on GPU --- `qwen600` is inspired by: - [llama.cpp - ggml](https://github.com/ggml-org/llama.cpp) - [llama2.c - Andrej Karpathy](https://github.com/karpathy/llama2.c) - [LLMs-from-scratch - Sebastian Raschka](https://github.com/rasbt/LLMs-from-scratch) - [qwen3.c - Adrian Cable](https://github.com/adriancable/qwen3.c) <p align="center"> <img src="assets/arch.png" width="283" height="401" alt="arch"> </p> ## Design Philosophy - The design of `qwen600.cu` is heavily inspired by the [suckless philosophy](https://suckless.org/philosophy/). - The goal is to create a tool that is simple, minimalist, and highly performant by avoiding feature bloat and unnecessary abstractions. - Configuration is done directly in the source code `config.h` as much as possible, and dependencies are kept to an absolute minimum. ## WANNA TRY ?! ### Initial Setup First, you need to clone [QWEN3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B). This is fantastic [hugging face doc blog](https://huggingface.co/docs/hub/en/repositories-getting-started) to start with cloning hf repos. then as a safe approach, you locate the weights file (model.safetensors) and sha256sum: ```bash sha256sum <model_dir>/<safetensors-file-name> ``` and output must be according to hf: ```text f47f71177f32bcd101b7573ec9171e6a57f4f4d31148d38e382306f42996874b ``` After that: ```bash git clone https://github.com/yassa9/qwen600 cd qwen600 ``` Assume that downloaded hugging face dir is `<model_dir>`. We convert the Hugging Face tokenizer into the format used by `qwen600`. ```bash python export.py <model_dir> ``` That gonna output some template files and most importantly: `tokenizer.bin` ### Building qwen600 Now we are ready to build ! You just want: - `CUDA` + `nvcc` - `cuBLAS` + `CUB` ```bash mkdir build && cd build cmake .. && make -j$(nproc) ``` Just that simple, no other bulky libraries and dependencies to build. ## Moment of Truth: Running the Model You can see arguments manual by: ```bash # you are now inside qwen600/build ./qwen600 ``` the output be that manual: ```aiignore usage: ./qwen600 <model_dir> [options] example: ./qwen600 <model_dir> -r 1 model directory must contain: - model.safetensors - tokenizer.bin - template_*.txt files arguments: ---------- -r <int> reasoning mode, 0 (default) = no thinking, 1 = thinking -s <int> random seed, default -k <int> k value in top-k sampling, default 20 -t <float> temperature in [0,inf], default 0.6 -p <float> p value in top-p (nucleus) sampling in [0,1], default 0.95 -i <string> input prompt -y <string> system prompt in chat mode, default is none ``` For example: ```bash ./qwen600 <model_dir> -r 1 -t 0.65 -p 0.9 -k 20 ``` or simply going with defaults: ```bash ./qwen600 <model_dir> -r 1 ``` Based on official hugging face [model card](https://huggingface.co/Qwen/Qwen3-0.6B), they advise
Excerpt of 9,141 characters
Read on GitHub
11
2
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:c2bd29e27a2acf4c, topic:llm, topic:transformer, topic:qwen
matched fp:c2bd29e27a2acf4c, topic:llm-inference, topic:llamacpp, desc:inference engine
matched fp:c2bd29e27a2acf4c, topic:gpu, topic:cuda