Rust + CUDA LLM inference engine for Blackwell (Tuned specifically on RTX PRO 6000, RTX 5090, B200): OpenAI-compatible (+converse and ant) serving, per-model X hardware exactness gates. NVFP4/mixed (fp8 hybrid, 4o6, etc - correctness, performance, hardware specific adapted) main quant support.
Install · Models · Serving · Performance · Hosted API
memra is the research engine of the tiyuvta lab: kernels, quantization arithmetic, speculative decoding and serving written from scratch for Blackwell, with exactness gates and receipts for every published number. The lab's hosted API (inference) and its deployments (services) serve open models tuned on your workload on an open serving stack the lab operates; memra's findings feed that tuning. The hosted API is OpenAI-compatible.
Prebuilt binaries require Linux x86_64, an NVIDIA driver with CUDA 13 runtime support, and the
CUDA runtime libraries. The installer selects the GPU build, verifies its checksum, and does not
require nvcc.
curl -fsSL https://raw.githubusercontent.com/avifenesh/memra/main/tools/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
kernel-check # expect: ALL GREENStart a server with a supported public checkpoint. The first run downloads and caches the model:
MEMRA_MODELS="q38=hf:tiyuvta/Qwen3.8-27B-NVFP4-MTP-GGUF:Q5K-mtp" \
memra-serverThe server listens on 127.0.0.1:8080. Send a streaming chat completion from another terminal:
curl -sS -N http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "q38",
"messages": [{"role": "user", "content": "Explain KV caching in one sentence."}],
"max_tokens": 128,
"stream": true
}'Use the cookbook for qualified model-and-card configurations. See installation for source builds and platform requirements.
Memra is narrow on purpose. The best execution path depends on the model, checkpoint, hardware, and workload, so Memra measures those combinations separately instead of promising that one loader or one default supports everything. A fast path is promoted only when its own correctness and performance evidence is current.
The project favors explicit support, reproducible receipts, and useful failures over silent fallbacks. Issues, model requests, hardware reports, and evidence-backed pull requests are welcome. Look elsewhere if you need broad hardware coverage, a large architecture catalog, or a Python library embedded in your application.
By model: Qwen3.8 27B · Qwen3.8 Flash Next (bring-up only) · Qwen3.5 9B · Qwen3.6 27B · Qwen3.6 35B-A3B · Qwen-AgentWorld 35B-A3B · Ornith 1.5 35B-A3B · Ornith 1.0 9B · Ornith 1.0 35B · Gemma 4 12B · Gemma 4 26B-A4B · Gemma 4 31B · Gemma 4 E4B · Step 3.7 Flash · DeepSeek V4 Flash · GLM-5.3 Flash (NativeReference) · Hy3 (NativeReference · NVFP4 NativeQualified)
By hardware: RTX PRO 6000 Blackwell · RTX 5090 / 50-series · H100 · Ada · B200
By workload: interactive agents · long-context and prefix reuse · batch throughput · multimodal · large models on multiple GPUs
memra is a from-scratch LLM inference engine: a Rust host runtime driving hand-written CUDA kernels, compiled ahead of time into fatbins embedded in the binary, with no Python and no framework in the serving path. It targets one card class at a time and is tuned separately for RTX PRO 6000 Blackwell and RTX 5090 (sm_120a), with B200 (sm_100a) as a runtime-qualified source lane and Hopper/Ada supported via source builds. The design constraints and the decisions they forced are in ARCHITECTURE.md.
| Crate | Owns |
|---|---|
memra-engine |
The CUDA engine: kernels, model programs, KV cache, speculative decoding, multi-GPU placement |
memra-server |
OpenAI-compatible HTTP serving (chat, completions, Messages, Responses, embeddings, rerank), admission, prefix cache |
memra-gguf |
GGUF and safetensors loading, quant-format decode, tensor inventory, model packs and the support-state enum |
memra-tokenizer |
GGUF-native BPE and SPM tokenizer with chat templates |
memra-sampling |
Host-side sampler chain (temperature, top-k, top-p, penalties) |
memra-kv |
KV-cache format policy (q8_0, q5_1, q4_0, fp8 block layouts) |
memra-lanes |
Serving-lane types, SLO admission policy, engine-truth step stats |
memra-reference |
Portable unfused reference executor for correctness gates |
memra-runtime |
CUDA runtime scaffolding (context, GEMM checks) |
memra-validate |
Numeric validation helpers (reference comparisons, tolerance gates) |
memra-cli |
Model onboarding compiler: memra model inspect, model pack, model verify |
memra-probe |
Unpublished dev spike (publish = false) |
Support is specific to a model, quantization, and drafter combination, never to a format. There
are exactly three positive states, the enum NativeSupport in
crates/memra-gguf/src/model_packs/mod.rs:
- NativeReference: the plan compiles and runs in memra's reference executor. Bring-up evidence only.
- NativeQualified: the required checkpoint and serving gates pass.
- NativeTuned: qualified, plus current receipts for the optimized rewrites the deployment selects.
"Loads", "shares an architecture name", and "works through another engine" are not support states. docs/MODELS.md is the support matrix; each entry in docs/models/ is the shortest recommended path for one model.
Every fast path is promoted only against its own oracle. Kernel-level bit identity
(kernel-check), argmax parity of generation against the reference path (run-gen),
speculative-decode self-consistency across draft lengths (run-spec), and served-path
byte identity (spec versus plain, restored versus cold) are the standing battery; GitHub CI is
compile-only, so the battery runs on a GPU before a merge or a tag. Greedy decoding is the
instrument, not the product: it is what makes byte-level gates possible. The gate catalog and
what each one proves is docs/TESTING.md.
Tagged releases carry prebuilt binaries (the installer above reads them) and publish the
workspace to crates.io. The repository history was rebuilt from a content snapshot on
2026-09-01 (a zero-history swap), so the GitHub Releases list here starts again with the first
tag cut on this history; crates.io is continuous, with memra-engine 0.123.0 published
2026-09-01T04:59Z from old-history commit bc0952fe5, which is tag v0.123.0. The notes for
v0.122.0 and v0.123.0 are kept in
docs/archive/RELEASE-NOTES-v0.122.0-v0.123.0.md.
Release mechanics are in docs/RELEASING.md.
| Document | Use it for |
|---|---|
| Installation | Prebuilt requirements, source builds, architecture selection |
| Model cards | Best starting path for each supported model |
| Hardware cards | Recommended use by GPU target |
| Workload cards | Recommended use by request shape |
| Cookbook | Copy-paste model and card configurations |
| Models | Supported checkpoints, formats, drafters, and hardware |
| Serving | HTTP contract, caching, auth, admission, multi-GPU, operations. v0.138.0: the GLM TP pool-split indexer is prime-only and its retained door is renamed (#395, MEMRA_GLM5_TP_INDEXER_SPLIT_PRIME, default ON, =0 restores the replicated prime); decode ignores it and the decode dispatch is deleted, so the door no longer carries a decode cost. The served context window is resolved from the model's own config (#455, MEMRA_CTX unset = the checkpoint's declared context, never a literal 8192). Admission can be shaped by memory instead of a worst-case slot proxy (#431, MEMRA_ADMIT_BY_MEMORY, default OFF, decide-by 2026-09-23). DSV4F takes the matrix expert program as its default and three doors leave with the flip (#482); the three norm-fusion doors are ported, measured on the served program and deleted (#490); the dense wide-prefill tile width becomes the code (#471). Qwen3.5 defaults MEMRA_SERVE_DEVPENALTY ON on sm_120 (#429), gains an opt-in GQA-packed FA2-class prefill (#421, MEMRA_PRIME_ATTN_FA2, default OFF, decide-by 2026-09-23) and a carried-prime graph with live state tables (#417). Gemma sampled-spec admission defaults ON (MEMRA_GEMMA_SPEC_SAMPLED). Speech lands as a program: the capability plan and its gate set (#440), the served ASR decode contract G7 (docs/SPEECH.md section 5), and an audio surface that sheds honestly (#489, MEMRA_AUDIO_*). v0.137.0: GLM host-cache images preserve model-owned and TP-rank planes through demotion and promotion; an opt-in startup arena reserves the full pinned budget before readiness (#325, MEMRA_GLM5_TP_KV_HOST, default OFF). Validated DFlash restores also use retained-prefix admission and refuse carrier restoration failures (#377). The owned speculative-prime fairness walker is included (#379, MEMRA_PRIME_YIELD OFF; receipts). DSV4 graph split-K defaults ON (#392, MEMRA_DSV4_MOE_M1_SPLITK, unset = graph, 0 rolls back with fresh graphs; accepted drift, unset/0 engagement). GLM TP-2 gains opt-in GPU sampling (#378, merge ff0937dd09b808cd97dde174de1b6aedff63a0b8, MEMRA_GLM5_TP_DEVICE_SAMPLE default OFF, decide-by 2026-09-22). Five p32k OFF/ON pairs across two windows measured median wall throughput 81.32 -> 91.25 tok/s and server decode 11.89 -> 10.61 ms/token; 160-token greedy/top-k-one twins were byte-identical. Unset/0 with a restart restores host sampling. Sampler receipts. Release qualification; deployment qualification remains separate. Since v0.136.0 DFlash serving streams cold and resumed prefill taps in 256-row batches (#370), removing the 13.42 GB cold tap transient at 128k and bounding the resumed suffix sink; qualification. Since v0.135.0 eligible single-device text prefix restores pay for their suffix workspace, keep the donor leased, and refuse a cold fallback if restoration fails (#350, #362); TP restores retain their existing admission charges; v0.137.0 extends this accounting to validated external DFlash restores (#377). Shared cache qualification rejects invalid HTTP and incomplete stream rows (#360). Since v0.134.0 startup GPU-canary timeouts are retried before latching a fault (#353), and a non-resident MoE MTP head keeps verify capture closed and selects eager verification (#354). The DSV4 small-kernel diet (#339, MEMRA_DSV4_SMALL_KERNEL_DIET) is included as a default-OFF experiment. Since v0.133.0 the dspark (qwen) verify runs its GDN state kernels once per layer over all draft rows instead of once per row (MEMRA_SPEC_GDN_PACKED, default ON, 0 rolls back; memra#341): +9% served decode at c=1, byte-identical. Since v0.132.0 the GLM-5.3-Flash TP-2 pair keeps a prefix cache that carries every rank (MEMRA_GLM5_TP_PREFIX) and primes with the split grouped prime's host diet (MEMRA_GLM5_TP_SPLIT_PRIME_HOSTDIET, 1M prime 310 -> 251 s); since v0.131.0 memra-server serves that walk behind MEMRA_GLM5_TP (plain route, MEMRA_SERVE_SPEC=0, spec and the batched hc walk refused by name; memra#313), the DSA indexer's prefill scorer can run on bf16 tensor cores (MEMRA_DSA_SCORE_TC, default OFF; memra#309) and its query rows split across the TP ranks (MEMRA_GLM5_TP_INDEXER_SPLIT, default OFF, byte-identical; memra#314): a 1.03M-token prime on the 2x B200 pair went 782 -> 310 s composed, and MEMRA_B200_DSA_DECODE defaults to level 2 on sm_100a (memra#315). Since v0.130.0 a temperature-0 request that carries a penalty (every non-thinking qwen request under the vendor presence_penalty) takes the dspark route, verified by penalised argmax and byte-identical to the host sampler (MEMRA_DSPARK_GREEDY_PENALTY, default ON, 0 rolls back; memra#310), and the door-0 decline is receipted (memra#305). Since v0.129.0 the dspark (qwen) draft KV is a live ring (MEMRA_DFLASH_KV_RING, default ON, 0 rolls back): 4,120 rows resident for any prompt length instead of the whole context cap, byte-identical to the cold oracle (memra#301). Since v0.128.0 a RESTORED dspark (qwen) session publishes the new turn's render-stable boundary as its own prefix-cache entry (memra#257), so the entry advances one turn per turn instead of freezing at turn 1's cut with a suffix that grew by a whole turn each time (same two doors, no new flag); sm_100a builds default the composed B200 decode doors ON. Since v0.127.0 the dspark (qwen) prefix-cache entry can be cut at the render-stable turn boundary (MEMRA_DSPARK_BOUNDARY_CAPTURE, default off) so a multi-turn continuation restores instead of re-priming the whole conversation, and that route publishes committed tokens at round cadence like the glm5 one. Since v0.126.1 admission keeps a driver-headroom floor (MEMRA_ADMIT_DRIVER_HEADROOM_MB, 2048) in front of every admitted request, trimming only the pool slice a starved driver needs, so a long prime no longer OOMs on the driver side while the pool sits on cached blocks. Since v0.126.0 a streaming request's long prefill stays alive through proxies (SSE comments until MEMRA_STREAM_TTFT_MS_MAX, with a bounded pre-header admission budget) and a step-OOM teardown reclaims the parked pools and the prefix cache before its retry. Since v0.125.0 a DFlash2 draft proposal carrying the top-k selector's exhausted-slot sentinel refuses that one request by name instead of panicking the GPU worker, and a BUSY worker is judged on forward progress rather than heartbeat silence |
| API surfaces | Anthropic Messages and OpenAI Responses compatibility |
| Embeddings and rerank | /v1/embeddings (OpenAI schema) and /v1/rerank (Cohere shape): prefill-only capture surfaces; every item of a multi-item request is metered under its own ledger id <x-request-id>.<index> (v0.124.1), Serving |
| Performance | Measurements, methodology, rigs, and receipts |
| Flags | Audited environment-variable reference |
| Testing | Correctness gates and evidence requirements |
| Architecture | Runtime structure and Blackwell implementation ledger |
| Decisions | Adopted and rejected design choices with evidence |
| Releases | Release artifacts; notes for the pre-swap tags in docs/archive/ |
Issues and requests are welcome:
- Report a bug
- Request a model or feature
- Submit a hardware validation
- Report a vulnerability privately
For a model request, include the exact checkpoint, quantization, target GPU, and why the model is useful. For a performance report, include the command, model artifact, hardware, and raw output.
Issues are welcome even without a proposed fix. Pull requests are welcome when they include the proof required for the affected path; GPU changes cannot be validated by GitHub's compile-only runners. Start with CONTRIBUTING.md.
FSL-1.1-ALv2. Each version becomes available under Apache-2.0 two years after it is first made available. Earlier revisions and package releases remain under the license that accompanied them. Third-party components retain their own terms; see THIRD-PARTY.md.
Built by Avi Fenesh at tiyuvta.