Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
fastLLaMa: An experimental high-performance framework for running Decoder-only LLMs with 4-bit quantization in Python using a C/C++ backend.
| Date | Stars |
|---|---|
| 2026-07-31 | 413 |
| 2026-08-03 | 412 |
| 2026-08-06 | 412 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# fastLLaMa
[](https://opensource.org/licenses/MIT)
`fastLLaMa` is an experimental high-performance framework designed to tackle the challenges associated with deploying large language models (LLMs) in production environments.
It offers a user-friendly Python interface to a C++ library, [llama.cpp](https://github.com/ggerganov/llama.cpp), enabling developers to create custom workflows, implement adaptable logging, and seamlessly switch contexts between sessions. This framework is geared towards enhancing the efficiency of operating LLMs at scale, with ongoing development focused on introducing features such as optimized cold boot times, Int4 support for NVIDIA GPUs, model artifact management, and multiple programming language support.
```
___ __ _ _ __ __
| | '___ ___ _| |_ | | | | ___ | \ \ ___
| |-<_> |<_-< | | | |_ | |_ <_> || |<_> |
|_| <___|/__/ |_| |___||___|<___||_|_|_|<___|
.+*+-.
-%#--
:=***%*++=.
:+=+**####%+
++=+*%#
.*+++==-
::--:. .**++=::
#%##*++=...... =*+==-::
.@@@*@%*==-==-==---:::::------::==*+==--::
%@@@@+--====+===---=---==+=======+++----:
.%@@*++*##***+===-=====++++++*++*+====++.
:@@%*##%@@%#*%#+==++++++=++***==-=+==+=-
%@%%%%%@%#+=*%*##%%%@###**++++==--==++
#@%%@%@@##**%@@@%#%%%%**++*++=====-=*-
-@@@@@@@%*#%@@@@@@@%%%%#+*%#++++++=*+.
+@@@@@%%*-#@@@@@@@@@@@%%@%**#*#+=-.
#%%###%: ..+#%@@@@%%@@@@%#+-
:***#*- ... *@@@%*+:
=***= -@%##**.
:#*++ -@#-:*=.
=##- .%*..##
+*- *: +-
:+- :+ =.
=-. *+ =-
:-:- =-- :::
```
---
## Features
- [x] Easy-to-use Python interface that allows developers to build custom workflows.
- [x] Pip install support.
- [x] Ability to ingest system prompts.
- [x] System prompts will remain in runtime memory, normal prompts are recycled.
- [x] Customisable logger support.
- [x] Low memory mode support using mmap.
- [x] Quick context switching between sessions.
- [x] Ability to save and load session states.
- [x] Quick LoRA adapter switching during runtime.
- [x] During the conversion of LoRA adapters to bin file, we are caching the result of matrix multiplication to avoid expensive caclulation for every context switch.
- [x]Excerpt of 12,983 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:e9a5a349d23d5d36, llm:description: 'fastLLaMa: An experimental high-performance framework for running Decoder-only LLMs with 4-bit quantization in Python using a C/C++ backend.' topics: c, cpp, lama, lamacpp, python
matched fp:e9a5a349d23d5d36, llm:description: 'fastLLaMa: An experimental high-performance framework for running Decoder-only LLMs with 4-bit quantization in Python using a C/C++ backend.' topics: c, cpp, lama, lamacpp, python
matched fp:e9a5a349d23d5d36, llm:description: 'fastLLaMa: An experimental high-performance framework for running Decoder-only LLMs with 4-bit quantization in Python using a C/C++ backend.' topics: c, cpp, lama, lamacpp, python
matched fp:e9a5a349d23d5d36, llm:description: 'fastLLaMa: An experimental high-performance framework for running Decoder-only LLMs with 4-bit quantization in Python using a C/C++ backend.' topics: c, cpp, lama, lamacpp, python