Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
LongLLaMA is a large language model capable of handling long contexts. It is based on OpenLLaMA and fine-tuned with the Focused Transformer (FoT) method.
| Date | Stars |
|---|---|
| 2026-07-31 | 1466 |
| 2026-08-06 | 1466 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<p align="center" width="100%"><img src="assets/longllama.png" alt="LongLLaMA" style="width: 50%; display: block; margin: auto;"></p>
# LongLLaMA: Focused Transformer Training for Context Scaling
<div align="center">
<table>
<tr>
<th style="font-size: 120%"> >_ 🎓 <a href="https://huggingface.co/syzymon/long_llama_code_7b_instruct">LongLLaMA-Code 7B Instruct</a> 📑🗨 </th>
</tr>
<tr>
<td align="center">
<a href="https://colab.research.google.com/github/CStanKonrad/long_llama/blob/main/long_llama_code_instruct_colab.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg"></a>   <a href="instruction_fine_tuning/LongLLamaCode7BInstruct.md">Learn more</a>
</td>
</tr>
</table>
</div>
<div align="center">
<table>
<tr>
<td align="center">
<span style="font-size:200%">⇧</span>
</td>
</tr>
</table>
</div>
<div align="center">
<table>
<tr>
<td align="center">
<span style="font-size:150%">{</span>
</td>
<td align="center">
<span style="font-size:110%">
<b>
<a href="https://huggingface.co/syzymon/long_llama_code_7b" tyle="margin-bottom:30px">LongLLaMA-Code 7B</a>
</b>
</span>
</td>
<td align="center">
<span style="font-size:150%">}</span>
</td>
</tr>
</table>
<hr>
</div>
<div align="center">
<table>
<tr>
<th> <a href="https://huggingface.co/syzymon/long_llama_3b_instruct">LongLLaMA-Instruct-3Bv1.1</a> </th>
<th> <a href="https://huggingface.co/syzymon/long_llama_3b_v1_1">LongLLaMA-3Bv1.1</a></th>
</tr>
<tr>
<td align="center">
<a href="https://colab.research.google.com/github/CStanKonrad/long_llama/blob/main/long_llama_instruct_colab.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg"></a>
</td>
<td align="center">
<a href="https://colab.research.google.com/github/CStanKonrad/long_llama/blob/main/long_llama_colab.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg"></a>
</td>
</tr>
</table>
</div>
<div align="center">
[TLDR](#TLDR) | [Overview](#Overview) | [Usage](#Usage) | [LongLLaMA performance](#LongLLaMA-performance) | [Authors](#Authors) | [Citation](#Citation) | [License](#License) | [Acknowledgments](#Acknowledgments)
[FoT continued pretraining](fot_continued_pretraining) | [Instruction tuning](instruction_fine_tuning)
</div>
## TLDR
This repository contains the research preview of **LongLLaMA, a large language model capable of handling long contexts of 256k tokens or even more**.
LongLLaMA is built upon the foundation of [OpenLLaMA](https://github.com/openlm-research/open_llama) and fine-tuned using the [Focused Transformer (FoT)](https://arxiv.org/abs/2307.03170) method.
LongLLaMA Code is built upon the foundation of [Code Llama](https://huggingface.co/codellama/CodeLlama-7b-hf).
We release a smaller 3B base variant (not instruction tuned) of the LongLLaMA model on a permissive license (Apache 2.0) and inference code supporting longer contexts on [Hugging Face](https://huggingface.co/syzymon/long_llama_3b). Our model weights can serve as the drop-in replacement of LLaMA in existing implementations (for short context up to 2048 tokens). Additionally, we provide evaluation results and comparisons against the original OpenLLaMA models.
In addition to this, we release code for [instruction tuning (PyTorch)](instruction_fine_tuning/) and [FoT continued pretraining (JAX)](fot_continued_pretraining/).
## Overview
### Base models
[Focused Transformer: Contrastive Training for Context Scaling](https://arxiv.org/abs/2307.03170) (FoT) presents a simple method for endowing language models with the ability to handle context consisting possibly of millions of tokens while training on significantly shorter input. FoT permits a subset of attention layers to access a memory cache of (key, value) pairs to extend the context length. The distinctive aspect of FoT is its training procedure, drawing from contrastExcerpt of 25,568 characters
Read on GitHub
Konrad Staniszewski
57
4
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:8425b50826207d14, llm:description: 'LongLLaMA is a large language model capable of handling long contexts. It is based on OpenLLaMA and fine-tuned with the Focused Transformer (FoT) method.'
matched fp:8425b50826207d14, llm:description: 'LongLLaMA is a large language model capable of handling long contexts. It is based on OpenLLaMA and fine-tuned with the Focused Transformer (FoT) method.'
matched fp:8425b50826207d14, llm:description: 'LongLLaMA is a large language model capable of handling long contexts. It is based on OpenLLaMA and fine-tuned with the Focused Transformer (FoT) method.'