Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
This repository contains code and tooling for the Abacus.AI LLM Context Expansion project. Also included are evaluation scripts and benchmark tasks that evaluate a model’s information retrieval capabilities with context expansion. We also include key experimental results and instructions for reproducing and building on them.
| Date | Stars |
|---|---|
| 2026-07-31 | 603 |
| 2026-08-03 | 604 |
| 2026-08-06 | 604 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Extending LLM Context Length The choice of how to encode positional information for transformers has been one of the key components of LLM architectures. An area that has been interesting to us and others in the community recently is whether LLMs can be extended to longer contexts. We have conducted a range of experiments with different schemes for extending context length capabilities of Llama, which has been pretrained on 2048 context length with the RoPE (Rotary Position Embedding) encoding. Here we share some of the results as well as the training and evaluation scripts in the hope that it will be useful to the community. For our best performing models - linear scaling with IFT at scales 4 and 16 - we are also sharing the weights in case others wish to use them, or to conduct their own tests. We believe the scale 16 model should perform well on real world tasks up to 16k context lengths, and potentially even up to about 20-24k context lengths. [Scale 16 model](https://huggingface.co/abacusai/Giraffe-v1-delta-13b-scaled-16) [Technical Paper](http://arxiv.org/abs/2308.10882) ## Overview We conducted a wide variety of experiments to try to extend the context length of the models. First, we tried simply using the base Llama model zero-shot. As expected, this performed well up to 2048 context length but deterioriated very rapidly afterwards. We next investigated fine tuning approaches where we trained the model on the RedPajama dataset at context lengths of 4096. This led to expected improvements in performance up to 4096 context but again, no further. Another approach to extending context length is to modify in some way the RoPE encoding. Here, we tried many different ideas: - Linear scaling, as described by kaiokendev.github.io. - Scaling the Fourier basis of RoPE by a power, such that low frequencies are stretched more than high frequencies. - Applying truncation to the Fourier basis. Our idea here was that we wanted the model to see only frequencies that were fast enough so that it got at least one full cycle during training; any slower frequencies were set to 0 (equivalent to no rotation at all, i.e. equally important at all context lengths). - Randomising the position vector. In particular, we combined fine-tuning on the RedPajama dataset and instruction-fine-tuning with the Vicuna dataset with the above approaches. This is what led to the most fruitful results. Finally, we implemented and tried the approach described in the [xPos](https://arxiv.org/abs/2212.10554) paper. This approach adds decaying amplitude penalty terms that cause fast frequencies to have less impact at long distances than slow frequencies in the Fourier basis (see our blog post for similarity heatmaps that show this). ## Highlighted Results Perhaps the most pointed observation we made is that different evaluation methodologies/tasks lead to different rankings of the approaches detailed above. This will be described in further detail below. That said, we made the following general observations: - Linear interpolation/scaling seems to be the most robust approach for increasing model context length. - Using a linear scale of N does not necessarily lead to a model context length increase by a factor of N. For example, our scale 16 experiments generally stopped performing well after a context length of 16000, not 32000 (~2048 * 16). We have ideas for how to ameliorate this effect planned for future work. - Truncation and randomisation both seem to have great perplexity scores but perform less well on the retrieval task. - Instruction fine tuning with the Vicuna dataset improves accuracy in the retrieval context significantly at lengths which the base model is capable of handling, but cannot 'fix' the base model at lengths where it fails. ## Evaluation Tasks For evaluation we used two different datasets: - LMSys datasets (the 'lines' task) for locating a substring in the context - Our own open book question answering dataset, WikiQA, whi
Excerpt of 16,477 characters
Read on GitHub
8
4
3
Arka Pal
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:2d8ec40e87529e8f, llm:repository description: 'code and tooling for the Abacus.AI LLM Context Expansion project... evaluation scripts and benchmark tasks that evaluate a model’s information retrieval capabilities with context expansion'
matched fp:2d8ec40e87529e8f, llm:repository description: 'code and tooling for the Abacus.AI LLM Context Expansion project... evaluation scripts and benchmark tasks that evaluate a model’s information retrieval capabilities with context expansion'
matched fp:2d8ec40e87529e8f, llm:repository description: 'code and tooling for the Abacus.AI LLM Context Expansion project... evaluation scripts and benchmark tasks that evaluate a model’s information retrieval capabilities with context expansion'