Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Implementation of paper Data Engineering for Scaling Language Models to 128K Context
| Date | Stars |
|---|---|
| 2026-07-31 | 502 |
| 2026-08-05 | 502 |
| 2026-08-06 | 502 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Long-Context Data Engineering
<p align="center" width="100%">
<a ><img src="assets/logo.jpg" alt="logo" style="width: 60%; min-width: 300px; display: block; margin: auto;"></a>
</p>
ChatGPT-4 Dalle-3 Prompt: "Draw a carton style logo showing a very very long paper"
<p align="center">
🤗 <a href="https://huggingface.co/yaofu/llama-2-7b-80k" target="_blank">HF Repo</a> • 📃 <a href="https://arxiv.org/abs/2402.10171" target="_blank">Paper</a> • 💿 <a href="https://huggingface.co/datasets/yaofu/slimpajama-per-source-length-upsample" target="_blank">Data</a>
</p>
Implementation of paper:
* Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim and Hao Peng. Feb 2024. _Data Engineering for Scaling Language Models to 128K Context_
<p align="center" width="100%">
<a ><img src="assets/needle.jpg" alt="logo" style="width: 100%; min-width: 300px; display: block; margin: auto;"></a>
</p>
Our model is the first public work showing how to achieve GPT-4 level long-context retrieval performance.
## Table of Content
- [x] Loading and playing with the following continue pretrained checkpoint:
- [x] LLaMA-2 7B 80K: continue pretrained on 80K, tested on 128K
- [x] LLaMA-2 13B 64K: continue pretrained on 64K, tested on 128K
- [x] Evaluating the pretrained checkpoint on Needle-in-a-HayStack
- [x] Loading the preprocessed data
- [x] Processing the long-context data
- [ ] Continue pretraining the model on processed long-context data
## Download the model to local
Create a folder to download the model.
```bash
pip install -r requirements.txt # pytorch is not included here because we assume you have already installed pytorch
mkdir ../llama-2-7b-80k
mkdir ../llama-2-13b-64k
```
Download the continue pretrained checkpoint to local
```python
from huggingface_hub import snapshot_download
snapshot_download(repo_id='yaofu/llama-2-7b-80k',
local_dir='../llama-2-7b-80k',
repo_type='model',
local_dir_use_symlinks=False,
resume_download=True)
snapshot_download(repo_id='yaofu/llama-2-13b-64k',
local_dir='../llama-2-13b-64k',
repo_type='model',
local_dir_use_symlinks=False,
resume_download=True)
```
We recommend you download the checkpoint to local first, instead of directly loading from HF, like the following:
```python
from transformers import AutoModelForCausalLM
# Below is slow and hard to control in a cluster
# Unless you insist, **we recommend you download the model to local first**
model = AutoModelForCausalLM.from_pretrained("yaofu/llama-2-7b-80k",
use_flash_attention_2="flash_attention_2",
torch_dtype=torch.bfloat16
)
```
## Load the continue pretrained checkpoint and play with it
The following code requries at least 8x4090 to support 80K context.
If you have 4x80G A100 you can make it to at least 128K
We use `tensor_parallel` implemented from [this repo](https://github.com/BlackSamorez/tensor_parallel) because it is much faster than huggingface's `device_map` and lightweight than vLLM. But it has a small bug that if your GPU memory is not large enough, it will stuck instead of through a memory overflow exception. So make sure you do have enough GPU memory.
```python
import torch
import tensor_parallel as tp
from transformers import AutoModelForCausalLM, AutoTokenizer
from eval.needle.utils import load_context, insert_needle
# This is the continue pretrained LLaMA 2 7B model with modified rope
def reset_rope(model, model_max_train_len, scaling_factor):
for l in model.model.layers:
l.self_attn.rotary_emb.scaling_factor = scaling_factor
l.self_attn.rotary_emb._set_cos_sin_cache(seq_len=model_max_train_len, device="cpu", dtype=torch.float32)
return
model = AutoModelForCausalLM.from_pretrained("../Excerpt of 12,076 characters
Read on GitHub
4
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:a15b29ca1805ac6b, llm:Repository description: 'Implementation of paper Data Engineering for Scaling Language Models to 128K Context' (Python implementation of methods to scale LLM context to 128K).
matched fp:a15b29ca1805ac6b, llm:Repository description: 'Implementation of paper Data Engineering for Scaling Language Models to 128K Context' (Python implementation of methods to scale LLM context to 128K).
matched fp:a15b29ca1805ac6b, llm:Repository description: 'Implementation of paper Data Engineering for Scaling Language Models to 128K Context' (Python implementation of methods to scale LLM context to 128K).