Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[ICLR 2025] Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
| Date | Stars |
|---|---|
| 2026-07-31 | 966 |
| 2026-08-06 | 966 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<div align="center"> <img src="assets/Samba-pic.webp" width="300"/> </div> <h1 align="left"> Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling</h1> [](https://arxiv.org/abs/2406.07522) [](https://docs.google.com/presentation/d/1SUGKM_sTKgb-wXSYvalWtD2O2qTKnYuMzid669GfJbU/edit#slide=id.g274b8c2f536_0_0) [](https://drive.google.com/file/d/1bfK6aSH4O7G_ndIH835KfdQMCcqw8txJ/view) Samba is a simple yet powerful hybrid model with an **unlimited** context length. Its architecture is frustratingly simple: Samba = Mamba + MLP + Sliding Window Attention + MLP stacking at the layer level. Our largest model, `Samba-3.8B`, is trained on 3.2 trillion tokens from the Phi3 dataset, outperforming `Phi3-mini` on major benchmarks (e.g. MMLU, GSM8K and HumanEval) by a large margin. Samba can also achieve perfect **long-context** retrieval ability with minimal instruction tuning, while still maintaining its **linear complexity** with respect to sequence length. This ability leads to the impressive performance of `Samba-3.8B-instruct` on downstream tasks such as long-context summarization. ## Performance :rocket: <div align="left"> <img src="assets/ppl.jpg" width="300"/> <img src="assets/gen_speed.jpg" width="298"/> </div> | Model | MMLU | GSM8K | HumanEval | GovReport | SQuALITY | |-------------------------------|------|-------|-----------|-----------|----------| | Phi-3-mini-4K-instruct | 68.8 | 82.5 | 58.5 | 14.4 | **21.6** | | Samba-3.8B-instruct (preview) | **71.9** | **87.6** | **62.8** | **18.9** | 21.2 | We report 5-shot accuracy for MMLU, 8-shot CoT accuracy for GSM8K, 0-shot pass@1 for HumanEval and ROUGE-L for both GovReport and SQuALITY. ## Updates - [July 9] We released the [Phi-4-mini-flash-reasoning](https://huggingface.co/microsoft/Phi-4-mini-flash-reasoning) model which is based on SambaY architecture. Please check the new codebase for neural scaling experiments [here](https://github.com/microsoft/ArchScale)! - [Jan. 22] Samba has been accepted to ICLR 2025! - [Dec. 8] Added the evaluation script and more baseline architectures. - [June 11] Released the codebase for training Samba-421M and Samba-1.3B on SlimPajama. ## Code Overview Our training infrastructure on SlimPajama is a modified version of [TinyLlama](https://github.com/jzhang38/TinyLlama) and [LitGPT](https://github.com/Lightning-AI/litgpt). One can easily specify different architectual configurations through modifying the [`model_name`](pretrain.py#L30) and the [`config file`](lit_gpt/config.py) which includes tons of baseline architectures mentioned in the paper. Our RetNet and GLA implementations are from the awesome [Flash Linear Attention](https://github.com/sustcsonglin/flash-linear-attention) repository. ## Pretraining Samba from scratch Please follow the [`Dockerfile`](Dockerfile) to setup the environment. The data preparation mainly follows TinyLlama except that we only use the SlimPajama dataset. ### Data Preparation Download the Slimpajama dataset to your chosen directory. We also provide the pre-tokenized data [here](https://huggingface.co/datasets/jsun/slimpajama_Llama2_Tokenizer). ```bash cd /path/to/dataset git lfs install git clone https://huggingface.co/datasets/cerebras/SlimPajama-627B ``` The SlimPajama dataset takes 893GB diskspace. Use the provided scripts to tokenize the datasets and divide them into chunks. ```bash python scripts/prepare_slimpajama.py --source_path /path/to/SlimPajama --tokenizer_path data/llama --destination_path data/slim --split validation --percentage 1.0 python scripts/prepare_slimpajama.py --source_path /path/to/SlimPajama --tokenizer_path data/llama --destination_path data
Excerpt of 5,629 characters
Read on GitHub
23
Ikko Eltociear Ashimine · Japan
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:ca51b31a3a8b9df5, llm:Repository description: "[ICLR 2025] Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling" (language: Python).
matched fp:ca51b31a3a8b9df5, llm:Repository description: "[ICLR 2025] Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling" (language: Python).
matched fp:ca51b31a3a8b9df5, llm:Repository description: "[ICLR 2025] Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling" (language: Python).