Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[ICLR 2024] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
| Date | Stars |
|---|---|
| 2026-07-31 | 643 |
| 2026-08-03 | 643 |
| 2026-08-06 | 643 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# 🦙 Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
🌟 [ArXiv Preprint](https://arxiv.org/abs/2310.06694) | [Blog Post](https://xiamengzhou.github.io/sheared-llama/)
Base models: [Sheared-LLaMA-1.3B](https://huggingface.co/princeton-nlp/Sheared-LLaMA-1.3B) | [Sheared-LLaMA-2.7B](https://huggingface.co/princeton-nlp/Sheared-LLaMA-2.7B) | [Sheared-Pythia-160m](https://huggingface.co/princeton-nlp/Sheared-Pythia-160m/tree/main)
Pruned Models without Continued Pre-training: [Sheared-LLaMA-1.3B-Pruned](https://huggingface.co/princeton-nlp/Sheared-LLaMA-1.3B-Pruned), [Sheared-LLaMA-2.7B-Pruned](https://huggingface.co/princeton-nlp/Sheared-LLaMA-2.7B-Pruned)
Instruction-tuned models: [Sheared-LLaMA-1.3B-ShareGPT](https://huggingface.co/princeton-nlp/Sheared-LLaMA-1.3B-ShareGPT) | [Sheared-LLaMA-2.7B-ShareGPT](https://huggingface.co/princeton-nlp/Sheared-LLaMA-2.7B-ShareGPT)
Thank you for your interest in our work! This is a joint work by [Mengzhou Xia](https://xiamengzhou.github.io/), [Tianyu Gao](https://gaotianyu.xyz/about/), [Zhiyuan Zeng](https://zhiyuan-zeng.github.io/), and [Danqi Chen](https://www.cs.princeton.edu/~danqic/). Here, we provide our codebase for Sheared-LLaMA's pruning and continued pre-training algorithms :) We find that pruning strong base models is an extremely cost-effective way to get strong small-scale language models compared to pre-training them from scratch. The following graph shows that given the existence of Llama-2-7B model (pre-trained with 2T tokens), pruning it produces a model as strong as an OpenLLaMA model with 3% of its pre-training cost.
<img src="images/teaserwlegend.jpg" alt="teaser" width="400" />
**Update**
- [12/19/2023] Updated the [evaluation scripts](https://github.com/princeton-nlp/LLM-Shearing/blob/master/icl_eval) and pruning logs in the repo.
- [11/22/2023] We released the instruction-tuned models [Sheared-LLaMA-1.3B-ShareGPT](https://huggingface.co/princeton-nlp/Sheared-LLaMA-1.3B-ShareGPT) and [Sheared-LLaMA-2.7B-ShareGPT](https://huggingface.co/princeton-nlp/Sheared-LLaMA-2.7B-ShareGPT).
- [11/19/2023] We released the [Sheared-Pythia-160m](https://huggingface.co/princeton-nlp/Sheared-Pythia-160m) model developed at early stages. It was produced using the same shearing recipe and the Pile dataset.
- [11/05/2023] We released the code on LLM-Shearing - excited to see it being applied to more models of different scales.
- [10/10/2023] We released the Sheared-LLaMA paper, two Sheared LLaMA models and [tweeted about it](https://twitter.com/xiamengzhou/status/1712102912439226510) 🚀!
## 🔗 Quick Links
- [🦙 Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](#-sheared-llama-accelerating-language-model-pre-training-via-structured-pruning)
- [🔗 Quick Links](#-quick-links)
- [Brief Introduction](#brief-introduction)
- [Install Requirements](#install-requirements)
- [Data Preparation](#data-preparation)
- [Model Preparation](#model-preparation)
- [Sample Scripts for Pruning and Continued Pre-training](#sample-scripts-for-pruning-and-continued-pre-training)
- [Convert Pruned Model](#convert-pruned-model)
- [Convert Composer Model to Huggingface Model](#convert-composer-model-to-huggingface-model)
- [Training Configurations](#training-configurations)
- [Data configurations](#data-configurations)
- [Basic training configurations](#basic-training-configurations)
- [Pruning configurations](#pruning-configurations)
- [Dynamic batch loading configurations](#dynamic-batch-loading-configurations)
- [Throughput](#throughput)
- [Future Work](#future-work)
- [Bugs or Questions?](#bugs-or-questions)
- [Citation](#citation)
## Brief Introduction
This codebase is built based on MosaicML's amazing [Composer package](https://github.com/mosaicml), which is specially designed and optimized for large language model pre-training. The entire implementation, including the `pruning` logic and Excerpt of 21,910 characters
Read on GitHub
Mengzhou Xia · Princeton University
68
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:55df855c40fbeb6e, topic:llm, topic:llama
matched fp:55df855c40fbeb6e, topic:pruning, desc:pruning
matched fp:55df855c40fbeb6e, topic:nlp