Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
This repository collects papers for "A Survey on Knowledge Distillation of Large Language Models". We break down KD into Knowledge Elicitation and Distillation Algorithms, and explore the Skill & Vertical Distillation of LLMs.
| Date | Stars |
|---|---|
| 2026-07-31 | 1296 |
| 2026-08-04 | 1296 |
| 2026-08-06 | 1296 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Awesome Knowledge Distillation of LLM Papers
[]()
[](https://arxiv.org/abs/2402.13116)
<!-- Big font size -->
<h2 align="center">
A Survey on Knowledge Distillation of Large Language Models
</h2>
<p align="center">
Xiaohan Xu<sup>1</sup>  
Ming Li<sup>2</sup>  
Chongyang Tao<sup>3</sup>  
Tao Shen<sup>4</sup>  
Reynold Cheng<sup>1</sup>  
Jinyang Li<sup>1</sup>  
Can Xu<sup>5</sup>  
Dacheng Tao<sup>6</sup>  
Tianyi Zhou<sup>2</sup>  
</p>
<p align="center">
<sup>1</sup> The University of Hong Kong   
<sup>2</sup> University of Maryland   
<sup>3</sup> Microsoft   
<sup>4</sup> University of Technology Sydney   
<sup>5</sup> Peking University   
<sup>6</sup> The University of Sydney
</p>
<div align="center">
<img src="imgs/framework.png" width="700"><br>
</div>
<br>
*A collection of papers related to knowledge distillation of large language models (LLMs).
If you want to use LLMs for benefitting your own smaller models training, or use self-generated knowledge to achieve the self-improvement, just take a look at this collection.*
***We will update this collection every week. Welcome to star ⭐️ this repo to keep track of the updates.***
> ❗️Legal Consideration: It's crucial to note the legal implications of utilizing LLM outputs, such as those from ChatGPT ([Restrictions](https://openai.com/policies/business-terms)), Llama ([License](https://llama.meta.com/llama-downloads/)), etc. We strongly advise users to adhere to the terms of use specified by the model providers, such as the restrictions on developing competitive products, and so on.
## 💡 News
- **2024-2-20**: 📃 We released a survey paper "**[A Survey on Knowledge Distillation of Large Language Models](https://arxiv.org/abs/2402.13116)**". Welcome to read and cite it. We are looking forward to your feedback and suggestions.
- Update Log
- **2024-3-19**: Add 14 papers.
## Contributing to This Collection
Feel free to **open an issue/PR** or e-mail [[email protected]](mailto:[email protected]), [[email protected]](mailto:[email protected]), [[email protected]](mailto:[email protected]) and [[email protected]](mailto:[email protected]) if you find any missing taxonomies or papers. We will keep updating this collection and survey.
## 📝 Introduction
**KD of LLMs**: This survey delves into knowledge distillation (KD) techniques in Large Language Models (LLMs), highlighting KD's crucial role in transferring advanced capabilities from proprietary LLMs like GPT-4 to open-source counterparts such as LLaMA and Mistral. We also explore how KD enables the compression and self-improvement of open-source LLMs by using them as teachers.
**KD and Data Augmentation**: Crucially, the survey navigates the intricate interplay between data augmentation (DA) and KD, illustrating how DA emerges as a powerful paradigm within the KD framework to bolster LLMs' performance. By leveraging DA to generate context-rich, skill-specific training data, KD transcends traditional boundaries, enabling open-source models to approximate the contextual adeptness, ethical alignment, and deep semantic insights characteristic of their proprietary counterparts.
**Taxonomy**: Our analysis is meticulously structured around three foundational pillars: **[algorithm](#kd-algorithms)**, **[skill](#skill-distillation)**, and **[verticalization](#verticalization-distillation)** -- providing a comprehensive examination of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields.
**[KD Algorithms](#kd-algorithms)**: For KD algorithms, we categorize it into two principal steps: "Knowledge Elicitation" focusing on eliciting knowledExcerpt of 81,148 characters
Read on GitHub
54
Liam Liang Ding
2
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:53fd52ec5d2e4cf7, topic:knowledge-distillation, name:knowledge distillation, desc:knowledge distillation
matched fp:53fd52ec5d2e4cf7, topic:data-augmentation
matched fp:53fd52ec5d2e4cf7, topic:alignment
matched fp:53fd52ec5d2e4cf7, topic:llm