Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Knowledge distillation methods implemented with Tensorflow (now there are 11 (+1) methods, and will be added more.)
| Date | Stars |
|---|---|
| 2026-07-24 | 264 |
| 2026-07-25 | 264 |
| 2026-07-28 | 264 |
| 2026-07-30 | 264 |
| 2026-08-06 | 264 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Knowledge Distillation Methods with Tensorflow
Knowledge distillation is the method to enhance student network by teacher knowledge.
So annually knowledge distillation methods have been proposed, but each paper's do experiments with different networks and compare with different methods.
Moreover, each method is implemented by each author, so if a new researcher wants to study knowledge distillation, they have to find or implement all of the methods. Surely it is tough work.
To reduce this burden, I publish some codes and modify from my research codes.
I'll update the code and knowledge distillation algorithm, and all of the things will be implemented using Tensorflow.
Upgrade version of this Repo. will be available at this [link](https://github.com/sseung0703/Knowledge_distillation_via_TF2.0)
# Implemented Knowledge Distillation Methods
Please check detail of each category in [MHGD](https://arxiv.org/abs/1907.02226) and If you think the above categorization is useful, please consider citing the following paper.
@inproceedings{GraphKD,
title = {Graph-based Knowledge Distillation by Multi-head Attention Network},
author = {Seunghyun Lee, Byung Cheol Song},
booktitle = {British Machine Vision Conference (BMVC)},
year = {2019}
}
## Response-based Knowledge
Defined knowledge by the neural response of the hidden layer or the output layer of the network
- Soft-logit : The first knowledge distillation method for deep neural network. Knowledge is defined by softened logits. Because it is easy to handle it, many applied methods were proposed using it such as semi-supervised learning, defencing adversarial attack and so on.
- [Geoffrey Hinton, et al. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015.](https://arxiv.org/abs/1503.02531)
- Deep Mutual Learning (DML) : train teacher and student network coincidently, to follow not only training results but teacher network's training procedure.
- [Zhang, Ying, et al. "Deep mutual learning." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.](http://openaccess.thecvf.com/content_cvpr_2018/html/Zhang_Deep_Mutual_Learning_CVPR_2018_paper.html)
- Factor Transfer (FT) : Encode a teacher network's feature map, and transfer the knowledge by mimicking it.
- [Jangho Kim et al. "Paraphrasing Complex Network: Network Compression via Factor Transfer" Advances in Neural Information Processing Systems (NeurIPS) 2018](https://papers.nips.cc/paper/7541-paraphrasing-complex-network-network-compression-via-factor-transfer)
## Multi-connection Knowledge
Increase the quantity of knowledge by sensing several points of the teacher network
- FitNet : To increase amounts of information, knowledge is defined by multi-connected networks and compared feature maps by L2-distance.
- [Adriana Romero, et al. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.](https://arxiv.org/abs/1412.6550)
- Attention transfer (AT) : Knowledge is defined by attention map which is L2-norm of each feature point.
- [Zagoruyko, Sergey et. al. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928, 2016.](https://arxiv.org/pdf/1612.03928.pdf) [[the original project link](https://github.com/szagoruyko/attention-transfer)]
- Activation boundary (AB) : To soften teacher network's constraint, they propose the new metric function inspired by hinge loss which usually used for SVM.
- [Byeongho Heo, et. al. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. AAAI2019](https://arxiv.org/abs/1811.03233) (rivised by Author) [[the original project link](https://github.com/bhheo/AB_distillation)]
- VID : Define variational lower boundary as the knowledge, to maximize mutual information between teacher and student network.
- [Ahn, et. al. Variational Information DisExcerpt of 7,267 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:150905e8cb22f324, topic:knowledge-distillation, desc:knowledge distillation, readme:knowledge distillation
matched fp:150905e8cb22f324, topic:tensorflow