Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Enumerate diverse machine learning training tricks.
| Date | Stars |
|---|---|
| 2026-07-31 | 420 |
| 2026-08-02 | 420 |
| 2026-08-06 | 420 |
| 2026-08-12 | 420 |
| 2026-08-13 | 420 |
| 2026-08-18 | 420 |
| 2026-09-20 | 420 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Deep Learning Tricks
This is an attempt to enumerate different machine learning training tricks I gather around as well as some [network architectures](networks.md). The goal is to briefly give a description of the trick as well as an intuition about why it is working. My knowledge is quite limited so this is prone to errors/imprecisions. This should be a collaborative work so feel free to complete or correct.<br />
Most of the tricks may seem trivial for those who have some experience in machine learning but I feel like while there is a lot of very good theoretical material available for machine learning, there is still a lack of practical advice. Those would really have helped me when I started.<br />
The excellent [CS231n](http://cs231n.github.io/) Stanford course already has a good list of training tricks.
#### Table of Contents
* [Data prepossessing](#data-prepossessing)
* [Initialization](#initialization)
* [Training](#training)
* [Regularization](#regularization)
* [Network architecture](#network-architecture)
* [NLP](#nlp)
* [Reinforcement learning](#reinforcement-learning)
* [Network compression](#network-compression)
## Data prepossessing
## Initialization
* **What**: Initializing the weights correctly can improve the performances and speed up the training. Bias usually initialized at 0. For the weights, some recommend using uniform within:
* For linear layers \[1\]: [-v,v] with v = 1/sqrt(inputSize)
* For convolution layers \[2\]: [-v,v] with v = 1/sqrt(kernelWidth\*kernelHeight\*inputDepth)
Batch normalization \[3\] seems to reduce the need for fine tuned weight initialization. More recent results propose alternative formulas.<br />
**Why**: Using the default initialization, each neurone has a variance that grows with the number of inputs. By scaling each weight by th sqrt(n), it ensure that the neurons will have approximately the same output distribution.<br />
**Ref**:
1. *Stochastic Gradient Descent Tricks, Leon Bottou*
2. ?? (default used by Torch)
3. *Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, S. Ioffe and C. Szegedy* (https://arxiv.org/abs/1502.03167)
* **What**: For LSTM, initialize the forget bias to one. This speed up the training.<br />
**Why**: The intuition is that at the beginning of the training, we want the information to propagate from cell to cell, so don't want the cell to forget its state.<br />
**Ref**: *An Empirical Exploration of Recurrent Network Architectures, Rafal Jozefowicz et al.* (cites the trick but not the original authors)
* **What**: For t-SNE, the original author recommend a perplexity between 5 and 50 (for dataset with about 5000~10000 points) [1], for bigger dataset, it's recommended to use a bigger one.<br />
**Why**: The perplexity determine the size of variance of the Gaussian distribution for each point. It can be seen as the number of affected nearest neighbors. A smaller perplexity will privilege local variations (more clusters), a big one will privilege the global ones. A perplexity bigger than the number of point don't really makes sense.<br />
The other things to take into account is that the plotted clusters don't conserve their original scale (dense clusters will be diluted) and the distance between clusters isn't necessarily representative of the original spacial geometry. Different perplexity can gives complementary informations on the data structure, different runs can gives different results [2].<br />
**Ref**:
1. *Visualizing High-Dimensional Data Using t-SNE, L.J.P. van der Maaten*
2. *How to Use t-SNE Effectively, Wattenberg, et al., Distill, 2016.* ([link](http://distill.pub/2016/misread-tsne/))
## Training
* **What**: In addition to the ground truth 'hard' targets, we can also train a network using the soft targets (SoftMax outputs with high temperature (about 1-10 depending on the problem)) from another model.<br />
**Ref**: *Distilling the KnowledgExcerpt of 12,780 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:fbcccd82fc345c24, topic:deep-learning