Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A New Optimization Technique for Deep Neural Networks
| Date | Stars |
|---|---|
| 2026-07-31 | 539 |
| 2026-08-01 | 539 |
| 2026-08-02 | 539 |
| 2026-08-06 | 539 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Gradient Centralization
## [Gradient Centralization: A New Optimization Technique for Deep Neural Networks](https://arxiv.org/abs/2004.01461)
***
## Introduction
* Gradient Centralization (GC) is a simple and effective optimization technique for Deep Neural Networks (DNNs), which operates directly on gradients by centralizing the gradient vectors to have zero mean. It can both speedup training process and improve the final generalization performance of DNNs. GC is very simple to implement and can be easily embedded into existing gradient based DNN optimizers with only few lines of code. It can also be directly used to finetune the pre-trained DNNs. Please refer to the [algorithm-GC](https://github.com/Yonghongwei/Gradient-Centralization/blob/master/algorithm-GC/) to obtain the codes of more advanced optimizers.
<div align="center"><img src="https://github.com/Yonghongwei/Gradient-Centralization/blob/master/fig/gradient.png" height="45%" width="45%" alt="Illustration of the GC operation on gradient matrix/tensor of weights in the fully-connected layer (left) and convolutional layer (right)."/></div>
* GC can be viewed as a projected gradient descent method with a constrained loss function. The Lipschitzness of the constrained loss function and its gradient is better so that the training process becomes more efficient and stable. Our experiments on various applications, including `general image classification`, `fine-grained image classification`, `detection and segmentation` and `Person ReID` demonstrate that GC can consistently improve the performance of DNN learning.
<div align="center"><img src="https://github.com/Yonghongwei/Gradient-Centralization/blob/master/fig/projected_Grad.png" height="50%" width="50%" alt=""/></div>
* The optimizers are provided in the files: [`SGD.py`](https://github.com/Yonghongwei/Gradient-Centralization/blob/master/GC_code/CIFAR100/algorithm/SGD.py), [`Adam.py`](https://github.com/Yonghongwei/Gradient-Centralization/blob/master/GC_code/CIFAR100/algorithm/Adam.py) and [`Adagrad.py`](https://github.com/Yonghongwei/Gradient-Centralization/blob/master/GC_code/CIFAR100/algorithm/Adagrad.py), including SGD_GC, SGD_GCC, SGDW_GCC, Adam_GC, Adam_GCC, Adam_GCC2, AdamW_GCC, AdamW_GCC2 and Adagrad_GCC. The optimizers with "_GC" use GC for both Conv layers and FC layers, and the optimizers with "_GCC" use GC only for Conv layers. For adaptive learning rate methods, keeping mean of weight vector unchanged usually works better. Please refer to Adam_GCC2 and AdamW_GCC2. We can use the following codes to import SGD_GC:
```python
from SGD import SGD_GC
```
***
## Update
* 2020/04/07:Release a pytorch implementation of optimizers with GC, and provide some examples on classification task, including
general image classification (Mini-ImageNet, CIFAR100 and ImageNet) and Fine-grained image classification (FGVC Aircraft， Stanford Cars， Stanford Dogs and CUB-200-2011).
* 2020/04/14:Release the code of GC on MMdetection and update some tables of experimental results.
* 2020/05/07:Release the code of GC on Person ReID and show some results on Market1501.
* 2020/08/08:Release the code of some advanced optimizers with GC.
***
## Citation
@article{GradientCentra,
title={Gradient-Centralization: A New Optimization Technique for Deep Neural Networks},
author={Hongwei Yong and Jianqiang Huang and Xiansheng Hua and Lei Zhang},
booktitle={the European Conference on Conputer Vision},
year={2020}
}
***
## Link to the other implementation of GC
* Gradient Centralization in TensorFlow [`https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow`](https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow)
* Gradient Centralization in Ranger optimizer [`https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer`](https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer)
## Experiments
***
### General Image Classification
* Mini-ImageNet
The codes are inExcerpt of 11,780 characters
Read on GitHub
138
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:c0ff2ff8d4c8cf3b, llm:Repository topics: deep-neural-networks, gradient-centralization, optimization-technique, optimizers; description: 'A New Optimization Technique for Deep Neural Networks'.
matched fp:c0ff2ff8d4c8cf3b, llm:Repository topics: deep-neural-networks, gradient-centralization, optimization-technique, optimizers; description: 'A New Optimization Technique for Deep Neural Networks'.