Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Always sparse. Never dense. But never say never. A Sparse Training repository for the Adaptive Sparse Connectivity concept and its algorithmic instantiation, i.e. Sparse Evolutionary Training, to boost Deep Learning scalability on various aspects (e.g. memory and computational time efficiency, representation and generalization power).
| Date | Stars |
|---|---|
| 2026-07-24 | 263 |
| 2026-07-25 | 263 |
| 2026-07-28 | 263 |
| 2026-07-30 | 263 |
| 2026-07-31 | 263 |
| 2026-08-06 | 263 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# sparse-evolutionary-artificial-neural-networks * Proof of concept implementations of various sparse artificial neural network models with adaptive sparse connectivity trained with the Sparse Evolutionary Training (SET) algorithm - https://arxiv.org/abs/1707.04780, 15 July 2017 * **SET** was the first algorithm which demonstrated that **sparse neural networks** can be trained from scratch to **outperform dense neural networks** within the framework of gradient descent and introduced the idea of optimizing the sparse connections between neurons together with the weights during training. * On short, **SET** laid the ground for what is today known as **sparse training** with **dynamic sparsity** (also referred to in some papers as dynamic sparse training, pruning and growth strategies, and so on). * The following implementations are distributed in the hope that they may be useful, but without any warranties; Their use is entirely at the user's own risk. ###### Implementation 1 - using binary masks - SET-MLP with Keras and Tensorflow (SET-MLP-Keras-Weights-Mask) * Proof of concept implementation of Sparse Evolutionary Training (SET) for Multi Layer Perceptron (MLP) on CIFAR10 using Keras and a mask over weights. * This implementation can be used to test SET in varying conditions, using the Keras framework versatility, e.g. various optimizers, activation layers, tensorflow. * Also it can be easily adapted for Convolutional Neural Networks or other models which have dense layers. * Variants of this implementation have been used to perform the experiments from Reference 1 with MLP and CNN. * However, due the fact that the weights are stored in the standard Keras format (dense matrices), this implementation can not scale properly. * If you would like to build an SET-MLP with over 100000 neurons, please use Implementation 2. ###### Implementation 2 - truly sparse implementation - SET-MLP using just sparse data structures from pure Python 3 (SET-MLP-Sparse-Python-Data-Structures) * An improved version of this Implementation can be found here https://github.com/SelimaC/Tutorial-SCADS-Summer-School-2020-Scalable-Deep-Learning * Proof of concept implementation of Sparse Evolutionary Training (SET) for Multi Layer Perceptron (MLP) on lung dataset using Python, SciPy sparse data structures, and (optionally) Cython. * This implementation was developed just in the last stages of the reviewing process, and we are briefly discussing about it in the "Peer Review File" which can be downloaded from Reference 1 website. * This implementation can be used to create SET-MLP with hundred of thousands of neurons on a standard laptop. It was made starting from the vanilla fully connected MLP implementation of Ritchie Vink (https://www.ritchievink.com/) and we would like to acknowledge his work and thank him. Also, we would like to thank Thomas Hagebols for analyzing the performance of SciPy sparse matrix operations. We thank also to Amarsagar Reddy Ramapuram Matavalam from Iowa State University ([email protected]), who provided us a faster implementation of the "weightsEvolution" method, after the initial release of this code. * If you would like to try large SET-MLP models, below are the expected running times measured on my laptop (16 GB RAM) using the original implementation of the "weightsEvolution" method. I have used exactly the model and the dataset from the file "set_mlp_sparse_data_structures.py" and I just changed the number of hidden neurons per layer: - 3,000 neurons/hidden layer, 12,317 neurons in total 0.3 minutes/epoch - 30,000 neurons/hidden layer, 93,317 neurons in total 3 minutes/epoch - 300,000 neurons/hidden layer, 903,317 neurons in total 49 minutes/epoch - 600,000 neurons/hidden layer, 1,803,317 neurons in total 112 minutes/epoch * If you would like to try out SET-MLP with various activation functions, optimization methods and so on (in the detriment of scalability)
Excerpt of 9,515 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:191face88325a8ce, topic:sparsity, readme:pruning, readme:sparsity
matched fp:191face88325a8ce, topic:deep-learning