Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Enabling easy statistical significance testing for deep neural networks.
| Date | Stars |
|---|---|
| 2026-07-31 | 339 |
| 2026-08-05 | 340 |
| 2026-08-06 | 340 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# deep-significance: Easy and Better Significance Testing for Deep Neural Networks []() [](https://coveralls.io/github/Kaleidophon/deep-significance?branch=main) []() [](https://www.gnu.org/licenses/gpl-3.0) [](https://github.com/python/black) [](https://zenodo.org/badge/latestdoi/341677886)  **Contents** * [:interrobang: Why](#interrobang-why) * [:inbox_tray: Installation](#inbox_tray-installation) * [:bookmark: Examples](#bookmark-examples) * [Intermezzo: Almost Stochastic Order - a better significance test for Deep Neural Networks](#intermezzo-almost-stochastic-order---a-better-significance-test-for-deep-neural-networks) * [Scenario 1: Comparing multiple runs of two models](#scenario-1---comparing-multiple-runs-of-two-models) * [Scenario 2: Comparing multiple runs across datasets](#scenario-2---comparing-multiple-runs-across-datasets) * [Scenario 3: Comparing sample-level scores](#scenario-3---comparing-sample-level-scores) * [Scenario 4: Comparing more than two models](#scenario-4---comparing-more-than-two-models) * [How to report results](#newspaper-how-to-report-results) * [Sample size](#control_knobs-sample-size) * [Other features](#sparkles-other-features) * [General Recommendations & other notes](#general-recommendations) * [:mortar_board: Cite](#mortar_board-cite) * [:medal_sports: Acknowledgements](#medal_sports-acknowledgements) * [:people_holding_hands: Papers using deep-significance](#people_holding_hands-papers-using-deep-significance) * [:books: Bibliography](#books-bibliography) ### :interrobang: Why? Although Deep Learning has undergone spectacular growth in the recent decade, a large portion of experimental evidence is not supported by statistical hypothesis tests. Instead, conclusions are often drawn based on single performance scores. This is problematic: Neural network display highly non-convex loss surfaces (Li et al., 2018) and their performance depends on the specific hyperparameters that were found, or stochastic factors like Dropout masks, making comparisons between architectures more difficult. Based on comparing only (the mean of) a few scores, **we often cannot conclude that one model type or algorithm is better than another**. This endangers the progress in the field, as seeming success due to random chance might lead practitioners astray. For instance, a recent study in Natural Language Processing by Narang et al. (2021) has found that many modifications proposed to transformers do not actually improve performance. Similar issues are known to plague other fields like e.g., Reinforcement Learning (Henderson et al., 2018) and Computer Vision (Borji, 2017) as well. To help mitigate this problem, this package supplies fully-tested re-implementations of useful functions for significance testing: * Statistical Significance tests such as Almost Stochastic Order (del Barrio et al, 2017; Dror et al., 2019), bootstrap (Efron & Tibshirani, 1994) and permutation-randomization (Noreen, 1989). * Bonferroni correction methods for multiplicity in datasets (Bonferroni, 1936). * Bootstrap power analysis (Yuan & Hayashi, 2003) and other functions to determine the right sample size. All functions are fully tested and also compatible with common deep learning data structures, such as PyTorch / Tensorflow tensors as well as NumPy and Jax arrays. For examples about the usage, consult the documentation [here](https://deep-significance.readthedocs.io/en/latest/) , the scenarios in the section [Examples]
Excerpt of 42,327 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:284771ec5ea9187d, topic:deep-learning