Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A deep-dive on the entire history of deep-learning
| Date | Stars |
|---|---|
| 2026-07-31 | 1575 |
| 2026-08-04 | 1575 |
| 2026-08-06 | 1576 |
Today
+1 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# deep-learning
A deep-dive on the entire history of deep learning, highlighting the series of innovations that got us from simple feed-forward networks to GPT-4o.
For each key milestone, I've included the critical papers in this repository, along with my notes, my explanation of important intuitions & math, and a toy implementation in pytorch when relevant.
The rest of this page is my breakdown of everything we can learn from this history, and what it tells us about the future of deep learning, inspired by [_The Lessons of History_](https://www.amazon.com/Lessons-History-Will-Durant/dp/143914995X) by Will & Ariel Durant.
> [!IMPORTANT]
>
> **This project is designed so everyone can get most of the value by just reading my overview on the rest of this page.**
>
> Then, people curious to learn about the technical details of each innovation can explore the rest of the repository via the links in the [resources](#resources) section.
> [!NOTE]
>
> For more context, checkout the [original twitter thread](https://x.com/MajmudarAdam/status/1794190796411027791)
> [!NOTE]
>
> Thanks to [Pavan Jayasinha](https://x.com/pavanjayasinha) and [Anand Majmudar](https://x.com/Almondgodd) for their constant feedback while I made this 😄
## Table of Contents
- [Overview](#overview)
- [1. Constraints](#2-constraints)
- [1.1. Data](#11-data)
- [1.2. Parameters](#12-parameters)
- [1.3. Optimization & Regularization](#13-optimization--regularization)
- [1.4. Architecture](#14-architecture)
- [1.5. Compute](#15-compute)
- [1.6. Compute Efficiency](#16-compute-efficiency)
- [1.7. Energy](#17-energy)
- [1.8. Constraints & Leverage](#18-constraints--leverage)
- [2. Narratives](#2-narratives)
- [3. Inspiration](#3-inspiration)
- [4. Intelligence](#4-intelligence)
- [5. Future](#5-future)
- [Resources](#resources)
- [Topics](#topics)
- [Implementations](#implementations)
- [Papers](#papers)
<br />
# Overview
The most interesting part of my deep-dive came from noticing a clear trend across all the key advancements, which has completely reframed how I understand deep learning:
> [!IMPORTANT]
>
> **There are 7 simple constraints that limit the capacity of digital intelligence:**
>
> 1. data
> 2. parameters
> 3. optimization & regularization
> 4. architecture
> 5. compute
> 6. compute efficiency
> 7. energy
>
> **The entire history of deep learning can be seen as the series of advancements that have gradually raised the ceiling on these constraints**, enabling the creation of increasingly intelligent systems.
It's impossible to understand where we're going without first understanding how we got here - and it's impossible to understand how we got here without understanding these constraints, which have always governed the rate of progress.
By understanding them, we can also explore a few related questions:
- How is progress made in deep learning?
- Where do the ideas that drive progress in deep learning come from?
- How have our narratives about digital intelligence changed over time?
- What does deep learning teach us about our own intelligence?
- Where is the future of deep learning headed?
So, let's start by understanding these constraints from first principles.
<br />
# 1. Constraints
We can define intelligence[^1] as the ability to accurately model reality[^2]. Practically, we're interested in models of reality that are useful for performing economically valuable tasks.
The goal of deep learning is to produce accurate models of reality for these useful tasks by:
1. Treating the true models that describe reality as complex probability distributions[^3]
2. Creating neural networks capable of modeling complex probability distributions
3. Training these networks to learn to model the probability distributions that underlie reality
In this view, creating intelligence with deep learning involves just two steps:
1. Collect useful information about reality (collect data)
2. Create a neural netwExcerpt of 68,809 characters
Read on GitHub
214
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:52f8f8a818bc75b3, llm:Repository description: 'A deep-dive on the entire history of deep-learning' (educational/history content). Language: Jupyter Notebook.