Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Markov Decision Processes, Dynamic Programming and Reinforcement Learning
| Date | Stars |
|---|---|
| 2026-07-31 | 309 |
| 2026-08-01 | 310 |
| 2026-08-02 | 310 |
| 2026-08-06 | 310 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# MDP-DP-RL The goal of this project was to develop all Dynamic Programming and Reinforcement Learning algorithms from scratch (i.e., with no use of standard libraries, except for basic numpy and scipy tools). The "develop from scratch" goal was motivated by educational purposes - students learning this topic can understand the concepts throroughly only when they develop and work with code developed from scratch. I teach courses on this topic to a variety of student backgrounds, and each such course is big on precise programming implementations of the techniques/algorithms. In particular, I use this codebase when I teach Stanford CME 241: Reinforcement Learning for Stochastic Control Problems in Finance (http://cme241.stanford.edu). Any feedback on code readability, performance and bugs will be greatly appreciated as the code is still fairly raw and untested in various parts (started working on this code in August 2018, and have mainly been in code-growth mode so far). The project started by implementing the foundational data structures for finite Markov Processes (a.k.a. Markov Chains), Markov Reward Processes (MRP), and Markov Decision Processes (MDP). This was followed by Dynamic Programming (DP) algorithms, where the focus was to represent Bellman equations in clear mathematical terms within the code. Next was the core educational material of Reinforcement Learning, implementing the Generalized Policy Iteration algorithms based on simulations (Monte Carlo and Temporal Difference, including eligibility traces). However, the emphasis was to first implement the tabular methods so that one can work with actual data structures (finite, hence tabular), rather than functions to represent MDP rewards and transition specifications as well as value functions and policies. Once the tabular RL methods were implemented, it was straightforward to write the same algorithms as functional approximation-based algorithms. However, this required a detour to build some foundation for function approximation. I chose to implement linear and deep neural network approximations, both of which require a specification of feature functions. Backpropagation was developed from scratch, again for educational purposes. On a whim, I also implemented Approximate Dynamic Programming (ADP) Algorithms, which was basically the same old Policy Iteration and Value Iteration algorithms but now using the output of function approximation for the right-hand-side of the Bellman update, and using the updated values as training data for gradient descent on the parameters of the function approximation. So far, I am finding ADP as the most valuable algorithm for the MDP problems I typically work with. I am a bit surprised that the "literature" focuses so much on model-free whereas I often know the model for many of the MDPs I work on, and so, ADP is ideal. I have chosen Python 3 as the language, mainly because I can't expect my students to have expertise in the potentially more-appropriate languages for this project, such as Scala, Ocaml and Haskell. These are functional programming languages and this topic/project is best done through a tasteful application of Functional Progamming. But Python 3 is not such a bad choice as functions are fast-class entities. My core technique in this project is indeed Functional Programming, but I had to very careful in getting around Python's "naughty" handling of function closures. I have also made heavy use of classes and TypeVars. Object-oriented polymorphism as well as type-parametrized polymorphism enabled me to cover a wide range of algorithms with plenty of common code. Python 3 also provided me the benefit of type annotations, which I have taken heavy advantage of in this project. Type annotations support turned out to be extremely valuable in the project as my IDE (PyCharm) caught a lot of errors/warnings statically, in fact as I was typing code, it would spot errors. More importa
Excerpt of 7,039 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:1dfbbab1d0b51a2b, llm:Repository description: 'Markov Decision Processes, Dynamic Programming and Reinforcement Learning' (no topics provided).
matched fp:1dfbbab1d0b51a2b, llm:Repository description: 'Markov Decision Processes, Dynamic Programming and Reinforcement Learning' (no topics provided).