Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Implementing Recurrent Neural Network from Scratch
| Date | Stars |
|---|---|
| 2026-07-24 | 529 |
| 2026-07-25 | 529 |
| 2026-07-28 | 529 |
| 2026-07-30 | 529 |
| 2026-07-31 | 529 |
| 2026-08-06 | 529 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Implementing Recurrent Neural Network from Scratch I’m assuming that you are somewhat familiar with basic Neural Networks. If you’re not, you may want to head over to [Implementing A Neural Network From Scratch](https://github.com/pangolulu/neural-network-from-scratch), which guides you through the ideas and implementation behind non-recurrent networks. ## Introduction This post is inspired by [recurrent-neural-networks-tutorial](http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/) from [WildML](http://www.wildml.com/). And you can deeply read it to know the basic knowledge about RNN, which I will not include in this tutorial. In this tutorial, we will focus on how to train RNN by [Backpropagation Through Time (BPTT)](http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/), based on the **computation graph** of RNN and do **automatic differentiation**. You can find that it is more simple and reliable to calculate the gradient in this way than you do it by hand. This post will take RNN language model (rnnlm) as example. More about the fancy applications of RNN can be found [here](http://karpathy.github.io/2015/05/21/rnn-effectiveness/). ## How to train RNN The architecture of RNN can be as the following figure.  You can find that the parameters `(W, U, V)` are shared in different time steps. And the output in each time step can be **softmax**. So you can use **cross entropy** loss as an error function and use some optimizing method (e.g. gradient descent) to calculate the optimized parameters `(W, U, V)`. Let recap the equations of our RNN:  We also defined our loss, or error, to be the cross entropy loss, given by:  Here `y_t` is the correct word at time step `t`, and `y^_t` is our prediction. We typically treat the full sequence (sentence) as one training example, so the total error is just the sum of the errors at each time step (word).  Remember that our goal is to calculate the gradients of the error with respect to our parameters `U`, `V` and `W` and then learn good parameters using optimizing method (in this post we use **Stochastic Gradient Descent**). Just like we sum up the errors, we also sum up the gradients at each time step for one training example: . That is we should calculate `dEt/dW`, `dEt/dU` and `dEt/dV`, then sum up all time steps. It is simple to calculate `dEt/dV`, because it only depends on the values at the current time step. But the story is different for `dEt/dW` and `dEt/dU`. Note that `s_3 = tanh(Ux_3 + Ws_2)` depend on `s_2`, which depends on `W`, `U` and `s_1`, and so on. So if we take the derivative with respect to `W` we can't treat `s_2` as a constant! We need to apply the chain rule again. You can have a view from the following figure.  Now use **computation graph** to represent `E1` as an example and calculate `dE1/dW`, `dE1/dU` is the same idea.  Note that this is exactly the same as the standard backpropagation algorithm that we use in deep [Feedforward Neural Networks](https://github.com/pangolulu/neural-network-from-scratch). The key difference is that we sum up the gradients for `W` at each time step. In a traditional NN we don’t share parameters across layers, so we don’t need to sum anything. But in my opinion BPTT is just a fancy name for standard backpropagatio
Excerpt of 15,548 characters
Read on GitHub
9
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:27b0965f18708f44, topic:neural-network, topic:tensorflow, readme:automatic differentiation