Recurrent Networks by Hand: Hidden States, Unrolling Through Time, Vanishing Gradients, LSTM Gates, GRU Cells & Sequence Prediction

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Lay out a toy parcel series one time step per row, add the previous value, the change and a 0–1 scaled input, turn two four-word sentences into numbers with a vocabulary and see that a plain sum cannot tell them apart while a leaky memory can.
• Run a one-unit recurrent cell over six scaled inputs: feed the previous hidden state back in, compute the pre-activation z_t, squash it with tanh written with EXP and follow the hidden state to the end of the sequence.
• Unroll the cell sideways over five four-word toy reviews, keep only the final hidden state as the summary of each review, turn it into a probability with a sigmoid output and see that reversing the words changes the score.
• Compute the factor u·(1 − h²) that a gradient is multiplied by at every step back, chain ten steps with PRODUCT for a small and a large recurrent weight and chart how the signal vanishes or explodes.
• Run a one-unit LSTM cell with given weights: compute the forget, input and output gates with sigmoids, the candidate with tanh, update the cell state c_t = f·c_(t−1) + i·g and the hidden state h_t = o·tanh(c_t), and carry both into the next step.
• Run a one-unit GRU over the same six inputs as the LSTM: compute the update and reset gates, the candidate with the reset applied to the old state, blend old and new state with the update gate and chart the GRU and LSTM hidden states side by side.
• Count the weights of a simple cell, a GRU and an LSTM for six layer sizes with h·(h + x + 1) per gate, turn the LSTM count into storage at 4 bytes per weight and add up a stacked two-layer LSTM.
• Run a recurrent cell with given trained weights over twelve weeks of toy visitor counts, turn each hidden state into a forecast for the following week, compare the mean absolute error with the naive last-week forecast, chart actual vs forecast and read off the forecast for week 13.