Reinforcement Learning by Hand: Bandits, Q-Learning & Value Iteration

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Estimate the value of three app offers from 15 visits with sample averages, then rebuild the same estimate step by step with the incremental update rule and compare it with a constant step size.
• Run an ε-greedy agent for 12 app visits with given random numbers: decide explore or exploit, pick the offer, collect the reward, add up the cumulative reward and chart it against always showing the best offer.
• Score four offers with the UCB1 rule: average reward plus an exploration bonus that is large for rarely tried offers, and see why UCB picks a different offer than greedy.
• Calculate the discounted return of one episode of Robo the warehouse robot forwards (discount factors) and backwards (G(t) = R + γ·G(t+1)) and see why both give the same number.
• Evaluate a coin-flip policy for Robo in a five-cell aisle with the Bellman expectation equation, sweep after sweep, and check that the exact values are a fixed point of the equation.
• Run value iteration for Robo on a slippery floor: in every sweep each cell takes the better of left and right (MAX), the values climb towards the optimal V*, and a line chart of the largest change shows the convergence.
• Replay eight logged steps of Robo through the Q-learning rule: look up the best next value, build the TD target and TD error, and update the Q-table column by column as the dock's reward travels back along the aisle.
• Turn Robo's optimal values into Q-values and a policy (the best action per cell), then measure how much better the optimal policy is than the coin flip, cell by cell and on average.