AWS Builder Center

Action-value methods with incremental step size in RL

Old draft from my digital garden, but this is something cool I found in reinforcement learning

AI & Web3 Product Manager
Old draft from my digital garden, but this is something cool I found in reinforcement learning
Suppose you want to estimate the average payout from a slot machine. The traditional approach is to record every payout, add them together, and divide by the number of pulls.
A running average avoids storing the full history. Update the estimate after each pull with:
New average = Old average + Step size × (New reward - Old average)
The difference between the new reward and the old average is the error:
  • A positive error means the payout was better than expected.
  • A negative error means it was worse than expected.
  • The step size controls how quickly the average responds.

Two ways to choose the step size

  1. Use 1/n:
Set the step size to 1/n, where n is the number of pulls so far. This gives every payout equal weight. As the number of pulls increases, each new reward changes the average less. For a stable machine, the estimate approaches its expected payout over time.
  1. Use a fixed step size
Choose a constant such as 0.1:
New average = Old average + 0.1 × (New reward - Old average)
This gives more weight to recent rewards. It works well when the machine's payout pattern can change and the estimate needs to adapt quickly.
With a step size of 1/n, you need to store the current average and the pull count. With a fixed step size, you only need the current average.
For complete pseudocode, mathematical proofs, and detailed notes, read the long ago full writeup at  jienweng.github.io .
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article