Research
The ideas behind the agent.
Four concepts explain most of what reinforcement learning does. Choose one to read a short explanation, then try the discount slider.
Concepts
Rewards
A reward is a number the environment returns after each action. The agent's only goal is to collect as much reward as possible over time, so the reward design is the real specification.
- Reward the outcome you want, not a shortcut that happens to correlate with it.
- Small step costs encourage efficient behaviour.
- Badly designed rewards get "gamed": the agent finds loopholes you did not intend.
Exploration
An agent that only repeats what worked so far can miss a better route. Exploration means sometimes trying something new.
- Epsilon-greedy: with a small probability, take a random action.
- Explore more early, less once the agent is confident.
- In the real world, exploration must be limited to safe actions.
Policies
A policy is the agent's rule for choosing an action in each situation. In the playground, the arrows are the policy.
- Value-based methods such as Q-learning learn scores and act greedily on them.
- Policy-gradient methods adjust the policy directly.
- Large problems use neural networks to approximate either one.
Safety
Learning by trial and error is fine in a simulation and risky in the real world.
- Train in simulation first, then test carefully on real systems.
- Constrain actions so mistakes are recoverable.
- Keep a human able to pause or override the agent.
0.90
Move the slider to see how much a future reward is worth today.