Reinforcement Learning: Explained With Our Own Experiment

The Short Answer

Reinforcement learning is a way of teaching software to make decisions by trial and error. Nobody writes the rules. The software tries things, gets a score after each try, and slowly learns to repeat what scored well.

Think of teaching a dog a trick with treats. You never explain the trick. You reward what moves in the right direction, and over time the trick appears. The difference is scale: software can repeat an attempt millions of times, without getting tired.

To keep this concrete, we use an experiment of our own throughout: Lock and Key, a game of freeze tag played by AI characters inside a 3D arena. We built it to learn how this technology really behaves, and it has already taught us things that textbooks only describe.

The Loop: Observe, Act, Get Rewarded

Every reinforcement learning setup is made of the same few parts. The Hugging Face deep reinforcement learning course describes them this way. The agent is the learner, the one making decisions. The environment is the world the agent acts in. The agent looks at the situation, picks an action, and the environment answers with a reward and a new situation. Then it repeats.

• Observation. What the agent can see right now. The course points out that this is often only part of the whole picture, like a platform game that shows only the area around your character.
• Action. A choice the agent can make. It can be a small set, such as four directions to walk in, or a smooth range, such as how far to turn a steering wheel.
• Reward. A number the environment hands back after an action. Think of it as points. It is the agent's only way to learn whether the action was good.
• Episode. One complete attempt, from start to finish, like one round of a game. Unity's ML-Agents documentation says an episode ends when a time limit is reached or the game decides it is over, and then everything resets for the next attempt.

The agent is not chasing the biggest reward right now. It is chasing the biggest total reward over time. The Hugging Face course adds that rewards that arrive sooner count for more than rewards far in the future, so the agent learns to prefer quick, reliable gains.

Our Experiment: Freeze Tag With AI Agents

You may know the playground game. In Lock and Key, one seeker chases three runners. If the seeker gets within one meter of a runner, that runner freezes. A free runner who touches a frozen teammate unfreezes them, and a rescued runner is protected for one second. A round lasts 20 seconds. If every runner is frozen, the seeker wins. If at least one runner is still free at 20 seconds, the runners win.

We built it in Unity, a popular game-building tool, using a toolkit called ML-Agents that lets game characters learn instead of following scripts. Here is how the parts above map onto our game. You can also see the arena, videos and progress on the Lock and Key experiment page.

At first the runners just move at random. Interestingly, before any training, runners that simply stand still already won 44% of rounds in our tests, because the seeker often cannot freeze all three in 20 seconds. That is why you always measure a starting point first. A win rate only means something when you can compare it to a baseline.

When Rewards Go Wrong

Our first training run used only the rewards above. We measure training in steps, which are small moments of play the software learns from. After roughly 540,000 steps, the runners had learned nothing. The reason is simple. Random runners almost never happen to touch a frozen teammate, so rescuing never earns a reward, and the software never gets a chance to learn it.

So we added a bonus: every rescue now gave the team +0.3. For a while it looked like it worked, and rescues began to rise. Then they kept rising, and something else rose with them.

An honest round pays at most about +1.5. By 2 million steps the runners were earning about 6 per round by getting frozen about 18 times and rescuing each other again and again. Freezes per round had gone from 2.5 before training to 18. The runners had not become better at avoiding the seeker. They had discovered that being frozen and then rescued was a points machine.

We stopped that run and added a penalty of -0.3 for every freeze, so that a freeze-and-rescue loop adds up to zero. This is a well-known problem. DeepMind researchers call it specification gaming, which they describe as behavior that satisfies the literal specification of an objective without achieving the intended outcome. In plain words: the agent does exactly what you rewarded, not what you meant.

Their best-known example is a boat-racing game where the goal was to finish the race. The agent was also rewarded for hitting green blocks along the track, so it drove in circles hitting the same blocks instead of finishing. A 2016 paper, Concrete Problems in AI Safety, names this reward hacking and lists avoiding it as one of five practical research problems.

Learning Together: Many Agents

Lock and Key has several characters learning at once, which is called a multi-agent problem. This brings a new difficulty. If the whole team gets one shared reward for winning, every member is rewarded, even one who did nothing helpful. Each agent then struggles to work out what it did right.

Unity's ML-Agents v2.0 release, announced in May 2021, added a method called MA-POCA for this case. It trains a shared judge that the post describes as a coach for the whole group. It also lets an agent leave mid-round, for example when a character is eliminated, while still learning whether its earlier actions helped the team win. Our three runners are trained together as one MA-POCA group.

Competition can drive learning too. In OpenAI's hide-and-seek experiment, teams of agents trained against each other using standard reinforcement learning at large scale. The authors report six distinct phases of strategy appearing on their own, such as building shelters from movable boxes and later using ramps to get over obstacles. Each new strategy pushed the opposing team to find a counter.

What It Has Achieved

In 2015, DeepMind researchers published a paper in Nature about an AI agent that learned to play classic Atari video games from only the raw screen pixels and the game score. Using the same method, network design and settings for every game, it reached a level comparable to a professional human games tester across 49 Atari 2600 games.

Where It Fits in Games and Simulation

Game engines are a natural home for reinforcement learning, because they already give you a controllable world, a fast simulation and a clear score. The ML-Agents toolkit describes itself as enabling games and simulations to serve as environments for training intelligent agents.

It is worth using when unpredictability is part of the value. For example, an opponent or hazard that should react differently to what a trainee does in a simulation-based training scenario. It is often unnecessary when a scenario needs to play out the same way every time, because fixed scripts are easier to build, test and keep consistent.

Frequently Asked Questions