deep-rl-class Documentation

So let's get started! 🚀

File: units/en/unit1/quiz.mdx Copy Chapter

Quiz [[quiz]]

The best way to learn and to avoid the illusion of competence is to test yourself. This will help you to find where you need to reinforce your knowledge.

Q1: What is Reinforcement Learning? Copy Section

Solution

Reinforcement learning is a framework for solving control tasks (also called decision problems) by building agents that learn from the environment by interacting with it through trial and error and receiving rewards (positive or negative) as unique feedback.

Q2: Define the RL Loop Copy Section

At every step:

• Our Agent receives ______ from the environment
• Based on that ______ the Agent takes an ______
• Our Agent will move to the right
• The Environment goes to a ______
• The Environment gives a ______ to the Agent

<Question
choices={[
{
text: "an action a0, action a0, state s0, state s1, reward r1",
explain: "At every step: Our Agent receives **state s0** from the environment. Based on that **state s0** the Agent takes an **action a0**. Our Agent will move to the right. The Environment goes to a **new state s1**. The Environment gives **a reward r1** to the Agent."
},
{
text: "state s0, state s0, action a0, new state s1, reward r1",
explain: "",
correct: true
},
{
text: "a state s0, state s0, action a0, state s1, action a1",
explain: "At every step: Our Agent receives **state s0** from the environment. Based on that **state s0** the Agent takes an **action a0**. Our Agent will move to the right. The Environment goes to a **new state s1**. The Environment gives **a reward r1** to the Agent."
}
]}
/>

Q3: What's the difference between a state and an observation? Copy Section

<Question
choices={[
{
text: "The state is a complete description of the state of the world (there is no hidden information)",
explain: "",
correct: true
},
{
text: "The state is a partial description of the state",
explain: ""
},
{
text: "The observation is a complete description of the state of the world (there is no hidden information)",
explain: ""
},
{
text: "The observation is a partial description of the state",
explain: "",
correct: true
},
{
text: "We receive a state when we play with chess environment",
explain: "Since we have access to the whole checkboard information.",
correct: true
},
{
text: "We receive an observation when we play with chess environment",
explain: "Since we have access to the whole checkboard information."
},
{
text: "We receive a state when we play with Super Mario Bros",
explain: "We only see a part of the level close to the player, so we receive an observation."
},
{
text: "We receive an observation when we play with Super Mario Bros",
explain: "We only see a part of the level close to the player.",
correct: true
}
]}
/>

Q4: A task is an instance of a Reinforcement Learning problem. What are the two types of tasks? Copy Section

<Question
choices={[
{
text: "Episodic",
explain: "In Episodic task, we have a starting point and an ending point (a terminal state). This creates an episode: a list of States, Actions, Rewards, and new States. For instance, think about Super Mario Bros: an episode begin at the launch of a new Mario Level and ending when you’re killed or you reached the end of the level.",
correct: true
},
{
text: "Recursive",
explain: ""
},
{
text: "Adversarial",
explain: ""
},
{
text: "Continuing",
explain: "Continuing tasks are tasks that continue forever (no terminal state). In this case, the agent must learn how to choose the best actions and simultaneously interact with the environment.",
correct: true
}
]}
/>

Q5: What is the exploration/exploitation tradeoff? Copy Section

Solution

In Reinforcement Learning, we need to balance how much we explore the environment and how much we exploit what we know about the environment.

• Exploration is exploring the environment by trying random actions in order to find more information about the environment.

• Exploitation is exploiting known information to maximize the reward.

Q6: What is a policy? Copy Section

Solution

• The Policy π is the brain of our Agent. It’s the function that tells us what action to take given the state we are in. So it defines the agent’s behavior at a given time.

Q7: What are value-based methods? Copy Section

Solution

• Value-based methods is one of the main approaches for solving RL problems.
• In Value-based methods, instead of training a policy function, we train a value function that maps a state to the expected value of being at that state.

Q8: What are policy-based methods? Copy Section

Solution

• In Policy-Based Methods, we learn a policy function directly.
• This policy function will map from each state to the best corresponding action at that state. Or a probability distribution over the set of possible actions at that state.

Congrats on finishing this Quiz 🥳, if you missed some elements, take time to read again the chapter to reinforce (😏) your knowledge, but do not worry: during the course we'll go over again of these concepts, and you'll reinforce your theoretical knowledge with hands-on.

File: units/en/unit1/rl-framework.mdx Copy Chapter

The Reinforcement Learning Framework [[the-reinforcement-learning-framework]]

The RL Process [[the-rl-process]] Copy Chapter

To understand the RL process, let’s imagine an agent learning to play a platform game:

• Our Agent receives state \(S_0\) from the Environment — we receive the first frame of our game (Environment).
• Based on that state \(S_0\), the Agent takes action \(A_0\) — our Agent will move to the right.
• The environment goes to a new state \(S_1\) — new frame.
• The environment gives some reward \(R_1\) to the Agent — we’re not dead (Positive Reward +1).

This RL loop outputs a sequence of state, action, reward and next state.

The agent's goal is to maximize its cumulative reward, called the expected return.

The reward hypothesis: the central idea of Reinforcement Learning [[reward-hypothesis]] Copy Chapter

⇒ Why is the goal of the agent to maximize the expected return?

Because RL is based on the reward hypothesis, which is that all goals can be described as the maximization of the expected return (expected cumulative reward).

That’s why in Reinforcement Learning, to have the best behavior, we aim to learn to take actions that maximize the expected cumulative reward.

Markov Property [[markov-property]] Copy Chapter

In papers, you’ll see that the RL process is called a Markov Decision Process (MDP).

We’ll talk again about the Markov Property in the following units. But if you need to remember something today about it, it's this: the Markov Property implies that our agent needs only the current state to decide what action to take and not the history of all the states and actions they took before.

Observations/States Space [[obs-space]] Copy Chapter

Observations/States are the information our agent gets from the environment. In the case of a video game, it can be a frame (a screenshot). In the case of the trading agent, it can be the value of a certain stock, etc.

There is a differentiation to make between observation and state, however:

• State s: is a complete description of the state of the world (there is no hidden information). In a fully observed environment.

In a chess game, we have access to the whole board information, so we receive a state from the environment. In other words, the environment is fully observed.

• Observation o: is a partial description of the state. In a partially observed environment.

In Super Mario Bros, we only see the part of the level close to the player, so we receive an observation.

In Super Mario Bros, we are in a partially observed environment. We receive an observation since we only see a part of the level.

To recap:

Action Space [[action-space]] Copy Chapter

The Action space is the set of all possible actions in an environment.

The actions can come from a discrete or continuous space:

• Discrete space: the number of possible actions is finite.

Again, in Super Mario Bros, we have a finite set of actions since we have only 4 directions.

• Continuous space: the number of possible actions is infinite.

To recap:

Taking this information into consideration is crucial because it will have importance when choosing the RL algorithm in the future.

Rewards and the discounting [[rewards]] Copy Chapter

The reward is fundamental in RL because it’s the only feedback for the agent. Thanks to it, our agent knows if the action taken was good or not.

The cumulative reward at each time step t can be written as:

Which is equivalent to:

However, in reality, we can’t just add them like that. The rewards that come sooner (at the beginning of the game) are more likely to happen since they are more predictable than the long-term future reward.

Let’s say your agent is this tiny mouse that can move one tile each time step, and your opponent is the cat (that can move too). The mouse's goal is to eat the maximum amount of cheese before being eaten by the cat.

As we can see in the diagram, it’s more probable to eat the cheese near us than the cheese close to the cat (the closer we are to the cat, the more dangerous it is).

Consequently, the reward near the cat, even if it is bigger (more cheese), will be more discounted since we’re not really sure we’ll be able to eat it.

To discount the rewards, we proceed like this:

1. We define a discount rate called gamma. It must be between 0 and 1. Most of the time between 0.95 and 0.99.

• The larger the gamma, the smaller the discount. This means our agent cares more about the long-term reward.
• On the other hand, the smaller the gamma, the bigger the discount. This means our agent cares more about the short term reward (the nearest cheese).

2. Then, each reward will be discounted by gamma to the exponent of the time step. As the time step increases, the cat gets closer to us, so the future reward is less and less likely to happen.

Our discounted expected cumulative reward is:

File: units/en/unit1/summary.mdx Copy Chapter

Summary [[summary]]

That was a lot of information! Let's summarize:

• Reinforcement Learning is a computational approach of learning from actions. We build an agent that learns from the environment by interacting with it through trial and error and receiving rewards (negative or positive) as feedback.

• The goal of any RL agent is to maximize its expected cumulative reward (also called expected return) because RL is based on the reward hypothesis, which is that all goals can be described as the maximization of the expected cumulative reward.

• The RL process is a loop that outputs a sequence of state, action, reward and next state.

• To calculate the expected cumulative reward (expected return), we discount the rewards: the rewards that come sooner (at the beginning of the game) are more probable to happen since they are more predictable than the long term future reward.

• To solve an RL problem, you want to find an optimal policy. The policy is the “brain” of your agent, which will tell us what action to take given a state. The optimal policy is the one which gives you the actions that maximize the expected return.

• There are two ways to find your optimal policy:

  1. By training your policy directly: policy-based methods.
  2. By training a value function that tells us the expected return the agent will get at each state and use this function to define our policy: value-based methods.

• Finally, we speak about Deep RL because we introduce deep neural networks to estimate the action to take (policy-based) or to estimate the value of a state (value-based) hence the name “deep”.

File: units/en/unit1/tasks.mdx Copy Chapter

Type of tasks [[tasks]]

A task is an instance of a Reinforcement Learning problem. We can have two types of tasks: episodic and continuing.

Episodic task [[episodic-task]] Copy Chapter

In this case, we have a starting point and an ending point (a terminal state). This creates an episode: a list of States, Actions, Rewards, and new States.

For instance, think about Super Mario Bros: an episode begins at the launch of a new Mario Level and ends when you’re killed or you reached the end of the level.

Continuing tasks [[continuing-tasks]] Copy Chapter

These are tasks that continue forever (no terminal state). In this case, the agent must learn how to choose the best actions and simultaneously interact with the environment.

For instance, an agent that does automated stock trading. For this task, there is no starting point and terminal state. The agent keeps running until we decide to stop it.

To recap:

File: units/en/unit1/two-methods.mdx Copy Chapter

Two main approaches for solving RL problems [[two-methods]]

In other words, how do we build an RL agent that can select the actions that maximize its expected cumulative reward?

The Policy π: the agent’s brain [[policy]] Copy Chapter

The Policy π is the brain of our Agent, it’s the function that tells us what action to take given the state we are in. So it defines the agent’s behavior at a given time.

This Policy is the function we want to learn, our goal is to find the optimal policy π*, the policy that maximizes expected return when the agent acts according to it. We find this π* through training.

There are two approaches to train our agent to find this optimal policy π*:

• Directly, by teaching the agent to learn which action to take, given the current state: Policy-Based Methods.
• Indirectly, teach the agent to learn which state is more valuable and then take the action that leads to the more valuable states: Value-Based Methods.

Policy-Based Methods [[policy-based]] Copy Chapter

In Policy-Based methods, we learn a policy function directly.

This function will define a mapping from each state to the best corresponding action. Alternatively, it could define a probability distribution over the set of possible actions at that state.

We have two types of policies:

• Deterministic: a policy at a given state will always return the same action.

• Stochastic: outputs a probability distribution over actions.

If we recap:

Value-based methods [[value-based]] Copy Chapter

In value-based methods, instead of learning a policy function, we learn a value function that maps a state to the expected value of being at that state.

The value of a state is the expected discounted return the agent can get if it starts in that state, and then acts according to our policy.

“Act according to our policy” just means that our policy is “going to the state with the highest value”.

Here we see that our value function defined values for each possible state.

Thanks to our value function, at each step our policy will select the state with the biggest value defined by the value function: -7, then -6, then -5 (and so on) to attain the goal.

If we recap:

File: units/en/unit1/what-is-rl.mdx Copy Chapter

What is Reinforcement Learning? [[what-is-reinforcement-learning]]

To understand Reinforcement Learning, let’s start with the big picture.

The big picture [[the-big-picture]] Copy Chapter

The idea behind Reinforcement Learning is that an agent (an AI) will learn from the environment by interacting with it (through trial and error) and receiving rewards (negative or positive) as feedback for performing actions.

Learning from interactions with the environment comes from our natural experiences.

For instance, imagine putting your little brother in front of a video game he never played, giving him a controller, and leaving him alone.

Your brother will interact with the environment (the video game) by pressing the right button (action). He got a coin, that’s a +1 reward. It’s positive, he just understood that in this game he must get the coins.

But then, he presses the right button again and he touches an enemy. He just died, so that's a -1 reward.

By interacting with his environment through trial and error, your little brother understands that he needs to get coins in this environment but avoid the enemies.

Without any supervision, the child will get better and better at playing the game.

That’s how humans and animals learn, through interaction. Reinforcement Learning is just a computational approach of learning from actions.

A formal definition [[a-formal-definition]] Copy Section

We can now make a formal definition:

But how does Reinforcement Learning work?

File: units/en/unit2/additional-readings.mdx Copy Chapter

Additional Readings [[additional-readings]]

These are optional readings if you want to go deeper.

Monte Carlo and TD Learning [[mc-td]] Copy Chapter

To dive deeper into Monte Carlo and Temporal Difference Learning:

• Why do temporal difference (TD) methods have lower variance than Monte Carlo methods?
• When are Monte Carlo methods preferred over temporal difference ones?

Q-Learning [[q-learning]] Copy Chapter

• Reinforcement Learning: An Introduction, Richard Sutton and Andrew G. Barto Chapter 5, 6 and 7
• Foundations of Deep RL Series, L2 Deep Q-Learning by Pieter Abbeel

File: units/en/unit2/bellman-equation.mdx Copy Chapter

The Bellman Equation: simplify our value estimation [[bellman-equation]]

The Bellman equation simplifies our state value or state-action value calculation.

With what we have learned so far, we know that if we calculate \(V(S_t)\) (the value of a state), we need to calculate the return starting at that state and then follow the policy forever after. (The policy we defined in the following example is a Greedy Policy; for simplification, we don't discount the reward).

So to calculate \(V(S_t)\), we need to calculate the sum of the expected rewards. Hence:

Then, to calculate the \(V(S_{t+1})\), we need to calculate the return starting at that state \(S_{t+1}\).

So you may have noticed, we're repeating the computation of the value of different states, which can be tedious if you need to do it for each state value or state-action value.

Instead of calculating the expected return for each state or each state-action pair, we can use the Bellman equation. (hint: if you know what Dynamic Programming is, this is very similar! if you don't know what it is, no worries!)

The Bellman equation is a recursive equation that works like this: instead of starting for each state from the beginning and calculating the return, we can consider the value of any state as:

The immediate reward \(R_{t+1}\) + the discounted value of the state that follows ( \(\gamma * V(S_{t+1}) \) ) .

If we go back to our example, we can say that the value of State 1 is equal to the expected cumulative return if we start at that state.

To calculate the value of State 1: the sum of rewards if the agent started in that state 1 and then followed the policy for all the time steps.

This is equivalent to \(V(S_{t})\) = Immediate reward \(R_{t+1}\) + Discounted value of the next state \(\gamma * V(S_{t+1})\)

In the interest of simplicity, here we don't discount, so gamma = 1.
But you'll study an example with gamma = 0.99 in the Q-Learning section of this unit.

• The value of \(V(S_{t+1}) \) = Immediate reward \(R_{t+2}\) + Discounted value of the next state ( \(gamma * V(S_{t+2})\) ).
• And so on.

To recap, the idea of the Bellman equation is that instead of calculating each value as the sum of the expected return, which is a long process, we calculate the value as the sum of immediate reward + the discounted value of the state that follows.

Before going to the next section, think about the role of gamma in the Bellman equation. What happens if the value of gamma is very low (e.g. 0.1 or even 0)? What happens if the value is 1? What happens if the value is very high, such as a million?

File: units/en/unit2/conclusion.mdx Copy Chapter

Conclusion [[conclusion]]

Congrats on finishing this chapter! There was a lot of information. And congrats on finishing the tutorials. You’ve just implemented your first RL agent from scratch and shared it on the Hub 🥳.

Implementing from scratch when you study a new architecture is important to understand how it works.

It's normal if you still feel confused by all these elements. This was the same for me and for everyone who studies RL.

Take time to really grasp the material before continuing.

In the next chapter, we’re going to dive deeper by studying our first Deep Reinforcement Learning algorithm based on Q-Learning: Deep Q-Learning. And you'll train a DQN agent with RL-Baselines3 Zoo to play Atari Games.

Finally, we would love to hear what you think of the course and how we can improve it. If you have some feedback then please 👉 fill this form

Keep Learning, stay awesome 🤗 Copy Section

File: units/en/unit2/glossary.mdx Copy Chapter

Glossary [[glossary]]

This is a community-created glossary. Contributions are welcomed!

Strategies to find the optimal policy Copy Section

• Policy-based methods. The policy is usually trained with a neural network to select what action to take given a state. In this case it is the neural network which outputs the action that the agent should take instead of using a value function. Depending on the experience received by the environment, the neural network will be re-adjusted and will provide better actions.
• Value-based methods. In this case, a value function is trained to output the value of a state or a state-action pair that will represent our policy. However, this value doesn't define what action the agent should take. In contrast, we need to specify the behavior of the agent given the output of the value function. For example, we could decide to adopt a policy to take the action that always leads to the biggest reward (Greedy Policy). In summary, the policy is a Greedy Policy (or whatever decision the user takes) that uses the values of the value-function to decide the actions to take.

Among the value-based methods, we can find two main strategies Copy Section

• The state-value function. For each state, the state-value function is the expected return if the agent starts in that state and follows the policy until the end.
• The action-value function. In contrast to the state-value function, the action-value calculates for each state and action pair the expected return if the agent starts in that state, takes that action, and then follows the policy forever after.

Epsilon-greedy strategy: Copy Section

• Common strategy used in reinforcement learning that involves balancing exploration and exploitation.
• Chooses the action with the highest expected reward with a probability of 1-epsilon.
• Chooses a random action with a probability of epsilon.
• Epsilon is typically decreased over time to shift focus towards exploitation.

Greedy strategy: Copy Section

• Involves always choosing the action that is expected to lead to the highest reward, based on the current knowledge of the environment. (Only exploitation)
• Always chooses the action with the highest expected reward.
• Does not include any exploration.
• Can be disadvantageous in environments with uncertainty or unknown optimal actions.

Off-policy vs on-policy algorithms Copy Section

• Off-policy algorithms: A different policy is used at training time and inference time
• On-policy algorithms: The same policy is used during training and inference

Monte Carlo and Temporal Difference learning strategies Copy Section

• Monte Carlo (MC): Learning at the end of the episode. With Monte Carlo, we wait until the episode ends and then we update the value function (or policy function) from a complete episode.

• Temporal Difference (TD): Learning at each step. With Temporal Difference Learning, we update the value function (or policy function) at each step without requiring a complete episode.

If you want to improve the course, you can open a Pull Request.

This glossary was made possible thanks to:

• Ramón Rueda
• Hasarindu Perera
• Arkady Arkhangorodsky

File: units/en/unit2/hands-on.mdx Copy Chapter

Hands-on [[hands-on]]

<CourseFloatingBanner classNames="absolute z-10 right-0 top-0"
  notebooks={[
    {label: "Google Colab", value: "https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/notebooks/unit2/unit2.ipynb"}
    ]}
    askForHelpUrl="http://hf.co/join/discord" />

Now that we studied the Q-Learning algorithm, let's implement it from scratch and train our Q-Learning agent in two environments:

1. Frozen-Lake-v1 (non-slippery and slippery version) ☃️ : where our agent will need to go from the starting state (S) to the goal state (G) by walking only on frozen tiles (F) and avoiding holes (H).
2. An autonomous taxi 🚖 will need to learn to navigate a city to transport its passengers from point A to point B.

Thanks to a leaderboard, you'll be able to compare your results with other classmates and exchange the best practices to improve your agent's scores. Who will win the challenge for Unit 2?

To validate this hands-on for the certification process, you need to push your trained Taxi model to the Hub and get a result of >= 4.5.

To find your result, go to the leaderboard and find your model, the result = mean_reward - std of reward

For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process

And you can check your progress here 👉 https://huggingface.co/spaces/ThomasSimonini/Check-my-progress-Deep-RL-Course

To start the hands-on click on the Open In Colab button 👇 :

We strongly recommend students use Google Colab for the hands-on exercises instead of running them on their personal computers.

By using Google Colab, you can focus on learning and experimenting without worrying about the technical aspects of setting up your environments.

Unit 2: Q-Learning with FrozenLake-v1 ⛄ and Taxi-v3 🚕

In this notebook, you'll code your first Reinforcement Learning agent from scratch to play FrozenLake ❄️ using Q-Learning, share it with the community, and experiment with different configurations.

⬇️ Here is an example of what you will achieve in just a couple of minutes. ⬇️

🎮 Environments: Copy Section

• FrozenLake-v1
• Taxi-v3

📚 RL-Library: Copy Section

• Python and NumPy
• Gymnasium

We're constantly trying to improve our tutorials, so if you find some issues in this notebook, please open an issue on the GitHub Repo.

Objectives of this notebook 🏆 Copy Chapter

At the end of the notebook, you will:

• Be able to use Gymnasium, the environment library.
• Be able to code a Q-Learning agent from scratch.
• Be able to push your trained agent and the code to the Hub with a nice video replay and an evaluation score 🔥.

This notebook is from the Deep Reinforcement Learning Course Copy Chapter

In this free course, you will:

• 📖 Study Deep Reinforcement Learning in theory and practice.
• 🧑‍💻 Learn to use famous Deep RL libraries such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.
• 🤖 Train agents in unique environments

And more check 📚 the syllabus 👉 https://simoninithomas.github.io/deep-rl-course

Don’t forget to sign up to the course (we are collecting your email to be able to send you the links when each Unit is published and give you information about the challenges and updates).

The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5

Prerequisites 🏗️ Copy Chapter

Before diving into the notebook, you need to:

🔲 📚 Study Q-Learning by reading Unit 2 🤗

A small recap of Q-Learning Copy Chapter

Q-Learning is the RL algorithm that:

• Trains Q-Function, an action-value function that is encoded, in internal memory, by a Q-table that contains all the state-action pair values.

• Given a state and action, our Q-Function will search the Q-table for the corresponding value.

• When the training is done, we have an optimal Q-Function, so an optimal Q-Table.

• And if we have an optimal Q-function, we
  have an optimal policy, since we know for each state, the best action to take.

But, in the beginning, our Q-Table is useless since it gives arbitrary value for each state-action pair (most of the time we initialize the Q-Table to 0 values). But, as we’ll explore the environment and update our Q-Table it will give us better and better approximations

This is the Q-Learning pseudocode:

Let's code our first Reinforcement Learning algorithm 🚀

To validate this hands-on for the certification process, you need to push your trained Taxi model to the Hub and get a result of >= 4.5.

To find your result, go to the leaderboard and find your model, the result = mean_reward - std of reward

For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process

Install dependencies and create a virtual display 🔽 Copy Chapter

In the notebook, we'll need to generate a replay video. To do so, with Colab, we need to have a virtual screen to render the environment (and thus record the frames).

Hence the following cell will install the libraries and create and run a virtual screen 🖥

We’ll install multiple ones:

• gymnasium: Contains the FrozenLake-v1 ⛄ and Taxi-v3 🚕 environments.
• pygame: Used for the FrozenLake-v1 and Taxi-v3 UI.
• numpy: Used for handling our Q-table.

The Hugging Face Hub 🤗 works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easily collaborate with others.

You can see here all the Deep RL models available (if they use Q Learning) here 👉 https://huggingface.co/models?other=q-learning

pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit2/requirements-unit2.txt

sudo apt-get update
sudo apt-get install -y python3-opengl
apt install ffmpeg xvfb
pip3 install pyvirtualdisplay

To make sure the new installed libraries are used, sometimes it's required to restart the notebook runtime. The next cell will force the runtime to crash, so you'll need to connect again and run the code starting from here. Thanks to this trick, we will be able to run our virtual screen.

import os

os.kill(os.getpid(), 9)

# Virtual display
from pyvirtualdisplay import Display

virtual_display = Display(visible=0, size=(1400, 900))
virtual_display.start()

Import the packages 📦 Copy Chapter

In addition to the installed libraries, we also use:

• random: To generate random numbers (that will be useful for epsilon-greedy policy).
• imageio: To generate a replay video.

import numpy as np
import gymnasium as gym
import random
import imageio
import os
import tqdm

import pickle5 as pickle
from tqdm.notebook import tqdm

We're now ready to code our Q-Learning algorithm 🔥

Part 1: Frozen Lake ⛄ (non slippery version)

Create and understand [FrozenLake environment ⛄]((https://gymnasium.farama.org/environments/toy_text/frozen_lake/) Copy Chapter

💡 A good habit when you start to use an environment is to check its documentation

👉 https://gymnasium.farama.org/environments/toy_text/frozen_lake/

We're going to train our Q-Learning agent to navigate from the starting state (S) to the goal state (G) by walking only on frozen tiles (F) and avoid holes (H).

We can have two sizes of environment:

• map_name="4x4": a 4x4 grid version
• map_name="8x8": a 8x8 grid version

The environment has two modes:

• is_slippery=False: The agent always moves in the intended direction due to the non-slippery nature of the frozen lake (deterministic).
• is_slippery=True: The agent may not always move in the intended direction due to the slippery nature of the frozen lake (stochastic).

For now let's keep it simple with the 4x4 map and non-slippery.
We add a parameter called render_mode that specifies how the environment should be visualised. In our case because we want to record a video of the environment at the end, we need to set render_mode to rgb_array.

As explained in the documentation “rgb_array”: Return a single frame representing the current state of the environment. A frame is a np.ndarray with shape (x, y, 3) representing RGB values for an x-by-y pixel image.

# Create the FrozenLake-v1 environment using 4x4 map and non-slippery version and render_mode="rgb_array"
env = gym.make()  # TODO use the correct parameters

Solution Copy Section

env = gym.make("FrozenLake-v1", map_name="4x4", is_slippery=False, render_mode="rgb_array")

You can create your own custom grid like this:

desc=["SFFF", "FHFH", "FFFH", "HFFG"]
gym.make('FrozenLake-v1', desc=desc, is_slippery=True)

but we'll use the default environment for now.

Let's see what the Environment looks like: Copy Section

# We create our environment with gym.make("<name_of_the_environment>")- `is_slippery=False`: The agent always moves in the intended direction due to the non-slippery nature of the frozen lake (deterministic).
print("_____OBSERVATION SPACE_____ \n")
print("Observation Space", env.observation_space)
print("Sample observation", env.observation_space.sample())  # Get a random observation

We see with Observation Space Shape Discrete(16) that the observation is an integer representing the agent’s current position as current_row * ncols + current_col (where both the row and col start at 0).

For example, the goal position in the 4x4 map can be calculated as follows: 3 * 4 + 3 = 15. The number of possible observations is dependent on the size of the map. For example, the 4x4 map has 16 possible observations.

For instance, this is what state = 0 looks like:

print("\n _____ACTION SPACE_____ \n")
print("Action Space Shape", env.action_space.n)
print("Action Space Sample", env.action_space.sample())  # Take a random action

The action space (the set of possible actions the agent can take) is discrete with 4 actions available 🎮:

• 0: GO LEFT
• 1: GO DOWN
• 2: GO RIGHT
• 3: GO UP

Reward function 💰:

• Reach goal: +1
• Reach hole: 0
• Reach frozen: 0

Create and Initialize the Q-table 🗄️ Copy Chapter

(👀 Step 1 of the pseudocode)

It's time to initialize our Q-table! To know how many rows (states) and columns (actions) to use, we need to know the action and observation space. We already know their values from before, but we'll want to obtain them programmatically so that our algorithm generalizes for different environments. Gym provides us a way to do that: env.action_space.n and env.observation_space.n

state_space =
print("There are ", state_space, " possible states")

action_space =
print("There are ", action_space, " possible actions")

# Let's create our Qtable of size (state_space, action_space) and initialized each values at 0 using np.zeros. np.zeros needs a tuple (a,b)
def initialize_q_table(state_space, action_space):
  Qtable =
  return Qtable

Qtable_frozenlake = initialize_q_table(state_space, action_space)

Solution Copy Section

state_space = env.observation_space.n
print("There are ", state_space, " possible states")

action_space = env.action_space.n
print("There are ", action_space, " possible actions")

# Let's create our Qtable of size (state_space, action_space) and initialized each values at 0 using np.zeros
def initialize_q_table(state_space, action_space):
    Qtable = np.zeros((state_space, action_space))
    return Qtable

Qtable_frozenlake = initialize_q_table(state_space, action_space)

Define the greedy policy 🤖 Copy Chapter

Remember we have two policies since Q-Learning is an off-policy algorithm. This means we're using a different policy for acting and updating the value function.

• Epsilon-greedy policy (acting policy)
• Greedy-policy (updating policy)

The greedy policy will also be the final policy we'll have when the Q-learning agent completes training. The greedy policy is used to select an action using the Q-table.

def greedy_policy(Qtable, state):
  # Exploitation: take the action with the highest state, action value
  action =

  return action

Solution

def greedy_policy(Qtable, state):
    # Exploitation: take the action with the highest state, action value
    action = np.argmax(Qtable[state][:])

    return action

Define the epsilon-greedy policy 🤖 Copy Chapter

Epsilon-greedy is the training policy that handles the exploration/exploitation trade-off.

The idea with epsilon-greedy:

• With probability 1 - ɛ : we do exploitation (i.e. our agent selects the action with the highest state-action pair value).

• With probability ɛ: we do exploration (trying a random action).

As the training continues, we progressively reduce the epsilon value since we will need less and less exploration and more exploitation.

def epsilon_greedy_policy(Qtable, state, epsilon):
  # Randomly generate a number between 0 and 1
  random_num =
  # if random_num > greater than epsilon --> exploitation
  if random_num > epsilon:
    # Take the action with the highest value given a state
    # np.argmax can be useful here
    action =
  # else --> exploration
  else:
    action = # Take a random action

  return action

Solution

def epsilon_greedy_policy(Qtable, state, epsilon):
    # Randomly generate a number between 0 and 1
    random_num = random.uniform(0, 1)
    # if random_num > greater than epsilon --> exploitation
    if random_num > epsilon:
        # Take the action with the highest value given a state
        # np.argmax can be useful here
        action = greedy_policy(Qtable, state)
    # else --> exploration
    else:
        action = env.action_space.sample()

    return action

Define the hyperparameters ⚙️ Copy Chapter

The exploration related hyperparamters are some of the most important ones.

• We need to make sure that our agent explores enough of the state space to learn a good value approximation. To do that, we need to have progressive decay of the epsilon.
• If you decrease epsilon too fast (too high decay_rate), you take the risk that your agent will be stuck, since your agent didn't explore enough of the state space and hence can't solve the problem.

# Training parameters
n_training_episodes = 10000  # Total training episodes
learning_rate = 0.7  # Learning rate

# Evaluation parameters
n_eval_episodes = 100  # Total number of test episodes

# Environment parameters
env_id = "FrozenLake-v1"  # Name of the environment
max_steps = 99  # Max steps per episode
gamma = 0.95  # Discounting rate
eval_seed = []  # The evaluation seed of the environment

# Exploration parameters
max_epsilon = 1.0  # Exploration probability at start
min_epsilon = 0.05  # Minimum exploration probability
decay_rate = 0.0005  # Exponential decay rate for exploration prob

Create the training loop method Copy Chapter

The training loop goes like this:

For episode in the total of training episodes:

Reduce epsilon (since we need less and less exploration)
Reset the environment

  For step in max timesteps:
    Choose the action At using epsilon greedy policy
    Take the action (a) and observe the outcome state(s') and reward (r)
    Update the Q-value Q(s,a) using Bellman equation Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]
    If done, finish the episode
    Our next state is the new state

def train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable):
  for episode in tqdm(range(n_training_episodes)):
    # Reduce epsilon (because we need less and less exploration)
    epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode)
    # Reset the environment
    state, info = env.reset()
    step = 0
    terminated = False
    truncated = False

    # repeat
    for step in range(max_steps):
      # Choose the action At using epsilon greedy policy
      action =

      # Take action At and observe Rt+1 and St+1
      # Take the action (a) and observe the outcome state(s') and reward (r)
      new_state, reward, terminated, truncated, info =

      # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]
      Qtable[state][action] =

      # If terminated or truncated finish the episode
      if terminated or truncated:
        break

      # Our next state is the new state
      state = new_state
  return Qtable

Solution

def train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable):
    for episode in tqdm(range(n_training_episodes)):
        # Reduce epsilon (because we need less and less exploration)
        epsilon = min_epsilon + (max_epsilon - min_epsilon) * np.exp(-decay_rate * episode)
        # Reset the environment
        state, info = env.reset()
        step = 0
        terminated = False
        truncated = False

        # repeat
        for step in range(max_steps):
            # Choose the action At using epsilon greedy policy
            action = epsilon_greedy_policy(Qtable, state, epsilon)

            # Take action At and observe Rt+1 and St+1
            # Take the action (a) and observe the outcome state(s') and reward (r)
            new_state, reward, terminated, truncated, info = env.step(action)

            # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]
            Qtable[state][action] = Qtable[state][action] + learning_rate * (
                reward + gamma * np.max(Qtable[new_state]) - Qtable[state][action]
            )

            # If terminated or truncated finish the episode
            if terminated or truncated:
                break

            # Our next state is the new state
            state = new_state
    return Qtable

Train the Q-Learning agent 🏃 Copy Chapter

Qtable_frozenlake = train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable_frozenlake)

Let's see what our Q-Learning table looks like now 👀 Copy Chapter

Qtable_frozenlake

The evaluation method 📝 Copy Chapter

• We defined the evaluation method that we're going to use to test our Q-Learning agent.

def evaluate_agent(env, max_steps, n_eval_episodes, Q, seed):
    """
    Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.
    :param env: The evaluation environment
    :param n_eval_episodes: Number of episode to evaluate the agent
    :param Q: The Q-table
    :param seed: The evaluation seed array (for taxi-v3)
    """
    episode_rewards = []
    for episode in tqdm(range(n_eval_episodes)):
        if seed:
            state, info = env.reset(seed=seed[episode])
        else:
            state, info = env.reset()
        step = 0
        truncated = False
        terminated = False
        total_rewards_ep = 0

        for step in range(max_steps):
            # Take the action (index) that have the maximum expected future reward given that state
            action = greedy_policy(Q, state)
            new_state, reward, terminated, truncated, info = env.step(action)
            total_rewards_ep += reward

            if terminated or truncated:
                break
            state = new_state
        episode_rewards.append(total_rewards_ep)
    mean_reward = np.mean(episode_rewards)
    std_reward = np.std(episode_rewards)

    return mean_reward, std_reward

Evaluate our Q-Learning agent 📈 Copy Chapter

• Usually, you should have a mean reward of 1.0
• The environment is relatively easy since the state space is really small (16). What you can try to do is to replace it with the slippery version, which introduces stochasticity, making the environment more complex.

# Evaluate our Agent
mean_reward, std_reward = evaluate_agent(env, max_steps, n_eval_episodes, Qtable_frozenlake, eval_seed)
print(f"Mean_reward={mean_reward:.2f} +/- {std_reward:.2f}")

Publish our trained model to the Hub 🔥 Copy Chapter

Now that we saw good results after the training, we can publish our trained model to the Hub 🤗 with one line of code.

Here's an example of a Model Card:

Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent.

Do not modify this code

from huggingface_hub import HfApi, snapshot_download
from huggingface_hub.repocard import metadata_eval_result, metadata_save

from pathlib import Path
import datetime
import json

def record_video(env, Qtable, out_directory, fps=1):
    """
    Generate a replay video of the agent
    :param env
    :param Qtable: Qtable of our agent
    :param out_directory
    :param fps: how many frame per seconds (with taxi-v3 and frozenlake-v1 we use 1)
    """
    images = []
    terminated = False
    truncated = False
    state, info = env.reset(seed=random.randint(0, 500))
    img = env.render()
    images.append(img)
    while not terminated or truncated:
        # Take the action (index) that have the maximum expected future reward given that state
        action = np.argmax(Qtable[state][:])
        state, reward, terminated, truncated, info = env.step(
            action
        )  # We directly put next_state = state for recording logic
        img = env.render()
        images.append(img)
    imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)

def push_to_hub(repo_id, model, env, video_fps=1, local_repo_path="hub"):
    """
    Evaluate, Generate a video and Upload a model to Hugging Face Hub.
    This method does the complete pipeline:
    - It evaluates the model
    - It generates the model card
    - It generates a replay video of the agent
    - It pushes everything to the Hub

    :param repo_id: repo_id: id of the model repository from the Hugging Face Hub
    :param env
    :param video_fps: how many frame per seconds to record our video replay
    (with taxi-v3 and frozenlake-v1 we use 1)
    :param local_repo_path: where the local repository is
    """
    _, repo_name = repo_id.split("/")

    eval_env = env
    api = HfApi()

    # Step 1: Create the repo
    repo_url = api.create_repo(
        repo_id=repo_id,
        exist_ok=True,
    )

    # Step 2: Download files
    repo_local_path = Path(snapshot_download(repo_id=repo_id))

    # Step 3: Save the model
    if env.spec.kwargs.get("map_name"):
        model["map_name"] = env.spec.kwargs.get("map_name")
        if env.spec.kwargs.get("is_slippery", "") == False:
            model["slippery"] = False

    # Pickle the model
    with open((repo_local_path) / "q-learning.pkl", "wb") as f:
        pickle.dump(model, f)

    # Step 4: Evaluate the model and build JSON with evaluation metrics
    mean_reward, std_reward = evaluate_agent(
        eval_env, model["max_steps"], model["n_eval_episodes"], model["qtable"], model["eval_seed"]
    )

    evaluate_data = {
        "env_id": model["env_id"],
        "mean_reward": mean_reward,
        "n_eval_episodes": model["n_eval_episodes"],
        "eval_datetime": datetime.datetime.now().isoformat(),
    }

    # Write a JSON file called "results.json" that will contain the
    # evaluation results
    with open(repo_local_path / "results.json", "w") as outfile:
        json.dump(evaluate_data, outfile)

    # Step 5: Create the model card
    env_name = model["env_id"]
    if env.spec.kwargs.get("map_name"):
        env_name += "-" + env.spec.kwargs.get("map_name")

    if env.spec.kwargs.get("is_slippery", "") == False:
        env_name += "-" + "no_slippery"

    metadata = {}
    metadata["tags"] = [env_name, "q-learning", "reinforcement-learning", "custom-implementation"]

    # Add metrics
    eval = metadata_eval_result(
        model_pretty_name=repo_name,
        task_pretty_name="reinforcement-learning",
        task_id="reinforcement-learning",
        metrics_pretty_name="mean_reward",
        metrics_id="mean_reward",
        metrics_value=f"{mean_reward:.2f} +/- {std_reward:.2f}",
        dataset_pretty_name=env_name,
        dataset_id=env_name,
    )

    # Merges both dictionaries
    metadata = {**metadata, **eval}

    model_card = f"""
  # **Q-Learning** Agent playing1 **{env_id}**
  This is a trained model of a **Q-Learning** agent playing **{env_id}** .

  ## Usage

  model = load_from_hub(repo_id="{repo_id}", filename="q-learning.pkl")

  # Don't forget to check if you need to add additional attributes (is_slippery=False etc)
  env = gym.make(model["env_id"])
  """

    evaluate_agent(env, model["max_steps"], model["n_eval_episodes"], model["qtable"], model["eval_seed"])

    readme_path = repo_local_path / "README.md"
    readme = ""
    print(readme_path.exists())
    if readme_path.exists():
        with readme_path.open("r", encoding="utf8") as f:
            readme = f.read()
    else:
        readme = model_card

    with readme_path.open("w", encoding="utf-8") as f:
        f.write(readme)

    # Save our metrics to Readme metadata
    metadata_save(readme_path, metadata)

    # Step 6: Record a video
    video_path = repo_local_path / "replay.mp4"
    record_video(env, model["qtable"], video_path, video_fps)

    # Step 7. Push everything to the Hub
    api.upload_folder(
        repo_id=repo_id,
        folder_path=repo_local_path,
        path_in_repo=".",
    )

    print("Your model is pushed to the Hub. You can view your model here: ", repo_url)

. Copy Section

By using push_to_hub you evaluate, record a replay, generate a model card of your agent and push it to the Hub.

This way:

• You can showcase our work 🔥
• You can visualize your agent playing 👀
• You can share an agent with the community that others can use 💾
• You can access a leaderboard 🏆 to see how well your agent is performing compared to your classmates 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard

To be able to share your model with the community there are three more steps to follow:

1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join

2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.

• Create a new token (https://huggingface.co/settings/tokens) with write role

from huggingface_hub import notebook_login

notebook_login()

If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: huggingface-cli login (or login)

3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥 using push_to_hub() function

• Let's create the model dictionary that contains the hyperparameters and the Q_table.

model = {
    "env_id": env_id,
    "max_steps": max_steps,
    "n_training_episodes": n_training_episodes,
    "n_eval_episodes": n_eval_episodes,
    "eval_seed": eval_seed,
    "learning_rate": learning_rate,
    "gamma": gamma,
    "max_epsilon": max_epsilon,
    "min_epsilon": min_epsilon,
    "decay_rate": decay_rate,
    "qtable": Qtable_frozenlake,
}

Let's fill the push_to_hub function:

• repo_id: the name of the Hugging Face Hub Repository that will be created/updated (repo_id = {username}/{repo_name})
  💡 A good repo_id is {username}/q-{env_id}
• model: our model dictionary containing the hyperparameters and the Qtable.
• env: the environment.
• commit_message: message of the commit

model

username = ""  # FILL THIS
repo_name = "q-FrozenLake-v1-4x4-noSlippery"
push_to_hub(repo_id=f"{username}/{repo_name}", model=model, env=env)

Congrats 🥳 you've just implemented from scratch, trained, and uploaded your first Reinforcement Learning agent.
FrozenLake-v1 no_slippery is very simple environment, let's try a harder one 🔥.

Part 2: Taxi-v3 🚖

Create and understand [Taxi-v3 🚕](https://gymnasium.farama.org/environments/toy_text/taxi/) Copy Chapter

💡 A good habit when you start to use an environment is to check its documentation

👉 https://gymnasium.farama.org/environments/toy_text/taxi/

In Taxi-v3 🚕, there are four designated locations in the grid world indicated by R(ed), G(reen), Y(ellow), and B(lue).

When the episode starts, the taxi starts off at a random square and the passenger is at a random location. The taxi drives to the passenger’s location, picks up the passenger, drives to the passenger’s destination (another one of the four specified locations), and then drops off the passenger. Once the passenger is dropped off, the episode ends.

env = gym.make("Taxi-v3", render_mode="rgb_array")

There are 500 discrete states since there are 25 taxi positions, 5 possible locations of the passenger (including the case when the passenger is in the taxi), and 4 destination locations.

state_space = env.observation_space.n
print("There are ", state_space, " possible states")

action_space = env.action_space.n
print("There are ", action_space, " possible actions")

The action space (the set of possible actions the agent can take) is discrete with 6 actions available 🎮:

• 0: move south
• 1: move north
• 2: move east
• 3: move west
• 4: pickup passenger
• 5: drop off passenger

Reward function 💰:

• -1 per step unless other reward is triggered.
• +20 delivering passenger.
• -10 executing “pickup” and “drop-off” actions illegally.

# Create our Q table with state_size rows and action_size columns (500x6)
Qtable_taxi = initialize_q_table(state_space, action_space)
print(Qtable_taxi)
print("Q-table shape: ", Qtable_taxi.shape)

Define the hyperparameters ⚙️ Copy Chapter

⚠ DO NOT MODIFY EVAL_SEED: the eval_seed array allows us to evaluate your agent with the same taxi starting positions for every classmate

# Training parameters
n_training_episodes = 25000  # Total training episodes
learning_rate = 0.7  # Learning rate

# Evaluation parameters
n_eval_episodes = 100  # Total number of test episodes

# DO NOT MODIFY EVAL_SEED
eval_seed = [
    16,
    54,
    165,
    177,
    191,
    191,
    120,
    80,
    149,
    178,
    48,
    38,
    6,
    125,
    174,
    73,
    50,
    172,
    100,
    148,
    146,
    6,
    25,
    40,
    68,
    148,
    49,
    167,
    9,
    97,
    164,
    176,
    61,
    7,
    54,
    55,
    161,
    131,
    184,
    51,
    170,
    12,
    120,
    113,
    95,
    126,
    51,
    98,
    36,
    135,
    54,
    82,
    45,
    95,
    89,
    59,
    95,
    124,
    9,
    113,
    58,
    85,
    51,
    134,
    121,
    169,
    105,
    21,
    30,
    11,
    50,
    65,
    12,
    43,
    82,
    145,
    152,
    97,
    106,
    55,
    31,
    85,
    38,
    112,
    102,
    168,
    123,
    97,
    21,
    83,
    158,
    26,
    80,
    63,
    5,
    81,
    32,
    11,
    28,
    148,
]  # Evaluation seed, this ensures that all classmates agents are trained on the same taxi starting position
# Each seed has a specific starting state

# Environment parameters
env_id = "Taxi-v3"  # Name of the environment
max_steps = 99  # Max steps per episode
gamma = 0.95  # Discounting rate

# Exploration parameters
max_epsilon = 1.0  # Exploration probability at start
min_epsilon = 0.05  # Minimum exploration probability
decay_rate = 0.005  # Exponential decay rate for exploration prob

Train our Q-Learning agent 🏃 Copy Chapter

Qtable_taxi = train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable_taxi)
Qtable_taxi

Create a model dictionary 💾 and publish our trained model to the Hub 🔥 Copy Chapter

• We create a model dictionary that will contain all the training hyperparameters for reproducibility and the Q-Table.

model = {
    "env_id": env_id,
    "max_steps": max_steps,
    "n_training_episodes": n_training_episodes,
    "n_eval_episodes": n_eval_episodes,
    "eval_seed": eval_seed,
    "learning_rate": learning_rate,
    "gamma": gamma,
    "max_epsilon": max_epsilon,
    "min_epsilon": min_epsilon,
    "decay_rate": decay_rate,
    "qtable": Qtable_taxi,
}

username = ""  # FILL THIS
repo_name = ""  # FILL THIS
push_to_hub(repo_id=f"{username}/{repo_name}", model=model, env=env)

Now that it's on the Hub, you can compare the results of your Taxi-v3 with your classmates using the leaderboard 🏆 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard

Part 3: Load from Hub 🔽

What's amazing with Hugging Face Hub 🤗 is that you can easily load powerful models from the community.

Loading a saved model from the Hub is really easy:

1. You go https://huggingface.co/models?other=q-learning to see the list of all the q-learning saved models.

2. You select one and copy its repo_id

3. Then we just need to use load_from_hub with:

• The repo_id
• The filename: the saved model inside the repo.

Do not modify this code

from urllib.error import HTTPError

from huggingface_hub import hf_hub_download


def load_from_hub(repo_id: str, filename: str) -> str:
    """
    Download a model from Hugging Face Hub.
    :param repo_id: id of the model repository from the Hugging Face Hub
    :param filename: name of the model zip file from the repository
    """
    # Get the model from the Hub, download and cache the model on your local disk
    pickle_model = hf_hub_download(repo_id=repo_id, filename=filename)

    with open(pickle_model, "rb") as f:
        downloaded_model_file = pickle.load(f)

    return downloaded_model_file

. Copy Section

model = load_from_hub(repo_id="ThomasSimonini/q-Taxi-v3", filename="q-learning.pkl")  # Try to use another model

print(model)
env = gym.make(model["env_id"])

evaluate_agent(env, model["max_steps"], model["n_eval_episodes"], model["qtable"], model["eval_seed"])

model = load_from_hub(
    repo_id="ThomasSimonini/q-FrozenLake-v1-no-slippery", filename="q-learning.pkl"
)  # Try to use another model

env = gym.make(model["env_id"], is_slippery=False)

evaluate_agent(env, model["max_steps"], model["n_eval_episodes"], model["qtable"], model["eval_seed"])

Some additional challenges 🏆 Copy Chapter

The best way to learn is to try things on your own! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results!

In the Leaderboard you will find your agents. Can you get to the top?

Here are some ideas to climb up the leaderboard:

• Train more steps
• Try different hyperparameters by looking at what your classmates have done.
• Push your new trained model on the Hub 🔥

Are walking on ice and driving taxis too boring to you? Try to change the environment, why not use FrozenLake-v1 slippery version? Check how they work using the gymnasium documentation and have fun 🎉.

Congrats 🥳, you've just implemented, trained, and uploaded your first Reinforcement Learning agent.

Understanding Q-Learning is an important step to understanding value-based methods.

In the next Unit with Deep Q-Learning, we'll see that while creating and updating a Q-table was a good strategy — however, it is not scalable.

For instance, imagine you create an agent that learns to play Doom.

Doom is a large environment with a huge state space (millions of different states). Creating and updating a Q-table for that environment would not be efficient.

That's why we'll study Deep Q-Learning in the next unit, an algorithm where we use a neural network that approximates, given a state, the different Q-values for each action.

See you in Unit 3! 🔥

Keep learning, stay awesome 🤗 Copy Chapter

File: units/en/unit2/introduction.mdx Copy Chapter

Introduction to Q-Learning [[introduction-q-learning]]

In the first unit of this class, we learned about Reinforcement Learning (RL), the RL process, and the different methods to solve an RL problem. We also trained our first agents and uploaded them to the Hugging Face Hub.

In this unit, we're going to dive deeper into one of the Reinforcement Learning methods: value-based methods and study our first RL algorithm: Q-Learning.

We'll also implement our first RL agent from scratch, a Q-Learning agent, and will train it in two environments:

1. Frozen-Lake-v1 (non-slippery version): where our agent will need to go from the starting state (S) to the goal state (G) by walking only on frozen tiles (F) and avoiding holes (H).
2. An autonomous taxi: where our agent will need to learn to navigate a city to transport its passengers from point A to point B.

Concretely, we will:

• Learn about value-based methods.
• Learn about the differences between Monte Carlo and Temporal Difference Learning.
• Study and implement our first RL algorithm: Q-Learning.

This unit is fundamental if you want to be able to work on Deep Q-Learning: the first Deep RL algorithm that played Atari games and beat the human level on some of them (breakout, space invaders, etc).

So let's get started! 🚀

File: units/en/unit2/mc-vs-td.mdx Copy Chapter

Monte Carlo vs Temporal Difference Learning [[mc-vs-td]]

The last thing we need to discuss before diving into Q-Learning is the two learning strategies.

Remember that an RL agent learns by interacting with its environment. The idea is that given the experience and the received reward, the agent will update its value function or policy.

Monte Carlo and Temporal Difference Learning are two different strategies on how to train our value function or our policy function. Both of them use experience to solve the RL problem.

On one hand, Monte Carlo uses an entire episode of experience before learning. On the other hand, Temporal Difference uses only a step ( \(S_t, A_t, R_{t+1}, S_{t+1}\) ) to learn.

We'll explain both of them using a value-based method example.

Monte Carlo: learning at the end of the episode [[monte-carlo]] Copy Chapter

Monte Carlo waits until the end of the episode, calculates \(G_t\) (return) and uses it as a target for updating \(V(S_t)\).

So it requires a complete episode of interaction before updating our value function.

If we take an example:

• We always start the episode at the same starting point.

• The agent takes actions using the policy. For instance, using an Epsilon Greedy Strategy, a policy that alternates between exploration (random actions) and exploitation.

• We get the reward and the next state.

• We terminate the episode if the cat eats the mouse or if the mouse moves > 10 steps.

• At the end of the episode, we have a list of State, Actions, Rewards, and Next States tuples
  For instance [[State tile 3 bottom, Go Left, +1, State tile 2 bottom], [State tile 2 bottom, Go Left, +0, State tile 1 bottom]...]

• The agent will sum the total rewards \(G_t\) (to see how well it did).

• It will then update \(V(s_t)\) based on the formula

• Then start a new game with this new knowledge

By running more and more episodes, the agent will learn to play better and better.

For instance, if we train a state-value function using Monte Carlo:

• We initialize our value function so that it returns 0 value for each state

• Our learning rate (lr) is 0.1 and our discount rate is 1 (= no discount)

• Our mouse explores the environment and takes random actions

• The mouse made more than 10 steps, so the episode ends .

• We have a list of state, action, rewards, next_state, we need to calculate the return \(G{t=0}\)

\(G_t = R_{t+1} + R_{t+2} + R_{t+3} ...\) (for simplicity, we don't discount the rewards)

\(G_0 = R_{1} + R_{2} + R_{3}…\)

\(G_0 = 1 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0\)

\(G_0 = 3\)

• We can now compute the new \(V(S_0)\):

\(V(S_0) = V(S_0) + lr * [G_0 — V(S_0)]\)

\(V(S_0) = 0 + 0.1 * [3 – 0]\)

\(V(S_0) = 0.3\)

Temporal Difference Learning: learning at each step [[td-learning]] Copy Chapter

Temporal Difference, on the other hand, waits for only one interaction (one step) \(S_{t+1}\) to form a TD target and update \(V(S_t)\) using \(R_{t+1}\) and \( \gamma * V(S_{t+1})\).

The idea with TD is to update the \(V(S_t)\) at each step.

But because we didn't experience an entire episode, we don't have \(G_t\) (expected return). Instead, we estimate \(G_t\) by adding \(R_{t+1}\) and the discounted value of the next state.

This is called bootstrapping. It's called this because TD bases its update in part on an existing estimate \(V(S_{t+1})\) and not a complete sample \(G_t\).

This method is called TD(0) or one-step TD (update the value function after any individual step).

If we take the same example,

• We initialize our value function so that it returns 0 value for each state.
• Our learning rate (lr) is 0.1, and our discount rate is 1 (no discount).
• Our mouse begins to explore the environment and takes a random action: going to the left
• It gets a reward \(R_{t+1} = 1\) since it eats a piece of cheese

We can now update \(V(S_0)\):

New \(V(S_0) = V(S_0) + lr * [R_1 + \gamma * V(S_1) - V(S_0)]\)

New \(V(S_0) = 0 + 0.1 * [1 + 1 * 0–0]\)

New \(V(S_0) = 0.1\)

So we just updated our value function for State 0.

Now we continue to interact with this environment with our updated value function.

To summarize:

• With Monte Carlo, we update the value function from a complete episode, and so we use the actual accurate discounted return of this episode.
• With TD Learning, we update the value function from a step, and we replace \(G_t\), which we don't know, with an estimated return called the TD target.

File: units/en/unit2/mid-way-quiz.mdx Copy Chapter

Mid-way Quiz [[mid-way-quiz]]

The best way to learn and to avoid the illusion of competence is to test yourself. This will help you to find where you need to reinforce your knowledge.

Q1: What are the two main approaches to find optimal policy? Copy Section

<Question
choices={[
{
text: "Policy-based methods",
explain: "With Policy-Based methods, we train the policy directly to learn which action to take given a state.",
correct: true
},
{
text: "Random-based methods",
explain: ""
},
{
text: "Value-based methods",
explain: "With value-based methods, we train a value function to learn which state is more valuable and use this value function to take the action that leads to it.",
correct: true
},
{
text: "Evolution-strategies methods",
explain: ""
}
]}
/>

Q2: What is the Bellman Equation? Copy Section

Solution

The Bellman equation is a recursive equation that works like this: instead of starting for each state from the beginning and calculating the return, we can consider the value of any state as:

Rt+1 + gamma * V(St+1)

The immediate reward + the discounted value of the state that follows

Q3: Define each part of the Bellman Equation Copy Section

Solution

Q4: What is the difference between Monte Carlo and Temporal Difference learning methods? Copy Section

<Question
choices={[
{
text: "With Monte Carlo methods, we update the value function from a complete episode",
explain: "",
correct: true
},
{
text: "With Monte Carlo methods, we update the value function from a step",
explain: ""
},
{
text: "With TD learning methods, we update the value function from a complete episode",
explain: ""
},
{
text: "With TD learning methods, we update the value function from a step",
explain: "",
correct: true
},
]}
/>

Q5: Define each part of Temporal Difference learning formula Copy Section

Solution

Q6: Define each part of Monte Carlo learning formula Copy Section

Solution

Congrats on finishing this Quiz 🥳, if you missed some elements, take time to read again the previous sections to reinforce (😏) your knowledge.

File: units/en/unit2/mid-way-recap.mdx Copy Chapter

Mid-way Recap [[mid-way-recap]]

Before diving into Q-Learning, let's summarize what we've just learned.

We have two types of value-based functions:

• State-value function: outputs the expected return if the agent starts at a given state and acts according to the policy forever after.
• Action-value function: outputs the expected return if the agent starts in a given state, takes a given action at that state and then acts accordingly to the policy forever after.
• In value-based methods, rather than learning the policy, we define the policy by hand and we learn a value function. If we have an optimal value function, we will have an optimal policy.

There are two types of methods to update the value function:

• With the Monte Carlo method, we update the value function from a complete episode, and so we use the actual discounted return of this episode.
• With the TD Learning method, we update the value function from a step, replacing the unknown \(G_t\) with an estimated return called the TD target.

File: units/en/unit2/q-learning-example.mdx Copy Chapter

A Q-Learning example [[q-learning-example]]

To better understand Q-Learning, let's take a simple example:

• You're a mouse in this tiny maze. You always start at the same starting point.
• The goal is to eat the big pile of cheese at the bottom right-hand corner and avoid the poison. After all, who doesn't like cheese?
• The episode ends if we eat the poison, eat the big pile of cheese, or if we take more than five steps.
• The learning rate is 0.1
• The discount rate (gamma) is 0.99

The reward function goes like this:

• +0: Going to a state with no cheese in it.
• +1: Going to a state with a small cheese in it.
• +10: Going to the state with the big pile of cheese.
• -10: Going to the state with the poison and thus dying.
• +0 If we take more than five steps.

To train our agent to have an optimal policy (so a policy that goes right, right, down), we will use the Q-Learning algorithm.

Step 1: Initialize the Q-table [[step1]] Copy Chapter

So, for now, our Q-table is useless; we need to train our Q-function using the Q-Learning algorithm.

Let's do it for 2 training timesteps:

Training timestep 1:

Step 2: Choose an action using the Epsilon Greedy Strategy [[step2]] Copy Chapter

Because epsilon is big (= 1.0), I take a random action. In this case, I go right.

Step 3: Perform action At, get Rt+1 and St+1 [[step3]] Copy Chapter

By going right, I get a small cheese, so \(R_{t+1} = 1\) and I'm in a new state.

Step 4: Update Q(St, At) [[step4]] Copy Chapter

We can now update \(Q(S_t, A_t)\) using our formula.

Training timestep 2:

Step 2: Choose an action using the Epsilon Greedy Strategy [[step2-2]] Copy Chapter

I take a random action again, since epsilon=0.99 is big. (Notice we decay epsilon a little bit because, as the training progress, we want less and less exploration).

I took the action 'down'. This is not a good action since it leads me to the poison.

Step 3: Perform action At, get Rt+1 and St+1 [[step3-3]] Copy Chapter

Because I ate poison, I get \(R_{t+1} = -10\), and I die.

Step 4: Update Q(St, At) [[step4-4]] Copy Chapter

Because we're dead, we start a new episode. But what we see here is that, with two explorations steps, my agent became smarter.

As we continue exploring and exploiting the environment and updating Q-values using the TD target, the Q-table will give us a better and better approximation. At the end of the training, we'll get an estimate of the optimal Q-function.

File: units/en/unit2/q-learning-recap.mdx Copy Chapter

Q-Learning Recap [[q-learning-recap]]

Q-Learning is the RL algorithm that :

• Trains a Q-function, an action-value function encoded, in internal memory, by a Q-table containing all the state-action pair values.

• Given a state and action, our Q-function will search its Q-table for the corresponding value.

• When the training is done, we have an optimal Q-function, or, equivalently, an optimal Q-table.

• And if we have an optimal Q-function, we
  have an optimal policy, since we know, for each state, the best action to take.

But, in the beginning, our Q-table is useless since it gives arbitrary values for each state-action pair (most of the time we initialize the Q-table to 0 values). But, as we explore the environment and update our Q-table it will give us a better and better approximation.

This is the Q-Learning pseudocode:

File: units/en/unit2/q-learning.mdx Copy Chapter

Introducing Q-Learning [[q-learning]]

What is Q-Learning? [[what-is-q-learning]] Copy Chapter

Q-Learning is an off-policy value-based method that uses a TD approach to train its action-value function:

• Off-policy: we'll talk about that at the end of this unit.
• Value-based method: finds the optimal policy indirectly by training a value or action-value function that will tell us the value of each state or each state-action pair.
• TD approach: updates its action-value function at each step instead of at the end of the episode.

Q-Learning is the algorithm we use to train our Q-function, an action-value function that determines the value of being at a particular state and taking a specific action at that state.

The Q comes from "the Quality" (the value) of that action at that state.

Let's recap the difference between value and reward:

• The value of a state, or a state-action pair is the expected cumulative reward our agent gets if it starts at this state (or state-action pair) and then acts accordingly to its policy.
• The reward is the feedback it gets from the environment after performing an action at a state.

Internally, our Q-function is encoded by a Q-table, a table where each cell corresponds to a state-action pair value. Think of this Q-table as the memory or cheat sheet of our Q-function.

Let's go through an example of a maze.

The Q-table is initialized. That's why all values are = 0. This table contains, for each state and action, the corresponding state-action values.
For this simple example, the state is only defined by the position of the mouse. Therefore, we have 2*3 rows in our Q-table, one row for each possible position of the mouse. In more complex scenarios, the state could contain more information than the position of the actor.

Here we see that the state-action value of the initial state and going up is 0:

So: the Q-function uses a Q-table that has the value of each state-action pair. Given a state and action, our Q-function will search inside its Q-table to output the value.

If we recap, Q-Learning is the RL algorithm that:

• Trains a Q-function (an action-value function), which internally is a Q-table that contains all the state-action pair values.
• Given a state and action, our Q-function will search its Q-table for the corresponding value.
• When the training is done, we have an optimal Q-function, which means we have optimal Q-table.
• And if we have an optimal Q-function, we have an optimal policy since we know the best action to take at each state.

In the beginning, our Q-table is useless since it gives arbitrary values for each state-action pair (most of the time, we initialize the Q-table to 0). As the agent explores the environment and we update the Q-table, it will give us a better and better approximation to the optimal policy.

Now that we understand what Q-Learning, Q-functions, and Q-tables are, let's dive deeper into the Q-Learning algorithm.

The Q-Learning algorithm [[q-learning-algo]] Copy Chapter

This is the Q-Learning pseudocode; let's study each part and see how it works with a simple example before implementing it. Don't be intimidated by it, it's simpler than it looks! We'll go over each step.

Step 1: We initialize the Q-table [[step1]] Copy Section

We need to initialize the Q-table for each state-action pair. Most of the time, we initialize with values of 0.

Step 2: Choose an action using the epsilon-greedy strategy [[step2]] Copy Section

The epsilon-greedy strategy is a policy that handles the exploration/exploitation trade-off.

The idea is that, with an initial value of ɛ = 1.0:

• With probability 1 — ɛ : we do exploitation (aka our agent selects the action with the highest state-action pair value).
• With probability ɛ: we do exploration (trying random action).

At the beginning of the training, the probability of doing exploration will be huge since ɛ is very high, so most of the time, we'll explore. But as the training goes on, and consequently our Q-table gets better and better in its estimations, we progressively reduce the epsilon value since we will need less and less exploration and more exploitation.

Step 3: Perform action At, get reward Rt+1 and next state St+1 [[step3]] Copy Section

Step 4: Update Q(St, At) [[step4]] Copy Section

Remember that in TD Learning, we update our policy or value function (depending on the RL method we choose) after one step of the interaction.

To produce our TD target, we used the immediate reward \(R_{t+1}\) plus the discounted value of the next state, computed by finding the action that maximizes the current Q-function at the next state. (We call that bootstrap).

Therefore, our \(Q(S_t, A_t)\) update formula goes like this:

This means that to update our \(Q(S_t, A_t)\):

• We need \(S_t, A_t, R_{t+1}, S_{t+1}\).
• To update our Q-value at a given state-action pair, we use the TD target.

How do we form the TD target?

1. We obtain the reward \(R_{t+1}\) after taking the action \(A_t\).
2. To get the best state-action pair value for the next state, we use a greedy policy to select the next best action. Note that this is not an epsilon-greedy policy, this will always take the action with the highest state-action value.

Then when the update of this Q-value is done, we start in a new state and select our action using a epsilon-greedy policy again.

This is why we say that Q Learning is an off-policy algorithm.

Off-policy vs On-policy [[off-vs-on]] Copy Chapter

The difference is subtle:

• Off-policy: using a different policy for acting (inference) and updating (training).

For instance, with Q-Learning, the epsilon-greedy policy (acting policy), is different from the greedy policy that is used to select the best next-state action value to update our Q-value (updating policy).

Is different from the policy we use during the training part:

• On-policy: using the same policy for acting and updating.

For instance, with Sarsa, another value-based algorithm, the epsilon-greedy policy selects the next state-action pair, not a greedy policy.

<figcaption>Sarsa</figcaption>

File: units/en/unit2/quiz2.mdx Copy Chapter

Second Quiz [[quiz2]]

The best way to learn and to avoid the illusion of competence is to test yourself. This will help you to find where you need to reinforce your knowledge.

Q1: What is Q-Learning? Copy Section

<Question
choices={[
{
text: "The algorithm we use to train our Q-function",
explain: "",
correct: true
},
{
text: "A value function",
explain: "It's an action-value function since it determines the value of being at a particular state and taking a specific action at that state",
},
{
text: "An algorithm that determines the value of being at a particular state and taking a specific action at that state",
explain: "Q-function is the function that determines the value of being at a particular state and taking a specific action at that state.",
},
{
text: "A table",
explain: "Q-learning is not a Q-table. The Q-function is the algorithm that will feed the Q-table."
}
]}
/>

Q2: What is a Q-table? Copy Section

<Question
choices={[
{
text: "An algorithm we use in Q-Learning",
explain: "",
},
{
text: "Q-table is the internal memory of our agent",
explain: "",
correct: true
},
{
text: "In Q-table each cell corresponds a state value",
explain: "Each cell corresponds to a state-action value pair value. Not a state value.",
}
]}
/>

Q3: Why if we have an optimal Q-function Q* we have an optimal policy? Copy Section

Solution

Because if we have an optimal Q-function, we have an optimal policy since we know for each state what is the best action to take.

Q4: Can you explain what is Epsilon-Greedy Strategy? Copy Section

Solution
Epsilon Greedy Strategy is a policy that handles the exploration/exploitation trade-off.

The idea is that we define epsilon ɛ = 1.0:

• With probability 1 — ɛ : we do exploitation (aka our agent selects the action with the highest state-action pair value).
• With probability ɛ : we do exploration (trying random action).

Q5: How do we update the Q value of a state, action pair? Copy Section

Solution

Q6: What's the difference between on-policy and off-policy Copy Section

Solution

Congrats on finishing this Quiz 🥳, if you missed some elements, take time to read again the chapter to reinforce (😏) your knowledge.

File: units/en/unit2/two-types-value-based-methods.mdx Copy Chapter

Two types of value-based methods [[two-types-value-based-methods]]

In value-based methods, we learn a value function that maps a state to the expected value of being at that state.

The value of a state is the expected discounted return the agent can get if it starts at that state and then acts according to our policy.

Remember that the goal of an RL agent is to have an optimal policy π*.

To find the optimal policy, we learned about two different methods:

• Policy-based methods: Directly train the policy to select what action to take given a state (or a probability distribution over actions at that state). In this case, we don't have a value function.

The policy takes a state as input and outputs what action to take at that state (deterministic policy: a policy that output one action given a state, contrary to stochastic policy that output a probability distribution over actions).

And consequently, we don't define by hand the behavior of our policy; it's the training that will define it.

• Value-based methods: Indirectly, by training a value function that outputs the value of a state or a state-action pair. Given this value function, our policy will take an action.

Since the policy is not trained/learned, we need to specify its behavior. For instance, if we want a policy that, given the value function, will take actions that always lead to the biggest reward, we'll create a Greedy Policy.

Consequently, whatever method you use to solve your problem, you will have a policy. In the case of value-based methods, you don't train the policy: your policy is just a simple pre-specified function (for instance, the Greedy Policy) that uses the values given by the value-function to select its actions.

So the difference is:

• In policy-based training, the optimal policy (denoted π*) is found by training the policy directly.
• In value-based training, finding an optimal value function (denoted Q* or V*, we'll study the difference below) leads to having an optimal policy.

In fact, most of the time, in value-based methods, you'll use an Epsilon-Greedy Policy that handles the exploration/exploitation trade-off; we'll talk about this when we talk about Q-Learning in the second part of this unit.

As we mentioned above, we have two types of value-based functions:

The state-value function [[state-value-function]] Copy Chapter

We write the state value function under a policy π like this:

For each state, the state-value function outputs the expected return if the agent starts at that state and then follows the policy forever afterward (for all future timesteps, if you prefer).

The action-value function [[action-value-function]] Copy Chapter

In the action-value function, for each state and action pair, the action-value function outputs the expected return if the agent starts in that state, takes that action, and then follows the policy forever after.

The value of taking action \(a\) in state \(s\) under a policy \(π\) is:

We see that the difference is:

• For the state-value function, we calculate the value of a state \(S_t\)
• For the action-value function, we calculate the value of the state-action pair ( \(S_t, A_t\) ) hence the value of taking that action at that state.

In either case, whichever value function we choose (state-value or action-value function), the returned value is the expected return.

However, the problem is that to calculate EACH value of a state or a state-action pair, we need to sum all the rewards an agent can get if it starts at that state.

This can be a computationally expensive process, and that's where the Bellman equation comes in to help us.

File: units/en/unit2/what-is-rl.mdx Copy Chapter

What is RL? A short recap [[what-is-rl]]

In RL, we build an agent that can make smart decisions. For instance, an agent that learns to play a video game. Or a trading agent that learns to maximize its benefits by deciding on what stocks to buy and when to sell.

To make intelligent decisions, our agent will learn from the environment by interacting with it through trial and error and receiving rewards (positive or negative) as unique feedback.

Its goal is to maximize its expected cumulative reward (because of the reward hypothesis).

The agent's decision-making process is called the policy π: given a state, a policy will output an action or a probability distribution over actions. That is, given an observation of the environment, a policy will provide an action (or multiple probabilities for each action) that the agent should take.

*Our goal is to find an optimal policy π **, aka., a policy that leads to the best expected cumulative reward.

And to find this optimal policy (hence solving the RL problem), there are two main types of RL methods:

• Policy-based methods: Train the policy directly to learn which action to take given a state.
• Value-based methods: Train a value function to learn which state is more valuable and use this value function to take the action that leads to it.

And in this unit, we'll dive deeper into the value-based methods.

File: units/en/unit3/additional-readings.mdx Copy Chapter

Additional Readings [[additional-readings]]

These are optional readings if you want to go deeper.

• Foundations of Deep RL Series, L2 Deep Q-Learning by Pieter Abbeel
• Playing Atari with Deep Reinforcement Learning
• Double Deep Q-Learning
• Prioritized Experience Replay
• Dueling Deep Q-Learning

File: units/en/unit3/conclusion.mdx Copy Chapter

Conclusion [[conclusion]]

Congrats on finishing this chapter! There was a lot of information. And congrats on finishing the tutorial. You’ve just trained your first Deep Q-Learning agent and shared it on the Hub 🥳.

Take time to really grasp the material before continuing.

Don't hesitate to train your agent in other environments (Pong, Seaquest, QBert, Ms Pac Man). The best way to learn is to try things on your own!

In the next unit, we're going to learn about Optuna. One of the most critical tasks in Deep Reinforcement Learning is to find a good set of training hyperparameters. Optuna is a library that helps you to automate the search.

Finally, we would love to hear what you think of the course and how we can improve it. If you have some feedback then please 👉 fill this form

Keep Learning, stay awesome 🤗 Copy Section

File: units/en/unit3/deep-q-algorithm.mdx Copy Chapter

The Deep Q-Learning Algorithm [[deep-q-algorithm]]

We learned that Deep Q-Learning uses a deep neural network to approximate the different Q-values for each possible action at a state (value-function estimation).

The difference is that, during the training phase, instead of updating the Q-value of a state-action pair directly as we have done with Q-Learning:

in Deep Q-Learning, we create a loss function that compares our Q-value prediction and the Q-target and uses gradient descent to update the weights of our Deep Q-Network to approximate our Q-values better.

The Deep Q-Learning training algorithm has two phases:

• Sampling: we perform actions and store the observed experience tuples in a replay memory.
• Training: Select a small batch of tuples randomly and learn from this batch using a gradient descent update step.

This is not the only difference compared with Q-Learning. Deep Q-Learning training might suffer from instability, mainly because of combining a non-linear Q-value function (Neural Network) and bootstrapping (when we update targets with existing estimates and not an actual complete return).

To help us stabilize the training, we implement three different solutions:

1. Experience Replay to make more efficient use of experiences.
2. Fixed Q-Target to stabilize the training.
3. Double Deep Q-Learning, to handle the problem of the overestimation of Q-values.

Let's go through them!

Experience Replay to make more efficient use of experiences [[exp-replay]] Copy Chapter

Why do we create a replay memory?

Experience Replay in Deep Q-Learning has two functions:

1. Make more efficient use of the experiences during the training.
   Usually, in online reinforcement learning, the agent interacts with the environment, gets experiences (state, action, reward, and next state), learns from them (updates the neural network), and discards them. This is not efficient.

Experience replay helps by using the experiences of the training more efficiently. We use a replay buffer that saves experience samples that we can reuse during the training.

⇒ This allows the agent to learn from the same experiences multiple times.

2. Avoid forgetting previous experiences (aka catastrophic interference, or catastrophic forgetting) and reduce the correlation between experiences.

• catastrophic forgetting: The problem we get if we give sequential samples of experiences to our neural network is that it tends to forget the previous experiences as it gets new experiences. For instance, if the agent is in the first level and then in the second, which is different, it can forget how to behave and play in the first level.

The solution is to create a Replay Buffer that stores experience tuples while interacting with the environment and then sample a small batch of tuples. This prevents the network from only learning about what it has done immediately before.

Experience replay also has other benefits. By randomly sampling the experiences, we remove correlation in the observation sequences and avoid action values from oscillating or diverging catastrophically.

In the Deep Q-Learning pseudocode, we initialize a replay memory buffer D with capacity N (N is a hyperparameter that you can define). We then store experiences in the memory and sample a batch of experiences to feed the Deep Q-Network during the training phase.

Fixed Q-Target to stabilize the training [[fixed-q]] Copy Chapter

When we want to calculate the TD error (aka the loss), we calculate the difference between the TD target (Q-Target) and the current Q-value (estimation of Q).

But we don’t have any idea of the real TD target. We need to estimate it. Using the Bellman equation, we saw that the TD target is just the reward of taking that action at that state plus the discounted highest Q value for the next state.

However, the problem is that we are using the same parameters (weights) for estimating the TD target and the Q-value. Consequently, there is a significant correlation between the TD target and the parameters we are changing.

Therefore, at every step of training, both our Q-values and the target values shift. We’re getting closer to our target, but the target is also moving. It’s like chasing a moving target! This can lead to significant oscillation in training.

It’s like if you were a cowboy (the Q estimation) and you wanted to catch a cow (the Q-target). Your goal is to get closer (reduce the error).

At each time step, you’re trying to approach the cow, which also moves at each time step (because you use the same parameters).

This leads to a bizarre path of chasing (a significant oscillation in training).

Instead, what we see in the pseudo-code is that we:

• Use a separate network with fixed parameters for estimating the TD Target
• Copy the parameters from our Deep Q-Network every C steps to update the target network.

Double DQN [[double-dqn]] Copy Chapter

Double DQNs, or Double Deep Q-Learning neural networks, were introduced by Hado van Hasselt. This method handles the problem of the overestimation of Q-values.

To understand this problem, remember how we calculate the TD Target:

We face a simple problem by calculating the TD target: how are we sure that the best action for the next state is the action with the highest Q-value?

We know that the accuracy of Q-values depends on what action we tried and what neighboring states we explored.

Consequently, we don’t have enough information about the best action to take at the beginning of the training. Therefore, taking the maximum Q-value (which is noisy) as the best action to take can lead to false positives. If non-optimal actions are regularly given a higher Q value than the optimal best action, the learning will be complicated.

The solution is: when we compute the Q target, we use two networks to decouple the action selection from the target Q-value generation. We:

• Use our DQN network to select the best action to take for the next state (the action with the highest Q-value).
• Use our Target network to calculate the target Q-value of taking that action at the next state.

Therefore, Double DQN helps us reduce the overestimation of Q-values and, as a consequence, helps us train faster and with more stable learning.

Since these three improvements in Deep Q-Learning, many more have been added, such as Prioritized Experience Replay and Dueling Deep Q-Learning. They’re out of the scope of this course but if you’re interested, check the links we put in the reading list.

File: units/en/unit3/deep-q-network.mdx Copy Chapter

The Deep Q-Network (DQN) [[deep-q-network]]

This is the architecture of our Deep Q-Learning network:

As input, we take a stack of 4 frames passed through the network as a state and output a vector of Q-values for each possible action at that state. Then, like with Q-Learning, we just need to use our epsilon-greedy policy to select which action to take.

When the Neural Network is initialized, the Q-value estimation is terrible. But during training, our Deep Q-Network agent will associate a situation with the appropriate action and learn to play the game well.

Preprocessing the input and temporal limitation [[preprocessing]] Copy Chapter

We need to preprocess the input. It’s an essential step since we want to reduce the complexity of our state to reduce the computation time needed for training.

To achieve this, we reduce the state space to 84x84 and grayscale it. We can do this since the colors in Atari environments don't add important information.
This is a big improvement since we reduce our three color channels (RGB) to 1.

We can also crop a part of the screen in some games if it does not contain important information.
Then we stack four frames together.

Why do we stack four frames together?
We stack frames together because it helps us handle the problem of temporal limitation. Let’s take an example with the game of Pong. When you see this frame:

Can you tell me where the ball is going?
No, because one frame is not enough to have a sense of motion! But what if I add three more frames? Here you can see that the ball is going to the right.

That’s why, to capture temporal information, we stack four frames together.

Then the stacked frames are processed by three convolutional layers. These layers allow us to capture and exploit spatial relationships in images. But also, because the frames are stacked together, we can exploit some temporal properties across those frames.

If you don't know what convolutional layers are, don't worry. You can check out Lesson 4 of this free Deep Learning Course by Udacity

Finally, we have a couple of fully connected layers that output a Q-value for each possible action at that state.

So, we see that Deep Q-Learning uses a neural network to approximate, given a state, the different Q-values for each possible action at that state. Now let's study the Deep Q-Learning algorithm.

File: units/en/unit3/from-q-to-dqn.mdx Copy Chapter

From Q-Learning to Deep Q-Learning [[from-q-to-dqn]]

We learned that Q-Learning is an algorithm we use to train our Q-Function, an action-value function that determines the value of being at a particular state and taking a specific action at that state.

The Q comes from "the Quality" of that action at that state.

Internally, our Q-function is encoded by a Q-table, a table where each cell corresponds to a state-action pair value. Think of this Q-table as the memory or cheat sheet of our Q-function.

The problem is that Q-Learning is a tabular method. This becomes a problem if the states and actions spaces are not small enough to be represented efficiently by arrays and tables. In other words: it is not scalable.
Q-Learning worked well with small state space environments like:

• FrozenLake, we had 16 states.
• Taxi-v3, we had 500 states.

But think of what we're going to do today: we will train an agent to learn to play Space Invaders, a more complex game, using the frames as input.

As Nikita Melkozerov mentioned, Atari environments have an observation space with a shape of (210, 160, 3)*, containing values ranging from 0 to 255 so that gives us \(256^{210 \times 160 \times 3} = 256^{100800}\) possible observations (for comparison, we have approximately \(10^{80}\) atoms in the observable universe).

• A single frame in Atari is composed of an image of 210x160 pixels. Given that the images are in color (RGB), there are 3 channels. This is why the shape is (210, 160, 3). For each pixel, the value can go from 0 to 255.

Therefore, the state space is gigantic; due to this, creating and updating a Q-table for that environment would not be efficient. In this case, the best idea is to approximate the Q-values using a parametrized Q-function \(Q_{\theta}(s,a)\) .

This neural network will approximate, given a state, the different Q-values for each possible action at that state. And that's exactly what Deep Q-Learning does.

Now that we understand Deep Q-Learning, let's dive deeper into the Deep Q-Network.

File: units/en/unit3/glossary.mdx Copy Chapter

Glossary

This is a community-created glossary. Contributions are welcomed!

• Tabular Method: Type of problem in which the state and action spaces are small enough to approximate value functions to be represented as arrays and tables.
  Q-learning is an example of tabular method since a table is used to represent the value for different state-action pairs.

• Deep Q-Learning: Method that trains a neural network to approximate, given a state, the different Q-values for each possible action at that state.
  It is used to solve problems when observational space is too big to apply a tabular Q-Learning approach.

• Temporal Limitation is a difficulty presented when the environment state is represented by frames. A frame by itself does not provide temporal information.
  In order to obtain temporal information, we need to stack a number of frames together.

• Phases of Deep Q-Learning:

  • Sampling: Actions are performed, and observed experience tuples are stored in a replay memory.
  • Training: Batches of tuples are selected randomly and the neural network updates its weights using gradient descent.

• Solutions to stabilize Deep Q-Learning:

  • Experience Replay: A replay memory is created to save experiences samples that can be reused during training.
    This allows the agent to learn from the same experiences multiple times. Also, it helps the agent avoid forgetting previous experiences as it gets new ones.

  • Random sampling from replay buffer allows to remove correlation in the observation sequences and prevents action values from oscillating or diverging
    catastrophically.

  • Fixed Q-Target: In order to calculate the Q-Target we need to estimate the discounted optimal Q-value of the next state by using Bellman equation. The problem
    is that the same network weights are used to calculate the Q-Target and the Q-value. This means that everytime we are modifying the Q-value, the Q-Target also moves with it.
    To avoid this issue, a separate network with fixed parameters is used for estimating the Temporal Difference Target. The target network is updated by copying parameters from
    our Deep Q-Network after certain C steps.

  • Double DQN: Method to handle overestimation of Q-Values. This solution uses two networks to decouple the action selection from the target Value generation:

    • DQN Network to select the best action to take for the next state (the action with the highest Q-Value)
    • Target Network to calculate the target Q-Value of taking that action at the next state.
      This approach reduces the Q-Values overestimation, it helps to train faster and have more stable learning.

If you want to improve the course, you can open a Pull Request.

This glossary was made possible thanks to:

• Dario Paez

File: units/en/unit3/hands-on.mdx Copy Chapter

Hands-on [[hands-on]]

<CourseFloatingBanner classNames="absolute z-10 right-0 top-0"
  notebooks={[
    {label: "Google Colab", value: "https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/notebooks/unit3/unit3.ipynb"}
    ]}
    askForHelpUrl="http://hf.co/join/discord" />

Now that you've studied the theory behind Deep Q-Learning, you’re ready to train your Deep Q-Learning agent to play Atari Games. We'll start with Space Invaders, but you'll be able to use any Atari game you want 🔥

We're using the RL-Baselines-3 Zoo integration, a vanilla version of Deep Q-Learning with no extensions such as Double-DQN, Dueling-DQN, or Prioritized Experience Replay.

Also, if you want to learn to implement Deep Q-Learning by yourself after this hands-on, you definitely should look at the CleanRL implementation: https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py

To validate this hands-on for the certification process, you need to push your trained model to the Hub and get a result of >= 200.

To find your result, go to the leaderboard and find your model, the result = mean_reward - std of reward

If you don't find your model, go to the bottom of the page and click on the refresh button.

For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process

And you can check your progress here 👉 https://huggingface.co/spaces/ThomasSimonini/Check-my-progress-Deep-RL-Course

To start the hands-on click on Open In Colab button 👇 :

Unit 3: Deep Q-Learning with Atari Games 👾 using RL Baselines3 Zoo

In this hands-on, you'll train a Deep Q-Learning agent playing Space Invaders using RL Baselines3 Zoo, a training framework based on Stable-Baselines3 that provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.

We're using the RL-Baselines-3 Zoo integration, a vanilla version of Deep Q-Learning with no extensions such as Double-DQN, Dueling-DQN, and Prioritized Experience Replay.

🎮 Environments: Copy Section

• SpacesInvadersNoFrameskip-v4

You can see the difference between Space Invaders versions here 👉 https://gymnasium.farama.org/environments/atari/space_invaders/#variants

📚 RL-Library: Copy Section

• RL-Baselines3-Zoo

Objectives of this hands-on 🏆 Copy Chapter

At the end of the hands-on, you will:

• Be able to understand deeper how RL Baselines3 Zoo works.
• Be able to push your trained agent and the code to the Hub with a nice video replay and an evaluation score 🔥.

Prerequisites 🏗️ Copy Chapter

Before diving into the hands-on, you need to:

🔲 📚 Study Deep Q-Learning by reading Unit 3 🤗

We're constantly trying to improve our tutorials, so if you find some issues in this hands-on, please open an issue on the Github Repo.

Let's train a Deep Q-Learning agent playing Atari' Space Invaders 👾 and upload it to the Hub.

We strongly recommend students to use Google Colab for the hands-on exercises instead of running them on their personal computers.

By using Google Colab, you can focus on learning and experimenting without worrying about the technical aspects of setting up your environments.

To validate this hands-on for the certification process, you need to push your trained model to the Hub and get a result of >= 200.

To find your result, go to the leaderboard and find your model, the result = mean_reward - std of reward

For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process

Set the GPU 💪 Copy Chapter

• To accelerate the agent's training, we'll use a GPU. To do that, go to Runtime > Change Runtime type

• Hardware Accelerator > GPU

Install RL-Baselines3 Zoo and its dependencies 📚

If you see ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. this is normal and it's not a critical error there's a conflict of version. But the packages we need are installed.

# For now we install this update of RL-Baselines3 Zoo
pip install git+https://github.com/DLR-RM/rl-baselines3-zoo

apt-get install swig cmake ffmpeg

To be able to use Atari games in Gymnasium we need to install atari package. And accept-rom-license to download the rom files (games files).

!pip install gymnasium[atari]
!pip install gymnasium[accept-rom-license]

Create a virtual display 🔽 Copy Chapter

During the hands-on, we'll need to generate a replay video. To do so, if you train it on a headless machine, we need to have a virtual screen to be able to render the environment (and thus record the frames).

Hence the following cell will install the librairies and create and run a virtual screen 🖥

apt install python-opengl
apt install ffmpeg
apt install xvfb
pip3 install pyvirtualdisplay

# Virtual display
from pyvirtualdisplay import Display

virtual_display = Display(visible=0, size=(1400, 900))
virtual_display.start()

Train our Deep Q-Learning Agent to Play Space Invaders 👾 Copy Chapter

To train an agent with RL-Baselines3-Zoo, we just need to do two things:

1. Create a hyperparameter config file that will contain our training hyperparameters called dqn.yml.

This is a template example:

SpaceInvadersNoFrameskip-v4:
  env_wrapper:
    - stable_baselines3.common.atari_wrappers.AtariWrapper
  frame_stack: 4
  policy: 'CnnPolicy'
  n_timesteps: !!float 1e7
  buffer_size: 100000
  learning_rate: !!float 1e-4
  batch_size: 32
  learning_starts: 100000
  target_update_interval: 1000
  train_freq: 4
  gradient_steps: 1
  exploration_fraction: 0.1
  exploration_final_eps: 0.01
  # If True, you need to deactivate handle_timeout_termination
  # in the replay_buffer_kwargs
  optimize_memory_usage: False

Here we see that:

• We use the Atari Wrapper that preprocess the input (Frame reduction ,grayscale, stack 4 frames)
• We use CnnPolicy, since we use Convolutional layers to process the frames
• We train it for 10 million n_timesteps
• Memory (Experience Replay) size is 100000, aka the amount of experience steps you saved to train again your agent with.

💡 My advice is to reduce the training timesteps to 1M, which will take about 90 minutes on a P100. !nvidia-smi will tell you what GPU you're using. At 10 million steps, this will take about 9 hours. I recommend running this on your local computer (or somewhere else). Just click on: File>Download.

In terms of hyperparameters optimization, my advice is to focus on these 3 hyperparameters:

• learning_rate
• buffer_size (Experience Memory size)
• batch_size

As a good practice, you need to check the documentation to understand what each hyperparameters does: https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html#parameters

2. We start the training and save the models on logs folder 📁

• Define the algorithm after --algo, where we save the model after -f and where the hyperparameter config is after -c.

python -m rl_zoo3.train --algo ________ --env SpaceInvadersNoFrameskip-v4  -f _________  -c _________

Solution

python -m rl_zoo3.train --algo dqn  --env SpaceInvadersNoFrameskip-v4 -f logs/ -c dqn.yml

Let's evaluate our agent 👀 Copy Chapter

• RL-Baselines3-Zoo provides enjoy.py, a python script to evaluate our agent. In most RL libraries, we call the evaluation script enjoy.py.
• Let's evaluate it for 5000 timesteps 🔥

python -m rl_zoo3.enjoy  --algo dqn  --env SpaceInvadersNoFrameskip-v4  --no-render  --n-timesteps _________  --folder logs/

Solution

python -m rl_zoo3.enjoy  --algo dqn  --env SpaceInvadersNoFrameskip-v4  --no-render  --n-timesteps 5000  --folder logs/

Publish our trained model on the Hub 🚀 Copy Chapter

Now that we saw we got good results after the training, we can publish our trained model on the hub 🤗 with one line of code.

By using rl_zoo3.push_to_hub you evaluate, record a replay, generate a model card of your agent and push it to the hub.

This way:

• You can showcase our work 🔥
• You can visualize your agent playing 👀
• You can share with the community an agent that others can use 💾
• You can access a leaderboard 🏆 to see how well your agent is performing compared to your classmates 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard

To be able to share your model with the community there are three more steps to follow:

1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join

2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.

• Create a new token (https://huggingface.co/settings/tokens) with write role

• Copy the token

• Run the cell below and past the token

from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.
notebook_login()
!git config --global credential.helper store

If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: huggingface-cli login

3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥

Let's run push_to_hub.py file to upload our trained agent to the Hub.

--repo-name : The name of the repo

-orga: Your Hugging Face username

-f: Where the trained model folder is (in our case logs)

python -m rl_zoo3.push_to_hub  --algo dqn  --env SpaceInvadersNoFrameskip-v4  --repo-name _____________________ -orga _____________________ -f logs/

Solution

python -m rl_zoo3.push_to_hub  --algo dqn  --env SpaceInvadersNoFrameskip-v4  --repo-name dqn-SpaceInvadersNoFrameskip-v4  -orga ThomasSimonini  -f logs/

###.

Congrats 🥳 you've just trained and uploaded your first Deep Q-Learning agent using RL-Baselines-3 Zoo. The script above should have displayed a link to a model repository such as https://huggingface.co/ThomasSimonini/dqn-SpaceInvadersNoFrameskip-v4. When you go to this link, you can:

• See a video preview of your agent at the right.
• Click "Files and versions" to see all the files in the repository.
• Click "Use in stable-baselines3" to get a code snippet that shows how to load the model.
• A model card (README.md file) which gives a description of the model and the hyperparameters you used.

Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent.

Compare the results of your agents with your classmates using the leaderboard 🏆

Load a powerful trained model 🔥 Copy Chapter

• The Stable-Baselines3 team uploaded more than 150 trained Deep Reinforcement Learning agents on the Hub.

You can find them here: 👉 https://huggingface.co/sb3

Some examples:

• Asteroids: https://huggingface.co/sb3/dqn-AsteroidsNoFrameskip-v4
• Beam Rider: https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4
• Breakout: https://huggingface.co/sb3/dqn-BreakoutNoFrameskip-v4
• Road Runner: https://huggingface.co/sb3/dqn-RoadRunnerNoFrameskip-v4

Let's load an agent playing Beam Rider: https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4

1. We download the model using rl_zoo3.load_from_hub, and place it in a new folder that we can call rl_trained

# Download model and save it into the logs/ folder
python -m rl_zoo3.load_from_hub --algo dqn --env BeamRiderNoFrameskip-v4 -orga sb3 -f rl_trained/

2. Let's evaluate if for 5000 timesteps

python -m rl_zoo3.enjoy --algo dqn --env BeamRiderNoFrameskip-v4 -n 5000  -f rl_trained/ --no-render

Why not trying to train your own Deep Q-Learning Agent playing BeamRiderNoFrameskip-v4? 🏆.

If you want to try, check https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4#hyperparameters in the model card, you have the hyperparameters of the trained agent.

But finding hyperparameters can be a daunting task. Fortunately, we'll see in the next Unit, how we can use Optuna for optimizing the Hyperparameters 🔥.

Some additional challenges 🏆 Copy Chapter

The best way to learn is to try things by your own!

In the Leaderboard you will find your agents. Can you get to the top?

Here's a list of environments you can try to train your agent with:

• BeamRiderNoFrameskip-v4
• BreakoutNoFrameskip-v4
• EnduroNoFrameskip-v4
• PongNoFrameskip-v4

Also, if you want to learn to implement Deep Q-Learning by yourself, you definitely should look at CleanRL implementation: https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py

Congrats on finishing this chapter!

If you’re still feel confused with all these elements...it's totally normal! This was the same for me and for all people who studied RL.

Take time to really grasp the material before continuing and try the additional challenges. It’s important to master these elements and having a solid foundations.

In the next unit, we’re going to learn about Optuna. One of the most critical task in Deep Reinforcement Learning is to find a good set of training hyperparameters. And Optuna is a library that helps you to automate the search.

This is a course built with you 👷🏿‍♀️ Copy Section

Finally, we want to improve and update the course iteratively with your feedback. If you have some, please fill this form 👉 https://forms.gle/3HgA7bEHwAmmLfwh9

We're constantly trying to improve our tutorials, so if you find some issues in this notebook, please open an issue on the Github Repo.

See you on Bonus unit 2! 🔥

Keep Learning, Stay Awesome 🤗 Copy Section

File: units/en/unit3/introduction.mdx Copy Chapter

Deep Q-Learning [[deep-q-learning]]

In the last unit, we learned our first reinforcement learning algorithm: Q-Learning, implemented it from scratch, and trained it in two environments, FrozenLake-v1 ☃️ and Taxi-v3 🚕.

We got excellent results with this simple algorithm, but these environments were relatively simple because the state space was discrete and small (16 different states for FrozenLake-v1 and 500 for Taxi-v3). For comparison, the state space in Atari games can contain \(10^{9}\) to \(10^{11}\) states.

But as we'll see, producing and updating a Q-table can become ineffective in large state space environments.

So in this unit, we'll study our first Deep Reinforcement Learning agent: Deep Q-Learning. Instead of using a Q-table, Deep Q-Learning uses a Neural Network that takes a state and approximates Q-values for each action based on that state.

And we'll train it to play Space Invaders and other Atari environments using RL-Zoo, a training framework for RL using Stable-Baselines that provides scripts for training, evaluating agents, tuning hyperparameters, plotting results, and recording videos.

So let’s get started! 🚀

File: units/en/unit3/quiz.mdx Copy Chapter

Quiz [[quiz]]

The best way to learn and to avoid the illusion of competence is to test yourself. This will help you to find where you need to reinforce your knowledge.

Q1: We mentioned Q Learning is a tabular method. What are tabular methods? Copy Section

Solution

Tabular methods is a type of problem in which the state and actions spaces are small enough to approximate value functions to be represented as arrays and tables. For instance, Q-Learning is a tabular method since we use a table to represent the state, and action value pairs.

Q2: Why can't we use a classical Q-Learning to solve an Atari Game? Copy Section

<Question
choices={[
{
text: "Atari environments are too fast for Q-Learning",
explain: ""
},
{
text: "Atari environments have a big observation space. So creating an updating the Q-Table would not be efficient",
explain: "",
correct: true
}
]}
/>

Q3: Why do we stack four frames together when we use frames as input in Deep Q-Learning? Copy Section

Solution

We stack frames together because it helps us handle the problem of temporal limitation: one frame is not enough to capture temporal information.
For instance, in pong, our agent will be unable to know the ball direction if it gets only one frame.

Q4: What are the two phases of Deep Q-Learning? Copy Section

<Question
choices={[
{
text: "Sampling",
explain: "We perform actions and store the observed experiences tuples in a replay memory.",
correct: true,
},
{
text: "Shuffling",
explain: "",
},
{
text: "Reranking",
explain: "",
},
{
text: "Training",
explain: "We select the small batch of tuple randomly and learn from it using a gradient descent update step.",
correct: true,
}
]}
/>

Q5: Why do we create a replay memory in Deep Q-Learning? Copy Section

Solution

1. Make more efficient use of the experiences during the training

Usually, in online reinforcement learning, the agent interacts in the environment, gets experiences (state, action, reward, and next state), learns from them (updates the neural network), and discards them. This is not efficient.
But, with experience replay, we create a replay buffer that saves experience samples that we can reuse during the training.

2. Avoid forgetting previous experiences and reduce the correlation between experiences

The problem we get if we give sequential samples of experiences to our neural network is that it tends to forget the previous experiences as it overwrites new experiences. For instance, if we are in the first level and then the second, which is different, our agent can forget how to behave and play in the first level.

Q6: How do we use Double Deep Q-Learning? Copy Section

Solution

When we compute the Q target, we use two networks to decouple the action selection from the target Q value generation. We:

• Use our DQN network to select the best action to take for the next state (the action with the highest Q value).

• Use our Target network to calculate the target Q value of taking that action at the next state.

Congrats on finishing this Quiz 🥳, if you missed some elements, take time to read again the chapter to reinforce (😏) your knowledge.

File: units/en/unit4/additional-readings.mdx Copy Chapter

Additional Readings

These are optional readings if you want to go deeper.

Introduction to Policy Optimization Copy Chapter

• Part 3: Intro to Policy Optimization - Spinning Up documentation

Policy Gradient Copy Chapter

• https://johnwlambert.github.io/policy-gradients/
• RL - Policy Gradient Explained
• Chapter 13, Policy Gradient Methods; Reinforcement Learning, an introduction by Richard Sutton and Andrew G. Barto

Implementation Copy Chapter

• PyTorch Reinforce implementation
• Implementations from DDPG to PPO

File: units/en/unit4/advantages-disadvantages.mdx Copy Chapter

The advantages and disadvantages of policy-gradient methods

At this point, you might ask, "but Deep Q-Learning is excellent! Why use policy-gradient methods?". To answer this question, let's study the advantages and disadvantages of policy-gradient methods.

Advantages Copy Chapter

There are multiple advantages over value-based methods. Let's see some of them:

The simplicity of integration Copy Section

We can estimate the policy directly without storing additional data (action values).

Policy-gradient methods can learn a stochastic policy Copy Section

Policy-gradient methods can learn a stochastic policy while value functions can't.

This has two consequences:

1. We don't need to implement an exploration/exploitation trade-off by hand. Since we output a probability distribution over actions, the agent explores the state space without always taking the same trajectory.

2. We also get rid of the problem of perceptual aliasing. Perceptual aliasing is when two states seem (or are) the same but need different actions.

Let's take an example: we have an intelligent vacuum cleaner whose goal is to suck the dust and avoid killing the hamsters.

Our vacuum cleaner can only perceive where the walls are.

The problem is that the two red (colored) states are aliased states because the agent perceives an upper and lower wall for each.

Under a deterministic policy, the policy will either always move right when in a red state or always move left. Either case will cause our agent to get stuck and never suck the dust.

Under a value-based Reinforcement learning algorithm, we learn a quasi-deterministic policy ("greedy epsilon strategy"). Consequently, our agent can spend a lot of time before finding the dust.

On the other hand, an optimal stochastic policy will randomly move left or right in red (colored) states. Consequently, it will not be stuck and will reach the goal state with a high probability.

Policy-gradient methods are more effective in high-dimensional action spaces and continuous actions spaces Copy Section

The problem with Deep Q-learning is that their predictions assign a score (maximum expected future reward) for each possible action, at each time step, given the current state.

But what if we have an infinite possibility of actions?

For instance, with a self-driving car, at each state, you can have a (near) infinite choice of actions (turning the wheel at 15°, 17.2°, 19,4°, honking, etc.). We'll need to output a Q-value for each possible action! And taking the max action of a continuous output is an optimization problem itself!

Instead, with policy-gradient methods, we output a probability distribution over actions.

Policy-gradient methods have better convergence properties Copy Section

In value-based methods, we use an aggressive operator to change the value function: we take the maximum over Q-estimates.
Consequently, the action probabilities may change dramatically for an arbitrarily small change in the estimated action values if that change results in a different action having the maximal value.

For instance, if during the training, the best action was left (with a Q-value of 0.22) and the training step after it's right (since the right Q-value becomes 0.23), we dramatically changed the policy since now the policy will take most of the time right instead of left.

On the other hand, in policy-gradient methods, stochastic policy action preferences (probability of taking action) change smoothly over time.

Disadvantages Copy Chapter

Naturally, policy-gradient methods also have some disadvantages:

• Frequently, policy-gradient methods converges to a local maximum instead of a global optimum.
• Policy-gradient goes slower, step by step: it can take longer to train (inefficient).
• Policy-gradient can have high variance. We'll see in the actor-critic unit why, and how we can solve this problem.

👉 If you want to go deeper into the advantages and disadvantages of policy-gradient methods, you can check this video.

File: units/en/unit4/conclusion.mdx Copy Chapter

Conclusion

Congrats on finishing this unit! There was a lot of information.
And congrats on finishing the tutorial. You've just coded your first Deep Reinforcement Learning agent from scratch using PyTorch and shared it on the Hub 🥳.

Don't hesitate to iterate on this unit by improving the implementation for more complex environments (for instance, what about changing the network to a Convolutional Neural Network to handle
frames as observation)?

In the next unit, we're going to learn more about Unity MLAgents, by training agents in Unity environments. This way, you will be ready to participate in the AI vs AI challenges where you'll train your agents
to compete against other agents in a snowball fight and a soccer game.

Sound fun? See you next time!

Finally, we would love to hear what you think of the course and how we can improve it. If you have some feedback then please 👉 fill this form

Keep Learning, stay awesome 🤗 Copy Section

File: units/en/unit4/glossary.mdx Copy Chapter

Glossary

This is a community-created glossary. Contributions are welcome!

• Deep Q-Learning: A value-based deep reinforcement learning algorithm that uses a deep neural network to approximate Q-values for actions in a given state. The goal of Deep Q-learning is to find the optimal policy that maximizes the expected cumulative reward by learning the action-values.

• Value-based methods: Reinforcement Learning methods that estimate a value function as an intermediate step towards finding an optimal policy.

• Policy-based methods: Reinforcement Learning methods that directly learn to approximate the optimal policy without learning a value function. In practice they output a probability distribution over actions.

  The benefits of using policy-gradient methods over value-based methods include:

  • simplicity of integration: no need to store action values;
  • ability to learn a stochastic policy: the agent explores the state space without always taking the same trajectory, and avoids the problem of perceptual aliasing;
  • effectiveness in high-dimensional and continuous action spaces; and
  • improved convergence properties.

• Policy Gradient: A subset of policy-based methods where the objective is to maximize the performance of a parameterized policy using gradient ascent. The goal of a policy-gradient is to control the probability distribution of actions by tuning the policy such that good actions (that maximize the return) are sampled more frequently in the future.

• Monte Carlo Reinforce: A policy-gradient algorithm that uses an estimated return from an entire episode to update the policy parameter.

If you want to improve the course, you can open a Pull Request.

This glossary was made possible thanks to:

• Diego Carpintero

File: units/en/unit4/hands-on.mdx Copy Chapter

Hands on

<CourseFloatingBanner classNames="absolute z-10 right-0 top-0"
  notebooks={[
    {label: "Google Colab", value: "https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/notebooks/unit4/unit4.ipynb"}
    ]}
    askForHelpUrl="http://hf.co/join/discord" />

Now that we've studied the theory behind Reinforce, you’re ready to code your Reinforce agent with PyTorch. And you'll test its robustness using CartPole-v1 and PixelCopter,.

You'll then be able to iterate and improve this implementation for more advanced environments.

To validate this hands-on for the certification process, you need to push your trained models to the Hub and:

• Get a result of >= 350 for Cartpole-v1
• Get a result of >= 5 for PixelCopter.

To find your result, go to the leaderboard and find your model, the result = mean_reward - std of reward. If you don't see your model on the leaderboard, go at the bottom of the leaderboard page and click on the refresh button.

If you don't find your model, go to the bottom of the page and click on the refresh button.

For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process

And you can check your progress here 👉 https://huggingface.co/spaces/ThomasSimonini/Check-my-progress-Deep-RL-Course

To start the hands-on click on Open In Colab button 👇 :

We strongly recommend students use Google Colab for the hands-on exercises instead of running them on their personal computers.

By using Google Colab, you can focus on learning and experimenting without worrying about the technical aspects of setting up your environments.

Unit 4: Code your first Deep Reinforcement Learning Algorithm with PyTorch: Reinforce. And test its robustness 💪

In this notebook, you'll code your first Deep Reinforcement Learning algorithm from scratch: Reinforce (also called Monte Carlo Policy Gradient).

Reinforce is a Policy-based method: a Deep Reinforcement Learning algorithm that tries to optimize the policy directly without using an action-value function.

More precisely, Reinforce is a Policy-gradient method, a subclass of Policy-based methods that aims to optimize the policy directly by estimating the weights of the optimal policy using gradient ascent.

To test its robustness, we're going to train it in 2 different simple environments:

• Cartpole-v1
• PixelcopterEnv

⬇️ Here is an example of what you will achieve at the end of this notebook. ⬇️

🎮 Environments: Copy Section

• CartPole-v1
• PixelCopter

📚 RL-Library: Copy Section

• Python
• PyTorch

We're constantly trying to improve our tutorials, so if you find some issues in this notebook, please open an issue on the GitHub Repo.

Objectives of this notebook 🏆 Copy Chapter

At the end of the notebook, you will:

• Be able to code a Reinforce algorithm from scratch using PyTorch.
• Be able to test the robustness of your agent using simple environments.
• Be able to push your trained agent to the Hub with a nice video replay and an evaluation score 🔥.

Prerequisites 🏗️ Copy Chapter

Before diving into the notebook, you need to:

🔲 📚 Study Policy Gradients by reading Unit 4

Let's code Reinforce algorithm from scratch 🔥

Some advice 💡 Copy Chapter

It's better to run this colab in a copy on your Google Drive, so that if it times out you still have the saved notebook on your Google Drive and do not need to fill everything in from scratch.

To do that you can either do Ctrl + S or File > Save a copy in Google Drive.

Set the GPU 💪 Copy Chapter

• To accelerate the agent's training, we'll use a GPU. To do that, go to Runtime > Change Runtime type

• Hardware Accelerator > GPU

Create a virtual display 🖥 Copy Chapter

During the notebook, we'll need to generate a replay video. To do so, with colab, we need to have a virtual screen to be able to render the environment (and thus record the frames).

The following cell will install the librairies and create and run a virtual screen 🖥

%%capture
!apt install python-opengl
!apt install ffmpeg
!apt install xvfb
!pip install pyvirtualdisplay
!pip install pyglet==1.5.1

# Virtual display
from pyvirtualdisplay import Display

virtual_display = Display(visible=0, size=(1400, 900))
virtual_display.start()

Install the dependencies 🔽 Copy Chapter

The first step is to install the dependencies. We’ll install multiple ones:

• gym
• gym-games: Extra gym environments made with PyGame.
• huggingface_hub: The Hub works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations, and other features that will allow you to easily collaborate with others.

You may be wondering why we install gym and not gymnasium, a more recent version of gym? Because the gym-games we are using are not updated yet with gymnasium.

The differences you'll encounter here:

• In gym we don't have terminated and truncated but only done.
• In gym using env.step() returns state, reward, done, info

You can learn more about the differences between Gym and Gymnasium here 👉 https://gymnasium.farama.org/content/migration-guide/

You can see here all the Reinforce models available 👉 https://huggingface.co/models?other=reinforce

And you can find all the Deep Reinforcement Learning models here 👉 https://huggingface.co/models?pipeline_tag=reinforcement-learning

!pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt

Import the packages 📦 Copy Chapter

In addition to importing the installed libraries, we also import:

• imageio: A library that will help us to generate a replay video

import numpy as np

from collections import deque

import matplotlib.pyplot as plt
%matplotlib inline

# PyTorch
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.distributions import Categorical

# Gym
import gym
import gym_pygame

# Hugging Face Hub
from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.
import imageio

Check if we have a GPU Copy Chapter

• Let's check if we have a GPU
• If it's the case you should see device:cuda0

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

print(device)

We're now ready to implement our Reinforce algorithm 🔥

First agent: Playing CartPole-v1 🤖

Create the CartPole environment and understand how it works Copy Chapter

[The environment 🎮](https://www.gymlibrary.dev/environments/classic_control/cart_pole/) Copy Section

Why do we use a simple environment like CartPole-v1? Copy Section

As explained in Reinforcement Learning Tips and Tricks, when you implement your agent from scratch, you need to be sure that it works correctly and find bugs with easy environments before going deeper as finding bugs will be much easier in simple environments.

Try to have some “sign of life” on toy problems

Validate the implementation by making it run on harder and harder envs (you can compare results against the RL zoo). You usually need to run hyperparameter optimization for that step.

The CartPole-v1 environment Copy Section

A pole is attached by an un-actuated joint to a cart, which moves along a frictionless track. The pendulum is placed upright on the cart and the goal is to balance the pole by applying forces in the left and right direction on the cart.

So, we start with CartPole-v1. The goal is to push the cart left or right so that the pole stays in the equilibrium.

The episode ends if:

• The pole Angle is greater than ±12°
• The Cart Position is greater than ±2.4
• The episode length is greater than 500

We get a reward 💰 of +1 every timestep that the Pole stays in the equilibrium.

env_id = "CartPole-v1"
# Create the env
env = gym.make(env_id)

# Create the evaluation env
eval_env = gym.make(env_id)

# Get the state space and action space
s_size = env.observation_space.shape[0]
a_size = env.action_space.n

print("_____OBSERVATION SPACE_____ \n")
print("The State Space is: ", s_size)
print("Sample observation", env.observation_space.sample())  # Get a random observation

print("\n _____ACTION SPACE_____ \n")
print("The Action Space is: ", a_size)
print("Action Space Sample", env.action_space.sample())  # Take a random action

Let's build the Reinforce Architecture Copy Chapter

This implementation is based on three implementations:

• PyTorch official Reinforcement Learning example
• Udacity Reinforce
• Improvement of the integration by Chris1nexus

So we want:

• Two fully connected layers (fc1 and fc2).
• To use ReLU as activation function of fc1
• To use Softmax to output a probability distribution over actions

class Policy(nn.Module):
    def __init__(self, s_size, a_size, h_size):
        super(Policy, self).__init__()
        # Create two fully connected layers



    def forward(self, x):
        # Define the forward pass
        # state goes to fc1 then we apply ReLU activation function

        # fc1 outputs goes to fc2

        # We output the softmax

    def act(self, state):
        """
        Given a state, take action
        """
        state = torch.from_numpy(state).float().unsqueeze(0).to(device)
        probs = self.forward(state).cpu()
        m = Categorical(probs)
        action = np.argmax(m)
        return action.item(), m.log_prob(action)

Solution Copy Section

class Policy(nn.Module):
    def __init__(self, s_size, a_size, h_size):
        super(Policy, self).__init__()
        self.fc1 = nn.Linear(s_size, h_size)
        self.fc2 = nn.Linear(h_size, a_size)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.softmax(x, dim=1)

    def act(self, state):
        state = torch.from_numpy(state).float().unsqueeze(0).to(device)
        probs = self.forward(state).cpu()
        m = Categorical(probs)
        action = np.argmax(m)
        return action.item(), m.log_prob(action)

I made a mistake, can you guess where?

• To find out let's make a forward pass:

debug_policy = Policy(s_size, a_size, 64).to(device)
debug_policy.act(env.reset())

• Here we see that the error says ValueError: The value argument to log_prob must be a Tensor

• It means that action in m.log_prob(action) must be a Tensor but it's not.

• Do you know why? Check the act function and try to see why it does not work.

Advice 💡: Something is wrong in this implementation. Remember that for the act function we want to sample an action from the probability distribution over actions.

(Real) Solution Copy Section

class Policy(nn.Module):
    def __init__(self, s_size, a_size, h_size):
        super(Policy, self).__init__()
        self.fc1 = nn.Linear(s_size, h_size)
        self.fc2 = nn.Linear(h_size, a_size)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.softmax(x, dim=1)

    def act(self, state):
        state = torch.from_numpy(state).float().unsqueeze(0).to(device)
        probs = self.forward(state).cpu()
        m = Categorical(probs)
        action = m.sample()
        return action.item(), m.log_prob(action)

By using CartPole, it was easier to debug since we know that the bug comes from our integration and not from our simple environment.

• Since we want to sample an action from the probability distribution over actions, we can't use action = np.argmax(m) since it will always output the action that has the highest probability.

• We need to replace this with action = m.sample() which will sample an action from the probability distribution P(.|s)

Let's build the Reinforce Training Algorithm Copy Section

This is the Reinforce algorithm pseudocode:

• When we calculate the return Gt (line 6), we see that we calculate the sum of discounted rewards starting at timestep t.

• Why? Because our policy should only reinforce actions on the basis of the consequences: so rewards obtained before taking an action are useless (since they were not because of the action), only the ones that come after the action matters.

• Before coding this you should read this section don't let the past distract you that explains why we use reward-to-go policy gradient.

We use an interesting technique coded by Chris1nexus to compute the return at each timestep efficiently. The comments explained the procedure. Don't hesitate also to check the PR explanation
But overall the idea is to compute the return at each timestep efficiently.

The second question you may ask is why do we minimize the loss? Didn't we talk about Gradient Ascent, not Gradient Descent earlier?

• We want to maximize our utility function $J(\theta)$, but in PyTorch and TensorFlow, it's better to minimize an objective function.
  • So let's say we want to reinforce action 3 at a certain timestep. Before training this action P is 0.25.
  • So we want to modify \(theta \) such that \(\pi_\theta(a_3|s; \theta) > 0.25 \)
  • Because all P must sum to 1, max \(pi_\theta(a_3|s; \theta)\) will minimize other action probability.
  • So we should tell PyTorch to min \(1 - \pi_\theta(a_3|s; \theta)\).
  • This loss function approaches 0 as \(\pi_\theta(a_3|s; \theta)\) nears 1.
  • So we are encouraging the gradient to max \(\pi_\theta(a_3|s; \theta)\)

def reinforce(policy, optimizer, n_training_episodes, max_t, gamma, print_every):
    # Help us to calculate the score during the training
    scores_deque = deque(maxlen=100)
    scores = []
    # Line 3 of pseudocode
    for i_episode in range(1, n_training_episodes+1):
        saved_log_probs = []
        rewards = []
        state = # TODO: reset the environment
        # Line 4 of pseudocode
        for t in range(max_t):
            action, log_prob = # TODO get the action
            saved_log_probs.append(log_prob)
            state, reward, done, _ = # TODO: take an env step
            rewards.append(reward)
            if done:
                break
        scores_deque.append(sum(rewards))
        scores.append(sum(rewards))

        # Line 6 of pseudocode: calculate the return
        returns = deque(maxlen=max_t)
        n_steps = len(rewards)
        # Compute the discounted returns at each timestep,
        # as the sum of the gamma-discounted return at time t (G_t) + the reward at time t

        # In O(N) time, where N is the number of time steps
        # (this definition of the discounted return G_t follows the definition of this quantity
        # shown at page 44 of Sutton&Barto 2017 2nd draft)
        # G_t = r_(t+1) + r_(t+2) + ...

        # Given this formulation, the returns at each timestep t can be computed
        # by re-using the computed future returns G_(t+1) to compute the current return G_t
        # G_t = r_(t+1) + gamma*G_(t+1)
        # G_(t-1) = r_t + gamma* G_t
        # (this follows a dynamic programming approach, with which we memorize solutions in order
        # to avoid computing them multiple times)

        # This is correct since the above is equivalent to (see also page 46 of Sutton&Barto 2017 2nd draft)
        # G_(t-1) = r_t + gamma*r_(t+1) + gamma*gamma*r_(t+2) + ...


        ## Given the above, we calculate the returns at timestep t as:
        #               gamma[t] * return[t] + reward[t]
        #
        ## We compute this starting from the last timestep to the first, in order
        ## to employ the formula presented above and avoid redundant computations that would be needed
        ## if we were to do it from first to last.

        ## Hence, the queue "returns" will hold the returns in chronological order, from t=0 to t=n_steps
        ## thanks to the appendleft() function which allows to append to the position 0 in constant time O(1)
        ## a normal python list would instead require O(N) to do this.
        for t in range(n_steps)[::-1]:
            disc_return_t = (returns[0] if len(returns)>0 else 0)
            returns.appendleft(    ) # TODO: complete here

        ## standardization of the returns is employed to make training more stable
        eps = np.finfo(np.float32).eps.item()

        ## eps is the smallest representable float, which is
        # added to the standard deviation of the returns to avoid numerical instabilities
        returns = torch.tensor(returns)
        returns = (returns - returns.mean()) / (returns.std() + eps)

        # Line 7:
        policy_loss = []
        for log_prob, disc_return in zip(saved_log_probs, returns):
            policy_loss.append(-log_prob * disc_return)
        policy_loss = torch.cat(policy_loss).sum()

        # Line 8: PyTorch prefers gradient descent
        optimizer.zero_grad()
        policy_loss.backward()
        optimizer.step()

        if i_episode % print_every == 0:
            print('Episode {}\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))

    return scores

Solution

def reinforce(policy, optimizer, n_training_episodes, max_t, gamma, print_every):
    # Help us to calculate the score during the training
    scores_deque = deque(maxlen=100)
    scores = []
    # Line 3 of pseudocode
    for i_episode in range(1, n_training_episodes + 1):
        saved_log_probs = []
        rewards = []
        state = env.reset()
        # Line 4 of pseudocode
        for t in range(max_t):
            action, log_prob = policy.act(state)
            saved_log_probs.append(log_prob)
            state, reward, done, _ = env.step(action)
            rewards.append(reward)
            if done:
                break
        scores_deque.append(sum(rewards))
        scores.append(sum(rewards))

        # Line 6 of pseudocode: calculate the return
        returns = deque(maxlen=max_t)
        n_steps = len(rewards)
        # Compute the discounted returns at each timestep,
        # as
        #      the sum of the gamma-discounted return at time t (G_t) + the reward at time t
        #
        # In O(N) time, where N is the number of time steps
        # (this definition of the discounted return G_t follows the definition of this quantity
        # shown at page 44 of Sutton&Barto 2017 2nd draft)
        # G_t = r_(t+1) + r_(t+2) + ...

        # Given this formulation, the returns at each timestep t can be computed
        # by re-using the computed future returns G_(t+1) to compute the current return G_t
        # G_t = r_(t+1) + gamma*G_(t+1)
        # G_(t-1) = r_t + gamma* G_t
        # (this follows a dynamic programming approach, with which we memorize solutions in order
        # to avoid computing them multiple times)

        # This is correct since the above is equivalent to (see also page 46 of Sutton&Barto 2017 2nd draft)
        # G_(t-1) = r_t + gamma*r_(t+1) + gamma*gamma*r_(t+2) + ...

        ## Given the above, we calculate the returns at timestep t as:
        #               gamma[t] * return[t] + reward[t]
        #
        ## We compute this starting from the last timestep to the first, in order
        ## to employ the formula presented above and avoid redundant computations that would be needed
        ## if we were to do it from first to last.

        ## Hence, the queue "returns" will hold the returns in chronological order, from t=0 to t=n_steps
        ## thanks to the appendleft() function which allows to append to the position 0 in constant time O(1)
        ## a normal python list would instead require O(N) to do this.
        for t in range(n_steps)[::-1]:
            disc_return_t = returns[0] if len(returns) > 0 else 0
            returns.appendleft(gamma * disc_return_t + rewards[t])

        ## standardization of the returns is employed to make training more stable
        eps = np.finfo(np.float32).eps.item()
        ## eps is the smallest representable float, which is
        # added to the standard deviation of the returns to avoid numerical instabilities
        returns = torch.tensor(returns)
        returns = (returns - returns.mean()) / (returns.std() + eps)

        # Line 7:
        policy_loss = []
        for log_prob, disc_return in zip(saved_log_probs, returns):
            policy_loss.append(-log_prob * disc_return)
        policy_loss = torch.cat(policy_loss).sum()

        # Line 8: PyTorch prefers gradient descent
        optimizer.zero_grad()
        policy_loss.backward()
        optimizer.step()

        if i_episode % print_every == 0:
            print("Episode {}\tAverage Score: {:.2f}".format(i_episode, np.mean(scores_deque)))

    return scores

Train it Copy Chapter

• We're now ready to train our agent.
• But first, we define a variable containing all the training hyperparameters.
• You can change the training parameters (and should 😉)

cartpole_hyperparameters = {
    "h_size": 16,
    "n_training_episodes": 1000,
    "n_evaluation_episodes": 10,
    "max_t": 1000,
    "gamma": 1.0,
    "lr": 1e-2,
    "env_id": env_id,
    "state_space": s_size,
    "action_space": a_size,
}

# Create policy and place it to the device
cartpole_policy = Policy(
    cartpole_hyperparameters["state_space"],
    cartpole_hyperparameters["action_space"],
    cartpole_hyperparameters["h_size"],
).to(device)
cartpole_optimizer = optim.Adam(cartpole_policy.parameters(), lr=cartpole_hyperparameters["lr"])

scores = reinforce(
    cartpole_policy,
    cartpole_optimizer,
    cartpole_hyperparameters["n_training_episodes"],
    cartpole_hyperparameters["max_t"],
    cartpole_hyperparameters["gamma"],
    100,
)

Define evaluation method 📝 Copy Chapter

• Here we define the evaluation method that we're going to use to test our Reinforce agent.

def evaluate_agent(env, max_steps, n_eval_episodes, policy):
    """
    Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.
    :param env: The evaluation environment
    :param n_eval_episodes: Number of episode to evaluate the agent
    :param policy: The Reinforce agent
    """
    episode_rewards = []
    for episode in range(n_eval_episodes):
        state = env.reset()
        step = 0
        done = False
        total_rewards_ep = 0

        for step in range(max_steps):
            action, _ = policy.act(state)
            new_state, reward, done, info = env.step(action)
            total_rewards_ep += reward

            if done:
                break
            state = new_state
        episode_rewards.append(total_rewards_ep)
    mean_reward = np.mean(episode_rewards)
    std_reward = np.std(episode_rewards)

    return mean_reward, std_reward

Evaluate our agent 📈 Copy Chapter

evaluate_agent(
    eval_env, cartpole_hyperparameters["max_t"], cartpole_hyperparameters["n_evaluation_episodes"], cartpole_policy
)

Publish our trained model on the Hub 🔥 Copy Section

Now that we saw we got good results after the training, we can publish our trained model on the hub 🤗 with one line of code.

Here's an example of a Model Card:

Push to the Hub Copy Section

Do not modify this code

from huggingface_hub import HfApi, snapshot_download
from huggingface_hub.repocard import metadata_eval_result, metadata_save

from pathlib import Path
import datetime
import json
import imageio

import tempfile

import os

def record_video(env, policy, out_directory, fps=30):
    """
    Generate a replay video of the agent
    :param env
    :param Qtable: Qtable of our agent
    :param out_directory
    :param fps: how many frame per seconds (with taxi-v3 and frozenlake-v1 we use 1)
    """
    images = []
    done = False
    state = env.reset()
    img = env.render(mode="rgb_array")
    images.append(img)
    while not done:
        # Take the action (index) that have the maximum expected future reward given that state
        action, _ = policy.act(state)
        state, reward, done, info = env.step(action)  # We directly put next_state = state for recording logic
        img = env.render(mode="rgb_array")
        images.append(img)
    imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)

def push_to_hub(repo_id,
                model,
                hyperparameters,
                eval_env,
                video_fps=30
                ):
  """
  Evaluate, Generate a video and Upload a model to Hugging Face Hub.
  This method does the complete pipeline:
  - It evaluates the model
  - It generates the model card
  - It generates a replay video of the agent
  - It pushes everything to the Hub

  :param repo_id: repo_id: id of the model repository from the Hugging Face Hub
  :param model: the pytorch model we want to save
  :param hyperparameters: training hyperparameters
  :param eval_env: evaluation environment
  :param video_fps: how many frame per seconds to record our video replay
  """

  _, repo_name = repo_id.split("/")
  api = HfApi()

  # Step 1: Create the repo
  repo_url = api.create_repo(
        repo_id=repo_id,
        exist_ok=True,
  )

  with tempfile.TemporaryDirectory() as tmpdirname:
    local_directory = Path(tmpdirname)

    # Step 2: Save the model
    torch.save(model, local_directory / "model.pt")

    # Step 3: Save the hyperparameters to JSON
    with open(local_directory / "hyperparameters.json", "w") as outfile:
      json.dump(hyperparameters, outfile)

    # Step 4: Evaluate the model and build JSON
    mean_reward, std_reward = evaluate_agent(eval_env,
                                            hyperparameters["max_t"],
                                            hyperparameters["n_evaluation_episodes"],
                                            model)
    # Get datetime
    eval_datetime = datetime.datetime.now()
    eval_form_datetime = eval_datetime.isoformat()

    evaluate_data = {
          "env_id": hyperparameters["env_id"],
          "mean_reward": mean_reward,
          "n_evaluation_episodes": hyperparameters["n_evaluation_episodes"],
          "eval_datetime": eval_form_datetime,
    }

    # Write a JSON file
    with open(local_directory / "results.json", "w") as outfile:
        json.dump(evaluate_data, outfile)

    # Step 5: Create the model card
    env_name = hyperparameters["env_id"]

    metadata = {}
    metadata["tags"] = [
          env_name,
          "reinforce",
          "reinforcement-learning",
          "custom-implementation",
          "deep-rl-class"
      ]

    # Add metrics
    eval = metadata_eval_result(
        model_pretty_name=repo_name,
        task_pretty_name="reinforcement-learning",
        task_id="reinforcement-learning",
        metrics_pretty_name="mean_reward",
        metrics_id="mean_reward",
        metrics_value=f"{mean_reward:.2f} +/- {std_reward:.2f}",
        dataset_pretty_name=env_name,
        dataset_id=env_name,
      )

    # Merges both dictionaries
    metadata = {**metadata, **eval}

    model_card = f"""
  # **Reinforce** Agent playing **{env_id}**
  This is a trained model of a **Reinforce** agent playing **{env_id}** .
  To learn to use this model and train yours check Unit 4 of the Deep Reinforcement Learning Course: https://huggingface.co/deep-rl-course/unit4/introduction
  """

    readme_path = local_directory / "README.md"
    readme = ""
    if readme_path.exists():
        with readme_path.open("r", encoding="utf8") as f:
          readme = f.read()
    else:
      readme = model_card

    with readme_path.open("w", encoding="utf-8") as f:
      f.write(readme)

    # Save our metrics to Readme metadata
    metadata_save(readme_path, metadata)

    # Step 6: Record a video
    video_path =  local_directory / "replay.mp4"
    record_video(env, model, video_path, video_fps)

    # Step 7. Push everything to the Hub
    api.upload_folder(
          repo_id=repo_id,
          folder_path=local_directory,
          path_in_repo=".",
    )

    print(f"Your model is pushed to the Hub. You can view your model here: {repo_url}")

By using push_to_hub, you evaluate, record a replay, generate a model card of your agent, and push it to the Hub.

This way:

• You can showcase our work 🔥
• You can visualize your agent playing 👀
• You can share an agent with the community that others can use 💾
• You can access a leaderboard 🏆 to see how well your agent is performing compared to your classmates 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard

To be able to share your model with the community there are three more steps to follow:

1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join

2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.

• Create a new token (https://huggingface.co/settings/tokens) with write role

notebook_login()

If you don't want to use Google Colab or a Jupyter Notebook, you need to use this command instead: huggingface-cli login (or login)

3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥 using package_to_hub() function

repo_id = ""  # TODO Define your repo id {username/Reinforce-{model-id}}
push_to_hub(
    repo_id,
    cartpole_policy,  # The model we want to save
    cartpole_hyperparameters,  # Hyperparameters
    eval_env,  # Evaluation environment
    video_fps=30
)

Now that we tested the robustness of our implementation, let's try a more complex environment: PixelCopter 🚁

Second agent: PixelCopter 🚁 Copy Chapter

Study the PixelCopter environment 👀 Copy Section

• The Environment documentation

env_id = "Pixelcopter-PLE-v0"
env = gym.make(env_id)
eval_env = gym.make(env_id)
s_size = env.observation_space.shape[0]
a_size = env.action_space.n

print("_____OBSERVATION SPACE_____ \n")
print("The State Space is: ", s_size)
print("Sample observation", env.observation_space.sample())  # Get a random observation

print("\n _____ACTION SPACE_____ \n")
print("The Action Space is: ", a_size)
print("Action Space Sample", env.action_space.sample())  # Take a random action

The observation space (7) 👀:

• player y position
• player velocity
• player distance to floor
• player distance to ceiling
• next block x distance to player
• next blocks top y location
• next blocks bottom y location

The action space(2) 🎮:

• Up (press accelerator)
• Do nothing (don't press accelerator)

The reward function 💰:

• For each vertical block it passes, it gains a positive reward of +1. Each time a terminal state is reached it receives a negative reward of -1.

Define the new Policy 🧠 Copy Section

• We need to have a deeper neural network since the environment is more complex

class Policy(nn.Module):
    def __init__(self, s_size, a_size, h_size):
        super(Policy, self).__init__()
        # Define the three layers here

    def forward(self, x):
        # Define the forward process here
        return F.softmax(x, dim=1)

    def act(self, state):
        state = torch.from_numpy(state).float().unsqueeze(0).to(device)
        probs = self.forward(state).cpu()
        m = Categorical(probs)
        action = m.sample()
        return action.item(), m.log_prob(action)

Solution

class Policy(nn.Module):
    def __init__(self, s_size, a_size, h_size):
        super(Policy, self).__init__()
        self.fc1 = nn.Linear(s_size, h_size)
        self.fc2 = nn.Linear(h_size, h_size * 2)
        self.fc3 = nn.Linear(h_size * 2, a_size)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return F.softmax(x, dim=1)

    def act(self, state):
        state = torch.from_numpy(state).float().unsqueeze(0).to(device)
        probs = self.forward(state).cpu()
        m = Categorical(probs)
        action = m.sample()
        return action.item(), m.log_prob(action)

Define the hyperparameters ⚙️ Copy Section

• Because this environment is more complex.
• Especially for the hidden size, we need more neurons.

pixelcopter_hyperparameters = {
    "h_size": 64,
    "n_training_episodes": 50000,
    "n_evaluation_episodes": 10,
    "max_t": 10000,
    "gamma": 0.99,
    "lr": 1e-4,
    "env_id": env_id,
    "state_space": s_size,
    "action_space": a_size,
}

Train it Copy Section

• We're now ready to train our agent 🔥.

# Create policy and place it to the device
# torch.manual_seed(50)
pixelcopter_policy = Policy(
    pixelcopter_hyperparameters["state_space"],
    pixelcopter_hyperparameters["action_space"],
    pixelcopter_hyperparameters["h_size"],
).to(device)
pixelcopter_optimizer = optim.Adam(pixelcopter_policy.parameters(), lr=pixelcopter_hyperparameters["lr"])

scores = reinforce(
    pixelcopter_policy,
    pixelcopter_optimizer,
    pixelcopter_hyperparameters["n_training_episodes"],
    pixelcopter_hyperparameters["max_t"],
    pixelcopter_hyperparameters["gamma"],
    1000,
)

Publish our trained model on the Hub 🔥 Copy Section

repo_id = ""  # TODO Define your repo id {username/Reinforce-{model-id}}
push_to_hub(
    repo_id,
    pixelcopter_policy,  # The model we want to save
    pixelcopter_hyperparameters,  # Hyperparameters
    eval_env,  # Evaluation environment
    video_fps=30
)

Some additional challenges 🏆 Copy Chapter

The best way to learn is to try things on your own! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. But also try to find better parameters.

In the Leaderboard you will find your agents. Can you get to the top?

Here are some ideas to climb up the leaderboard:

• Train more steps
• Try different hyperparameters by looking at what your classmates have done 👉 https://huggingface.co/models?other=reinforce
• Push your new trained model on the Hub 🔥
• Improving the implementation for more complex environments (for instance, what about changing the network to a Convolutional Neural Network to handle
  frames as observation)?

Congrats on finishing this unit! There was a lot of information.
And congrats on finishing the tutorial. You've just coded your first Deep Reinforcement Learning agent from scratch using PyTorch and shared it on the Hub 🥳.

Don't hesitate to iterate on this unit by improving the implementation for more complex environments (for instance, what about changing the network to a Convolutional Neural Network to handle
frames as observation)?

In the next unit, we're going to learn more about Unity MLAgents, by training agents in Unity environments. This way, you will be ready to participate in the AI vs AI challenges where you'll train your agents
to compete against other agents in a snowball fight and a soccer game.

Sound fun? See you next time!

Finally, we would love to hear what you think of the course and how we can improve it. If you have some feedback then please 👉 fill this form

See you in Unit 5! 🔥

Keep Learning, stay awesome 🤗 Copy Section

File: units/en/unit4/introduction.mdx Copy Chapter

Introduction [[introduction]]

In the last unit, we learned about Deep Q-Learning. In this value-based deep reinforcement learning algorithm, we used a deep neural network to approximate the different Q-values for each possible action at a state.

Since the beginning of the course, we have only studied value-based methods, where we estimate a value function as an intermediate step towards finding an optimal policy.

In value-based methods, the policy ** (π) only exists because of the action value estimates since the policy is just a function** (for instance, greedy-policy) that will select the action with the highest value given a state.

With policy-based methods, we want to optimize the policy directly without having an intermediate step of learning a value function.

So today, we'll learn about policy-based methods and study a subset of these methods called policy gradient. Then we'll implement our first policy gradient algorithm called Monte Carlo Reinforce from scratch using PyTorch.
Then, we'll test its robustness using the CartPole-v1 and PixelCopter environments.

You'll then be able to iterate and improve this implementation for more advanced environments.

Let's get started!