# Technical Documentation: huggingface/deep-rl-class > โ„น๏ธ **Provenance:** In-Tree Documentation (56 Chapters) ยท [CodeWiki Reference](https://codewiki.google/github.com/huggingface/deep-rl-class) ยท Recency: Active (< 180 days) ## File: README.md # [The Hugging Face Deep Reinforcement Learning Course ๐Ÿค— (v2.0)](https://huggingface.co/deep-rl-course/unit0/introduction) If you like the course, don't hesitate to **โญ star this repository. This helps us ๐Ÿค—**. This repository contains the Deep Reinforcement Learning Course mdx files and notebooks. **The website is here**: https://huggingface.co/deep-rl-course/unit0/introduction?fw=pt - The syllabus ๐Ÿ“š: https://simoninithomas.github.io/deep-rl-course - The course ๐Ÿ“š: https://huggingface.co/deep-rl-course/unit0/introduction?fw=pt - **Sign up here** โžก๏ธโžก๏ธโžก๏ธ http://eepurl.com/ic5ZUD ## Course Maintenance Notice ๐Ÿšง Please note that this **Deep Reinforcement Learning course is now in a low-maintenance state**. However, it **remains an excellent resource to learn both the theory and practical aspects of Deep Reinforcement Learning**. Keep in mind the following points: - *Unit 7 (AI vs AI)* : This feature is currently non-functional. However, you can still train your agent to play soccer and observe its performance. - *Leaderboard* : The leaderboard is no longer operational. Aside from these points, all theory content and practical exercises remain fully accessible and effective for learning. If you have any problem with one of the hands-on **please check the issue sections where the community give some solutions to bugs**. ## Citing the project To cite this repository in publications: ```bibtex @misc{deep-rl-course, author = {Simonini, Thomas and Sanseviero, Omar}, title = {The Hugging Face Deep Reinforcement Learning Class}, year = {2023}, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {\url{https://github.com/huggingface/deep-rl-class}}, } ``` --- ## File: LICENSE.md Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --- ## File: notebooks/unit1/requirements-unit1.txt stable-baselines3==2.0.0a5 swig gymnasium[box2d] huggingface_sb3 --- ## File: notebooks/unit2/requirements-unit2.txt gymnasium pygame numpy huggingface_hub pickle5 pyyaml==6.0 imageio imageio_ffmpeg pyglet==1.5.1 tqdm --- ## File: notebooks/unit4/requirements-unit4.txt git+https://github.com/ntasfi/PyGame-Learning-Environment.git git+https://github.com/simoninithomas/gym-games huggingface_hub imageio-ffmpeg pyyaml==6.0 --- ## File: notebooks/unit6/requirements-unit6.txt stable-baselines3==2.0.0a4 huggingface_sb3 panda-gym huggingface_hub --- ## File: units/en/_toctree.yml - title: Unit 0. Welcome to the course sections: - local: unit0/introduction title: Welcome to the course ๐Ÿค— - local: unit0/setup title: Setup - local: unit0/discord101 title: Discord 101 - title: Unit 1. Introduction to Deep Reinforcement Learning sections: - local: unit1/introduction title: Introduction - local: unit1/what-is-rl title: What is Reinforcement Learning? - local: unit1/rl-framework title: The Reinforcement Learning Framework - local: unit1/tasks title: The type of tasks - local: unit1/exp-exp-tradeoff title: The Exploration/ Exploitation tradeoff - local: unit1/two-methods title: The two main approaches for solving RL problems - local: unit1/deep-rl title: The โ€œDeepโ€ in Deep Reinforcement Learning - local: unit1/summary title: Summary - local: unit1/glossary title: Glossary - local: unit1/hands-on title: Hands-on - local: unit1/quiz title: Quiz - local: unit1/conclusion title: Conclusion - local: unit1/additional-readings title: Additional Readings - title: Bonus Unit 1. Introduction to Deep Reinforcement Learning with Huggy sections: - local: unitbonus1/introduction title: Introduction - local: unitbonus1/how-huggy-works title: How Huggy works? - local: unitbonus1/train title: Train Huggy - local: unitbonus1/play title: Play with Huggy - local: unitbonus1/conclusion title: Conclusion - title: Live 1. How the course work, Q&A, and playing with Huggy sections: - local: live1/live1 title: Live 1. How the course work, Q&A, and playing with Huggy ๐Ÿถ - title: Unit 2. Introduction to Q-Learning sections: - local: unit2/introduction title: Introduction - local: unit2/what-is-rl title: What is RL? A short recap - local: unit2/two-types-value-based-methods title: The two types of value-based methods - local: unit2/bellman-equation title: The Bellman Equation, simplify our value estimation - local: unit2/mc-vs-td title: Monte Carlo vs Temporal Difference Learning - local: unit2/mid-way-recap title: Mid-way Recap - local: unit2/mid-way-quiz title: Mid-way Quiz - local: unit2/q-learning title: Introducing Q-Learning - local: unit2/q-learning-example title: A Q-Learning example - local: unit2/q-learning-recap title: Q-Learning Recap - local: unit2/glossary title: Glossary - local: unit2/hands-on title: Hands-on - local: unit2/quiz2 title: Q-Learning Quiz - local: unit2/conclusion title: Conclusion - local: unit2/additional-readings title: Additional Readings - title: Unit 3. Deep Q-Learning with Atari Games sections: - local: unit3/introduction title: Introduction - local: unit3/from-q-to-dqn title: From Q-Learning to Deep Q-Learning - local: unit3/deep-q-network title: The Deep Q-Network (DQN) - local: unit3/deep-q-algorithm title: The Deep Q Algorithm - local: unit3/glossary title: Glossary - local: unit3/hands-on title: Hands-on - local: unit3/quiz title: Quiz - local: unit3/conclusion title: Conclusion - local: unit3/additional-readings title: Additional Readings - title: Bonus Unit 2. Automatic Hyperparameter Tuning with Optuna sections: - local: unitbonus2/introduction title: Introduction - local: unitbonus2/optuna title: Optuna - local: unitbonus2/hands-on title: Hands-on - title: Unit 4. Policy Gradient with PyTorch sections: - local: unit4/introduction title: Introduction - local: unit4/what-are-policy-based-methods title: What are the policy-based methods? - local: unit4/advantages-disadvantages title: The advantages and disadvantages of policy-gradient methods - local: unit4/policy-gradient title: Diving deeper into policy-gradient - local: unit4/pg-theorem title: (Optional) the Policy Gradient Theorem - local: unit4/glossary title: Glossary - local: unit4/hands-on title: Hands-on - local: unit4/quiz title: Quiz - local: unit4/conclusion title: Conclusion - local: unit4/additional-readings title: Additional Readings - title: Unit 5. Introduction to Unity ML-Agents sections: - local: unit5/introduction title: Introduction - local: unit5/how-mlagents-works title: How ML-Agents works? - local: unit5/snowball-target title: The SnowballTarget environment - local: unit5/pyramids title: The Pyramids environment - local: unit5/curiosity title: (Optional) What is curiosity in Deep Reinforcement Learning? - local: unit5/hands-on title: Hands-on - local: unit5/bonus title: Bonus. Learn to create your own environments with Unity and MLAgents - local: unit5/quiz title: Quiz - local: unit5/conclusion title: Conclusion - title: Unit 6. Actor Critic methods with Robotics environments sections: - local: unit6/introduction title: Introduction - local: unit6/variance-problem title: The Problem of Variance in Reinforce - local: unit6/advantage-actor-critic title: Advantage Actor Critic (A2C) - local: unit6/hands-on title: Advantage Actor Critic (A2C) using Robotics Simulations with Panda-Gym ๐Ÿค– - local: unit6/quiz title: Quiz - local: unit6/conclusion title: Conclusion - local: unit6/additional-readings title: Additional Readings - title: Unit 7. Introduction to Multi-Agents and AI vs AI sections: - local: unit7/introduction title: Introduction - local: unit7/introduction-to-marl title: An introduction to Multi-Agents Reinforcement Learning (MARL) - local: unit7/multi-agent-setting title: Designing Multi-Agents systems - local: unit7/self-play title: Self-Play - local: unit7/hands-on title: Let's train our soccer team to beat your classmates' teams (AI vs. AI) - local: unit7/quiz title: Quiz - local: unit7/conclusion title: Conclusion - local: unit7/additional-readings title: Additional Readings - title: Unit 8. Part 1 Proximal Policy Optimization (PPO) sections: - local: unit8/introduction title: Introduction - local: unit8/intuition-behind-ppo title: The intuition behind PPO - local: unit8/clipped-surrogate-objective title: Introducing the Clipped Surrogate Objective Function - local: unit8/visualize title: Visualize the Clipped Surrogate Objective Function - local: unit8/hands-on-cleanrl title: PPO with CleanRL - local: unit8/conclusion title: Conclusion - local: unit8/additional-readings title: Additional Readings - title: Unit 8. Part 2 Proximal Policy Optimization (PPO) with Doom sections: - local: unit8/introduction-sf title: Introduction - local: unit8/hands-on-sf title: PPO with Sample Factory and Doom - local: unit8/conclusion-sf title: Conclusion - title: Bonus Unit 3. Advanced Topics in Reinforcement Learning sections: - local: unitbonus3/introduction title: Introduction - local: unitbonus3/model-based title: Model-Based Reinforcement Learning - local: unitbonus3/offline-online title: Offline vs. Online Reinforcement Learning - local: unitbonus3/generalisation title: Generalisation Reinforcement Learning - local: unitbonus3/rlhf title: Reinforcement Learning from Human Feedback - local: unitbonus3/decision-transformers title: Decision Transformers and Offline RL - local: unitbonus3/language-models title: Language models in RL - local: unitbonus3/curriculum-learning title: (Automatic) Curriculum Learning for RL - local: unitbonus3/envs-to-try title: Interesting environments to try - local: unitbonus3/learning-agents title: An introduction to Unreal Learning Agents - local: unitbonus3/godotrl title: An Introduction to Godot RL - local: unitbonus3/student-works title: Students projects - local: unitbonus3/rl-documentation title: Brief introduction to RL documentation - title: Bonus Unit 5. Imitation Learning with Godot RL Agents sections: - local: unitbonus5/introduction title: Introduction - local: unitbonus5/the-environment title: The environment - local: unitbonus5/getting-started title: Getting started - local: unitbonus5/train-our-robot title: Train our robot - local: unitbonus5/customize-the-environment title: (Optional) Customize the environment - local: unitbonus5/conclusion title: Conclusion - title: Certification and congratulations sections: - local: communication/conclusion title: Congratulations - local: communication/certification title: Get your certificate of completion --- ## File: units/en/communication/certification.mdx # The certification process The certification process is **completely free**: - To get a *certificate of completion*: you need **to pass 80% of the assignments**. - To get a *certificate of excellence*: you need **to pass 100% of the assignments**. There's **no deadlines, the course is self-paced**. When we say pass, **we mean that your model must be pushed to the Hub and get a result equal or above the minimal requirement**. To check your progression and which unit you passed/not passed: https://huggingface.co/spaces/ThomasSimonini/Check-my-progress-Deep-RL-Course Now that you're ready for the certification process, you need to: 1. Go here: https://huggingface.co/spaces/huggingface-projects/Deep-RL-Course-Certification/ 2. Type your *hugging face username*, your *first name*, *last name* 3. Click on "Generate my certificate". - If you passed 80% of the assignments, **congratulations** you've just got the certificate of completion. - If you passed 100% of the assignments, **congratulations** you've just got the excellence certificate. - If you are below 80%, don't be discouraged! Check which units you need to do again to get your certificate. 4. You can download your certificate in pdf format and png format. Don't hesitate to share your certificate on Twitter (tag me @ThomasSimonini and @huggingface) and on Linkedin. --- ## File: units/en/communication/conclusion.mdx # Congratulations **Congratulations on finishing this course!** With perseverance, hard work, and determination, **you've acquired a solid background in Deep Reinforcement Learning**. But finishing this course is **not the end of your journey**. It's just the beginning: don't hesitate to explore bonus unit 3, where we show you topics you may be interested in studying. And don't hesitate to **share what you're doing, and ask questions in the discord server** **Thank you** for being part of this course. **I hope you liked this course as much as I loved writing it**. Don't hesitate **to give us feedback on how we can improve the course** using [this form](https://forms.gle/BzKXWzLAGZESGNaE9) And don't forget **to check in the next section how you can get (if you pass) your certificate of completion โ€Žโ€๐ŸŽ“.** One last thing, to keep in touch with the Reinforcement Learning Team and with me: - [Follow me on Twitter](https://twitter.com/thomassimonini) - [Follow Hugging Face Twitter account](https://twitter.com/huggingface) - [Join the Hugging Face Discord](https://www.hf.co/join/discord) ## Keep Learning, Stay Awesome ๐Ÿค— Thomas Simonini, --- ## File: units/en/live1/live1.mdx # Live 1: How the course work, Q&A, and playing with Huggy In this first live stream, we explained how the course work (scope, units, challenges, and more) and answered your questions. And finally, we saw some LunarLander agents you've trained and play with your Huggies ๐Ÿถ To know when the next live is scheduled **check the discord server**. We will also send **you an email**. If you can't participate, don't worry, we record the live sessions. --- ## File: units/en/unit0/discord101.mdx # Discord 101 [[discord-101]] Hey there! My name is Huggy, the dog ๐Ÿ•, and I'm looking forward to train with you during this RL Course! Although I don't know much about fetching sticks (yet), I know one or two things about Discord. So I wrote this guide to help you learn about it! Discord is a free chat platform. If you've used Slack, **it's quite similar**. There is a Hugging Face Community Discord server with 50000 members you can [join with a single click here](https://discord.gg/ydHrjt3WP5). So many humans to play with! Starting in Discord can be a bit intimidating, so let me take you through it. When you [sign-up to our Discord server](http://hf.co/join/discord), you'll choose your interests. Make sure to **click "Reinforcement Learning,"** and you'll get access to the Reinforcement Learning Category containing all the course-related channels. If you feel like joining even more channels, go for it! ๐Ÿš€ Then click next, you'll then get to **introduce yourself in the `#introduce-yourself` channel**. They are in the reinforcement learning category. **Don't forget to sign up to these channels** by clicking on ๐Ÿค– Reinforcement Learning in `role-assigment`. - `rl-announcements`: where we give the **latest information about the course**. - `rl-discussions`: where you can **exchange about RL and share information**. - `rl-study-group`: where you can **ask questions and exchange with your classmates**. - `rl-i-made-this`: where you can **share your projects and models**. The HF Community Server has a thriving community of human beings interested in many areas, so you can also learn from those. There are paper discussions, events, and many other things. Was this useful? There are a couple of tips I can share with you: - There are **voice channels** you can use as well, although most people prefer text chat. - You can **use markdown style** for text chats. So if you're writing code, you can use that style. Sadly this does not work as well for links. - You can open threads as well! It's a good idea when **it's a long conversation**. I hope this is useful! And if you have questions, just ask! See you later! Huggy ๐Ÿถ --- ## File: units/en/unit0/introduction.mdx # Welcome to the ๐Ÿค— Deep Reinforcement Learning Course [[introduction]] Welcome to the most fascinating topic in Artificial Intelligence: **Deep Reinforcement Learning**. This course will **teach you about Deep Reinforcement Learning from beginner to expert**. Itโ€™s completely free and open-source! In this introduction unit youโ€™ll: - Learn more about the **course content**. - **Define the path** youโ€™re going to take (either self-audit or certification process). - Learn more about the **AI vs. AI challenges** you're going to participate in. - Learn more **about us**. - **Create your Hugging Face account** (itโ€™s free). - **Sign-up to our Discord server**, the place where you can chat with your classmates and us (the Hugging Face team). Letโ€™s get started! ## What to expect? [[expect]] In this course, you will: - ๐Ÿ“– Study Deep Reinforcement Learning in **theory and practice.** - ๐Ÿง‘โ€๐Ÿ’ป Learn to **use famous Deep RL libraries** such as [Stable Baselines3](https://stable-baselines3.readthedocs.io/en/master/), [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo), [Sample Factory](https://samplefactory.dev/) and [CleanRL](https://github.com/vwxyzjn/cleanrl). - ๐Ÿค– **Train agents in unique environments** such as [SnowballFight](https://huggingface.co/spaces/ThomasSimonini/SnowballFight), [Huggy the Doggo ๐Ÿถ](https://huggingface.co/spaces/ThomasSimonini/Huggy), [VizDoom (Doom)](https://vizdoom.cs.put.edu.pl/)ย and classical ones such as [Space Invaders](https://gymnasium.farama.org/environments/atari/space_invaders/), [PyBullet](https://pybullet.org/wordpress/) and more. - ๐Ÿ’พ Share your **trained agents with one line of code to the Hub** and also download powerful agents from the community. - ๐Ÿ† Participate in challenges where you will **evaluate your agents against other teams. You'll also get to play against the agents you'll train.** - ๐ŸŽ“ **Earn a certificate of completion** by completing 80% of the assignments. And more! At the end of this course, **youโ€™ll get a solid foundation from the basics to the SOTA (state-of-the-art) of methods**. Donโ€™t forget to **[sign up to the course](http://eepurl.com/ic5ZUD)** (we are collecting your email to be able toย **send you the links when each Unit is published and give you information about the challenges and updates).** Sign up ๐Ÿ‘‰ [here](http://eepurl.com/ic5ZUD) ## Course Maintenance Notice ๐Ÿšง Please note that this **Deep Reinforcement Learning course is now in a low-maintenance state**. However, it **remains an excellent resource to learn both the theory and practical aspects of Deep Reinforcement Learning**. Keep in mind the following points: - *Unit 7 (AI vs AI)* : This feature is currently non-functional. However, you can still train your agent to play soccer and observe its performance. - *Leaderboard* : The leaderboard is no longer operational. Aside from these points, all theory content and practical exercises remain fully accessible and effective for learning. If you have any problem with one of the hands-on **please check the issue sections where the community give some solutions to bugs**. ## What does the course look like? [[course-look-like]] The course is composed of: - *A theory part*: where you learn a **concept in theory**. - *A hands-on*: where youโ€™ll learn **to use famous Deep RL libraries** to train your agents in unique environments. These hands-on will be **Google Colab notebooks with companion tutorial videos** if you prefer learning with video format! - *Challenges*: you'll get to put your agent to compete against other agents in different challenges. There will also be [a leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) for you to compare the agents' performance. ## What's the syllabus? [[syllabus]] This is the course's syllabus: ## Two paths: choose your own adventure [[two-paths]] You can choose to follow this course either: - *To get a certificate of completion*: you need to complete 80% of the assignments. - *To get a certificate of honors*: you need to complete 100% of the assignments. - *As a simple audit*: you can participate in all challenges and do assignments if you want. There's **no deadlines, the course is self-paced**. Both paths **are completely free**. Whatever path you choose, we advise you **to follow the recommended pace to enjoy the course and challenges with your fellow classmates.** You don't need to tell us which path you choose. **If you get more than 80% of the assignments done, you'll get a certificate.** ## The Certification Process [[certification-process]] The certification process is **completely free**: - *To get a certificate of completion*: you need to complete 80% of the assignments. - *To get a certificate of honors*: you need to complete 100% of the assignments. Again, there's **no deadline** since the course is self paced. But our advice **is to follow the recommended pace section**. ## How to get most of the course? [[advice]] To get most of the course, we have some advice: 1. [Join study groups in Discord ](https://discord.gg/ydHrjt3WP5): studying in groups is always easier. To do that, you need to join our discord server. If you're new to Discord, no worries! We have some tools that will help you learn about it. 2. **Do the quizzes and assignments**: the best way to learn is to do and test yourself. 3. **Define a schedule to stay in sync**: you can use our recommended pace schedule below or create yours. ## What tools do I need? [[tools]] You need only 3 things: - *A computer* with an internet connection. - *Google Colab (free version)*: most of our hands-on will use Google Colab, the **free version is enough.** - A *Hugging Face Account*: to push and load models. If you donโ€™t have an account yet, you can create one **[here](https://hf.co/join)** (itโ€™s free). ## What is the recommended pace? [[recommended-pace]] Each chapter in this course is designed **to be completed in 1 week, with approximately 3-4 hours of work per week**. However, you can take as much time as necessary to complete the course. If you want to dive into a topic more in-depth, we'll provide additional resources to help you achieve that. ## Who are we [[who-are-we]] About the author: - [Thomas Simonini](https://twitter.com/ThomasSimonini) is a Developer Advocate at Hugging Face ๐Ÿค— specializing in Deep Reinforcement Learning. He founded the Deep Reinforcement Learning Course in 2018, which became one of the most used courses in Deep RL. About the team: - [Omar Sanseviero](https://twitter.com/osanseviero) is a Machine Learning Engineer at Hugging Face where he works in the intersection of ML, Community and Open Source. Previously, Omar worked as a Software Engineer at Google in the teams of Assistant and TensorFlow Graphics. He is from Peru and likes llamas ๐Ÿฆ™. - [ Sayak Paul](https://twitter.com/RisingSayak) is a Developer Advocate Engineer at Hugging Face. He's interested in the area of representation learning (self-supervision, semi-supervision, model robustness). And he loves watching crime and action thrillers ๐Ÿ”ช. ## What are the challenges in this course? [[challenges]] In this new version of the course, you have two types of challenges: - [A leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) to compare your agent's performance to other classmates'. - [AI vs. AI challenges](https://huggingface.co/learn/deep-rl-course/unit7/introduction?fw=pt) where you can train your agent and compete against other classmates' agents. ## I still have questions [[questions]] Please ask your question in our [discord server #rl-discussions.](https://discord.gg/ydHrjt3WP5) --- ## File: units/en/unit0/setup.mdx # Setup [[setup]] After all this information, it's time to get started. We're going to do two things: 1. **Create your Hugging Face account** if it's not already done 2. **Sign up to Discord and introduce yourself** (don't be shy ๐Ÿค—) ### Let's create my Hugging Face account (If it's not already done) create an account to HF [here](https://huggingface.co/join) ### Let's join our Discord server You can now sign up for our Discord Server. This is the place where you **can chat with the community and with us, create and join study groups to grow with each other and more** ๐Ÿ‘‰๐Ÿป Join our discord server [here.](https://discord.gg/ydHrjt3WP5) When you join, remember to introduce yourself in #introduce-yourself and sign-up for reinforcement channels in #channels-and-roles. We have multiple RL-related channels: - `rl-announcements`: where we give the latest information about the course. - `rl-discussions`: where you can chat about RL and share information. - `rl-study-group`: where you can create and join study groups. - `rl-i-made-this`: where you can share your projects and models. If this is your first time using Discord, we wrote a Discord 101 to get the best practices. Check the next section. Congratulations! **You've just finished the on-boarding**. You're now ready to start to learn Deep Reinforcement Learning. Have fun! ### Keep Learning, stay awesome ๐Ÿค— --- ## File: units/en/unit1/additional-readings.mdx # Additional Readings [[additional-readings]] These are **optional readings** if you want to go deeper. ## Deep Reinforcement Learning [[deep-rl]] - [Reinforcement Learning: An Introduction, Richard Sutton and Andrew G. Barto Chapter 1, 2 and 3](http://incompleteideas.net/book/RLbook2020.pdf) - [Foundations of Deep RL Series, L1 MDPs, Exact Solution Methods, Max-ent RL by Pieter Abbeel](https://youtu.be/2GwBez0D20A) - [Spinning Up RL by OpenAI Part 1: Key concepts of RL](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html) ## Gym [[gym]] - [Getting Started With OpenAI Gym: The Basic Building Blocks](https://blog.paperspace.com/getting-started-with-openai-gym/) - [Make your own Gym custom environment](https://www.gymlibrary.dev/content/environment_creation/) --- ## File: units/en/unit1/conclusion.mdx # Conclusion [[conclusion]] Congrats on finishing this unit! **That was the biggest one**, and there was a lot of information. And congrats on finishing the tutorial. Youโ€™ve just trained your first Deep RL agents and shared them with the community! ๐Ÿฅณ It's **normal if you still feel confused by some of these elements**. This was the same for me and for all people who studied RL. **Take time to really grasp the material** before continuing. Itโ€™s important to master these elements and have a solid foundation before entering the fun part. Naturally, during the course, weโ€™re going to use and explain these terms again, but itโ€™s better to understand them before diving into the next units. In the next (bonus) unit, weโ€™re going to reinforce what we just learned by **training Huggy the Dog to fetch a stick**. You will then be able to play with him ๐Ÿค—. Finally, we would love **to hear what you think of the course and how we can improve it**. If you have some feedback then, please ๐Ÿ‘‰ [fill this form](https://forms.gle/BzKXWzLAGZESGNaE9) ### Keep Learning, stay awesome ๐Ÿค— --- ## File: units/en/unit1/deep-rl.mdx # The โ€œDeepโ€ in Reinforcement Learning [[deep-rl]] What we've talked about so far is Reinforcement Learning. But where does the "Deep" come into play? Deep Reinforcement Learning introducesย **deep neural networks to solve Reinforcement Learning problems**ย โ€” hence the name โ€œdeepโ€. For instance, in the next unit, weโ€™ll learn about two value-based algorithms:ย Q-Learningย (classic Reinforcement Learning) and thenย Deep Q-Learning. Youโ€™ll see the difference is that, in the first approach,ย **we use a traditional algorithm**ย to create a Q table that helps us find what action to take for each state. In the second approach,ย **we will use a Neural Network**ย (to approximate the Q value).
Schema inspired by the Q learning notebook by Udacity
If you are not familiar with Deep Learning you should definitely watch [the FastAI Practical Deep Learning for Coders](https://course.fast.ai) (Free). --- ## File: units/en/unit1/exp-exp-tradeoff.mdx # The Exploration/Exploitation trade-off [[exp-exp-tradeoff]] Finally, before looking at the different methods to solve Reinforcement Learning problems, we must cover one more very important topic:ย *the exploration/exploitation trade-off.* - *Exploration* is exploring the environment by trying random actions in order to **find more information about the environment.** - *Exploitation* is **exploiting known information to maximize the reward.** Remember, the goal of our RL agent is to maximize the expected cumulative reward. However,ย **we can fall into a common trap**. Letโ€™s take an example: In this game, our mouse can have anย **infinite amount of small cheese**ย (+1 each). But at the top of the maze, there is a gigantic sum of cheese (+1000). However, if we only focus on exploitation, our agent will never reach the gigantic sum of cheese. Instead, it will only exploitย **the nearest source of rewards,**ย even if this source is small (exploitation). But if our agent does a little bit of exploration, it canย **discover the big reward**ย (the pile of big cheese). This is what we call theย exploration/exploitation trade-off. We need to balance how much weย **explore the environment**ย and how much weย **exploit what we know about the environment.** Therefore, we must **define a rule that helps to handle this trade-off**. Weโ€™ll see the different ways to handle it in the future units. If itโ€™s still confusing, **think of a real problem: the choice of picking a restaurant:**
Source: [ Berkley AI Course](https://inst.eecs.berkeley.edu/~cs188/sp20/assets/lecture/lec15_6up.pdf)
- *Exploitation*: You go to the same one that you know is good every day and **take the risk to miss another better restaurant.** - *Exploration*: Try restaurants you never went to before, with the risk of having a bad experience **but the probable opportunity of a fantastic experience.** To recap: --- ## File: units/en/unit1/glossary.mdx # Glossary [[glossary]] This is a community-created glossary. Contributions are welcome! ### Agent An agent learns to **make decisions by trial and error, with rewards and punishments from the surroundings**. ### Environment An environment is a simulated world **where an agent can learn by interacting with it**. ### Markov Property It implies that the action taken by our agent is **conditional solely on the present state and independent of the past states and actions**. ### Observations/State - **State**: Complete description of the state of the world. - **Observation**: Partial description of the state of the environment/world. ### Actions - **Discrete Actions**: Finite number of actions, such as left, right, up, and down. - **Continuous Actions**: Infinite possibility of actions; for example, in the case of self-driving cars, the driving scenario has an infinite possibility of actions occurring. ### Rewards and Discounting - **Rewards**: Fundamental factor in RL. Tells the agent whether the action taken is good/bad. - RL algorithms are focused on maximizing the **cumulative reward**. - **Reward Hypothesis**: RL problems can be formulated as a maximisation of (cumulative) return. - **Discounting** is performed because rewards obtained at the start are more likely to happen as they are more predictable than long-term rewards. ### Tasks - **Episodic**: Has a starting point and an ending point. - **Continuous**: Has a starting point but no ending point. ### Exploration v/s Exploitation Trade-Off - **Exploration**: It's all about exploring the environment by trying random actions and receiving feedback/returns/rewards from the environment. - **Exploitation**: It's about exploiting what we know about the environment to gain maximum rewards. - **Exploration-Exploitation Trade-Off**: It balances how much we want to **explore** the environment and how much we want to **exploit** what we know about the environment. ### Policy - **Policy**: It is called the agent's brain. It tells us what action to take, given the state. - **Optimal Policy**: Policy that **maximizes** the **expected return** when an agent acts according to it. It is learned through *training*. ### Policy-based Methods: - An approach to solving RL problems. - In this method, the Policy is learned directly. - Will map each state to the best corresponding action at that state. Or a probability distribution over the set of possible actions at that state. ### Value-based Methods: - Another approach to solving RL problems. - Here, instead of training a policy, we train a **value function** that maps each state to the expected value of being in that state. Contributions are welcome ๐Ÿค— If you want to improve the course, you can [open a Pull Request.](https://github.com/huggingface/deep-rl-class/pulls) This glossary was made possible thanks to: - [@lucifermorningstar1305](https://github.com/lucifermorningstar1305) - [@daspartho](https://github.com/daspartho) - [@misza222](https://github.com/misza222) --- ## File: units/en/unit1/hands-on.mdx # Train your first Deep Reinforcement Learning Agent ๐Ÿค– [[hands-on]] Now that you've studied the bases of Reinforcement Learning, youโ€™re ready to train your first agent and share it with the community through the Hub ๐Ÿ”ฅ: A Lunar Lander agent that will learn to land correctly on the Moon ๐ŸŒ• And finally, you'll **upload this trained agent to the Hugging Face Hub ๐Ÿค—, a free, open platform where people can share ML models, datasets, and demos.** Thanks to our [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard), you'll be able to compare your results with other classmates and exchange the best practices to improve your agent's scores. Who will win the challenge for Unit 1 ๐Ÿ†? To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push your trained model to the Hub and **get a result of >= 200**. To find your result, go to the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) and find your model, **the result = mean_reward - std of reward** **If you don't find your model, go to the bottom of the page and click on the refresh button.** For more information about the certification process, check this section ๐Ÿ‘‰ https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process And you can check your progress here ๐Ÿ‘‰ https://huggingface.co/spaces/ThomasSimonini/Check-my-progress-Deep-RL-Course So let's get started! ๐Ÿš€ **To start the hands-on click on Open In Colab button** ๐Ÿ‘‡ : [](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/master/notebooks/unit1/unit1.ipynb) We strongly **recommend students use Google Colab for the hands-on exercises** instead of running them on their personal computers. By using Google Colab, **you can focus on learning and experimenting without worrying about the technical aspects** of setting up your environments. # Unit 1: Train your first Deep Reinforcement Learning Agent ๐Ÿค– In this notebook, you'll train your **first Deep Reinforcement Learning agent** a Lunar Lander agent that will learn to **land correctly on the Moon ๐ŸŒ•**. Using [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) a Deep Reinforcement Learning library, share them with the community, and experiment with different configurations ### The environment ๐ŸŽฎ - [LunarLander-v2](https://gymnasium.farama.org/environments/box2d/lunar_lander/) ### The library used ๐Ÿ“š - [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues). ## Objectives of this notebook ๐Ÿ† At the end of the notebook, you will: - Be able to use **Gymnasium**, the environment library. - Be able to use **Stable-Baselines3**, the deep reinforcement learning library. - Be able to **push your trained agent to the Hub** with a nice video replay and an evaluation score ๐Ÿ”ฅ. ## This notebook is from Deep Reinforcement Learning Course In this free course, you will: - ๐Ÿ“– Study Deep Reinforcement Learning in **theory and practice**. - ๐Ÿง‘โ€๐Ÿ’ป Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0. - ๐Ÿค– Train **agents in unique environments** - ๐ŸŽ“ **Earn a certificate of completion** by completing 80% of the assignments. And more! Check ๐Ÿ“š the syllabus ๐Ÿ‘‰ https://simoninithomas.github.io/deep-rl-course Donโ€™t forget to **[sign up to the course](http://eepurl.com/ic5ZUD)** (we are collecting your email to be able toย **send you the links when each Unit is published and give you information about the challenges and updates).** The best way to keep in touch and ask questions is **to join our discord server** to exchange with the community and with us ๐Ÿ‘‰๐Ÿป https://discord.gg/ydHrjt3WP5 ## Prerequisites ๐Ÿ—๏ธ Before diving into the notebook, you need to: ๐Ÿ”ฒ ๐Ÿ“ **[Read Unit 0](https://huggingface.co/deep-rl-course/unit0/introduction)** that gives you all the **information about the course and helps you to onboard** ๐Ÿค— ๐Ÿ”ฒ ๐Ÿ“š **Develop an understanding of the foundations of Reinforcement learning** (MC, TD, Rewards hypothesis...) by [reading Unit 1](https://huggingface.co/deep-rl-course/unit1/introduction). ## A small recap of Deep Reinforcement Learning ๐Ÿ“š Let's do a small recap on what we learned in the first Unit: - Reinforcement Learning is a **computational approach to learning from actions**. We build an agent that learns from the environment by **interacting with it through trial and error** and receiving rewards (negative or positive) as feedback. - The goal of any RL agent is to **maximize its expected cumulative reward** (also called expected return) because RL is based on the _reward hypothesis_, which is that all goals can be described as the maximization of an expected cumulative reward. - The RL process is a **loop that outputs a sequence of state, action, reward, and next state**. - To calculate the expected cumulative reward (expected return), **we discount the rewards**: the rewards that come sooner (at the beginning of the game) are more probable to happen since they are more predictable than the long-term future reward. - To solve an RL problem, you want to **find an optimal policy**; the policy is the "brain" of your AI that will tell us what action to take given a state. The optimal one is the one that gives you the actions that max the expected return. There are **two** ways to find your optimal policy: - By **training your policy directly**: policy-based methods. - By **training a value function** that tells us the expected return the agent will get at each state and use this function to define our policy: value-based methods. - Finally, we spoke about Deep RL because **we introduce deep neural networks to estimate the action to take (policy-based) or to estimate the value of a state (value-based) hence the name "deep."** # Let's train our first Deep Reinforcement Learning agent and upload it to the Hub ๐Ÿš€ ## Get a certificate ๐ŸŽ“ To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push your trained model to the Hub and **get a result of >= 200**. To find your result, go to the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) and find your model, **the result = mean_reward - std of reward** For more information about the certification process, check this section ๐Ÿ‘‰ https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process ## Set the GPU ๐Ÿ’ช - To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type` - `Hardware Accelerator > GPU` ## Install dependencies and create a virtual screen ๐Ÿ”ฝ The first step is to install the dependencies, weโ€™ll install multiple ones. - `gymnasium[box2d]`: Contains the LunarLander-v2 environment ๐ŸŒ› - `stable-baselines3[extra]`: The deep reinforcement learning library. - `huggingface_sb3`: Additional code for Stable-baselines3 to load and upload models from the Hugging Face ๐Ÿค— Hub. To make things easier, we created a script to install all these dependencies. ```bash apt install swig cmake ``` ```bash pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit1/requirements-unit1.txt ``` During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). Hence the following cell will install virtual screen libraries and create and run a virtual screen ๐Ÿ–ฅ ```bash sudo apt-get update apt install python3-opengl apt install ffmpeg apt install xvfb pip3 install pyvirtualdisplay ``` To make sure the new installed libraries are used, **sometimes it's required to restart the notebook runtime**. The next cell will force the **runtime to crash, so you'll need to connect again and run the code starting from here**. Thanks to this trick, **we will be able to run our virtual screen.** ```python import os os.kill(os.getpid(), 9) ``` ```python # Virtual display from pyvirtualdisplay import Display virtual_display = Display(visible=0, size=(1400, 900)) virtual_display.start() ``` ## Import the packages ๐Ÿ“ฆ One additional library we import is huggingface_hub **to be able to upload and download trained models from the hub**. The Hugging Face Hub ๐Ÿค— works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easily collaborate with others. You can see here all the Deep reinforcement Learning models available here๐Ÿ‘‰ https://huggingface.co/models?pipeline_tag=reinforcement-learning&sort=downloads ```python import gymnasium from huggingface_sb3 import load_from_hub, package_to_hub from huggingface_hub import ( notebook_login, ) # To log to our Hugging Face account to be able to upload models to the Hub. from stable_baselines3 import PPO from stable_baselines3.common.env_util import make_vec_env from stable_baselines3.common.evaluation import evaluate_policy from stable_baselines3.common.monitor import Monitor ``` ## Understand Gymnasium and how it works ๐Ÿค– ๐Ÿ‹ The library containing our environment is called Gymnasium. **You'll use Gymnasium a lot in Deep Reinforcement Learning.** Gymnasium is the **new version of Gym library** [maintained by the Farama Foundation](https://farama.org/). The Gymnasium library provides two things: - An interface that allows you to **create RL environments**. - A **collection of environments** (gym-control, atari, box2D...). Let's look at an example, but first let's recall the RL loop. At each step: - Our Agent receivesย a **state (S0)**ย from theย **Environment**ย โ€” we receive the first frame of our game (Environment). - Based on thatย **state (S0),**ย the Agent takes anย **action (A0)**ย โ€” our Agent will move to the right. - The environment transitions to aย **new**ย **state (S1)**ย โ€” new frame. - The environment gives someย **reward (R1)**ย to the Agent โ€” weโ€™re not deadย *(Positive Reward +1)*. With Gymnasium: 1๏ธโƒฃ We create our environment using `gymnasium.make()` 2๏ธโƒฃ We reset the environment to its initial state with `observation = env.reset()` At each step: 3๏ธโƒฃ Get an action using our model (in our example we take a random action) 4๏ธโƒฃ Using `env.step(action)`, we perform this action in the environment and get - `observation`: The new state (st+1) - `reward`: The reward we get after executing the action - `terminated`: Indicates if the episode terminated (agent reach the terminal state) - `truncated`: Introduced with this new version, it indicates a timelimit or if an agent go out of bounds of the environment for instance. - `info`: A dictionary that provides additional information (depends on the environment). For more explanations check this ๐Ÿ‘‰ https://gymnasium.farama.org/api/env/#gymnasium.Env.step If the episode is terminated: - We reset the environment to its initial state with `observation = env.reset()` **Let's look at an example!** Make sure to read the code ```python import gymnasium as gym # First, we create our environment called LunarLander-v2 env = gym.make("LunarLander-v2") # Then we reset this environment observation, info = env.reset() for _ in range(20): # Take a random action action = env.action_space.sample() print("Action taken:", action) # Do this action in the environment and get # next_state, reward, terminated, truncated and info observation, reward, terminated, truncated, info = env.step(action) # If the game is terminated (in our case we land, crashed) or truncated (timeout) if terminated or truncated: # Reset the environment print("Environment is reset") observation, info = env.reset() env.close() ``` ## Create the LunarLander environment ๐ŸŒ› and understand how it works ### The environment ๐ŸŽฎ In this first tutorial, weโ€™re going to train our agent, a [Lunar Lander](https://gymnasium.farama.org/environments/box2d/lunar_lander/), **to land correctly on the moon**. To do that, the agent needs to learn **to adapt its speed and position (horizontal, vertical, and angular) to land correctly.** --- ๐Ÿ’ก A good habit when you start to use an environment is to check its documentation ๐Ÿ‘‰ https://gymnasium.farama.org/environments/box2d/lunar_lander/ --- Let's see what the Environment looks like: ```python # We create our environment with gym.make("") env = gym.make("LunarLander-v2") env.reset() print("_____OBSERVATION SPACE_____ \n") print("Observation Space Shape", env.observation_space.shape) print("Sample observation", env.observation_space.sample()) # Get a random observation ``` We see with `Observation Space Shape (8,)` that the observation is a vector of size 8, where each value contains different information about the lander: - Horizontal pad coordinate (x) - Vertical pad coordinate (y) - Horizontal speed (x) - Vertical speed (y) - Angle - Angular speed - If the left leg contact point has touched the land (boolean) - If the right leg contact point has touched the land (boolean) ```python print("\n _____ACTION SPACE_____ \n") print("Action Space Shape", env.action_space.n) print("Action Space Sample", env.action_space.sample()) # Take a random action ``` The action space (the set of possible actions the agent can take) is discrete with 4 actions available ๐ŸŽฎ: - Action 0: Do nothing, - Action 1: Fire left orientation engine, - Action 2: Fire the main engine, - Action 3: Fire right orientation engine. Reward function (the function that will give a reward at each timestep) ๐Ÿ’ฐ: After every step a reward is granted. The total reward of an episode is the **sum of the rewards for all the steps within that episode**. For each step, the reward: - Is increased/decreased the closer/further the lander is to the landing pad. - Is increased/decreased the slower/faster the lander is moving. - Is decreased the more the lander is tilted (angle not horizontal). - Is increased by 10 points for each leg that is in contact with the ground. - Is decreased by 0.03 points each frame a side engine is firing. - Is decreased by 0.3 points each frame the main engine is firing. The episode receive an **additional reward of -100 or +100 points for crashing or landing safely respectively.** An episode is **considered a solution if it scores at least 200 points.** #### Vectorized Environment - We create a vectorized environment (a method for stacking multiple independent environments into a single environment) of 16 environments, this way, **we'll have more diverse experiences during the training.** ```python # Create the environment env = make_vec_env("LunarLander-v2", n_envs=16) ``` ## Create the Model ๐Ÿค– - We have studied our environment and we understood the problem: **being able to land the Lunar Lander to the Landing Pad correctly by controlling left, right and main orientation engine**. Now let's build the algorithm we're going to use to solve this Problem ๐Ÿš€. - To do so, we're going to use our first Deep RL library, [Stable Baselines3 (SB3)](https://stable-baselines3.readthedocs.io/en/master/). - SB3 is a set of **reliable implementations of reinforcement learning algorithms in PyTorch**. --- ๐Ÿ’ก A good habit when using a new library is to dive first on the documentation: https://stable-baselines3.readthedocs.io/en/master/ and then try some tutorials. ---- To solve this problem, we're going to use SB3 **PPO**. [PPO (aka Proximal Policy Optimization) is one of the SOTA (state of the art) Deep Reinforcement Learning algorithms that you'll study during this course](https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#example%5D). PPO is a combination of: - *Value-based reinforcement learning method*: learning an action-value function that will tell us the **most valuable action to take given a state and action**. - *Policy-based reinforcement learning method*: learning a policy that will **give us a probability distribution over actions**. Stable-Baselines3 is easy to set up: 1๏ธโƒฃ You **create your environment** (in our case it was done above) 2๏ธโƒฃ You define the **model you want to use and instantiate this model** `model = PPO("MlpPolicy")` 3๏ธโƒฃ You **train the agent** with `model.learn` and define the number of training timesteps ``` # Create environment env = gym.make('LunarLander-v2') # Instantiate the agent model = PPO('MlpPolicy', env, verbose=1) # Train the agent model.learn(total_timesteps=int(2e5)) ``` ```python # TODO: Define a PPO MlpPolicy architecture # We use MultiLayerPerceptron (MLPPolicy) because the input is a vector, # if we had frames as input we would use CnnPolicy model = ``` #### Solution ```python # SOLUTION # We added some parameters to accelerate the training model = PPO( policy="MlpPolicy", env=env, n_steps=1024, batch_size=64, n_epochs=4, gamma=0.999, gae_lambda=0.98, ent_coef=0.01, verbose=1, ) ``` ## Train the PPO agent ๐Ÿƒ - Let's train our agent for 1,000,000 timesteps, don't forget to use GPU on Colab. It will take approximately ~20min, but you can use fewer timesteps if you just want to try it out. - During the training, take a โ˜• break you deserved it ๐Ÿค— ```python # TODO: Train it for 1,000,000 timesteps # TODO: Specify file name for model and save the model to file model_name = "ppo-LunarLander-v2" ``` #### Solution ```python # SOLUTION # Train it for 1,000,000 timesteps model.learn(total_timesteps=1000000) # Save the model model_name = "ppo-LunarLander-v2" model.save(model_name) ``` ## Evaluate the agent ๐Ÿ“ˆ - Remember to wrap the environment in a [Monitor](https://stable-baselines3.readthedocs.io/en/master/common/monitor.html). - Now that our Lunar Lander agent is trained ๐Ÿš€, we need to **check its performance**. - Stable-Baselines3 provides a method to do that: `evaluate_policy`. - To fill that part you need to [check the documentation](https://stable-baselines3.readthedocs.io/en/master/guide/examples.html#basic-usage-training-saving-loading) - In the next step, we'll see **how to automatically evaluate and share your agent to compete in a leaderboard, but for now let's do it ourselves** ๐Ÿ’ก When you evaluate your agent, you should not use your training environment but create an evaluation environment. ```python # TODO: Evaluate the agent # Create a new environment for evaluation eval_env = # Evaluate the model with 10 evaluation episodes and deterministic=True mean_reward, std_reward = # Print the results ``` #### Solution ```python # @title eval_env = Monitor(gym.make("LunarLander-v2")) mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True) print(f"mean_reward={mean_reward:.2f} +/- {std_reward}") ``` - In my case, I got a mean reward of `200.20 +/- 20.80` after training for 1 million steps, which means that our lunar lander agent is ready to land on the moon ๐ŸŒ›๐Ÿฅณ. ## Publish our trained model on the Hub ๐Ÿ”ฅ Now that we saw we got good results after the training, we can publish our trained model on the hub ๐Ÿค— with one line of code. ๐Ÿ“š The libraries documentation ๐Ÿ‘‰ https://github.com/huggingface/huggingface_sb3/tree/main#hugging-face--x-stable-baselines3-v20 Here's an example of a Model Card (with Space Invaders): By using `package_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**. This way: - You can **showcase our work** ๐Ÿ”ฅ - You can **visualize your agent playing** ๐Ÿ‘€ - You can **share with the community an agent that others can use** ๐Ÿ’พ - You can **access a leaderboard ๐Ÿ† to see how well your agent is performing compared to your classmates** ๐Ÿ‘‰ https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard To be able to share your model with the community there are three more steps to follow: 1๏ธโƒฃ (If it's not already done) create an account on Hugging Face โžก https://huggingface.co/join 2๏ธโƒฃ Sign in and then, you need to store your authentication token from the Hugging Face website. - Create a new token (https://huggingface.co/settings/tokens) **with write role** - Copy the token - Run the cell below and paste the token ```python notebook_login() !git config --global credential.helper store ``` If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login` 3๏ธโƒฃ We're now ready to push our trained agent to the ๐Ÿค— Hub ๐Ÿ”ฅ using `package_to_hub()` function Let's fill the `package_to_hub` function: - `model`: our trained model. - `model_name`: the name of the trained model that we defined in `model_save` - `model_architecture`: the model architecture we used, in our case PPO - `env_id`: the name of the environment, in our case `LunarLander-v2` - `eval_env`: the evaluation environment defined in eval_env - `repo_id`: the name of the Hugging Face Hub Repository that will be created/updated `(repo_id = {username}/{repo_name})` ๐Ÿ’ก **A good name is `{username}/{model_architecture}-{env_id}` ** - `commit_message`: message of the commit ```python import gymnasium as gym from stable_baselines3.common.vec_env import DummyVecEnv from stable_baselines3.common.env_util import make_vec_env from huggingface_sb3 import package_to_hub ## TODO: Define a repo_id ## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2 repo_id = # TODO: Define the name of the environment env_id = # Create the evaluation env and set the render_mode="rgb_array" eval_env = DummyVecEnv([lambda: gym.make(env_id, render_mode="rgb_array")]) # TODO: Define the model architecture we used model_architecture = "" ## TODO: Define the commit message commit_message = "" # method save, evaluate, generate a model card and record a replay video of your agent before pushing the repo to the hub package_to_hub(model=model, # Our trained model model_name=model_name, # The name of our trained model model_architecture=model_architecture, # The model architecture we used: in our case PPO env_id=env_id, # Name of the environment eval_env=eval_env, # Evaluation Environment repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2 commit_message=commit_message) ``` #### Solution ```python import gymnasium as gym from stable_baselines3 import PPO from stable_baselines3.common.vec_env import DummyVecEnv from stable_baselines3.common.env_util import make_vec_env from huggingface_sb3 import package_to_hub # PLACE the variables you've just defined two cells above # Define the name of the environment env_id = "LunarLander-v2" # TODO: Define the model architecture we used model_architecture = "PPO" ## Define a repo_id ## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2 ## CHANGE WITH YOUR REPO ID repo_id = "ThomasSimonini/ppo-LunarLander-v2" # Change with your repo id, you can't push with mine ๐Ÿ˜„ ## Define the commit message commit_message = "Upload PPO LunarLander-v2 trained agent" # Create the evaluation env and set the render_mode="rgb_array" eval_env = DummyVecEnv([lambda: Monitor(gym.make(env_id, render_mode="rgb_array"))]) # PLACE the package_to_hub function you've just filled here package_to_hub( model=model, # Our trained model model_name=model_name, # The name of our trained model model_architecture=model_architecture, # The model architecture we used: in our case PPO env_id=env_id, # Name of the environment eval_env=eval_env, # Evaluation Environment repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2 commit_message=commit_message, ) ``` Congrats ๐Ÿฅณ you've just trained and uploaded your first Deep Reinforcement Learning agent. The script above should have displayed a link to a model repository such as https://huggingface.co/osanseviero/test_sb3. When you go to this link, you can: * See a video preview of your agent at the right. * Click "Files and versions" to see all the files in the repository. * Click "Use in stable-baselines3" to get a code snippet that shows how to load the model. * A model card (`README.md` file) which gives a description of the model Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent. Compare the results of your LunarLander-v2 with your classmates using the leaderboard ๐Ÿ† ๐Ÿ‘‰ https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard ## Load a saved LunarLander model from the Hub ๐Ÿค— Thanks to [ironbar](https://github.com/ironbar) for the contribution. Loading a saved model from the Hub is really easy. You go to https://huggingface.co/models?library=stable-baselines3 to see the list of all the Stable-baselines3 saved models. 1. You select one and copy its repo_id 2. Then we just need to use load_from_hub with: - The repo_id - The filename: the saved model inside the repo and its extension (*.zip) Because the model I download from the Hub was trained with Gym (the former version of Gymnasium) we need to install shimmy a API conversion tool that will help us to run the environment correctly. Shimmy Documentation: https://github.com/Farama-Foundation/Shimmy ```python !pip install shimmy ``` ```python from huggingface_sb3 import load_from_hub repo_id = "Classroom-workshop/assignment2-omar" # The repo_id filename = "ppo-LunarLander-v2.zip" # The model filename.zip # When the model was trained on Python 3.8 the pickle protocol is 5 # But Python 3.6, 3.7 use protocol 4 # In order to get compatibility we need to: # 1. Install pickle5 (we done it at the beginning of the colab) # 2. Create a custom empty object we pass as parameter to PPO.load() custom_objects = { "learning_rate": 0.0, "lr_schedule": lambda _: 0.0, "clip_range": lambda _: 0.0, } checkpoint = load_from_hub(repo_id, filename) model = PPO.load(checkpoint, custom_objects=custom_objects, print_system_info=True) ``` Let's evaluate this agent: ```python # @title eval_env = Monitor(gym.make("LunarLander-v2")) mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True) print(f"mean_reward={mean_reward:.2f} +/- {std_reward}") ``` ## Some additional challenges ๐Ÿ† The best way to learn **is to try things by your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results! In the [Leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top? Here are some ideas to achieve so: * Train more steps * Try different hyperparameters for `PPO`. You can see them at https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#parameters. * Check the [Stable-Baselines3 documentation](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) and try another model such as DQN. * **Push your new trained model** on the Hub ๐Ÿ”ฅ **Compare the results of your LunarLander-v2 with your classmates** using the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) ๐Ÿ† Is moon landing too boring for you? Try to **change the environment**, why not use MountainCar-v0, CartPole-v1 or CarRacing-v0? Check how they work [using the gym documentation](https://www.gymlibrary.dev/) and have fun ๐ŸŽ‰. ________________________________________________________________________ Congrats on finishing this chapter! That was the biggest one, **and there was a lot of information.** If youโ€™re still feel confused with all these elements...it's totally normal! **This was the same for me and for all people who studied RL.** Take time to really **grasp the material before continuing and try the additional challenges**. Itโ€™s important to master these elements and have a solid foundations. Naturally, during the course, weโ€™re going to dive deeper into these concepts but **itโ€™s better to have a good understanding of them now before diving into the next chapters.** Next time, in the bonus unit 1, you'll train Huggy the Dog to fetch the stick. ## Keep learning, stay awesome ๐Ÿค— --- ## File: units/en/unit1/introduction.mdx # Introduction to Deep Reinforcement Learning [[introduction-to-deep-reinforcement-learning]] Welcome to the most fascinating topic in Artificial Intelligence:ย **Deep Reinforcement Learning.** Deep RL is a type of Machine Learning where an agent learnsย **how to behave**ย in an environmentย **by performing actions**ย andย **seeing the results.** In this first unit,ย **you'll learn the foundations of Deep Reinforcement Learning.** Then, you'll **train your Deep Reinforcement Learning agent, a lunar lander to land correctly on the Moon** using [ Stable-Baselines3 ](https://stable-baselines3.readthedocs.io/en/master/), a Deep Reinforcement Learning library. And finally, you'll **upload this trained agent to the Hugging Face Hub ๐Ÿค—, a free, open platform where people can share ML models, datasets, and demos.** It's essential **to master these elements**ย before diving into implementing Deep Reinforcement Learning agents. The goal of this chapter is to give you solid foundations. After this unit, in a bonus unit, you'll be **able to train Huggy the Dog ๐Ÿถ to fetch the stick and play with him ๐Ÿค—**.