Practical_RL Documentation

A course in reinforcement learning in the wild

6,596 stars Jupyter Notebook 16 files · ~9,699 tokens #course-materials#deep-learning#deep-reinforcement-learning#git-course#hacktoberfest#keras#mooc#pytorch
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy Practical_RL on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name practical_rl -p 8888:8888 --restart always yandexdataschool/practical_rl:latest

Technical Documentation: yandexdataschool/Practical_RL

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

Practical_RL

An open course on reinforcement learning in the wild.
Taught on-campus at HSE and YSDA and maintained to be friendly to online students (both english and russian).

Manifesto:

  • Optimize for the curious. For all the materials that aren’t covered in detail there are links to more information and related materials (D.Silver/Sutton/blogs/whatever). Assignments will have bonus sections if you want to dig deeper.
  • Practicality first. Everything essential to solving reinforcement learning problems is worth mentioning. We won't shun away from covering tricks and heuristics. For every major idea there should be a lab that makes you to “feel” it on a practical problem.
  • Git-course. Know a way to make the course better? Noticed a typo in a formula? Found a useful link? Made the code more readable? Made a version for alternative framework? You're awesome! Pull-request it!

Course info

Additional materials

Syllabus

The syllabus is approximate: the lectures may occur in a slightly different order and some topics may end up taking two weeks.

  • week01_intro Introduction

    • Lecture: RL problems around us. Decision processes. Stochastic optimization, Crossentropy method. Parameter space search vs action space search.
    • Seminar: Welcome into openai gym. Tabular CEM for Taxi-v0, deep CEM for box2d environments.
    • Homework description - see week1/README.md.

  • week02_value_based Value-based methods

    • Lecture: Discounted reward MDP. Value-based approach. Value iteration. Policy iteration. Discounted reward fails.
    • Seminar: Value iteration.
    • Homework description - see week2/README.md.

  • week03_model_free Model-free reinforcement learning

    • Lecture: Q-learning. SARSA. Off-policy Vs on-policy algorithms. N-step algorithms. TD(Lambda).
    • Seminar: Qlearning Vs SARSA Vs Expected Value SARSA
    • Homework description - see week3/README.md.

  • recap_deep_learning - deep learning recap

    • Lecture: Deep learning 101
    • Seminar: Intro to pytorch/tensorflow, simple image classification with convnets

  • week04_approx_rl Approximate (deep) RL

    • Lecture: Infinite/continuous state space. Value function approximation. Convergence conditions. Multiple agents trick; experience replay, target networks, double/dueling/bootstrap DQN, etc.
    • Seminar: Approximate Q-learning with experience replay. (CartPole, Atari)

  • week05_explore Exploration

    • Lecture: Contextual bandits. Thompson Sampling, UCB, bayesian UCB. Exploration in model-based RL, MCTS. "Deep" heuristics for exploration.
    • Seminar: bayesian exploration for contextual bandits. UCB for MCTS.

  • week06_policy_based Policy Gradient methods

    • Lecture: Motivation for policy-based, policy gradient, logderivative trick, REINFORCE/crossentropy method, variance reduction(baseline), advantage actor-critic (incl. GAE)
    • Seminar: REINFORCE, advantage actor-critic

  • week07_seq2seq Reinforcement Learning for Sequence Models

    • Lecture: Problems with sequential data. Recurrent neural networks. Backprop through time. Vanishing & exploding gradients. LSTM, GRU. Gradient clipping
    • Seminar: character-level RNN language model

  • week08_pomdp Partially Observed MDP

    • Lecture: POMDP intro. POMDP learning (agents with memory). POMDP planning (POMCP, etc)
    • Seminar: Deep kung-fu & doom with recurrent A3C and DRQN

  • week09_policy_II Advanced policy-based methods

    • Lecture: Trust region policy optimization. NPO/PPO. Deterministic policy gradient. DDPG
    • Seminar: Approximate TRPO for simple robot control.

  • week10_planning Model-based RL & Co

    • Lecture: Model-Based RL, Planning in General, Imitation Learning and Inverse Reinforcement Learning
    • Seminar: MCTS for toy tasks

  • yet_another_week Inverse RL and Imitation Learning

    • All that cool RL stuff that you won't learn from this course :)

Course staff

Course materials and teaching by: [unordered]

Contributions

Overview (`docker/README.md`)

To simplify installation process, you can deploy a container (~virtual machine) with all dependencies pre-installed.

tl;dr dockerhub url

Install Docker

We recommend you to use either native docker (recommended for linux) or kitematic(recommended for windows).

Below are the instructions for both approaches.

Kitematic

Find dmittov/practical_rl in the search menu. Download and launch the container.

Click on "web preview" screen in the top-right or go to settings, ports and find at which port your jupyter is located, usually 32***.

Native

docker run --rm -it -v /path/to/your/repo:/notebooks -p <local_port>:8888 dmittov/practical_rl:spring2020-cpu

For example,
docker run --rm -it -v /home/myuser/Documents/practical_rl:/notebooks -p 8888:8888 dmittov/practical_rl:spring2020-cpu

Then you can access your jupyter in a browser at localhost:<local_port>/?token=<token_you_see_in_container_logs>, e.g. localhost:8888/?token=ad1a5a0aab43efb47a9a805388fcf508d0b5f84a16e4542b&token=ad1a5a0aab43efb47a9a805388fcf508d0b5f84a16e4542b

GPU

docker run --rm -it -v /path/to/your/repo:/notebooks -p <local_port>:8888 --gpus all dmittov/practical_rl:spring2020-cuda-10.2

Manual

Build container

docker build -t practical_rl --build-arg device=cpu .

to build GPU version

docker build -t practical_rl --build-arg device=gpu .

Run it

$ docker run --rm -it -v <local_dir>:/notebooks -p <local_port>:8888 practical_rl

to run GPU version

$ docker run --rm -it -v <local_dir>:/notebooks -p <local_port>:8888 --gpus all practical_rl

examples:

$ docker run --rm -it -v `pwd`:/notebooks -p 8888:8888 practical_rl

Copy the token from console and run
http://localhost:8888/?token=

Overview (`week01_intro/README.md`)

Materials:

  • Lecture slides

  • Russian:

  • English:

    • [main] Video-intro by David Silver (english) - video
    • [main] Lecture by J Schulman with crossentropy method explained (english) - url
    • Optional lecture by David Silver (english) - video

More materials:

  • [recommended] - awesome openai post about evolution strategies - blog post, article
  • [recommended] - formal explanation of crossentropy method in general and for optimization
  • Deep learning course (if you want to learn in parallel) - https://github.com/yandexdataschool/HSE_deeplearning
  • Video on genetic algorithms (english) - video
  • Another guide to genetic algorithm (english) - video
  • About Differential evolution (english) - pdf
  • Video on Ant Colony Algorithm (english) - video
  • Longer video on Ant Colony Algorithm (english) - video
  • Longer video on corssentropy method (russian) - video

Practice assignment

Instant dive in: seminar_gymnasium_interface, crossentropy_method,
deep_crossentropy_method

  • Open seminar_gymnasium_interface.ipynb and follow instructions from there
  • After you're done there, proceed to crossentropy_method.ipynb
  • You can find homework and bonus assignment descriptions at the end of that notebook.
  • Note: so far it's enough to say pip install gymnasium on top of any data-science-stuffed python, but we'd appreciate if you gradually switch to full installation.

Overview (`week02_value_based/README.md`)

Materials

  • Lecture slides
  • Our videos: lecture seminar (russian)
  • [main] lecture by David Silver - url
  • Alternative lecture by Pieter Abbeel (english): part 1, part 2
  • Alternative lecture by John Schulmann (english): video
  • Definitive guide in policy/value iteration from Sutton: start from page 81 here.

Homework description:

The main assignment is seminar_vi.ipynb notebook in this week's folder. It has no requirements besides the most basic data science libraries (e.g. numpy) so you should be able to run it locally.

Note: if you have any difficulty using graphviz, just set has_graphviz=False.

Overview (`week03_model_free/README.md`)

__Lecture slides__ - [here](https://yadi.sk/i/9bPN0pxsR1-zIA)

Materials

  • Russian materials:
  • English materials:
    • Lecture by David Silver (english) - video part I, video part II
    • Alternative lecture by Pieter Abbeel (english) - video
    • Alternative lecture by John Schulmann (english) - video
    • Blog post on q-learning Vs SARSA - url

More materials

  • N-step temporal difference from Sutton's book - suttonbook chapter 7
  • Eligibility traces from Sutton's book - suttonbook chapter 12
  • Blog post on eligibility traces - url

Assignments

Just as usual, start with homework.ipynb For seminar, implement q-learning agent and test it on Taxi and CartPole with binarizer. And then, implement EV-SARSA agent, experience replay + bonus tasks for homework.

Overview (`week04_[recap]_deep_learning/README.md`)

Note: This week's materials cover the basics of neural nets and deep learning and teach you how to use auto-diff frameworks. If you're already fluent in Tensorflow or PyTorch, feel free to skip this week entirely.

Materials

Bonus materials

Practice

Colab URL (PyTorch)
From now on, we'll have two tracks: Tensorflow and PyTorch.

Please pick seminar_tensorflow.ipynb or seminar_pytorch.ipynb.

Note: in this and all following weeks you're only required to get through practice in one of the frameworks. Looking into other alternatives is great for self-education but never mandatory.

What to choose?

  • The simplest choice is PyTorch: it's basically ye olde numpy with automatic gradients and a lot of pre-implemented DL stuff... except all the functions have different names.

  • If you want to be familiar with production-related stuff from day 1, choose TensorFlow. It's much more convenient to deploy (to non-python or to mobiles). The catch is that all those conveniences become inconveniences once you want to write something simple in jupyter.

  • It's not like choosing house at Hogwarts, you'll be able to switch between frameworks easily once you master the underlying principles.

Overview (`week04_approx_rl/README.md`)

Materials

More materials

  • [recommended] How to actually do deep reinforcement learning by J. Schulman - pdf
  • [recommended] Spinning up - a massive repository of deep RL knowledge
  • [recommended] A table of collateral effects of reinforcement learning in games - table
  • DQN and modiffications - lecture by J. Schulman - video
    • interactive demos in your browser: demo1(karpathy), demo2(Hünermann)
  • Reinforcement learning architectures list - repo
  • Article on dueling DQN - arxiv
  • Article on double DQN - arxiv
  • Article on prioritized experience replay - arxiv
  • Article on Rainbow: Combining Improvements in Deep Reinforcement Learning - arxiv
  • Article on bootstrap DQN - pdf, summary
  • Article on asynchronuous methods in deep RL - arxiv
  • Successor representations for reinforcement learning - article, video
  • Video on asynchronuous methods (Mnih) - video
  • An overview of deep reinforcement learning - arxiv

DQN tutorials

  • [in pytorch] A great series starting from simple DQN to all the cool new stuff - url
  • A guide to deep RL from ~scratch (nervana blog) - url
  • Building deep q-network from ~scratch (blog) - url
  • Another guide guide to DQN from ~scratch (blog) - url

Practice

  • Seminar:
  • Homework (main):
  • Homework (debug):

From now on, we have two tracks, for pytorch and tensorflow. However, pytorch track is somewhat better supported by the course team. You can choose whichever track you want, but unless you're expertly familiar with your framework, we recommend you to start by completing the task in pytorch and only then reproduce your solution in your chosen framework.

Begin with seminar_<framework>.ipynb and then proceed with homework_<framework>.ipynb.

Note: you're not required to submit assignments in all three frameworks. Pick one and go with it. Maybe switch it occasionally if you want more challenge.

Overview (`week05_explore/README.md`)

Slides - [here](https://yadi.sk/i/H0zVBROe3TWWHz)

Exploration and exploitation

  • [main] David Silver lecture on exploration and expoitation - video
  • Alternative lecture by J. Schulman - video
  • Alternative lecture by N. de Freitas (with bayesian opt) - video
  • Our lectures (russian)
    • "mathematical" lecture (by Alexander Vorobev) '17 - slides, video
    • "practical" lecture '18 - video
    • Seminar - video

More materials

  • Gittins Index - the less heuristical approach to bandit exploration - article

  • "Deep" version: variational information maximizing exploration - video

    • Same topics in russian - video

  • Lecture covering intrinsically motivated reinforcement learning - video

    • Slides
    • Same topics in russian - video
    • Note: UCB-1 is not for bernoulli rewards, but for arbitrary r in [0,1], so you can just scale any reward to [0,1] to obtain a peace of mind. It's derived directly from Hoeffding's inequality.

  • Very interesting blog post written by Lilian Weng that summarises this week's materials: The Multi-Armed Bandit Problem and Its Solutions

Seminar

In this seminar, you'll be solving basic and contextual bandits with uncertainty-based exploration like Bayesian UCB and Thompson Sampling. You will also get acquainted with Bayesian Neural Networks.

Everything else is in the notebook :)

Overview (`week06_policy_based/README.md`)

Materials

More materials

  • Actually proving the policy gradient for discounted rewards - article

  • On variance of policy gradient and optimal baselines: article, another article

  • Learn Advatangeg Actor Critic with a comic

  • Generalizing log-derivative trick - url

  • Combining policy gradient and q-learning - arxiv

  • Variational perspective on reinforcement learning (from DeepBayes) - pdf

  • Adversarial review of policy gradient - blog

Run seminar notebook in Colab:

Run optional homework notebook in Colab:

Overview (`week07_[recap]_rnn/README.md`)

Materials

More materials

  • Seq2seq lecture - video
  • "Awesome rnn" entry point - repo
  • OpenAI research on sentiment analysis that sheds some light on what's inside LSTM language model.

Homework description

This week's practice gets you acquainted with basics of recurrent neural networks. For simplicity, we'll train them on character language modelling task. You'll need seminar_pytorch.ipynb or seminar_tf.ipynb.

Overview (`week07_seq2seq/README.md`)

Materials

Practice

As usual, go to practice_{your framework}.ipynb above and follow instructions from there. PyTorch, Tensorflow

More materials

Overview (`week08_pomdp/README.md`)

Materials

lecture slides

Links on all articles mentioned during the lecture could be found in "References" at the very end of the presentation slides. All other interesing links which contribute to the topic of POMDP are presented below

Basics

  • Our lecture and seminar (russian)
  • A Lecture on Basics by Pavel Shvechikov (english) Video
  • A lecture on basics by Andrew NG (english, LQ) - video
  • A lecture on basics by 5vision (russian) - video
  • [alternative] Chalkboard-style 2-part lecture by B. Ravindran. - part1, part2
  • [alternative] Yet another mini-lecture touching on POMDP by S.S. Baveja - video

POMDP Learning

Practice

The assignment is platform and framewerk independent, so choose the framework that suits you best, but pay attention on how many you will need to implement youself in case of nonstandart ones.

Overview (`week09_policy_II/README.md`)

Materials

This section covers some steroids for policy gradient methods, along with a cool general trick called

Practice

  • TRPO:

  • PPO:

More: Reinforcement learning in large/continuous action spaces

While you already know algorithms that will work with continuously many actions, it can't hurt to learn something more specialized.

  • Lecture by J. Schulman - video
  • Q-learning with normalized advantage functions - article, code1, code2
  • Deterministic policy gradient - article, post+code
  • Stochastic value gradient - article
  • Embedding large discrete action spaces for RL - article
  • Lecture by A. Seleznev, 5vision (russian) - video

Overview (`week10_planning/README.md`)

Assignments

Just as usual, start with seminar_MCTS.ipynb

Materials: planning

Supplementary materials

  • Integrating learning and planning (D. Silver) - video
  • Approximating the MCTS optimal actions - 5vision solution for deephack.RL, code by Mikhail Pavlov - repo
  • Alpha Go - explanation
  • Monte Carlo Planning in POMDP pdf

Overview (`yet_another_week/README.md`)

Our slides: inverse/imitation rl; multi-agent 101; multi-agent 102; hierarchical rl

In this week you can find several sections covering advanced topics in RL, along with less advanced topics that we couldn't squeeze into the main track

Other

  • Learning by imitation - video, assignment(berkeley cs294)
  • Inverse reinforcement learning
    • Lecture by Chelsea Finn - video
    • Udacity videos - video
  • Distributional RL - video
  • Learning World Models for RL - blog
  • Knowledge transfer in RL - video(berkeley cs294)
  • Hierarchical reinforcemnt learning
    • Cool article ( Fe__U__dal networks ) - arxiv
    • Short lecture by Roy Fox - video
  • Multi-Agent reinforcement learning
    • Lecture by Thoro Graepel - role of HRL in AI research - video
    • Lecture by Balaraman Ravindran - video

A list of lists

Overview (`yet_another_week/_resource/README.md`)

This is a utility folder to store images and other resources used in notebooks.

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#course-materials Browse course-materials catalog → #deep-learning Browse deep-learning catalog → #deep-reinforcement-learning Browse deep-reinforcement-learning catalog → #git-course Browse git-course catalog → #hacktoberfest Browse hacktoberfest catalog → #keras Browse keras catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →