# Technical Documentation: yandexdataschool/Practical_RL

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/yandexdataschool/Practical_RL)

### Overview (`README.md`)

# Practical_RL

An open course on reinforcement learning in the wild.
Taught on-campus at [HSE](https://cs.hse.ru) and [YSDA](https://yandexdataschool.com/)  and maintained to be friendly to online students (both english and russian).


#### Manifesto:
* __Optimize for the curious.__ For all the materials that aren’t covered in detail there are links to more information and related materials (D.Silver/Sutton/blogs/whatever). Assignments will have bonus sections if you want to dig deeper.
* __Practicality first.__ Everything essential to solving reinforcement learning problems is worth mentioning. We won't shun away from covering tricks and heuristics. For every major idea there should be a lab that makes you to “feel” it on a practical problem.
* __Git-course.__ Know a way to make the course better? Noticed a typo in a formula? Found a useful link? Made the code more readable? Made a version for alternative framework? You're awesome! [Pull-request](https://help.github.com/articles/about-pull-requests/) it!

[](https://github.com/yandexdataschool/Practical_RL/graphs/contributors)

# Course info

* __FAQ:__ [About the course](https://github.com/yandexdataschool/Practical_RL/wiki/Practical-RL), [Technical issues thread](https://github.com/yandexdataschool/Practical_RL/issues/1), [Lecture Slides](https://yadi.sk/d/loPpY45J3EAYfU), [Online Student Survival Guide](https://github.com/yandexdataschool/Practical_RL/wiki/Online-student's-survival-guide)

* Anonymous [feedback form](https://docs.google.com/forms/d/e/1FAIpQLSdurWw97Sm9xCyYwC8g3iB5EibITnoPJW2IkOVQYE_kcXPh6Q/viewform).

* Virtual course environment: 
    * [__Google Colab__](https://colab.research.google.com/) - set open -> github -> yandexdataschool/pracical_rl -> {branch name} and select any notebook you want.
    * [Installing dependencies](https://github.com/yandexdataschool/Practical_RL/issues/1) on your local machine (recommended).
    * Alternative: [Azure Notebooks](https://notebooks.azure.com/).


# Additional materials
* [RL reading group](https://github.com/yandexdataschool/Practical_RL/wiki/RL-reading-group)


# Syllabus

The syllabus is approximate: the lectures may occur in a slightly different order and some topics may end up taking two weeks.

* [__week01_intro__](./week01_intro) Introduction
  * Lecture: RL problems around us. Decision processes. Stochastic optimization, Crossentropy method. Parameter space search vs action space search.
  * Seminar: Welcome into openai gym. Tabular CEM for Taxi-v0, deep CEM for box2d environments.
  * Homework description - see week1/README.md. 

* [__week02_value_based__](./week02_value_based) Value-based methods
  * Lecture: Discounted reward MDP. Value-based approach. Value iteration. Policy iteration. Discounted reward fails.
  * Seminar: Value iteration.  
  * Homework description - see week2/README.md. 
  
* [__week03_model_free__](./week03_model_free) Model-free reinforcement learning
  * Lecture: Q-learning. SARSA. Off-policy Vs on-policy algorithms. N-step algorithms. TD(Lambda).
  * Seminar: Qlearning Vs SARSA Vs Expected Value SARSA
  * Homework description - see week3/README.md. 

* [__recap_deep_learning__](./week04_\[recap\]_deep_learning) - deep learning recap 
  * Lecture: Deep learning 101
  * Seminar: Intro to pytorch/tensorflow, simple image classification with convnets

* [__week04_approx_rl__](./week04_approx_rl) Approximate (deep) RL
  * Lecture: Infinite/continuous state space. Value function approximation. Convergence conditions. Multiple agents trick; experience replay, target networks, double/dueling/bootstrap DQN, etc.
  * Seminar:  Approximate Q-learning with experience replay. (CartPole, Atari)
  
* [__week05_explore__](./week05_explore) Exploration
  * Lecture: Contextual bandits. Thompson Sampling, UCB, bayesian UCB. Exploration in model-based RL, MCTS. "Deep" heuristics for exploration.
  * Seminar: bayesian exploration for contextual bandits. UCB for MCTS.

* [__week06_policy_based__](./week06_policy_based) Policy Gradient methods
  * Lecture: Motivation for policy-based, policy gradient, logderivative trick, REINFORCE/crossentropy method, variance reduction(baseline), advantage actor-critic (incl. GAE)
  * Seminar: REINFORCE, advantage actor-critic

* [__week07_seq2seq__](./week07_seq2seq) Reinforcement Learning for Sequence Models
  * Lecture: Problems with sequential data. Recurrent neural networks. Backprop through time. Vanishing & exploding gradients. LSTM, GRU. Gradient clipping
  * Seminar: character-level RNN language model

* [__week08_pomdp__](./week08_pomdp) Partially Observed MDP
  * Lecture: POMDP intro. POMDP learning (agents with memory). POMDP planning (POMCP, etc)
  * Seminar: Deep kung-fu & doom with recurrent A3C and DRQN
  
* [__week09_policy_II__](./week09_policy_II) Advanced policy-based methods
  * Lecture: Trust region policy optimization. NPO/PPO. Deterministic policy gradient. DDPG
  * Seminar: Approximate TRPO for simple robot control.

* [__week10_planning__](./week10_planning) Model-based RL & Co
  * Lecture: Model-Based RL, Planning in General, Imitation Learning and Inverse Reinforcement Learning
  * Seminar: MCTS for toy tasks

* [__yet_another_week__](./yet_another_week) Inverse RL and Imitation Learning
  * All that cool RL stuff that you won't learn from this course :)


# Course staff
Course materials and teaching by: _[unordered]_
- [Pavel Shvechikov](https://github.com/pshvechikov) - lectures, seminars, hw checkups, reading group
- [Nikita Putintsev](https://github.com/qwasser) - seminars, hw checkups, organizing our hot mess
- [Alexander Fritsler](https://github.com/Fritz449) - lectures, seminars, hw checkups
- [Oleg Vasilev](https://github.com/Omrigan) - seminars, hw checkups, technical support
- [Dmitry Nikulin](https://github.com/pastafarianist) - tons of fixes, far and wide
- [Mikhail Konobeev](https://github.com/MichaelKonobeev) - seminars, hw checkups
- [Ivan Kharitonov](https://github.com/neer201) - seminars, hw checkups
- [Ravil Khisamov](https://github.com/zshrav) - seminars, hw checkups
- [Anna Klepova](https://github.com/q0o0p) - hw checkups
- [Fedor Ratnikov](https://github.com/justheuristic) - admin stuff

# Contributions
* Using pictures from [Berkeley AI course](http://ai.berkeley.edu/home.html)
* Massively refering to [CS294](http://rll.berkeley.edu/deeprlcourse/)
* Several tensorflow assignments by [Scitator](https://github.com/Scitator)
* A lot of fixes from [arogozhnikov](https://github.com/arogozhnikov)
* Other awesome people: see github [contributors](https://github.com/yandexdataschool/Practical_RL/graphs/contributors)
* [Alexey Umnov](https://github.com/alexeyum) helped us a lot during spring2018

---

### Overview (`docker/README.md`)

To simplify installation process, you can deploy a container (~virtual machine) with all dependencies pre-installed.

_tl;dr [dockerhub url](https://hub.docker.com/r/justheuristic/practical_rl/)_

## Install Docker

We recommend you to use either native docker (recommended for linux) or kitematic(recommended for windows).
* Installing [kitematic](https://kitematic.com/), a simple interface to docker (all platforms)
* Pure docker: Guide for [windows](https://docs.docker.com/docker-for-windows/), [linux](https://docs.docker.com/engine/installation/), or [macOS](https://docs.docker.com/docker-for-mac/).
* If you want to use your GPU make sure you have [nvidia-container-toolkit](https://github.com/NVIDIA/nvidia-container-toolkit) and [NVidia driver](https://www.nvidia.com/en-us/drivers/unix/) + [CUDA 10.2](https://developer.nvidia.com/cuda-downloads) installed

Below are the instructions for both approaches.

## Kitematic
Find dmittov/practical_rl in the search menu. Download and launch the container.

Click on "web preview" screen in the top-right __or__ go to settings, ports and find at which port your jupyter is located, usually 32***.

## Native
`docker run --rm -it -v /path/to/your/repo:/notebooks -p <local_port>:8888 dmittov/practical_rl:spring2020-cpu`

For example,
```docker run --rm -it -v /home/myuser/Documents/practical_rl:/notebooks -p 8888:8888 dmittov/practical_rl:spring2020-cpu```


Then you can access your jupyter in a browser at `localhost:<local_port>/?token=<token_you_see_in_container_logs>`, e.g. `localhost:8888/?token=ad1a5a0aab43efb47a9a805388fcf508d0b5f84a16e4542b&token=ad1a5a0aab43efb47a9a805388fcf508d0b5f84a16e4542b`

#### GPU
`docker run --rm -it -v /path/to/your/repo:/notebooks -p <local_port>:8888 --gpus all dmittov/practical_rl:spring2020-cuda-10.2`

## Manual
Build container

`docker build -t practical_rl --build-arg device=cpu .`

to build GPU version

`docker build -t practical_rl --build-arg device=gpu .`

Run it

`$ docker run --rm -it -v <local_dir>:/notebooks -p <local_port>:8888 practical_rl`

to run GPU version

`$ docker run --rm -it -v <local_dir>:/notebooks -p <local_port>:8888 --gpus all practical_rl`

examples:

```$ docker run --rm -it -v `pwd`:/notebooks -p 8888:8888 practical_rl```

Copy the token from console and run
http://localhost:8888/?token=<token>

---

### Overview (`week01_intro/README.md`)

## Materials:
* [__Lecture slides__](https://yadi.sk/i/-EUHXUXOTC5t9Q)
* __Russian:__
  * Intro to RL - [video](https://yadi.sk/i/HpRw6zYv3SPmFf)
  * Seminar - [video](https://yadi.sk/i/gRTVCHfM3SPmHo)

* __English:__
  * [__main__] Video-intro by David Silver (english) - [video](https://www.youtube.com/watch?v=2pWv7GOvuf0)
  * [__main__] Lecture by J Schulman with crossentropy method explained (english) - [url](https://www.youtube.com/watch?v=aUrX-rP_ss4&list=PLCTc_C7itk-GaAMxmlChrkPnGKtjz8hv1)
  * Optional lecture by David Silver (english) - [video](https://www.youtube.com/watch?v=lfHX2hHRMVQ)


## More materials:
* __[recommended]__ - awesome openai post about evolution strategies - [blog post](https://openai.com/research/evolution-strategies), [article](https://arxiv.org/abs/1703.03864)
* __[recommended]__ - formal explanation of crossentropy method in [general](https://people.smp.uq.edu.au/DirkKroese/ps/CEEncycl.pdf) and for [optimization](https://people.smp.uq.edu.au/DirkKroese/ps/CEopt.pdf)
* Deep learning course (if you want to learn in parallel) - https://github.com/yandexdataschool/HSE_deeplearning
* Video on genetic algorithms (english) - [video](https://www.youtube.com/watch?v=ejxfTy4lI6I)
* Another guide to genetic algorithm (english) - [video](https://www.youtube.com/watch?v=zwYV11a__HQ)
* About Differential evolution (english) - [pdf](http://jvanderw.une.edu.au/DE_1.pdf)
* Video on Ant Colony Algorithm (english) - [video](https://www.youtube.com/watch?v=D58nLNLkb0I)
* Longer video on Ant Colony Algorithm (english) - [video](https://www.youtube.com/watch?v=xpyKmjJuqhk)
* Longer video on corssentropy method (russian) - [video](https://yadi.sk/i/5yf_4oGI3EDJhJ)
  

## Practice assignment
Instant dive in: [__seminar_gymnasium_interface__](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week01_intro/seminar_gymnasium_interface.ipynb), [__crossentropy_method__](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week01_intro/crossentropy_method.ipynb),
[__deep_crossentropy_method__](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week01_intro/deep_crossentropy_method.ipynb)

* Open `seminar_gymnasium_interface.ipynb` and follow instructions from there
* After you're done there, proceed to `crossentropy_method.ipynb`
* You can find homework and bonus assignment descriptions at the end of that notebook.
* Note: so far it's enough to say `pip install gymnasium` on top of any data-science-stuffed python, but we'd appreciate if you gradually switch to [full installation](https://github.com/Farama-Foundation/Gymnasium).

---

### Overview (`week02_value_based/README.md`)

## Materials
* [__Lecture slides__](https://docs.google.com/presentation/d/1dyTsV4MfLb006VHMQvrMTEvSMjsN4B4ymoiWl-HrqVs)
* Our videos: [lecture](https://yadi.sk/i/qsYgN53mviDpQw) [seminar](https://yadi.sk/i/umiVNsMWngtodg) (russian)
* __[main]__ lecture by David Silver - [url](https://www.youtube.com/watch?v=Nd1-UUMVfz4)
* Alternative lecture by Pieter Abbeel (english): [part 1](https://www.youtube.com/watch?v=i0o-ui1N35U), [part 2](https://www.youtube.com/watch?v=Csiiv6WGzKM)
* Alternative lecture by John Schulmann (english): [video](https://www.youtube.com/watch?v=IL3gVyJMmhg)
* Definitive guide in policy/value iteration from Sutton: start from page 81 [here](http://incompleteideas.net/book/the-book-2nd.html).


## Homework description:

The main assignment is `seminar_vi.ipynb`[](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week02_value_based/seminar_vi.ipynb) notebook in this week's folder. It has no requirements besides the most basic data science libraries (e.g. numpy) so you should be able to run it locally.

__Note:__ if you have any difficulty using graphviz, just set `has_graphviz=False`.

---

### Overview (`week03_model_free/README.md`)

#### __Lecture slides__ - [here](https://yadi.sk/i/9bPN0pxsR1-zIA)
### Materials
* Russian materials:
   - Lecture - [video](https://yadi.sk/i/jcQ1Bg8n3SrhuQ)
   - Seminars
      - Q-learning seminar - [video](https://yadi.sk/i/dQmolwOy3EtGNK) (older track - [assignment](./crawler_and_pacman/seminar_py2))
      - SARSA & stuff - [video](https://yadi.sk/i/XbqNQmjm3ExNsq)
* English materials:
   - Lecture by David Silver (english) - [video part I](https://www.youtube.com/watch?v=PnHCvfgC_ZA), [video part II](https://www.youtube.com/watch?v=0g4j2k_Ggc4&t=43s)
   - Alternative lecture by Pieter Abbeel (english) - [video](https://www.youtube.com/watch?v=ifma8G7LegE)
   - Alternative lecture by John Schulmann (english) - [video](https://www.youtube.com/watch?v=IL3gVyJMmhg)
   - Blog post on q-learning Vs SARSA - [url](https://studywolf.wordpress.com/2013/07/01/reinforcement-learning-sarsa-vs-q-learning/)

### More materials
* N-step temporal difference from Sutton's book - [suttonbook](http://incompleteideas.net/book/RLbook2018.pdf) __chapter 7__
* Eligibility traces from Sutton's book - [suttonbook](http://incompleteideas.net/book/RLbook2018.pdf) __chapter 12__
* Blog post on eligibility traces - [url](http://pierrelucbacon.com/traces/)

### Assignments

Just as usual, start with `homework.ipynb` [](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week03_model_free/homework.ipynb) For seminar, implement q-learning agent and test it on Taxi and CartPole with binarizer. And then, implement EV-SARSA agent, experience replay + bonus tasks for homework.

---

### Overview (`week04_[recap]_deep_learning/README.md`)

__Note:__ This week's materials cover the basics of neural nets and deep learning and teach you how to use auto-diff frameworks. If you're already fluent in Tensorflow or PyTorch, feel free to skip this week entirely.

## Materials
* [__Lecture slides__](https://yadi.sk/i/yAO2AJ3M3EKP8g)

- __In russian:__
  * Basic lecture on deep learning - [video](https://yadi.sk/i/yyHZub6R3Ej5dV)
  * Deep learning frameworks - [video](https://yadi.sk/i/hDIkaR4H3EtnXM)
  * [Pytorch tutorial](https://yadi.sk/i/O3mQ76u43So3h9) __recommended__
  * [Tensorflow tutorial](https://www.youtube.com/watch?v=FQ660T4uu7k) (english only for now. Links are welcome)

- __In english:__
  * Intro to neural nets and backprop (english) - [video](https://www.youtube.com/watch?v=uXt8qF2Zzfo)
  * Intro to convnets - [video](https://www.youtube.com/watch?v=FmpDIaiMIeA)
  * Deep learning frameworks - [video](https://www.youtube.com/watch?v=Vf_-OkqbwPo)
  * [Tensorflow tutorial](https://www.youtube.com/watch?v=FQ660T4uu7k)
  * [PyTorch tutorial](https://www.youtube.com/watch?v=VMcRWYEKmhw)

## Bonus materials
* Karpathy's course on deep learning (english) - http://cs231n.github.io/
* A neat little play-ground where you can train small NNs and see what they actually learn - [playground](http://playground.tensorflow.org/)
* Nuts and Bolts of deep learning by Andrew Ng (english) - [video](https://www.youtube.com/watch?v=F1ka6a13S9I)
* Deep learning philosophy: [our humble take](https://www.youtube.com/watch?v=9qyE1Ev1Xdw) (english)
* Deep learning demystified - [video](https://www.youtube.com/watch?v=Q9Z20HCPnww)
* Karpathy's lecture on deep learning for computer vision - https://www.youtube.com/watch?v=u6aEYuemt0M
* Our humble DL course: [HSE'fall17](https://github.com/yandexdataschool/HSE_deeplearning), [Skoltech/YSDA'spring16](https://github.com/ddtm/dl-course/) courses on deep learning (english).
* Srsly, just google `"deep learning %s" % s for s in what_you_want_to_know`.


### Practice
__[Colab URL (PyTorch)](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week04_%5Brecap%5D_deep_learning/seminar_pytorch.ipynb)__
From now on, we'll have two tracks: Tensorflow and PyTorch.

Please pick `seminar_tensorflow.ipynb` or `seminar_pytorch.ipynb`.

__Note:__ in this and all following weeks you're only required to get through practice in _one_ of the frameworks. Looking into other alternatives is great for self-education but never mandatory.

#### What to choose?
* The simplest choice is PyTorch: it's basically ye olde numpy with automatic gradients and a lot of pre-implemented DL stuff... except all the functions have different names.
* If you want to be familiar with production-related stuff from day 1, choose TensorFlow. It's much more convenient to deploy (to non-python or to mobiles). The catch is that all those conveniences become inconveniences once you want to write something simple in jupyter.

* It's not like choosing house at Hogwarts, you'll be able to switch between frameworks easily once you master the underlying principles.

---

### Overview (`week04_approx_rl/README.md`)

## Materials
* [__slides__](https://github.com/yandexdataschool/Practical_RL/files/15286337/ysda_practical_rl_lecture_04.pdf)
* Our [lecture](https://yadi.sk/i/Gd9yWV1dpuB7BQ), [seminar](https://yadi.sk/i/mvtKAIRN2yKU2g) (russian)
* [__seminar slides__](https://yadi.sk/i/B89gXClSpmYZKw)


* David Silver lecture - [video](https://www.youtube.com/watch?v=UoPei5o4fps)
* More practical and less theoretical lecture from MIT 6.S191 - [video](https://www.youtube.com/watch?v=xWe58WGWmlk)
* Understanding approximate q-learning - [url](https://danieltakeshi.github.io/2016/10/31/going-deeper-into-reinforcement-learning-understanding-q-learning-and-linear-function-approximation/)
* Karpathy's post on approximate RL - [url](http://karpathy.github.io/2016/05/31/rl/)

## More materials
* __[recommended]__ How to _actually_ do deep reinforcement learning by J. Schulman - [pdf](http://rll.berkeley.edu/deeprlcourse/docs/nuts-and-bolts.pdf)
* __[recommended]__ [Spinning up](https://spinningup.openai.com/en/latest/) - a massive repository of deep RL knowledge
* __[recommended]__ A table of collateral effects of reinforcement learning in games - [table](https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml?fbclid=IwAR114qpVxlzU8vO8BiMRbpPKUIci8sHsOoUZ16-_0yVtABr6AW1-AZKFXWg)
* DQN and modiffications - lecture by J. Schulman - [video](https://www.youtube.com/watch?v=h1-pj4Y9-kM)
* * interactive demos in your browser: [demo1](http://cs.stanford.edu/people/karpathy/convnetjs/demo/rldemo.html)(karpathy), [demo2](http://janhuenermann.com/projects/learning-to-drive)(Hünermann)
* Reinforcement learning architectures list - [repo](https://github.com/5vision/deep-reinforcement-learning-networks)
* Article on dueling DQN - [arxiv](https://arxiv.org/pdf/1511.06581.pdf)
* Article on double DQN - [arxiv](https://arxiv.org/abs/1509.06461)
* Article on prioritized experience replay - [arxiv](https://arxiv.org/abs/1511.05952)
* Article on Rainbow: Combining Improvements in Deep Reinforcement Learning - [arxiv](https://arxiv.org/abs/1710.02298)
* Article on bootstrap DQN - [pdf](https://papers.nips.cc/paper/6501-deep-exploration-via-bootstrapped-dqn.pdf), [summary](http://pemami4911.github.io/paper-summaries/2016/08/16/Deep-exploration.html)
* Article on asynchronuous methods in deep RL - [arxiv](https://arxiv.org/abs/1602.01783)
* Successor representations for reinforcement learning - [article](https://arxiv.org/abs/1606.02396), [video](https://www.youtube.com/watch?v=kNqXCn7K-BM&feature=youtu.be)
* Video on asynchronuous methods (Mnih) - [video](https://www.youtube.com/watch?v=9sx1_u2qVhQ)
* An overview of deep reinforcement learning - [arxiv](https://arxiv.org/pdf/1701.07274v1.pdf)

## DQN tutorials
* [in pytorch] A great series starting from simple DQN to all the cool new stuff - [url](https://github.com/higgsfield/RL-Adventure)
* A guide to deep RL from ~scratch (nervana blog) - [url](https://www.nervanasys.com/demystifying-deep-reinforcement-learning/)
* Building deep q-network from ~scratch (blog) - [url](https://jaromiru.com/2016/09/27/lets-make-a-dqn-theory/)
* Another guide guide to DQN from ~scratch (blog) - [url](https://rubenfiszel.github.io/posts/rl4j/2016-08-24-Reinforcement-Learning-and-DQN.html)


## Practice

* Seminar: [](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week04_approx_rl/seminar_pytorch.ipynb)
* Homework (main): [](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week04_approx_rl/homework_pytorch_main.ipynb#scrollTo=KVvvo7k_ap8w)
* Homework (debug): [](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week04_approx_rl/homework_pytorch_debug.ipynb#scrollTo=KVvvo7k_ap8w)



From now on, we have two tracks, for pytorch and tensorflow. However, pytorch track is somewhat better supported by the course team. You can choose whichever track you want, but unless you're expertly familiar with your framework, we recommend you to start by completing the task in pytorch and only then reproduce your solution in your chosen framework.

Begin with `seminar_<framework>.ipynb` and then proceed with `homework_<framework>.ipynb`.


__Note: you're not required to submit assignments in all three frameworks. Pick one and go with it. Maybe switch it occasionally if you want more challenge.__

---

### Overview (`week05_explore/README.md`)

[](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week05_explore/week5.ipynb)

### Slides - [here](https://yadi.sk/i/H0zVBROe3TWWHz)

## Exploration and exploitation
* [__main__] David Silver lecture on exploration and expoitation - [video](https://www.youtube.com/watch?v=sGuiWX07sKw)
* Alternative lecture by J. Schulman - [video](https://www.youtube.com/watch?v=SfCa1HQMkuw)
* Alternative lecture by N. de Freitas (with bayesian opt) - [video](https://www.youtube.com/watch?v=vz3D36VXefI)
* Our lectures (russian)
  - "mathematical" lecture (by Alexander Vorobev) '17 - [slides](https://yadi.sk/i/JAeItALT3JmvCL), [video](https://yadi.sk/i/bVHmu9gt3Hi9Ym)
  - "practical" lecture '18 - [video](https://yadi.sk/i/_myWJ13O3TdzXo)
  - Seminar - [video](https://yadi.sk/i/du7FLXs13TdzZS)



## More materials
* Gittins Index - the less heuristical approach to bandit exploration - [article](http://www.ece.mcgill.ca/~amahaj1/projects/bandits/book/2013-bandit-computations.pdf)
* "Deep" version: variational information maximizing exploration - [video](https://www.youtube.com/watch?v=sRIjxxjVrnY)
  * Same topics in russian - [video](https://yadi.sk/i/_2_0yqeW3HDbcn)
* Lecture covering intrinsically motivated reinforcement learning - [video](https://www.youtube.com/watch?v=aJI_9SoBDaQ)
  * [Slides](https://yadi.sk/i/8sx42nau3HEYKg)
  * Same topics in russian - [video](https://www.youtube.com/watch?v=WCE9hhPbCmc)
  * Note: UCB-1 is not for bernoulli rewards, but for arbitrary r in [0,1], so you can just scale any reward to [0,1] to obtain a peace of mind. It's derived directly from Hoeffding's inequality.

* Very interesting blog post written by Lilian Weng that summarises this week's materials: [The Multi-Armed Bandit Problem and Its Solutions](https://lilianweng.github.io/posts/2018-01-23-multi-armed-bandit/)

## Seminar
In this seminar, you'll be solving basic and contextual bandits with uncertainty-based exploration like Bayesian UCB and Thompson Sampling. You will also get acquainted with Bayesian Neural Networks.

Everything else is in the notebook :)

---

### Overview (`week06_policy_based/README.md`)

## Materials
* [Slides](https://yadi.sk/i/keSzKSgA2oYuwQ)
* Video lecture by D. Silver - [video](https://www.youtube.com/watch?v=KHZVXao4qXs)
* Our [lecture](https://yadi.sk/i/yPIPkO_f3TPsNK), [seminar(pytorch)](https://yadi.sk/i/flW8ezGk3TPsQ5)
* Alternative lecture by J. Schulman part 1 - [video](https://www.youtube.com/watch?v=BB-BhTn6DCM)
* Alternative lecture by J. Schulman part 2 - [video](https://www.youtube.com/watch?v=Wnl-Qh2UHGg)
* Andrej Karpathy's [post](http://karpathy.github.io/2016/05/31/rl/) on policy gradients


## More materials
* Actually proving the policy gradient for discounted rewards - [article](https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf)
* On variance of policy gradient and optimal baselines: [article](https://papers.nips.cc/paper/4264-analysis-and-improvement-of-policy-gradient-estimation.pdf), another [article](https://arxiv.org/pdf/1301.2315.pdf)
* Learn Advatangeg Actor Critic with a [comic](https://hackernoon.com/intuitive-rl-intro-to-advantage-actor-critic-a2c-4ff545978752)


* Generalizing log-derivative trick - [url](http://blog.shakirm.com/2015/11/machine-learning-trick-of-the-day-5-log-derivative-trick/)
* Combining policy gradient and q-learning - [arxiv](https://arxiv.org/abs/1611.01626)
* Variational perspective on reinforcement learning (from DeepBayes) - [pdf](http://incompleteideas.net/book/the-book-2nd.html)
* Adversarial review of policy gradient - [blog](http://www.argmin.net/2018/02/20/reinforce/)


Run seminar notebook in Colab: [](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week06_policy_based/reinforce_pytorch.ipynb)

Run optional homework notebook in Colab: [](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week06_policy_based/a2c-optional.ipynb)

---

### Overview (`week07_[recap]_rnn/README.md`)

## Materials
* [Slides](https://yadi.sk/i/-Iqdhg483GDyoN)
* CS231 lecture on RNNs - [video](https://www.youtube.com/watch?v=iX5V1WpxxkY)
* Our [lecture](https://yadi.sk/i/XHmT5hO53GcCKV), [seminar (PyTorch)](https://yadi.sk/i/nCch5I8S3TsXh5) (in russian)
* [alternative] Brief lecture on RNN by nervana - [video](https://www.youtube.com/watch?v=Ukgii7Yd_cU)
* [alternative] More detailed lecture by Y. Bengio - [video](https://www.youtube.com/watch?v=xK-bzjIQkmM)
* Great reading by Karpathy - [blog post](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)
* LSTM explained in detail by colah - [blog post](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)

## More materials
* Seq2seq lecture - [video](https://www.youtube.com/watch?v=G5RY_SUJih4)
* "Awesome rnn" entry point - [repo](https://github.com/kjw0612/awesome-rnn)
* OpenAI research on sentiment analysis that sheds some light on what's inside LSTM language model.

# Homework description
[](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week07_%5Brecap%5D_rnn/seminar_pytorch.ipynb)

This week's practice gets you acquainted with basics of recurrent neural networks. For simplicity, we'll train them on character language modelling task. You'll need `seminar_pytorch.ipynb` or `seminar_tf.ipynb`.

---

### Overview (`week07_seq2seq/README.md`)

## Materials
* [Slides](https://yadi.sk/i/2oUkKL8m3UFFe8)
* Our [lecture](https://yadi.sk/i/hmjUfKht3UNCSq) & [seminar](https://yadi.sk/i/dTkWTFNj3UNCTv) (russian)
* English lectures
  * Lecture by Mohammad Norouzi - [cs294 video](https://www.youtube.com/watch?v=fZNyHoXgV7M&index=24&list=PLkFD6_40KJIwTmSbCv9OVJB3YaO4sFwkX)
  * Alternative lecture by Graham Neubig - [CMU nn4nlp video](https://www.youtube.com/watch?v=isxzsAelQX0)
  * Optional lecture on conversation systems - [video](https://www.youtube.com/watch?v=2tKNpzUvDc4	)
* Will hopefully record our lecture in english soon!
* Self-critical sequence traning [original article](https://arxiv.org/abs/1612.00563)

## Practice

[](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week07_seq2seq/practice_torch.ipynb)


As usual, go to practice_{your framework}.ipynb above and follow instructions from there. [PyTorch](./practice_torch.ipynb), [Tensorflow](./practice_tf.ipynb)

## More materials
* An [awesome post](http://distill.pub/2016/augmented-rnns/) explaining attention and long-term memory models.
* [BLEU](http://www.aclweb.org/anthology/P02-1040.pdf) and [CIDEr](https://arxiv.org/pdf/1411.5726.pdf) articles.
* Image captioning
  * MSCOCO captioning [challenge](https://cocodataset.org/#captions-2015)
  * Captioning tutorial [notebook](https://github.com/yandexdataschool/Practical_DL/tree/980121c7b3147ed28a7c1360df5038d3432b8cc3/week07_seq2seq)
* Other articles on reinforcement learning for natural language:
  * [task-oriented conversation system](https://arxiv.org/abs/1703.07055)
  * [generating dialogues](https://arxiv.org/abs/1606.01541)
  * [sequential adversarial networks](https://arxiv.org/abs/1609.05473) (a.k.a. SeqGAN)
  * A large overview for machine translation (touching on RL, including RL failures) - [arxiv](https://arxiv.org/abs/1609.08144)
  * How _not_ to evaluate conversation models - [arxiv](https://arxiv.org/abs/1603.08023)
* Overview of other non-games applications ("that article again") - [arxiv](https://arxiv.org/abs/1701.07274)

---

### Overview (`week08_pomdp/README.md`)

# Materials
[lecture slides](https://yadi.sk/i/3-4q71hhwtFxuw)

_Links on all articles mentioned during the lecture could be found in "References" at the very end of the presentation slides. All other interesing links which contribute to the topic of POMDP are presented below_

## Basics
* Our [lecture](https://yadi.sk/i/AHzpTjiT3U8L8e) and [seminar](https://yadi.sk/i/Ka-I7nBp3U8LAG) (russian)
* A Lecture on Basics by Pavel Shvechikov (english) [Video](https://www.youtube.com/watch?v=aV4wz7FAXmo)
* A lecture on basics by Andrew NG (english, LQ) - [video](https://www.youtube.com/watch?v=yCqPMD6coO8)
* A lecture on basics by 5vision (russian) - [video](https://www.youtube.com/watch?v=_dkaynuKUFE)
* _[alternative]_ Chalkboard-style 2-part lecture by B. Ravindran.  - [part1](https://www.youtube.com/watch?v=9G_KevA8DFY), [part2](https://www.youtube.com/watch?v=dMOUp7YzUpQ)
* _[alternative]_ Yet another mini-lecture touching on POMDP by S.S. Baveja - [video](https://www.youtube.com/watch?v=SE56KgF7aVc)

## POMDP Learning
* DRQN lecture by Fritz448 (russian) - [video](https://www.youtube.com/watch?v=bE5DIJvZexc)
* [Data efficient learning in continous POMDP](https://arxiv.org/abs/1602.02523v1)
* [Managing wind farms with bayesian POMDP](http://ascelibrary.org/doi/abs/10.1061/(ASCE)CP.1943-5487.0000390)
* [Bayesian learning and decision-making in dynamic environments](http://www.jmlr.org/papers/volume12/ross11a/ross11a.pdf)




---

# Practice

[](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week08_pomdp/practice_pytorch.ipynb)

The assignment is platform and framewerk independent, so choose the framework that suits you best, but pay attention on how many you will need to implement youself in case of nonstandart ones.

---

### Overview (`week09_policy_II/README.md`)

* [__slides #1 (trpo)__](https://docs.google.com/presentation/d/15Z_AVBsO9VuOSZ5uY-Q4by3tHKiRSENchhAKHhCxIOc/present?token=AC4w5VgM6o7lCOmwtNFI3lfzyPv2PHOpRQ%3A1511795215658&includes_info_params=1#slide=id.g1d8d5bc58c_0_4)
* [__slides #2 (dpg)__](https://yadi.sk/i/uV6IA-C23UTn7c)

## Materials
This section covers some steroids for policy gradient methods, along with a cool general trick called

* Lecture on NPG and TRPO by J. Schulman - [video](https://www.youtube.com/watch?v=_t5fpZuuf-4)
* Alternative lecture on TRPO and open problems by... J. Schulman - [video](https://www.youtube.com/watch?v=gb5Q2XL5c8A)
* Our videos: [lecture](https://yadi.sk/i/c7GR1kAAJc00Og), [seminar(PyTorch)](https://yadi.sk/i/OGZJJjkQH_7h5g) (russian)
* Original articles - [TRPO](https://arxiv.org/abs/1502.05477), [NPG](https://papers.nips.cc/paper/2073-a-natural-policy-gradient.pdf)

## Practice
* TRPO: [](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week09_policy_II/seminar_TRPO_pytorch.ipynb)

* PPO: [](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week09_policy_II/ppo.ipynb)

## More: Reinforcement learning in large/continuous action spaces
While you already know algorithms that will work with continuously many actions, it can't hurt to learn something more specialized.
 * Lecture by J. Schulman - [video](https://www.youtube.com/watch?v=jmMsNQ2eug4)
 * Q-learning with normalized advantage functions - [article](https://arxiv.org/abs/1603.00748), [code1](https://github.com/carpedm20/NAF-tensorflow), [code2](http://bit.ly/2qx2087)
 * Deterministic policy gradient - [article](https://arxiv.org/pdf/1512.07679.pdf), [post+code](https://yanpanlau.github.io/2016/10/11/Torcs-Keras.html)
 * Stochastic value gradient - [article](https://arxiv.org/abs/1510.09142)
 * Embedding large discrete action spaces for RL - [article](https://arxiv.org/pdf/1512.07679.pdf)
 * Lecture by A. Seleznev, 5vision (russian) - [video](https://www.youtube.com/watch?v=j1L2FnanXPo&t=119m45s)

---

### Overview (`week10_planning/README.md`)

## Assignments

Just as usual, start with  `seminar_MCTS.ipynb`
[](https://colab.research.google.com/github/yandexdataschool/Practical_RL/blob/master/week10_planning/seminar_MCTS.ipynb)

## Materials: planning

* Planning by dynamic programming (D. Silver) - [video](https://www.youtube.com/watch?v=Nd1-UUMVfz4)
* Planning via tree search [videos 2-6 from CS188](https://www.youtube.com/channel/UCHBzJsIcRIVuzzHVYabikTQ)
* Our lecture:
  * [Lecture(planning, MCTS, planning in POMDP)](https://yadi.sk/i/lOAUu7o13JBHFz) & [Seminar(MCTS)](https://yadi.sk/i/bkmjEZrk3JBHGF)
  * Slides [part1](https://yadi.sk/i/3PM9zCP33J3ub3) (intro), [part2](https://yadi.sk/i/M03xvZ2y3JMQre) (pomdp)
* Monte-carlo tree search
  * Monte-carlo tree search step-by-step by J.Levine (very intuitive) - [video](https://www.youtube.com/watch?v=UXW2yZndl7U)
  * Udacity video on monte-carlo tree search (first part of a chain) - [video](https://www.youtube.com/watch?v=onBYsen2_eA)
  * A Survey of Monte Carlo Tree Search Methods (2011-2012) [pdf](http://ccg.doc.gold.ac.uk/ccg_old/papers/browne_tciaig12_1.pdf)
  * Reminder: UCB-1 - [slides](https://www.cs.bham.ac.uk/internal/courses/robotics/lectures/ucb1.pdf)
  * Guide to MCTS - [url](https://jeffbradberry.com/posts/2015/09/intro-to-monte-carlo-tree-search/)
  or [cached](https://webcache.googleusercontent.com/search?q=cache:jeffbradberry.com/posts/2015/09/intro-to-monte-carlo-tree-search/)

## Supplementary materials

* Integrating learning and planning (D. Silver) - [video](https://www.youtube.com/watch?v=ItMutbeOHtc)
* Approximating the MCTS optimal actions - 5vision solution for deephack.RL, code by Mikhail Pavlov - [repo](https://github.com/5vision/uct_atari)
* Alpha Go - [explanation](https://medium.com/@jonathan_hui/alphago-how-it-works-technically-26ddcc085319)
* Monte Carlo Planning in POMDP [pdf](https://papers.nips.cc/paper/4031-monte-carlo-planning-in-large-pomdps.pdf)

---

### Overview (`yet_another_week/README.md`)

__Our slides:__ __[inverse/imitation rl](https://yadi.sk/i/ngB_BcNx3UggK6);__ __[multi-agent 101](https://yadi.sk/i/XrFgmdCy3Vtd4k);__ __[multi-agent 102](https://docs.google.com/presentation/d/1AiSZnWGHWU_34QZ0fqdCGIXWUX_bC-0zz3kc3VVcPCA/edit?usp=sharing);__ __[hierarchical rl](https://yadi.sk/i/LkNiKxMz3Vtcr3)__


In this week you can find several sections covering advanced topics in RL, along with less advanced topics that we couldn't squeeze into the main track

## Other
* Learning by imitation - [video](https://www.youtube.com/watch?v=kl_G95uKTHw), [assignment](http://rll.berkeley.edu/deeprlcourse/docs/hw1.pdf)(berkeley cs294)
* Inverse reinforcement learning 
  * Lecture by Chelsea Finn - [video](https://www.youtube.com/watch?v=d9DlQSJQAoI)
  * Udacity videos - [video](https://www.youtube.com/watch?v=h7uGyBcIeII)
* Distributional RL - [video](https://www.youtube.com/watch?v=bsuvM1jO-4w)
* Learning World Models for RL - [blog](https://worldmodels.github.io/)
* Knowledge transfer in RL - [video](https://www.youtube.com/watch?v=Hx4XpVdJOI0)(berkeley cs294)
* Hierarchical reinforcemnt learning 
  * Cool article ( Fe__U__dal networks ) - [arxiv](https://arxiv.org/abs/1703.01161)
  * Short lecture by Roy Fox - [video](https://www.youtube.com/watch?v=x_QjJry0hTc)
* Multi-Agent reinforcement learning
  * Lecture by Thoro Graepel - role of HRL in AI research - [video](https://www.youtube.com/watch?v=CvL-KV3IBcM)
  * Lecture by Balaraman Ravindran - [video](https://www.youtube.com/watch?v=K5MlmO0UJtI)

## A list of lists
* [awesome_rl](https://github.com/aikorea/awesome-rl/) - a curated list of resources dedicated to reinforcement learning.
* [junhyukoh's list](https://github.com/junhyukoh/deep-reinforcement-learning-papers)
* [muupan's list](https://github.com/muupan/deep-reinforcement-learning-papers)
* Courses:
 * [CS294: deep reinforcement learning](http://rll.berkeley.edu/deeprlcourse/)
 * [Silver's RL course](http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html)
 * [Sutton's book, 2nd edition](http://incompleteideas.net/sutton/book/the-book-2nd.html)
* [Implementations of many basic RL algorithms (raw and/or tensorflow)](https://github.com/dennybritz/reinforcement-learning)
* Reddit: [General ML](https://www.reddit.com/r/MachineLearning/), [RL](https://www.reddit.com/r/reinforcementlearning/), [CS294](https://www.reddit.com/r/berkeleydeeprlcourse/)
* [This great link you could have contributed]

---

### Overview (`yet_another_week/_resource/README.md`)

This is a utility folder to store images and other resources used in notebooks.

---

--- METRICS ---
- Files Extracted: 16
- Estimated Token Budget: ~9610 tokens
- Canonical Reference: https://codewiki.google/github.com/yandexdataschool/Practical_RL