# Technical Documentation: ageron/handson-ml3 > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/ageron/handson-ml3) ### Overview (`README.md`) Machine Learning Notebooks, 3rd edition ================================= This project aims at teaching you the fundamentals of Machine Learning in python. It contains the example code and solutions to the exercises in the third edition of my O'Reilly book [Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow (3rd edition)](https://homl.info/er3): **Note**: If you are looking for the second edition notebooks, check out [ageron/handson-ml2](https://github.com/ageron/handson-ml2). For the first edition, see [ageron/handson-ml](https://github.com/ageron/handson-ml). ## Quick Start ### Want to play with these notebooks online without having to install anything? * Open In Colab (recommended) ⚠ _Colab provides a temporary environment: anything you do will be deleted after a while, so make sure you download any data you care about._
Other services may work as well, but I have not fully tested them: * Open in Kaggle * Launch binder * Launch in Deepnote
### Just want to quickly look at some notebooks, without executing any code? * Render nbviewer * [github.com's notebook viewer](https://github.com/ageron/handson-ml3/blob/main/index.ipynb) also works but it's not ideal: it's slower, the math equations are not always displayed correctly, and large notebooks often fail to open. ### Want to run this project using a Docker image? Read the [Docker instructions](https://github.com/ageron/handson-ml3/tree/main/docker). ### Want to install this project on your own machine? Start by installing [Anaconda](https://www.anaconda.com/products/distribution) (or [Miniconda](https://docs.conda.io/en/latest/miniconda.html)), [git](https://git-scm.com/downloads), and if you have a TensorFlow-compatible GPU, install the [GPU driver](https://www.nvidia.com/Download/index.aspx), as well as the appropriate version of CUDA and cuDNN (see TensorFlow's documentation for more details). Next, clone this project by opening a terminal and typing the following commands (do not type the first `$` signs on each line, they just indicate that these are terminal commands): $ git clone https://github.com/ageron/handson-ml3.git $ cd handson-ml3 Next, run the following commands: $ conda env create -f environment.yml $ conda activate homl3 $ python -m ipykernel install --user --name=python3 Finally, start Jupyter: $ jupyter notebook If you need further instructions, read the [detailed installation instructions](INSTALL.md). # FAQ **Which Python version should I use?** I recommend Python 3.10. If you follow the installation instructions above, that's the version you will get. Any version ≥3.7 should work as well. **I'm getting an error when I call `load_housing_data()`** If you're getting an HTTP error, make sure you're running the exact same code as in the notebook (copy/paste it if needed). If the problem persists, please check your network configuration. If it's an SSL error, see the next question. **I'm getting an SSL error on MacOSX** You probably need to install the SSL certificates (see this [StackOverflow question](https://stackoverflow.com/questions/27835619/urllib-and-ssl-certificate-verify-failed-error)). If you downloaded Python from the official website, then run `/Applications/Python\ 3.10/Install\ Certificates.command` in a terminal (change `3.10` to whatever version you installed). If you installed Python using MacPorts, run `sudo port install curl-ca-bundle` in a terminal. **I've installed this project locally. How do I update it to the latest version?** See [INSTALL.md](INSTALL.md) **How do I update my Python libraries to the latest versions, when using Anaconda?** See [INSTALL.md](INSTALL.md) ## Contributors I would like to thank everyone [who contributed to this project](https://github.com/ageron/handson-ml3/graphs/contributors), either by providing useful feedback, filing issues or submitting Pull Requests. Special thanks go to Haesun Park and Ian Beauregard who reviewed every notebook and submitted many PRs, including help on some of the exercise solutions. Thanks as well to Steven Bunkley and Ziembla who created the `docker` directory, and to github user SuperYorio who helped on some exercise solutions. Thanks a lot to Victor Khaustov who submitted plenty of excellent PRs, fixing many errors. And lastly, thanks to Google ML Developer Programs team who supported this work by providing Google Cloud Credit. --- ### Overview (`docker/README.md`) # Hands-on Machine Learning in Docker This is the Docker configuration which allows you to run and tweak the book's notebooks without installing any dependencies on your machine!
OK, any except `docker` and `docker-compose`.
And optionally `make`.
And a few more things if you want GPU support (see below for details). ## Prerequisites Follow the instructions on [Install Docker](https://docs.docker.com/engine/installation/) and [Install Docker Compose](https://docs.docker.com/compose/install/) for your environment if you haven't got `docker` and `docker-compose` already. Some general knowledge about `docker` infrastructure might be useful (that's an interesting topic on its own) but is not strictly *required* to just run the notebooks. ## Usage ### Prepare the image (once) The first option is to pull the image from Docker Hub (this will download about 1.9 GB of compressed data): ```bash $ docker pull ageron/handson-ml3 ``` **Note**: this is the CPU-only image. For GPU support, read the GPU section below. Alternatively, you can build the image yourself. This will be slower, but it will ensure the image is up to date, with the latest libraries. For this, assuming you already downloaded this project into the directory `/path/to/project/handson-ml3`: ```bash $ cd /path/to/project/handson-ml3/docker $ docker-compose build ``` This will take quite a while, but is only required once. After the process is finished you have an `ageron/handson-ml3:latest` image, that will be the base for your experiments. You can confirm that by running the following command: ```bash $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE ageron/handson-ml3 latest 3ebafebc604a 2 minutes ago 4.87GB ``` ### Run the notebooks Still assuming you already downloaded this project into the directory `/path/to/project/handson-ml3`, run the following commands to start the Jupyter server inside the container, which is named `handson-ml3`: ```bash $ cd /path/to/project/handson-ml3/docker $ docker-compose up ``` Next, just point your browser to the URL printed on the screen (or go to if you enabled password authentication inside the `jupyter_notebook_config.py` file, before building the image) and you're ready to play with the book's code! The server runs in the directory containing the notebooks, and the changes you make from the browser will be persisted there. You can close the server just by pressing `Ctrl-C` in the terminal window. ### Using `make` (optional) If you have `make` installed on your computer, you can use it as a thin layer to run `docker-compose` commands. For example, executing `make rebuild` will actually run `docker-compose build --no-cache`, which will rebuild the image without using the cache. This ensures that your image is based on the latest version of the `continuumio/miniconda3` image which the `ageron/handson-ml3` image is based on. If you don't have `make` (and you don't want to install it), just examine the contents of `Makefile` to see which `docker-compose` commands you can run instead. ### Run additional commands in the container Run `make exec` (or `docker-compose exec handson-ml3 bash`) while the server is running to run an additional `bash` shell inside the `handson-ml3` container. Now you're inside the environment prepared within the image. One of the useful things that can be done there would be starting TensorBoard (for example with simple `tb` command, see bashrc file). Another one may be comparing versions of the notebooks using the `nbdiff` command if you haven't got `nbdime` installed locally (it is **way** better than plain `diff` for notebooks). See [Tools for diffing and merging of Jupyter notebooks](https://github.com/jupyter/nbdime) for more details. You can see changes you made relative to the version in git using `git diff` which is integrated with `nbdiff`. You may also try `nbd NOTEBOOK_NAME.ipynb` command (custom, see bashrc file) to compare one of your notebooks with its `checkpointed` version.
To be precise, the output will tell you *what modifications should be re-played on the **manually saved** version of the notebook (located in `.ipynb_checkpoints` subdirectory) to update it to the **current** i.e. **auto-saved** version (given as command's argument - located in working directory)*. ## GPU Support on Linux (experimental) ### Prerequisites If you're running on Linux, and you have a TensorFlow-compatible GPU card (NVidia card with Compute Capability ≥ 3.5) that you would like TensorFlow to use inside the Docker container, then you should download and install the latest driver for your card from [nvidia.com](https://www.nvidia.com/Download/index.aspx?lang=en-us). You will also need to install [NVidia Docker support](https://github.com/NVIDIA/nvidia-docker): if you are using Docker 19.03 or above, you must install the `nvidia-container-toolkit` package, and for earlier versions, you must install `nvidia-docker2`. Next, edit the `docker-compose.yml` file: ```bash $ cd /path/to/project/handson-ml3/docker $ edit docker-compose.yml # use your favorite editor ``` * Replace `dockerfile: ./docker/Dockerfile` with `dockerfile: ./docker/Dockerfile.gpu` * Replace `image: ageron/handson-ml3:latest` with `image: ageron/handson-ml3:latest-gpu` * If you want to use `docker-compose`, you will need version 1.28 or above for GPU support, and you must uncomment the whole `deploy` section in `docker-compose.yml`. ### Prepare the image (once) If you want to pull the prebuilt image from Docker Hub (this will download over 3.5 GB of compressed data): ```bash $ docker pull ageron/handson-ml3:latest-gpu ``` If you prefer to build the image yourself: ```bash $ cd /path/to/project/handson-ml3/docker $ docker-compose build ``` ### Run the notebooks with `docker-compose` (version 1.28 or above) If you have `docker-compose` version 1.28 or above, that's great! You can simply run: ```bash $ cd /path/to/project/handson-ml3/docker $ docker-compose up [...] or http://127.0.0.1:8888/?token=[...] ``` Then point your browser to the URL and Jupyter should appear. If you then open or create a notebook and execute the following code, a list containing your GPU device(s) should be displayed (success!): ```python import tensorflow as tf tf.config.list_physical_devices("GPU") ``` To stop the server, just press Ctrl-C. ### Run the notebooks without `docker-compose` If you have a version of `docker-compose` earlier than 1.28, you will have to use `docker run` directly. If you are using Docker 19.03 or above, you can run: ```bash $ cd /path/to/project/handson-ml3 $ docker run --name handson-ml3 --gpus all -p 8888:8888 -p 6006:6006 --log-opt mode=non-blocking --log-opt max-buffer-size=50m -v `pwd`:/home/devel/handson-ml3 ageron/handson-ml3:latest-gpu /opt/conda/envs/homl3/bin/jupyter notebook --ip='0.0.0.0' --port=8888 --no-browser ``` If you are using an older version of Docker, then replace `--gpus all` with `--runtime=nvidia`. Now point your browser to the displayed URL: Jupyter should appear, and you can open a notebook and run `import tensorflow as tf` and `tf.config.list_physical_devices("GPU")` as above to confirm that TensorFlow does indeed see your GPU device(s). Lastly, to interrupt the server, press Ctrl-C, then run: ```bash $ docker rm handson-ml3 ``` This will remove the container so you can start a new one later (but it will not remove the image or the notebooks, don't worry!). Have fun! --- --- METRICS --- - Files Extracted: 2 - Estimated Token Budget: ~3253 tokens - Canonical Reference: https://codewiki.google/github.com/ageron/handson-ml3