Technical Documentation: ageron/handson-ml2
ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference
Overview (`README.md`)
Machine Learning Notebooks
⚠ THIS EDITION IS NOW OUTDATED. THE [THIRD EDITION OF MY BOOK](https://github.com/ageron/handson-ml3) IS NOW AVAILABLE, AS WELL A [PYTORCH VERSION](https://github.com/ageron/handson-mlp) AND [MANY TRANSLATIONS](https://homl.info/).
This project is for the second edition, which is now outdated (it came out in 2019).
This project aims at teaching you the fundamentals of Machine Learning in
python. It contains the example code and solutions to the exercises in the second edition of my O'Reilly book Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow:
Note: If you are looking for the first edition notebooks, check out ageron/handson-ml. For the third edition, check out ageron/handson-ml3.
Quick Start
Want to play with these notebooks online without having to install anything?
Use any of the following services (I recommended Colab or Kaggle, since they offer free GPUs and TPUs).
WARNING: Please be aware that these services provide temporary environments: anything you do will be deleted after a while, so make sure you download any data you care about.
Just want to quickly look at some notebooks, without executing any code?
github.com's notebook viewer also works but it's not ideal: it's slower, the math equations are not always displayed correctly, and large notebooks often fail to open.
Want to run this project using a Docker image?
Read the Docker instructions.
Want to install this project on your own machine?
Start by installing Anaconda (or Miniconda), git, and if you have a TensorFlow-compatible GPU, install the GPU driver, as well as the appropriate version of CUDA and cuDNN (see TensorFlow's documentation for more details).
Next, clone this project by opening a terminal and typing the following commands (do not type the first $ signs on each line, they just indicate that these are terminal commands):
$ git clone https://github.com/ageron/handson-ml2.git
$ cd handson-ml2Next, run the following commands:
$ conda env create -f environment.yml
$ conda activate tf2
$ python -m ipykernel install --user --name=python3Finally, start Jupyter:
$ jupyter notebookIf you need further instructions, read the detailed installation instructions.
FAQ
Which Python version should I use?
I recommend Python 3.8. If you follow the installation instructions above, that's the version you will get. Most code will work with other versions of Python 3, but some libraries do not support Python 3.9 or 3.10 yet, which is why I recommend Python 3.8.
I'm getting an error when I call load_housing_data()
Make sure you call fetch_housing_data() before you call load_housing_data(). If you're getting an HTTP error, make sure you're running the exact same code as in the notebook (copy/paste it if needed). If the problem persists, please check your network configuration.
I'm getting an SSL error on MacOSX
You probably need to install the SSL certificates (see this StackOverflow question). If you downloaded Python from the official website, then run /Applications/Python\ 3.8/Install\ Certificates.command in a terminal (change 3.8 to whatever version you installed). If you installed Python using MacPorts, run sudo port install curl-ca-bundle in a terminal.
I've installed this project locally. How do I update it to the latest version?
See INSTALL.md
How do I update my Python libraries to the latest versions, when using Anaconda?
See INSTALL.md
Contributors
I would like to thank everyone who contributed to this project, either by providing useful feedback, filing issues or submitting Pull Requests. Special thanks go to Haesun Park and Ian Beauregard who reviewed every notebook and submitted many PRs, including help on some of the exercise solutions. Thanks as well to Steven Bunkley and Ziembla who created the docker directory, and to github user SuperYorio who helped on some exercise solutions.
Overview (`docker/README.md`)
Hands-on Machine Learning in Docker
This is the Docker configuration which allows you to run and tweak the book's notebooks without installing any dependencies on your machine!OK, any except docker and docker-compose.And optionally make.And a few more things if you want GPU support (see below for details).
Prerequisites
Follow the instructions on Install Docker and Install Docker Compose for your environment if you haven't got docker and docker-compose already.
Some general knowledge about docker infrastructure might be useful (that's an interesting topic on its own) but is not strictly required to just run the notebooks.
Usage
Prepare the image (once)
The first option is to pull the image from Docker Hub (this will download about 1.9 GB of compressed data):
$ docker pull ageron/handson-ml2Note: this is the CPU-only image. For GPU support, read the GPU section below.
Alternatively, you can build the image yourself. This will be slower, but it will ensure the image is up to date, with the latest libraries. For this, assuming you already downloaded this project into the directory /path/to/project/handson-ml2:
$ cd /path/to/project/handson-ml2/docker
$ docker-compose buildThis will take quite a while, but is only required once.
After the process is finished you have an ageron/handson-ml2:latest image, that will be the base for your experiments. You can confirm that by running the following command:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ageron/handson-ml2 latest 3ebafebc604a 2 minutes ago 4.87GBRun the notebooks
Still assuming you already downloaded this project into the directory /path/to/project/handson-ml2, run the following commands to start the Jupyter server inside the container, which is named handson-ml2:
$ cd /path/to/project/handson-ml2/docker
$ docker-compose upNext, just point your browser to the URL printed on the screen (or go to http://localhost:8888 if you enabled password authentication inside the jupyter_notebook_config.py file, before building the image) and you're ready to play with the book's code!
The server runs in the directory containing the notebooks, and the changes you make from the browser will be persisted there.
You can close the server just by pressing Ctrl-C in the terminal window.
Using `make` (optional)
If you have make installed on your computer, you can use it as a thin layer to run docker-compose commands. For example, executing make rebuild will actually run docker-compose build --no-cache, which will rebuild the image without using the cache. This ensures that your image is based on the latest version of the continuumio/miniconda3 image which the ageron/handson-ml2 image is based on.
If you don't have make (and you don't want to install it), just examine the contents of Makefile to see which docker-compose commands you can run instead.
Run additional commands in the container
Run make exec (or docker-compose exec handson-ml2 bash) while the server is running to run an additional bash shell inside the handson-ml2 container. Now you're inside the environment prepared within the image.
One of the useful things that can be done there would be starting TensorBoard (for example with simple tb command, see bashrc file).
Another one may be comparing versions of the notebooks using the nbdiff command if you haven't got nbdime installed locally (it is way better than plain diff for notebooks). See Tools for diffing and merging of Jupyter notebooks for more details.
You can see changes you made relative to the version in git using git diff which is integrated with nbdiff.
You may also try nbd NOTEBOOK_NAME.ipynb command (custom, see bashrc file) to compare one of your notebooks with its checkpointed version.
To be precise, the output will tell you what modifications should be re-played on the manually saved version of the notebook (located in .ipynb_checkpoints subdirectory) to update it to the current i.e. auto-saved version (given as command's argument - located in working directory).
GPU Support on Linux (experimental)
Prerequisites
If you're running on Linux, and you have a TensorFlow-compatible GPU card (NVidia card with Compute Capability ≥ 3.5) that you would like TensorFlow to use inside the Docker container, then you should download and install the latest driver for your card from nvidia.com. You will also need to install NVidia Docker support: if you are using Docker 19.03 or above, you must install the nvidia-container-toolkit package, and for earlier versions, you must install nvidia-docker2.
Next, edit the docker-compose.yml file:
$ cd /path/to/project/handson-ml2/docker
$ edit docker-compose.yml # use your favorite editor- Replace
dockerfile: ./docker/Dockerfilewithdockerfile: ./docker/Dockerfile.gpu - Replace
image: ageron/handson-ml2:latestwithimage: ageron/handson-ml2:latest-gpu - If you want to use
docker-compose, you will need version 1.28 or above for GPU support, and you must uncomment the wholedeploysection indocker-compose.yml.
Prepare the image (once)
If you want to pull the prebuilt image from Docker Hub (this will download over 3.5 GB of compressed data):
$ docker pull ageron/handson-ml2:latest-gpuIf you prefer to build the image yourself:
$ cd /path/to/project/handson-ml2/docker
$ docker-compose buildRun the notebooks with `docker-compose` (version 1.28 or above)
If you have docker-compose version 1.28 or above, that's great! You can simply run:
$ cd /path/to/project/handson-ml2/docker
$ docker-compose up
[...]
or http://127.0.0.1:8888/?token=[...]Then point your browser to the URL and Jupyter should appear. If you then open or create a notebook and execute the following code, a list containing your GPU device(s) should be displayed (success!):
import tensorflow as tf
tf.config.list_physical_devices("GPU")To stop the server, just press Ctrl-C.
Run the notebooks without `docker-compose`
If you have a version of docker-compose earlier than 1.28, you will have to use docker run directly.
If you are using Docker 19.03 or above, you can run:
$ cd /path/to/project/handson-ml2
$ docker run --name handson-ml2 --gpus all -p 8888:8888 -p 6006:6006 --log-opt mode=non-blocking --log-opt max-buffer-size=50m -v `pwd`:/home/devel/handson-ml2 ageron/handson-ml2:latest-gpu /opt/conda/envs/tf2/bin/jupyter notebook --ip='0.0.0.0' --port=8888 --no-browserIf you are using an older version of Docker, then replace --gpus all with --runtime=nvidia.
Now point your browser to the displayed URL: Jupyter should appear, and you can open a notebook and run import tensorflow as tf and tf.config.list_physical_devices("GPU) as above to confirm that TensorFlow does indeed see your GPU device(s).
Lastly, to interrupt the server, press Ctrl-C, then run:
$ docker rm handson-ml2This will remove the container so you can start a new one later (but it will not remove the image or the notebooks, don't worry!).
Have fun!
Overview (`datasets/housing/README.md`)
California Housing
Source
This dataset is a modified version of the California Housing dataset available from Luís Torgo's page (University of Porto). Luís Torgo obtained it from the StatLib repository (which is closed now). The dataset may also be downloaded from StatLib mirrors.
This dataset appeared in a 1997 paper titled Sparse Spatial Autoregressions by Pace, R. Kelley and Ronald Barry, published in the Statistics and Probability Letters journal. They built it using the 1990 California census data. It contains one row per census block group. A block group is the smallest geographical unit for which the U.S. Census Bureau publishes sample data (a block group typically has a population of 600 to 3,000 people).
Tweaks
The dataset in this directory is almost identical to the original, with two differences:
- 207 values were randomly removed from the
total_bedroomscolumn, so we can discuss what to do with missing data. - An additional categorical attribute called
ocean_proximitywas added, indicating (very roughly) whether each block group is near the ocean, near the Bay area, inland or on an island. This allows discussing what to do with categorical data.
Note that the block groups are called "districts" in the Jupyter notebooks, simply because in some contexts the name "block group" was confusing.
Data description
>>> housing.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 20640 entries, 0 to 20639
Data columns (total 10 columns):
longitude 20640 non-null float64
latitude 20640 non-null float64
housing_median_age 20640 non-null float64
total_rooms 20640 non-null float64
total_bedrooms 20433 non-null float64
population 20640 non-null float64
households 20640 non-null float64
median_income 20640 non-null float64
median_house_value 20640 non-null float64
ocean_proximity 20640 non-null object
dtypes: float64(9), object(1)
memory usage: 1.6+ MB
>>> housing["ocean_proximity"].value_counts()
<1H OCEAN 9136
INLAND 6551
NEAR OCEAN 2658
NEAR BAY 2290
ISLAND 5
Name: ocean_proximity, dtype: int64
>>> housing.describe()
longitude latitude housing_median_age total_rooms \
count 16513.000000 16513.000000 16513.000000 16513.000000
mean -119.575972 35.639693 28.652335 2622.347605
std 2.002048 2.138279 12.576306 2138.559393
min -124.350000 32.540000 1.000000 6.000000
25% -121.800000 33.940000 18.000000 1442.000000
50% -118.510000 34.260000 29.000000 2119.000000
75% -118.010000 37.720000 37.000000 3141.000000
max -114.310000 41.950000 52.000000 39320.000000
total_bedrooms population households median_income
count 16355.000000 16513.000000 16513.000000 16513.000000
mean 534.885112 1419.525465 496.975050 3.875651
std 412.716467 1115.715084 375.737945 1.905088
min 2.000000 3.000000 2.000000 0.499900
25% 295.000000 784.000000 278.000000 2.566800
50% 433.000000 1164.000000 408.000000 3.541400
75% 644.000000 1718.000000 602.000000 4.745000
max 6210.000000 35682.000000 5358.000000 15.000100Overview (`datasets/jsb_chorales/README.md`)
Johann Sebastian Bach Chorales Dataset
Source
This dataset contains 382 chorales by Johann Sebastian Bach (in the public domain), where each chorale is composed of 100 to 640 chords with a temporal resolution of 1/16th. Each chord is composed of 4 integers, each indicating the index of a note on a piano, except for the value 0 which means "no note played".
This dataset is based on czhuang's JSB-Chorales-dataset (Jsb16thSeparated.npz) which used the train, validation, test split from Boulanger-Lewandowski (2012).
Motivation: I thought it would be nice to have a version of this dataset in CSV format.
Reference
Boulanger-Lewandowski, N., Vincent, P., & Bengio, Y. (2012). Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription. Proceedings of the 29th International Conference on Machine Learning (ICML-12), 1159–1166.
Usage
Download jsb_chorales.tgz and untar it:
$ tar xvzf jsb_chorales.tgzData structure
The dataset is split in three (train, valid, test), with a total of 382 CSV files:
$ tree
.
├── train
│ ├── chorale_000.csv
│ ├── chorale_001.csv
│ ├── chorale_002.csv
│ │ ...
│ ├── chorale_227.csv
│ └── chorale_228.csv
├── valid
│ ├── chorale_229.csv
│ ├── chorale_230.csv
│ ├── chorale_231.csv
│ │ ...
│ ├── chorale_303.csv
│ └── chorale_304.csv
└── test
├── chorale_305.csv
├── chorale_306.csv
├── chorale_307.csv
│ ...
├── chorale_380.csv
└── chorale_381.csvData sample
$ head train/chorale_000.csv
note0,note1,note2,note3
74,70,65,58
74,70,65,58
74,70,65,58
74,70,65,58
75,70,58,55
75,70,58,55
75,70,60,55
75,70,60,55
77,69,62,50Enjoy!
Overview (`datasets/lifesat/README.md`)
Life satisfaction and GDP per capita
Life satisfaction
Source
This dataset was obtained from the OECD's website at: http://stats.oecd.org/index.aspx?DataSetCode=BLI
Data description
Int64Index: 3292 entries, 0 to 3291
Data columns (total 17 columns):
"LOCATION" 3292 non-null object
Country 3292 non-null object
INDICATOR 3292 non-null object
Indicator 3292 non-null object
MEASURE 3292 non-null object
Measure 3292 non-null object
INEQUALITY 3292 non-null object
Inequality 3292 non-null object
Unit Code 3292 non-null object
Unit 3292 non-null object
PowerCode Code 3292 non-null int64
PowerCode 3292 non-null object
Reference Period Code 0 non-null float64
Reference Period 0 non-null float64
Value 3292 non-null float64
Flag Codes 1120 non-null object
Flags 1120 non-null object
dtypes: float64(3), int64(1), object(13)
memory usage: 462.9+ KBExample usage using python Pandas
>>> life_sat = pd.read_csv("oecd_bli_2015.csv", thousands=',')
>>> life_sat_total = life_sat[life_sat["INEQUALITY"]=="TOT"]
>>> life_sat_total = life_sat_total.pivot(index="Country", columns="Indicator", values="Value")
>>> life_sat_total.info()
<class 'pandas.core.frame.DataFrame'>
Index: 37 entries, Australia to United States
Data columns (total 24 columns):
Air pollution 37 non-null float64
Assault rate 37 non-null float64
Consultation on rule-making 37 non-null float64
Dwellings without basic facilities 37 non-null float64
Educational attainment 37 non-null float64
Employees working very long hours 37 non-null float64
Employment rate 37 non-null float64
Homicide rate 37 non-null float64
Household net adjusted disposable income 37 non-null float64
Household net financial wealth 37 non-null float64
Housing expenditure 37 non-null float64
Job security 37 non-null float64
Life expectancy 37 non-null float64
Life satisfaction 37 non-null float64
Long-term unemployment rate 37 non-null float64
Personal earnings 37 non-null float64
Quality of support network 37 non-null float64
Rooms per person 37 non-null float64
Self-reported health 37 non-null float64
Student skills 37 non-null float64
Time devoted to leisure and personal care 37 non-null float64
Voter turnout 37 non-null float64
Water quality 37 non-null float64
Years in education 37 non-null float64
dtypes: float64(24)
memory usage: 7.2+ KBGDP per capita
Source
Dataset obtained from the IMF's website at: http://goo.gl/j1MSKe
Data description
Int64Index: 190 entries, 0 to 189
Data columns (total 7 columns):
Country 190 non-null object
Subject Descriptor 189 non-null object
Units 189 non-null object
Scale 189 non-null object
Country/Series-specific Notes 188 non-null object
2015 187 non-null float64
Estimates Start After 188 non-null float64
dtypes: float64(2), object(5)
memory usage: 11.9+ KBExample usage using python Pandas
>>> gdp_per_capita = pd.read_csv(
... datapath+"gdp_per_capita.csv", thousands=',', delimiter='\t',
... encoding='latin1', na_values="n/a", index_col="Country")
...
>>> gdp_per_capita.rename(columns={"2015": "GDP per capita"}, inplace=True)--- METRICS ---
- Files Extracted: 5
- Estimated Token Budget: ~5830 tokens
- Canonical Reference: https://codewiki.google/github.com/ageron/handson-ml2