# Technical Documentation: jupyter/docker-stacks > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/jupyter/docker-stacks) ### Overview (`README.md`) # Jupyter Docker Stacks [](https://github.com/jupyter/docker-stacks/actions/workflows/docker.yml?query=branch%3Amain "Docker images build status") [](https://jupyter-docker-stacks.readthedocs.io/en/latest/ "Documentation build status") [](https://results.pre-commit.ci/latest/github/jupyter/docker-stacks/main "pre-commit.ci build status") [](https://discourse.jupyter.org/ "Jupyter Discourse Forum") [](https://mybinder.org/v2/gh/jupyter/docker-stacks/main?urlpath=lab/tree/README.ipynb "Launch a quay.io/jupyter/base-notebook container on mybinder.org") [](https://github.com/psf/black "Black code formatter") [](https://scorecard.dev/viewer/?uri=github.com/jupyter/docker-stacks "OpenSSF Scorecard") [](https://opensource.org/license/BSD-3-Clause "Modified BSD License") Jupyter Docker Stacks are a set of ready-to-run [Docker images](https://quay.io/organization/jupyter) containing Jupyter applications and interactive computing tools. You can use a stack image to do any of the following (and more): - Start a personal Jupyter Server with the JupyterLab frontend (default) - Run JupyterLab for a team using JupyterHub - Start a personal Jupyter Server with the Jupyter Notebook frontend in a local Docker container - Write your own project Dockerfile ## Quick Start You can [try the quay.io/jupyter/base-notebook image](https://mybinder.org/v2/gh/jupyter/docker-stacks/main?urlpath=lab/tree/README.ipynb) on . Otherwise, the examples below may help you get started if you [have Docker installed](https://docs.docker.com/get-started/get-docker/), know [which Docker image](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html) you want to use, and want to launch a single Jupyter Application in a container. The [User Guide on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/) describes additional uses and features in detail. ```{note} Since `2023-10-20` our images are only pushed to `Quay.io` registry. Older images are available on Docker Hub, but they will no longer be updated. ``` ### Example 1 This command pulls the `jupyter/scipy-notebook` image tagged `2026-07-28` from Quay.io if it is not already present on the local host. It then starts a container running a Jupyter Server with the JupyterLab frontend and exposes the container's internal port `8888` to port `10000` of the host machine: ```bash docker run -p 10000:8888 quay.io/jupyter/scipy-notebook:2026-07-28 ``` You can modify the host port to which the container's port is mapped by [changing the value of the `-p` option](https://docs.docker.com/engine/containers/run/#exposed-ports) to `-p 8888:8888`. Visiting `http://:10000/?token=` in a browser loads JupyterLab, where: - The `hostname` is the name of the computer running Docker - The `token` is the secret token printed in the console. The container remains intact for restart after the Server exits. ### Example 2 This command pulls the `jupyter/datascience-notebook` image tagged `2026-07-28` from Quay.io if it is not already present on the local host. It then starts an _ephemeral_ container running a Jupyter Server with the JupyterLab frontend and exposes the server on host port 10000. ```bash docker run -it --rm -p 10000:8888 -v "${PWD}":/home/jovyan/work quay.io/jupyter/datascience-notebook:2026-07-28 ``` The use of the `-v` flag in the command mounts the current working directory on the host (`${PWD}` in the example command) as `/home/jovyan/work` in the container. The server logs appear in the terminal. Visiting `http://:10000/?token=` in a browser loads JupyterLab. Due to the usage of [the `--rm` flag](https://docs.docker.com/reference/cli/docker/container/run/#rm) Docker automatically cleans up the container and removes the file system when the container exits, but any changes made to the `~/work` directory and its files in the container will remain intact on the host. [The `-i` flag](https://docs.docker.com/reference/cli/docker/container/run/#interactive) keeps the container's `STDIN` open, and lets you send input to the container through standard input. [The `-t` flag](https://docs.docker.com/reference/cli/docker/container/run/#tty) attaches a pseudo-TTY to the container. ```{note} By default, [jupyter's root_dir](https://jupyter-server.readthedocs.io/en/latest/other/full-config.html) is `/home/jovyan`. So, new notebooks will be saved there, unless you change the directory in the file browser. To change the default directory, you must specify `ServerApp.root_dir` by adding this line to the previous command: `start-notebook.py --ServerApp.root_dir=/home/jovyan/work`. ``` ## Choosing Jupyter frontend JupyterLab is the default for all the Jupyter Docker Stacks images. It is still possible to switch back to Jupyter Notebook (or to launch a different startup command). You can achieve this by passing the environment variable `DOCKER_STACKS_JUPYTER_CMD=notebook` (or any other valid `jupyter` subcommand) at container startup; more information is available in the [documentation](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html#alternative-commands). ## Resources - [Documentation on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/) - [Issue Tracker on GitHub](https://github.com/jupyter/docker-stacks/issues) - [Jupyter Discourse Forum](https://discourse.jupyter.org/) - [Jupyter Website](https://jupyter.org) - [Images on Quay.io](https://quay.io/organization/jupyter) ## Acknowledgments - Starting from `2022-07-05`, `aarch64` self-hosted runners were sponsored by [`@mathbunnyru`](https://github.com/mathbunnyru/). Please, consider [sponsoring his work](https://github.com/sponsors/mathbunnyru) on GitHub - Starting from `2023-10-31`, `aarch64` self-hosted runners are sponsored by an amazing [`2i2c non-profit organization`](https://2i2c.org) - Starting from `2025-02-11`, we use GitHub-hosted `aarch64` runners ## CPU Architectures - We publish containers for both `x86_64` and `aarch64` platforms - Single-platform images have either `aarch64-` or `x86_64-` tag prefixes, for example, `quay.io/jupyter/base-notebook:aarch64-python-3.13.14` - Starting from `2022-09-21`, we create multi-platform images (except `tensorflow-notebook`) - Starting from `2023-06-01`, we create a multi-platform `tensorflow-notebook` image as well - Starting from `2024-02-24`, we create CUDA enabled variants of `pytorch-notebook` image for `x86_64` platform - Starting from `2024-03-26`, we create CUDA enabled variant of `tensorflow-notebook` image for `x86_64` platform - Starting from `2025-12-02`, we create CUDA enabled variants of `pytorch-notebook` and `tensorflow-notebook` images for `aarch64` platform as well ## Using old images [](https://www.python.org/downloads/ "Python versions supported") This project only builds one set of images at a time. The weekly build (`latest` tag) is based on `Ubuntu 26.04` and `Python 3.13`. If you want to use the older `Ubuntu` and/or `Python` version, you can use the following images: ### Ubuntu 24.04 | Build Date | Python | Tag | | ---------- | ------ | -------------- | | 2024-10-22 | 3.11 | `b74418220768` | | 2025-08-11 | 3.12 | `82d322f00937` | | 2026-09-28 | 3.13 | `ac62373f9f60` | ### Ubuntu 22.04 | Build Date | Python | Tag | | ---------- | ------ | -------------- | | 2022-10-09 | 3.7 | `b86753318aa1` | | 2022-10-09 | 3.8 | `7285848c0a11` | | 2022-10-09 | 3.9 | `ed2908bbb62e` | | 2023-05-30 | 3.10 | `4d70cf8da953` | | 2024-08-26 | 3.11 | `00987883e58d` | ### Ubuntu 20.04 | Build Date | Python | Tag | | ---------- | ------ | -------------- | | 2022-10-09 | 3.7 | `1aac87eb7fa5` | | 2022-10-09 | 3.8 | `a374cab4fcb6` | | 2022-10-09 | 3.9 | `5ae537728c69` | | 2022-10-09 | 3.10 | `f3079808ca8c` | ## Contributing Please see the [the documentation](https://jupyter-docker-stacks.readthedocs.io/en/latest/) for information about how to contribute [issues](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/issues.html), [features](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/features.html), [recipes](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/recipes.html), [tests](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/tests.html), and [community-maintained stacks](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/stacks.html). ## LICENSE This project is licensed under the terms of the Modified BSD License (also known as New or Revised or 3-Clause BSD). ## About the Jupyter Development Team The Jupyter Development Team is the set of all contributors to the Jupyter project. This includes all of the Jupyter subprojects. The core team that coordinates development on GitHub can be found here: . ## Our Copyright Policy Jupyter uses a shared copyright model. Each contributor maintains copyright over their contributions to Jupyter. But, it is important to note that these contributions are typically only changes to the repositories. Thus, the Jupyter source code, in its entirety is not the copyright of any single person or institution. Instead, it is the collective copyright of the entire Jupyter Development Team. If individual contributors want to maintain a record of what changes/contributions they have specific copyright on, they should indicate their copyright in the commit message of the change, when they commit the change to one of the Jupyter repositories. With this in mind, the following banner should be used in any source code file to indicate the copyright and license terms: ```text # Copyright (c) Jupyter Development Team. # Distributed under the terms of the Modified BSD License. ``` ## Alternatives - [b-data](https://github.com/b-data)'s JupyterLab docker stacks - For [R](https://github.com/b-data/jupyterlab-r-docker-stack), [Python](https://github.com/b-data/jupyterlab-python-docker-stack), [MAX/Mojo](https://github.com/b-data/jupyterlab-mojo-docker-stack) and [Julia](https://github.com/b-data/jupyterlab-julia-docker-stack). With [code-server](https://github.com/coder/code-server) next to JupyterLab. Just Python – no [Conda](https://github.com/conda/conda) / [Mamba](https://github.com/mamba-org/mamba). - [rocker/binder](https://rocker-project.org/images/versioned/binder.html) - From the R focused [rocker-project](https://rocker-project.org), lets you run both RStudio and Jupyter either standalone or in a JupyterHub - [jupyter/repo2docker](https://github.com/jupyterhub/repo2docker) - Turn git repositories into Jupyter-enabled Docker Images - [openshift/source-to-image](https://github.com/openshift/source-to-image) - A tool for building artifacts from source code and injecting them into docker images - [jupyter-on-openshift/jupyter-notebooks](https://github.com/jupyter-on-openshift/jupyter-notebooks) - OpenShift compatible S2I builder for basic notebook images --- ### Overview (`examples/README.md`) # Examples These examples are not tested and might not work. Please, send PRs if you start using these examples and see some issues. --- ### Overview (`tagging/README.md`) # Docker stacks tagging and manifest creation Please, refer to the [tagging section of documentation](https://jupyter-docker-stacks.readthedocs.io/en/latest/maintaining/tagging.html) to see how tags and manifests are created. --- ### Overview (`tests/README.md`) # Docker stacks testing Please, refer to the [testing section of documentation](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/tests.html) to see how the tests are run. --- ### Overview (`examples/docker-compose/README.md`) # Docker Compose example This example demonstrate how to deploy docker-stack notebook containers to any Docker Machine-controlled host using Docker Compose. ## Prerequisites - [Docker Engine](https://docs.docker.com/engine/) 1.10.0+ - [Docker Machine](https://docs.docker.com/machine/) 0.6.0+ - [Docker Compose](https://docs.docker.com/compose/) 1.6.0+ See the [installation instructions](https://docs.docker.com/engine/installation/) for your environment. ## Quickstart Build and run a `jupyter/minimal-notebook` image on a VirtualBox VM on local desktop. ```bash # create a Docker Machine-controlled VirtualBox VM bin/vbox.sh mymachine # activate the docker machine eval "$(docker-machine env mymachine)" # build the notebook image on the machine notebook/build.sh # bring up the notebook container notebook/up.sh ``` To stop and remove the container: ```bash notebook/down.sh ``` ## FAQ ### How do I specify which docker-stack notebook image to deploy? You can customize the docker-stack notebook image to deploy by modifying the `notebook/Dockerfile`. For example, you can build and deploy a `jupyter/all-spark-notebook` by modifying the Dockerfile like so: ```dockerfile FROM quay.io/jupyter/all-spark-notebook # Your RUN commands and so on ``` Once you modify the Dockerfile, don't forget to rebuild the image. ```bash # activate the docker machine eval "$(docker-machine env mymachine)" notebook/build.sh ``` ### Can I run multiple notebook containers on the same VM? Yes. Set environment variables to specify unique names and ports when running the `up.sh` command. ```bash NAME=my-notebook PORT=9000 notebook/up.sh NAME=your-notebook PORT=9001 notebook/up.sh ``` To stop and remove the containers: ```bash NAME=my-notebook notebook/down.sh NAME=your-notebook notebook/down.sh ``` ### Where are my notebooks stored? The `up.sh` creates a Docker volume named after the notebook container with a `-work` suffix, e.g., `my-notebook-work`. ### Can multiple notebook containers share the same notebook volume? Yes. Set the `WORK_VOLUME` environment variable to the same value for each notebook. ```bash NAME=my-notebook PORT=9000 WORK_VOLUME=our-work notebook/up.sh NAME=your-notebook PORT=9001 WORK_VOLUME=our-work notebook/up.sh ``` ### How do I run over HTTPS? To run the Jupyter Server with a self-signed certificate, pass the `--secure` option to the `up.sh` script. You must also provide a password, which will be used to secure the Jupyter Server. You can specify the password by setting the `PASSWORD` environment variable, or by passing it to the `up.sh` script. ```bash PASSWORD=a_secret notebook/up.sh --secure # or notebook/up.sh --secure --password a_secret ``` ### Can I use Let's Encrypt certificate chains? Sure. If you want to secure access to publicly addressable notebook containers, you can generate a free certificate using the [Let's Encrypt](https://letsencrypt.org) service. This example includes the `bin/letsencrypt.sh` script, which runs the `letsencrypt` client to create a full-chain certificate and private key, and stores them in a Docker volume. ```{note} The script hard codes several `letsencrypt` options, one of which automatically agrees to the Let's Encrypt Terms of Service. ``` The following command will create a certificate chain and store it in a Docker volume named `mydomain-secrets`. ```bash FQDN=host.mydomain.com EMAIL=myemail@somewhere.com \ SECRETS_VOLUME=mydomain-secrets \ bin/letsencrypt.sh ``` Now run `up.sh` with the `--letsencrypt` option. You must also provide the name of the secrets volume and a password. ```bash PASSWORD=a_secret SECRETS_VOLUME=mydomain-secrets notebook/up.sh --letsencrypt # or notebook/up.sh --letsencrypt --password a_secret --secrets mydomain-secrets ``` Be aware that Let's Encrypt has a pretty [low rate limit per domain](https://community.letsencrypt.org/t/public-beta-rate-limits/4772/3) at the moment. You can avoid exhausting your limit by testing against the Let's Encrypt staging servers. To hit their staging servers, set the environment variable `CERT_SERVER=--staging`. ```bash FQDN=host.mydomain.com EMAIL=myemail@somewhere.com \ CERT_SERVER=--staging \ bin/letsencrypt.sh ``` Also, be aware that Let's Encrypt certificates are short-lived (90 days). If you need them for a longer period of time, you'll need to manually set up a cron job to run the renewal steps. (You can reuse the command above.) ### Can I deploy to any Docker Machine host? Yes, you should be able to deploy to any Docker Machine-controlled host. To make it easier to get up and running, this example includes scripts to provision Docker Machines to VirtualBox and IBM SoftLayer, but more scripts are welcome! To create a Docker machine using a VirtualBox VM on local desktop: ```bash bin/vbox.sh mymachine ``` To create a Docker machine using a virtual device on IBM SoftLayer: ```bash export SOFTLAYER_USER=my_softlayer_username export SOFTLAYER_API_KEY=my_softlayer_api_key export SOFTLAYER_DOMAIN=my.domain # Create virtual device bin/softlayer.sh myhost # Add DNS entry (SoftLayer DNS zone must exist for SOFTLAYER_DOMAIN) bin/sl-dns.sh myhost ``` ## Troubleshooting ### Unable to connect to VirtualBox VM on Mac OS X when using Cisco VPN client The Cisco VPN client blocks access to IP addresses that it does not know about, and may block access to a new VM if it is created while the Cisco VPN client is running. 1. Stop Cisco VPN client. (It does not allow modifications to route table). 2. Run `ifconfig` to list `vboxnet` virtual network devices. 3. Run `sudo route -nv add -net 192.168.99 -interface vboxnetX`, where X is the number of the virtual device assigned to the VirtualBox VM. 4. Start Cisco VPN client. --- ### Overview (`examples/make-deploy/README.md`) # Make deploy example This folder contains a Makefile and a set of supporting files demonstrating how to run a docker-stack notebook container on a docker-machine controlled host. ## Prerequisites - make 3.81+ - Ubuntu users: Be aware of [make 3.81 defect 483086](https://bugs.launchpad.net/ubuntu/+source/make-dfsg/+bug/483086) which exists in 14.04 LTS but is fixed in 15.04+ - docker-machine 0.5.0+ - docker 1.9.0+ ## Quickstart To show what's possible, here's how to run the `jupyter/minimal-notebook` on a brand-new local virtualbox. ```bash # create a new VM make virtualbox-vm NAME=dev # make the new VM the active docker machine eval $(docker-machine env dev) # pull a docker stack and build a local image from it make image # start a Server in a container make notebook ``` The last command will log the IP address and port to visit in your browser. ## FAQ ### Can I run multiple notebook containers on the same VM? Yes. Specify a unique name and port on the `make notebook` command. ```bash make notebook NAME=my-notebook PORT=9000 make notebook NAME=your-notebook PORT=9001 ``` ### Can multiple notebook containers share their notebook directory? Yes. ```bash make notebook NAME=my-notebook PORT=9000 WORK_VOLUME=our-work make notebook NAME=your-notebook PORT=9001 WORK_VOLUME=our-work ``` ### How do I run over HTTPS? Instead of `make notebook`, run `make self-signed-notebook PASSWORD=your_desired_password`. This target gives you a notebook with a self-signed certificate. ### That self-signed certificate is a pain. Let's Encrypt? Yes. Please. ```bash make letsencrypt FQDN=host.mydomain.com EMAIL=myemail@somewhere.com make letsencrypt-notebook ``` The first command creates a Docker volume named after the notebook container with a `-secrets` suffix. It then runs the `letsencrypt` client with a slew of options (one of which has you automatically agreeing to the Let's Encrypt Terms of Service, see the Makefile). The second command mounts the secrets volume and configures Jupyter to use the full-chain certificate and private key. Be aware: Let's Encrypt has a pretty [low rate limit per domain](https://community.letsencrypt.org/t/public-beta-rate-limits/4772/3) at the moment. You can avoid exhausting your limit by testing against the Let's Encrypt staging servers. To hit their staging servers, set the environment variable `CERT_SERVER=--staging`. ```bash make letsencrypt FQDN=host.mydomain.com EMAIL=myemail@somewhere.com CERT_SERVER=--staging ``` Also, keep in mind Let's Encrypt certificates are short-lived: 90 days at the moment. You'll need to manually set up a cron job to run the renewal steps at the moment. (You can reuse the first command above.) ### My pip/conda/apt-get installs disappear every time I restart the container. Can I make them permanent? ```bash # add your pip, conda, apt-get, etc. permanent features to the Dockerfile where # indicated by the comments in the Dockerfile vi Dockerfile make image make notebook ``` ### How do I upgrade my Docker container? ```bash make image DOCKER_ARGS=--pull make notebook ``` The first line pulls the latest version of the Docker image used in the local Dockerfile. Then it rebuilds the local Docker image containing any customizations you may have added to it. The second line kills your currently running notebook container, and starts a fresh one using the new image. ### Can I run on another VM provider other than VirtualBox? Yes. As an example, there's a `softlayer.makefile` included in this repo as an example. You would use it like so: ```bash make softlayer-vm NAME=myhost \ SOFTLAYER_DOMAIN=your_desired_domain \ SOFTLAYER_USER=your_user_id \ SOFTLAYER_API_KEY=your_api_key eval $(docker-machine env myhost) # optional, creates a real DNS entry for the VM using the machine name as the hostname make softlayer-dns SOFTLAYER_DOMAIN=your_desired_domain make image make notebook ``` If you'd like to add support for another docker-machine driver, use the `softlayer.makefile` as a template. ### Where are my notebooks stored? `make notebook` creates a Docker volume named after the notebook container with a `-data` suffix. ### Uh ... make? Yes, sorry Windows users. It got the job done for a simple example. We can certainly accept other deployment mechanism examples in the parent folder or in other repos. ### Are there any other options? Yes indeed. `cat` the Makefiles and look at the target parameters. --- ### Overview (`examples/openshift/README.md`) # OpenShift example This example provides templates for deploying the Jupyter Project docker-stacks images to OpenShift. ## Prerequisites Any OpenShift 3 environment. The templates were tested with OpenShift 3.7. It is believed they should work with at least OpenShift 3.6 or later. Do be aware that the Jupyter Project docker-stacks images are very large. The OpenShift environment you are using must provide sufficient quota on the per-user space for images and the file system for running containers. If the quota is too small, the pulling of the images to a node in the OpenShift cluster when deploying them, will fail due to lack of space. Even if the image is able to run, if the quota is only just larger than the space required for the image, you will not be able to install many packages into the container before running out of space. OpenShift Online, the public hosted version of OpenShift from Red Hat has a quota of only 3GB for the image and container file system. As a result, only the `minimal-notebook` can be started and there is little space remaining to install additional packages. Although OpenShift Online is suitable for demonstrating these templates work, what you can do in that environment will be limited due to the size of the images. If you want to experiment with using Jupyter Notebooks in an OpenShift environment, you should instead use [Minishift](https://www.openshift.org/minishift/). Minishift provides you the ability to run OpenShift in a virtual machine on your own local computer. ## Loading the Templates To load the templates, login to OpenShift from the command line and run: ```bash oc create -f https://raw.githubusercontent.com/jupyter/docker-stacks/main/examples/openshift/templates.json ``` This should create the `jupyter-notebook` template The template can be used from the command line using the `oc new-app` command, or from the OpenShift web console by selecting _Add to Project_. This `README` is only going to explain deploying from the command line. ## Deploying a Notebook To deploy a notebook from the command line using the template, run: ```bash oc new-app --template jupyter-notebook ``` The output will be similar to: ```lang-none --> Deploying template "jupyter/jupyter-notebook" to project jupyter Jupyter Notebook --------- Template for deploying Jupyter Notebook images. * With parameters: * APPLICATION_NAME=notebook * NOTEBOOK_IMAGE=quay.io/jupyter/minimal-notebook:latest * NOTEBOOK_PASSWORD=ded4d7cada554aa48e0db612e1ed1080 # generated --> Creating resources ... configmap "notebook-cfg" created deploymentconfig "notebook" created route "notebook" created service "notebook" created --> Success Access your application via route 'notebook-jupyter.b9ad.pro-us-east-1.openshiftapps.com' Run 'oc status' to view your app. ``` When no template parameters are provided, the name of the deployed notebook will be `notebook`. The image used will be: ```lang-none quay.io/jupyter/minimal-notebook:latest ``` A password you can use when accessing the notebook will be auto generated and is displayed in the output from running `oc new-app`. To see the hostname for accessing the notebook run: ```bash oc get routes ``` The output will be similar to: ```lang-none NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD notebook notebook-jupyter.abcd.pro-us-east-1.openshiftapps.com notebook 8888-tcp edge/Redirect None ``` A secure route will be used to expose the notebook outside the OpenShift cluster, so in this case the URL would be: ```lang-none https://notebook-jupyter.abcd.pro-us-east-1.openshiftapps.com/ ``` When prompted, enter the password for the notebook. ## Passing Template Parameters To override the name for the notebook, the image used, and the password, you can pass template parameters using the `--param` option. ```bash oc new-app --template jupyter-notebook \ --param APPLICATION_NAME=mynotebook \ --param NOTEBOOK_IMAGE=quay.io/jupyter/scipy-notebook:latest \ --param NOTEBOOK_PASSWORD=mypassword ``` You can deploy any of the Jupyter Project docker-stacks images. If you don't care what version of the image is used, add the `:latest` tag at the end of the image name, otherwise use the hash corresponding to the image version you want to use. ## Deleting the Notebook Instance To delete the notebook instance, run `oc delete` using a label selector for the application name. ```bash oc delete all,configmap --selector app=mynotebook ``` ## Adding Persistent Storage You can upload notebooks and other files using the web interface of the notebook. Any uploaded files or changes you make to them will be lost when the notebook instance is restarted. If you want to save your work, you need to add persistent storage to the notebook. To add persistent storage run: ```bash oc set volume dc/mynotebook --add \ --type=pvc --claim-size=1Gi --claim-mode=ReadWriteOnce \ --claim-name mynotebook-data --name data \ --mount-path /home/jovyan ``` When you have deleted the notebook instance, if using a persistent volume, you will need to delete it in a separate step. ```bash oc delete pvc/mynotebook-data ``` ## Customizing the Configuration If you want to set any custom configuration for the notebook, you can edit the config map created by the template. ```bash oc edit configmap/mynotebook-cfg ``` The `data` field of the config map contains Python code used as the `jupyter_server_config.py` file. If you are using a persistent volume, you can also create a configuration file at: ```lang-none /home/jovyan/.jupyter/jupyter_server_config.py ``` This will be merged at the end of the configuration from the config map. Because the configuration is Python code, ensure any indenting is correct. Any errors in the configuration file will cause the notebook to fail when starting. If the error is in the config map, edit it again to fix it and trigger a new deployment if necessary by running: ```bash oc rollout latest dc/mynotebook ``` If you make an error in the configuration file stored in the persistent volume, you will need to scale down the notebook, so it isn't running. ```bash oc scale dc/mynotebook --replicas 0 ``` Then run: ```bash oc debug dc/mynotebook ``` to run the notebook in debug mode. This will provide you with an interactive terminal session inside a running container, but the notebook will not have been started. Edit the configuration file in the volume to fix any errors and exit the terminal session. Start up the notebook again. ```bash oc scale dc/mynotebook --replicas 1 ``` ## Changing the Notebook Password The password for the notebook is supplied as a template parameter, or if not supplied will be automatically generated by the template. It will be passed into the container through an environment variable. If you want to change the password, you can do so by editing the environment variable on the deployment configuration. ```bash oc set env dc/mynotebook JUPYTER_NOTEBOOK_PASSWORD=mypassword ``` This will trigger a new deployment so ensure you have downloaded any work if not using a persistent volume. If using a persistent volume, you could instead set up a password in the file `/home/jovyan/.jupyter/jupyter_server_config.py` as per guidelines in . ## Deploying from a Custom Image If you want to deploy a custom variant of the Jupyter Project docker-stacks images, you can replace the image name with that of your own. If the image is not stored on Quay.io, but some other public image registry, prefix the name of the image with the image registry host details. If the image is in your OpenShift project, because you imported the image into OpenShift, or used the docker build strategy of OpenShift to build a derived custom image, you can use the name of the image stream for the image name, including any image tag if necessary. This can be illustrated by first importing an image into the OpenShift project. ```bash oc import-image quay.io/jupyter/datascience-notebook:latest --confirm ``` Then deploy it using the name of the image stream created. ```bash oc new-app --template jupyter-notebook \ --param APPLICATION_NAME=mynotebook \ --param NOTEBOOK_IMAGE=datascience-notebook \ --param NOTEBOOK_PASSWORD=mypassword ``` Importing an image into OpenShift before deploying it means that when a notebook is started, the image need only be pulled from the internal OpenShift image registry rather than Quay.io for each deployment. Because the images are so large, this can speed up deployments when the image hasn't previously been deployed to a node in the OpenShift cluster. --- ### Overview (`examples/source-to-image/README.md`) # Custom Jupyter Notebook images This example provides scripts for building custom Jupyter Notebook images containing notebooks, data files, and with Python packages required by the notebooks already installed. The scripts provided work with the Source-to-Image tool, and you can create the images from the command line on your own computer. Templates are also provided to enable running builds in OpenShift, as well as deploying the resulting image to OpenShift to make it available. The build scripts, when used with the Source-to-Image tool, provide similar capabilities to `repo2docker`. When builds are run under OpenShift with the supplied templates, it provides similar capabilities to `mybinder.org`, but where notebook instances are deployed in your existing OpenShift project and JupyterHub is not required. For separate examples of using JupyterHub with OpenShift, see the project: - ## Source-to-Image Project Source-to-Image (S2I) is an open source project which provides a tool for creating container images. It works by taking a base image, injecting additional source code or files into a running container created from the base image, and running a builder script in the container to process the source code or files to prepare the new image. Details on the S2I tool, and executable binaries for Linux, macOS and Windows, can be found on GitHub at: - The tool is standalone, and can be used on any system which provides a docker daemon for running containers. To provide an end-to-end capability to build and deploy applications in containers, support for S2I is also integrated into container platforms such as OpenShift. ## Getting Started with S2I As an example of how S2I can be used to create a custom image with a bundled set of notebooks, run: ```bash s2i build \ --scripts-url https://raw.githubusercontent.com/jupyter/docker-stacks/main/examples/source-to-image \ --context-dir docs/source/examples/Notebook \ https://github.com/jupyter/notebook \ quay.io/jupyter/minimal-notebook:latest \ notebook-examples ``` This example command will pull down the Git repository and build the image `notebook-examples` using the files contained in the `docs/source/examples/Notebook` directory of that Git repository. The base image which the files will be combined with is `quay.io/jupyter/minimal-notebook:latest`, but you can specify any of the Jupyter Project `docker-stacks` images as the base image. The resulting image from running the command can be seen by running `docker images` command: ```bash docker images # REPOSITORY TAG IMAGE ID CREATED SIZE # notebook-examples latest f5899ed1241d 2 minutes ago 2.59GB ``` You can now run the image. ```bash docker run --rm -p 8888:8888 notebook-examples ``` Open your browser on the URL displayed, and you will find the notebooks from the Git repository and can work with them. ## The S2I Builder Scripts Normally when using S2I, the base image would be S2I enabled and contain the builder scripts needed to prepare the image and define how the application in the image should be run. As the Jupyter Project `docker-stacks` images are not S2I enabled (although they could be), in the above example the `--scripts-url` option has been used to specify that the example builder scripts contained in this directory of this Git repository should be used. Using the `--scripts-url` option, the builder scripts can be hosted on any HTTP server, or you could also use builder scripts local to your computer file using an appropriate `file://` format URI argument to `--scripts-url`. The builder scripts in this directory of this repository are `assemble` and `run` and are provided as examples of what can be done. You can use the scripts as is, or create your own. The supplied `assemble` script performs a few key steps. The first steps copy files into the location they need to be when the image is run, from the directory where they are initially placed by the `s2i` command. ```bash cp -Rf /tmp/src/. "/home/${NB_USER}" rm -rf /tmp/src ``` The next steps are: ```bash if [ -f "/home/${NB_USER}/environment.yml" ]; then mamba env update --name root --file "/home/${NB_USER}/environment.yml" mamba clean --all -f -y else if [ -f "/home/${NB_USER}/requirements.txt" ]; then pip --no-cache-dir install -r "/home/${NB_USER}/requirements.txt" fi fi ``` This determines whether a `environment.yml` or `requirements.txt` file exists with the files and if so, runs the appropriate package management tool to install any Python packages listed in those files. This means that so long as a set of notebook files provides one of these files listing what Python packages they need, those packages will be automatically installed into the image, so they are available when the image is run. A final step is: ```bash fix-permissions "${CONDA_DIR}" fix-permissions "/home/${NB_USER}" ``` This fixes up permissions on any new files created by the build. This is necessary to ensure that when the image is run, you can still install additional files. This is important for when an image is run in `sudo` mode, or it is hosted in a more secure container platform such as Kubernetes/OpenShift where it will be run as a set user ID that isn't known in advance. As long as you preserve the first and last set of steps, you can do whatever you want in the `assemble` script to install packages, create files etc. Do be aware though that S2I builds do not run as `root` and so you cannot install additional system packages. If you need to install additional system packages, use a `Dockerfile` and normal `docker build` to first create a new custom base image from the Jupyter Project `docker-stacks` images, with the extra system packages, and then use that image with the S2I build to combine your notebooks and have Python packages installed. The `run` script in this directory is very simple and just runs the notebook application. ```bash exec start-notebook.py "$@" ``` ## Integration with OpenShift The OpenShift platform provides integrated support for S2I type builds. Templates are provided for using the S2I build mechanism with the scripts in this directory. To load the templates run: ```bash oc create -f https://raw.githubusercontent.com/jupyter/docker-stacks/main/examples/source-to-image/templates.json ``` This will create the templates: ```bash jupyter-notebook-builder jupyter-notebook-quickstart ``` The templates can be used from the OpenShift web console or command line. This `README` is only going to explain deploying from the command line. To use the OpenShift command line to build into an image, and deploy, the set of notebooks used above, run: ```bash oc new-app --template jupyter-notebook-quickstart \ --param APPLICATION_NAME=notebook-examples \ --param GIT_REPOSITORY_URL=https://github.com/jupyter/notebook \ --param CONTEXT_DIR=docs/source/examples/Notebook \ --param BUILDER_IMAGE=quay.io/jupyter/minimal-notebook:latest \ --param NOTEBOOK_PASSWORD=mypassword ``` You can provide a password using the `NOTEBOOK_PASSWORD` parameter. If you don't set that parameter, a password will be generated, with it being displayed by the `oc new-app` command. Once the image has been built, it will be deployed. To see the hostname for accessing the notebook, run `oc get routes`. ```lang-none NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD notebook-examples notebook-examples-jupyter.abcd.pro-us-east-1.openshiftapps.com notebook-examples 8888-tcp edge/Redirect None ``` As the deployment will use a secure connection, the URL for accessing the notebook in this case would be . If you only want to build an image but not deploy it, you can use the `jupyter-notebook-builder` template. You can then deploy it using the `jupyter-notebook` template provided with the [openshift](../openshift) examples directory. See the `openshift` examples directory for further information on customizing configuration for a Jupyter Notebook deployment and deleting a deployment. --- ### Overview (`images/all-spark-notebook/README.md`) # Jupyter Notebook Python, R, Spark Stack GitHub Actions in the project builds and pushes this image to the Registry. Please visit the project documentation site for help to use and contribute to this image and others. - [Jupyter Docker Stacks on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html) - [Selecting an Image :: Core Stacks :: jupyter/all-spark-notebook](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-all-spark-notebook) - [Image Specifics :: Apache Spark](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/specifics.html#apache-spark) --- ### Overview (`images/base-notebook/README.md`) # Base Jupyter Notebook Stack GitHub Actions in the project builds and pushes this image to the Registry. Please visit the project documentation site for help to use and contribute to this image and others. - [Jupyter Docker Stacks on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html) - [Selecting an Image :: Core Stacks :: jupyter/base-notebook](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-base-notebook) --- ### Overview (`images/datascience-notebook/README.md`) # Jupyter Notebook Data Science Stack GitHub Actions in the project builds and pushes this image to the Registry. Please visit the project documentation site for help to use and contribute to this image and others. - [Jupyter Docker Stacks on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html) - [Selecting an Image :: Core Stacks :: jupyter/datascience-notebook](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-datascience-notebook) --- ### Overview (`images/docker-stacks-foundation/README.md`) # Foundation Jupyter Stack GitHub Actions in the project builds and pushes this image to the Registry. Please visit the project documentation site for help to use and contribute to this image and others. - [Jupyter Docker Stacks on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html) - [Selecting an Image :: Core Stacks :: jupyter/docker-stacks-foundation](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-docker-stacks-foundation) --- ### Overview (`images/julia-notebook/README.md`) # Jupyter Notebook Julia Stack GitHub Actions in the project builds and pushes this image to the Registry. Please visit the project documentation site for help to use and contribute to this image and others. - [Jupyter Docker Stacks on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html) - [Selecting an Image :: Core Stacks :: jupyter/julia-notebook](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-julia-notebook) --- ### Overview (`images/minimal-notebook/README.md`) # Minimal Jupyter Notebook Stack GitHub Actions in the project builds and pushes this image to the Registry. Please visit the project documentation site for help to use and contribute to this image and others. - [Jupyter Docker Stacks on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html) - [Selecting an Image :: Core Stacks :: jupyter/minimal-notebook](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-minimal-notebook) --- ### Overview (`images/pyspark-notebook/README.md`) # Jupyter Notebook Python, Spark Stack GitHub Actions in the project builds and pushes this image to the Registry. Please visit the project documentation site for help to use and contribute to this image and others. - [Jupyter Docker Stacks on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html) - [Selecting an Image :: Core Stacks :: jupyter/pyspark-notebook](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-pyspark-notebook) - [Image Specifics :: Apache Spark](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/specifics.html#apache-spark) --- ### Overview (`images/pytorch-notebook/README.md`) # Jupyter Notebook PyTorch Deep Learning Stack GitHub Actions in the project builds and pushes this image to the Registry. Please visit the project documentation site for help to use and contribute to this image and others. - [Jupyter Docker Stacks on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html) - [Selecting an Image :: Core Stacks :: jupyter/pytorch-notebook](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-pytorch-notebook) --- ### Overview (`images/r-notebook/README.md`) # Jupyter Notebook R Stack GitHub Actions in the project builds and pushes this image to the Registry. Please visit the project documentation site for help to use and contribute to this image and others. - [Jupyter Docker Stacks on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html) - [Selecting an Image :: Core Stacks :: jupyter/r-notebook](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-r-notebook) --- ### Overview (`images/scipy-notebook/README.md`) # Jupyter Notebook Scientific Python Stack GitHub Actions in the project builds and pushes this image to the Registry. Please visit the project documentation site for help to use and contribute to this image and others. - [Jupyter Docker Stacks on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html) - [Selecting an Image :: Core Stacks :: jupyter/scipy-notebook](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-scipy-notebook) --- ### Overview (`images/tensorflow-notebook/README.md`) # Jupyter Notebook TensorFlow Deep Learning Stack GitHub Actions in the project builds and pushes this image to the Registry. Please visit the project documentation site for help to use and contribute to this image and others. - [Jupyter Docker Stacks on ReadTheDocs](https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html) - [Selecting an Image :: Core Stacks :: jupyter/tensorflow-notebook](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-tensorflow-notebook) - [Image Specifics :: Tensorflow](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/specifics.html#tensorflow) --- ### Index (`docs/index.rst`) .. include:: ../README.md :parser: myst_parser.sphinx_ Table of Contents ----------------- .. toctree:: :maxdepth: 2 :caption: User Guide using/selecting using/running using/common using/specifics using/recipes using/custom-images using/troubleshooting using/faq .. toctree:: :maxdepth: 2 :caption: Contributor Guide contributing/dev-setup contributing/issues contributing/features contributing/tests contributing/lint contributing/recipes contributing/stacks contributing/packages .. toctree:: :maxdepth: 2 :caption: Maintainer Guide maintaining/new-images-and-packages-policy maintaining/tagging maintaining/tasks .. toctree:: :maxdepth: 1 :caption: Changelog changelog .. toctree:: :maxdepth: 2 :caption: Getting Help Issue Tracker on GitHub Jupyter Discourse Forum Jupyter Website --- ### Requirements (`docs/requirements.txt`) docutils==0.22.4 myst-parser==5.1.0 sphinx==9.1.0 sphinx-book-theme==1.4.0 sphinx-copybutton==0.5.2 sphinx-last-updated-by-git==0.3.8 sphinxcontrib-mermaid==2.1.1 --- ### Home (`wiki/Home.md`) # Jupyter Docker Stacks build manifests Welcome! Please see [the documentation](https://jupyter-docker-stacks.readthedocs.io/en/latest/) for help with using, contributing to, and maintaining the Jupyter Docker stacks images. ## Build History This is an auto-generated index of information from the build system. In this index, you can find image tags, links to commits, and build manifests that describe the image. All the builds are grouped by year and then month. Note: we only store the last 4500 manifest files because of GitHub limits. That's why old manifest files might not be available. If you want to clone this repo and access the Git history, use the following command: `git clone git@github.com:{REPOSITORY}.wiki.git` In the tables below, each line represents: - `YYYY-MM`: link to a page with a list of images built - `Builds`: # of times build workflow finished - `Images`: # of single platform images pushed - `Commits`: # of commits made and a GitHub link --- ### Contributing Dev Setup (`docs/contributing/dev-setup.md`) # Development Setup This page covers everything you need to run the full CI-equivalent checks locally before opening a pull request. ## Prerequisites - [Docker](https://docs.docker.com/get-started/get-docker/) - Python 3.12+ - GNU Make - Git ## One-time setup ```bash # Clone the repository (SSH, recommended) git clone git@github.com:jupyter/docker-stacks.git # or via HTTPS git clone https://github.com/jupyter/docker-stacks.git cd docker-stacks # Install Python development dependencies pip install -r requirements-dev.txt # Install pre-commit hooks (runs linters automatically on git commit) pre-commit install --install-hooks ``` ## Pre-PR checklist Run these steps locally before pushing. ```bash # 1. Run all linters including mypy pre-commit run --all-files --hook-stage manual # 2. Build the image you changed make build/ # 3. Run tests for that image make test/ ``` ```{note} If the parent image isn't built locally, Docker pulls it from the registry. This means changes to a parent image won't be reflected unless you also rebuild it locally. ``` Replace `` with the image you modified (e.g., `docker-stacks-foundation`, `base-notebook`, `scipy-notebook`). ```{note} `make test/` runs the tests for that image plus every parent image's tests against it (e.g., testing `scipy-notebook` also runs `base-notebook` and `docker-stacks-foundation` test files against the `scipy-notebook` image). CI additionally runs the same test set against every downstream image, so a change in `docker-stacks-foundation` is verified across all images in CI even though locally you'd only run `make test/docker-stacks-foundation`. ``` ## Common examples ```bash # Working on the foundation image (start scripts, logging, etc.) pre-commit run --all-files --hook-stage manual make build/docker-stacks-foundation make test/docker-stacks-foundation # Working on the base notebook image pre-commit run --all-files --hook-stage manual make build/docker-stacks-foundation make build/base-notebook make test/base-notebook # Build and test every image (slow; mainly useful before opening a PR # that changes the foundation or base image) make build-all make test-all ``` --- ### Contributing Features (`docs/contributing/features.md`) # New Features Thank you for contributing to the Jupyter Docker Stacks! We review pull requests for new features (e.g., new packages, new scripts, new flags) to balance the value of the images to the Jupyter community with the cost of maintaining the images over time. ## Suggesting a New Feature Please follow the process below to suggest a new feature for inclusion in one of the core stacks: 1. Open a [GitHub feature request issue](https://github.com/jupyter/docker-stacks/issues/new?template=feature_request.yml) describing the feature you'd like to contribute. 2. Discuss with the maintainers whether the addition makes sense in [one of the core stacks](../using/selecting.md#core-stacks), as a [way to build a custom set of images](../using/custom-images.md), as a [recipe in the documentation](recipes.md), as a [community stack](stacks.md), or as something else entirely. ## Selection Criteria Roughly speaking, we evaluate new features based on the following criteria: - **Usefulness to Jupyter users**: Is the feature generally applicable across domains? Does it work with JupyterLab, Jupyter Notebook, JupyterHub, etc.? - **Fit with the image purpose**: Does the feature match the theme of the stack to which it will be added? Would it fit better in a new community stack? - **The complexity of build/runtime configuration**: How many lines of code does the feature require in one of the Dockerfiles or startup scripts? Does it require new scripts entirely? Do users need to adjust how they use the images? - **Impact on image metrics**: How many bytes does the feature and its dependencies add to the image(s)? How many minutes do they add to the build time? - **Ability to support the addition**: Can existing maintainers answer user questions and address future build issues? Are the contributors interested in helping with long-term maintenance? Can we write tests to ensure the feature continues to work over the years? ## Submitting a Pull Request If there's agreement that the feature belongs in one or more of the core stacks: 1. Implement the feature in a local clone of the `jupyter/docker-stacks` project. 2. Please, build the image locally before submitting a pull request. It shortens the debugging cycle by taking some load off GitHub Actions, which graciously provides free build services for open-source projects like this one. If you use `make`, call: ```bash make build/ ``` 3. [Submit a pull request](https://docs.github.com/en/pull-requests/how-tos/create-pull-requests/creating-a-pull-request) (PR) with your changes. 4. Watch for GitHub to report a build success or failure for your PR on GitHub. 5. Discuss changes with the maintainers and address any build issues. ```{note} If your changes cannot affect any image beyond `docker-stacks-foundation` and `base-notebook`, you can add `[FAST_BUILD]` to the PR title to skip building the rest of the images and speed up CI. ``` --- ### Contributing Issues (`docs/contributing/issues.md`) # Project Issues We appreciate you taking the time to report an issue you encountered while using the Jupyter Docker Stacks. Please review the following guidelines when reporting your problem. - Please use GitHub's "Report a Vulnerability" button under Security > Advisories on the appropriate repo, e.g. [report here for Jupyter Docker Stacks](https://github.com/jupyter/docker-stacks/security/advisories). You may also send an email to , but the GitHub reporting system is preferred. - If you think your problem is unique to the Jupyter Docker Stacks images, please search the [jupyter/docker-stacks issue tracker](https://github.com/jupyter/docker-stacks/issues?q=is%3Aissue%20) to see if someone else has already reported the same problem. If not, please open a [GitHub bug report issue](https://github.com/jupyter/docker-stacks/issues/new?template=bug_report.yml) and provide all the information requested in the issue template. Additionally, check the [Troubleshooting Common Problems](../using/troubleshooting.md) page in the documentation before submitting an issue. - If the issue you're seeing is with one of the open-source libraries included in the Docker images and is reproducible outside the images, please file a bug with the appropriate open-source project. - If you have a general question about how to use the Jupyter Docker Stacks in your environment, in conjunction with other tools, customizations, and so on, please post your question on the [Jupyter Discourse site](https://discourse.jupyter.org). --- ### Contributing Lint (`docs/contributing/lint.md`) # Lint To enforce some rules, **linters** are used in this project. Linters can be run either during the **development phase** (by the developer) or the **integration phase** (by GitHub Actions). To integrate and enforce this process in the project lifecycle, we are using **git hooks** through [pre-commit](https://pre-commit.com/). ## Using pre-commit hooks ### Pre-commit hook installation _pre-commit_ is a Python package that needs to be installed. To achieve this, use the generic task to install all Python development dependencies. ```sh # Install all development dependencies for the project pip install -r requirements-dev.txt # It can also be installed directly pip install "$(grep '^pre-commit==' requirements-dev.txt)" ``` Then the git hooks scripts configured for the project in `.pre-commit-config.yaml` need to be installed in the local git repository. ```sh pre-commit install ``` ### Run Now, _pre-commit_ (and so configured hooks) will run automatically on `git commit` on each changed file. However, you can also run it against all files manually. ```{note} Hadolint pre-commit uses Docker to run, so `docker` should be running while executing this command. ``` ```sh pre-commit run --all-files --hook-stage manual ``` ```{note} We're running `pre-commit` with `--hook-stage manual`, because `pre-commit` is run on modified files only, which doesn't work well with `mypy --follow-imports error`. More information can be found in [`.pre-commit-config.yaml` file](https://github.com/jupyter/docker-stacks/blob/main/.pre-commit-config.yaml) ``` ## Image Lint To comply with [Docker best practices](https://docs.docker.com/build/building/best-practices/), we are using the [Hadolint](https://github.com/hadolint/hadolint) tool to analyze each `Dockerfile`. ### Ignoring Rules Sometimes it is necessary to ignore [some rules](https://github.com/hadolint/hadolint#rules). The following rules are ignored by default for all images in the `.hadolint.yaml` file. - [`DL3006`][dl3006]: We use a specific policy to manage image tags. - The `docker-stacks-foundation` `FROM` clause is fixed but based on an argument (`ARG`). - Building downstream images from (`FROM`) the latest is done on purpose. - [`DL3008`][dl3008]: System packages are always updated (`apt-get`) to the latest version. - [`DL3013`][dl3013]: We always install the latest packages using `pip` The preferred way to ignore other rules is to flag them in the `Dockerfile`. You can use a special comment directly above the Dockerfile instruction you want to make an exception for. Ignore rule comments look like `# hadolint ignore=DL3001,SC1081`. For example: ```dockerfile FROM ubuntu # hadolint ignore=DL3003,SC1035 RUN cd /tmp && echo "hello!" ``` [dl3006]: https://github.com/hadolint/hadolint/wiki/DL3006 [dl3008]: https://github.com/hadolint/hadolint/wiki/DL3008 [dl3013]: https://github.com/hadolint/hadolint/wiki/DL3013 --- ### Contributing Packages (`docs/contributing/packages.md`) # Package Updates Generally, we do not pin package versions in our `Dockerfile`s. Dependency resolution is a difficult thing to do. All this means that packages might have old versions. Images are rebuilt weekly, so usually, packages receive updates quite frequently. ```{note} We pin major.minor version of Python, so this will stay the same even after invoking the `mamba update` command. ``` ## Outdated packages To help identify packages that can be updated, you can use the following helper tool. It will list all the outdated packages installed in the `Dockerfile`. Dependencies are filtered to display only the requested packages. ```bash make check-outdated/base-notebook # INFO test_outdated:test_outdated.py:80 3/8 (38%) packages could be updated # INFO test_outdated:test_outdated.py:82 # Package Current Newest # ---------- --------- -------- # conda 4.7.12 4.8.2 # jupyterlab 1.2.5 2.0.0 # python 3.7.4 3.8.2 ``` --- ### Contributing Recipes (`docs/contributing/recipes.md`) # New Recipes We welcome contributions of [recipes](../using/recipes.md), which are short examples of using, configuring, or extending the Docker Stacks for inclusion in the documentation site. Follow the process below to add a new recipe: 1. Open the `docs/using/recipes.md` source file. 2. Add a second-level Markdown heading naming your recipe at the bottom of the file (e.g., `## Slideshows with JupyterLab and RISE`) 3. Write the body of your recipe under the heading, including whatever command line, links, etc. you need. 4. If you have a Dockerfile, please put it in a `recipe_code` subdirectory. This file will be built automatically by [contributed-recipes workflow](https://github.com/jupyter/docker-stacks/blob/main/.github/workflows/contributed-recipes.yml). 5. [Submit a pull request](https://docs.github.com/en/pull-requests/how-tos/create-pull-requests/creating-a-pull-request) (PR) with your changes. Maintainers will respond and work with you to address any formatting or content issues. --- ### Contributing Stacks (`docs/contributing/stacks.md`) # Community Stacks We love to see the community create and share new Jupyter Docker images. We've put together a [cookiecutter project](https://github.com/jupyter/cookiecutter-docker-stacks) and the documentation below to help you get started defining, building, and sharing your Jupyter environments in Docker. Following these steps will: 1. Set up a project on GitHub containing a Dockerfile based on any image we provide. 2. Configure GitHub Actions to build and test your image when users submit pull requests to your repository. 3. Configure a container registry (Docker Hub in this example) to host your images for others to use. 4. Update the [list of community stacks](../using/selecting.md#community-stacks) in this documentation to include your image. This approach mirrors how we build and share the core stack images. Feel free to follow it or pave your path using alternative services and build tools. ## Creating a Project First, install [cookiecutter](https://github.com/cookiecutter/cookiecutter) using _pip_ or _mamba_: ```bash pip install cookiecutter # or mamba install cookiecutter ``` Run the cookiecutter command pointing to the [jupyter/cookiecutter-docker-stacks](https://github.com/jupyter/cookiecutter-docker-stacks) project on GitHub. ```bash cookiecutter https://github.com/jupyter/cookiecutter-docker-stacks.git ``` Enter a name for your new stack image. This will serve as both the git repository name and the part of the Docker image name after the slash. ```text stack_name [my-jupyter-stack]: ``` Enter the user or organization name under which this stack will reside on Docker Hub. You must have access to manage this Docker Hub organization to push images here. ```text stack_org [my-project]: ``` Select an image from the `jupyter/docker-stacks` project that will serve as the base for your new image. ```text stack_base_image [quay.io/jupyter/base-notebook]: ``` Enter a longer description of the stack for your README. ```text stack_description [my-jupyter-stack is a community-maintained Jupyter Docker Stack image]: ``` Create a GitHub repository to store your project. Initialize your project as a Git repository and push it to GitHub. ```bash cd git init git add . git commit -m 'Seed repo' git remote add origin git push -u origin main ``` ## Exploring GitHub Actions 1. By default, the `.github/workflows/docker.yaml` file generated by the cookiecutter template will trigger the CI pipeline whenever you push to your `main` branch and when any Pull Requests are made to your repository. For more details on this configuration, visit the [GitHub Actions documentation on triggers](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows). 2. Go to your repository and click on the **Actions** tab. From there, you can click on the workflows on the left-hand side of the screen. ```{note} The first run is expected to fail because we haven't yet added Docker credentials to push the image ``` 3. In the next screen, you will see information about the workflow run and duration. If you click the button with the workflow name again, you will see the logs for the workflow steps. ## Configuring Docker Hub ```{note} Jupyter Docker Stacks are hosted on Quay.io, but in this example, we show you how to host your image on Docker Hub. ``` Now, configure Docker Hub so that your stack image can be pushed to the Docker Hub repository whenever you merge a GitHub pull request to the main branch of your project. 1. Visit [https://hub.docker.com/](https://hub.docker.com/) and log in. 2. Create a new repository - make sure to use the correct namespace (account or organization). Enter the name of the image matching the one you entered when prompted with `stack_name` by the cookiecutter. 3. Enter a description for your image. 4. Click on your avatar in the top-right corner and select Account Settings. 5. Click on **Security** and then click on the **New Access Token** button. 6. Enter a meaningful name for your token and click on **Generate** 7. Copy the personal access token displayed on the next screen. ```{note} **You will not be able to see it again after you close the pop-up window**. ``` 8. Head back to your GitHub repository and click on the **Settings tab**. 9. Click on the **Secrets and variables->Actions** section and then on the **New repository secret** button in the top right corner. 10. Create a **DOCKERHUB_TOKEN** secret and paste the Personal Access Token from Docker Hub in the **value** field. 11. Now you're ready to go and you can restart a failed workflow. ## Defining Your Image Make edits to the Dockerfile in your project to add third-party libraries and configure Jupyter applications. Refer to the Dockerfiles for the core stacks (e.g., [jupyter/datascience-notebook](https://github.com/jupyter/docker-stacks/blob/main/images/datascience-notebook/Dockerfile)) to get a feel for what's possible and the best practices. [Submit pull requests](https://docs.github.com/en/pull-requests/how-tos/create-pull-requests/creating-a-pull-request) to your project repository on GitHub. Ensure your image builds correctly on GitHub Actions before merging to the main branch. After merging to the main branch, your image will be built and pushed to the Docker Hub automatically. ## Sharing Your Image Finally, if you'd like to add a link to your project to this documentation site, please do the following: 1. Fork the [jupyter/docker-stacks](https://github.com/jupyter/docker-stacks) GitHub repository. 2. Open the `docs/using/selecting.md` source file and locate the **Community Stacks** section in your fork. 3. Add a table entry with a link to your project, a binder link, and a short description of what your Docker image contains. 4. [Submit a pull request](https://docs.github.com/en/pull-requests/how-tos/create-pull-requests/creating-a-pull-request) (PR) with your changes. Maintainers will respond and work with you to address any formatting or content issues. --- ### Contributing Tests (`docs/contributing/tests.md`) # Image Tests We greatly appreciate Pull Requests that extend the automated tests that vet the basic functionality of the Docker images. ## How the Tests Work A [GitHub Actions workflow](https://github.com/jupyter/docker-stacks/blob/main/.github/workflows/docker-build-test-upload.yml) runs tests against pull requests submitted to the `jupyter/docker-stacks` repository. We use the `pytest` module to run tests on the image. `conftest.py` and `pytest.ini` in the `tests` folder define the environment in which tests are run. [Read `pytest` documentation](https://docs.pytest.org/en/latest/contents.html). The actual image-specific test files are located in folders like `tests/by_image//` (e.g., `tests/by_image/docker-stacks-foundation/`, etc.). ```{note} If your test is located in `tests/by_image//`, it will be run against the `jupyter/` image and against all the [images inherited from this image](../using/selecting.md#image-relationships). ``` Many tests make use of global [pytest fixtures](https://docs.pytest.org/en/latest/reference/fixtures.html) defined in the [conftest.py](https://github.com/jupyter/docker-stacks/blob/main/tests/conftest.py) file. ## Unit tests You can add a unit test if you want to run a Python script in one of our images. You should create a `tests/by_image//units/` directory, if it doesn't already exist, and put your file there. Files in this folder will be executed in the container when tests are run. You can see a [TensorFlow package example here](https://github.com/jupyter/docker-stacks/blob/HEAD/tests/by_image/tensorflow-notebook/units/unit_tensorflow.py). ## Contributing New Tests Please follow the process below to add new tests: 1. Add your test code to one of the modules in the `tests/by_image//` directory or create a new module. 2. Build one or more images you intend to test and run the tests locally. If you use `make`, call: ```bash make build/ make test/ ``` 3. [Submit a pull request](https://docs.github.com/en/pull-requests/how-tos/create-pull-requests/creating-a-pull-request) (PR) with your changes. 4. Watch for GitHub to report a build success or failure for your PR on GitHub. 5. Discuss changes with the maintainers and address any issues running the tests on GitHub. --- ### Maintaining New Images And Packages Policy (`docs/maintaining/new-images-and-packages-policy.md`) # New images / packages policy There are many things we consider while adding new images and packages. Here is a non-exhaustive list of things we do care about: 1. **Software health**, details, and maintenance status - reasonable versioning is adopted, and the version is considered to be stable - has been around for several years - the package maintains documentation - a changelog is actively maintained - a release procedure with helpful automation is established - multiple people are involved in the maintenance of the project - provides a `conda-forge` package besides a `pypi` package, where both are kept up to date - supports both `x86_64` and `aarch64` architectures 2. **Installation consequences** - GitHub Actions build time - Image sizes - All requirements should be installed as well 3. Jupyter Docker Stacks _**image fit**_ - new package or stack changes (or inherits from) the most suitable stack 4. **Software impact** for users of docker-stacks images - How this image can help existing users, or maybe reduce the need to build new images 5. Why it shouldn't just be a documented **recipe** 6. Impact on **security** - Does the package open additional ports, or add new web endpoints, that could be exploited? With all this in mind, we have a voting group, that consists of [@mathbunnyru](https://github.com/mathbunnyru), [@consideRatio](https://github.com/consideRatio), [@yuvipanda](https://github.com/yuvipanda), and [@manics](https://github.com/manics). This voting group is responsible for accepting or declining new packages and stacks. The change is accepted, if there are **at least 2 positive votes**. --- ### Maintaining Tagging (`docs/maintaining/tagging.md`) # Tags and manifests The main purpose of the source code in [the `tagging` folder](https://github.com/jupyter/docker-stacks/tree/main/tagging) is to properly write the tags file, the build history line, and the manifest for a single-platform image, apply these tags, and then merge single-platform images into one multi-arch image. ## What is a tag and a manifest A tag is a label attached to a Docker image identifying specific attributes or versions. For example, an image `jupyter/base-notebook` with Python 3.10.5 will have a full image name `quay.io/jupyter/base-notebook:python-3.10.5`. These tags are pushed to our [Quay.io registry](https://quay.io/organization/jupyter). A manifest is a description of important image attributes written in Markdown format. For example, we dump all `conda` packages with their versions into the manifest. ## Main principles - All images are organized in a hierarchical tree. More info on [image relationships](../using/selecting.md#image-relationships). - `TaggerInterface` and `ManifestInterface` are interfaces for functions to generate tags and manifest pieces by running commands in Docker containers. - Tags and manifests are reevaluated for each image in the hierarchy since values may change between parent and child images. - To tag an image and create its manifest and build history line, run `make hook/` (e.g., `make hook/base-notebook`). ## Utils ### DockerRunner `DockerRunner` is a helper class to easily run a docker container and execute commands inside this container: ```{literalinclude} tagging_examples/docker_runner.py :language: py :lines: 3- ``` ### GitHelper `GitHelper` methods are run in the current `git` repo and give the information about the last commit hash and commit message: ```{literalinclude} tagging_examples/git_helper.py :language: py :lines: 3- ``` The prefix of the commit hash (namely, 12 characters) is used as an image tag to make it easy to inherit from a fixed version of a docker image. ## Taggers and Manifests ### Tagger `Tagger` is a function that runs commands inside a docker container to calculate a tag for an image. All the taggers follow `TaggerInterface`: ```{literalinclude} ../../tagging/taggers/tagger_interface.py :language: py :start-at: TaggerInterface ``` So, the `tagger(container)` gets a docker container as an input and returns a tag. For example: ```{literalinclude} ../../tagging/taggers/sha.py :language: py :start-at: def ``` - `taggers/` subdirectory contains all taggers. - `apps/write_tags_file.py`, `apps/apply_tags.py`, and `apps/merge_tags.py` are Python executables used to write tags for an image, apply tags from a file, and create multi-arch images. ### Manifest All manifest functions except `build_info_manifest` follow `ManifestInterface` and `manifest(container)` method returns a piece of the manifest. ```{literalinclude} ../../tagging/manifests/manifest_interface.py :language: py :start-at: ManifestInterface ``` For example: ```{literalinclude} ../../tagging/manifests/apt_packages.py :language: py :start-at: def ``` where: - `quoted_output(container, cmd)` simply runs the command inside a container using `DockerRunner.exec_cmd` and wraps it to triple quotes to create a valid markdown piece. It also adds the command which was run to the markdown piece. - `manifests/` subdirectory contains all the manifests. - `apps/write_manifest.py` is a Python executable to create the build manifest and history line for an image. ## Images Hierarchy All images' dependencies on each other and what taggers and manifests are applicable to them are defined in `hierarchy/images_hierarchy.py`. `hierarchy/get_taggers.py` and `hierarchy/get_manifests.py` define functions to get the taggers and manifests for a specific image. --- ### Maintaining Tasks (`docs/maintaining/tasks.md`) # Maintainer Playbook ## Merging Pull Requests To build new images and publish them to the Registry, do the following: 1. Make sure GitHub Actions status checks pass for the PR. 2. Merge the PR. 3. Monitor the merge commit GitHub Actions status. ```{note} GitHub Actions are pretty reliable, so please investigate if some error occurs. Building Docker images in PRs is the same as building them in the default branch. The only difference is that single-platform images are pushed to Registry and then tags are merged for `x86_64` and `aarch64`. ``` 4. Merging another PR while builds in the main branch are still running does not cancel them - runs in the main branch (and scheduled runs) are queued instead (GitHub keeps the latest pending run, replacing any previously queued one). Still, it is tidier to let the pending builds finish before merging another PR: this way, you will know which commit might have broken the build and also have the correct values for the moving tags (like the `Python` version). ### `[FAST_BUILD]` pull requests If a PR title contains `[FAST_BUILD]`, the [docker.yml](https://github.com/jupyter/docker-stacks/blob/main/.github/workflows/docker.yml) workflow only builds and tests the `docker-stacks-foundation` and `base-notebook` images and skips all the other image jobs. This only affects `pull_request` events - pushes to the `main` branch and weekly scheduled builds always build all the images. Keep in mind: - Skipped jobs still satisfy the required status checks. So, before merging such a PR, **make sure** the `[FAST_BUILD]` title is appropriate for the changes made (i.e. the changes cannot affect the skipped images). - Changing a PR title does **not** re-trigger the checks (the workflow does not run on the `edited` activity type). Push a new commit or re-run the workflow jobs to refresh the checks after retitling a PR. ## Updating Python version When a new `Python` version is released, we wait for: - all the dependencies to be available (as wheels or in `conda-forge`). - the first `Python` patch release for this version. This allows us to avoid many bugs that can happen in a major release. ## Updating the Ubuntu Base Image `jupyter/docker-stacks-foundation` is based on the LTS Ubuntu docker image. We wait for the first point release of the new LTS Ubuntu before updating the version. Other images are directly or indirectly inherited from `jupyter/docker-stacks-foundation`. We rebuild our images automatically each week, which means they frequently receive updates. When there's a security fix in the Ubuntu base image, it's a good idea to manually trigger the rebuild of images [from the GitHub Actions workflow UI](https://github.com/jupyter/docker-stacks/actions/workflows/docker.yml). Pushing the `Run Workflow` button will trigger this process. ## Adding a New Core Image to the Registry ```{note} In general, we do not add new core images and ask contributors to either create a [recipe](../using/recipes.md) or [community stack](../contributing/stacks.md). We have a [policy](./new-images-and-packages-policy.md), which we consider when adding new images or new packages to existing images. ``` [Take a look at an example](https://github.com/jupyter/docker-stacks/pull/1936/files) of adding a new image. When there's a new stack definition, check before merging the PR: 1. PR includes an update to the stack overview diagram [in the documentation](../using/selecting.md#image-relationships). 2. PR updates the [Makefile](https://github.com/jupyter/docker-stacks/blob/main/Makefile). 3. Necessary Tagger(s)/Manifest(s) are added for the new image in the [tagging](https://github.com/jupyter/docker-stacks/tree/main/tagging) folder. 4. A new repository is created in the `jupyter` organization in the Registry, and it's named after the stack folder in the git repo. 5. Robot `Write` permission is added in the `Repository Settings`. ## Adding a New Registry Owner Account 1. Visit 2. Add the maintainer's username. ## Restarting a failed build If an automated build in GitHub Actions fails, you can restart the failed jobs on GitHub. You can also download the artifacts and investigate them for any issues. --- ### Using Common (`docs/using/common.md`) # Common Features Except for `jupyter/docker-stacks-foundation`, a container launched from any Jupyter Docker Stacks image runs a Jupyter Server with the JupyterLab frontend. The container does so by executing a `start-notebook.py` script. This script configures the internal container environment and then runs `jupyter lab`, passing any command-line arguments received. This page describes the options supported by the startup script and how to bypass it to run alternative commands. ## Jupyter Server Options You can pass [Jupyter Server options](https://jupyter-server.readthedocs.io/en/latest/operators/public-server.html) to the `start-notebook.py` script when launching the container. 1. For example, to secure the Jupyter Server with a [custom password](https://jupyter-server.readthedocs.io/en/latest/operators/public-server.html#preparing-a-hashed-password) hashed using `jupyter_server.auth.passwd()` instead of the default token, you can run the following (this hash was generated for the `my-password` password): ```bash docker run -it --rm -p 8888:8888 quay.io/jupyter/base-notebook \ start-notebook.py --PasswordIdentityProvider.hashed_password='argon2:$argon2id$v=19$m=10240,t=10,p=8$JdAN3fe9J45NvK/EPuGCvA$O/tbxglbwRpOFuBNTYrymAEH6370Q2z+eS1eF4GM6Do' ``` 2. To set the [base URL](https://jupyter-server.readthedocs.io/en/latest/operators/public-server.html#running-the-notebook-with-a-customized-url-prefix) of the Jupyter Server, you can run the following: ```bash docker run -it --rm -p 8888:8888 quay.io/jupyter/base-notebook \ start-notebook.py --ServerApp.base_url=/customized/url/prefix/ ``` ## Docker Options You may instruct the `start-notebook.py` script to customize the container environment before launching the Server. You do so by passing arguments to the `docker run` command. ### User-related configurations - `-e NB_USER=` - The desired username and associated home folder. The default value is `jovyan`. Setting `NB_USER` redefines the `jovyan` default user and ensures that the desired user has the correct file permissions for the new home directory created at `/home/`. For this option to take effect, you **must** run the container with `--user root`. Additionally, set `-e CHOWN_HOME=yes` if you also change `NB_UID`/`NB_GID` or mount the user's home directory from the host. If the working directory was under `/home/jovyan`, the startup script automatically updates it to the new `/home/` location. _Example usage:_ ```bash docker run -it --rm \ -p 8888:8888 \ --user root \ -e NB_USER="my-username" \ -e CHOWN_HOME=yes \ quay.io/jupyter/base-notebook ``` ```{note} If you set `NB_USER` to `root`, you must also set `NB_UID=0` and `NB_GID=0`; the `root` home dir will then be set to `/home/root`. See discussion [here](https://github.com/jupyter/docker-stacks/issues/2042). ``` - `-e NB_UID=` - Instructs the startup script to switch the numeric user ID of `${NB_USER}` to the given value. The default value is `1000`. This feature is useful when mounting host volumes with specific owner permissions. You **must** run the container with `--user root` for this option to take effect. (The `start.sh` script will run the container command as `${NB_USER}` via `sudo` after adjusting the user ID.) Instead, you might consider using the modern Docker-native options [`--user`](https://docs.docker.com/engine/containers/run/#user) and [`--group-add`](https://docs.docker.com/engine/containers/run/#additional-groups) - see the last bullet in this section for more details. - `-e NB_GID=` - Instructs the startup script to change the primary group of `${NB_USER}` to `${NB_GID}` (the new group is added with a name of `${NB_GROUP}` if it is defined. Otherwise, the group is named `${NB_USER}`). This feature is useful when mounting host volumes with specific group permissions. You **must** run the container with `--user root` for this option to take effect. (The `start.sh` script will run the container command as `${NB_USER}` via `sudo` after adjusting the group ID.) Instead, you might consider using modern Docker options `--user` and `--group-add` - see the last bullet in this section for more details. The user is added to the supplemental group `users` (gid 100) to grant write access to the home directory and `/opt/conda`. If you override the user/group logic, ensure the user stays in the group `users` if you want them to be able to modify files in the image. - `-e NB_GROUP=` - The name used for `${NB_GID}`, which defaults to `${NB_USER}`. This group name is only used if `${NB_GID}` is specified, and is completely optional: it only has a cosmetic effect. - `--user 5000 --group-add users` - Launches the container with a specific user ID and adds that user to the `users` group so that it can modify files in the default home directory and `/opt/conda`. You can use these arguments as alternatives to setting `${NB_UID}` and `${NB_GID}`. ### Permission-specific configurations - `-e NB_UMASK=` - Configures Jupyter to use a `umask` value different from the default `022`. For example, if setting `umask` to `002`, new files will be readable and writable by group members instead of the owner only. [Check this Wikipedia article](https://en.wikipedia.org/wiki/Umask) for an in-depth description of `umask` and suitable values for multiple needs. While the default `umask` value should be sufficient for most use cases, you can set the `NB_UMASK` value to fit your requirements. ```{note} When `NB_UMASK` is set, it only applies to the Jupyter process itself - you cannot use it to set a `umask` for additional files created during `run-hooks.sh`. For example, via `pip` or `conda`. If you need to set a `umask` for these, you **must** set the `umask` value for each command. ``` - `-e CHOWN_HOME=yes` - Instructs the startup script to change the `${NB_USER}` home directory owner and group to the current value of `${NB_UID}` and `${NB_GID}`. This change will take effect even if the user home directory is mounted from the host using `-v` as described below. The change is **not** applied recursively by default. You can modify the `chown` behavior by setting `CHOWN_HOME_OPTS` (e.g., `-e CHOWN_HOME_OPTS='-R'`). - `-e CHOWN_EXTRA=","` - Instructs the startup script to change the owner and group of each comma-separated container directory to the current value of `${NB_UID}` and `${NB_GID}`. The change is **not** applied recursively by default. You can modify the `chown` behavior by setting `CHOWN_EXTRA_OPTS` (e.g., `-e CHOWN_EXTRA_OPTS='-R'`). - `-e GRANT_SUDO=yes` - Instructs the startup script to grant the `NB_USER` user passwordless `sudo` capability. You do **not** need this option to allow the user to `conda` or `pip` install additional packages. This option is helpful for cases when you wish to give `${NB_USER}` the ability to install OS packages with `apt` or modify other root-owned files in the container. You **must** run the container with `--user root` for this option to take effect. (The `start.sh` script will run the container command as `${NB_USER}` via `sudo` after adding `${NB_USER}` to sudoers.) **You should only enable `sudo` if you trust the user or if the container runs on an isolated host.** ### Additional runtime configurations - `-e GEN_CERT=yes` - Instructs the startup script to generate a self-signed SSL certificate. Configures Jupyter Server to use it to accept encrypted HTTPS connections. - `-e DOCKER_STACKS_JUPYTER_CMD=` - Instructs the startup script to run `jupyter ${DOCKER_STACKS_JUPYTER_CMD}` instead of the default `jupyter lab` command. See [Switching back to the classic notebook or using a different startup command][switch_back] for available options. This setting is helpful in container orchestration environments where setting environment variables is more straightforward than changing command line parameters. - `-e RESTARTABLE=yes` - Runs Jupyter in a loop so that quitting Jupyter does not cause the container to exit. This may be useful when installing extensions that require restarting Jupyter. - `-v /some/host/folder/for/work:/home/jovyan/work` - Mounts a host machine directory as a folder in the container. This configuration is useful for preserving notebooks and other work even after the container has been destroyed. **You must grant the within-container notebook user or group (`NB_UID` or `NB_GID`) write access to the host directory (e.g., `sudo chown 1000 /some/host/folder/for/work`).** - `-e JUPYTER_ENV_VARS_TO_UNSET=ADMIN_SECRET_1,ADMIN_SECRET_2` - Unsets specified environment variables in the default startup script. The variables are unset after the hooks have been executed but before the command provided to the startup script runs. - `-e NOTEBOOK_ARGS="--log-level='DEBUG' --dev-mode"` - Adds custom options to the `jupyter` command. This way, the user could use any option supported by the `jupyter` subcommand. - `-e JUPYTER_PORT=8117` - Changes the port in the container that Jupyter is using to the value of the `${JUPYTER_PORT}` environment variable. This may be useful if you run multiple instances of Jupyter in swarm mode and want to use a different port for each instance. ## Startup Hooks You can further customize the container environment by adding shell scripts (`*.sh`) to be sourced or executables (`chmod +x`) to be run to the paths below: - `/usr/local/bin/start-notebook.d/` - handled **before** any of the standard options noted above is applied - `/usr/local/bin/before-notebook.d/` - handled **after** all the standard options noted above are applied and ran right before the Server launches Look at the [`run-hooks.sh` script](https://github.com/jupyter/docker-stacks/blob/main/images/docker-stacks-foundation/run-hooks.sh) and how it's used in the [`start.sh`](https://github.com/jupyter/docker-stacks/blob/main/images/docker-stacks-foundation/start.sh) script for execution details. ## SSL Certificates You may mount an SSL key and certificate file into a container and configure the Jupyter Server to use them to accept HTTPS connections. For example, to mount a host folder containing a `notebook.key` and `notebook.crt` and use them, you might run the following: ```bash docker run -it --rm -p 8888:8888 \ -v /some/host/folder:/etc/ssl/notebook \ quay.io/jupyter/base-notebook \ start-notebook.py \ --ServerApp.keyfile=/etc/ssl/notebook/notebook.key \ --ServerApp.certfile=/etc/ssl/notebook/notebook.crt ``` Alternatively, you may mount a single PEM file containing both the key and certificate. For example: ```bash docker run -it --rm -p 8888:8888 \ -v /some/host/folder/notebook.pem:/etc/ssl/notebook.pem \ quay.io/jupyter/base-notebook \ start-notebook.py \ --ServerApp.certfile=/etc/ssl/notebook.pem ``` In either case, Jupyter Server expects the key and certificate to be a **base64 encoded text file**. The certificate file or PEM may contain one or more certificates (e.g., server, intermediate, and root). For additional information about using SSL, see the following: - The [docker-stacks/examples](https://github.com/jupyter/docker-stacks/tree/main/examples) for information about how to use [Let's Encrypt](https://letsencrypt.org/) certificates when you run these stacks on a publicly visible domain. - The [`jupyter_server_config.py`](https://github.com/jupyter/docker-stacks/blob/main/images/base-notebook/jupyter_server_config.py) file for how this Docker image generates a self-signed certificate. - The [Jupyter Server documentation](https://jupyter-server.readthedocs.io/en/latest/operators/public-server.html#securing-a-jupyter-server) for best practices about securing a public Server in general. ## Alternative Commands ### Switching back to the classic notebook or using a different startup command JupyterLab, built on top of Jupyter Server, is now the default for all the images of the stack. However, switching back to the classic notebook or using a different startup command is still possible. You can achieve this by setting the environment variable `DOCKER_STACKS_JUPYTER_CMD` at container startup. The table below shows some options. Since `Jupyter Notebook v7`, `jupyter-server` is used as a backend. | `DOCKER_STACKS_JUPYTER_CMD` | Frontend | | --------------------------- | ---------------- | | `lab` (default) | JupyterLab | | `notebook` | Jupyter Notebook | | `nbclassic` | NbClassic | | `server` | None | | `retro`\* | RetroLab | ```{note} - Changing frontend for **JupyterHub singleuser image** is described in [JupyterHub docs](https://jupyterhub.readthedocs.io/en/latest/howto/configuration/config-user-env.html#switching-back-to-the-classic-notebook). - \* `retro` is not installed at this time, but it could be the case in the future or in a community stack. - Any other valid `jupyter` subcommand that starts the Jupyter Application can be used. ``` Example: ```bash # Run Jupyter Server with the Jupyter Notebook frontend docker run -it --rm \ -p 8888:8888 \ -e DOCKER_STACKS_JUPYTER_CMD=notebook \ quay.io/jupyter/base-notebook # Executing the command: start-notebook.py # Executing: jupyter notebook # ... # Use Jupyter NBClassic frontend docker run -it --rm \ -p 8888:8888 \ -e DOCKER_STACKS_JUPYTER_CMD=nbclassic \ quay.io/jupyter/base-notebook # Executing the command: start-notebook.py # Executing: jupyter nbclassic # ... ``` ### `start.sh` Most of the configuration options in the `start-notebook.py` script are handled by an internal `start.sh` script that automatically runs before the command provided to the container (it's set as the container entrypoint). This allows you to specify an arbitrary command that takes advantage of all these features. For example, to run the text-based `ipython` console in a container, do the following: ```bash docker run -it --rm quay.io/jupyter/base-notebook ipython ``` This script is handy when you derive a new Dockerfile from this image and install additional Jupyter applications with subcommands like `jupyter console`, `jupyter kernelgateway`, etc. ## Conda Environments The default Python 3.x [Conda environment](https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/environments.html) resides in `/opt/conda`. The `/opt/conda/bin` directory is part of the default `jovyan` user's `${PATH}`. That directory is also searched for binaries when run using `sudo` (`sudo my_binary` will search for `my_binary` in `/opt/conda/bin/`). The `jovyan` user has full read/write access to the `/opt/conda` directory. You can use either `mamba`, `pip`, or `conda` (`mamba` is recommended) to install new packages without any additional permissions. ```bash # install a package into the default (python 3.x) environment and clean it up after # the installation mamba install --yes some-package && \ mamba clean --all -f -y && \ fix-permissions "${CONDA_DIR}" && \ fix-permissions "/home/${NB_USER}" pip install --no-cache-dir some-package && \ fix-permissions "${CONDA_DIR}" && \ fix-permissions "/home/${NB_USER}" conda install --yes some-package && \ conda clean --all -f -y && \ fix-permissions "${CONDA_DIR}" && \ fix-permissions "/home/${NB_USER}" ``` ### Using Alternative Channels Conda is configured by default to use only the [`conda-forge`](https://anaconda.org/conda-forge) channel. However, you can use alternative channels, either one-shot by overwriting the default channel in the installation command or by configuring `mamba` to use different channels. The examples below show how to use the [anaconda default channels](https://repo.anaconda.com/pkgs/main) instead of `conda-forge` to install packages. ```bash # using defaults channels to install a package mamba install --channel defaults humanize # configure conda to add default channels at the top of the list conda config --system --prepend channels defaults # install a package mamba install --yes humanize && \ mamba clean --all -f -y && \ fix-permissions "${CONDA_DIR}" && \ fix-permissions "/home/${NB_USER}" ``` [switch_back]: #switching-back-to-the-classic-notebook-or-using-a-different-startup-command --- ### Using Custom Images (`docs/using/custom-images.md`) # Building a custom set of images This section describes how to build a custom set of images. It may be helpful if you need to change the Ubuntu or Python version, or to make a significant change to the build process itself. This project only builds one set of images at a time. If you want to use older images, [take a look here](../index.rst#using-old-images). ## Automating your build using template cookiecutter project If you wish to build your own image on top of one of our images and automate your build process, please, [take a look at cookiecutter template](../contributing/stacks.md). ## Custom arguments Our repository provides several customization points: - `ROOT_IMAGE` (docker argument) - the parent image for `docker-stacks-foundation` image - `PYTHON_VERSION` (docker argument) - the Python version to install in `docker-stacks-foundation` image - `REGISTRY`, `OWNER`, `BASE_IMAGE` (docker arguments) - they allow you to specify the parent image for all the other images - `REGISTRY`, `OWNER` (part of `env` in some GitHub workflows) - these allow you to properly tag and refer to images during the following steps: - [`build-test-upload`](https://github.com/jupyter/docker-stacks/blob/main/.github/workflows/docker-build-test-upload.yml) - [`contributed-recipes`](https://github.com/jupyter/docker-stacks/blob/main/.github/workflows/contributed-recipes.yml) - [`tag-push-merge`](https://github.com/jupyter/docker-stacks/blob/main/.github/workflows/docker-tag-push-merge.yml) These customization points can't be changed during runtime. Read more about [Docker build arguments](https://docs.docker.com/build/building/variables/#arg-usage-example) and [GitHub environment variables for a single workflow](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/store-information-in-variables#defining-environment-variables-for-a-single-workflow). ## Building stack images with custom arguments A selection of prebuilt images is available from [Quay.io](https://quay.io/organization/jupyter). However, it's impossible to cater to everybody's needs. For extensive customization with an automated build pipeline, you may wish to create a [community-maintained stack](../contributing/stacks.md). However, for minor customizations, this may be overkill. For example, you may wish to use the same Jupyter stacks but built on a different base image, or built with a different Python version. To achieve this you can use [Docker Bake](https://docs.docker.com/build/bake/) to build the stacks locally with custom arguments. ```{note} Custom arguments may result in build errors due to incompatibility. If so your use-case may require a fully customized stack. ``` As a basic example, suppose you want to build a custom image based on the `minimal-notebook` image using `Python 3.12`, and you have a Dockerfile like: ```{code-block} Dockerfile :caption: Dockerfile ARG BASE_IMAGE=minimal-notebook FROM $BASE_IMAGE ... ``` Then include the file below in your project: ```{literalinclude} recipe_code/docker-bake.custom-python.hcl :force: :language: hcl :caption: docker-bake.hcl ``` To build this stack, in the same directory run: ```bash docker buildx bake ``` Docker Bake then determines the correct build order from the `contexts` parameters and builds the stack as requested. This image can then be run the same way as any other image provided by this project, for example: ```bash docker run -it --rm -p 8888:8888 custom-jupyter ``` or referenced in a Docker Compose file. ## Forking our repository If for some reason, you need to change more things in our images, feel free to fork it and change it any way you want. If your customization is easy to backport to the main repo and might be helpful for other users, feel free to create a PR. It is almost always a great idea to keep your diff as small as possible and to merge/rebase the latest version of our repo in your project. --- ### Using Faq (`docs/using/faq.md`) # Frequently Asked Questions (FAQ) ## How to persist user data There are two types of data you might want to persist. 1. If you want to persist your environment (i.e. packages installed by `mamba`, `conda`, `pip`, `apt-get`, and so on), then you should create an inherited image and install packages only once while building your Dockerfile. Take a look at [an example of using `mamba` and `pip`](./recipes.md#using-mamba-install-recommended-or-pip-install-in-a-child-docker-image) in a child image. ```{note} If you install a package inside a running container (for example you run `pip install ` in a terminal), it won't be preserved when you next run your image. To make it work, install this package in your inherited image and rerun the `docker build` command. ``` 2. If you want to persist user data (files created by you, like `Python` scripts, notebooks, text files, and so on), then you should use a [Docker bind mount](https://docs.docker.com/engine/storage/bind-mounts/) or [Docker Volume](https://docs.docker.com/engine/storage/volumes/). You can find [an example of using a bind mount here](./running.md#example-2). There is also [a mount troubleshooting section](./troubleshooting.md#permission-denied-when-mounting-volumes) if you experience any issues. ## Why we don't add your favorite package We have lots of users with different packages they want to use. Adding them all is impossible, so we have several images to choose from. [Choose the image](selecting.md) that is closest to your needs, and feel free to [add your package on top of our images](recipes.md#using-mamba-install-recommended-or-pip-install-in-a-child-docker-image). ## Who is `jovyan` As described [in this issue comment](https://github.com/jupyter/docker-stacks/issues/358#issuecomment-288844834): ```text Jo·vy·an /ˈjōvēən/ noun – an inhabitant of Jupyter ``` `Jovyan` is often a special term used to describe members of the Jupyter community. It is also used as the user ID in the Jupyter Docker stacks or referenced in conversations. You can find more information on [the Jupyter Community documentation](https://docs.jupyter.org/en/latest/community/content-community.html#what-is-a-jovyan). ## How to give root permissions to the user We have a [recipe for enabling root permissions](recipes.md#using-sudo-within-a-container). --- ### Using Recipes (`docs/using/recipes.md`) # Contributed Recipes Users sometimes share interesting ways of using the Jupyter Docker Stacks. We encourage users to [contribute these recipes](../contributing/recipes.md) to the documentation in case they prove helpful to other community members by submitting a pull request to `docs/using/recipes.md`. The sections below capture this knowledge. All the recipes here assume you would like to use an image built by this project and install some things on top of it. If you would like to build a custom set of images, [take a look at the docs](custom-images.md). ## Using `sudo` within a container Password authentication is disabled for the `NB_USER` (e.g., `jovyan`). We made this choice to avoid distributing images with a weak default password that users ~~might~~ will forget to change before running a container on a publicly accessible host. You can grant the within-container `NB_USER` passwordless `sudo` access by adding `--user root` and `-e GRANT_SUDO=yes` to your Docker command line or appropriate container orchestrator config. For example: ```bash docker run -it --rm \ --user root \ -e GRANT_SUDO=yes \ quay.io/jupyter/base-notebook ``` **You should only enable `sudo` if you trust the user and/or if the container is running on an isolated host.** See [Docker security documentation](https://docs.docker.com/engine/security/userns-remap/) for more information about running containers as `root`. ## Using `mamba install` (recommended) or `pip install` in a Child Docker image Create a new Dockerfile like the one shown below. To use a requirements.txt file, first, create your `requirements.txt` file with the listing of packages desired. ```{literalinclude} recipe_code/mamba_install.dockerfile :language: docker ``` `pip` usage is similar: ```{literalinclude} recipe_code/pip_install.dockerfile :language: docker ``` Then build a new image. ```bash docker build --rm --tag my-custom-image . ``` You can then run the image as follows: ```bash docker run -it --rm \ -p 8888:8888 \ my-custom-image ``` ## Add a custom conda environment and Jupyter kernel The default version of `Python` that ships with the image may not be the version you want. The instructions below permit adding a conda environment with a different `Python` version and making it accessible to Jupyter. You may also use older images like `quay.io/jupyter/base-notebook:python-3.11`. We also maintain a [full build history](https://github.com/jupyter/docker-stacks/wiki). ```{literalinclude} recipe_code/custom_environment.dockerfile :language: docker ``` ## Dask JupyterLab Extension [Dask JupyterLab Extension](https://github.com/dask/dask-labextension) provides a JupyterLab extension to manage Dask clusters, as well as embed Dask's dashboard plots directly into JupyterLab panes. Create the Dockerfile as: ```{literalinclude} recipe_code/dask_jupyterlab.dockerfile :language: docker ``` And build the image as: ```bash docker build --rm --tag my-custom-image . ``` Once built, run using the command: ```bash docker run -it --rm \ -p 8888:8888 \ -p 8787:8787 \ my-custom-image ``` ## Let's Encrypt a Server ```{warning} This recipe is not tested and might be broken. ``` See the README for basic automation here which includes steps for requesting and renewing a Let's Encrypt certificate. Ref: ## Slideshows with JupyterLab and RISE [RISE](https://github.com/jupyterlab-contrib/rise): "Live" Reveal.js JupyterLab Slideshow Extension. ```{note} We're providing the recipe to install the JupyterLab extension. You can find the original Jupyter Notebook extension [here](https://github.com/damianavila/RISE) ``` ```{literalinclude} recipe_code/rise_jupyterlab.dockerfile :language: docker ``` ## xgboost ```{literalinclude} recipe_code/xgboost.dockerfile :language: docker ``` ## JAX with CUDA [JAX](https://docs.jax.dev/en/latest/) GPU support via the bundled NVIDIA CUDA wheels. Run with `--gpus all` and a host NVIDIA driver new enough for CUDA 13. For CUDA 12 instead, swap `jax[cuda13]` for `jax[cuda12]`. ```{literalinclude} recipe_code/jax_cuda.dockerfile :language: docker ``` ## Running behind an nginx proxy ```{warning} This recipe is not tested and might be broken. ``` Sometimes it is helpful to run the Jupyter instance behind an nginx proxy, for example: - you would prefer to access the notebook at a server URL with a path (`https://example.com/jupyter`) rather than a port (`https://example.com:8888`) - you may have many services in addition to Jupyter running on the same server and want nginx to help improve server performance in managing the connections Here is a [quick example of NGINX configuration](https://gist.github.com/cboettig/8643341bd3c93b62b5c2) to get started. You'll need a server, a `.crt`, and a `.key` file for your server, and `docker` & `docker-compose` installed. Then download the files at that gist and run `docker-compose up` to test it out. Customize the `nginx.conf` file to set the desired paths and add other services. ## Host volume mounts and notebook errors If you are mounting a host directory as `/home/jovyan/work` in your container, and you receive permission errors or connection errors when you create a notebook, be sure that the `jovyan` user (`UID=1000` by default) has read/write access to the directory on the host. Alternatively, specify the UID of the `jovyan` user on container startup using the `-e NB_UID` option described in the [Common Features, Docker Options section](common.md#docker-options) Ref: ## Manpage installation Most images, including our Ubuntu base image, ship without manpages installed to save space. You can use the following Dockerfile to inherit from one of our images to enable manpages: ```{literalinclude} recipe_code/manpage_install.dockerfile :language: docker ``` Adding the documentation on top of the existing image wastes a lot of space and requires reinstalling every system package, which can take additional time and bandwidth. Enabling manpages in the base Ubuntu layer prevents this image bloat. To achieve this, use the previous `Dockerfile`'s commands with the original `ubuntu` image as your base image: ```dockerfile FROM ubuntu:26.04 ``` Be sure to check the current base image in `jupyter/docker-stacks-foundation` before building. ## JupyterHub We also have contributed recipes for using JupyterHub. ### Use JupyterHub's DockerSpawner You can find [an example of using DockerSpawner](https://github.com/jupyterhub/jupyterhub-deploy-docker/tree/main/basic-example). ### Containers with a specific version of JupyterHub The version of `jupyterhub` in your image should match the version of JupyterHub itself. To use a specific version of JupyterHub, do the following: ```{literalinclude} recipe_code/jupyterhub_version.dockerfile :language: docker ``` ## Spark A few suggestions have been made regarding using Docker Stacks with Spark. ### Using PySpark with AWS S3 ```{warning} This recipe is not tested and might be broken. ``` Using Spark session for Hadoop 2.7.3 ```python import os # To figure out what version of Hadoop, run: # ls /usr/local/spark/jars/hadoop* os.environ["PYSPARK_SUBMIT_ARGS"] = ( '--packages "org.apache.hadoop:hadoop-aws:2.7.3" pyspark-shell' ) import pyspark myAccessKey = input() mySecretKey = input() spark = ( pyspark.sql.SparkSession.builder.master("local[*]") .config("spark.hadoop.fs.s3a.access.key", myAccessKey) .config("spark.hadoop.fs.s3a.secret.key", mySecretKey) .getOrCreate() ) df = spark.read.parquet("s3://myBucket/myKey") ``` Using Spark context for Hadoop 2.6.0 ```python import os os.environ["PYSPARK_SUBMIT_ARGS"] = ( "--packages com.amazonaws:aws-java-sdk:1.10.34,org.apache.hadoop:hadoop-aws:2.6.0 pyspark-shell" ) import pyspark sc = pyspark.SparkContext("local[*]") from pyspark.sql import SQLContext sqlContext = SQLContext(sc) hadoopConf = sc._jsc.hadoopConfiguration() myAccessKey = input() mySecretKey = input() hadoopConf.set("fs.s3.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem") hadoopConf.set("fs.s3.awsAccessKeyId", myAccessKey) hadoopConf.set("fs.s3.awsSecretAccessKey", mySecretKey) df = sqlContext.read.parquet("s3://myBucket/myKey") ``` Ref: ### Using Local Spark JARs ```{warning} This recipe is not tested and might be broken. ``` ```python import os os.environ["PYSPARK_SUBMIT_ARGS"] = ( "--jars /home/jovyan/spark-streaming-kafka-assembly_2.10-1.6.1.jar pyspark-shell" ) import pyspark from pyspark.streaming.kafka import KafkaUtils from pyspark.streaming import StreamingContext sc = pyspark.SparkContext() ssc = StreamingContext(sc, 1) broker = "" directKafkaStream = KafkaUtils.createDirectStream( ssc, ["test1"], {"metadata.broker.list": broker} ) directKafkaStream.pprint() ssc.start() ``` Ref: ### Using spark-packages.org ```{warning} This recipe is not tested and might be broken. ``` If you'd like to use packages from [spark-packages](https://repos.spark-packages.org/), see [https://gist.github.com/parente/c95fdaba5a9a066efaab](https://gist.github.com/parente/c95fdaba5a9a066efaab) for an example of how to specify the package identifier in the environment before creating a SparkContext. Ref: ### Use jupyter/all-spark-notebooks with an existing Spark/YARN cluster ```{warning} This recipe is not tested and might be broken. ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Credit: [britishbadger](https://github.com/britishbadger) from [docker-stacks/issues/369](https://github.com/jupyter/docker-stacks/issues/369) ## Run Server inside an already secured environment (i.e., with no token) The default security is very good. There are use cases, encouraged by containers, where the Jupyter container and the system it runs within lie inside the security boundary. It is convenient to launch the server without a password or token in these use cases. In this case, you should use the `start-notebook.py` script to launch the server with no token: For JupyterLab: ```bash docker run -it --rm \ quay.io/jupyter/base-notebook \ start-notebook.py --IdentityProvider.token='' ``` For Jupyter Notebook: ```bash docker run -it --rm \ -e DOCKER_STACKS_JUPYTER_CMD=notebook \ quay.io/jupyter/base-notebook \ start-notebook.py --IdentityProvider.token='' ``` ## Enable nbclassic-extension spellchecker for markdown (or any other nbclassic-extension) ```{note} This recipe only works for NBClassic with Jupyter Notebook < 7. It is recommended to use [jupyterlab-spellchecker](https://github.com/jupyterlab-contrib/spellchecker) in modern environments. ``` ```{literalinclude} recipe_code/spellcheck_notebook_v6.dockerfile :language: docker ``` ## Enable Delta Lake in Spark notebooks ```{warning} This recipe is not tested and might be broken. ``` Please note that the [Delta Lake](https://delta.io/) packages are only available for Spark version > `3.0`. By adding the properties to `spark-defaults.conf`, the user no longer needs to enable Delta support in each notebook. ```dockerfile FROM quay.io/jupyter/pyspark-notebook RUN mamba install --yes 'delta-spark' && \ mamba clean --all -f -y && \ fix-permissions "${CONDA_DIR}" && \ fix-permissions "/home/${NB_USER}" USER root RUN cat <> "${SPARK_HOME}/conf/spark-defaults.conf" spark.sql.extensions io.delta.sql.DeltaSparkSessionExtension spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog EOF USER ${NB_UID} # Trigger download of delta lake files RUN python <. The container runs in the background until stopped and/or removed by additional Docker commands: ```bash # stop the container docker stop notebook # notebook # remove the container permanently docker rm notebook # notebook ``` ## Using the Podman CLI An alternative to using the Docker CLI is to use the Podman CLI. Podman is mostly compatible with Docker. ### Podman example If we use Podman instead of Docker in the situation given in _Example 2_, it will look like this: The example makes use of rootless Podman; in other words, the Podman command is run from a regular user account. In a Bash shell, set the shell variables _uid_ and _gid_ to the UID and GID of the user _jovyan_ in the container. ```bash uid=1000 gid=100 ``` This command pulls the `quay.io/jupyter/r-notebook` image tagged `2026-07-28` from Quay.io if it is not already present on the local host. It then starts a container running a Jupyter Server with the JupyterLab frontend and exposes the server on host port 10000. The server logs appear in the terminal and include a URL to the server but with the internal container port (8888) instead of the correct host port (10000). ```bash podman run -it --rm -p 10000:8888 \ -v "${PWD}":/home/jovyan/work --user $uid:$gid \ --userns keep-id:uid=${uid},gid=${gid} \ quay.io/jupyter/r-notebook:2026-07-28 ``` If the commands fail with the error message: `Error: unrecognized namespace mode keep-id:uid=1000,gid=100 passed` see [troubleshooting tip](https://github.com/containers/podman/blob/main/troubleshooting.md#39-podman-run-fails-with-error-unrecognized-namespace-mode-keep-iduid1000gid1000-passed). ```{note} The `podman run` command in the example above maps all subuids and subgids of the user into the container. That works fine but is actually more than needed. The `podman run` option `--userns=auto` will, for instance, not be possible to use as long as there are no unused subuids and subgids available. The example could be improved by investigating more in detail which UIDs and GIDs need to be available in the container and then only map them, for example by setting the size option (`--userns keep-id:uid=${uid},gid=${gid},size=$size`). ``` Pressing `Ctrl-C` twice shuts down the Server and immediately destroys the container. New files and changes in `~/work` in the container will be preserved. Any other changes made in the container will be lost. ## Using Binder A [Binder](https://mybinder.org/) is a service that allows you to create and share custom computing environments for projects in version control. You can use any of the Jupyter Docker Stacks images as a basis for a Binder-compatible Dockerfile. See the [docker-stacks example](https://mybinder.readthedocs.io/en/latest/examples/sample_repos.html#using-a-docker-image-from-the-jupyter-docker-stacks-repository) and [Using a Dockerfile](https://mybinder.readthedocs.io/en/latest/tutorials/dockerfile.html) section in the [Binder documentation](https://mybinder.readthedocs.io/en/latest/index.html) for instructions. ## Using JupyterHub You can configure JupyterHub to launch Docker containers from the Jupyter Docker Stacks images. If you've been following the [Zero to JupyterHub with Kubernetes](https://z2jh.jupyter.org/en/latest/) guide, see the [Use an existing Docker image](https://z2jh.jupyter.org/en/latest/jupyterhub/customizing/user-environment.html#choose-and-use-an-existing-docker-image) section for details. If you have a custom JupyterHub deployment, see the [Picking or building a Docker image](https://jupyterhub-dockerspawner.readthedocs.io/en/latest/docker-image.html) instructions for the [dockerspawner](https://github.com/jupyterhub/dockerspawner) instead. ## Using Other Tools and Services You can use the Jupyter Docker Stacks with any Docker-compatible technology (e.g., [Docker Compose](https://docs.docker.com/compose/), [docker-py](https://github.com/docker/docker-py), or your favorite cloud container service). See the documentation of the tool, library, or service for details about how to reference, configure, and launch containers from these images. --- ### Using Selecting (`docs/using/selecting.md`) # Selecting an Image - [Core Stacks](#core-stacks) - [Image Relationships](#image-relationships) - [Community Stacks](#community-stacks) > **Pull vs Build: Quick Guidance** > > Images are published on **Quay.io** registry and most users should start by pulling an existing image that closely matches their needs. > Pulling a pre-built image is fast and suitable for common use cases such as running notebooks, teaching, or standard data science workflows. > > Building images locally is recommended only when additional customization is required, for example: > > - adding system-level packages that cannot be installed at runtime, > - preinstalling large or complex language-specific libraries, > - creating organization-specific images or heavy customization for CI. > > If unsure, try an existing Quay.io image first; if it falls short, follow the custom images guide to build a tailored image. Using one of the Jupyter Docker Stacks requires two choices: 1. Which Docker image you wish to use 2. How you wish to start Docker containers from that image This section provides details about the first. ## Core Stacks The Jupyter team maintains a set of Docker image definitions in the GitHub repository. The following sections describe these images, including their contents, relationships, and versioning strategy. ### jupyter/docker-stacks-foundation [Source on GitHub](https://github.com/jupyter/docker-stacks/tree/main/images/docker-stacks-foundation) | [Dockerfile commit history](https://github.com/jupyter/docker-stacks/commits/main/images/docker-stacks-foundation/Dockerfile) | [Quay.io image tags](https://quay.io/repository/jupyter/docker-stacks-foundation?tab=tags) `jupyter/docker-stacks-foundation` is a small image supporting a majority of [options common across all core stacks](common.md). It is the basis for all other stacks on which Jupyter-related applications can be built (e.g., kernel-based containers, [nbclient](https://github.com/jupyter/nbclient) applications, etc.). As such, it does not contain application-level software like JupyterLab, Jupyter Notebook, or JupyterHub. It contains: - Package managers - [conda](https://github.com/conda/conda): "cross-platform, language-agnostic binary package manager". - [mamba](https://github.com/mamba-org/mamba): "reimplementation of the conda package manager in C++". We use this package manager by default when installing packages. - Unprivileged user `jovyan` (`uid=1000`, configurable, [see options in the common features section](./common.md) of this documentation) in group `users` (`gid=100`) with ownership over the `/home/jovyan` and `/opt/conda` paths - `tini` and a `start.sh` script as the container entry point - useful for running alternative commands in the container as applications are added (e.g. `ipython`, `jupyter kernelgateway`, `jupyter lab`) - A `run-hooks.sh` script, which can source/run files in a given directory - Options for a passwordless sudo - Common system libraries like `ca-certificates`, `locales` - `wget` to download external files - No preinstalled scientific computing packages ### jupyter/base-notebook [Source on GitHub](https://github.com/jupyter/docker-stacks/tree/main/images/base-notebook) | [Dockerfile commit history](https://github.com/jupyter/docker-stacks/commits/main/images/base-notebook/Dockerfile) | [Quay.io image tags](https://quay.io/repository/jupyter/base-notebook?tab=tags) `jupyter/base-notebook` adds base Jupyter Applications like JupyterLab, Jupyter Notebook, JupyterHub, and NBClassic and serves as the basis for all other stacks besides `jupyter/docker-stacks-foundation`. It contains: - Everything in `jupyter/docker-stacks-foundation` - Minimally functional Server (e.g., no LaTeX support for saving notebooks as PDFs) - `notebook`, `nbclassic`, `jupyterhub-singleuser`, and `jupyterlab` packages - A `start-notebook.py` script as the default command - A `start-singleuser.py` script useful for launching containers in JupyterHub - Options for a self-signed HTTPS certificate ```{warning} `jupyter/base-notebook` also contains `start-notebook.sh` and `start-singleuser.sh` files to maintain backward compatibility. External config that explicitly refers to those files should instead update to refer to `start-notebook.py` and `start-singleuser.py`. The shim `.sh` files will be removed at some future date. ``` ### jupyter/minimal-notebook [Source on GitHub](https://github.com/jupyter/docker-stacks/tree/main/images/minimal-notebook) | [Dockerfile commit history](https://github.com/jupyter/docker-stacks/commits/main/images/minimal-notebook/Dockerfile) | [Quay.io image tags](https://quay.io/repository/jupyter/minimal-notebook?tab=tags) `jupyter/minimal-notebook` adds command-line tools useful when working in Jupyter applications. It contains: - Everything in `jupyter/base-notebook` - Common useful utilities like [curl](https://curl.se), [git](https://git-scm.com/), [nano](https://www.nano-editor.org/) (actually `nano-tiny`), [tzdata](https://www.iana.org/time-zones), [unzip](https://code.launchpad.net/ubuntu/+source/unzip), and [vi](https://www.vim.org) (actually `vim-tiny`), - [TeX Live](https://www.tug.org/texlive/) for notebook document conversion ### jupyter/r-notebook [Source on GitHub](https://github.com/jupyter/docker-stacks/tree/main/images/r-notebook) | [Dockerfile commit history](https://github.com/jupyter/docker-stacks/commits/main/images/r-notebook/Dockerfile) | [Quay.io image tags](https://quay.io/repository/jupyter/r-notebook?tab=tags) `jupyter/r-notebook` includes popular packages from the R ecosystem listed below: - Everything in `jupyter/minimal-notebook` and its ancestor images - The [R](https://www.r-project.org/) interpreter and base environment - [IRKernel](https://irkernel.github.io/) to support R code in Jupyter notebooks - [tidyverse](https://tidyverse.org/) packages from [conda-forge](https://conda-forge.org/feedstock-outputs/index.html) - [caret](https://topepo.github.io/caret/index.html), [crayon](https://cran.r-project.org/web/packages/crayon/index.html), [devtools](https://cran.r-project.org/web/packages/devtools/index.html), [forecast](https://cran.r-project.org/web/packages/forecast/index.html), [hexbin](https://cran.r-project.org/web/packages/hexbin/index.html), [htmltools](https://cran.r-project.org/web/packages/htmltools/index.html), [htmlwidgets](https://www.htmlwidgets.org), [nycflights13](https://cran.r-project.org/web/packages/nycflights13/index.html), [randomforest](https://cran.r-project.org/web/packages/randomForest/index.html), [rcurl](https://cran.r-project.org/web/packages/RCurl/index.html), [rmarkdown](https://rmarkdown.rstudio.com), [rodbc](https://cran.r-project.org/web/packages/RODBC/index.html), [rsqlite](https://cran.r-project.org/web/packages/RSQLite/index.html), [shiny](https://shiny.posit.co), [tidymodels](https://www.tidymodels.org/), [unixODBC](https://www.unixodbc.org/) packages from [conda-forge](https://conda-forge.org/feedstock-outputs/index.html) ### jupyter/julia-notebook [Source on GitHub](https://github.com/jupyter/docker-stacks/tree/main/images/julia-notebook) | [Dockerfile commit history](https://github.com/jupyter/docker-stacks/commits/main/images/julia-notebook/Dockerfile) | [Quay.io image tags](https://quay.io/repository/jupyter/julia-notebook?tab=tags) `jupyter/julia-notebook` includes popular packages from the Julia ecosystem listed below: - Everything in `jupyter/minimal-notebook` and its ancestor images - The [Julia](https://julialang.org/) compiler and base environment - [IJulia](https://github.com/JuliaLang/IJulia.jl) to support Julia code in Jupyter notebook - [Pluto.jl](https://plutojl.org/) reactive Julia notebook interface, made accessible with [jupyter-pluto-proxy](https://github.com/yuvipanda/jupyter-pluto-proxy) - [HDF5](https://github.com/JuliaIO/HDF5.jl) package ### jupyter/scipy-notebook [Source on GitHub](https://github.com/jupyter/docker-stacks/tree/main/images/scipy-notebook) | [Dockerfile commit history](https://github.com/jupyter/docker-stacks/commits/main/images/scipy-notebook/Dockerfile) | [Quay.io image tags](https://quay.io/repository/jupyter/scipy-notebook?tab=tags) `jupyter/scipy-notebook` includes popular packages from the scientific Python ecosystem. - Everything in `jupyter/minimal-notebook` and its ancestor images - [altair](https://altair-viz.github.io), [beautifulsoup4](https://www.crummy.com/software/BeautifulSoup/), [bokeh](https://docs.bokeh.org/en/latest/), [bottleneck](https://bottleneck.readthedocs.io/en/latest/), [cloudpickle](https://github.com/cloudpipe/cloudpickle), [conda-forge::blas=\*=openblas](https://www.openblas.net), [cython](https://cython.org), [dask](https://www.dask.org/), [dill](https://pypi.org/project/dill/), [h5py](https://www.h5py.org), [jupyterlab-git](https://github.com/jupyterlab/jupyterlab-git), [matplotlib-base](https://matplotlib.org/), [numba](https://numba.pydata.org/), [numexpr](https://github.com/pydata/numexpr), [openpyxl](https://openpyxl.readthedocs.io/en/stable/), [pandas](https://pandas.pydata.org/), [patsy](https://patsy.readthedocs.io/en/latest/), [protobuf](https://protobuf.dev/getting-started/pythontutorial/), [pytables](https://www.pytables.org/), [scikit-image](https://scikit-image.org), [scikit-learn](https://scikit-learn.org/stable/), [scipy](https://scipy.org/), [seaborn](https://seaborn.pydata.org/), [sqlalchemy](https://www.sqlalchemy.org/), [statsmodels](https://www.statsmodels.org/stable/index.html), [sympy](https://www.sympy.org/en/index.html), [widgetsnbextension](https://ipywidgets.readthedocs.io/en/latest/user_install.html#installing-in-classic-jupyter-notebook), [xlrd](https://www.python-excel.org) packages - [ipympl](https://github.com/matplotlib/ipympl) and [ipywidgets](https://ipywidgets.readthedocs.io/en/stable/) for interactive visualizations and plots in Python notebooks ### jupyter/tensorflow-notebook [Source on GitHub](https://github.com/jupyter/docker-stacks/tree/main/images/tensorflow-notebook) | [Dockerfile commit history](https://github.com/jupyter/docker-stacks/commits/main/images/tensorflow-notebook/Dockerfile) | [Quay.io image tags](https://quay.io/repository/jupyter/tensorflow-notebook?tab=tags) `jupyter/tensorflow-notebook` includes popular Python deep learning libraries. - Everything in `jupyter/scipy-notebook` and its ancestor images - [TensorFlow](https://www.tensorflow.org/) machine learning library - [Jupyter Server Proxy](https://jupyter-server-proxy.readthedocs.io/en/latest/) to support [TensorBoard](https://www.tensorflow.org/tensorboard) ### jupyter/pytorch-notebook [Source on GitHub](https://github.com/jupyter/docker-stacks/tree/main/images/pytorch-notebook) | [Dockerfile commit history](https://github.com/jupyter/docker-stacks/commits/main/images/pytorch-notebook/Dockerfile) | [Quay.io image tags](https://quay.io/repository/jupyter/pytorch-notebook?tab=tags) `jupyter/pytorch-notebook` includes popular Python deep learning libraries. - Everything in `jupyter/scipy-notebook` and its ancestor images - [pytorch](https://pytorch.org/) machine learning library ### jupyter/datascience-notebook [Source on GitHub](https://github.com/jupyter/docker-stacks/tree/main/images/datascience-notebook) | [Dockerfile commit history](https://github.com/jupyter/docker-stacks/commits/main/images/datascience-notebook/Dockerfile) | [Quay.io image tags](https://quay.io/repository/jupyter/datascience-notebook?tab=tags) `jupyter/datascience-notebook` includes libraries for data analysis from the Python, R, and Julia communities. - Everything in `jupyter/scipy-notebook` and its ancestor images - The R and Julia packages from the `jupyter/r-notebook` and `jupyter/julia-notebook` images (but not the additional system packages installed via `apt` in `jupyter/r-notebook`, such as `unixodbc-dev`) - [rpy2](https://rpy2.github.io/doc/latest/html/index.html) package ### jupyter/pyspark-notebook [Source on GitHub](https://github.com/jupyter/docker-stacks/tree/main/images/pyspark-notebook) | [Dockerfile commit history](https://github.com/jupyter/docker-stacks/commits/main/images/pyspark-notebook/Dockerfile) | [Quay.io image tags](https://quay.io/repository/jupyter/pyspark-notebook?tab=tags) `jupyter/pyspark-notebook` includes Python support for Apache Spark. - Everything in `jupyter/scipy-notebook` and its ancestor images - [Apache Spark](https://spark.apache.org/) with Hadoop binaries - [grpcio-status](https://github.com/grpc/grpc/tree/master/src/python/grpcio_status) - [grpcio](https://grpc.io/docs/languages/python/quickstart/) - [pyarrow](https://arrow.apache.org/docs/python/) ### jupyter/all-spark-notebook [Source on GitHub](https://github.com/jupyter/docker-stacks/tree/main/images/all-spark-notebook) | [Dockerfile commit history](https://github.com/jupyter/docker-stacks/commits/main/images/all-spark-notebook/Dockerfile) | [Quay.io image tags](https://quay.io/repository/jupyter/all-spark-notebook?tab=tags) `jupyter/all-spark-notebook` includes Python and R support for Apache Spark. - Everything in `jupyter/pyspark-notebook` and its ancestor images - [IRKernel](https://irkernel.github.io/) to support R code in Jupyter notebooks - [rcurl](https://cran.r-project.org/web/packages/RCurl/index.html), [sparklyr](https://spark.posit.co), [ggplot2](https://ggplot2.tidyverse.org) packages ### CUDA enabled variants We provide CUDA accelerated versions of the `pytorch-notebook` and `tensorflow-notebook` images. Prepend a CUDA prefix (versioned prefix like `cuda12-` for `pytorch-notebook` or just `cuda-` for `tensorflow-notebook`) to the image tag to allow PyTorch or TensorFlow operations to use compatible NVIDIA GPUs for accelerated computation. We only build `pytorch-notebook` for the last two major versions of CUDA. The `tensorflow-notebook` image only supports the latest CUDA version listed in the [officially tested build configurations](https://www.tensorflow.org/install/source#gpu). For example, you could use the image `quay.io/jupyter/pytorch-notebook:cuda12-python-3.13.14` or `quay.io/jupyter/tensorflow-notebook:cuda-latest`. ### Image Relationships The following diagram depicts the build dependency tree of the core images. (i.e., the `FROM` statements in their Dockerfiles). Any given image inherits the complete content of all ancestor images pointing to it. ```{mermaid} flowchart TD ubuntu["ubuntu
(LTS with point release)"] --> docker-stacks-foundation docker-stacks-foundation --> base-notebook base-notebook --> minimal-notebook minimal-notebook --> scipy-notebook minimal-notebook --> r-notebook minimal-notebook --> julia-notebook scipy-notebook --> tensorflow-notebook["tensorflow-notebook
+cuda variant"] scipy-notebook --> pytorch-notebook["pytorch-notebook
+cuda12/cuda13 variants"] scipy-notebook --> datascience-notebook scipy-notebook --> pyspark-notebook pyspark-notebook --> all-spark-notebook ``` ### Builds Every Monday and whenever a pull request is merged, images are rebuilt and pushed to [the public container registry](https://quay.io/organization/jupyter). ### Versioning via image tags Whenever a docker image is pushed to the container registry, it is tagged with: - the `latest` tag - a 12-character git commit SHA like `1ffe43816ba9` - a date formatted like `2023-01-30` - OS version like `ubuntu-24.04` - a set of software version tags like `python-3.13.14` and `lab-4.6.1` (exact versions vary with each build) ```{warning} - Tags before `2022-07-05` were sometimes incorrect. Please, do not rely on them. - Single-platform images have either `aarch64-` or `x86_64-` tag prefixes, for example, `quay.io/jupyter/base-notebook:aarch64-python-3.13.14` ``` For stability and reproducibility, you should either reference a date-formatted tag from a date before the current date (in UTC) or a git commit SHA older than the latest git commit SHA in the default branch of the [jupyter/docker-stacks GitHub repository](https://github.com/jupyter/docker-stacks/). ### Verifying image signatures Pushed images (both multi-platform and single-platform ones) are signed with [cosign](https://docs.sigstore.dev/cosign/signing/overview/) using keyless signing with the GitHub Actions OIDC identity. Signatures are stored in the Sigstore bundle format, which requires cosign v3+ to verify. To verify that an image was built and pushed by this repository's CI: ```bash cosign verify quay.io/jupyter/base-notebook:latest \ --certificate-identity-regexp='^https://github\.com/jupyter/docker-stacks/' \ --certificate-oidc-issuer=https://token.actions.githubusercontent.com ``` ## Community Stacks The core stacks are but a tiny sample of what's possible when combining Jupyter with other technologies. We encourage members of the Jupyter community to create their own stacks based on the core images and link them below. See the [contributing guide](../contributing/stacks.md) for information about how to create your own Jupyter Docker Stack. | Flavor | Binder | Description | | -------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | [csharp] | [![bb]][csharp_b] | More than 200 Jupyter Notebooks with example **C#** code | | [education] | [![bb]][education_b] | **`nbgrader`** and `RISE` on top of the `datascience-notebook` image | | [ihaskell] | [![bb]][ihaskell_b] | Based on [**IHaskell**][ihaskell_project]. Includes popular packages and example notebooks | | [java] | [![bb]][java_b] | [**IJava**][ijava] kernel on top of the `minimal-notebook` image | | [sage] | [![bb]][sage_b] | [**sagemath**][sagemath] kernel on top of the `minimal-notebook` image | | [cgspatial] | [![bb]][cgspatial_b] | Major **geospatial** Python & R libraries on top of the `datascience-notebook` image | | [kotlin] | [![bb]][kotlin_b] | [**Kotlin** kernel for Jupyter/IPython][kotlin_kernel] on top of the `base-notebook` image | | [transformers] | [![bb]][transformers_b] | [**Transformers**][transformers_lib] and NLP libraries such as `Tensorflow`, `Keras`, `Jax` and `PyTorch` | | [scraper] | [![bb]][scraper_b] | **Scraper** tools (`selenium`, `chromedriver`, `beatifulsoup4`, `requests`) on `minimal-notebook` image | | [almond] | [![bb]][almond_b] | Scala kernel for Jupyter using **Almond** on top of the `base-notebook` image | | [lisp-stat] | [![bb]][lisp-stat_b] | Common Lisp statistical computing environment on top of the `minimal-notebook` image | | [sequencing] | [![bb]][sequencing_b] | Collection for bioinformatics sequencing data analysis, covering bulk RNA-seq, single-cell RNA-seq, spatial sequencing, and multi-omics | [bb]: https://static.mybinder.org/badge_logo.svg [csharp]: https://github.com/tlinnet/csharp-notebook [csharp_b]: https://mybinder.org/v2/gh/tlinnet/csharp-notebook/master [education]: https://github.com/umsi-mads/education-notebook [education_b]: https://mybinder.org/v2/gh/umsi-mads/education-notebook/master [ihaskell]: https://github.com/IHaskell/ihaskell-notebook [ihaskell_b]: https://mybinder.org/v2/gh/jamesdbrock/learn-you-a-haskell-notebook/master?urlpath=lab/tree/ihaskell_examples/ihaskell/IHaskell.ipynb [ihaskell_project]: https://github.com/IHaskell/IHaskell [java]: https://github.com/jbindinga/java-notebook [java_b]: https://mybinder.org/v2/gh/jbindinga/java-notebook/master [ijava]: https://github.com/SpencerPark/IJava [sage]: https://github.com/sharpTrick/sage-notebook [sage_b]: https://mybinder.org/v2/gh/sharpTrick/sage-notebook/master [sagemath]: https://www.sagemath.org [cgspatial]: https://github.com/SCiO-systems/cgspatial-notebook [cgspatial_b]: https://mybinder.org/v2/gh/SCiO-systems/cgspatial-notebook/master [kotlin]: https://github.com/knonm/kotlin-notebook [kotlin_b]: https://mybinder.org/v2/gh/knonm/kotlin-notebook/main [kotlin_kernel]: https://github.com/Kotlin/kotlin-jupyter [transformers]: https://github.com/ToluClassics/transformers_notebook [transformers_b]: https://mybinder.org/v2/gh/ToluClassics/transformers_notebook/main [transformers_lib]: https://huggingface.co/docs/transformers/index [scraper]: https://github.com/rgriffogoes/scraper-notebook [scraper_b]: https://mybinder.org/v2/gh/rgriffogoes/scraper-notebook/main [almond]: https://almond.sh [almond_b]: https://mybinder.org/v2/gh/almond-sh/examples/master?urlpath=lab%2Ftree%2Fnotebooks%2Findex.ipynb [lisp-stat]: https://lisp-stat.dev [lisp-stat_b]: https://mybinder.org/v2/gh/Lisp-Stat/IPS9/HEAD?urlpath=%2Fdoc%2Ftree%2Findex.ipynb [sequencing]: https://github.com/huchlab/sequencing-docker-stacks [sequencing_b]: https://mybinder.org/v2/gh/huchlab/sequencing-docker-stacks/main?urlpath=lab%2Ftree%2FREADME.ipynb ### Other GPU-accelerated notebooks | Flavor | Description | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [GPU-Jupyter][gpu] | Power of your NVIDIA GPU and GPU calculations using Tensorflow and Pytorch in collaborative notebooks. This is done by generating a Dockerfile that consists of the **nvidia/cuda** base image, the well-maintained **docker-stacks** that is integrated as a submodule, and GPU-able libraries like **Tensorflow**, **Keras** and **PyTorch** on top of it. | | [myLab TH Lübeck Images][gpu_thl] | Images based on the **jupyter/docker-stacks**, built and maintained at the [myLab TH Lübeck][gpu_mylab] using build scripts similar to iot-salzburg. Several images include GPU libraries. | | [PRP-GPU][prp_gpu] | PRP (Pacific Research Platform) maintained [registry][prp_reg] for jupyter stack based on NVIDIA CUDA-enabled image. Added the PRP image with Pytorch and some other Python packages and GUI Desktop notebook based on . | | [b-data][b-data] | GPU accelerated, multi-arch (`linux/amd64`, `linux/arm64/v8`) Docker images for [R][r_cuda], [Python][python_cuda] , [MAX][max_cuda] and [Julia][julia_cuda]. Derived from nvidia/cuda `devel`-flavored images. With [code-server][code-server] next to JupyterLab. Just Python – no [Conda][conda] / [Mamba][mamba]. | [gpu]: https://github.com/iot-salzburg/gpu-jupyter [gpu_thl]: https://hub.docker.com/r/hanseware/jlab-images [gpu_mylab]: https://mylab.th-luebeck.de [prp_gpu]: https://gitlab.nrp-nautilus.io/prp/jupyter-stack/-/tree/prp [prp_reg]: https://gitlab.nrp-nautilus.io/prp/jupyter-stack/container_registry [b-data]: https://github.com/b-data [r_cuda]: https://github.com/b-data/jupyterlab-r-docker-stack/blob/main/CUDA.md [python_cuda]: https://github.com/b-data/jupyterlab-python-docker-stack/blob/main/CUDA.md [max_cuda]: https://github.com/b-data/jupyterlab-mojo-docker-stack/blob/main/CUDA.md [julia_cuda]: https://github.com/b-data/jupyterlab-julia-docker-stack/blob/main/CUDA.md [code-server]: https://github.com/coder/code-server [conda]: https://github.com/conda/conda [mamba]: https://github.com/mamba-org/mamba --- ### Using Specifics (`docs/using/specifics.md`) # Image Specifics This page provides details about features specific to one or more images. ## Apache Spark™ ### Specific Docker Image Options - `-p 4040:4040` - The `jupyter/pyspark-notebook` and `jupyter/all-spark-notebook` images open [SparkUI (Spark Monitoring and Instrumentation UI)](https://spark.apache.org/docs/latest/monitoring.html) at default port `4040`, this option maps the `4040` port inside the docker container to the `4040` port on the host machine. ```{note} Every new spark context that is created is put onto an incrementing port (i.e. 4040, 4041, 4042, etc.), and it might be necessary to open multiple ports. ``` For example, `docker run --detach -p 8888:8888 -p 4040:4040 -p 4041:4041 quay.io/jupyter/pyspark-notebook`. #### IPython low-level output capture and forward Spark images (`pyspark-notebook` and `all-spark-notebook`) have been configured to disable IPython low-level output capture and forward system-wide. The rationale behind this choice is that Spark logs can be verbose, especially at startup when Ivy is used to load additional jars. Those logs are still available but only in the container's logs. If you want to make them appear in the notebook, you can overwrite the configuration in a user-level IPython kernel profile. To do that, you have to uncomment the following line in your `~/.ipython/profile_default/ipython_kernel_config.py` and restart the kernel. ```python c.IPKernelApp.capture_fd_output = True ``` If you have no IPython profile, you can initiate a fresh one by running the following command. ```bash ipython profile create # [ProfileCreate] Generating default config file: '/home/jovyan/.ipython/profile_default/ipython_config.py' # [ProfileCreate] Generating default config file: '/home/jovyan/.ipython/profile_default/ipython_kernel_config.py' ``` ### Build an Image with a Different Version of Spark You can build a `pyspark-notebook` image with a different `Spark` version by overriding the default value of the following arguments at build time. `all-spark-notebook` inherits from `pyspark-notebook`, so you have to first build `pyspark-notebook` and then `all-spark-notebook` to get the same version in `all-spark-notebook`. - Spark distribution is defined by the combination of Spark, Hadoop, and Scala versions, see [Download Apache Spark](https://spark.apache.org/downloads.html) and the [archive repo](https://archive.apache.org/dist/spark/) for more information. - `openjdk_version`: The version of the OpenJDK (JRE headless) distribution (`21` by default). - This version needs to match the version supported by the Spark distribution used above. - See [Spark Overview](https://spark.apache.org/docs/latest/#downloading) and [Ubuntu packages](https://packages.ubuntu.com/search?keywords=openjdk). - `spark_version` (optional): The Spark version to install, for example `4.0.0`. If not specified (this is the default), the latest stable Spark version will be installed. - `hadoop_version`: The Hadoop version (`3` by default). - `scala_version` (optional): The Scala version, for example `2.13` (not specified by default). Starting with _Spark >= 3.2_, the distribution file might contain the Scala version. - `spark_download_url`: URL to use for Spark downloads. You may need to use url if you want to download old Spark versions. ```{note} Only Spark >= 4.0 can be built this way: the pandas version is resolved from the `dev/infra/Dockerfile` file of the Spark release, and the Derby jar replacement expects the version bundled by modern Spark. Building older versions requires modifying `images/pyspark-notebook/Dockerfile`. ``` For example, here is how to build a `pyspark-notebook` image with Spark `4.0.0` and OpenJDK `17`. ```{warning} This recipe is not tested and might be broken. ``` ```bash # From the root of the project # Build the image with different arguments docker build --rm --force-rm \ -t my-pyspark-notebook ./images/pyspark-notebook \ --build-arg openjdk_version=17 \ --build-arg spark_version=4.0.0 \ --build-arg spark_download_url="https://archive.apache.org/dist/spark/" # Check the newly built image docker run -it --rm my-pyspark-notebook pyspark --version # WARNING: Using incubator modules: jdk.incubator.vector # Welcome to # ____ __ # / __/__ ___ _____/ /__ # _\ \/ _ \/ _ `/ __/ '_/ # /___/ .__/\_,_/_/ /_/\_\ version 4.0.0 # /_/ # Using Scala version 2.13.16, OpenJDK 64-Bit Server VM, 17.0.19 # Branch HEAD # Compiled by user wenchen on 2025-05-19T07:58:03Z # Revision fa33ea000a0bda9e5a3fa1af98e8e85b8cc5e4d4 # Url https://github.com/apache/spark # Type --help for more information. ``` ### Usage Examples The `jupyter/pyspark-notebook` and `jupyter/all-spark-notebook` images support the use of [Apache Spark](https://spark.apache.org/) in Python and R notebooks. The following sections provide some examples of how to get started using them. #### Using Spark Local Mode Spark **local mode** is useful for experimentation on small data when you do not have a Spark cluster available. ```{warning} In these examples, Spark spawns all the main execution components in the same single JVM. You can read additional info about local mode [here](https://books.japila.pl/apache-spark-internals/local/). If you want to use all the CPU, one of the simplest ways is to set up a [Spark Standalone Cluster](https://spark.apache.org/docs/latest/spark-standalone.html). ``` ##### Local Mode in Python In a Python notebook. ```python from pyspark.sql import SparkSession # Spark session & context spark = SparkSession.builder.master("local").getOrCreate() sc = spark.sparkContext # Sum of the first 100 whole numbers rdd = sc.parallelize(range(100 + 1)) rdd.sum() # 5050 ``` ##### Local Mode in R ```{warning} SparkR is deprecated from Apache Spark 4.0.0 and will be removed in a future version. Prefer the [sparklyr][sparklyr] examples below. ``` In an R notebook with [SparkR][sparkr]. ```R library(SparkR) # Spark session & context sc <- sparkR.session("local") # Sum of the first 100 whole numbers sdf <- createDataFrame(list(1:100)) dapplyCollect(sdf, function(x) { x <- sum(x)} ) # 5050 ``` In an R notebook with [sparklyr][sparklyr]. ```R library(sparklyr) # Spark configuration conf <- spark_config() # Set the catalog implementation in-memory conf$spark.sql.catalogImplementation <- "in-memory" # Spark session & context sc <- spark_connect(master = "local", config = conf) # Sum of the first 100 whole numbers sdf_len(sc, 100, repartition = 1) %>% spark_apply(function(e) sum(e)) # 5050 ``` #### Connecting to a Spark Cluster in Standalone Mode Connection to Spark Cluster on **[Standalone Mode](https://spark.apache.org/docs/latest/spark-standalone.html)** requires the following set of steps: 0. Verify that the docker image (check the Dockerfile) and the Spark Cluster, which is being deployed, run the same version of Spark. 1. [Deploy Spark in Standalone Mode](https://spark.apache.org/docs/latest/spark-standalone.html). 2. Run the Docker container with `--net=host` in a location that is network-addressable by all of your Spark workers. (This is a [Spark networking requirement](https://spark.apache.org/docs/latest/cluster-overview.html#components).) ```{note} When using `--net=host`, you must also use the flags `--pid=host -e TINI_SUBREAPER=true`. See for details. ``` **Note**: In the following examples, we are using the Spark master URL `spark://master:7077` which shall be replaced by the URL of the Spark master. ##### Standalone Mode in Python The **same Python version** needs to be used on the notebook (where the driver is located) and on the Spark workers. The Python version used on the driver and worker side can be adjusted by setting the environment variables `PYSPARK_PYTHON` and/or `PYSPARK_DRIVER_PYTHON`, see [Spark Configuration][spark-conf] for more information. ```python from pyspark.sql import SparkSession # Spark session & context spark = SparkSession.builder.master("spark://master:7077").getOrCreate() sc = spark.sparkContext # Sum of the first 100 whole numbers rdd = sc.parallelize(range(100 + 1)) rdd.sum() # 5050 ``` ##### Standalone Mode in R In an R notebook with [SparkR][sparkr]. ```R library(SparkR) # Spark session & context sc <- sparkR.session("spark://master:7077") # Sum of the first 100 whole numbers sdf <- createDataFrame(list(1:100)) dapplyCollect(sdf, function(x) { x <- sum(x)} ) # 5050 ``` In an R notebook with [sparklyr][sparklyr]. ```R library(sparklyr) # Spark session & context # Spark configuration conf <- spark_config() # Set the catalog implementation in-memory conf$spark.sql.catalogImplementation <- "in-memory" sc <- spark_connect(master = "spark://master:7077", config = conf) # Sum of the first 100 whole numbers sdf_len(sc, 100, repartition = 1) %>% spark_apply(function(e) sum(e)) # 5050 ``` ### Define Spark Dependencies ```{note} This example is given for [Elasticsearch](https://www.elastic.co/docs/reference/elasticsearch-hadoop/installation). ``` Spark dependencies can be declared thanks to the `spark.jars.packages` property (see [Spark Configuration](https://spark.apache.org/docs/latest/configuration.html#runtime-environment) for more information). They can be defined as a comma-separated list of Maven coordinates at the creation of the Spark session. ```python from pyspark.sql import SparkSession spark = ( SparkSession.builder.appName("elasticsearch") .config( "spark.jars.packages", "org.elasticsearch:elasticsearch-spark-30_2.12:7.13.0" ) .getOrCreate() ) ``` Dependencies can also be defined in the `spark-defaults.conf`. However, it has to be done by `root`, so it should only be considered to build custom images. ```dockerfile USER root RUN echo "spark.jars.packages org.elasticsearch:elasticsearch-spark-30_2.12:7.13.0" >> "${SPARK_HOME}/conf/spark-defaults.conf" USER ${NB_UID} ``` Jars will be downloaded dynamically at the creation of the Spark session and stored by default in `${HOME}/.ivy2/jars` (can be changed by setting `spark.jars.ivy`). ## Tensorflow The `jupyter/tensorflow-notebook` image supports the use of [Tensorflow](https://www.tensorflow.org/) in a single machine or distributed mode. ### Single Machine Mode ```python import tensorflow as tf print(tf.constant("Hello, TensorFlow")) print(tf.reduce_sum(tf.random.normal([1000, 1000]))) ``` ### Distributed Mode TensorFlow 2 uses [distribution strategies](https://www.tensorflow.org/guide/distributed_training) for distributed computations. For example, `tf.distribute.MirroredStrategy` performs synchronous training across multiple devices on a single machine (it also works on a machine without GPUs). ```python import tensorflow as tf strategy = tf.distribute.MirroredStrategy() with strategy.scope(): model = tf.keras.Sequential([tf.keras.Input(shape=(1,)), tf.keras.layers.Dense(1)]) model.compile(loss="mse", optimizer="sgd") ``` [sparkr]: https://spark.apache.org/docs/latest/sparkr.html [sparklyr]: https://spark.posit.co [spark-conf]: https://spark.apache.org/docs/latest/configuration.html --- ### Using Troubleshooting (`docs/using/troubleshooting.md`) # Troubleshooting Common Problems When troubleshooting, you may see unexpected behaviors or receive an error message. This section provides advice on how to identify and fix some of the most commonly encountered issues. Most of the `docker run` flags used in this document are explained in detail in the [Common Features, Docker Options section](common.md#docker-options) of the documentation. ## Permission denied when mounting volumes If you are running a Docker container while mounting a local volume or host directory using the `-v` flag like so: ```bash docker run -it --rm \ -p 8888:8888 \ -v : \ quay.io/jupyter/minimal-notebook:latest ``` you might face permissions issues when trying to access the mounted volume: ```bash # assuming we mounted the volume in /home/jovyan/stagingarea # root is the owner of the mounted volume ls -ld ~/stagingarea/ # drwxr-xr-x 2 root root 4096 Feb 1 12:55 stagingarea/ touch stagingarea/kale.txt # touch: cannot touch 'stagingarea/kale.txt': Permission denied ``` In this case, the user of the container (`jovyan`) and the owner of the mounted volume (`root`) have different permission levels and ownership over the container's directories and mounts. The following sections cover a few of these scenarios and how to fix them. **Some things to try:** 1. **Change ownership of the volume mount** You can change the ownership of the volume mount using the `chown` command. In the case of the docker-stacks images, you can set the `CHOWN_EXTRA` and `CHOWN_EXTRA_OPTS` environment variables. For example, to change the ownership of the volume mount to the `jovyan` user (non-privileged default user in the Docker images): ```bash # running in detached mode - can also be run in interactive mode docker run --detach \ -v : \ -p 8888:8888 \ --user root \ -e CHOWN_EXTRA="" \ -e CHOWN_EXTRA_OPTS="-R" \ quay.io/jupyter/minimal-notebook ``` where: - `CHOWN_EXTRA=,`: will change the ownership and group of the specified container directory (non-recursive by default). You need to provide full paths starting with `/`. - `CHOWN_EXTRA_OPTS="-R"`: will recursively change the ownership and group of the directory specified in `CHOWN_EXTRA`. - `--user root`: you **must** run the container with the root user to change ownership at runtime. Now accessing the mount should work as expected: ```bash # assuming we mounted the volume in /home/jovyan/stagingarea ls -ld ~/stagingarea # drwxr-xr-x 2 jovyan users 4096 Feb 1 12:55 stagingarea/ touch stagingarea/kale.txt # jovyan is now the owner of /home/jovyan/stagingarea # ls -la ~/stagingarea/ # -rw-r--r-- 1 jovyan users 0 Feb 1 14:41 kale.txt ``` ```{admonition} Additional notes - If you are mounting your volume inside the `/home/` directory, you can use the `-e CHOWN_HOME=yes` and `CHOWN_HOME_OPTS="-R"` flags instead of the `-e CHOWN_EXTRA` and `-e CHOWN_EXTRA_OPTS` in the example above. - This solution should work in most cases where you have created a docker volume (i.e. using the [`docker volume create --name ` command](https://docs.docker.com/engine/storage/volumes/#create-and-manage-volumes)) and mounted it using the `-v` flag in `docker run`. ``` 2. **Matching the container's UID/GID with the host's** Docker handles mounting host directories differently from mounting volumes, even though the syntax is essentially the same (i.e. `-v`). When you initialize a Docker container using the `-v` flag, the host directories are bind-mounted directly into the container. Therefore, the permissions and ownership are copied over and will be **the same** as the ones in your local host (including user ids) which may result in permissions errors when trying to access directories or create/modify files inside. Suppose your local user has a `UID` and `GID` of `1234` and `5678`, respectively. To fix the UID discrepancies between your local directories and the container's directories, you can run the container with an explicit `NB_UID` and `NB_GID` to match that of the local user: ```bash docker run -it --rm \ --user root \ -p 8888:8888 \ -e NB_UID=1234 \ -e NB_GID=5678 \ -v "${PWD}"/test:/home/jovyan/work \ quay.io/jupyter/minimal-notebook:latest # you should see an output similar to this # Update jovyan's UID:GID to 1234:5678 # Running as jovyan: bash ``` where: - `NB_UID` and `NB_GID` should match the local user's UID and GID. - You **must** use `--user root` to ensure that the `UID` and `GID` are updated at runtime. ````{admonition} Additional notes - The caveat with this approach is that since these changes are applied at runtime, you will need to re-run the same command with the appropriate flags and environment variables if you need to recreate the container (i.e. after removing/destroying it). - If you pass a numeric UID, it **must** be in the range of 0-2147483647 - This approach only updates the UID and GID of the **existing `jovyan` user** instead of creating a new user. From the above example: ```bash id # uid=1234(jovyan) gid=5678(jovyan) groups=5678(jovyan),100(users) ``` ```` ## Permission issues after changing the UID/GID and USER in the container If you have also **created a new user**, you might be experiencing any of the following issues: - the `root` user is the owner of `/home` or a mounted volume - when starting the container, you get an error such as `Failed copy data from /home/jovyan to /home/ or to create symlink!` - getting permission denied when trying to `conda install` packages **Some things to try:** 1. **Ensure the new user has ownership of `/home` and volume mounts** For example, say you want to create a user `callisto` with a `GID` and `UID` of `1234`. You will have to add the following flags to the docker run command: ```bash docker run -it --rm \ -p 8888:8888 \ --user root \ -e NB_USER=callisto \ -e NB_UID=1234 \ -e NB_GID=1234 \ -e CHOWN_HOME=yes \ -e CHOWN_HOME_OPTS="-R" \ -w "/home/callisto" \ -v "${PWD}"/test:/home/callisto/work \ quay.io/jupyter/minimal-notebook # Updated the jovyan user: # - username: jovyan -> callisto # - home dir: /home/jovyan -> /home/callisto # Update callisto's UID:GID to 1234:1234 # Attempting to copy /home/jovyan to /home/callisto... # Success! # Ensuring /home/callisto is owned by 1234:1234 (chown options: -R) # Running as callisto: bash ``` where: - `-e NB_USER=callisto`: will rename the `jovyan` user to `callisto` and attempt to copy the content of the home directory to `/home/callisto`, falling back to creating a symlink to `/home/jovyan` if the copy fails - `-e NB_UID=1234` and `-e NB_GID=1234`: will set the `UID` and `GID` of the new user (`callisto`) to `1234` - `-e CHOWN_HOME_OPTS="-R"` and `-e CHOWN_HOME=yes`: ensure that the new user is the owner of the `/home` directory and subdirectories (setting `CHOWN_HOME_OPTS="-R` will ensure this change is applied recursively) - `-w "/home/callisto"` sets the working directory to be the new user's home ```{admonition} Additional notes In the example above, the `-v` flag is used to mount the local volume onto the new user's `/home` directory. However, if you are mounting a volume elsewhere, you also need to use the `-e CHOWN_EXTRA=` flag to avoid any permission issues (see the section [Permission denied when mounting volumes](#permission-denied-when-mounting-volumes) on this page). ``` 2. **Dynamically assign the user ID and GID** The above case ensures that the `/home` directory is owned by a newly created user with a specific `UID` and `GID`, but if you want to assign the `UID` and `GID` of the new user dynamically, you can make the following adjustments: ```bash docker run -it --rm \ -p 8888:8888 \ --user root \ -e NB_USER=callisto \ -e NB_UID="$(id -u)" \ -e NB_GID="$(id -g)" \ -e CHOWN_HOME=yes \ -e CHOWN_HOME_OPTS="-R" \ -w "/home/callisto" \ -v "${PWD}"/test:/home/callisto/work \ quay.io/jupyter/minimal-notebook ``` where: - `"$(id -u)"` and `"$(id -g)"` will dynamically assign the `UID` and `GID` of the user executing the `docker run` command to the new user (`callisto`) ## Additional tips and troubleshooting commands for permission-related errors - Pass absolute paths to the `-v` flag: ```bash -v "${PWD}"/:/home/jovyan/work ``` This example uses the syntax `${PWD}`, which is replaced with the full path to the current directory at runtime. The destination path should also be an absolute path starting with a `/` such as `/home/jovyan/work`. - You might want to consider using the Docker native `--user ` and `--group-add users` flags instead of `-e NB_UID` and `-e NB_GID`: ```bash # note this will use the same UID from # the user calling the command, thus matching the local host docker run -it --rm \ -p 8888:8888 \ --user "$(id -u)" --group-add users \ -v :/home/jovyan/work quay.io/jupyter/datascience-notebook ``` This command will launch the container with a specific user UID and add that user to the `users` group to modify the files in the default `/home` and `/opt/conda` directories. This further avoids issues when trying to `conda install` additional packages. - Use `docker inspect ` and look for the [`Mounts` section](https://docs.docker.com/engine/storage/volumes/#start-a-container-with-a-volume) to verify that the volume was created and mounted accordingly: ```json { "Mounts": [ { "Type": "volume", "Name": "my-vol", "Source": "/var/lib/docker/volumes/stagingarea/_data", "Destination": "/home/jovyan/stagingarea", "Driver": "local", "Mode": "z", "RW": true, "Propagation": "" } ] } ``` ## Problems installing conda packages from specific channels The docker-stacks images use the default `flexible` conda channels priority. ```bash conda config --show | grep priority # channel_priority: flexible # to see its meaning conda config --describe channel_priority # checking the current channels conda config --show channels # channels: # - conda-forge ``` **Installing packages from alternative channels:** You can install packages from other conda channels (e.g. `bioconda`) by specifying the channel: ```bash conda install -c bioconda bioconductor-geoquery ``` If you have set the channels priority to `strict`, installing packages from a channel with lower priority may cause problems. In this case, you can disable the `channel_priority` setting at the command level: ```bash conda install --no-channel-priority -c bioconda bioconductor-geoquery ``` Additional details are provided in the [Using Alternative Channels](../using/common.md#using-alternative-channels) section of the [Common Features](common.md) page. ## Tokens are being rejected If you are a regular user of VSCode and the Jupyter extension, you might experience either of these issues when using any of the docker-stacks images: - when clicking on the URL displayed on your command line logs, you face a "This site cannot be reached" page on your web browser - using the produced token and/or URL results in an "Invalid credentials" error on the Jupyter "Token authentication is enabled" page ```bash # example log output from the docker run command # [...] # Or copy and paste one of these URLs: # http://3d4cf3809e3f:8888/?token=996426e890f8dc22fa6835a44442b6026cba02ee61fee6a2 # or http://127.0.0.1:8888/?token=996426e890f8dc22fa6835a44442b6026cba02ee61fee6a2 ``` **Some things to try:** 1. **Find lingering Jupyter processes in the background** The first thing you want to try is to check that no other Jupyter processes are running in the background: ```bash ps aux | grep jupyter ``` If there are existing processes running, you can kill them with: ```bash # example output from the above command # my-user 3412 ... --daemon-module=vscode_datascience_helpers.jupyter_daemon # using the pid from the above log kill 3412 ``` 2. **Turn off Jupyter auto-start in VSCode** Alternatively - you might want to ensure that the `Jupyter: Disable Jupyter Auto Start` setting is turned on to avoid this issue in the future. You can achieve this from the `Settings > Jupyter` menu in VScode: 3. **Route container to unused local port** Instead of mapping Docker port `8888` to local port `8888`, map to another unused local port. You can see an example of mapping to local port `8001`: ```bash docker run -it --rm -p 8001:8888 quay.io/jupyter/datascience-notebook ``` When the terminal provides the link to access Jupyter: , change the default port value of `8888` in the URL to the port value mapped with the `docker run` command. In this example, we use 8001, so the edited link would be: . Note: Port mapping for Jupyter has other applications outside of Docker. For example, it can be used to allow multiple Jupyter instances when using SSH to control cloud devices. --- --- METRICS --- - Files Extracted: 41 - Estimated Token Budget: ~44379 tokens - Canonical Reference: https://codewiki.google/github.com/jupyter/docker-stacks