udemy-docker-mastery Documentation

1. Project Overview & Quickstart (BretFisher/udemy-docker-mastery) Copy Chapter

File: README.md Copy Chapter

Udemy Course Docker Mastery: with Kubernetes+Swarm from a Docker Captain

Build, test, and deploy containers with the best mega-course on Docker, Kubernetes, Compose, Swarm and Registry using a cloud native DevOps mindset

This repository is for use in my Udemy Courses "Docker Mastery" and "Swarm Mastery"

• Get these courses with my "cheapest on the internet" coupon links: bretfisher.com/courses
• Get the weekly newsletter on all the cloud native DevOps content I'm creating.

Course Sections Copy Chapter

1. Quick Start!
   1. README
   2. Commands and Links
2. Course Introduction
3. The Best Way to Setup Docker for Your OS
4. Creating and Using Containers Like a Boss
5. Container Images, Where To Find Them and How To Build Them
6. Container Lifetime & Persistent Data: Volumes, Volumes, Volumes
7. Making It Easier with Docker Compose: The Multi-Container Tool
8. Swarm Intro and Creating a 3-Node Swarm Cluster
9. Swarm Basic Features and How to Use Them In Your Workflow
10. Swarm App Lifecycle
11. Container Registries: Image Storage and Distribution
12. Docker in Production
13. The What and Why of Kubernetes
14. Kubernetes Architecture and Install
15. Your First Pods
16. Inspecting Kubernetes Resources
17. Exposing Kubernetes Ports
18. Kubernetes Management Techniques
19. Moving to Declarative Kubernetes YAML
20. Your Next Steps and The Future of Kubernetes
21. Automated CI Workflows
22. GitHub Actions Workflow Examples
23. Docker Security Good Defaults and Tools
24. Docker 19.03 Release New Features
25. DevOps and Docker Clips
26. Dockerfiles and Docker Images in 2022
27. Dockerfile and Compose File Reviews
28. Extra's, Common Questions, and Resources

Feel free to create issues or PRs if you find a problem with examples in this repository!

File: compose-assignment-1/README.md Copy Chapter

Assignment: Build a Compose File For a Multi-Container Project

Goal: Create a compose config for a local Drupal CMS website

Docker Mastery Lecture

• This empty directory is where you should create a docker-compose.yml
• Use the drupal:9 image along with the postgres:14 image
• 2020 UPDATE: No version needed in compose files! All 2.x and 3.x features supported without version key.
• Use ports to expose Drupal on 8080 (or your preferred port for localhost access)
• Be sure to setup the POSTGRES_PASSWORD environment variable on the postgres service
• Also note that the postgres official image defaults to user:postgres and database:postgres
• Walk through the Drupal config steps in browser at http://localhost:8080
• Tip: Drupal setup will assume the database runs on localhost, which is incorrect. You'll need to change it under Advanced settings to the name of the Docker service you gave to postgres
• Use Docker Hub documentation to figure out the right environment and volume settings
• Extra Credit: Use volumes to store Drupal unique data

File: compose-assignment-2/README.md Copy Chapter

Assignment: Compose For Image Building

Goal: This time imagine you're just wanting to learn Drupal's admin and GUI, or maybe you're a software tester and you need to test a new theme for Drupal. When configured properly, this will let you build a custom image and start everything with docker-compose up including storing important db and config data in volumes so the site will remember your changes across Compose restarts.

Docker Mastery Lecture

• Use the compose file you created in the last assignment (drupal and postgres) as a starting point.
• Let's pin image version from Docker Hub this time.
  It's always a good idea to do that so a new major version doesn't surprise you.

Dockerfile Copy Chapter

• First you need to build a custom Dockerfile in this directory,
  FROM drupal:9
• Then RUN apt package manager command to install git: apt-get update && apt-get install -y git
• Remember to cleanup after your apt install with rm -rf /var/lib/apt/lists/* and use \ and && properly.
  You can find examples of them in drupal official image. More on this below under Compose file.
• Then change WORKDIR /var/www/html/themes
• Then use git to clone the theme with:
  RUN git clone --branch 8.x-4.x --single-branch --depth 1 https://git.drupalcode.org/project/bootstrap.git
• Combine that line with this line, as we need to change permissions on files and don't want to
  use another image layer to do that (it creates size bloat).
  This drupal container runs as www-data user but the build actually runs as root,
  so often we have to do things like chown to change file owners to the proper user:
  chown -R www-data:www-data bootstrap. Remember the first line needs a \ at end
  to signify the next line is included in the command, and at start of next line you should have
  && to signify "if first command succeeds then also run this command"
• Then, just to be safe, change the working directory back to its default (from drupal image) at /var/www/html

Compose File Copy Chapter

• We're going to build a custom image in this compose file for drupal service.
  Use Compose file from previous assignment for Drupal to start with, and we'll add to it, as well as change image name.
• Rename image to custom-drupal as we want to make a new image based on the official drupal:9.
• We want to build the default Dockerfile in this directory by adding build: . to the drupal service.
  When we add a build + image value to a compose service,
  it knows to use the image name to write to in our image cache, rather then pull from Docker Hub.
• For the postgres:14 service, you need the same password as in previous assignment,
  but also add a volume for drupal-data:/var/lib/postgresql/data so the database will
  persist across Compose restarts.

Start Containers, Configure Drupal Copy Chapter

• Start containers like before, configure Drupal web install like before.
• After site comes up, click on Appearance in top bar, and notice a new theme calledBootstrap is there.
  That's the one we added with our custom Dockerfile.
• Click Install and set as default.
  Then click Back to site (in top left) and the site interface should look different.
  You've successfully installed and activated a new theme in your own custom image without
  installing anything on your host other than Docker!
• If you exit (ctrl-c) and then docker-compose down it will delete containers,
  but not the volumes, so on next docker-compose up everything will be as it was.
• To totally clean up volumes, add -v to down command.

BONUS: Use MariaDB rather than Postgres Copy Chapter

In newer versions of Drupal, they've begun to prefer MySQL or MariaDB over Postgres. You could try installing it using the recommendations guide on Drupal.org.

File: compose-sample-3/html/README.md Copy Chapter

[Start Bootstrap](http://startbootstrap.com/) - [Agency](http://startbootstrap.com/template-overviews/agency/)

Agency is a one page agency portfolio theme for Bootstrap created by Start Bootstrap. This theme features several content sections, a responsive portfolio grid with hover effects, full page portfolio item modals, a responsive timeline, and a working PHP contact form.

Getting Started Copy Chapter

Several options are available to get started quickly:

• Download the latest release on Start Bootstrap
• Clone the repo: git clone https://github.com/BlackrockDigital/startbootstrap-agency.git
• Fork the repo

Developing Using Source Files Copy Chapter

To use the source files, you will need to have npm installed globally along with Gulp.js. To start:

• Run npm install in the root directory
• Run gulp dev and edit the files as needed

If you need to update the plugins included with this template, simply run the following tasks:

• First run npm update to update the dependencies
• Then run gulp copy to copy the new versions to their proper destinations

Bugs and Issues Copy Chapter

Have a bug or an issue with this template? Open a new issue here on GitHub or leave a comment on the template overview page at Start Bootstrap.

Creator Copy Chapter

Start Bootstrap was created by and is maintained by David Miller, Owner of Blackrock Digital.

• https://twitter.com/davidmillerskt
• https://github.com/davidtmiller

Start Bootstrap is based on the Bootstrap framework created by Mark Otto and Jacob Thorton.

Copyright and License Copy Chapter

Copyright 2013-2017 Blackrock Digital LLC. Code released under the MIT license.

File: dockerfile-assignment-1/README.md Copy Chapter

Instructions for Assignment

Build Your Own Dockerfile and Run Containers From It

This directory contains a Node.js app, and you need to get it running in a container.

No modifications to the app should be necessary, only edits to the Dockerfile in this directory.

Overview of this assignment Copy Chapter

• Imagine another developer gave you the instruction below, and you'll need to interpret them into what the Dockerfile should contain.
• Once the Dockerfile builds correctly, start container locally to make sure it works on http://localhost
• Then ensure the image is named properly for your Docker Hub account with a new repository name
• Then push to Docker Hub, and go to https://hub.docker.com to verify you pushed it correctly
• Then remove local image from cache (docker image rm <name>)
• Then start a new container from your Hub image, and watch how it auto downloads and runs
• Test again that it works at http://localhost

Instructions from the app developer Copy Chapter

• you should use the node official image, with the alpine 6.x branch (node:6-alpine)
  • (Yes this is a 2-year old image of node, but all official images are always available on Docker Hub forever, to ensure even old apps still work. It is common to still need to deploy old app versions, even years later.)
• This app listens on port 3000, but the container should listen on port 80 of the Docker host, so it will respond to http://localhost:80 on your computer
• Then it should use the alpine package manager to install tini: apk add --no-cache tini.
• Then it should create directory /usr/src/app for app files with mkdir -p /usr/src/app, or with the Dockerfile command WORKDIR /usr/src/app.
• Node.js uses a "package manager", so it needs to copy in package.json file.
• Then it needs to run 'npm install' to install dependencies from that file.
• To keep it clean and small, run npm cache clean --force after the above, in the same RUN command.
• Then it needs to copy in all files from current directory into the image.
• Then it needs to start the container with the command /sbin/tini -- node ./bin/www. Be sure to use JSON array syntax for CMD. (CMD [ "something", "something" ])
• In the end you should be using FROM, RUN, WORKDIR, COPY, EXPOSE, and CMD commands

Bonus Extra Credit Copy Chapter

• This assignment will not have you setting up a complete image useful for local development, test, and prod. It's just meant to get you started with basic Dockerfile concepts and not focus too much on proper Node.js use in a container. If you happen to be a Node.js Developer, then after you get through more of this course, you should come back and use my Node.js Docker Good Defaults sample project on GitHub to change this Dockerfile for better local development with more advanced topics.

File: dockerfiles/entrypoint/assignment02/README.md Copy Chapter

ENTRYPOINT Assignment 02

Running startup scripts with ENTRYPOINT

This assignment has you building a Dockerfile from scratch to perform startup tasks with an ENTRYPOINT script. This script will perform a few actions before starting the FastAPI Python web API server.

Dockerfile Requirements Copy Chapter

• This is a Python app, so you should start with the python:slim image. I always prefer slim images when available, as the default language images are usually too big and bloated for real-world use.
• Set the working directory to /app.
• Python apps often keep their dependencies in a requirements.txt file, so let's copy that in first, so we can install dependencies before we copy all the other files in.
• Install the dependencies with a run command for pip install --no-cache-dir -r requirements.txt
• Now let's copy all files in after requirements are installed.
• This app requires persistent data, so lets create a volume statement for /app/data.
• Set the entrypoint to run the startup script ./docker-entrypoint.sh.
• Set the cmd to start the uvicorn web server with uvicorn main:app --host 0.0.0.0 --port 8000.

Testing the Image Copy Chapter

• Once this image builds, you can run it with a -p 8000:8000 to make the web server available on http://localhost:8000
• The most interesting URL for testing is the built-in docs of FastAPI at http://localhost:8000/docs
• Python pip has a --no-cache-dir option to avoid caching the package index locally, so a command would look like pip install --no-cache-dir -r requirements.txt.
• Checkout the docker-entrypoint.sh script for the startup tasks. It's a simple script that checks several common things like ensuring the /app/data directory exists and copying over any initial data files (which you might want to peak at, 😎).

File: dockerfiles/entrypoint/assignment01/cmatrix/README.md Copy Chapter

ENTRYPOINT Assignment 01 Part 1 - cmatrix

Bulding a Matrix screensaver as a Docker Image

While this is just a fun tool, it's a good way to practice creating a Docker image for a single binary.

All we need to get started is the latest alpine image and a quick understanding of the apk package manager.

Requirements for this Dockerfile Copy Chapter

• No source code is needed, so we won't need to COPY any files into the image.
• Start FROM the alpine image. You can also practice pinning to a recent version (found on Docker Hub's alpine image page.) or :latest. I tend to care less about FROM image pinning for CLI tools like this.
• Use the apk package manager to install the cmatrix package.
• You'll want to set the ENTRYPOINT to run cmatrix when the container starts.
• Let's make this image easy to run by adding a CMD instruction with some default cmatrix arguments. For example, -abs -C red.
• The cmatrix tool needs a terminal to run, so you need to ensure docker provisions a tty. Always use the -it flags with this image when running the container, or you'll get a "Error opening terminal" error.

Tips Copy Chapter

• Tip: Alpine Linux uses the apk package manager, not apt or yum. The command to install a package is apk add .
• Tip: Use the --no-cache apk flag to avoid caching the package index locally, so a command would look like apk add --no-cache .
• Tip: As most Linux CLI tools, cmatrix has a --help option, so I welcome you to explore the options and customize the CMD instruction to your liking. Remember you could always build this image, then overwrite the ENTRYPOINT at runtime and then play around with cmatrix by running docker run -it --entrypoint sh

File: dockerfiles/entrypoint/assignment01/apachebench/README.md Copy Chapter

ENTRYPOINT Assignment 01 Part 2 - ApacheBench

Create an image for the ApacheBench HTTP server benchmarking tool

This Dockerfile Assignment will create a container image for the ApacheBench HTTP server benchmarking tool. This is a simple tool that can be used to test the performance of a web server by sending a large number of requests to it and measuring the response time.

We'll use the Ubuntu image with the apt-get package manager.

Requirements Copy Chapter

• No source code is needed, so we won't need to COPY any files into the image.
• Use the ubuntu image as the base image. You could pin to a specific version (LTS versions end in .04, so :20.04), or use :latest.
• Use the apt-get package manager to install the apache2-utils package, which contains the ab ApacheBench tool.
• Create an ENTRYPOINT that runs the ab command.
• Create an CMD that gives some default arguments to the ab command. For example, -n 10 -c 2 https://www.bretfisher.com/. -n 10 means 10 requests total, and -c 2 means 2 concurrent requests.
• OR, an easer way for tools like ab, curl, httping, that always need a URL added at the CLI, I find it easier to add some default arguments to the ENTRYPOINT, and leave the CMD as ["--help"] so that all the user needs to do is add the URL at the end of the docker run command. If they forget to override the CMD, they'll get the help output.

Tips Copy Chapter

• Tip: Ubuntu Linux uses the apt package manager. It requires a three step process (commands) with 1. update the db cache, 2. install packages, 3. cleanup the cache files. You can chain these three commands together with &&, so a typical install would look like apt-get update && apt-get install -y <list-of-packages> && rm -rf /var/lib/apt/lists/* and you can add line breaks for readability with a backslash \ at the end of the Dockerfile line.
• ApacheBench expects a URL as the last argument, so you'll need to pass that in when you run the container, and it always needs http:// or https:// in the URL, and a trailing slash after the domain name.
• Tip: As most Linux CLI tools, ab has a --help option, so I welcome you to explore the options and customize the CMD instruction to your liking. Remember you could always build this image, then overwrite the ENTRYPOINT at runtime and then play around with ab by running docker run -it --entrypoint sh

File: dockerfiles/entrypoint/assignment01/README.md Copy Chapter

ENTRYPOINT Assignment 01

Creating CLI programs with Docker ENTRYPOINT

This assignment has two different Dockerfiles for you to build that are meant to run a single executable.

1. The cmatrix subdirectory uses the Alpine Linux image and installs the cmatrix screen saver.
2. The apachebench subdirectory uses the Ubuntu image and installs the apache2-utils package for running the ab benchmarking tool.

File: intro/README.md Copy Chapter

Intro topics for the course

1. What is Docker?
2. Quick Container run on Play with Docker
3. Why Docker? Why Now?

2. Official Technical Reference & Guides (BretFisher/docs) Copy Chapter

File: content/overview/core-concepts.mdx Copy Chapter

title: Core Concepts ogTitle: Core Concepts of Depot Copy Chapter

Organizations Copy Chapter

Everything you do in Depot is within the context of an organization. An organization typically represents a single company or team. Billing is on a per-organization basis.

GitHub Connection Copy Chapter

Depot organizations can be connected to GitHub organizations in order to use Depot GitHub Actions Runners. This allows Depot runners to be used GitHub Actions workflows and jobs to accelerate your entire CI/CD pipeline.

Projects Copy Chapter

Projects are used for our remote container builds. A project is a cache namespace that is isolated from other projects. You can use a single namespace for a single git repository or Dockerfile or one namespace for multiple git repositories or Dockerfiles.

Projects are created within an organization and are used to store the Docker layer cache for your Docker image build on a per architecture basis. See our pricing page for more information on how many projects are included in each plan.

Cache Storage Policy Copy Section

A cache storage policy is specified per project. It defines how much cache to keep for each architecture you build. When the cache goes beyond that size, the oldest cache entries are deleted. The default cache storage policy is 50 GB, but is configurable up to 500 GB.

Builds Copy Chapter

A build in Depot is a Docker image build. When you run depot build your build context is sent to a remote builder running BuildKit. BuildKit performs the build and sends the resulting image back to your machine or a remote registry based on the options you passed with the build command. The resulting cache from the build is stored in the project's persistent cache and is available to all subsequent builds by users or CI providers.

Depot can build images on both x86 or arm machines supporting the following platforms:

• linux/amd64
• linux/arm64
• linux/arm/v6
• linux/arm/v7
• linux/386

Jobs Copy Chapter

A job in Depot is considered to be a GitHub Actions job. When you run a GitHub Actions job with a runs-on label like depot-ubuntu-latest, the job is run on a Depot runner. All of the steps for that job will execute on the Depot-managed GitHub Actions Runner. You must have an active GitHub connection configured in order to access Depot runners.

Cloud Connections Copy Chapter

A cloud connection links your AWS cloud account and your Depot organization. With a cloud connection configured, you can choose to have a given Depot project launch builders in your cloud instead of ours.

We currently only support cloud connections for AWS and you must be on a Business plan to use this feature.

File: content/overview/faq.mdx Copy Chapter

title: Frequently Asked Questions ogTitle: Frequently Asked Questions Copy Chapter

How many builds can a project run concurrently? Copy Chapter

You can run as many builds concurrently as you want against a single Depot project.

How do I use Depot with `docker-compose`? Copy Chapter

You can use depot bake -f docker-compose.yml to build all of the images in your Compose file and then use docker-compose up to run the resulting images.

How do you authenticate with Depot? Copy Chapter

We have all our authentication options documented for depot in our CLI authentication documentation.

How do I push my images to a private registry? Copy Chapter

You can use the --push flag to push your images to a private registry. Our depot CLI uses your local Docker credentials provider. So, any registry you've logged into with docker login or similar will be available when running a Depot build. See our guide on private registries for more details.

Can I build Docker images for M1/M2 Macs? Copy Chapter

Yes! Depot supports native Arm container builds out of the box. We detect the architecture of the machine requesting a build via depot build. If that architecture is Arm, we route the build to a builder running Arm natively. You can build Docker images for M1/M2 Macs and run the resulting image immediately, as it is made specifically for your architecture. See our documentation on Arm containers for more details.

Can I build multi-platform Docker images? Copy Chapter

Yes! Check out our integration guide on how we do it.

How should I use Depot with a monorepo setup? Copy Chapter

If you're building multiple images from a single monorepo, and the builds are lightweight, we tend to recommend using a single project. But we detail some other options in our monorepo guide.

Can I use Depot with my existing `docker build` or `docker buildx build` commands? Copy Chapter

Yes! We have a depot configure-docker command that configures Depot as a plugin for the Docker CLI and sets Depot as the default builder for both docker build and docker buildx build. See our docker build guide for more details.

What are these extra files in my registry? Copy Chapter

Registries like Amazon Elastic Container Registry (ECR) and Google Container Registry (GCR) don't accurately display provenance information for a given image. Provenance is a set of metadata that describes how an image was built. This metadata is stored in the registry alongside the image. It's enabled by default in docker build and thus by default in depot build as well.

If you would like to clean up the clutter, you can run your build with --provenance=false:

depot build -t <your-registry> --push --provenance=false .

Does Depot support building images in any lazy-pulling compatible format? e.g. estargz, nydus or others? Copy Chapter

Depot supports building images in any lazy-pulling compatible format. You can build an estargz image by setting the --output flag at build time:

depot build \
    --output "type=image,name=repo/image:tag,push=true,compression=estargz,oci-mediatypes=true force-compression=true" \
    .

Does Depot supporting building images with ztsd compression? Copy Chapter

Depot supports building images with zstd compression, a popular compression format to help speed up the launching of containers in AWS Fargate and Kubernetes. You can build an image with zstd compression by setting the --output flag at build time:

depot build \
    --output type=image,name=$IMAGE_URI:$IMAGE_TAG,oci-mediatypes=true,compression=zstd,compression-level=3 force-compression=true,push=true \
    .

What is an ephemeral build? Copy Chapter

We label builds as ephemeral when they are launched by GitHub Actions for an open-source pull request. It is a build that did not have access to read from or write to the project cache, to prevent untrusted code from accessing sensitive data.

File: content/overview/index.mdx Copy Chapter

title: Introduction ogTitle: What is Depot? Copy Chapter

Welcome to Depot!

Depot is a build acceleration platform focused on making builds radically more efficient and performant. We have three products that are available on all pricing plans:

1. Remote container builds — our remote container build service can build Docker images up to 40x faster
2. GitHub Actions Runners — fully-managed GitHub Actions runners with 30% faster compute, 10x faster caching, and half the cost of GitHub-hosted runners
3. Depot Build API - we offer all of our performance for container image builds behind a public API so that you can build container images from your own code on behalf of your own users

Key Features Copy Chapter

Depot makes your builds faster with as little additional configuration as possible. The overarching philosophy is to meet you where you're already building and give you tools & compute tailored to making your builds as fast as possible.

We have a few key features that compose Depot that are worth knowing about:

Build isolation & acceleration Copy Section

All builds run on ephemeral EC2 instances optimized for their specific use case, such as Docker image builds for our remote container builds service or GitHub Actions jobs for our managed GitHub Actions runners.

Builders are spun up on demand and terminated when the build is complete, so you only pay for the compute you use. Builders are never shared across Depot customers or projects. We've optimized our builders and provisioning system to start a given build as quickly as possible, typically under 10 seconds.

Our fast Docker image builders have 16 CPUs with 32 GB of memory and a default NVMe SSD of 50GB (configurable up to 500GB).

Our GitHub Actions runners can be configured to run on the default runner type of 2 CPUs, up to 32 CPUs for both Intel & Arm runners.

Persistent shared caching Copy Section

We have built a cache orchestration architecture designed to make your builds as fast as possible.

We automatically persist your Docker layer cache to fast NVMe storage and make it instantly available across builds. The layer cache is also shared across your entire team with access to the same project, so you can also benefit from your team's work.

For our GitHub Actions runners, we've brought 10x faster caching to GitHub Actions jobs by plugging in the same cache orchestration system that we use for our Docker image builds. You don't have to do anything to get this benefit; it's just there.

Multiple platforms Copy Section

We support native multi-platform Docker image builds with zero emulation. We build Docker images for both Intel & Arm without the need for qemu or any other emulation. We build Intel images on fast Xeon Scalable Ice Lake CPUs and Arm images on AWS Graviton3 CPUs.

We also support Intel & Arm runners with our managed GitHub Actions Runners. You can run any given job on either by specifying the architecture in your GitHub Actions runs-on label.

Drop-in replacement Copy Section

Using Depot for your Docker image builds is as straightforward as replacing your docker build command with depot build. We support all the same flags and options as docker build, and we've added a few features behind the scenes to make your build faster.

To use our GitHub Actions Runners, you can add a label to your GitHub Actions job to run it on our runners. We support all the same features as GitHub-hosted runners, but with better performance like 1 GB/s networks, 10x faster caching, and 30% faster compute.

CI & Dev tool integrations Copy Section

We have extensive integrations with most major CI providers and developer tools to make it easy to use Depot in your existing workflows. To see how to get faster Docker image builds with Depot in your CI provider, check out our CI provider docs.

For more information on how to use Depot with developer tools like Docker, Dev Containers, and Docker Compose, check out any of the docs below:

• How to use Depot in local development
• How to use Depot with Docker & Docker Compose
• How to use Depot with Dev Containers

OIDC integrations Copy Section

For our remote container builds service, we support OIDC trust relationships with GitHub, CircleCI, Buildkite, and Mint so that you don't need any static access tokens in your CI provider to leverage Depot. You can learn more about configuring a trust relationship in our authentication docs.

Ephemeral Registry Copy Section

Our remote container builds service comes with a built-in ephemeral registry that you can use to save the images from your depot build and depot bake commands to a temporary registry. You can then pull those images back down or push them onto your final registry as you see fit. You can read more in our ephemeral registry guide.

Next steps Copy Chapter

Once you sign up for an account, getting started with Depot is relatively straightforward.

Once you have an account setup, we recommend starting with one of our Quickstart guides based on which product you'd like to try out first:

• Looking for up to 40x faster Docker image builds? Check out our remote container builds quickstart.
• Looking to make your GitHub Actions jobs faster at half the cost? Check out our GitHub Actions Runners quickstart.

File: content/overview/security.mdx Copy Chapter

title: Security ogTitle: Overview of Depot architecture and security. Copy Chapter

For questions, concerns, or information about our security policies or to disclose a security vulnerability, please get in touch with us at [email protected].

Overview Copy Chapter

A Depot organization represents a collection of projects that contain builder VMs and SSD cache disks. These VMs and disks are associated with a single organization and are not shared across organizations.

When a build request arrives, the build is routed to the correct builder VM based on organization, project, and requested CPU architecture. Communication between the depot CLI and builder VM uses an encrypted HTTPS (TLS) connection.

Cache volumes are encrypted at rest using our infrastructure providers' encryption capabilities.

Our Responsibilities Copy Chapter

Single-tenant Builders Copy Section

A builder in Depot and its SSD cache are tied to a single project and the organization that owns it.
Builders are never shared across organizations. Instead, builds running on a given builder are connected to one and only one organization, the organization that owns the projects.

Connections from the Depot CLI to the builder VM are routed through a stateless load balancer directly to the project's builder VM and are encrypted using TLS (HTTPS).

Physical Security Copy Section

Our services and applications run in the cloud using one of our infrastructure providers, AWS and GCP. Depot has no physical access to the underlying physical infrastructure. For more information, see AWS's security details and GCP's security details.

Data Encryption Copy Section

All data transferred in and out of Depot is encrypted using hardened TLS. This includes connections between the Depot CLI and builder VMs, which are conducted via HTTPS. In addition, Depot's domain is protected by HTTP Strict Transport Security (HSTS).

Cache volumes attached to project builders are encrypted at rest using our infrastructure providers' encryption capabilities.

Data Privacy Copy Section

Depot does not access builders or cache volumes directly, except for use in debugging when explicit permission is granted from the organization owner.

Today, Depot operates cloud infrastructure in regions that are geographically located inside the United States of America as well as the European Union (if a project chooses the EU as its region).

API Token Security Copy Section

Depot supports API-token-based authentication for various aspects of the application:

• User access tokens are used by the Depot CLI to authenticate with Depot's internal API, access resources that the user is allowed to access based on their organization memberships and roles, and can be used to initiate a build request.
• OIDC tokens issued by authorized third-party services can be exchanged for temporary API tokens if the Depot project has configured a trust relationship with that third party. The ephemeral API token can only access the project(s) to which the OIDC entity was granted access. Today, Depot supports creating trust relationships with GitHub Actions, CircleCI, and Buildkite.
• Build mTLS certificates are used by the Depot CLI to authenticate with the builder VM — these certificates are issued for a single build in response to a successful build request and live only for the lifetime of the build.

Software Dependencies Copy Section

Depot keeps up to date with software dependencies and has automated tools scanning for dependency vulnerabilities.

Development Environments Copy Section

Development environments are separated physically from Depot's production environment.

Your Responsibilities Copy Chapter

Organization Access Copy Section

You can add and remove user access to your organization via the settings page. Users can have one of two roles:

• User — users can view all projects in your organization and run builds against any project.
• Owner — owners can create and delete projects, edit project settings, and edit organization settings.

We expect to expand the available roles and permissions in the future; don't hesitate to contact us if you have any special requirements.

In addition to users, Depot also allows creating trust relationships with GitHub Actions. These relationships enable workflow runs initiated in GitHub Actions to access specific projects in your organization to run builds. Trust relationships can be configured in the project settings.

Caching and Builder Access Copy Section

Access to create project builds effectively equates to access to the builder VM due to the nature of how docker build works. Anyone with access to build a project can access that project's build cache files and potentially add, edit, or remove cache entries. You should be careful that you trust the users and trust relationships that you have given access to a project and use tools like OIDC trust relationships to limit access to only the necessary scope.

File: content/managed/on-aws.mdx Copy Chapter

title: Depot Managed on AWS ogTitle: Deploying Depot Managed on AWS Copy Chapter

With Depot Managed on Amazon Web Services (AWS), the Depot data plane is deployed within an isolated sub-account of your AWS organization. You can use the Depot CLI, web application, and API, but the underlying build compute and cache infrastructure reside entirely within your own AWS account.

Architecture Copy Chapter

Setup and Usage Copy Chapter

NOTE: This guide is intended for Depot customers who are working with the Depot team, you cannot deploy Depot Managed on AWS without it being enabled for your Depot organization. Contact us if you are interested in using Depot Managed.

Step 1: Create a dedicated sub-account Copy Section

Depot Managed requires the use of a dedicated sub-account within your AWS organization. This should be a new account containing no other resources or services.

Follow the AWS documentation to create a new account within your organization.

Step 2: CloudFormation stack deployment Copy Section

Once you have created a new sub-account, you can deploy the following CloudFormation template to provision the required IAM permissions in the AWS sub-account. First, save the following as a file named depot-managed-bootstrap.json:

{
  "Resources": {
    "GrantProvisionerAccess": {
      "Type": "AWS::IAM::Role",
      "DeletionPolicy": "Retain",
      "Properties": {
        "RoleName": "DepotProvisioner",
        "ManagedPolicyArns": ["arn:aws:iam::aws:policy/AdministratorAccess"],
        "AssumeRolePolicyDocument": {
          "Statement": [
            {
              "Action": ["sts:AssumeRole"],
              "Effect": "Allow",
              "Principal": {
                "AWS": [
                  "arn:aws:iam::375021575472:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_Provisioner_572dd0a52dd9fc8e"
                ]
              }
            }
          ]
        }
      }
    },
    "GrantOpsAccess": {
      "Type": "AWS::IAM::Role",
      "DeletionPolicy": "Retain",
      "Properties": {
        "RoleName": "DepotOps",
        "ManagedPolicyArns": ["arn:aws:iam::aws:policy/AdministratorAccess"],
        "AssumeRolePolicyDocument": {
          "Statement": [
            {
              "Action": ["sts:AssumeRole"],
              "Effect": "Allow",
              "Principal": {
                "AWS": [
                  "arn:aws:iam::375021575472:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_Ops_e45adfee11ab7421"
                ]
              }
            }
          ]
        }
      }
    }
  }
}

Next, deploy the CloudFormation stack in the new sub-account:

aws cloudformation create-stack \
 --stack-name depot-managed-bootstrap \
 --template-body file://depot-managed-bootstrap.json \
 --capabilities CAPABILITY_NAMED_IAM

Step 3: Notify Depot Copy Section

Finally, let the Depot team know that the CloudFormation stack has been deployed, and they will initiate the deployment of the Depot data plane into the new sub-account.

The Depot team will additionally work with you on any follow-up steps, including:

• AWS quota increases to match your expected usage
• Configuring KMS keys for encryption
• Configuring S3 buckets for cache storage
• Configuring VPC peering for private networking
• Configuring AWS PrivateLink for secure API access
• Enabling enforced usage of Depot Managed in your Depot organization

Additional questions Copy Chapter

If you have any questions, please contact us, and we'll be happy to help.

File: content/managed/overview.mdx Copy Chapter

title: Depot Managed Overview ogTitle: Overview of Depot Managed Copy Chapter

With Depot Managed, the Depot data plane can be deployed in your own Amazon Web Services (AWS) account. You can still use the Depot CLI, web application, and API, however the underlying build compute and cache data reside entirely within your own cloud account.

We are considering support for additional cloud providers like Google Cloud (GCP) in the future. If you are interested in this, please let us know.

How Depot Managed works Copy Chapter

Depot Managed is the entirety of the Depot data plane, deployed in a single-tenant isolated sub-account within your AWS organization. Once deployed, you have the option of using Depot Managed with some or all of your Depot organization's projects. You will continue to use the same Depot CLI and web application, but the CLI will communicate directly with the compute and cache infrastructure running in your AWS account.

If you are an existing Depot user, moving to a Depot Managed deployment requires no changes to your existing developer workflows or CI pipelines.

Depot Managed is still a fully managed service and comes with the full support and SLA of the Depot Business plan. The Depot team is on-call for any issues that arise with the Depot Managed deployment.

For more information, see:

• Depot Managed on AWS

Benefits of Depot Managed Copy Chapter

Depot Managed comes with a few key benefits:

• Data residency: All build data and cache data reside within your own cloud account, ensuring that you have full control over your data.
• Compliance: For organizations that have strict compliance requirements.
• VPC peering & IAM: You can configure VPC peering and IAM roles to allow the Depot data plane to access your private cloud resources.
• AWS PrivateLink: You can use AWS PrivateLink to keep all communication between the Depot data plane and control plane within the AWS network.
• Cost control: You can take advantage of any existing cloud discounts or credits you have, and you can control the size and type of instances used for builds.
• AWS Marketplace: You can pay for Depot Managed through the AWS Marketplace and take advantage of any existing AWS billing arrangements you have.

How to get started Copy Chapter

Depot Managed is available on the Depot Business plan. If you are interested in Depot Managed, please contact us to chat with us and see if Depot Managed is a good fit for your organization.

File: content/languages/node.mdx Copy Chapter

title: Best practice Dockerfiles for Node.js ogTitle: Best practice Dockerfiles for Node.js Copy Chapter

We've assembled some best practice Dockerfiles for building Docker images for Node.js using different package managers. These Dockerfiles are what we recommend when building Docker images for Node applications, but they are not the only way to do it, so your mileage may vary.

Guides Copy Chapter

• Dockerfile for Node.js using pnpm

File: content/languages/node-pnpm-dockerfile.mdx Copy Chapter

title: Best practice Dockerfile for Node.js with pnpm ogTitle: Best practice Dockerfile for Node.js with pnpm Copy Chapter

Below is an example Dockerfile that we use and recommend at Depot when we are building Docker images for Node applications that use pnpm as their package manager.

FROM node:20 AS base

FROM base AS deps

RUN corepack enable
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm fetch --frozen-lockfile
RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm install --frozen-lockfile --prod

FROM base AS build

RUN corepack enable
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm fetch --frozen-lockfile
RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm install --frozen-lockfile
COPY . .
RUN pnpm build

FROM base

WORKDIR /app
COPY --from=deps /app/node_modules /app/node_modules
COPY --from=build /app/dist /app/dist
ENV NODE_ENV production
CMD ["node", "./dist/index.js"]

Explanation of the Dockerfile Copy Chapter

There are several things in this example Dockerfile that are worth calling out. Most notably, we use a multi-stage build to separate the installation of dependencies from the actual build of the application. This allows us to take advantage of Docker's layer caching to speed up our builds.

Stage 1: `FROM node:20 AS base`

FROM node:20 AS base

Here we use the node:20 base image and set the stage name to be reused in stages that follow. If we had common dependencies that we wanted to be accessible in any stages using this base stage, we could install them here.

Stage 2: `FROM base AS deps`

FROM base AS deps

RUN corepack enable
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm fetch --frozen-lockfile
RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm install --frozen-lockfile --prod

Next up is installing our dependencies via pnpm.

First, we enable corepack in the Node base image. Corepack allows us to use pnpm right out of the box without having to install it ourselves. It's a nice convenience but watch out that you have the expected pnpm version.

We then create our working directory, app, and copy in just our package.json and pnpm-lock.yaml files. Note that we do not copy in our entire codebase. We only care about installing our production dependencies at this stage. This is a best practice that allows us to take advantage of Docker's layer caching.

Finally, we get to installing our packages via pnpm. This is broken into two RUN statements that are making use of BuildKit cache mounts.

1. pnpm fetch --frozen-lockfile is a pnpm feature that is designed to improve building a Docker image. It fetches packages from the pnpm-lock.yaml file and stores them in the pnpm virtual store. Ignoring the package.json manifest. It's a nice optimization that avoids having to reinstall all packages in the package.json when a change occurs there that isn't related to the actual dependencies.

2. pnpm install --frozen-lockfile --prod is the actual installation of our dependencies. We use the --frozen-lockfile flag to ensure that we are installing the exact same versions of our dependencies that are in our package.json. We also use the --prod flag to ensure that we are not installing any devDependencies.

This is a best practice that allows us to take advantage of Docker's layer caching.

We use the --frozen-lockfile flag to ensure that we are installing the exact same versions of our dependencies that we have installed locally. We also use the --prod flag to ensure that we are only installing our production dependencies. This is a best practice that allows us to take advantage of Docker's layer caching.

Stage 3: `FROM base AS build`

FROM base AS build

RUN corepack enable
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm fetch --frozen-lockfile
RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm install --frozen-lockfile
COPY . .
RUN pnpm build

It's time to build our application. We start by enabling corepack again so that we have pnpm in this stage. We then configure our working directory to be app and copy in the package.json and pnpm-lock.yaml files as we saw in the deps stage.

We then run pnpm fetch --frozen-lockfile and pnpm install --frozen-lockfile again. However, we omit the --prod flag as we want to install all dependencies as we may need some dev ones to build our final application. If that wasn't the case, we could copy in our dependencies from our earlier deps stage.

Once our dependencies have been installed into this stage, we then copy in our source code via the COPY statement. We then run our build command, pnpm build.

Stage 4: `FROM base`

FROM base

WORKDIR /app
COPY --from=deps /app/node_modules /app/node_modules
COPY --from=build /app/dist /app/dist
ENV NODE_ENV production
CMD ["node", "./dist/index.js"]

Our final stage is copying all of the files from our earlier stages into our final image. We start by setting our working directory to app, then have several COPY statements:

1. We copy in our node_modules from our deps stage
2. We then copy in the dist directory from our build stage containing the outputs from our pnpm build

Finally, we set our NODE_ENV to production for any dependencies that have an optimized production mode. We then set our CMD to run our application.

File: content/languages/rust-dockerfile.mdx Copy Chapter

title: Best practice Dockerfile for Rust with cargo-chef and sccache ogTitle: Best practice Dockerfile for Rust with cargo-chef and sccache Copy Chapter

Below is an example Dockerfile that we have used and recommend at Depot for building images for Rust applications.

FROM rust:1.75 AS base
RUN cargo install --locked cargo-chef sccache
ENV RUSTC_WRAPPER=sccache SCCACHE_DIR=/sccache

FROM base AS planner
WORKDIR /app
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

FROM base as builder
WORKDIR /app
COPY --from=planner /app/recipe.json recipe.json
RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/usr/local/cargo/git \
    --mount=type=cache,target=$SCCACHE_DIR,sharing=locked \
    cargo chef cook --release --recipe-path recipe.json
COPY . .
RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/usr/local/cargo/git \
    --mount=type=cache,target=$SCCACHE_DIR,sharing=locked \
    cargo build

In addition to the standard best practices when writing Dockerfiles, here we are also leveraging cargo-chef and sccache to speed up our Rust build.

Using cargo-chef for dependency management Copy Chapter

When you install multiple crates with one command like cargo build, Docker treats any change in the output of cargo build as a change to the entire command. This means that Docker will attempt to execute that command again (re-downloading and installing **all **crates) every time you make an unrelated change to your source code or Dockerfile.

There are various workarounds online to manually manage and copy individual packages into your container while building in order to avoid invalidating the cache on every build, but these are cumbersome and prone to bugs. Our preferred solution is to use cargo-chef, which allows you to separate building the dependencies and building the source code so that Docker sees them as different steps and can cache them separately.

Using sccache for additional dependency management Copy Chapter

Even though cargo-chef separates your third-party dependencies from your source code, compiling and downloading your third-party dependencies is still considered one operation. This means that if a single dependency changes, there will be a cache miss and all of them will have to be re-downloaded and compiled, even though they haven't changed.

If you have a more fine-grained cache, you only have to rebuild the changed dependencies. Enter sccache, which caches individual compilation artifacts so that they can be reused at a more granular level during future compilations. This allows you to recompile individual dependencies only when needed, rather than everything or nothing.

Explanation of the Dockerfile Copy Chapter

At a high level, here are the things we're optimizing in our Docker build for a Rust application::

Multi-stage builds via multiple FROM statements

• cargo-chef for dependency management
• sccache for dependency caching
• BuildKit cache mounts for finer-grained caching between builds

Stage 1: `FROM rust:1.75 AS base` Copy Section

FROM rust:1.75 AS base
RUN cargo install --locked cargo-chef sccache
ENV RUSTC_WRAPPER=sccache SCCACHE_DIR=/sccache

Here, we use rust:1.75 as our base image and set the stage name to be used in later stages. In addition to this base image, we install sccache and cargo-chef.

We then set the SCCACHE_DIR environment variable so that sccache stores compilation artifacts in the /sccache directory and the RUSTC_WRAPPER environment variable so that Cargo “wraps” the execution of the Rust compiler commands in an sccache call. That way, we can take advantage of sccache's cached dependencies when building the final image.

Stage 2: `FROM base AS planner` Copy Section

FROM base AS planner
WORKDIR /app
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

Next, we're building off the base stage by creating the recipe that will later be used to build our application with cargo-chef. We use two commands from cargo-chef that together act as a replacement for the standard cargo build command when building dependencies.

Using `cargo-chef` to separate building dependencies from building source code

cargo chef prepare looks at your Cargo.toml and auto-generated Cargo.lock files, determines all of your dependencies, and then creates a recipe.json file, which is a dependency tree of your project.

By creating the dependency tree separately from the actual installation of dependencies, we can cache them independently so that all dependencies don't need to be rebuilt whenever the source code changes.

Stage 3: `FROM base as builder` Copy Section

FROM base as builder
WORKDIR /app
COPY --from=planner /app/recipe.json recipe.json
RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/usr/local/cargo/git \
    --mount=type=cache,target=$SCCACHE_DIR,sharing=locked \
    cargo chef cook --release --recipe-path recipe.json
COPY . .
RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/usr/local/cargo/git \
    --mount=type=cache,target=$SCCACHE_DIR,sharing=locked \
    cargo build

cargo chef cook takes the recipe.json file and runs cargo build under the hood on each package independently. In this stage, we're copying the recipe.json file from the previous planner stage. If the recipe has not changed, then the step to build the dependencies with cargo chef cook can be skipped. After the dependencies have been built, we finally build the source code with the final cargo build command.

Using `sccache` to retain dependencies between builds

We use a cache mount to attach the sccache directory to the build. This type of cache mount gives you a more fine-grained level of caching that allows you to skip recomputing bits of work when the layer cache invalidates. Using sccache with a cache mount allows you to skip rebuilding compiled artifacts that have already been built even when certain layers in the build are invalidated.

Caching the Cargo registry and git directories

In this stage, we also use additional cache mounts to store the Cargo registry and git directories. The Cargo registry stores the crates that have already been downloaded, and the git directory stores any git dependencies. Usually only one or a few packages have actually changed and need to be re-downloaded and installed. All the other packages can be reused from previous builds with the cache mounts.