## 1. Project Overview & Quickstart (BretFisher/udemy-docker-mastery) ## File: README.md # Udemy Course Docker Mastery: with Kubernetes+Swarm from a Docker Captain [](https://discord.gg/devops) > Build, test, and deploy containers with the best mega-course on Docker, Kubernetes, Compose, Swarm and Registry using a cloud native DevOps mindset This repository is for use in my Udemy Courses "[Docker Mastery](https://www.udemy.com/course/docker-mastery/?referralCode=1410924A733D33635CCB )" and "[Swarm Mastery](https://www.udemy.com/course/docker-swarm-mastery/?referralCode=4927D9CB156D4AE0228C)" - Get these courses with my "cheapest on the internet" coupon links: [bretfisher.com/courses](https://www.bretfisher.com/courses) - Get the [weekly newsletter](https://www.bretfisher.com/newsletter) on all the cloud native DevOps content I'm creating. ## Course Sections 1. Quick Start! 1. [README](./intro/README.md) 2. [Commands and Links](./references/S01%20Commands%20and%20Links.md) 2. Course Introduction 3. The Best Way to Setup Docker for Your OS 4. Creating and Using Containers Like a Boss 5. Container Images, Where To Find Them and How To Build Them 6. Container Lifetime & Persistent Data: Volumes, Volumes, Volumes 7. Making It Easier with Docker Compose: The Multi-Container Tool 8. Swarm Intro and Creating a 3-Node Swarm Cluster 9. Swarm Basic Features and How to Use Them In Your Workflow 10. Swarm App Lifecycle 11. Container Registries: Image Storage and Distribution 12. Docker in Production 13. The What and Why of Kubernetes 14. Kubernetes Architecture and Install 15. Your First Pods 16. Inspecting Kubernetes Resources 17. Exposing Kubernetes Ports 18. Kubernetes Management Techniques 19. Moving to Declarative Kubernetes YAML 20. Your Next Steps and The Future of Kubernetes 21. Automated CI Workflows 22. GitHub Actions Workflow Examples 23. Docker Security Good Defaults and Tools 24. Docker 19.03 Release New Features 25. DevOps and Docker Clips 26. Dockerfiles and Docker Images in 2022 27. Dockerfile and Compose File Reviews 28. Extra's, Common Questions, and Resources Feel free to create issues or PRs if you find a problem with examples in this repository! --- ## File: compose-assignment-1/README.md # Assignment: Build a Compose File For a Multi-Container Project > Goal: Create a compose config for a local Drupal CMS website [Docker Mastery Lecture](https://www.udemy.com/course/docker-mastery/learn/lecture/6775804) - This empty directory is where you should create a docker-compose.yml - Use the `drupal:9` image along with the `postgres:14` image - 2020 UPDATE: No version needed in compose files! All 2.x and 3.x features supported without version key. - Use `ports` to expose Drupal on 8080 (or your preferred port for localhost access) - Be sure to setup the POSTGRES_PASSWORD environment variable on the postgres service - Also note that the postgres official image defaults to user:postgres and database:postgres - Walk through the Drupal config steps in browser at `http://localhost:8080` - Tip: Drupal setup will assume the database runs on localhost, which is incorrect. You'll need to change it under Advanced settings to the name of the Docker service you gave to postgres - Use Docker Hub documentation to figure out the right environment and volume settings - Extra Credit: Use volumes to store Drupal unique data --- ## File: compose-assignment-2/README.md # Assignment: Compose For Image Building > Goal: This time imagine you're just wanting to learn Drupal's admin and GUI, or maybe you're a software tester and you need to test a new theme for Drupal. When configured properly, this will let you build a custom image and start everything with `docker-compose up` including storing important db and config data in volumes so the site will remember your changes across Compose restarts. [Docker Mastery Lecture](https://www.udemy.com/course/docker-mastery/learn/lecture/6775804) - Use the compose file you created in the last assignment (drupal and postgres) as a starting point. - Let's pin image version from Docker Hub this time. It's always a good idea to do that so a new major version doesn't surprise you. ## Dockerfile - First you need to build a custom Dockerfile in this directory, `FROM drupal:9` - Then RUN apt package manager command to install git: `apt-get update && apt-get install -y git` - Remember to cleanup after your apt install with `rm -rf /var/lib/apt/lists/*` and use `\` and `&&` properly. You can find examples of them in drupal official image. More on this below under Compose file. - Then change `WORKDIR /var/www/html/themes` - Then use git to clone the theme with: `RUN git clone --branch 8.x-4.x --single-branch --depth 1 https://git.drupalcode.org/project/bootstrap.git` - Combine that line with this line, as we need to change permissions on files and don't want to use another image layer to do that (it creates size bloat). This drupal container runs as www-data user but the build actually runs as root, so often we have to do things like `chown` to change file owners to the proper user: `chown -R www-data:www-data bootstrap`. Remember the first line needs a `\` at end to signify the next line is included in the command, and at start of next line you should have `&&` to signify "if first command succeeds then also run this command" - Then, just to be safe, change the working directory back to its default (from drupal image) at `/var/www/html` ## Compose File - We're going to build a custom image in this compose file for drupal service. Use Compose file from previous assignment for Drupal to start with, and we'll add to it, as well as change image name. - Rename image to `custom-drupal` as we want to make a new image based on the official `drupal:9`. - We want to build the default Dockerfile in this directory by adding `build: .` to the `drupal` service. When we add a build + image value to a compose service, it knows to use the image name to write to in our image cache, rather then pull from Docker Hub. - For the `postgres:14` service, you need the same password as in previous assignment, but also add a volume for `drupal-data:/var/lib/postgresql/data` so the database will persist across Compose restarts. ## Start Containers, Configure Drupal - Start containers like before, configure Drupal web install like before. - After site comes up, click on `Appearance` in top bar, and notice a new theme called`Bootstrap` is there. That's the one we added with our custom Dockerfile. - Click `Install and set as default`. Then click `Back to site` (in top left) and the site interface should look different. You've successfully installed and activated a new theme in your own custom image without installing anything on your host other than Docker! - If you exit (ctrl-c) and then `docker-compose down` it will delete containers, but not the volumes, so on next `docker-compose up` everything will be as it was. - To totally clean up volumes, add `-v` to `down` command. ## BONUS: Use MariaDB rather than Postgres In newer versions of Drupal, they've begun to prefer MySQL or MariaDB over Postgres. You could try installing it using the [recommendations guide on Drupal.org](https://www.drupal.org/docs/system-requirements/database-server-requirements). --- ## File: compose-sample-3/html/README.md # [Start Bootstrap](http://startbootstrap.com/) - [Agency](http://startbootstrap.com/template-overviews/agency/) [Agency](http://startbootstrap.com/template-overviews/agency/) is a one page agency portfolio theme for [Bootstrap](http://getbootstrap.com/) created by [Start Bootstrap](http://startbootstrap.com/). This theme features several content sections, a responsive portfolio grid with hover effects, full page portfolio item modals, a responsive timeline, and a working PHP contact form. ## Getting Started Several options are available to get started quickly: * [Download the latest release on Start Bootstrap](http://startbootstrap.com/template-overviews/agency/) * Clone the repo: `git clone https://github.com/BlackrockDigital/startbootstrap-agency.git` * Fork the repo ## Developing Using Source Files To use the source files, you will need to have npm installed globally along with Gulp.js. To start: * Run `npm install` in the root directory * Run `gulp dev` and edit the files as needed If you need to update the plugins included with this template, simply run the following tasks: * First run `npm update` to update the dependencies * Then run `gulp copy` to copy the new versions to their proper destinations ## Bugs and Issues Have a bug or an issue with this template? [Open a new issue](https://github.com/BlackrockDigital/startbootstrap-agency/issues) here on GitHub or leave a comment on the [template overview page at Start Bootstrap](http://startbootstrap.com/template-overviews/agency/). ## Creator Start Bootstrap was created by and is maintained by **[David Miller](http://davidmiller.io/)**, Owner of [Blackrock Digital](http://blackrockdigital.io/). * https://twitter.com/davidmillerskt * https://github.com/davidtmiller Start Bootstrap is based on the [Bootstrap](http://getbootstrap.com/) framework created by [Mark Otto](https://twitter.com/mdo) and [Jacob Thorton](https://twitter.com/fat). ## Copyright and License Copyright 2013-2017 Blackrock Digital LLC. Code released under the [MIT](https://github.com/BlackrockDigital/startbootstrap-agency/blob/gh-pages/LICENSE) license. --- ## File: dockerfile-assignment-1/README.md # Instructions for Assignment [Build Your Own Dockerfile and Run Containers From It](https://www.udemy.com/course/docker-mastery/learn/lecture/6806638) This directory contains a Node.js app, and you need to get it running in a container. No modifications to the app should be necessary, only edits to the Dockerfile in this directory. ## Overview of this assignment - Imagine another developer gave you the instruction below, and you'll need to interpret them into what the Dockerfile should contain. - Once the Dockerfile builds correctly, start container locally to make sure it works on [http://localhost](http://localhost) - Then ensure the image is named properly for your Docker Hub account with a new repository name - Then push to Docker Hub, and go to [https://hub.docker.com](https://hub.docker.com) to verify you pushed it correctly - Then remove local image from cache (`docker image rm `) - Then start a new container from your Hub image, and watch how it auto downloads and runs - Test again that it works at [http://localhost](http://localhost) ## Instructions from the app developer - you should use the `node` official image, with the alpine 6.x branch (`node:6-alpine`) - (Yes this is a 2-year old image of node, but all official images are always available on Docker Hub forever, to ensure even old apps still work. It is common to still need to deploy old app versions, even years later.) - This app listens on port 3000, but the container should listen on port 80 of the Docker host, so it will respond to [http://localhost:80](http://localhost:80) on your computer - Then it should use the alpine package manager to install tini: `apk add --no-cache tini`. - Then it should create directory /usr/src/app for app files with `mkdir -p /usr/src/app`, or with the Dockerfile command `WORKDIR /usr/src/app`. - Node.js uses a "package manager", so it needs to copy in package.json file. - Then it needs to run 'npm install' to install dependencies from that file. - To keep it clean and small, run `npm cache clean --force` after the above, in the same RUN command. - Then it needs to copy in all files from current directory into the image. - Then it needs to start the container with the command `/sbin/tini -- node ./bin/www`. Be sure to use JSON array syntax for CMD. (`CMD [ "something", "something" ]`) - In the end you should be using FROM, RUN, WORKDIR, COPY, EXPOSE, and CMD commands ## Bonus Extra Credit - This assignment will not have you setting up a complete image useful for local development, test, and prod. It's just meant to get you started with basic Dockerfile concepts and not focus too much on proper Node.js use in a container. **If you happen to be a Node.js Developer**, then after you get through more of this course, you should come back and use my [Node.js Docker Good Defaults](https://github.com/BretFisher/node-docker-good-defaults) sample project on GitHub to change this Dockerfile for better local development with more advanced topics. --- ## File: dockerfiles/entrypoint/assignment02/README.md # ENTRYPOINT Assignment 02 > Running startup scripts with ENTRYPOINT This assignment has you building a Dockerfile from scratch to perform startup tasks with an ENTRYPOINT script. This script will perform a few actions before starting the FastAPI Python web API server. ## Dockerfile Requirements - This is a Python app, so you should start with the `python:slim` image. I always prefer slim images when available, as the default language images are usually too big and bloated for real-world use. - Set the working directory to `/app`. - Python apps often keep their dependencies in a requirements.txt file, so let's copy that in first, so we can install dependencies before we copy all the other files in. - Install the dependencies with a run command for `pip install --no-cache-dir -r requirements.txt` - Now let's copy all files in after requirements are installed. - This app requires persistent data, so lets create a volume statement for `/app/data`. - Set the entrypoint to run the startup script `./docker-entrypoint.sh`. - Set the cmd to start the uvicorn web server with `uvicorn main:app --host 0.0.0.0 --port 8000`. ## Testing the Image - Once this image builds, you can run it with a `-p 8000:8000` to make the web server available on http://localhost:8000 - The most interesting URL for testing is the built-in docs of FastAPI at http://localhost:8000/docs - Python pip has a `--no-cache-dir` option to avoid caching the package index locally, so a command would look like `pip install --no-cache-dir -r requirements.txt`. - Checkout the `docker-entrypoint.sh` script for the startup tasks. It's a simple script that checks several common things like ensuring the `/app/data` directory exists and copying over any initial data files (which you might want to peak at, 😎). --- ## File: dockerfiles/entrypoint/assignment01/cmatrix/README.md # ENTRYPOINT Assignment 01 Part 1 - cmatrix > Bulding a Matrix screensaver as a Docker Image While this is just a fun tool, it's a good way to practice creating a Docker image for a single binary. All we need to get started is the latest alpine image and a quick understanding of the apk package manager. ## Requirements for this Dockerfile - No source code is needed, so we won't need to COPY any files into the image. - Start FROM the `alpine` image. You can also practice pinning to a recent version (found on Docker Hub's alpine image page.) or `:latest`. I tend to care less about FROM image pinning for CLI tools like this. - Use the `apk` package manager to install the `cmatrix` package. - You'll want to set the ENTRYPOINT to run `cmatrix` when the container starts. - Let's make this image easy to run by adding a `CMD` instruction with some default cmatrix arguments. For example, `-abs -C red`. - The cmatrix tool needs a terminal to run, so you need to ensure docker provisions a tty. Always use the `-it` flags with this image when running the container, or you'll get a "Error opening terminal" error. ## Tips - Tip: Alpine Linux uses the `apk` package manager, not `apt` or `yum`. The command to install a package is `apk add `. - Tip: Use the `--no-cache` apk flag to avoid caching the package index locally, so a command would look like `apk add --no-cache `. - Tip: As most Linux CLI tools, cmatrix has a --help option, so I welcome you to explore the options and customize the CMD instruction to your liking. Remember you could always build this image, then overwrite the ENTRYPOINT at runtime and then play around with cmatrix by running `docker run -it --entrypoint sh ` --- ## File: dockerfiles/entrypoint/assignment01/apachebench/README.md # ENTRYPOINT Assignment 01 Part 2 - ApacheBench > Create an image for the ApacheBench HTTP server benchmarking tool This Dockerfile Assignment will create a container image for the ApacheBench HTTP server benchmarking tool. This is a simple tool that can be used to test the performance of a web server by sending a large number of requests to it and measuring the response time. We'll use the Ubuntu image with the apt-get package manager. ## Requirements - No source code is needed, so we won't need to COPY any files into the image. - Use the `ubuntu` image as the base image. You could pin to a specific version (LTS versions end in .04, so `:20.04`), or use `:latest`. - Use the `apt-get` package manager to install the `apache2-utils` package, which contains the `ab` ApacheBench tool. - Create an ENTRYPOINT that runs the `ab` command. - Create an CMD that gives some default arguments to the `ab` command. For example, `-n 10 -c 2 https://www.bretfisher.com/`. `-n 10` means 10 requests total, and `-c 2` means 2 concurrent requests. - OR, an easer way for tools like ab, curl, httping, that always need a URL added at the CLI, I find it easier to add some default arguments to the ENTRYPOINT, and leave the CMD as `["--help"]` so that all the user needs to do is add the URL at the end of the `docker run` command. If they forget to override the CMD, they'll get the help output. ## Tips - Tip: Ubuntu Linux uses the `apt` package manager. It requires a three step process (commands) with 1. update the db cache, 2. install packages, 3. cleanup the cache files. You can chain these three commands together with &&, so a typical install would look like `apt-get update && apt-get install -y && rm -rf /var/lib/apt/lists/*` and you can add line breaks for readability with a backslash `\` at the end of the Dockerfile line. - ApacheBench expects a URL as the last argument, so you'll need to pass that in when you run the container, and it always needs http:// or https:// in the URL, and a trailing slash after the domain name. - Tip: As most Linux CLI tools, ab has a --help option, so I welcome you to explore the options and customize the CMD instruction to your liking. Remember you could always build this image, then overwrite the ENTRYPOINT at runtime and then play around with ab by running `docker run -it --entrypoint sh ` --- ## File: dockerfiles/entrypoint/assignment01/README.md # ENTRYPOINT Assignment 01 > Creating CLI programs with Docker ENTRYPOINT This assignment has two different Dockerfiles for you to build that are meant to run a single executable. 1. The cmatrix subdirectory uses the Alpine Linux image and installs the cmatrix screen saver. 2. The apachebench subdirectory uses the Ubuntu image and installs the apache2-utils package for running the ab benchmarking tool. --- ## File: intro/README.md # Intro topics for the course 1. [What is Docker?](what-is-docker/what-is-docker.md) 2. [Quick Container run on Play with Docker](quick-container-run/quick-container-run.md) 3. [Why Docker? Why Now?](why-docker/why-docker.md) ## 2. Official Technical Reference & Guides (BretFisher/docs) ## File: content/overview/core-concepts.mdx --- title: Core Concepts ogTitle: Core Concepts of Depot --- ## Organizations Everything you do in Depot is within the context of an organization. An organization typically represents a single company or team. Billing is on a per-organization basis. ## GitHub Connection Depot organizations can be connected to GitHub organizations in order to use [Depot GitHub Actions Runners](/docs/github-actions/overview). This allows Depot runners to be used GitHub Actions workflows and jobs to accelerate your entire CI/CD pipeline. ## Projects Projects are used for our [remote container builds](/docs/container-builds/overview). A project is a cache namespace that is isolated from other projects. You can use a single namespace for a single git repository or Dockerfile or one namespace for multiple git repositories or Dockerfiles. Projects are created within an organization and are used to store the Docker layer cache for your Docker image build on a per architecture basis. See our [pricing page](/pricing) for more information on how many projects are included in each plan. ### Cache Storage Policy A cache storage policy is specified per project. It defines how much cache to keep for each architecture you build. When the cache goes beyond that size, the oldest cache entries are deleted. The default cache storage policy is 50 GB, but is configurable up to 500 GB. ## Builds A build in Depot is a Docker image build. When you run `depot build` your build context is sent to a remote builder running BuildKit. BuildKit performs the build and sends the resulting image back to your machine or a remote registry based on the options you passed with the build command. The resulting cache from the build is stored in the project's persistent cache and is available to all subsequent builds by users or CI providers. Depot can build images on both `x86` or `arm` machines supporting the following platforms: - `linux/amd64` - `linux/arm64` - `linux/arm/v6` - `linux/arm/v7` - `linux/386` ## Jobs A job in Depot is considered to be a GitHub Actions job. When you run a GitHub Actions job with a `runs-on` label like `depot-ubuntu-latest`, the job is run on a Depot runner. All of the steps for that job will execute on the Depot-managed GitHub Actions Runner. You must have an active [GitHub connection configured](/docs/github-actions/quickstart#connect-to-github) in order to access Depot runners. ## Cloud Connections A cloud connection links your AWS cloud account and your Depot organization. With a cloud connection configured, you can choose to have a given Depot project launch builders in your cloud instead of ours. We currently only support cloud connections for **AWS** and you must be on a **Business** plan to use this feature. --- ## File: content/overview/faq.mdx --- title: Frequently Asked Questions ogTitle: Frequently Asked Questions --- ## How many builds can a project run concurrently? You can run as many builds concurrently as you want against a single Depot project. ## How do I use Depot with `docker-compose`? You can use [`depot bake -f docker-compose.yml`](/docs/cli/reference#depot-bake) to build all of the images in your Compose file and then use `docker-compose up` to run the resulting images. ## How do you authenticate with Depot? We have all our authentication options documented for `depot` in our [CLI authentication documentation](/docs/cli/authentication). ## How do I push my images to a private registry? You can use the `--push` flag to push your images to a private registry. Our `depot` CLI uses your local Docker credentials provider. So, any registry you've logged into with `docker login` or similar will be available when running a Depot build. See our guide on [private registries](/docs/guides/private-registries) for more details. ## Can I build Docker images for M1/M2 Macs? Yes! Depot supports native Arm container builds out of the box. We detect the architecture of the machine requesting a build via `depot build`. If that architecture is Arm, we route the build to a builder running Arm natively. You can build Docker images for M1/M2 Macs and run the resulting image immediately, as it is made specifically for your architecture. See our documentation on [Arm containers](/docs/guides/arm-containers) for more details. ## Can I build multi-platform Docker images? Yes! Check out our [integration guide](/docs/guides/arm-containers#what-about-multi-architecture-containers) on how we do it. ## How should I use Depot with a monorepo setup? If you're building multiple images from a single monorepo, and the builds are lightweight, we tend to recommend using a single project. But we detail some other options in our [monorepo guide](/blog/how-to-use-depot-in-monorepos). ## Can I use Depot with my existing `docker build` or `docker buildx build` commands? Yes! We have a [`depot configure-docker`](/docs/cli/reference#depot-configure-docker) command that configures Depot as a plugin for the Docker CLI and sets Depot as the default builder for both `docker build` and `docker buildx build`. See our [`docker build` guide](/docs/guides/docker-build) for more details. ## What are these extra files in my registry? Registries like Amazon Elastic Container Registry (ECR) and Google Container Registry (GCR) don't accurately display provenance information for a given image. Provenance is a set of metadata that describes how an image was built. This metadata is stored in the registry alongside the image. It's enabled by default in `docker build` and thus by default in `depot build` as well. If you would like to clean up the clutter, you can run your build with `--provenance=false`: ```shell depot build -t --push --provenance=false . ``` ## Does Depot support building images in any lazy-pulling compatible format? e.g. estargz, nydus or others? Depot supports building images in any lazy-pulling compatible format. You can build an estargz image by setting the `--output` flag at build time: ```shell depot build \ --output "type=image,name=repo/image:tag,push=true,compression=estargz,oci-mediatypes=true force-compression=true" \ . ``` ## Does Depot supporting building images with ztsd compression? Depot supports building images with `zstd` compression, a popular compression format to help speed up the launching of containers in AWS Fargate and Kubernetes. You can build an image with zstd compression by setting the `--output` flag at build time: ```shell depot build \ --output type=image,name=$IMAGE_URI:$IMAGE_TAG,oci-mediatypes=true,compression=zstd,compression-level=3 force-compression=true,push=true \ . ``` ## What is an ephemeral build? We label builds as `ephemeral` when they are launched by GitHub Actions for an open-source pull request. It is a build that did not have access to read from or write to the project cache, to prevent untrusted code from accessing sensitive data. --- ## File: content/overview/index.mdx --- title: Introduction ogTitle: What is Depot? --- Welcome to Depot! Depot is a build acceleration platform focused on making builds radically more efficient and performant. We have three products that are available on **all pricing plans**: 1. **[Remote container builds](/docs/container-builds/overview)** β€” our remote container build service can build Docker images up to 40x faster 2. **[GitHub Actions Runners](/docs/github-actions/overview)** β€” fully-managed GitHub Actions runners with 30% faster compute, 10x faster caching, and half the cost of GitHub-hosted runners 3. **[Depot Build API](/docs/api/overview)** - we offer all of our performance for container image builds behind a public API so that you can build container images from your own code on behalf of your own users ## Key Features Depot makes your builds faster with as little additional configuration as possible. The overarching philosophy is to meet you where you're already building and give you tools & compute tailored to making your builds as fast as possible. We have a few key features that compose Depot that are worth knowing about: ### Build isolation & acceleration All builds run on ephemeral EC2 instances optimized for their specific use case, such as Docker image builds for our [remote container builds service](/docs/container-builds/overview) or GitHub Actions jobs for our [managed GitHub Actions runners](/docs/github-actions/overview). Builders are spun up on demand and terminated when the build is complete, so you only pay for the compute you use. Builders are never shared across Depot customers or projects. We've optimized our builders and provisioning system to start a given build as quickly as possible, typically under 10 seconds. Our fast Docker image builders have 16 CPUs with 32 GB of memory and a default NVMe SSD of 50GB (configurable up to 500GB). Our GitHub Actions runners can be configured to run on the default runner type of 2 CPUs, up to 32 CPUs for both Intel & Arm runners. ### Persistent shared caching We have built a cache orchestration architecture designed to make your builds as fast as possible. We automatically persist your Docker layer cache to fast NVMe storage and make it instantly available across builds. The layer cache is also shared across your entire team with access to the same project, so you can also benefit from your team's work. For our GitHub Actions runners, we've brought 10x faster caching to GitHub Actions jobs by plugging in the same cache orchestration system that we use for our Docker image builds. You don't have to do anything to get this benefit; it's just there. ### Multiple platforms We support native multi-platform Docker image builds with **zero emulation.** We build Docker images for both Intel & Arm without the need for `qemu` or any other emulation. We build Intel images on fast Xeon Scalable Ice Lake CPUs and Arm images on AWS Graviton3 CPUs. We also support Intel & Arm runners with our managed GitHub Actions Runners. You can run any given job on either by specifying the architecture in your [GitHub Actions `runs-on` label.](/docs/github-actions/quickstart#depot-supported-labels) ### Drop-in replacement Using Depot for your Docker image builds is as straightforward as replacing your `docker build` command with `depot build`. We support all the same flags and options as `docker build`, and we've added a few features behind the scenes to make your build faster. To use our GitHub Actions Runners, you can add a label to your GitHub Actions job to run it on our runners. We support all the same features as GitHub-hosted runners, but with better performance like 1 GB/s networks, 10x faster caching, and 30% faster compute. ### CI & Dev tool integrations We have extensive integrations with most major CI providers and developer tools to make it easy to use Depot in your existing workflows. To see how to get faster Docker image builds with Depot in your CI provider, check out our [CI provider docs.](/docs/guides/continuous-integration) For more information on how to use Depot with developer tools like Docker, Dev Containers, and Docker Compose, check out any of the docs below: - [How to use Depot in local development](/docs/guides/local-development) - [How to use Depot with Docker & Docker Compose](/docs/guides/docker-build) - [How to use Depot with Dev Containers](/docs/guides/devcontainers) ### OIDC integrations For our [remote container builds service](/docs/container-builds/overview), we support OIDC trust relationships with GitHub, CircleCI, Buildkite, and Mint so that you don't need any static access tokens in your CI provider to leverage Depot. You can learn more about configuring a trust relationship in our [authentication docs.](/docs/cli/authentication) ### Ephemeral Registry Our remote container builds service comes with a built-in ephemeral registry that you can use to save the images from your `depot build` and `depot bake` commands to a temporary registry. You can then pull those images back down or push them onto your final registry as you see fit. You can read more in our [ephemeral registry guide.](/docs/guides/ephemeral-registry) ## Next steps Once you [sign up for an account](/start), getting started with Depot is relatively straightforward. Once you have an account setup, we recommend starting with one of our Quickstart guides based on which product you'd like to try out first: - Looking for up to 40x faster Docker image builds? [Check out our remote container builds quickstart.](/docs/container-builds/quickstart) - Looking to make your GitHub Actions jobs faster at half the cost? [Check out our GitHub Actions Runners quickstart.](/docs/github-actions/quickstart) --- ## File: content/overview/security.mdx --- title: Security ogTitle: Overview of Depot architecture and security. --- For questions, concerns, or information about our security policies or to disclose a security vulnerability, please get in touch with us at [security@depot.dev](mailto:security@depot.dev). ## Overview A Depot organization represents a collection of projects that contain builder VMs and SSD cache disks. These VMs and disks are associated with a single organization and are not shared across organizations. When a build request arrives, the build is routed to the correct builder VM based on organization, project, and requested CPU architecture. Communication between the `depot` CLI and builder VM uses an encrypted HTTPS (TLS) connection. Cache volumes are encrypted at rest using our infrastructure providers' encryption capabilities. ## Our Responsibilities ### Single-tenant Builders A builder in Depot and its SSD cache are tied to a single project and the organization that owns it. Builders are never shared across organizations. Instead, builds running on a given builder are connected to one and only one organization, the organization that owns the projects. Connections from the Depot CLI to the builder VM are routed through a stateless load balancer directly to the project's builder VM and are encrypted using TLS (HTTPS). ### Physical Security Our services and applications run in the cloud using one of our infrastructure providers, AWS and GCP. Depot has no physical access to the underlying physical infrastructure. For more information, see [AWS's security details](https://aws.amazon.com/security/) and [GCP's security details](https://cloud.google.com/docs/security/infrastructure/design). ### Data Encryption All data transferred in and out of Depot is encrypted using hardened TLS. This includes connections between the Depot CLI and builder VMs, which are conducted via HTTPS. In addition, Depot's domain is protected by HTTP Strict Transport Security (HSTS). Cache volumes attached to project builders are encrypted at rest using our infrastructure providers' encryption capabilities. ### Data Privacy Depot does not access builders or cache volumes directly, except for use in debugging when explicit permission is granted from the organization owner. Today, Depot operates cloud infrastructure in regions that are geographically located inside the United States of America as well as the European Union (if a project chooses the EU as its region). ### API Token Security Depot supports API-token-based authentication for various aspects of the application: - **User access tokens** are used by the Depot CLI to authenticate with Depot's internal API, access resources that the user is allowed to access based on their organization memberships and roles, and can be used to initiate a build request. - **OIDC tokens** issued by authorized third-party services can be exchanged for temporary API tokens if the Depot project has configured a trust relationship with that third party. The ephemeral API token can only access the project(s) to which the OIDC entity was granted access. Today, Depot supports creating trust relationships with GitHub Actions, CircleCI, and Buildkite. - **Build mTLS certificates** are used by the Depot CLI to authenticate with the builder VM β€” these certificates are issued for a single build in response to a successful build request and live only for the lifetime of the build. ### Software Dependencies Depot keeps up to date with software dependencies and has automated tools scanning for dependency vulnerabilities. ### Development Environments Development environments are separated physically from Depot's production environment. ## Your Responsibilities ### Organization Access You can add and remove user access to your organization via the settings page. Users can have one of two roles: - **User** β€” users can view all projects in your organization and run builds against any project. - **Owner** β€” owners can create and delete projects, edit project settings, and edit organization settings. We expect to expand the available roles and permissions in the future; don't hesitate to contact us if you have any special requirements. In addition to users, Depot also allows creating trust relationships with GitHub Actions. These relationships enable workflow runs initiated in GitHub Actions to access specific projects in your organization to run builds. Trust relationships can be configured in the project settings. ### Caching and Builder Access Access to create project builds effectively equates to access to the builder VM due to the nature of how `docker build` works. Anyone with access to build a project can access that project's build cache files and potentially add, edit, or remove cache entries. You should be careful that you trust the users and trust relationships that you have given access to a project and use tools like OIDC trust relationships to limit access to only the necessary scope. --- ## File: content/managed/on-aws.mdx --- title: Depot Managed on AWS ogTitle: Deploying Depot Managed on AWS --- With Depot Managed on Amazon Web Services (AWS), the Depot data plane is deployed within an isolated sub-account of your AWS organization. You can use the Depot CLI, web application, and API, but the underlying build compute and cache infrastructure reside entirely within your own AWS account. ## Architecture [](/images/self-hosted-architecture.png) ## Setup and Usage **NOTE:** This guide is intended for Depot customers who are working with the Depot team, you cannot deploy Depot Managed on AWS without it being enabled for your Depot organization. [Contact us](mailto:contact@depot.dev) if you are interested in using Depot Managed. ### Step 1: Create a dedicated sub-account Depot Managed requires the use of a dedicated sub-account within your AWS organization. This should be a new account containing no other resources or services. Follow the [AWS documentation](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_accounts_create.html#orgs_manage_accounts_create-new) to create a new account within your organization. ### Step 2: CloudFormation stack deployment Once you have created a new sub-account, you can deploy the following CloudFormation template to provision the required IAM permissions in the AWS sub-account. First, save the following as a file named `depot-managed-bootstrap.json`: ```json { "Resources": { "GrantProvisionerAccess": { "Type": "AWS::IAM::Role", "DeletionPolicy": "Retain", "Properties": { "RoleName": "DepotProvisioner", "ManagedPolicyArns": ["arn:aws:iam::aws:policy/AdministratorAccess"], "AssumeRolePolicyDocument": { "Statement": [ { "Action": ["sts:AssumeRole"], "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::375021575472:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_Provisioner_572dd0a52dd9fc8e" ] } } ] } } }, "GrantOpsAccess": { "Type": "AWS::IAM::Role", "DeletionPolicy": "Retain", "Properties": { "RoleName": "DepotOps", "ManagedPolicyArns": ["arn:aws:iam::aws:policy/AdministratorAccess"], "AssumeRolePolicyDocument": { "Statement": [ { "Action": ["sts:AssumeRole"], "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::375021575472:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_Ops_e45adfee11ab7421" ] } } ] } } } } } ``` Next, deploy the CloudFormation stack in the new sub-account: ```bash aws cloudformation create-stack \ --stack-name depot-managed-bootstrap \ --template-body file://depot-managed-bootstrap.json \ --capabilities CAPABILITY_NAMED_IAM ``` ### Step 3: Notify Depot Finally, let the Depot team know that the CloudFormation stack has been deployed, and they will initiate the deployment of the Depot data plane into the new sub-account. The Depot team will additionally work with you on any follow-up steps, including: - AWS quota increases to match your expected usage - Configuring KMS keys for encryption - Configuring S3 buckets for cache storage - Configuring VPC peering for private networking - Configuring AWS PrivateLink for secure API access - Enabling enforced usage of Depot Managed in your Depot organization ## Additional questions If you have any questions, please [contact us](mailto:contact@depot.dev), and we'll be happy to help. --- ## File: content/managed/overview.mdx --- title: Depot Managed Overview ogTitle: Overview of Depot Managed --- With Depot Managed, the Depot data plane can be deployed in your own Amazon Web Services (AWS) account. You can still use the Depot CLI, web application, and API, however the underlying build compute and cache data reside entirely within your own cloud account. _We are considering support for additional cloud providers like Google Cloud (GCP) in the future. If you are interested in this, please [let us know](mailto:help@depot.dev)._ ## How Depot Managed works Depot Managed is the entirety of the Depot [data plane](https://en.wikipedia.org/wiki/Forwarding_plane#Data_plane), deployed in a single-tenant isolated sub-account within your AWS organization. Once deployed, you have the option of using Depot Managed with some or all of your Depot organization's projects. You will continue to use the same Depot CLI and web application, but the CLI will communicate directly with the compute and cache infrastructure running in your AWS account. If you are an existing Depot user, moving to a Depot Managed deployment requires no changes to your existing developer workflows or CI pipelines. Depot Managed is still a fully managed service and comes with the full support and SLA of the Depot Business plan. The Depot team is on-call for any issues that arise with the Depot Managed deployment. For more information, see: - [Depot Managed on AWS](/docs/managed/on-aws) ## Benefits of Depot Managed Depot Managed comes with a few key benefits: - **Data residency**: All build data and cache data reside within your own cloud account, ensuring that you have full control over your data. - **Compliance**: For organizations that have strict compliance requirements. - **VPC peering & IAM**: You can configure VPC peering and IAM roles to allow the Depot data plane to access your private cloud resources. - **AWS PrivateLink**: You can use AWS PrivateLink to keep all communication between the Depot data plane and control plane within the AWS network. - **Cost control**: You can take advantage of any existing cloud discounts or credits you have, and you can control the size and type of instances used for builds. - **AWS Marketplace**: You can pay for Depot Managed through the AWS Marketplace and take advantage of any existing AWS billing arrangements you have. ## How to get started Depot Managed is available on the Depot Business plan. If you are interested in Depot Managed, please [contact us](mailto:contact@depot.dev) to chat with us and see if Depot Managed is a good fit for your organization. --- ## File: content/languages/node.mdx --- title: Best practice Dockerfiles for Node.js ogTitle: Best practice Dockerfiles for Node.js --- We've assembled some best practice Dockerfiles for building Docker images for Node.js using different package managers. These Dockerfiles are what we recommend when building Docker images for Node applications, but they are not the only way to do it, so your mileage may vary. ## Guides - [Dockerfile for Node.js using `pnpm`](/docs/languages/node-pnpm-dockerfile) --- ## File: content/languages/node-pnpm-dockerfile.mdx --- title: Best practice Dockerfile for Node.js with pnpm ogTitle: Best practice Dockerfile for Node.js with pnpm --- Below is an example `Dockerfile` that we use and recommend at Depot when we are building Docker images for Node applications that use `pnpm` as their package manager. ```dockerfile FROM node:20 AS base FROM base AS deps RUN corepack enable WORKDIR /app COPY package.json pnpm-lock.yaml ./ RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm fetch --frozen-lockfile RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm install --frozen-lockfile --prod FROM base AS build RUN corepack enable WORKDIR /app COPY package.json pnpm-lock.yaml ./ RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm fetch --frozen-lockfile RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm install --frozen-lockfile COPY . . RUN pnpm build FROM base WORKDIR /app COPY --from=deps /app/node_modules /app/node_modules COPY --from=build /app/dist /app/dist ENV NODE_ENV production CMD ["node", "./dist/index.js"] ``` ## Explanation of the Dockerfile There are several things in this example Dockerfile that are worth calling out. Most notably, we use a multi-stage build to separate the installation of dependencies from the actual build of the application. This allows us to take advantage of Docker's layer caching to speed up our builds. #### Stage 1: `FROM node:20 AS base` ```dockerfile FROM node:20 AS base ``` Here we use the `node:20` base image and set the stage name to be reused in stages that follow. If we had common dependencies that we wanted to be accessible in any stages using this `base` stage, we could install them here. #### Stage 2: `FROM base AS deps` ```dockerfile FROM base AS deps RUN corepack enable WORKDIR /app COPY package.json pnpm-lock.yaml ./ RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm fetch --frozen-lockfile RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm install --frozen-lockfile --prod ``` Next up is installing our dependencies via `pnpm`. First, we enable [`corepack`](https://nodejs.org/api/corepack.html) in the Node `base` image. Corepack allows us to use `pnpm` right out of the box without having to install it ourselves. It's a nice convenience but watch out that you have the expected `pnpm` version. We then create our working directory, `app`, and copy in just our `package.json` and `pnpm-lock.yaml` files. Note that we do not copy in our entire codebase. We only care about installing our production dependencies at this stage. This is a best practice that allows us to take advantage of Docker's layer caching. Finally, we get to installing our packages via `pnpm`. This is broken into two `RUN` statements that are making use of [BuildKit cache mounts](/blog/how-to-use-buildkit-cache-mounts-in-ci). 1. `pnpm fetch --frozen-lockfile` is a [pnpm feature](https://pnpm.io/cli/fetch) that is designed to improve building a Docker image. It fetches packages from the `pnpm-lock.yaml` file and stores them in the `pnpm` virtual store. Ignoring the `package.json` manifest. It's a nice optimization that avoids having to reinstall all packages in the `package.json` when a change occurs there that isn't related to the actual dependencies. 2. `pnpm install --frozen-lockfile --prod` is the actual installation of our dependencies. We use the `--frozen-lockfile` flag to ensure that we are installing the exact same versions of our dependencies that are in our `package.json`. We also use the `--prod` flag to ensure that we are not installing any `devDependencies`. This is a best practice that allows us to take advantage of Docker's layer caching. We use the `--frozen-lockfile` flag to ensure that we are installing the exact same versions of our dependencies that we have installed locally. We also use the `--prod` flag to ensure that we are only installing our production dependencies. This is a best practice that allows us to take advantage of Docker's layer caching. #### Stage 3: `FROM base AS build` ```dockerfile FROM base AS build RUN corepack enable WORKDIR /app COPY package.json pnpm-lock.yaml ./ RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm fetch --frozen-lockfile RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm install --frozen-lockfile COPY . . RUN pnpm build ``` It's time to build our application. We start by enabling `corepack` again so that we have `pnpm` in this stage. We then configure our working directory to be `app` and copy in the `package.json` and `pnpm-lock.yaml` files as we saw in the `deps` stage. We then run `pnpm fetch --frozen-lockfile` and `pnpm install --frozen-lockfile` again. However, we omit the `--prod` flag as we want to install _all dependencies_ as we may need some dev ones to build our final application. If that wasn't the case, we could copy in our dependencies from our earlier `deps` stage. Once our dependencies have been installed into this stage, we then copy in our source code via the `COPY` statement. We then run our build command, `pnpm build`. #### Stage 4: `FROM base` ```dockerfile FROM base WORKDIR /app COPY --from=deps /app/node_modules /app/node_modules COPY --from=build /app/dist /app/dist ENV NODE_ENV production CMD ["node", "./dist/index.js"] ``` Our final stage is copying all of the files from our earlier stages into our final image. We start by setting our working directory to `app`, then have several `COPY` statements: 1. We copy in our `node_modules` from our `deps` stage 2. We then copy in the `dist` directory from our `build` stage containing the outputs from our `pnpm build` Finally, we set our `NODE_ENV` to production for any dependencies that have an optimized production mode. We then set our `CMD` to run our application. --- ## File: content/languages/rust-dockerfile.mdx --- title: Best practice Dockerfile for Rust with cargo-chef and sccache ogTitle: Best practice Dockerfile for Rust with cargo-chef and sccache --- Below is an example `Dockerfile` that we have used and recommend at Depot for building images for Rust applications. ```dockerfile FROM rust:1.75 AS base RUN cargo install --locked cargo-chef sccache ENV RUSTC_WRAPPER=sccache SCCACHE_DIR=/sccache FROM base AS planner WORKDIR /app COPY . . RUN cargo chef prepare --recipe-path recipe.json FROM base as builder WORKDIR /app COPY --from=planner /app/recipe.json recipe.json RUN --mount=type=cache,target=/usr/local/cargo/registry \ --mount=type=cache,target=/usr/local/cargo/git \ --mount=type=cache,target=$SCCACHE_DIR,sharing=locked \ cargo chef cook --release --recipe-path recipe.json COPY . . RUN --mount=type=cache,target=/usr/local/cargo/registry \ --mount=type=cache,target=/usr/local/cargo/git \ --mount=type=cache,target=$SCCACHE_DIR,sharing=locked \ cargo build ``` In addition to the standard best practices when writing Dockerfiles, here we are also leveraging cargo-chef and sccache to speed up our Rust build. ## Using cargo-chef for dependency management When you install multiple crates with one command like `cargo build`, Docker treats any change in the output of `cargo build` as a change to the **entire** command. This means that Docker will attempt to execute that command again (re-downloading and installing **all **crates) every time you make an unrelated change to your source code or Dockerfile. There are various workarounds online to manually manage and copy individual packages into your container while building in order to avoid invalidating the cache on every build, but these are cumbersome and prone to bugs. Our preferred solution is to use [cargo-chef](https://github.com/LukeMathWalker/cargo-chef/blob/main/README.md), which allows you to separate building the dependencies and building the source code so that Docker sees them as different steps and can cache them separately. ## Using sccache for additional dependency management Even though cargo-chef separates your third-party dependencies from your source code, compiling and downloading your third-party dependencies is still considered one operation. This means that if a single dependency changes, there will be a cache miss and all of them will have to be re-downloaded and compiled, even though they haven't changed. If you have a more fine-grained cache, you only have to rebuild the changed dependencies. Enter [sccache](https://github.com/mozilla/sccache), which caches individual compilation artifacts so that they can be reused at a more granular level during future compilations. This allows you to recompile individual dependencies only when needed, rather than everything or nothing. ## Explanation of the Dockerfile At a high level, here are the things we're optimizing in our Docker build for a Rust application:: Multi-stage builds via multiple `FROM` statements - cargo-chef for dependency management - sccache for dependency caching - BuildKit cache mounts for finer-grained caching between builds ### Stage 1: `FROM rust:1.75 AS base` ```dockerfile FROM rust:1.75 AS base RUN cargo install --locked cargo-chef sccache ENV RUSTC_WRAPPER=sccache SCCACHE_DIR=/sccache ``` Here, we use `rust:1.75` as our base image and set the stage name to be used in later stages. In addition to this base image, we install sccache and cargo-chef. We then set the `SCCACHE_DIR` environment variable so that sccache stores compilation artifacts in the `/sccache` directory and the `RUSTC_WRAPPER` environment variable so that Cargo β€œwraps” the execution of the Rust compiler commands in an sccache call. That way, we can take advantage of sccache's cached dependencies when building the final image. ### Stage 2: `FROM base AS planner` ```dockerfile FROM base AS planner WORKDIR /app COPY . . RUN cargo chef prepare --recipe-path recipe.json ``` Next, we're building off the base stage by creating the recipe that will later be used to build our application with cargo-chef. We use two commands from cargo-chef that together act as a replacement for the standard `cargo build` command when building dependencies. #### Using `cargo-chef` to separate building dependencies from building source code `cargo chef prepare` looks at your `Cargo.toml` and auto-generated `Cargo.lock` files, determines all of your dependencies, and then creates a `recipe.json` file, which is a dependency tree of your project. By creating the dependency tree separately from the actual installation of dependencies, we can cache them independently so that all dependencies don't need to be rebuilt whenever the source code changes. ### Stage 3: `FROM base as builder` ```dockerfile FROM base as builder WORKDIR /app COPY --from=planner /app/recipe.json recipe.json RUN --mount=type=cache,target=/usr/local/cargo/registry \ --mount=type=cache,target=/usr/local/cargo/git \ --mount=type=cache,target=$SCCACHE_DIR,sharing=locked \ cargo chef cook --release --recipe-path recipe.json COPY . . RUN --mount=type=cache,target=/usr/local/cargo/registry \ --mount=type=cache,target=/usr/local/cargo/git \ --mount=type=cache,target=$SCCACHE_DIR,sharing=locked \ cargo build ``` `cargo chef cook` takes the` recipe.json` file and runs `cargo build` under the hood on each package independently. In this stage, we're copying the `recipe.json` file from the previous `planner` stage. If the recipe has not changed, then the step to build the dependencies with `cargo chef cook` can be skipped. After the dependencies have been built, we finally build the source code with the final `cargo build` command. #### Using `sccache` to retain dependencies between builds We use a [cache mount](https://depot.dev/blog/how-to-use-buildkit-cache-mounts-in-ci) to attach the sccache directory to the build. This type of cache mount gives you a more fine-grained level of caching that allows you to skip recomputing bits of work when the layer cache invalidates. Using sccache with a cache mount allows you to skip rebuilding compiled artifacts that have already been built even when certain layers in the build are invalidated. #### Caching the Cargo registry and git directories In this stage, we also use additional cache mounts to store the Cargo registry and git directories. The Cargo registry stores the crates that have already been downloaded, and the git directory stores any git dependencies. Usually only one or a few packages have actually changed and need to be re-downloaded and installed. All the other packages can be reused from previous builds with the cache mounts.