docker-gen Documentation

docker-gen

docker-gen is a file generator that renders templates using docker container meta-data.

It can be used to generate various kinds of files for:

• Centralized logging - fluentd, logstash or other centralized logging tools that tail the containers JSON log file or files within the container.
• Log Rotation - logrotate files to rotate container JSON log files
• Reverse Proxy Configs - nginx, haproxy, etc. reverse proxy configs to route requests from the host to containers
• Service Discovery - Scripts (python, bash, etc..) to register containers within etcd, hipache, etc..

Installation Copy Section

There are three common ways to run docker-gen:

• on the host
• bundled in a container with another application
• separate standalone containers

Host Install

Download the version you need, untar, and install to your PATH.

wget https://github.com/nginx-proxy/docker-gen/releases/download/0.16.0/docker-gen-linux-amd64-0.16.0.tar.gz
tar xvzf docker-gen-linux-amd64-0.16.0.tar.gz
./docker-gen

Bundled Container Install

Docker-gen can be bundled inside of a container along-side applications.

nginx-proxy/nginx-proxy trusted build is an example of
running docker-gen within a container along-side nginx.
jwilder/docker-register is an example of running
docker-gen within a container to do service registration with etcd.

Separate Container Install

It can also be run as two separate containers using the nginx-proxy/docker-gen
image, together with virtually any other image.

This is how you could run the official nginx image and
have docker-gen generate a reverse proxy config in the same way that nginx-proxy works. You may want to do
this to prevent having the docker socket bound to a publicly exposed container service.

Start nginx with a shared volume:

docker run -d -p 80:80 --name nginx -v /tmp/nginx:/etc/nginx/conf.d -t nginx

Fetch the template and start the docker-gen container with the shared volume:

mkdir -p /tmp/templates && cd /tmp/templates
curl -o nginx.tmpl https://raw.githubusercontent.com/nginx-proxy/docker-gen/main/templates/nginx.tmpl
docker run -d --name nginx-gen --volumes-from nginx \
   -v /var/run/docker.sock:/tmp/docker.sock:rw \
   -v /tmp/templates:/etc/docker-gen/templates \
   -t nginxproxy/docker-gen -notify-sighup nginx -watch -only-exposed /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf

Start a container, taking note of any Environment variables a container expects. See the top of a template for details.

docker run --env VIRTUAL_HOST='example.com' --env VIRTUAL_PORT=80 ...

Usage Copy Section

/* Detailed source-code truncated for AI context efficiency. */

If no <dest> file is specified, the output is sent to stdout. Mainly useful for debugging.

Configuration file Copy Section

Using the -config flag from above you can tell docker-gen to use the specified config file instead of command-line options. Multiple templates can be defined and they will be executed in the order that they appear in the config file.

An example configuration file, docker-gen.cfg can be found in the examples folder.

Configuration File Syntax

[[config]]
# Starts a configuration section

dest = "path/to/a/file"
# path to write the template. If not specfied, STDOUT is used

notifycmd = "/etc/init.d/foo reload"
# run command after template is regenerated (e.g restart xyz)

onlyexposed = true
# only include containers with exposed ports

template = "/path/to/a/template/file.tmpl"
# path to a template to generate

watch = true
# watch for container changes

wait = "500ms:2s"
# debounce changes with a min:max duration. Only applicable if watch = true


[config.NotifyContainers]
# Starts a notify container section

containername = 1
# container name followed by the signal to send

container_id = 1
# or the container id can be used followed by the signal to send

Putting it all together here is an example configuration file.

[[config]]
template = "/etc/nginx/nginx.conf.tmpl"
dest = "/etc/nginx/sites-available/default"
onlyexposed = true
notifycmd = "/etc/init.d/nginx reload"

[[config]]
template = "/etc/logrotate.conf.tmpl"
dest = "/etc/logrotate.d/docker"
watch = true

[[config]]
template = "/etc/docker-gen/templates/nginx.tmpl"
dest = "/etc/nginx/conf.d/default.conf"
watch = true
wait = "500ms:2s"

[config.NotifyContainers]
nginx = 1  # 1 is a signal number to be sent; here SIGHUP
e75a60548dc9 = 1  # a key can be either container name (nginx) or ID

Templating Copy Section

The templates used by docker-gen are written using the Go text/template language. In addition to the built-in functions supplied by Go, docker-gen uses sprig and some additional functions to make it simpler (or possible) to generate your desired output. Some templates rely on environment variables within the container to make decisions on what to generate from the template.

Several templates may be parsed at once by using a semicolon (;) to delimit the template value. This can be used as a proxy for Golang's nested template functionality. In all cases, the main rendered template should go first.

[[config]]
template = "/etc/docker-gen/templates/nginx.tmpl;/etc/docker-gen/templates/header.tmpl"
dest = "/etc/nginx/conf.d/default.conf"
watch = true
wait = "500ms:2s"

Emit Structure

Within the templates, the object emitted by docker-gen will be a structure consisting of following Go structs:

/* Detailed source-code truncated for AI context efficiency. */

The root also exposes .CurrentContainer, the RuntimeContainer of the docker-gen container itself (or nil if it cannot be determined). Like .Docker, it is resolved independently from the container list, so it remains available even when -only-exposed/-only-published would filter the docker-gen container out; depending on filters, it may also be present in the containers the templates iterate over.

For example, this is a JSON version of an emitted RuntimeContainer struct:

{
  "ID": "71e9768075836eb38557adcfc71a207386a0c597dbeda240cf905df79b18cebf",
  "Addresses": [
    {
      "IP": "172.17.0.4",
      "Port": "22",
      "Proto": "tcp",
      "HostIP": "192.168.10.24",
      "HostPort": "2222"
    }
  ],
  "Devices": [
    {
      "PathOnHost": "/dev/ttyACM0",
      "PathInContainer": "/dev/ttyUSB0",
      "Permissions": "rwm"
    }
  ],
  "Gateway": "172.17.42.1",
  "Node": {
    "ID": "I2VY:P7PF:TZD5:PGWB:QTI7:QDSP:C5UD:DYKR:XKKK:TRG2:M2BL:DFUN",
    "Name": "docker-test",
    "Address": {
      "IP": "192.168.10.24"
    }
  },
  "Labels": {
    "operatingsystem": "Ubuntu 14.04.2 LTS",
    "storagedriver": "devicemapper",
    "anything_foo": "something_bar"
  },
  "IP": "172.17.0.4",
  "Name": "docker_register",
  "Hostname": "71e976807583",
  "Image": {
    "Registry": "jwilder",
    "Repository": "docker-register"
  },
  "Env": {
    "ETCD_HOST": "172.17.42.1:4001",
    "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "DOCKER_HOST": "unix:///var/run/docker.sock",
    "HOST_IP": "172.17.42.1"
  },
  "Volumes": {
    "/mnt": {
      "Path": "/mnt",
      "HostPath": "/Users/joebob/tmp",
      "ReadWrite": true
    }
  }
}

Ordering of `Addresses` and `Networks`

The Addresses and Networks slices are emitted in a deterministic order, so generated output is stable across regenerations and does not trigger spurious reloads/restarts when nothing meaningful changed:

• Addresses are sorted by port (compared numerically), then by protocol, host port, host IP and IP.
• Networks are sorted alphabetically by Name.

As a consequence, index .Networks 0 returns the alphabetically-first network (for example bridge) rather than an arbitrary one. To select a specific network by name, filter with where:

{{ $net := index (where $value.Networks "Name" "my-network") 0 }}
server {{ $net.IP }}:{{ (index $value.Addresses 0).Port }};

Functions

• Functions from Go
• Functions from Sprig v3, except for those that have the same name as one of the following functions.
• closest $array $value: Returns the longest matching substring in $array that matches $value
• coalesce ...: Returns the first non-nil argument.
• comment $delimiter $string: Returns $string with each line prefixed by $delimiter (helpful for debugging combined with Sprig toPrettyJson: {{ toPrettyJson $ | comment "#" }}).
• contains $map $key: Returns true if $map contains $key. Takes maps from string to any type.
• dir $path: Returns an array of filenames in the specified $path.
• exists $path: Returns true if $path refers to an existing file or directory. Takes a string.
• eval $templateName [$data]: Evaluates the named template like Go's built-in template action, but instead of writing out the result it returns the result as a string so that it can be post-processed. The $data argument may be omitted, which is equivalent to passing nil.
• groupBy $containers $fieldPath: Groups an array of RuntimeContainer instances based on the values of a field path expression $fieldPath. A field path expression is a dot-delimited list of map keys or struct member names specifying the path from container to a nested value, which must be a string. Returns a map from the value of the field path expression to an array of containers having that value. Containers that do not have a value for the field path in question are omitted.
• groupByWithDefault $containers $fieldPath $defaultValue: Returns the same as groupBy, but containers that do not have a value for the field path are instead included in the map under the $defaultValue key.
• groupByKeys $containers $fieldPath: Returns the same as groupBy but only returns the keys of the map.
• groupByMulti $containers $fieldPath $sep: Like groupBy, but the string value specified by $fieldPath is first split by $sep into a list of strings. A container whose $fieldPath value contains a list of strings will show up in the map output under each of those strings.
• groupByLabel $containers $label: Returns the same as groupBy but grouping by the given label's value. Containers that do not have the $label set are omitted.
• groupByLabelWithDefault $containers $label $defaultValue: Returns the same as groupBy but grouping by the given label's value. Containers that do not have the $label set are included in the map under the $defaultValue key.
• include $file: Returns content of $file, and empty string if file reading error.
• intersect $slice1 $slice2: Returns the strings that exist in both string slices.
• mustBeOneOf $slice $value: Validates that $value is one of the allowed string values in $slice, returns $value on success and an error otherwise.
• mustBeInt $value: Validates that $value is a base-10 integer string, returns $value on success and an error otherwise.
• mustBeIntInRange $min $max $value: Validates that $value is a base-10 integer string in the inclusive range $min..$max, returns $value on success and an error otherwise.
• fromYaml $string / mustFromYaml $string: Similar to Sprig's fromJson / mustFromJson, but for YAML.
• toYaml $dict / mustToYaml $dict: Similar to Sprig's toJson / mustToJson, but for YAML.
• keys $map: Returns the keys from $map. If $map is nil, a nil is returned. If $map is not a map, an error will be thrown.
• sortStringsAsc $strings: Returns a slice of strings $strings sorted in ascending order.
• sortStringsDesc $strings: Returns a slice of strings $strings sorted in descending (reverse) order.
• sortObjectsByKeysAsc $objects $fieldPath: Returns the array $objects, sorted in ascending order based on the values of a field path expression $fieldPath.
• sortObjectsByKeysDesc $objects $fieldPath: Returns the array $objects, sorted in descending (reverse) order based on the values of a field path expression $fieldPath.
• when $condition $trueValue $falseValue: Returns the $trueValue when the $condition is true and the $falseValue otherwise
• where $items $fieldPath $value: Filters an array or slice based on the values of a field path expression $fieldPath. A field path expression is a dot-delimited list of map keys or struct member names specifying the path from container to a nested value. Returns an array of items having that value.
• whereNot $items $fieldPath $value: Filters an array or slice based on the values of a field path expression $fieldPath. A field path expression is a dot-delimited list of map keys or struct member names specifying the path from container to a nested value. Returns an array of items not having that value.
• whereExist $items $fieldPath: Like where, but returns only items where $fieldPath exists (is not nil).
• whereNotExist $items $fieldPath: Like where, but returns only items where $fieldPath does not exist (is nil).
• whereAny $items $fieldPath $sep $values: Like where, but the string value specified by $fieldPath is first split by $sep into a list of strings. The comparison value is a string slice with possible matches. Returns items which OR intersect these values.
• whereAll $items $fieldPath $sep $values: Like whereAny, except all $values must exist in the $fieldPath.
• whereLabelExists $containers $label: Filters a slice of containers based on the existence of the label $label.
• whereLabelDoesNotExist $containers $label: Filters a slice of containers based on the non-existence of the label $label.
• whereLabelValueMatches $containers $label $pattern: Filters a slice of containers based on the existence of the label $label with values matching the regular expression $pattern.

Nginx-specific functions:

• nginxQuote $string: Returns $string trimmed then enclosed in quotes if it contains any unescaped character that have special meaning in Nginx configuration files. Otherwise, returns $string trimmed. Errors if $string contains line breaks or both unescaped single and double quotes.
• nginxMustParseHSTS $string: Returns $string trimmed and validated as a Strict-Transport-Security header value. Errors if $string is not a valid Strict-Transport-Security header value.

Sprig functions that have the same name as docker-gen function (but different behaviour) are made available with the sprig prefix:

• sprigCoalesce ...: Alias for Sprig's coalesce.
• sprigContains $string $string: Alias for Sprig's contains.
• sprigDir $path: Alias for Sprig's dir.
• sprigReplace $old $new $string: Alias for Sprig's replace.
• sprigSplit $sep $string: Alias for Sprig's split.
• sprigSplitn $sep $count $string": Alias for Sprig's splitn.

Some functions are aliases for Go's strings package functions:

• parseBool $string: Alias for strconv.ParseBool. Returns the boolean value represented by $string. It accepts 1, t, T, TRUE, true, True, 0, f, F, FALSE, false, False. Any other value returns an error.
• replace $string $old $new $count: Alias for strings.Replace. Replaces up to $count occurences of $old with $new in $string.
• split $string $sep: Alias for strings.Split. Splits $string into a slice of substrings delimited by $sep.
• splitN $string $sep $count: Alias for strings.SplitN. Splits $string into a slice of substrings delimited by $sep, with number of substrings returned determined by $count.
• toLower $string: Alias for strings.ToLower. Replace capital letters in $string to lowercase.
• toUpper $string: Alias for strings.ToUpper. Replace lowercase letters in $string to uppercase.

Those have been aliased to Sprig functions with the same behaviour as the original docker-gen function:

• json $value: Alias for Sprig's mustToJson
• parseJson $string: Alias for Sprig's mustFromJson.
• sha1 $string: Alias for Sprig's sha1sum.

Examples Copy Section

• Automated Nginx Reverse Proxy for Docker
• Docker Log Management With Fluentd
• Docker Service Discovery Using Etcd and Haproxy

NGINX Reverse Proxy Config

nginxproxy/nginx-proxy trusted build.

Start nginx-proxy:

docker run -d -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock -t nginxproxy/nginx-proxy

Then start containers with a VIRTUAL_HOST (and the VIRTUAL_PORT if more than one port is exposed) env variable:

docker run -e VIRTUAL_HOST=foo.bar.com -e VIRTUAL_PORT=80 -t ...

If you wanted to run docker-gen directly on the host, you could do it with:

docker-gen -only-published -watch -notify "/etc/init.d/nginx reload" templates/nginx.tmpl /etc/nginx/sites-enabled/default

Fluentd Log Management

This template generate a fluentd.conf file used by fluentd. It would then ship log files off
the host.

docker-gen -watch -notify "restart fluentd" templates/fluentd.tmpl /etc/fluent/fluent.conf

Service Discovery in Etcd

This template is an example of generating a script that is then executed. This template generates
a python script that is then executed which register containers in Etcd using its HTTP API.

docker-gen -notify "/bin/bash /tmp/etcd.sh" -interval 10 templates/etcd.tmpl /tmp/etcd.sh

Development Copy Section

This project uses Go Modules for managing 3rd party dependencies.
This means that at least go 1.11 is required.

For go 1.11 and go 1.12 it is additionally required to manually enable support by setting GO111MODULE=on.
For later versions, this is not required.

git clone <your fork>
cd <your fork>
make get-deps
make

Powered by Copy Section

About this repository Copy Chapter

[!NOTE]
This repository is officially maintained by ZeroSSL as part of our commitment to secure and reliable SSL/TLS solutions.
We welcome contributions and feedback from the community!
For more information about our services, including free and paid SSL/TLS certificates, visit https://zerossl.com.