docker Documentation

Technical Documentation: aquasecurity/docker

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

Docker: the container engine [](https://github.com/docker/docker/releases/latest)

Docker is an open source project to pack, ship and run any application
as a lightweight container.

Docker containers are both hardware-agnostic and platform-agnostic.
This means they can run anywhere, from your laptop to the largest
cloud compute instance and everything in between - and they don't require
you to use a particular language, framework or packaging system. That
makes them great building blocks for deploying and scaling web apps,
databases, and backend services without depending on a particular stack
or provider.

Docker began as an open-source implementation of the deployment engine which
powered dotCloud,
a popular Platform-as-a-Service. It benefits directly from the experience
accumulated over several years of large-scale operation and support of hundreds
of thousands of applications and databases.

Security Disclosure

Security is very important to us. If you have any issue regarding security,
please disclose the information responsibly by sending an email to
[email protected] and not by creating a github issue.

Better than VMs

A common method for distributing applications and sandboxing their
execution is to use virtual machines, or VMs. Typical VM formats are
VMware's vmdk, Oracle VirtualBox's vdi, and Amazon EC2's ami. In theory
these formats should allow every developer to automatically package
their application into a "machine" for easy distribution and deployment.
In practice, that almost never happens, for a few reasons:

Size: VMs are very large which makes them impractical to store
and transfer.
Performance: running VMs consumes significant CPU and memory,
which makes them impractical in many scenarios, for example local
development of multi-tier applications, and large-scale deployment
of cpu and memory-intensive applications on large numbers of
machines.
Portability: competing VM environments don't play well with each
other. Although conversion tools do exist, they are limited and
add even more overhead.
Hardware-centric: VMs were designed with machine operators in
mind, not software developers. As a result, they offer very
limited tooling for what developers need most: building, testing
and running their software. For example, VMs offer no facilities
for application versioning, monitoring, configuration, logging or
service discovery.

By contrast, Docker relies on a different sandboxing method known as
containerization. Unlike traditional virtualization, containerization
takes place at the kernel level. Most modern operating system kernels
now support the primitives necessary for containerization, including
Linux with openvz,
vserver and more recently
lxc, Solaris with
zones,
and FreeBSD with
Jails.

Docker builds on top of these low-level primitives to offer developers a
portable format and runtime environment that solves all four problems.
Docker containers are small (and their transfer can be optimized with
layers), they have basically zero memory and cpu overhead, they are
completely portable, and are designed from the ground up with an
application-centric design.

Perhaps best of all, because Docker operates at the OS level, it can still be
run inside a VM!

Plays well with others

Docker does not require you to buy into a particular programming
language, framework, packaging system, or configuration language.

Is your application a Unix process? Does it use files, tcp connections,
environment variables, standard Unix streams and command-line arguments
as inputs and outputs? Then Docker can run it.

Can your application's build be expressed as a sequence of such
commands? Then Docker can build it.

Escape dependency hell

A common problem for developers is the difficulty of managing all
their application's dependencies in a simple and automated way.

This is usually difficult for several reasons:

Cross-platform dependencies. Modern applications often depend on
a combination of system libraries and binaries, language-specific
packages, framework-specific modules, internal components
developed for another project, etc. These dependencies live in
different "worlds" and require different tools - these tools
typically don't work well with each other, requiring awkward
custom integrations.
Conflicting dependencies. Different applications may depend on
different versions of the same dependency. Packaging tools handle
these situations with various degrees of ease - but they all
handle them in different and incompatible ways, which again forces
the developer to do extra work.
Custom dependencies. A developer may need to prepare a custom
version of their application's dependency. Some packaging systems
can handle custom versions of a dependency, others can't - and all
of them handle it differently.

Docker solves the problem of dependency hell by giving the developer a simple
way to express all their application's dependencies in one place, while
streamlining the process of assembling them. If this makes you think of
XKCD 927, don't worry. Docker doesn't
replace your favorite packaging systems. It simply orchestrates
their use in a simple and repeatable way. How does it do that? With
layers.

Docker defines a build as running a sequence of Unix commands, one
after the other, in the same container. Build commands modify the
contents of the container (usually by installing new files on the
filesystem), the next command modifies it some more, etc. Since each
build command inherits the result of the previous commands, the
order in which the commands are executed expresses dependencies.

Here's a typical Docker build process:

bash

FROM ubuntu:12.04
RUN apt-get update && apt-get install -y python python-pip curl
RUN curl -sSL https://github.com/shykes/helloflask/archive/master.tar.gz | tar -xzv
RUN cd helloflask-master && pip install -r requirements.txt

Note that Docker doesn't care how dependencies are built - as long
as they can be built by running a Unix command in a container.

Getting started

Docker can be installed either on your computer for building applications or
on servers for running them. To get started, check out the installation
instructions in the
documentation.

Usage examples

Docker can be used to run short-lived commands, long-running daemons
(app servers, databases, etc.), interactive shell sessions, etc.

You can find a list of real-world
examples in the
documentation.

Under the hood

Under the hood, Docker is built on the following components:

The
cgroups
and
namespaces
capabilities of the Linux kernel
The Go programming language
The Docker Image Specification
The Libcontainer Specification

Contributing to Docker [](https://godoc.org/github.com/docker/docker)

Master (Linux)	Experimental (linux)	Windows	FreeBSD
		/badge/icon)	/badge/icon)

Want to hack on Docker? Awesome! We have instructions to help you get
started contributing code or documentation.

These instructions are probably not perfect, please let us know if anything
feels wrong or incomplete. Better yet, submit a PR and improve them yourself.

Getting the development builds

Want to run Docker from a master build? You can download
master builds at master.dockerproject.org.
They are updated with each commit merged into the master branch.

Don't know how to use that super cool new feature in the master build? Check
out the master docs at
docs.master.dockerproject.org.

How the project is run

Docker is a very, very active project. If you want to learn more about how it is run,
or want to get more involved, the best place to start is the project directory.

We are always open to suggestions on process improvements, and are always looking for more maintainers.

Talking to other Docker users and contributors

| Internet Relay Chat (IRC) | IRC is a direct line to our most knowledgeable Docker users; we have
both the #docker and #docker-dev group on
irc.freenode.net.
IRC is a rich chat protocol but it can overwhelm new users. You can search
our chat archives.

text

Read our [IRC quickstart guide](https://docs.docker.com/project/get-help/#irc-quickstart) for an easy way to get started. |

| --- | --- |
| Docker Community Forums | The Docker Engine
group is for users of the Docker Engine project. |
| Google Groups | The docker-dev group is for contributors and other people
contributing to the Docker project. You can join this group without a
Google account by sending an email to [email protected].
You'll receive a join-request message; simply reply to the message to
confirm your subscribtion. |
| Twitter | You can follow Docker's Twitter feed
to get updates on our products. You can also tweet us questions or just
share blogs or stories. |
| Stack Overflow | Stack Overflow has over 7000 Docker questions listed. We regularly
monitor Docker questions
and so do many other knowledgeable Docker users. |

Legal

Brought to you courtesy of our legal counsel. For more context,
please see the NOTICE document in this repo.

Use and transfer of Docker may be subject to certain restrictions by the
United States and other governments.

It is your responsibility to ensure that your use and/or transfer does not
violate applicable laws.

For more information, please see https://www.bis.doc.gov

Licensing

Docker is licensed under the Apache License, Version 2.0. See
LICENSE for the full
license text.

There are a number of projects under development that are based on Docker's
core technology. These projects expand the tooling built around the
Docker platform to broaden its application and utility.

Docker Registry: Registry
server for Docker (hosting/delivery of repositories and images)
Docker Machine: Machine management
for a container-centric world
Docker Swarm: A Docker-native clustering
system
Docker Compose (formerly Fig):
Define and run multi-container apps
Kitematic: The easiest way to use
Docker on Mac and Windows

If you know of another project underway that should be listed here, please help
us keep this list up-to-date by submitting a PR.

Awesome-Docker

You can find more projects, tools and articles related to Docker on the awesome-docker list. Add your project there.

Overview (`api/README.md`)

This directory contains code pertaining to the Docker API:

Used by the docker client when communicating with the docker daemon
Used by third party tools wishing to interface with the docker daemon

Overview (`contrib/README.md`)

The contrib directory contains scripts, images, and other helpful things
which are not part of the core docker distribution. Please note that they
could be out of date, since they do not receive the same attention as the
rest of the repository.

Overview (`docs/README.md`)

Docker Documentation

The source for Docker documentation is in this directory. Our
documentation uses extended Markdown, as implemented by
MkDocs. The current release of the Docker documentation
resides on https://docs.docker.com.

Understanding the documentation branches and processes

Docker has two primary branches for documentation:

Branch	Description	URL (published via commit-hook)
`docs`	Official release documentation	https://docs.docker.com
`master`	Merged but unreleased development work

Additions and updates to upcoming releases are made in a feature branch off of
the master branch. The Docker maintainers also support a docs branch that
contains the last release of documentation.

After a release, documentation updates are continually merged into master as
they occur. This work includes new documentation for forthcoming features, bug
fixes, and other updates.

Periodically, the Docker maintainers update docs.docker.com between official
releases of Docker. They do this by cherry-picking commits from master,
merging them into docs, and then publishing the result.

In the rare case where a change is not forward-compatible, changes may be made
on other branches by special arrangement with the Docker maintainers.

Quickstart for documentation contributors

If you are a new or beginner contributor, we encourage you to read through the
our detailed contributors
guide. The guide explains in
detail, with examples, how to contribute. If you are an experienced contributor
this quickstart should be enough to get you started.

The following is the essential workflow for contributing to the documentation:

Fork the docker/docker repository.
Clone the repository to your local machine.
Select an issue from docker/docker to work on or submit a proposal of your
own.
Create a feature branch from master in which to work.

By basing from master your work is automatically included in the next
release. It also allows docs maintainers to easily cherry-pick your changes
into the docs release branch.
Modify existing or add new .md files to the docs directory.
As you work, build the documentation site locally to see your changes.

The docker/docker repository contains a Dockerfile and a Makefile.
Together, these create a development environment in which you can build and
run a container running the Docker documentation website. To build the
documentation site, enter make docs in the docs directory of your docker/docker fork:
text
```
$ make docs
.... (lots of output) ....
docker run --rm -it  -e AWS_S3_BUCKET -p 8000:8000 "docker-docs:master" mkdocs serve
Running at: http://0.0.0.0:8000/
Live reload enabled.
Hold ctrl+c to quit.
```
The build creates an image containing all the required tools, adds the local
docs/ directory and generates the HTML files. Then, it runs a Docker
container with this image.

The container exposes port 8000 on the localhost so that you can connect and
see your changes. If you use Docker Machine, the docker-machine ip <machine-name> command gives you the address of your server.
Check your writing for style and mechanical errors.

Use our documentation style
guide to check style. There are
several good grammar and spelling online
checkers that can check your writing
mechanics.
Squash your commits on your branch.
Make a pull request from your fork back to Docker's master branch.
Work with the reviewers until your change is approved and merged.

Debugging and testing

If you have any issues you need to debug, you can use make docs-shell and then
run mkdocs serve. You can use make docs-test to generate a report of missing
links that are referenced in the documentation—there should be none.

Style guide

If you have questions about how to write for Docker's documentation, please see
the style guide. The style guide provides
guidance about grammar, syntax, formatting, styling, language, or tone. If
something isn't clear in the guide, please submit an issue to let us know or
submit a pull request to help us improve it.

Publishing documentation (for Docker maintainers)

To publish Docker's documentation you need to have Docker up and running on your
machine. You'll also need a docs/awsconfig file containing the settings you
need to access the AWS bucket you'll be deploying to.

The process for publishing is to build first to an AWS bucket, verify the build,
and then publish the final release.

Have Docker installed and running on your machine.
Ask the core maintainers for the awsconfig file.
Copy the awsconfig file to the docs/ directory.

The awsconfig file contains the profiles of the S3 buckets for our
documentation sites. (If needed, the release script creates an S3 bucket and
pushes the files to it.) Each profile has this format:
text
```
[profile dowideit-docs]
 aws_access_key_id = IHOIUAHSIDH234rwf....
 aws_secret_access_key = OIUYSADJHLKUHQWIUHE......
 region = ap-southeast-2
```
The profile name must be the same as the name of the bucket you are
deploying to.
Call the make from the docker directory.
text
```
$ make AWS_S3_BUCKET=dowideit-docs docs-release
```
This publishes only to the http://bucket-url/v1.2/ version of the
documentation.
If you're publishing the current release's documentation, you need to also
update the root docs pages by running
text
```
$ make AWS_S3_BUCKET=dowideit-docs BUILD_ROOT=yes docs-release
```

Errors publishing using a Docker Machine VM

Sometimes, in a Windows or Mac environment, the publishing procedure returns this
error:

text

Post http:///var/run/docker.sock/build?rm=1&t=docker-docs%3Apost-1.2.0-docs_update-2:
dial unix /var/run/docker.sock: no such file or directory.

If this happens, set the Docker host. Run the following command to get the
variables in your shell:

text

docker-machine env <machine-name>

Then, set your environment accordingly.

Cherry-picking documentation changes to update an existing release.

Whenever the core team makes a release, they publish the documentation based on
the release branch. At that time, the release branch is copied into the
docs branch. The documentation team makes updates between Docker releases by
cherry-picking changes from master into any of the documentation branches.
Typically, we cherry-pick into the docs branch.

For example, to update the current release's docs, do the following:

Go to your docker/docker fork and get the latest from master.
text
```
$ git fetch upstream
```
Checkout a new branch based on upstream/docs.

You should give your new branch a descriptive name.
text
```
$ git checkout -b post-1.2.0-docs-update-1 upstream/docs
```
In a browser window, open [https://github.com/docker/docker/commits/master].
Locate the merges you want to publish.

You should only cherry-pick individual commits; do not cherry-pick merge
commits. To minimize merge conflicts, start with the oldest commit and work
your way forward in time.
Copy the commit SHA from GitHub.
Cherry-pick the commit.
text
```
$ git cherry-pick -x fe845c4
```
Repeat until you have cherry-picked everything you want to merge.

Push your changes to your fork.

text

$ git push origin post-1.2.0-docs-update-1

Make a pull request to merge into the docs branch.

Do NOT merge into master.
Have maintainers review your pull request.
Once the PR has the needed "LGTMs", merge it on GitHub.
Return to your local fork and make sure you are still on the docs branch.
text
```
$ git checkout docs
```
Fetch your merged pull request from docs.
text
```
$ git fetch upstream/docs
```
Ensure your branch is clean and set to the latest.
text
```
$ git reset --hard upstream/docs
```
Copy the awsconfig file into the docs directory.

Make the beta documentation

text

$ make AWS_S3_BUCKET=beta-docs.docker.io BUILD_ROOT=yes docs-release

Open the beta
website site
and make sure what you published is correct.
When you're happy with your content, publish the docs to our live site:
text
```
$ make AWS_S3_BUCKET=docs.docker.com BUILD_ROOT=yes
```

DISTRIBUTION_ID=C2K6......FL2F docs-release

Test the uncached version of the live docs at [http://docs.docker.com.s3-website-us-east-1.amazonaws.com/]

Caching and the docs

New docs do not appear live on the site until the cache (a complex, distributed
CDN system) is flushed. The make docs-release command flushes the cache if
the DISTRIBUTION_ID is set to the Cloudfront distribution ID. The cache flush
can take at least 15 minutes to run and you can check its progress with the CDN
Cloudfront Purge Tool Chrome app.

Removing files from the docs.docker.com site

Sometimes it becomes necessary to remove files from the historical published documentation.
The most reliable way to do this is to do it directly using aws s3 commands running in a
docs container:

Start the docs container like make docs-shell, but bind mount in your awsconfig:

text

docker run --rm -it -v $(CURDIR)/docs/awsconfig:/docs/awsconfig docker-docs:master bash

and then the following example shows deleting 2 documents from s3, and then requesting the
CloudFlare cache to invalidate them:

text

export BUCKET BUCKET=docs.docker.com
export AWS_CONFIG_FILE=$(pwd)/awsconfig
aws s3 --profile $BUCKET ls s3://$BUCKET
aws s3 --profile $BUCKET rm s3://$BUCKET/v1.0/reference/api/docker_io_oauth_api/index.html
aws s3 --profile $BUCKET rm s3://$BUCKET/v1.1/reference/api/docker_io_oauth_api/index.html

aws configure set preview.cloudfront true
export DISTRIBUTION_ID=YUTIYUTIUTIUYTIUT
aws cloudfront  create-invalidation --profile docs.docker.com --distribution-id $DISTRIBUTION_ID --invalidation-batch '{"Paths":{"Quantity":1, "Items":["/v1.0/reference/api/docker_io_oauth_api/"]},"CallerReference":"6Mar2015sventest1"}'
aws cloudfront  create-invalidation --profile docs.docker.com --distribution-id $DISTRIBUTION_ID --invalidation-batch '{"Paths":{"Quantity":1, "Items":["/v1.1/reference/api/docker_io_oauth_api/"]},"CallerReference":"6Mar2015sventest1"}'

Generate the man pages

For information on generating man pages (short for manual page), see the README.md
document in the man page directory
in this project.

Overview (`experimental/README.md`)

Docker Experimental Features

This page contains a list of features in the Docker engine which are
experimental. Experimental features are not ready for production. They are
provided for test and evaluation in your sandbox environments.

The information below describes each feature and the GitHub pull requests and
issues associated with it. If necessary, links are provided to additional
documentation on an issue. As an active Docker user and community member,
please feel free to provide any feedback on these features you wish.

Install Docker experimental

Unlike the regular Docker binary, the experimental channels is built and
updated nightly on https://experimental.docker.com. From one day to the
next, new features may appear, while existing experimental features may be
refined or entirely removed.

Verify that you have curl installed.
text
```
$ which curl
```
If curl isn't installed, install it after updating your manager:
text
```
$ sudo apt-get update
 $ sudo apt-get install curl
```
Get the latest Docker package.
text
```
$ curl -sSL https://experimental.docker.com/ | sh
```
The system prompts you for your sudo password. Then, it downloads and
installs Docker and its dependencies.
Note: If your company is behind a filtering proxy, you may find that the
apt-key
command fails for the Docker repo during installation. To work around this,
add the key directly using the following:
text
```
$ curl -sSL https://experimental.docker.com/gpg | sudo apt-key add -
```
Verify docker is installed correctly.
text
```
$ sudo docker run hello-world
```
This command downloads a test image and runs it in a container.

Get the Linux binary

To download the latest experimental docker binary for Linux,
use the following URLs:

text

https://experimental.docker.com/builds/Linux/i386/docker-latest.tgz

https://experimental.docker.com/builds/Linux/x86_64/docker-latest.tgz

After downloading the appropriate binary, you can follow the instructions
here to run the docker daemon.

Note

You can get the MD5 and SHA256 hashes by appending .md5 and .sha256 to the URLs respectively

You can get the compressed binaries by appending .tgz to the URLs

Build an experimental binary

You can also build the experimental binary from the standard development environment by adding
DOCKER_EXPERIMENTAL=1 to the environment where you run make to build Docker binaries. For example,
to build a Docker binary with the experimental features enabled:

text

$ DOCKER_EXPERIMENTAL=1 make binary

Current experimental features

How to comment on an experimental feature

Each feature's documentation includes a list of proposal pull requests or PRs associated with the feature. If you want to comment on or suggest a change to a feature, please add it to the existing feature PR.

Issues or problems with a feature? Inquire for help on the #docker IRC channel or in on the Docker Google group.

Overview (`man/README.md`)

Docker Documentation

This directory contains the Docker user manual in the Markdown format.
Do not edit the man pages in the man1 directory. Instead, amend the
Markdown (*.md) files.

Generating man pages from the Markdown files

The recommended approach for generating the man pages is via a Docker
container using the supplied Dockerfile to create an image with the correct
environment. This uses go-md2man, a pure Go Markdown to man page generator.

Building the md2man image

There is a Dockerfile provided in the /man directory of your
'docker/docker' fork.

Using this Dockerfile, create a Docker image tagged docker/md2man:

text

docker build -t docker/md2man .

Utilizing the image

From within the /man directory run the following command:

text

docker run -v $(pwd):/man -w /man -i docker/md2man ./md2man-all.sh

The md2man Docker container will process the Markdown files and generate
the man pages inside the /man/man1 directory of your fork using
Docker volumes. For more information on Docker volumes see the man page for
docker run and also look at the article [Sharing Directories via Volumes]
(https://docs.docker.com/use/working_with_volumes/).

Overview (`pkg/README.md`)

pkg/ is a collection of utility packages used by the Docker project without being specific to its internals.

Utility packages are kept separate from the docker core codebase to keep it as small and concise as possible.
If some utilities grow larger and their APIs stabilize, they may be moved to their own repository under the
Docker organization, to facilitate re-use by other projects. However that is not the priority.

The directory pkg is named after the same directory in the camlistore project. Since Brad is a core
Go maintainer, we thought it made sense to copy his methods for organizing Go code :) Thanks Brad!

Because utility packages are small and neatly separated from the rest of the codebase, they are a good
place to start for aspiring maintainers and contributors. Get in touch if you want to help maintain them!

Overview (`project/README.md`)

Hacking on Docker

The project/ directory holds information and tools for everyone involved in the process of creating and
distributing Docker, specifically:

Guides

If you're a contributor or aspiring contributor, you should read CONTRIBUTORS.md.

If you're a maintainer or aspiring maintainer, you should read MAINTAINERS.

If you're a packager or aspiring packager, you should read PACKAGERS.md.

If you're a maintainer in charge of a release, you should read RELEASE-CHECKLIST.md.

Roadmap

A high-level roadmap is available at ROADMAP.md.

Build tools

hack/make.sh is the primary build tool for docker. It is used for compiling the official binary,
running the test suite, and pushing releases.

Overview (`cmd/dockerd/README.md`)

docker.go contains Docker's main function.

This file provides first line CLI argument parsing and environment variable setting.

Overview (`contrib/desktop-integration/README.md`)

Desktop Integration

The ./contrib/desktop-integration contains examples of typical dockerized
desktop applications.

Examples

Chromium: ./chromium/Dockerfile shows a way to dockerize a common application
Gparted: ./gparted/Dockerfile shows a way to dockerize a common application w devices

Overview (`contrib/docker-device-tool/README.md`)

Docker device tool for devicemapper storage driver backend

The ./contrib/docker-device-tool contains a tool to manipulate devicemapper thin-pool.

Compile

text

$ make shell
## inside build container
$ go build contrib/docker-device-tool/device_tool.go

# if devicemapper version is old and compliation fails, compile with `libdm_no_deferred_remove` tag
$ go build -tags libdm_no_deferred_remove contrib/docker-device-tool/device_tool.go

Overview (`contrib/vagrant-docker/README.md`)

Vagrant integration

Currently there are at least 4 different projects that we are aware of that deals
with integration with Vagrant at different levels. One
approach is to use Docker as a provisioner
which means you can create containers and pull base images on VMs using Docker's
CLI and the other is to use Docker as a provider,
meaning you can use Vagrant to control Docker containers.

Provisioners

Providers

Setting up Vagrant-docker with the Remote API

The initial Docker upstart script will not work because it runs on 127.0.0.1, which is not accessible to the host machine. Instead, we need to change the script to connect to 0.0.0.0. To do this, modify /etc/init/docker.conf to look like this:

text

description     "Docker daemon"

start on filesystem
stop on runlevel [!2345]

respawn

script
    /usr/bin/docker daemon -H=tcp://0.0.0.0:2375
end script

Once that's done, you need to set up a SSH tunnel between your host machine and the vagrant machine that's running Docker. This can be done by running the following command in a host terminal:

text

ssh -L 2375:localhost:2375 -p 2222 vagrant@localhost

(The first 2375 is what your host can connect to, the second 2375 is what port Docker is running on in the vagrant machine, and the 2222 is the port Vagrant is providing for SSH. If VirtualBox is the VM you're using, you can see what value "2222" should be by going to: Network > Adapter 1 > Advanced > Port Forwarding in the VirtualBox GUI.)

Note that because the port has been changed, to run docker commands from within the command line you must run them like this:

text

sudo docker -H 0.0.0.0:2375 < commands for docker >

Overview (`docs/static_files/README.md`)

Static files dir

Files you put in /static_files/ will be copied to the web visible /_static/

Be careful not to override pre-existing static files from the template.

Generally, layout related files should go in the /theme directory.

If you want to add images to your particular documentation page. Just put them next to
your .rst source file and reference them relatively.

Overview (`hack/Jenkins/readme.md`)

These files under this directory are for reference only.

They are used by Jenkins for CI runs.

Overview (`hack/make/README.md`)

This directory holds scripts called by make.sh in the parent directory.

Each script is named after the bundle it creates.
They should not be called directly - instead, pass it as argument to make.sh, for example:

text

./hack/make.sh test
./hack/make.sh binary ubuntu

# Or to run all bundles:
./hack/make.sh

To add a bundle:

Create a shell-compatible file here
Add it to $DEFAULT_BUNDLES in make.sh

Overview (`pkg/archive/README.md`)

This code provides helper functions for dealing with archive files.

Overview (`pkg/discovery/README.md`)

page_title: Docker discovery page_description: discovery page_keywords: docker, clustering, discovery

Discovery

Docker comes with multiple Discovery backends.

Backends

Using etcd

Point your Docker Engine instances to a common etcd instance. You can specify
the address Docker uses to advertise the node using the --cluster-advertise
flag.

bash

$ docker daemon -H=<node_ip:2376> --cluster-advertise=<node_ip:2376> --cluster-store etcd://<etcd_ip1>,<etcd_ip2>/<path>

Using consul

Point your Docker Engine instances to a common Consul instance. You can specify
the address Docker uses to advertise the node using the --cluster-advertise
flag.

bash

$ docker daemon -H=<node_ip:2376> --cluster-advertise=<node_ip:2376> --cluster-store consul://<consul_ip>/<path>

Using zookeeper

Point your Docker Engine instances to a common Zookeeper instance. You can specify
the address Docker uses to advertise the node using the --cluster-advertise
flag.

bash

$ docker daemon -H=<node_ip:2376> --cluster-advertise=<node_ip:2376> --cluster-store zk://<zk_addr1>,<zk_addr2>/<path>

Overview (`pkg/locker/README.md`)

Locker

locker provides a mechanism for creating finer-grained locking to help
free up more global locks to handle other tasks.

The implementation looks close to a sync.Mutex, however the user must provide a
reference to use to refer to the underlying lock when locking and unlocking,
and unlock may generate an error.

If a lock with a given name does not exist when Lock is called, one is
created.
Lock references are automatically cleaned up on Unlock if nothing else is
waiting for the lock.

Usage

package important

import (
	"sync"
	"time"

	"github.com/docker/docker/pkg/locker"
)

type important struct {
	locks *locker.Locker
	data  map[string]interface{}
	mu    sync.Mutex
}

func (i *important) Get(name string) interface{} {
	i.locks.Lock(name)
	defer i.locks.Unlock(name)
	return data[name]
}

func (i *important) Create(name string, data interface{}) {
	i.locks.Lock(name)
	defer i.locks.Unlock(name)

	i.createImportant(data)

	s.mu.Lock()
	i.data[name] = data
	s.mu.Unlock()
}

func (i *important) createImportant(data interface{}) {
	time.Sleep(10 * time.Second)
}

For functions dealing with a given name, always lock at the beginning of the
function (or before doing anything with the underlying state), this ensures any
other function that is dealing with the same name will block.

When needing to modify the underlying data, use the global lock to ensure nothing
else is modfying it at the same time.
Since name lock is already in place, no reads will occur while the modification
is being performed.

Overview (`pkg/mflag/README.md`)

Package mflag (aka multiple-flag) implements command-line flag parsing.
It's a hacky fork of the official golang package

It adds:

both short and long flag version
./example -s red ./example --string blue
multiple names for the same option

text

$>./example -h
Usage of example:
  -s, --string="": a simple string

It is very flexible on purpose, so you can do things like:

text

$>./example -h
Usage of example:
  -s, -string, --string="": a simple string

Or:

text

$>./example -h
Usage of example:
  -oldflag, --newflag="": a simple string

You can also hide some flags from the usage, so if we want only --newflag:

text

$>./example -h
Usage of example:
  --newflag="": a simple string
$>./example -oldflag str
str

See example.go for more details.

Overview (`pkg/reexec/README.md`)

reexec

The reexec package facilitates the busybox style reexec of the docker binary that we require because
of the forking limitations of using Go. Handlers can be registered with a name and the argv 0 of
the exec of the binary will be used to find and execute custom init paths.

Overview (`pkg/signal/README.md`)

This package provides helper functions for dealing with signals across various operating systems

Overview (`pkg/stringid/README.md`)

This package provides helper functions for dealing with string identifiers

Overview (`pkg/stringutils/README.md`)

This package provides helper functions for dealing with strings

Overview (`pkg/symlink/README.md`)

Package symlink implements EvalSymlinksInScope which is an extension of filepath.EvalSymlinks,
as well as a Windows long-path aware version of filepath.EvalSymlinks
from the Go standard library.

The code from filepath.EvalSymlinks has been adapted in fs.go.
Please read the LICENSE.BSD file that governs fs.go and LICENSE.APACHE for fs_test.go.

Overview (`pkg/sysinfo/README.md`)

SysInfo stores information about which features a kernel supports.

Overview (`pkg/useragent/README.md`)

This package provides helper functions to pack version information into a single User-Agent header.

Overview (`contrib/syntax/nano/README.md`)

Dockerfile.nanorc

Dockerfile syntax highlighting for nano

Single User Installation

Create a nano syntax directory in your home directory:

mkdir -p ~/.nano/syntax

Copy Dockerfile.nanorc to ~/.nano/syntax/

cp Dockerfile.nanorc ~/.nano/syntax/

Add the following to your ~/.nanorc to tell nano where to find the Dockerfile.nanorc file

text

## Dockerfile files
include "~/.nano/syntax/Dockerfile.nanorc"

System Wide Installation

Create a nano syntax directory:

mkdir /usr/local/share/nano

Copy Dockerfile.nanorc to /usr/local/share/nano

cp Dockerfile.nanorc /usr/local/share/nano/

Add the following to your /etc/nanorc:

text

## Dockerfile files
include "/usr/local/share/nano/Dockerfile.nanorc"

Overview (`contrib/syntax/textmate/README.md`)

Docker.tmbundle

Dockerfile syntax highlighting for TextMate and Sublime Text.

Install

Sublime Text

Available for Sublime Text under package control.
Search for Dockerfile Syntax Highlighting

TextMate 2

You can install this bundle in TextMate by opening the preferences and going to the bundles tab. After installation it will be automatically updated for you.

enjoy.

Overview (`contrib/syntax/vim/README.md`)

dockerfile.vim

Syntax highlighting for Dockerfiles

Installation

With pathogen, the usual way...

With Vundle

text

Plugin 'docker/docker' , {'rtp': '/contrib/syntax/vim/'}

Features

The syntax highlighting includes:

The directives (e.g. FROM)
Strings
Comments

License

BSD, short and sweet

Overview (`daemon/graphdriver/devmapper/README.md`)

devicemapper - a storage backend based on Device Mapper

Theory of operation

The device mapper graphdriver uses the device mapper thin provisioning
module (dm-thinp) to implement CoW snapshots. The preferred model is
to have a thin pool reserved outside of Docker and passed to the
daemon via the --storage-opt dm.thinpooldev option.

As a fallback if no thin pool is provided, loopback files will be
created. Loopback is very slow, but can be used without any
pre-configuration of storage. It is strongly recommended that you do
not use loopback in production. Ensure your Docker daemon has a
--storage-opt dm.thinpooldev argument provided.

In loopback, a thin pool is created at /var/lib/docker/devicemapper
(devicemapper graph location) based on two block devices, one for
data and one for metadata. By default these block devices are created
automatically by using loopback mounts of automatically created sparse
files.

The default loopback files used are
/var/lib/docker/devicemapper/devicemapper/data and
/var/lib/docker/devicemapper/devicemapper/metadata. Additional metadata
required to map from docker entities to the corresponding devicemapper
volumes is stored in the /var/lib/docker/devicemapper/devicemapper/json
file (encoded as Json).

In order to support multiple devicemapper graphs on a system, the thin
pool will be named something like: docker-0:33-19478248-pool, where
the 0:33 part is the minor/major device nr and 19478248 is the
inode number of the /var/lib/docker/devicemapper directory.

On the thin pool, docker automatically creates a base thin device,
called something like docker-0:33-19478248-base of a fixed
size. This is automatically formatted with an empty filesystem on
creation. This device is the base of all docker images and
containers. All base images are snapshots of this device and those
images are then in turn used as snapshots for other images and
eventually containers.

Information on `docker info`

As of docker-1.4.1, docker info when using the devicemapper storage driver
will display something like:

text

$ sudo docker info
[...]
Storage Driver: devicemapper
 Pool Name: docker-253:1-17538953-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Data file: /dev/loop4
 Metadata file: /dev/loop4
 Data Space Used: 2.536 GB
 Data Space Total: 107.4 GB
 Data Space Available: 104.8 GB
 Metadata Space Used: 7.93 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.14 GB
 Udev Sync Supported: true
 Data loop file: /home/docker/devicemapper/devicemapper/data
 Metadata loop file: /home/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.82-git (2013-10-04)
[...]

status items

Each item in the indented section under Storage Driver: devicemapper are
status information about the driver.

Pool Name name of the devicemapper pool for this driver.
Pool Blocksize tells the blocksize the thin pool was initialized with. This only changes on creation.
Base Device Size tells the maximum size of a container and image
Data file blockdevice file used for the devicemapper data
Metadata file blockdevice file used for the devicemapper metadata
Data Space Used tells how much of Data file is currently used
Data Space Total tells max size the Data file
Data Space Available tells how much free space there is in the Data file. If you are using a loop device this will report the actual space available to the loop device on the underlying filesystem.
Metadata Space Used tells how much of Metadata file is currently used
Metadata Space Total tells max size the Metadata file
Metadata Space Available tells how much free space there is in the Metadata file. If you are using a loop device this will report the actual space available to the loop device on the underlying filesystem.
Udev Sync Supported tells whether devicemapper is able to sync with Udev. Should be true.
Data loop file file attached to Data file, if loopback device is used
Metadata loop file file attached to Metadata file, if loopback device is used
Library Version from the libdevmapper used

About the devicemapper options

The devicemapper backend supports some options that you can specify
when starting the docker daemon using the --storage-opt flags.
This uses the dm prefix and would be used something like docker daemon --storage-opt dm.foo=bar.

These options are currently documented both in the man
page and in the online
documentation.
If you add an options, update both the man page and the documentation.

Overview (`docs/reference/api/README.md`)

This directory holds the authoritative specifications of APIs defined and implemented by Docker. Currently this includes:

The remote API by which a docker node can be queried over HTTP
The registry API by which a docker node can download and upload
images for storage and sharing
The index search API by which a docker node can search the public
index for images to download
The docker.io OAuth and accounts API which 3rd party services can
use to access account information

Overview (`docs/security/https/README.md`)

This is an initial attempt to make it easier to test the examples in the https.md
doc.

At this point, it has to be a manual thing, and I've been running it in boot2docker.

My process is as following:

text

$ boot2docker ssh
$$ git clone https://github.com/docker/docker
$$ cd docker/docs/articles/https
$$ make cert

lots of things to see and manually answer, as openssl wants to be interactive

NOTE: make sure you enter the hostname (boot2docker in my case) when prompted for Computer Name)

text

$$ sudo make run

Start another terminal:

text

$ boot2docker ssh
$$ cd docker/docs/articles/https
$$ make client

The last will connect first with --tls and then with --tlsverify, both should succeed.

Overview (`pkg/plugins/pluginrpc-gen/README.md`)

Plugin RPC Generator

Generates go code from a Go interface definition for proxying between the plugin
API and the subsystem being extended.

Usage

Given an interface definition:

type volumeDriver interface {
	Create(name string, opts opts) (err error)
	Remove(name string) (err error)
	Path(name string) (mountpoint string, err error)
	Mount(name string) (mountpoint string, err error)
	Unmount(name string) (err error)
}

Note: All function options and return values must be named in the definition.

Run the generator:

bash

$ pluginrpc-gen --type volumeDriver --name VolumeDriver -i volumes/drivers/extpoint.go -o volumes/drivers/proxy.go

Where:

--type is the name of the interface to use
--name is the subsystem that the plugin "Implements"
-i is the input file containing the interface definition
-o is the output file where the the generated code should go

Note: The generated code will use the same package name as the one defined in the input file

Optionally, you can skip functions on the interface that should not be
implemented in the generated proxy code by passing in the function name to --skip.
This flag can be specified multiple times.

You can also add build tags that should be prepended to the generated code by
supplying --tag. This flag can be specified multiple times.

Known issues

go-generate

You can also use this with go-generate, which is pretty awesome.
To do so, place the code at the top of the file which contains the interface
definition (i.e., the input file):

//go:generate pluginrpc-gen -i $GOFILE -o proxy.go -type volumeDriver -name VolumeDriver

Then cd to the package dir and run go generate

Note: the pluginrpc-gen binary must be within your $PATH

Overview (`contrib/builder/deb/amd64/README.md`)

`dockercore/builder-deb`

This image's tags contain the dependencies for building Docker .debs for each of the Debian-based platforms Docker targets.

To add new tags, see contrib/builder/deb/amd64 in https://github.com/docker/docker, specifically the generate.sh script, whose usage is described in a comment at the top of the file.

Overview (`contrib/builder/rpm/amd64/README.md`)

`dockercore/builder-rpm`

This image's tags contain the dependencies for building Docker .rpms for each of the RPM-based platforms Docker targets.

To add new tags, see contrib/builder/rpm/amd64 in https://github.com/docker/docker, specifically the generate.sh script, whose usage is described in a comment at the top of the file.

Overview (`vendor/src/google.golang.org/cloud/README.md`)

Google Cloud for Go

NOTE: These packages are experimental, and may occasionally make
backwards-incompatible changes.

NOTE: Github repo is a mirror of https://code.googlesource.com/gocloud.

Go packages for Google Cloud Platform services. Supported APIs include:

Google Cloud Datastore
Google Cloud Storage
Google Cloud Pub/Sub
Google Cloud Container Engine

import "google.golang.org/cloud"

Documentation and examples are available at
https://godoc.org/google.golang.org/cloud.

Authorization

Authorization, throughout the package, is delegated to the godoc.org/golang.org/x/oauth2.
Refer to the godoc documentation
for examples on using oauth2 with the Cloud package.

Google Cloud Datastore

Google Cloud Datastore (docs) is a fully
managed, schemaless database for storing non-relational data. Cloud Datastore
automatically scales with your users and supports ACID transactions, high availability
of reads and writes, strong consistency for reads and ancestor queries, and eventual
consistency for all other queries.

Follow the activation instructions to use the Google
Cloud Datastore API with your project.

https://godoc.org/google.golang.org/cloud/datastore

type Post struct {
	Title       string
	Body        string `datastore:",noindex"`
	PublishedAt time.Time
}
keys := []*datastore.Key{
	datastore.NewKey(ctx, "Post", "post1", 0, nil),
	datastore.NewKey(ctx, "Post", "post2", 0, nil),
}
posts := []*Post{
	{Title: "Post 1", Body: "...", PublishedAt: time.Now()},
	{Title: "Post 2", Body: "...", PublishedAt: time.Now()},
}
if _, err := datastore.PutMulti(ctx, keys, posts); err != nil {
	log.Println(err)
}

Google Cloud Storage

Google Cloud Storage (docs) allows you to store
data on Google infrastructure with very high reliability, performance and availability,
and can be used to distribute large data objects to users via direct download.

https://godoc.org/google.golang.org/cloud/storage

// Read the object1 from bucket.
rc, err := storage.NewReader(ctx, "bucket", "object1")
if err != nil {
	log.Fatal(err)
}
slurp, err := ioutil.ReadAll(rc)
rc.Close()
if err != nil {
	log.Fatal(err)
}

Google Cloud Pub/Sub (Alpha)

Google Cloud Pub/Sub is in Alpha status. As a result, it might change in
backward-incompatible ways and is not recommended for production use. It is not
subject to any SLA or deprecation policy.

Google Cloud Pub/Sub (docs) allows you to connect
your services with reliable, many-to-many, asynchronous messaging hosted on Google's
infrastructure. Cloud Pub/Sub automatically scales as you need it and provides a foundation
for building your own robust, global services.

https://godoc.org/google.golang.org/cloud/pubsub

// Publish "hello world" on topic1.
msgIDs, err := pubsub.Publish(ctx, "topic1", &pubsub.Message{
	Data: []byte("hello world"),
})
if err != nil {
	log.Println(err)
}
// Pull messages via subscription1.
msgs, err := pubsub.Pull(ctx, "subscription1", 1)
if err != nil {
	log.Println(err)
}

Contributing

Contributions are welcome. Please, see the
CONTRIBUTING
document for details. We're using Gerrit for our code reviews. Please don't open pull
requests against this repo, new pull requests will be automatically closed.

Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms.
See Contributor Code of Conduct
for more information.

Overview (`vendor/src/google.golang.org/grpc/README.md`)

#gRPC-Go

The Go implementation of gRPC: A high performance, open source, general RPC framework that puts mobile and HTTP/2 first. For more information see the gRPC Quick Start guide.

Installation

To install this package, you need to install Go and setup your Go workspace on your computer. The simplest way to install the library is to run:

text

$ go get google.golang.org/grpc

Prerequisites

This requires Go 1.5 or later .

Constraints

The grpc package should only depend on standard Go packages and a small number of exceptions. If your contribution introduces new dependencies which are NOT in the list, you need a discussion with gRPC-Go authors and consultants.

Documentation

See API documentation for package and API descriptions and find examples in the examples directory.

Status

Beta release

Overview (`vendor/src/gopkg.in/fsnotify.v1/README.md`)

File system notifications for Go

Go 1.3+ required.

Cross platform: Windows, Linux, BSD and OS X.

Adapter	OS	Status
inotify	Linux 2.6.27 or later, Android*	Supported
kqueue	BSD, OS X, iOS*	Supported
ReadDirectoryChangesW	Windows	Supported
FSEvents	OS X	Planned
FEN	Solaris 11	In Progress
fanotify	Linux 2.6.37+
USN Journals	Windows	Maybe
Polling	All	Maybe

* Android and iOS are untested.

Please see the documentation for usage. Consult the Wiki for the FAQ and further information.

API stability

fsnotify is a fork of howeyc/fsnotify with a new API as of v1.0. The API is based on this design document.

All releases are tagged based on Semantic Versioning. Further API changes are planned, and will be tagged with a new major revision number.

Go 1.6 supports dependencies located in the vendor/ folder. Unless you are creating a library, it is recommended that you copy fsnotify into vendor/github.com/fsnotify/fsnotify within your project.

Contributing

Please refer to CONTRIBUTING before opening an issue or pull request.

Example

See example_test.go.

Overview (`vendor/src/github.com/Azure/go-ansiterm/README.md`)

go-ansiterm

This is a cross platform Ansi Terminal Emulation library. It reads a stream of Ansi characters and produces the appropriate function calls. The results of the function calls are platform dependent.

For example the parser might receive "ESC, [, A" as a stream of three characters. This is the code for Cursor Up (http://www.vt100.net/docs/vt510-rm/CUU). The parser then calls the cursor up function (CUU()) on an event handler. The event handler determines what platform specific work must be done to cause the cursor to move up one position.

The parser (parser.go) is a partial implementation of this state machine (http://vt100.net/emu/vt500_parser.png). There are also two event handler implementations, one for tests (test_event_handler.go) to validate that the expected events are being produced and called, the other is a Windows implementation (winterm/win_event_handler.go).

See parser_test.go for examples exercising the state machine and generating appropriate function calls.

Overview (`vendor/src/github.com/BurntSushi/toml/README.md`)

TOML parser and encoder for Go with reflection

TOML stands for Tom's Obvious, Minimal Language. This Go package provides a
reflection interface similar to Go's standard library json and xml
packages. This package also supports the encoding.TextUnmarshaler and
encoding.TextMarshaler interfaces so that you can define custom data
representations. (There is an example of this below.)

Spec: https://github.com/mojombo/toml

Compatible with TOML version
v0.2.0

Documentation: http://godoc.org/github.com/BurntSushi/toml

Installation:

bash

go get github.com/BurntSushi/toml

Try the toml validator:

bash

go get github.com/BurntSushi/toml/cmd/tomlv
tomlv some-toml-file.toml

Testing

This package passes all tests in
toml-test for both the decoder
and the encoder.

Examples

This package works similarly to how the Go standard library handles XML
and JSON. Namely, data is loaded into Go values via reflection.

For the simplest example, consider some TOML file as just a list of keys
and values:

toml

Age = 25
Cats = [ "Cauchy", "Plato" ]
Pi = 3.14
Perfection = [ 6, 28, 496, 8128 ]
DOB = 1987-07-05T05:45:00Z

Which could be defined in Go as:

type Config struct {
  Age int
  Cats []string
  Pi float64
  Perfection []int
  DOB time.Time // requires `import time`
}

And then decoded with:

var conf Config
if _, err := toml.Decode(tomlData, &conf); err != nil {
  // handle error
}

You can also use struct tags if your struct field name doesn't map to a TOML
key value directly:

toml

some_key_NAME = "wat"

type TOML struct {
  ObscureKey string `toml:"some_key_NAME"`
}

Using the `encoding.TextUnmarshaler` interface

Here's an example that automatically parses duration strings into
time.Duration values:

toml

[[song]]
name = "Thunder Road"
duration = "4m49s"

[[song]]
name = "Stairway to Heaven"
duration = "8m03s"

Which can be decoded with:

type song struct {
  Name     string
  Duration duration
}
type songs struct {
  Song []song
}
var favorites songs
if _, err := toml.Decode(blob, &favorites); err != nil {
  log.Fatal(err)
}

for _, s := range favorites.Song {
  fmt.Printf("%s (%s)\n", s.Name, s.Duration)
}

And you'll also need a duration type that satisfies the
encoding.TextUnmarshaler interface:

type duration struct {
	time.Duration
}

func (d *duration) UnmarshalText(text []byte) error {
	var err error
	d.Duration, err = time.ParseDuration(string(text))
	return err
}

More complex usage

Here's an example of how to load the example from the official spec page:

toml

# This is a TOML document. Boom.

title = "TOML Example"

[owner]
name = "Tom Preston-Werner"
organization = "GitHub"
bio = "GitHub Cofounder & CEO\nLikes tater tots and beer."
dob = 1979-05-27T07:32:00Z # First class dates? Why not?

[database]
server = "192.168.1.1"
ports = [ 8001, 8001, 8002 ]
connection_max = 5000
enabled = true

[servers]

  # You can indent as you please. Tabs or spaces. TOML don't care.
  [servers.alpha]
  ip = "10.0.0.1"
  dc = "eqdc10"

  [servers.beta]
  ip = "10.0.0.2"
  dc = "eqdc10"

[clients]
data = [ ["gamma", "delta"], [1, 2] ] # just an update to make sure parsers support it

# Line breaks are OK when inside arrays
hosts = [
  "alpha",
  "omega"
]

And the corresponding Go types are:

type tomlConfig struct {
	Title string
	Owner ownerInfo
	DB database `toml:"database"`
	Servers map[string]server
	Clients clients
}

type ownerInfo struct {
	Name string
	Org string `toml:"organization"`
	Bio string
	DOB time.Time
}

type database struct {
	Server string
	Ports []int
	ConnMax int `toml:"connection_max"`
	Enabled bool
}

type server struct {
	IP string
	DC string
}

type clients struct {
	Data [][]interface{}
	Hosts []string
}

Note that a case insensitive match will be tried if an exact match can't be
found.

A working example of the above can be found in _examples/example.{go,toml}.

Overview (`vendor/src/github.com/Microsoft/go-winio/README.md`)

go-winio

This repository contains utilities for efficiently performing Win32 IO operations in
Go. Currently, this is focused on accessing named pipes and other file handles, and
for using named pipes as a net transport.

This code relies on IO completion ports to avoid blocking IO on system threads, allowing Go
to reuse the thread to schedule another goroutine. This limits support to Windows Vista and
newer operating systems. This is similar to the implementation of network sockets in Go's net
package.

Please see the LICENSE file for licensing information.

Thanks to natefinch for the inspiration for this library. See https://github.com/natefinch/npipe
for another named pipe implementation.

Overview (`vendor/src/github.com/RackSec/srslog/README.md`)

srslog

Go has a syslog package in the standard library, but it has the following
shortcomings:

It doesn't have TLS support
According to bradfitz on the Go team, it is no longer being maintained.

I agree that it doesn't need to be in the standard library. So, I've
followed Brad's suggestion and have made a separate project to handle syslog.

This code was taken directly from the Go project as a base to start from.

However, this does have TLS support.

Usage

Basic usage retains the same interface as the original syslog package. We
only added to the interface where required to support new functionality.

Switch from the standard library:

text

import(
    //"log/syslog"
    syslog "github.com/RackSec/srslog"
)

You can still use it for local syslog:

text

w, err := syslog.Dial("", "", syslog.LOG_ERR, "testtag")

Or to unencrypted UDP:

text

w, err := syslog.Dial("udp", "192.168.0.50:514", syslog.LOG_ERR, "testtag")

Or to unencrypted TCP:

text

w, err := syslog.Dial("tcp", "192.168.0.51:514", syslog.LOG_ERR, "testtag")

But now you can also send messages via TLS-encrypted TCP:

text

w, err := syslog.DialWithTLSCertPath("tcp+tls", "192.168.0.52:514", syslog.LOG_ERR, "testtag", "/path/to/servercert.pem")

And if you need more control over your TLS configuration :

text

pool := x509.NewCertPool()
serverCert, err := ioutil.ReadFile("/path/to/servercert.pem")
if err != nil {
    return nil, err
}
pool.AppendCertsFromPEM(serverCert)
config := tls.Config{
    RootCAs: pool,
}

w, err := DialWithTLSConfig(network, raddr, priority, tag, &config)

(Note that in both TLS cases, this uses a self-signed certificate, where the
remote syslog server has the keypair and the client has only the public key.)

And then to write log messages, continue like so:

text

if err != nil {
    log.Fatal("failed to connect to syslog:", err)
}
defer w.Close()

w.Alert("this is an alert")
w.Crit("this is critical")
w.Err("this is an error")
w.Warning("this is a warning")
w.Notice("this is a notice")
w.Info("this is info")
w.Debug("this is debug")
w.Write([]byte("these are some bytes"))

Generating TLS Certificates

We've provided a script that you can use to generate a self-signed keypair:

text

pip install cryptography
python script/gen-certs.py

That outputs the public key and private key to standard out. Put those into
.pem files. (And don't put them into any source control. The certificate in
the test directory is used by the unit tests, and please do not actually use
it anywhere else.)

Running Tests

Run the tests as usual:

text

go test

But we've also provided a test coverage script that will show you which
lines of code are not covered:

text

script/coverage --html

That will open a new browser tab showing coverage information.

License

This project uses the New BSD License, the same as the Go project itself.

Code of Conduct

Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms.

Overview (`vendor/src/github.com/Sirupsen/logrus/README.md`)

Logrus [](https://travis-ci.org/Sirupsen/logrus) [](https://godoc.org/github.com/Sirupsen/logrus)

Logrus is a structured logger for Go (golang), completely API compatible with
the standard library logger. [Godoc][godoc]. Please note the Logrus API is not
yet stable (pre 1.0). Logrus itself is completely stable and has been used in
many large deployments. The core API is unlikely to change much but please
version control your Logrus to make sure you aren't fetching latest master on
every build.

Nicely color-coded in development (when a TTY is attached, otherwise just
plain text):

With log.SetFormatter(&log.JSONFormatter{}), for easy parsing by logstash
or Splunk:

json

{"animal":"walrus","level":"info","msg":"A group of walrus emerges from the
ocean","size":10,"time":"2014-03-10 19:57:38.562264131 -0400 EDT"}

{"level":"warning","msg":"The group's number increased tremendously!",
"number":122,"omg":true,"time":"2014-03-10 19:57:38.562471297 -0400 EDT"}

{"animal":"walrus","level":"info","msg":"A giant walrus appears!",
"size":10,"time":"2014-03-10 19:57:38.562500591 -0400 EDT"}

{"animal":"walrus","level":"info","msg":"Tremendously sized cow enters the ocean.",
"size":9,"time":"2014-03-10 19:57:38.562527896 -0400 EDT"}

{"level":"fatal","msg":"The ice breaks!","number":100,"omg":true,
"time":"2014-03-10 19:57:38.562543128 -0400 EDT"}

With the default log.SetFormatter(&log.TextFormatter{}) when a TTY is not
attached, the output is compatible with the
logfmt format:

text

time="2015-03-26T01:27:38-04:00" level=debug msg="Started observing beach" animal=walrus number=8
time="2015-03-26T01:27:38-04:00" level=info msg="A group of walrus emerges from the ocean" animal=walrus size=10
time="2015-03-26T01:27:38-04:00" level=warning msg="The group's number increased tremendously!" number=122 omg=true
time="2015-03-26T01:27:38-04:00" level=debug msg="Temperature changes" temperature=-4
time="2015-03-26T01:27:38-04:00" level=panic msg="It's over 9000!" animal=orca size=9009
time="2015-03-26T01:27:38-04:00" level=fatal msg="The ice breaks!" err=&{0x2082280c0 map[animal:orca size:9009] 2015-03-26 01:27:38.441574009 -0400 EDT panic It's over 9000!} number=100 omg=true
exit status 1

Example

The simplest way to use Logrus is simply the package-level exported logger:

package main

import (
  log "github.com/Sirupsen/logrus"
)

func main() {
  log.WithFields(log.Fields{
    "animal": "walrus",
  }).Info("A walrus appears")
}

Note that it's completely api-compatible with the stdlib logger, so you can
replace your log imports everywhere with log "github.com/Sirupsen/logrus"
and you'll now have the flexibility of Logrus. You can customize it all you
want:

package main

import (
  "os"
  log "github.com/Sirupsen/logrus"
)

func init() {
  // Log as JSON instead of the default ASCII formatter.
  log.SetFormatter(&log.JSONFormatter{})

  // Output to stderr instead of stdout, could also be a file.
  log.SetOutput(os.Stderr)

  // Only log the warning severity or above.
  log.SetLevel(log.WarnLevel)
}

func main() {
  log.WithFields(log.Fields{
    "animal": "walrus",
    "size":   10,
  }).Info("A group of walrus emerges from the ocean")

  log.WithFields(log.Fields{
    "omg":    true,
    "number": 122,
  }).Warn("The group's number increased tremendously!")

  log.WithFields(log.Fields{
    "omg":    true,
    "number": 100,
  }).Fatal("The ice breaks!")

  // A common pattern is to re-use fields between logging statements by re-using
  // the logrus.Entry returned from WithFields()
  contextLogger := log.WithFields(log.Fields{
    "common": "this is a common field",
    "other": "I also should be logged always",
  })

  contextLogger.Info("I'll be logged with common and other field")
  contextLogger.Info("Me too")
}

For more advanced usage such as logging to multiple locations from the same
application, you can also create an instance of the logrus Logger:

package main

import (
  "github.com/Sirupsen/logrus"
)

// Create a new instance of the logger. You can have any number of instances.
var log = logrus.New()

func main() {
  // The API for setting attributes is a little different than the package level
  // exported logger. See Godoc.
  log.Out = os.Stderr

  log.WithFields(logrus.Fields{
    "animal": "walrus",
    "size":   10,
  }).Info("A group of walrus emerges from the ocean")
}

Fields

Logrus encourages careful, structured logging though logging fields instead of
long, unparseable error messages. For example, instead of: log.Fatalf("Failed to send event %s to topic %s with key %d"), you should log the much more
discoverable:

log.WithFields(log.Fields{
  "event": event,
  "topic": topic,
  "key": key,
}).Fatal("Failed to send event")

We've found this API forces you to think about logging in a way that produces
much more useful logging messages. We've been in countless situations where just
a single added field to a log statement that was already there would've saved us
hours. The WithFields call is optional.

In general, with Logrus using any of the printf-family functions should be
seen as a hint you should add a field, however, you can still use the
printf-family functions with Logrus.

Hooks

You can add hooks for logging levels. For example to send errors to an exception
tracking service on Error, Fatal and Panic, info to StatsD or log to
multiple places simultaneously, e.g. syslog.

Logrus comes with built-in hooks. Add those, or your custom hook, in
init:

import (
  log "github.com/Sirupsen/logrus"
  "gopkg.in/gemnasium/logrus-airbrake-hook.v2" // the package is named "aibrake"
  logrus_syslog "github.com/Sirupsen/logrus/hooks/syslog"
  "log/syslog"
)

func init() {

  // Use the Airbrake hook to report errors that have Error severity or above to
  // an exception tracker. You can create custom hooks, see the Hooks section.
  log.AddHook(airbrake.NewHook(123, "xyz", "production"))

  hook, err := logrus_syslog.NewSyslogHook("udp", "localhost:514", syslog.LOG_INFO, "")
  if err != nil {
    log.Error("Unable to connect to local syslog daemon")
  } else {
    log.AddHook(hook)
  }
}

Note: Syslog hook also support connecting to local syslog (Ex. "/dev/log" or "/var/run/syslog" or "/var/run/log"). For the detail, please check the syslog hook README.

Hook	Description
Airbrake	Send errors to the Airbrake API V3. Uses the official `gobrake` behind the scenes.
Airbrake "legacy"	Send errors to an exception tracking service compatible with the Airbrake API V2. Uses `airbrake-go` behind the scenes.
Papertrail	Send errors to the Papertrail hosted logging service via UDP.
Syslog	Send errors to remote syslog server. Uses standard library `log/syslog` behind the scenes.
Bugsnag	Send errors to the Bugsnag exception tracking service.
Sentry	Send errors to the Sentry error logging and aggregation service.
Hiprus	Send errors to a channel in hipchat.
Logrusly	Send logs to Loggly
Slackrus	Hook for Slack chat.
Journalhook	Hook for logging to `systemd-journald`
Graylog	Hook for logging to Graylog
Raygun	Hook for logging to Raygun.io
LFShook	Hook for logging to the local filesystem
Honeybadger	Hook for sending exceptions to Honeybadger
Mail	Hook for sending exceptions via mail
Rollrus	Hook for sending errors to rollbar
Fluentd	Hook for logging to fluentd
Mongodb	Hook for logging to mongodb
InfluxDB	Hook for logging to influxdb
Octokit	Hook for logging to github via octokit
DeferPanic	Hook for logging to DeferPanic
Redis-Hook	Hook for logging to a ELK stack (through Redis)
Amqp-Hook	Hook for logging to Amqp broker (Like RabbitMQ)
KafkaLogrus	Hook for logging to kafka
Typetalk	Hook for logging to Typetalk
ElasticSearch	Hook for logging to ElasticSearch

Level logging

Logrus has six logging levels: Debug, Info, Warning, Error, Fatal and Panic.

log.Debug("Useful debugging information.")
log.Info("Something noteworthy happened!")
log.Warn("You should probably take a look at this.")
log.Error("Something failed but I'm not quitting.")
// Calls os.Exit(1) after logging
log.Fatal("Bye.")
// Calls panic() after logging
log.Panic("I'm bailing.")

You can set the logging level on a Logger, then it will only log entries with
that severity or anything above it:

// Will log anything that is info or above (warn, error, fatal, panic). Default.
log.SetLevel(log.InfoLevel)

It may be useful to set log.Level = logrus.DebugLevel in a debug or verbose
environment if your application has that.

Entries

Besides the fields added with WithField or WithFields some fields are
automatically added to all logging events:

time. The timestamp when the entry was created.
msg. The logging message passed to {Info,Warn,Error,Fatal,Panic} after
the AddFields call. E.g. Failed to send event.
level. The logging level. E.g. info.

Environments

Logrus has no notion of environment.

If you wish for hooks and formatters to only be used in specific environments,
you should handle that yourself. For example, if your application has a global
variable Environment, which is a string representation of the environment you
could do:

import (
  log "github.com/Sirupsen/logrus"
)

init() {
  // do something here to set environment depending on an environment variable
  // or command-line flag
  if Environment == "production" {
    log.SetFormatter(&log.JSONFormatter{})
  } else {
    // The TextFormatter is default, you don't actually have to do this.
    log.SetFormatter(&log.TextFormatter{})
  }
}

This configuration is how logrus was intended to be used, but JSON in
production is mostly only useful if you do log aggregation with tools like
Splunk or Logstash.

Formatters

The built-in logging formatters are:

logrus.TextFormatter. Logs the event in colors if stdout is a tty, otherwise
without colors.
- Note: to force colored output when there is no TTY, set the ForceColors
  field to true. To force no colored output even if there is a TTY set the
  DisableColors field to true
logrus.JSONFormatter. Logs fields as JSON.
logrus/formatters/logstash.LogstashFormatter. Logs fields as Logstash Events.
go
```
logrus.SetFormatter(&logstash.LogstashFormatter{Type: "application_name"})
```

Third party logging formatters:

prefixed. Displays log entry source along with alternative layout.
zalgo. Invoking the P͉̫o̳̼̊w̖͈̰͎e̬͔̭͂r͚̼̹̲ ̫͓͉̳͈ō̠͕͖̚f̝͍̠ ͕̲̞͖͑Z̖̫̤̫ͪa͉̬͈̗l͖͎g̳̥o̰̥̅!̣͔̲̻͊̄ ̙̘̦̹̦.

You can define your formatter by implementing the Formatter interface,
requiring a Format method. Format takes an *Entry. entry.Data is a
Fields type (map[string]interface{}) with all your fields as well as the
default ones (see Entries section above):

type MyJSONFormatter struct {
}

log.SetFormatter(new(MyJSONFormatter))

func (f *MyJSONFormatter) Format(entry *Entry) ([]byte, error) {
  // Note this doesn't include Time, Level and Message which are available on
  // the Entry. Consult `godoc` on information about those fields or read the
  // source of the official loggers.
  serialized, err := json.Marshal(entry.Data)
    if err != nil {
      return nil, fmt.Errorf("Failed to marshal fields to JSON, %v", err)
    }
  return append(serialized, '\n'), nil
}

Logger as an `io.Writer`

Logrus can be transformed into an io.Writer. That writer is the end of an io.Pipe and it is your responsibility to close it.

w := logger.Writer()
defer w.Close()

srv := http.Server{
    // create a stdlib log.Logger that writes to
    // logrus.Logger.
    ErrorLog: log.New(w, "", 0),
}

Each line written to that writer will be printed the usual way, using formatters
and hooks. The level for those entries is info.

Rotation

Log rotation is not provided with Logrus. Log rotation should be done by an
external program (like logrotate(8)) that can compress and delete old log
entries. It should not be a feature of the application-level logger.

Tools

Tool	Description
Logrus Mate	Logrus mate is a tool for Logrus to manage loggers, you can initial logger's level, hook and formatter by config file, the logger will generated with different config at different environment.

Testing

Logrus has a built in facility for asserting the presence of log messages. This is implemented through the test hook and provides:

decorators for existing logger (test.NewLocal and test.NewGlobal) which basically just add the test hook
a test logger (test.NewNullLogger) that just records log messages (and does not output any):

logger, hook := NewNullLogger()
logger.Error("Hello error")

assert.Equal(1, len(hook.Entries))
assert.Equal(logrus.ErrorLevel, hook.LastEntry().Level)
assert.Equal("Hello error", hook.LastEntry().Message)

hook.Reset()
assert.Nil(hook.LastEntry())

Overview (`vendor/src/github.com/armon/go-metrics/README.md`)

go-metrics

This library provides a metrics package which can be used to instrument code,
expose application metrics, and profile runtime performance in a flexible manner.

Sinks

The metrics package makes use of a MetricSink interface to support delivery
to any type of backend. Currently the following sinks are provided:

StatsiteSink : Sinks to a statsite instance (TCP)
StatsdSink: Sinks to a statsd / statsite instance (UDP)
InmemSink : Provides in-memory aggregation, can be used to export stats
FanoutSink : Sinks to multiple sinks. Enables writing to multiple statsite instances for example.
BlackholeSink : Sinks to nowhere

In addition to the sinks, the InmemSignal can be used to catch a signal,
and dump a formatted output of recent metrics. For example, when a process gets
a SIGUSR1, it can dump to stderr recent performance metrics for debugging.

Examples

Here is an example of using the package:

text

func SlowMethod() {
    // Profiling the runtime of a method
    defer metrics.MeasureSince([]string{"SlowMethod"}, time.Now())
}

// Configure a statsite sink as the global metrics sink
sink, _ := metrics.NewStatsiteSink("statsite:8125")
metrics.NewGlobal(metrics.DefaultConfig("service-name"), sink)

// Emit a Key/Value pair
metrics.EmitKey([]string{"questions", "meaning of life"}, 42)

Here is an example of setting up an signal handler:

text

// Setup the inmem sink and signal handler
inm := NewInmemSink(10*time.Second, time.Minute)
sig := DefaultInmemSignal(inm)
metrics.NewGlobal(metrics.DefaultConfig("service-name"), inm)

// Run some code
inm.SetGauge([]string{"foo"}, 42)
inm.EmitKey([]string{"bar"}, 30)

inm.IncrCounter([]string{"baz"}, 42)
inm.IncrCounter([]string{"baz"}, 1)
inm.IncrCounter([]string{"baz"}, 80)

inm.AddSample([]string{"method", "wow"}, 42)
inm.AddSample([]string{"method", "wow"}, 100)
inm.AddSample([]string{"method", "wow"}, 22)

....

When a signal comes in, output like the following will be dumped to stderr:

text

[2014-01-28 14:57:33.04 -0800 PST][G] 'foo': 42.000
[2014-01-28 14:57:33.04 -0800 PST][P] 'bar': 30.000
[2014-01-28 14:57:33.04 -0800 PST][C] 'baz': Count: 3 Min: 1.000 Mean: 41.000 Max: 80.000 Stddev: 39.509
[2014-01-28 14:57:33.04 -0800 PST][S] 'method.wow': Count: 3 Min: 22.000 Mean: 54.667 Max: 100.000 Stddev: 40.513

Overview (`vendor/src/github.com/armon/go-radix/README.md`)

go-radix [](https://travis-ci.org/armon/go-radix)

Provides the radix package that implements a radix tree.
The package only provides a single Tree implementation, optimized for sparse nodes.

As a radix tree, it provides the following:

O(k) operations. In many cases, this can be faster than a hash table since
the hash function is an O(k) operation, and hash tables have very poor cache locality.
Minimum / Maximum value lookups
Ordered iteration

Documentation

The full documentation is available on Godoc.

Example

Below is a simple example of usage

// Create a tree
r := radix.New()
r.Insert("foo", 1)
r.Insert("bar", 2)
r.Insert("foobar", 2)

// Find the longest prefix match
m, _, _ := r.LongestPrefix("foozip")
if m != "foo" {
    panic("should be foo")
}

Overview (`vendor/src/github.com/boltdb/bolt/README.md`)

Bolt [](https://coveralls.io/r/boltdb/bolt?branch=master) [](https://godoc.org/github.com/boltdb/bolt)

Bolt is a pure Go key/value store inspired by Howard Chu's
LMDB project. The goal of the project is to provide a simple,
fast, and reliable database for projects that don't require a full database
server such as Postgres or MySQL.

Since Bolt is meant to be used as such a low-level piece of functionality,
simplicity is key. The API will be small and only focus on getting values
and setting values. That's it.

Project Status

Bolt is stable and the API is fixed. Full unit test coverage and randomized
black box testing are used to ensure database consistency and thread safety.
Bolt is currently in high-load production environments serving databases as
large as 1TB. Many companies such as Shopify and Heroku use Bolt-backed
services every day.

Getting Started
Resources
Comparison with other databases
Caveats & Limitations
Reading the Source
Other Projects Using Bolt

Getting Started

Installing

To start using Bolt, install Go and run go get:

$ go get github.com/boltdb/bolt/...

This will retrieve the library and install the bolt command line utility into
your $GOBIN path.

Opening a database

The top-level object in Bolt is a DB. It is represented as a single file on
your disk and represents a consistent snapshot of your data.

To open your database, simply use the bolt.Open() function:

package main

import (
	"log"

	"github.com/boltdb/bolt"
)

func main() {
	// Open the my.db data file in your current directory.
	// It will be created if it doesn't exist.
	db, err := bolt.Open("my.db", 0600, nil)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	...
}

Please note that Bolt obtains a file lock on the data file so multiple processes
cannot open the same database at the same time. Opening an already open Bolt
database will cause it to hang until the other process closes it. To prevent
an indefinite wait you can pass a timeout option to the Open() function:

db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second})

Transactions

Bolt allows only one read-write transaction at a time but allows as many
read-only transactions as you want at a time. Each transaction has a consistent
view of the data as it existed when the transaction started.

Individual transactions and all objects created from them (e.g. buckets, keys)
are not thread safe. To work with data in multiple goroutines you must start
a transaction for each one or use locking to ensure only one goroutine accesses
a transaction at a time. Creating transaction from the DB is thread safe.

Read-only transactions and read-write transactions should not depend on one
another and generally shouldn't be opened simultaneously in the same goroutine.
This can cause a deadlock as the read-write transaction needs to periodically
re-map the data file but it cannot do so while a read-only transaction is open.

Read-write transactions

To start a read-write transaction, you can use the DB.Update() function:

err := db.Update(func(tx *bolt.Tx) error {
	...
	return nil
})

Inside the closure, you have a consistent view of the database. You commit the
transaction by returning nil at the end. You can also rollback the transaction
at any point by returning an error. All database operations are allowed inside
a read-write transaction.

Always check the return error as it will report any disk failures that can cause
your transaction to not complete. If you return an error within your closure
it will be passed through.

Read-only transactions

To start a read-only transaction, you can use the DB.View() function:

err := db.View(func(tx *bolt.Tx) error {
	...
	return nil
})

You also get a consistent view of the database within this closure, however,
no mutating operations are allowed within a read-only transaction. You can only
retrieve buckets, retrieve values, and copy the database within a read-only
transaction.

Batch read-write transactions

Each DB.Update() waits for disk to commit the writes. This overhead
can be minimized by combining multiple updates with the DB.Batch()
function:

err := db.Batch(func(tx *bolt.Tx) error {
	...
	return nil
})

Concurrent Batch calls are opportunistically combined into larger
transactions. Batch is only useful when there are multiple goroutines
calling it.

The trade-off is that Batch can call the given
function multiple times, if parts of the transaction fail. The
function must be idempotent and side effects must take effect only
after a successful return from DB.Batch().

For example: don't display messages from inside the function, instead
set variables in the enclosing scope:

var id uint64
err := db.Batch(func(tx *bolt.Tx) error {
	// Find last key in bucket, decode as bigendian uint64, increment
	// by one, encode back to []byte, and add new key.
	...
	id = newValue
	return nil
})
if err != nil {
	return ...
}
fmt.Println("Allocated ID %d", id)

Managing transactions manually

The DB.View() and DB.Update() functions are wrappers around the DB.Begin()
function. These helper functions will start the transaction, execute a function,
and then safely close your transaction if an error is returned. This is the
recommended way to use Bolt transactions.

However, sometimes you may want to manually start and end your transactions.
You can use the Tx.Begin() function directly but please be sure to close
the transaction.

// Start a writable transaction.
tx, err := db.Begin(true)
if err != nil {
    return err
}
defer tx.Rollback()

// Use the transaction...
_, err := tx.CreateBucket([]byte("MyBucket"))
if err != nil {
    return err
}

// Commit the transaction and check for error.
if err := tx.Commit(); err != nil {
    return err
}

The first argument to DB.Begin() is a boolean stating if the transaction
should be writable.

Using buckets

Buckets are collections of key/value pairs within the database. All keys in a
bucket must be unique. You can create a bucket using the DB.CreateBucket()
function:

db.Update(func(tx *bolt.Tx) error {
	b, err := tx.CreateBucket([]byte("MyBucket"))
	if err != nil {
		return fmt.Errorf("create bucket: %s", err)
	}
	return nil
})

You can also create a bucket only if it doesn't exist by using the
Tx.CreateBucketIfNotExists() function. It's a common pattern to call this
function for all your top-level buckets after you open your database so you can
guarantee that they exist for future transactions.

To delete a bucket, simply call the Tx.DeleteBucket() function.

Using key/value pairs

To save a key/value pair to a bucket, use the Bucket.Put() function:

db.Update(func(tx *bolt.Tx) error {
	b := tx.Bucket([]byte("MyBucket"))
	err := b.Put([]byte("answer"), []byte("42"))
	return err
})

This will set the value of the "answer" key to "42" in the MyBucket
bucket. To retrieve this value, we can use the Bucket.Get() function:

db.View(func(tx *bolt.Tx) error {
	b := tx.Bucket([]byte("MyBucket"))
	v := b.Get([]byte("answer"))
	fmt.Printf("The answer is: %s\n", v)
	return nil
})

The Get() function does not return an error because its operation is
guaranteed to work (unless there is some kind of system failure). If the key
exists then it will return its byte slice value. If it doesn't exist then it
will return nil. It's important to note that you can have a zero-length value
set to a key which is different than the key not existing.

Use the Bucket.Delete() function to delete a key from the bucket.

Please note that values returned from Get() are only valid while the
transaction is open. If you need to use a value outside of the transaction
then you must use copy() to copy it to another byte slice.

Autoincrementing integer for the bucket

By using the NextSequence() function, you can let Bolt determine a sequence
which can be used as the unique identifier for your key/value pairs. See the
example below.

// CreateUser saves u to the store. The new user ID is set on u once the data is persisted.
func (s *Store) CreateUser(u *User) error {
    return s.db.Update(func(tx *bolt.Tx) error {
        // Retrieve the users bucket.
        // This should be created when the DB is first opened.
        b := tx.Bucket([]byte("users"))

        // Generate ID for the user.
        // This returns an error only if the Tx is closed or not writeable.
        // That can't happen in an Update() call so I ignore the error check.
        id, _ = b.NextSequence()
        u.ID = int(id)

        // Marshal user data into bytes.
        buf, err := json.Marshal(u)
        if err != nil {
            return err
        }

        // Persist bytes to users bucket.
        return b.Put(itob(u.ID), buf)
    })
}

// itob returns an 8-byte big endian representation of v.
func itob(v int) []byte {
    b := make([]byte, 8)
    binary.BigEndian.PutUint64(b, uint64(v))
    return b
}

type User struct {
    ID int
    ...
}

Iterating over keys

Bolt stores its keys in byte-sorted order within a bucket. This makes sequential
iteration over these keys extremely fast. To iterate over keys we'll use a
Cursor:

db.View(func(tx *bolt.Tx) error {
	// Assume bucket exists and has keys
	b := tx.Bucket([]byte("MyBucket"))

	c := b.Cursor()

	for k, v := c.First(); k != nil; k, v = c.Next() {
		fmt.Printf("key=%s, value=%s\n", k, v)
	}

	return nil
})

The cursor allows you to move to a specific point in the list of keys and move
forward or backward through the keys one at a time.

The following functions are available on the cursor:

text

First()  Move to the first key.
Last()   Move to the last key.
Seek()   Move to a specific key.
Next()   Move to the next key.
Prev()   Move to the previous key.

Each of those functions has a return signature of (key []byte, value []byte).
When you have iterated to the end of the cursor then Next() will return a
nil key. You must seek to a position using First(), Last(), or Seek()
before calling Next() or Prev(). If you do not seek to a position then
these functions will return a nil key.

During iteration, if the key is non-nil but the value is nil, that means
the key refers to a bucket rather than a value. Use Bucket.Bucket() to
access the sub-bucket.

Prefix scans

To iterate over a key prefix, you can combine Seek() and bytes.HasPrefix():

db.View(func(tx *bolt.Tx) error {
	// Assume bucket exists and has keys
	c := tx.Bucket([]byte("MyBucket")).Cursor()

	prefix := []byte("1234")
	for k, v := c.Seek(prefix); bytes.HasPrefix(k, prefix); k, v = c.Next() {
		fmt.Printf("key=%s, value=%s\n", k, v)
	}

	return nil
})

Range scans

Another common use case is scanning over a range such as a time range. If you
use a sortable time encoding such as RFC3339 then you can query a specific
date range like this:

db.View(func(tx *bolt.Tx) error {
	// Assume our events bucket exists and has RFC3339 encoded time keys.
	c := tx.Bucket([]byte("Events")).Cursor()

	// Our time range spans the 90's decade.
	min := []byte("1990-01-01T00:00:00Z")
	max := []byte("2000-01-01T00:00:00Z")

	// Iterate over the 90's.
	for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() {
		fmt.Printf("%s: %s\n", k, v)
	}

	return nil
})

Note that, while RFC3339 is sortable, the Golang implementation of RFC3339Nano does not use a fixed number of digits after the decimal point and is therefore not sortable.

ForEach()

You can also use the function ForEach() if you know you'll be iterating over
all the keys in a bucket:

db.View(func(tx *bolt.Tx) error {
	// Assume bucket exists and has keys
	b := tx.Bucket([]byte("MyBucket"))

	b.ForEach(func(k, v []byte) error {
		fmt.Printf("key=%s, value=%s\n", k, v)
		return nil
	})
	return nil
})

Nested buckets

You can also store a bucket in a key to create nested buckets. The API is the
same as the bucket management API on the DB object:

func (*Bucket) CreateBucket(key []byte) (*Bucket, error)
func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error)
func (*Bucket) DeleteBucket(key []byte) error

Database backups

Bolt is a single file so it's easy to backup. You can use the Tx.WriteTo()
function to write a consistent view of the database to a writer. If you call
this from a read-only transaction, it will perform a hot backup and not block
your other database reads and writes.

By default, it will use a regular file handle which will utilize the operating
system's page cache. See the Tx
documentation for information about optimizing for larger-than-RAM datasets.

One common use case is to backup over HTTP so you can use tools like cURL to
do database backups:

func BackupHandleFunc(w http.ResponseWriter, req *http.Request) {
	err := db.View(func(tx *bolt.Tx) error {
		w.Header().Set("Content-Type", "application/octet-stream")
		w.Header().Set("Content-Disposition", `attachment; filename="my.db"`)
		w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size())))
		_, err := tx.WriteTo(w)
		return err
	})
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
	}
}

Then you can backup using this command:

$ curl http://localhost/backup > my.db

Or you can open your browser to http://localhost/backup and it will download
automatically.

If you want to backup to another file you can use the Tx.CopyFile() helper
function.

Statistics

The database keeps a running count of many of the internal operations it
performs so you can better understand what's going on. By grabbing a snapshot
of these stats at two points in time we can see what operations were performed
in that time range.

For example, we could start a goroutine to log stats every 10 seconds:

go func() {
	// Grab the initial stats.
	prev := db.Stats()

	for {
		// Wait for 10s.
		time.Sleep(10 * time.Second)

		// Grab the current stats and diff them.
		stats := db.Stats()
		diff := stats.Sub(&prev)

		// Encode stats to JSON and print to STDERR.
		json.NewEncoder(os.Stderr).Encode(diff)

		// Save stats for the next loop.
		prev = stats
	}
}()

It's also useful to pipe these stats to a service such as statsd for monitoring
or to provide an HTTP endpoint that will perform a fixed-length sample.

Read-Only Mode

Sometimes it is useful to create a shared, read-only Bolt database. To this,
set the Options.ReadOnly flag when opening your database. Read-only mode
uses a shared lock to allow multiple processes to read from the database but
it will block any processes from opening the database in read-write mode.

db, err := bolt.Open("my.db", 0666, &bolt.Options{ReadOnly: true})
if err != nil {
	log.Fatal(err)
}

Mobile Use (iOS/Android)

Bolt is able to run on mobile devices by leveraging the binding feature of the
gomobile tool. Create a struct that will
contain your database logic and a reference to a *bolt.DB with a initializing
contstructor that takes in a filepath where the database file will be stored.
Neither Android nor iOS require extra permissions or cleanup from using this method.

func NewBoltDB(filepath string) *BoltDB {
	db, err := bolt.Open(filepath+"/demo.db", 0600, nil)
	if err != nil {
		log.Fatal(err)
	}

	return &BoltDB{db}
}

type BoltDB struct {
	db *bolt.DB
	...
}

func (b *BoltDB) Path() string {
	return b.db.Path()
}

func (b *BoltDB) Close() {
	b.db.Close()
}

Database logic should be defined as methods on this wrapper struct.

To initialize this struct from the native language (both platforms now sync
their local storage to the cloud. These snippets disable that functionality for the
database file):

Android

java

String path;
if (android.os.Build.VERSION.SDK_INT >=android.os.Build.VERSION_CODES.LOLLIPOP){
    path = getNoBackupFilesDir().getAbsolutePath();
} else{
    path = getFilesDir().getAbsolutePath();
}
Boltmobiledemo.BoltDB boltDB = Boltmobiledemo.NewBoltDB(path)

iOS

objc

- (void)demo {
    NSString* path = [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory,
                                                          NSUserDomainMask,
                                                          YES) objectAtIndex:0];
	GoBoltmobiledemoBoltDB * demo = GoBoltmobiledemoNewBoltDB(path);
	[self addSkipBackupAttributeToItemAtPath:demo.path];
	//Some DB Logic would go here
	[demo close];
}

- (BOOL)addSkipBackupAttributeToItemAtPath:(NSString *) filePathString
{
    NSURL* URL= [NSURL fileURLWithPath: filePathString];
    assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]);

    NSError *error = nil;
    BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES]
                                  forKey: NSURLIsExcludedFromBackupKey error: &error];
    if(!success){
        NSLog(@"Error excluding %@ from backup %@", [URL lastPathComponent], error);
    }
    return success;
}

Resources

For more information on getting started with Bolt, check out the following articles:

Intro to BoltDB: Painless Performant Persistence by Nate Finch.
Bolt -- an embedded key/value database for Go by Progville

Comparison with other databases

Postgres, MySQL, & other relational databases

Relational databases structure data into rows and are only accessible through
the use of SQL. This approach provides flexibility in how you store and query
your data but also incurs overhead in parsing and planning SQL statements. Bolt
accesses all data by a byte slice key. This makes Bolt fast to read and write
data by key but provides no built-in support for joining values together.

Most relational databases (with the exception of SQLite) are standalone servers
that run separately from your application. This gives your systems
flexibility to connect multiple application servers to a single database
server but also adds overhead in serializing and transporting data over the
network. Bolt runs as a library included in your application so all data access
has to go through your application's process. This brings data closer to your
application but limits multi-process access to the data.

LevelDB, RocksDB

LevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Bolt in that
they are libraries bundled into the application, however, their underlying
structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes
random writes by using a write ahead log and multi-tiered, sorted files called
SSTables. Bolt uses a B+tree internally and only a single file. Both approaches
have trade-offs.

If you require a high random write throughput (>10,000 w/sec) or you need to use
spinning disks then LevelDB could be a good choice. If your application is
read-heavy or does a lot of range scans then Bolt could be a good choice.

One other important consideration is that LevelDB does not have transactions.
It supports batch writing of key/values pairs and it supports read snapshots
but it will not give you the ability to do a compare-and-swap operation safely.
Bolt supports fully serializable ACID transactions.

LMDB

Bolt was originally a port of LMDB so it is architecturally similar. Both use
a B+tree, have ACID semantics with fully serializable transactions, and support
lock-free MVCC using a single writer and multiple readers.

The two projects have somewhat diverged. LMDB heavily focuses on raw performance
while Bolt has focused on simplicity and ease of use. For example, LMDB allows
several unsafe actions such as direct writes for the sake of performance. Bolt
opts to disallow actions which can leave the database in a corrupted state. The
only exception to this in Bolt is DB.NoSync.

There are also a few differences in API. LMDB requires a maximum mmap size when
opening an mdb_env whereas Bolt will handle incremental mmap resizing
automatically. LMDB overloads the getter and setter functions with multiple
flags whereas Bolt splits these specialized cases into their own functions.

Caveats & Limitations

It's important to pick the right tool for the job and Bolt is no exception.
Here are a few things to note when evaluating and using Bolt:

Bolt is good for read intensive workloads. Sequential write performance is
also fast but random writes can be slow. You can use DB.Batch() or add a
write-ahead log to help mitigate this issue.
Bolt uses a B+tree internally so there can be a lot of random page access.
SSDs provide a significant performance boost over spinning disks.
Try to avoid long running read transactions. Bolt uses copy-on-write so
old pages cannot be reclaimed while an old transaction is using them.
Byte slices returned from Bolt are only valid during a transaction. Once the
transaction has been committed or rolled back then the memory they point to
can be reused by a new page or can be unmapped from virtual memory and you'll
see an unexpected fault address panic when accessing it.
Be careful when using Bucket.FillPercent. Setting a high fill percent for
buckets that have random inserts will cause your database to have very poor
page utilization.
Use larger buckets in general. Smaller buckets causes poor page utilization
once they become larger than the page size (typically 4KB).
Bulk loading a lot of random writes into a new bucket can be slow as the
page will not split until the transaction is committed. Randomly inserting
more than 100,000 key/value pairs into a single new bucket in a single
transaction is not advised.
Bolt uses a memory-mapped file so the underlying operating system handles the
caching of the data. Typically, the OS will cache as much of the file as it
can in memory and will release memory as needed to other processes. This means
that Bolt can show very high memory usage when working with large databases.
However, this is expected and the OS will release memory as needed. Bolt can
handle databases much larger than the available physical RAM, provided its
memory-map fits in the process virtual address space. It may be problematic
on 32-bits systems.
The data structures in the Bolt database are memory mapped so the data file
will be endian specific. This means that you cannot copy a Bolt file from a
little endian machine to a big endian machine and have it work. For most
users this is not a concern since most modern CPUs are little endian.
Because of the way pages are laid out on disk, Bolt cannot truncate data files
and return free pages back to the disk. Instead, Bolt maintains a free list
of unused pages within its data file. These free pages can be reused by later
transactions. This works well for many use cases as databases generally tend
to grow. However, it's important to note that deleting large chunks of data
will not allow you to reclaim that space on disk.

For more information on page allocation, see this comment.

Reading the Source

Bolt is a relatively small code base (<3KLOC) for an embedded, serializable,
transactional key/value database so it can be a good starting point for people
interested in how databases work.

The best places to start are the main entry points into Bolt:

Open() - Initializes the reference to the database. It's responsible for
creating the database if it doesn't exist, obtaining an exclusive lock on the
file, reading the meta pages, & memory-mapping the file.
DB.Begin() - Starts a read-only or read-write transaction depending on the
value of the writable argument. This requires briefly obtaining the "meta"
lock to keep track of open transactions. Only one read-write transaction can
exist at a time so the "rwlock" is acquired during the life of a read-write
transaction.
Bucket.Put() - Writes a key/value pair into a bucket. After validating the
arguments, a cursor is used to traverse the B+tree to the page and position
where they key & value will be written. Once the position is found, the bucket
materializes the underlying page and the page's parent pages into memory as
"nodes". These nodes are where mutations occur during read-write transactions.
These changes get flushed to disk during commit.
Bucket.Get() - Retrieves a key/value pair from a bucket. This uses a cursor
to move to the page & position of a key/value pair. During a read-only
transaction, the key and value data is returned as a direct reference to the
underlying mmap file so there's no allocation overhead. For read-write
transactions, this data may reference the mmap file or one of the in-memory
node values.
Cursor - This object is simply for traversing the B+tree of on-disk pages
or in-memory nodes. It can seek to a specific key, move to the first or last
value, or it can move forward or backward. The cursor handles the movement up
and down the B+tree transparently to the end user.
Tx.Commit() - Converts the in-memory dirty nodes and the list of free pages
into pages to be written to disk. Writing to disk then occurs in two phases.
First, the dirty pages are written to disk and an fsync() occurs. Second, a
new meta page with an incremented transaction ID is written and another
fsync() occurs. This two phase write ensures that partially written data
pages are ignored in the event of a crash since the meta page pointing to them
is never written. Partially written meta pages are invalidated because they
are written with a checksum.

If you have additional notes that could be helpful for others, please submit
them via pull request.

Other Projects Using Bolt

Below is a list of public, open source projects that use Bolt:

Operation Go: A Routine Mission - An online programming game for Golang using Bolt for user accounts and a leaderboard.
Bazil - A file system that lets your data reside where it is most convenient for it to reside.
DVID - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb.
Skybox Analytics - A standalone funnel analysis tool for web analytics.
Scuttlebutt - Uses Bolt to store and process all Twitter mentions of GitHub projects.
Wiki - A tiny wiki using Goji, BoltDB and Blackfriday.
ChainStore - Simple key-value interface to a variety of storage engines organized as a chain of operations.
MetricBase - Single-binary version of Graphite.
Gitchain - Decentralized, peer-to-peer Git repositories aka "Git meets Bitcoin".
event-shuttle - A Unix system service to collect and reliably deliver messages to Kafka.
ipxed - Web interface and api for ipxed.
BoltStore - Session store using Bolt.
photosite/session - Sessions for a photo viewing site.
LedisDB - A high performance NoSQL, using Bolt as optional storage.
ipLocator - A fast ip-geo-location-server using bolt with bloom filters.
cayley - Cayley is an open-source graph database using Bolt as optional backend.
bleve - A pure Go search engine similar to ElasticSearch that uses Bolt as the default storage backend.
tentacool - REST api server to manage system stuff (IP, DNS, Gateway...) on a linux server.
SkyDB - Behavioral analytics database.
Seaweed File System - Highly scalable distributed key~file system with O(1) disk read.
InfluxDB - Scalable datastore for metrics, events, and real-time analytics.
Freehold - An open, secure, and lightweight platform for your files and data.
Prometheus Annotation Server - Annotation server for PromDash & Prometheus service monitoring system.
Consul - Consul is service discovery and configuration made easy. Distributed, highly available, and datacenter-aware.
Kala - Kala is a modern job scheduler optimized to run on a single node. It is persistent, JSON over HTTP API, ISO 8601 duration notation, and dependent jobs.
drive - drive is an unofficial Google Drive command line client for *NIX operating systems.
stow - a persistence manager for objects
backed by boltdb.
buckets - a bolt wrapper streamlining
simple tx and key scans.
mbuckets - A Bolt wrapper that allows easy operations on multi level (nested) buckets.
Request Baskets - A web service to collect arbitrary HTTP requests and inspect them via REST API or simple web UI, similar to RequestBin service
Go Report Card - Go code quality report cards as a (free and open source) service.
Boltdb Boilerplate - Boilerplate wrapper around bolt aiming to make simple calls one-liners.
lru - Easy to use Bolt-backed Least-Recently-Used (LRU) read-through cache with chainable remote stores.
Storm - A simple ORM around BoltDB.
GoWebApp - A basic MVC web application in Go using BoltDB.
SimpleBolt - A simple way to use BoltDB. Deals mainly with strings.
Algernon - A HTTP/2 web server with built-in support for Lua. Uses BoltDB as the default database backend.

If you are using Bolt in a project please send a pull request to add it to the list.

Overview (`vendor/src/github.com/deckarep/golang-set/README.md`)

golang-set

The missing set collection for the Go language. Until Go has sets built-in...use this.

Coming from Python one of the things I miss is the superbly wonderful set collection. This is my attempt to mimic the primary features of the set from Python.
You can of course argue that there is no need for a set in Go, otherwise the creators would have added one to the standard library. To those I say simply ignore this repository
and carry-on and to the rest that find this useful please contribute in helping me make it better by:

Helping to make more idiomatic improvements to the code.
Helping to increase the performance of it. ~~(So far, no attempt has been made, but since it uses a map internally, I expect it to be mostly performant.)~~
Helping to make the unit-tests more robust and kick-ass.
Helping to fill in the documentation.
Simply offering feedback and suggestions. (Positive, constructive feedback is appreciated.)

I have to give some credit for helping seed the idea with this post on stackoverflow.

Update - as of 3/9/2014, you can use a compile-time generic version of this package in the gen framework. This framework allows you to use the golang-set in a completely generic and type-safe way by allowing you to generate a supporting .go file based on your custom types.

Features (as of 9/22/2014)

a CartesionProduct() method has been added with unit-tests: Read more about the cartesion product

Features (as of 9/15/2014)

a PowerSet() method has been added with unit-tests: Read more about the Power set

Features (as of 4/22/2014)

One common interface to both implementations
Two set implementations to choose from
- a thread-safe implementation designed for concurrent use
- a non-thread-safe implementation designed for performance
75 benchmarks for both implementations
35 unit tests for both implementations
14 concurrent tests for the thread-safe implementation

Please see the unit test file for additional usage examples. The Python set documentation will also do a better job than I can of explaining how a set typically works. Please keep in mind
however that the Python set is a built-in type and supports additional features and syntax that make it awesome.

Examples but not exhaustive:

text

/* Detailed source-code truncated for AI context efficiency. */

Thanks!

-Ralph

Overview (`vendor/src/github.com/docker/distribution/README.md`)

Distribution

The Docker toolset to pack, ship, store, and deliver content.

This repository's main product is the Docker Registry 2.0 implementation
for storing and distributing Docker images. It supersedes the
docker/docker-registry
project with a new API design, focused around security and performance.

This repository contains the following components:

Component	Description
registry	An implementation of the Docker Registry HTTP API V2 for use with docker 1.6+.
libraries	A rich set of libraries for interacting with distribution components. Please see godoc for details. Note: These libraries are unstable.
specifications	Distribution related specifications are available in docs/spec
documentation	Docker's full documentation set is available at docs.docker.com. This repository contains the subset related just to the registry.

How does this integrate with Docker engine?

This project should provide an implementation to a V2 API for use in the Docker
core project. The API should be embeddable
and simplify the process of securely pulling and pushing content from docker
daemons.

What are the long term goals of the Distribution project?

The Distribution project has the further long term goal of providing a
secure tool chain for distributing content. The specifications, APIs and tools
should be as useful with Docker as they are without.

Our goal is to design a professional grade and extensible content distribution
system that allow users to:

Enjoy an efficient, secured and reliable way to store, manage, package and
exchange content
Hack/roll their own on top of healthy open-source components
Implement their own home made solution through good specs, and solid
extensions mechanism.

More about Registry 2.0

The new registry implementation provides the following benefits:

faster push and pull
new, more efficient implementation
simplified deployment
pluggable storage backend
webhook notifications

For information on upcoming functionality, please see ROADMAP.md.

Who needs to deploy a registry?

By default, Docker users pull images from Docker's public registry instance.
Installing Docker gives users this
ability. Users can also push images to a repository on Docker's public registry,
if they have a Docker Hub account.

For some users and even companies, this default behavior is sufficient. For
others, it is not.

For example, users with their own software products may want to maintain a
registry for private, company images. Also, you may wish to deploy your own
image repository for images used to test or in continuous integration. For these
use cases and others, deploying your own registry instance
may be the better choice.

Migration to Registry 2.0

For those who have previously deployed their own registry based on the Registry
1.0 implementation and wish to deploy a Registry 2.0 while retaining images,
data migration is required. A tool to assist with migration efforts has been
created. For more information see [docker/migrator]
(https://github.com/docker/migrator).

Contribute

Please see CONTRIBUTING.md for details on how to contribute
issues, fixes, and patches to this project. If you are contributing code, see
the instructions for building a development environment.

Support

If any issues are encountered while using the Distribution project, several
avenues are available for support:

IRC	#docker-distribution on FreeNode
Issue Tracker	github.com/docker/distribution/issues
Google Groups	https://groups.google.com/a/dockerproject.org/forum/#!forum/distribution
Mailing List	[email protected]

License

This project is distributed under Apache License, Version 2.0.

Overview (`vendor/src/github.com/docker/go-events/README.md`)

Docker Events Package

The Docker events package implements a composable event distribution package
for Go.

Originally created to implement the notifications in Docker Registry
2,
we've found the pattern to be useful in other applications. This package is
most of the same code with slightly updated interfaces. Much of the internals
have been made available.

Usage

The events package centers around a Sink type. Events are written with
calls to Sink.Write(event Event). Sinks can be wired up in various
configurations to achieve interesting behavior.

The canonical example is that employed by the
docker/distribution/notifications
package. Let's say we have a type httpSink where we'd like to queue
notifications. As a rule, it should send a single http request and return an
error if it fails:

func (h *httpSink) Write(event Event) error {
	p, err := json.Marshal(event)
	if err != nil {
		return err
	}
	body := bytes.NewReader(p)
	resp, err := h.client.Post(h.url, "application/json", body)
	if err != nil {
		return err
	}
	defer resp.Body.Close()
	
	if resp.Status != 200 {
		return errors.New("unexpected status")
	}

	return nil
}

// implement (*httpSink).Close()

With just that, we can start using components from this package. One can call
(*httpSink).Write to send events as the body of a post request to a
configured URL.

Retries

HTTP can be unreliable. The first feature we'd like is to have some retry:

hs := newHTTPSink(/*...*/)
retry := NewRetryingSink(hs, NewBreaker(5, time.Second))

We now have a sink that will retry events against the httpSink until they
succeed. The retry will backoff for one second after 5 consecutive failures
using the breaker strategy.

Queues

This isn't quite enough. We we want a sink that doesn't block while we are
waiting for events to be sent. Let's add a Queue:

queue := NewQueue(retry)

Now, we have an unbounded queue that will work through all events sent with
(*Queue).Write. Events can be added asynchronously to the queue without
blocking the current execution path. This is ideal for use in an http request.

Broadcast

It usually turns out that you want to send to more than one listener. We can
use Broadcaster to support this:

var broadcast = NewBroadcaster() // make it available somewhere in your application.
broadcast.Add(queue) // add your queue!
broadcast.Add(queue2) // and another!

With the above, we can now call broadcast.Write in our http handlers and have
all the events distributed to each queue. Because the events are queued, not
listener blocks another.

Extending

For the most part, the above is sufficient for a lot of applications. However,
extending the above functionality can be done implementing your own Sink. The
behavior and semantics of the sink can be completely dependent on the
application requirements. The interface is provided below for reference:

type Sink {
	Write(Event) error
	Close() error
}

Application behavior can be controlled by how Write behaves. The examples
above are designed to queue the message and return as quickly as possible.
Other implementations may block until the event is committed to durable
storage.

Overview (`vendor/src/github.com/docker/go-units/README.md`)

Introduction

go-units is a library to transform human friendly measurements into machine friendly values.

Usage

See the docs in godoc for examples and documentation.

License

go-units is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

Overview (`vendor/src/github.com/docker/libkv/README.md`)

libkv

libkv provides a Go native library to store metadata.

The goal of libkv is to abstract common store operations for multiple distributed and/or local Key/Value store backends.

For example, you can use it to store your metadata or for service discovery to register machines and endpoints inside your cluster.

You can also easily implement a generic Leader Election on top of it (see the docker/leadership repository).

As of now, libkv offers support for Consul, Etcd, Zookeeper (Distributed store) and BoltDB (Local store).

Usage

libkv is meant to be used as an abstraction layer over existing distributed Key/Value stores. It is especially useful if you plan to support consul, etcd and zookeeper using the same codebase.

It is ideal if you plan for something written in Go that should support:

A simple metadata storage, distributed or local
A lightweight discovery service for your nodes
A distributed lock mechanism

You can find examples of usage for libkv under in docs/examples.go. Optionally you can also take a look at the docker/swarm or docker/libnetwork repositories which are using docker/libkv for all the use cases listed above.

Supported versions

libkv supports:

Consul versions >= 0.5.1 because it uses Sessions with Delete behavior for the use of TTLs (mimics zookeeper's Ephemeral node support), If you don't plan to use TTLs: you can use Consul version 0.4.0+.
Etcd versions >= 2.0 because it uses the new coreos/etcd/client, this might change in the future as the support for APIv3 comes along and adds more capabilities.
Zookeeper versions >= 3.4.5. Although this might work with previous version but this remains untested as of now.
Boltdb, which shouldn't be subject to any version dependencies.

Interface

A storage backend in libkv should implement (fully or partially) this interface:

type Store interface {
	Put(key string, value []byte, options *WriteOptions) error
	Get(key string) (*KVPair, error)
	Delete(key string) error
	Exists(key string) (bool, error)
	Watch(key string, stopCh <-chan struct{}) (<-chan *KVPair, error)
	WatchTree(directory string, stopCh <-chan struct{}) (<-chan []*KVPair, error)
	NewLock(key string, options *LockOptions) (Locker, error)
	List(directory string) ([]*KVPair, error)
	DeleteTree(directory string) error
	AtomicPut(key string, value []byte, previous *KVPair, options *WriteOptions) (bool, *KVPair, error)
	AtomicDelete(key string, previous *KVPair) (bool, error)
	Close()
}

Compatibility matrix

Backend drivers in libkv are generally divided between local drivers and distributed drivers. Distributed backends offer enhanced capabilities like Watches and/or distributed Locks.

Local drivers are usually used in complement to the distributed drivers to store informations that only needs to be available locally.

Calls	Consul	Etcd	Zookeeper	BoltDB
Put	X	X	X	X
Get	X	X	X	X
Delete	X	X	X	X
Exists	X	X	X	X
Watch	X	X	X
WatchTree	X	X	X
NewLock (Lock/Unlock)	X	X	X
List	X	X	X	X
DeleteTree	X	X	X	X
AtomicPut	X	X	X	X
Close	X	X	X	X

Limitations

Distributed Key/Value stores often have different concepts for managing and formatting keys and their associated values. Even though libkv tries to abstract those stores aiming for some consistency, in some cases it can't be applied easily.

Please refer to the docs/compatibility.md to see what are the special cases for cross-backend compatibility.

Other than those special cases, you should expect the same experience for basic operations like Get/Put, etc.

Calls like WatchTree may return different events (or number of events) depending on the backend (for now, Etcd and Consul will likely return more events than Zookeeper that you should triage properly). Although you should be able to use it successfully to watch on events in an interchangeable way (see the docker/leadership repository or the pkg/discovery/kv package in docker/docker).

TLS

Only Consul and etcd have support for TLS and you should build and provide your own config.TLS object to feed the client. Support is planned for zookeeper.

##Roadmap

Make the API nicer to use (using options)
Provide more options (consistency for example)
Improve performance (remove extras Get/List operations)
Better key formatting
New backends?

##Contributing

Want to hack on libkv? Docker's contributions guidelines apply.

##Copyright and license

Copyright © 2014-2016 Docker, Inc. All rights reserved, except as follows. Code is released under the Apache 2.0 license. The README.md file, and files in the "docs" folder are licensed under the Creative Commons Attribution 4.0 International License under the terms and conditions set forth in the file "LICENSE.docs". You may obtain a duplicate copy of the same license, titled CC-BY-SA-4.0, at http://creativecommons.org/licenses/by/4.0/.

Overview (`vendor/src/github.com/docker/libnetwork/README.md`)

libnetwork - networking for containers

Libnetwork provides a native Go implementation for connecting containers

The goal of libnetwork is to deliver a robust Container Network Model that provides a consistent programming interface and the required network abstractions for applications.

Design

Please refer to the design for more information.

Using libnetwork

There are many networking solutions available to suit a broad range of use-cases. libnetwork uses a driver / plugin model to support all of these solutions while abstracting the complexity of the driver implementations by exposing a simple and consistent Network Model to users.

text

/* Detailed source-code truncated for AI context efficiency. */

Future

Please refer to roadmap for more information.

Contributing

Want to hack on libnetwork? Docker's contributions guidelines apply.

Copyright and license

Overview (`vendor/src/github.com/docker/libtrust/README.md`)

libtrust

Libtrust is library for managing authentication and authorization using public key cryptography.

Authentication is handled using the identity attached to the public key.
Libtrust provides multiple methods to prove possession of the private key associated with an identity.

TLS x509 certificates
Signature verification
Key Challenge

Authorization and access control is managed through a distributed trust graph.
Trust servers are used as the authorities of the trust graph and allow caching portions of the graph for faster access.

Copyright and license

Code and documentation copyright 2014 Docker, inc. Code released under the Apache 2.0 license.
Docs released under Creative commons.

Overview (`vendor/src/github.com/docker/notary/README.md`)

Notary

The Notary project comprises a server and a client for running and interacting
with trusted collections. Please see the service architecture documentation
for more information.

Notary aims to make the internet more secure by making it easy for people to
publish and verify content. We often rely on TLS to secure our communications
with a web server which is inherently flawed, as any compromise of the server
enables malicious content to be substituted for the legitimate content.

With Notary, publishers can sign their content offline using keys kept highly
secure. Once the publisher is ready to make the content available, they can
push their signed trusted collection to a Notary Server.

Consumers, having acquired the publisher's public key through a secure channel,
can then communicate with any notary server or (insecure) mirror, relying
only on the publisher's key to determine the validity and integrity of the
received content.

Goals

Notary is based on The Update Framework, a secure general design for the problem of software distribution and updates. By using TUF, notary achieves a number of key advantages:

Survivable Key Compromise: Content publishers must manage keys in order to sign their content. Signing keys may be compromised or lost so systems must be designed in order to be flexible and recoverable in the case of key compromise. TUF's notion of key roles is utilized to separate responsibilities across a hierarchy of keys such that loss of any particular key (except the root role) by itself is not fatal to the security of the system.
Freshness Guarantees: Replay attacks are a common problem in designing secure systems, where previously valid payloads are replayed to trick another system. The same problem exists in the software update systems, where old signed can be presented as the most recent. notary makes use of timestamping on publishing so that consumers can know that they are receiving the most up to date content. This is particularly important when dealing with software update where old vulnerable versions could be used to attack users.
Configurable Trust Thresholds: Oftentimes there are a large number of publishers that are allowed to publish a particular piece of content. For example, open source projects where there are a number of core maintainers. Trust thresholds can be used so that content consumers require a configurable number of signatures on a piece of content in order to trust it. Using thresholds increases security so that loss of individual signing keys doesn't allow publishing of malicious content.
Signing Delegation: To allow for flexible publishing of trusted collections, a content publisher can delegate part of their collection to another signer. This delegation is represented as signed metadata so that a consumer of the content can verify both the content and the delegation.
Use of Existing Distribution: Notary's trust guarantees are not tied at all to particular distribution channels from which content is delivered. Therefore, trust can be added to any existing content delivery mechanism.
Untrusted Mirrors and Transport: All of the notary metadata can be mirrored and distributed via arbitrary channels.

Security

Please see our service architecture docs for more information about our threat model, which details the varying survivability and severities for key compromise as well as mitigations.

Our last security audit was on July 31, 2015 by NCC (results).

Any security vulnerabilities can be reported to [email protected].

Getting started with the Notary CLI

Please get the Notary Client CLI binary from the official releases page or you can build one yourself.
The version of Notary server and signer should be greater than or equal to Notary CLI's version to ensure feature compatibility (ex: CLI version 0.2, server/signer version >= 0.2), and all official releases are associated with GitHub tags.

To use the Notary CLI with Docker hub images, please have a look at our
getting started docs.

For more advanced usage, please see the
advanced usage docs.

To use the CLI against a local Notary server rather than against Docker Hub:

Please ensure that you have docker and docker-compose installed.
git clone https://github.com/docker/notary.git and from the cloned repository path,
start up a local Notary server and signer and copy the config file and testing certs to your
local notary config directory:
sh
```
$ docker-compose build
$ docker-compose up -d
$ mkdir -p ~/.notary && cp cmd/notary/config.json cmd/notary/root-ca.crt ~/.notary
```
Add 127.0.0.1 notary-server to your /etc/hosts, or if using docker-machine,
add $(docker-machine ip) notary-server).

You can run through the examples in the
getting started docs and
advanced usage docs, but
without the -s (server URL) argument to the notary command since the server
URL is specified already in the configuration, file you copied.

You can also leave off the -d ~/.docker/trust argument if you do not care
to use notary with Docker images.

Building Notary

Prerequisites:

Go >= 1.6.1
godep installed
libtool development headers installed
- Ubuntu: apt-get install libltdl-dev
- CentOS/RedHat: yum install libtool-ltdl-devel
- Mac OS (Homebrew): brew install libtool

Run make binaries, which creates the Notary Client CLI binary at bin/notary.
Note that make binaries assumes a standard Go directory structure, in which
Notary is checked out to the src directory in your GOPATH. For example:

text

$GOPATH/
    src/
        github.com/
            docker/
                notary/

Overview (`vendor/src/github.com/go-check/check/README.md`)

Go-check

This is a fork of https://github.com/go-check/check

The intention of this fork is not to change any of the original behavior, but add
some specific behaviors needed for some of my projects already using this test suite.
For documentation on the main behavior of go-check see the aforementioned repo.

The original branch is intact at orig_v1

Overview (`vendor/src/github.com/go-ini/ini/README.md`)

ini [](https://drone.io/github.com/go-ini/ini/latest) [](http://gocover.io/github.com/go-ini/ini)

Package ini provides INI file read and write functionality in Go.

简体中文

Feature

Load multiple data sources([]byte or file) with overwrites.
Read with recursion values.
Read with parent-child sections.
Read with auto-increment key names.
Read with multiple-line values.
Read with tons of helper methods.
Read and convert values to Go types.
Read and WRITE comments of sections and keys.
Manipulate sections, keys and comments with ease.
Keep sections and keys in order as you parse and save.

Installation

text

go get gopkg.in/ini.v1

Getting Started

Loading from data sources

A Data Source is either raw data in type []byte or a file name with type string and you can load as many as data sources you want. Passing other types will simply return an error.

cfg, err := ini.Load([]byte("raw data"), "filename")

Or start with an empty object:

cfg := ini.Empty()

When you cannot decide how many data sources to load at the beginning, you still able to Append() them later.

err := cfg.Append("other file", []byte("other raw data"))

Working with sections

To get a section, you would need to:

section, err := cfg.GetSection("section name")

For a shortcut for default section, just give an empty string as name:

section, err := cfg.GetSection("")

When you're pretty sure the section exists, following code could make your life easier:

section := cfg.Section("")

What happens when the section somehow does not exist? Don't panic, it automatically creates and returns a new section to you.

To create a new section:

err := cfg.NewSection("new section")

To get a list of sections or section names:

sections := cfg.Sections()
names := cfg.SectionStrings()

Working with keys

To get a key under a section:

key, err := cfg.Section("").GetKey("key name")

Same rule applies to key operations:

key := cfg.Section("").Key("key name")

To create a new key:

err := cfg.Section("").NewKey("name", "value")

To get a list of keys or key names:

keys := cfg.Section("").Keys()
names := cfg.Section("").KeyStrings()

To get a clone hash of keys and corresponding values:

hash := cfg.GetSection("").KeysHash()

Working with values

To get a string value:

val := cfg.Section("").Key("key name").String()

To validate key value on the fly:

val := cfg.Section("").Key("key name").Validate(func(in string) string {
	if len(in) == 0 {
		return "default"
	}
	return in
})

To get value with types:

text

/* Detailed source-code truncated for AI context efficiency. */

What if my value is three-line long?

ini

[advance]
ADDRESS = """404 road,
NotFound, State, 5000
Earth"""

Not a problem!

cfg.Section("advance").Key("ADDRESS").String()

/* --- start ---
404 road,
NotFound, State, 5000
Earth
------  end  --- */

That's cool, how about continuation lines?

ini

[advance]
two_lines = how about \
	continuation lines?
lots_of_lines = 1 \
	2 \
	3 \
	4

Piece of cake!

cfg.Section("advance").Key("two_lines").String() // how about continuation lines?
cfg.Section("advance").Key("lots_of_lines").String() // 1 2 3 4

Note that single quotes around values will be stripped:

ini

foo = "some value" // foo: some value
bar = 'some value' // bar: some value

That's all? Hmm, no.

Helper methods of working with values

To get value with given candidates:

v = cfg.Section("").Key("STRING").In("default", []string{"str", "arr", "types"})
v = cfg.Section("").Key("FLOAT64").InFloat64(1.1, []float64{1.25, 2.5, 3.75})
v = cfg.Section("").Key("INT").InInt(5, []int{10, 20, 30})
v = cfg.Section("").Key("INT64").InInt64(10, []int64{10, 20, 30})
v = cfg.Section("").Key("UINT").InUint(4, []int{3, 6, 9})
v = cfg.Section("").Key("UINT64").InUint64(8, []int64{3, 6, 9})
v = cfg.Section("").Key("TIME").InTimeFormat(time.RFC3339, time.Now(), []time.Time{time1, time2, time3})
v = cfg.Section("").Key("TIME").InTime(time.Now(), []time.Time{time1, time2, time3}) // RFC3339

Default value will be presented if value of key is not in candidates you given, and default value does not need be one of candidates.

To validate value in a given range:

vals = cfg.Section("").Key("FLOAT64").RangeFloat64(0.0, 1.1, 2.2)
vals = cfg.Section("").Key("INT").RangeInt(0, 10, 20)
vals = cfg.Section("").Key("INT64").RangeInt64(0, 10, 20)
vals = cfg.Section("").Key("UINT").RangeUint(0, 3, 9)
vals = cfg.Section("").Key("UINT64").RangeUint64(0, 3, 9)
vals = cfg.Section("").Key("TIME").RangeTimeFormat(time.RFC3339, time.Now(), minTime, maxTime)
vals = cfg.Section("").Key("TIME").RangeTime(time.Now(), minTime, maxTime) // RFC3339

To auto-split value into slice:

vals = cfg.Section("").Key("STRINGS").Strings(",")
vals = cfg.Section("").Key("FLOAT64S").Float64s(",")
vals = cfg.Section("").Key("INTS").Ints(",")
vals = cfg.Section("").Key("INT64S").Int64s(",")
vals = cfg.Section("").Key("UINTS").Uints(",")
vals = cfg.Section("").Key("UINT64S").Uint64s(",")
vals = cfg.Section("").Key("TIMES").Times(",")

Save your configuration

Finally, it's time to save your configuration to somewhere.

A typical way to save configuration is writing it to a file:

// ...
err = cfg.SaveTo("my.ini")
err = cfg.SaveToIndent("my.ini", "\t")

Another way to save is writing to a io.Writer interface:

// ...
cfg.WriteTo(writer)
cfg.WriteToIndent(writer, "\t")

Advanced Usage

Recursive Values

For all value of keys, there is a special syntax %(<name>)s, where <name> is the key name in same section or default section, and %(<name>)s will be replaced by corresponding value(empty string if key not found). You can use this syntax at most 99 level of recursions.

ini

NAME = ini

[author]
NAME = Unknwon
GITHUB = https://github.com/%(NAME)s

[package]
FULL_NAME = github.com/go-ini/%(NAME)s

cfg.Section("author").Key("GITHUB").String()		// https://github.com/Unknwon
cfg.Section("package").Key("FULL_NAME").String()	// github.com/go-ini/ini

Parent-child Sections

You can use . in section name to indicate parent-child relationship between two or more sections. If the key not found in the child section, library will try again on its parent section until there is no parent section.

ini

NAME = ini
VERSION = v1
IMPORT_PATH = gopkg.in/%(NAME)s.%(VERSION)s

[package]
CLONE_URL = https://%(IMPORT_PATH)s

[package.sub]

cfg.Section("package.sub").Key("CLONE_URL").String()	// https://gopkg.in/ini.v1

Auto-increment Key Names

If key name is - in data source, then it would be seen as special syntax for auto-increment key name start from 1, and every section is independent on counter.

ini

[features]
-: Support read/write comments of keys and sections
-: Support auto-increment of key names
-: Support load multiple files to overwrite key values

cfg.Section("features").KeyStrings()	// []{"#1", "#2", "#3"}

Map To Struct

Want more objective way to play with INI? Cool.

ini

Name = Unknwon
age = 21
Male = true
Born = 1993-01-01T20:17:05Z

[Note]
Content = Hi is a good man!
Cities = HangZhou, Boston

type Note struct {
	Content string
	Cities  []string
}

type Person struct {
	Name string
	Age  int `ini:"age"`
	Male bool
	Born time.Time
	Note
	Created time.Time `ini:"-"`
}

func main() {
	cfg, err := ini.Load("path/to/ini")
	// ...
	p := new(Person)
	err = cfg.MapTo(p)
	// ...

	// Things can be simpler.
	err = ini.MapTo(p, "path/to/ini")
	// ...

	// Just map a section? Fine.
	n := new(Note)
	err = cfg.Section("Note").MapTo(n)
	// ...
}

Can I have default value for field? Absolutely.

Assign it before you map to struct. It will keep the value as it is if the key is not presented or got wrong type.

// ...
p := &Person{
	Name: "Joe",
}
// ...

It's really cool, but what's the point if you can't give me my file back from struct?

Reflect From Struct

Why not?

type Embeded struct {
	Dates  []time.Time `delim:"|"`
	Places []string
	None   []int
}

type Author struct {
	Name      string `ini:"NAME"`
	Male      bool
	Age       int
	GPA       float64
	NeverMind string `ini:"-"`
	*Embeded
}

func main() {
	a := &Author{"Unknwon", true, 21, 2.8, "",
		&Embeded{
			[]time.Time{time.Now(), time.Now()},
			[]string{"HangZhou", "Boston"},
			[]int{},
		}}
	cfg := ini.Empty()
	err = ini.ReflectFrom(cfg, a)
	// ...
}

So, what do I get?

ini

NAME = Unknwon
Male = true
Age = 21
GPA = 2.8

[Embeded]
Dates = 2015-08-07T22:14:22+08:00|2015-08-07T22:14:22+08:00
Places = HangZhou,Boston
None =

Name Mapper

To save your time and make your code cleaner, this library supports NameMapper between struct field and actual section and key name.

There are 2 built-in name mappers:

AllCapsUnderscore: it converts to format ALL_CAPS_UNDERSCORE then match section or key.
TitleUnderscore: it converts to format title_underscore then match section or key.

To use them:

type Info struct {
	PackageName string
}

func main() {
	err = ini.MapToWithMapper(&Info{}, ini.TitleUnderscore, []byte("packag_name=ini"))
	// ...

	cfg, err := ini.Load([]byte("PACKAGE_NAME=ini"))
	// ...
	info := new(Info)
	cfg.NameMapper = ini.AllCapsUnderscore
	err = cfg.MapTo(info)
	// ...
}

Same rules of name mapper apply to ini.ReflectFromWithMapper function.

Other Notes On Map/Reflect

Any embedded struct is treated as a section by default, and there is no automatic parent-child relations in map/reflect feature:

type Child struct {
	Age string
}

type Parent struct {
	Name string
	Child
}

type Config struct {
	City string
	Parent
}

Example configuration:

ini

City = Boston

[Parent]
Name = Unknwon

[Child]
Age = 21

What if, yes, I'm paranoid, I want embedded struct to be in the same section. Well, all roads lead to Rome.

type Child struct {
	Age string
}

type Parent struct {
	Name string
	Child `ini:"Parent"`
}

type Config struct {
	City string
	Parent
}

Example configuration:

ini

City = Boston

[Parent]
Name = Unknwon
Age = 21

Getting Help

FAQs

What does `BlockMode` field do?

By default, library lets you read and write values so we need a locker to make sure your data is safe. But in cases that you are very sure about only reading data through the library, you can set cfg.BlockMode = false to speed up read operations about 50-70% faster.

Why another INI library?

Many people are using my another INI library goconfig, so the reason for this one is I would like to make more Go style code. Also when you set cfg.BlockMode = false, this one is about 10-30% faster.

To make those changes I have to confirm API broken, so it's safer to keep it in another place and start using gopkg.in to version my package at this time.(PS: shorter import path)

License

This project is under Apache v2 License. See the LICENSE file for the full license text.

Overview (`vendor/src/github.com/gorilla/context/README.md`)

context

gorilla/context is a general purpose registry for global request variables.

Read the full documentation here: http://www.gorillatoolkit.org/pkg/context

Overview (`vendor/src/github.com/gorilla/mux/README.md`)

mux

gorilla/mux is a powerful URL router and dispatcher.

Read the full documentation here: http://www.gorillatoolkit.org/pkg/mux

Overview (`vendor/src/github.com/hashicorp/go-immutable-radix/README.md`)

go-immutable-radix [](https://travis-ci.org/hashicorp/go-immutable-radix)

Provides the iradix package that implements an immutable radix tree.
The package only provides a single Tree implementation, optimized for sparse nodes.

As a radix tree, it provides the following:

O(k) operations. In many cases, this can be faster than a hash table since
the hash function is an O(k) operation, and hash tables have very poor cache locality.
Minimum / Maximum value lookups
Ordered iteration

A tree supports using a transaction to batch multiple updates (insert, delete)
in a more efficient manner than performing each operation one at a time.

For a mutable variant, see go-radix.

Documentation

The full documentation is available on Godoc.

Example

Below is a simple example of usage

// Create a tree
r := iradix.New()
r, _, _ = r.Insert([]byte("foo"), 1)
r, _, _ = r.Insert([]byte("bar"), 2)
r, _, _ = r.Insert([]byte("foobar"), 2)

// Find the longest prefix match
m, _, _ := r.Root().LongestPrefix([]byte("foozip"))
if string(m) != "foo" {
    panic("should be foo")
}

Overview (`vendor/src/github.com/hashicorp/go-memdb/README.md`)

go-memdb

Provides the memdb package that implements a simple in-memory database
built on immutable radix trees. The database provides Atomicity, Consistency
and Isolation from ACID. Being that it is in-memory, it does not provide durability.
The database is instantiated with a schema that specifies the tables and indicies
that exist and allows transactions to be executed.

The database provides the following:

Multi-Version Concurrency Control (MVCC) - By leveraging immutable radix trees
the database is able to support any number of concurrent readers without locking,
and allows a writer to make progress.
Transaction Support - The database allows for rich transactions, in which multiple
objects are inserted, updated or deleted. The transactions can span multiple tables,
and are applied atomically. The database provides atomicity and isolation in ACID
terminology, such that until commit the updates are not visible.
Rich Indexing - Tables can support any number of indexes, which can be simple like
a single field index, or more advanced compound field indexes. Certain types like
UUID can be efficiently compressed from strings into byte indexes for reduces
storage requirements.

For the underlying immutable radix trees, see go-immutable-radix.

Documentation

The full documentation is available on Godoc.

Example

Below is a simple example of usage

// Create a sample struct
type Person struct {
    Email string
    Name  string
    Age   int
}

// Create the DB schema
schema := &memdb.DBSchema{
    Tables: map[string]*memdb.TableSchema{
        "person": &memdb.TableSchema{
            Name: "person",
            Indexes: map[string]*memdb.IndexSchema{
                "id": &memdb.IndexSchema{
                    Name:    "id",
                    Unique:  true,
                    Indexer: &memdb.StringFieldIndex{Field: "Email"},
                },
            },
        },
    },
}

// Create a new data base
db, err := memdb.NewMemDB(schema)
if err != nil {
    panic(err)
}

// Create a write transaction
txn := db.Txn(true)

// Insert a new person
p := &Person{"[email protected]", "Joe", 30}
if err := txn.Insert("person", p); err != nil {
    panic(err)
}

// Commit the transaction
txn.Commit()

// Create read-only transaction
txn = db.Txn(false)
defer txn.Abort()

// Lookup by email
raw, err := txn.First("person", "id", "[email protected]")
if err != nil {
    panic(err)
}

// Say hi!
fmt.Printf("Hello %s!", raw.(*Person).Name)

Overview (`vendor/src/github.com/hashicorp/go-multierror/README.md`)

go-multierror

go-multierror is a package for Go that provides a mechanism for
representing a list of error values as a single error.

This allows a function in Go to return an error that might actually
be a list of errors. If the caller knows this, they can unwrap the
list and access the errors. If the caller doesn't know, the error
formats to a nice human-readable format.

go-multierror implements the
errwrap interface so that it can
be used with that library, as well.

Installation and Docs

Install using go get github.com/hashicorp/go-multierror.

Full documentation is available at
http://godoc.org/github.com/hashicorp/go-multierror

Usage

go-multierror is easy to use and purposely built to be unobtrusive in
existing Go applications/libraries that may not be aware of it.

Building a list of errors

The Append function is used to create a list of errors. This function
behaves a lot like the Go built-in append function: it doesn't matter
if the first argument is nil, a multierror.Error, or any other error,
the function behaves as you would expect.

var result error

if err := step1(); err != nil {
	result = multierror.Append(result, err)
}
if err := step2(); err != nil {
	result = multierror.Append(result, err)
}

return result

Customizing the formatting of the errors

By specifying a custom ErrorFormat, you can customize the format
of the Error() string function:

var result *multierror.Error

// ... accumulate errors here, maybe using Append

if result != nil {
	result.ErrorFormat = func([]error) string {
		return "errors!"
	}
}

Accessing the list of errors

multierror.Error implements error so if the caller doesn't know about
multierror, it will work just fine. But if you're aware a multierror might
be returned, you can use type switches to access the list of errors:

if err := something(); err != nil {
	if merr, ok := err.(*multierror.Error); ok {
		// Use merr.Errors
	}
}

Returning a multierror only if there are errors

If you build a multierror.Error, you can use the ErrorOrNil function
to return an error implementation only if there are errors to return:

var result *multierror.Error

// ... accumulate errors here

// Return the `error` only if errors were added to the multierror, otherwise
// return nil since there are no errors.
return result.ErrorOrNil()

Overview (`vendor/src/github.com/hashicorp/memberlist/README.md`)

memberlist [](https://godoc.org/github.com/hashicorp/memberlist)

memberlist is a Go library that manages cluster
membership and member failure detection using a gossip based protocol.

The use cases for such a library are far-reaching: all distributed systems
require membership, and memberlist is a re-usable solution to managing
cluster membership and node failure detection.

memberlist is eventually consistent but converges quickly on average.
The speed at which it converges can be heavily tuned via various knobs
on the protocol. Node failures are detected and network partitions are partially
tolerated by attempting to communicate to potentially dead nodes through
multiple routes.

Building

If you wish to build memberlist you'll need Go version 1.2+ installed.

Please check your installation with:

text

go version

Usage

Memberlist is surprisingly simple to use. An example is shown below:

/* Create the initial memberlist from a safe configuration.
   Please reference the godoc for other default config types.
   http://godoc.org/github.com/hashicorp/memberlist#Config
*/
list, err := memberlist.Create(memberlist.DefaultLocalConfig())
if err != nil {
	panic("Failed to create memberlist: " + err.Error())
}

// Join an existing cluster by specifying at least one known member.
n, err := list.Join([]string{"1.2.3.4"})
if err != nil {
	panic("Failed to join cluster: " + err.Error())
}

// Ask for members of the cluster
for _, member := range list.Members() {
	fmt.Printf("Member: %s %s\n", member.Name, member.Addr)
}

// Continue doing whatever you need, memberlist will maintain membership
// information in the background. Delegates can be used for receiving
// events when members join or leave.

The most difficult part of memberlist is configuring it since it has many
available knobs in order to tune state propagation delay and convergence times.
Memberlist provides a default configuration that offers a good starting point,
but errs on the side of caution, choosing values that are optimized for
higher convergence at the cost of higher bandwidth usage.

For complete documentation, see the associated Godoc.

Protocol

memberlist is based on "SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol",
with a few minor adaptations, mostly to increase propagation speed and
convergence rate.

A high level overview of the memberlist protocol (based on SWIM) is
described below, but for details please read the full
SWIM paper
followed by the memberlist source. We welcome any questions related
to the protocol on our issue tracker.

Protocol Description

memberlist begins by joining an existing cluster or starting a new
cluster. If starting a new cluster, additional nodes are expected to join
it. New nodes in an existing cluster must be given the address of at
least one existing member in order to join the cluster. The new member
does a full state sync with the existing member over TCP and begins gossiping its
existence to the cluster.

Gossip is done over UDP to a with a configurable but fixed fanout and interval.
This ensures that network usage is constant with regards to number of nodes, as opposed to
exponential growth that can occur with traditional heartbeat mechanisms.
Complete state exchanges with a random node are done periodically over
TCP, but much less often than gossip messages. This increases the likelihood
that the membership list converges properly since the full state is exchanged
and merged. The interval between full state exchanges is configurable or can
be disabled entirely.

Failure detection is done by periodic random probing using a configurable interval.
If the node fails to ack within a reasonable time (typically some multiple
of RTT), then an indirect probe as well as a direct TCP probe are attempted. An
indirect probe asks a configurable number of random nodes to probe the same node,
in case there are network issues causing our own node to fail the probe. The direct
TCP probe is used to help identify the common situation where networking is
misconfigured to allow TCP but not UDP. Without the TCP probe, a UDP-isolated node
would think all other nodes were suspect and could cause churn in the cluster when
it attempts a TCP-based state exchange with another node. It is not desirable to
operate with only TCP connectivity because convergence will be much slower, but it
is enabled so that memberlist can detect this situation and alert operators.

If both our probe, the indirect probes, and the direct TCP probe fail within a
configurable time, then the node is marked "suspicious" and this knowledge is
gossiped to the cluster. A suspicious node is still considered a member of
cluster. If the suspect member of the cluster does not dispute the suspicion
within a configurable period of time, the node is finally considered dead,
and this state is then gossiped to the cluster.

This is a brief and incomplete description of the protocol. For a better idea,
please read the
SWIM paper
in its entirety, along with the memberlist source code.

Changes from SWIM

As mentioned earlier, the memberlist protocol is based on SWIM but includes
minor changes, mostly to increase propagation speed and convergence rates.

The changes from SWIM are noted here:

memberlist does a full state sync over TCP periodically. SWIM only propagates
changes over gossip. While both eventually reach convergence, the full state
sync increases the likelihood that nodes are fully converged more quickly,
at the expense of more bandwidth usage. This feature can be totally disabled
if you wish.
memberlist has a dedicated gossip layer separate from the failure detection
protocol. SWIM only piggybacks gossip messages on top of probe/ack messages.
memberlist also piggybacks gossip messages on top of probe/ack messages, but
also will periodically send out dedicated gossip messages on their own. This
feature lets you have a higher gossip rate (for example once per 200ms)
and a slower failure detection rate (such as once per second), resulting
in overall faster convergence rates and data propagation speeds. This feature
can be totally disabed as well, if you wish.
memberlist stores around the state of dead nodes for a set amount of time,
so that when full syncs are requested, the requester also receives information
about dead nodes. Because SWIM doesn't do full syncs, SWIM deletes dead node
state immediately upon learning that the node is dead. This change again helps
the cluster converge more quickly.

Overview (`vendor/src/github.com/imdario/mergo/README.md`)

Mergo

A helper to merge structs and maps in Golang. Useful for configuration default values, avoiding messy if-statements.

Also a lovely comune (municipality) in the Province of Ancona in the Italian region Marche.

Status

It is ready for production use. It works fine after extensive use in the wild.

Important note

Mergo is intended to assign only zero value fields on destination with source value. Since April 6th it works like this. Before it didn't work properly, causing some random overwrites. After some issues and PRs I found it didn't merge as I designed it. Thanks to imdario/mergo#8 overwriting functions were added and the wrong behavior was clearly detected.

If you were using Mergo before April 6th 2015, please check your project works as intended after updating your local copy with go get -u github.com/imdario/mergo. I apologize for any issue caused by its previous behavior and any future bug that Mergo could cause (I hope it won't!) in existing projects after the change (release 0.2.0).

Mergo in the wild

Installation

text

go get github.com/imdario/mergo

// use in your .go code
import (
    "github.com/imdario/mergo"
)

Usage

You can only merge same-type structs with exported fields initialized as zero value of their type and same-types maps. Mergo won't merge unexported (private) fields but will do recursively any exported one. Also maps will be merged recursively except for structs inside maps (because they are not addressable using Go reflection).

text

if err := mergo.Merge(&dst, src); err != nil {
    // ...
}

Additionally, you can map a map[string]interface{} to a struct (and otherwise, from struct to map), following the same restrictions as in Merge(). Keys are capitalized to find each corresponding exported field.

text

if err := mergo.Map(&dst, srcMap); err != nil {
    // ...
}

Warning: if you map a struct to map, it won't do it recursively. Don't expect Mergo to map struct members of your struct as map[string]interface{}. They will be just assigned as values.

More information and examples in godoc documentation.

Nice example

package main

import (
	"fmt"
	"github.com/imdario/mergo"
)

type Foo struct {
	A string
	B int64
}

func main() {
	src := Foo{
		A: "one",
	}

	dest := Foo{
		A: "two",
		B: 2,
	}

	mergo.Merge(&dest, src)

	fmt.Println(dest)
	// Will print
	// {two 2}
}

Note: if test are failing due missing package, please execute:

text

go get gopkg.in/yaml.v1

Contact me

If I can help you, you have an idea or you are using Mergo in your projects, don't hesitate to drop me a line (or a pull request): @im_dario

About

Written by Dario Castañé.

License

BSD 3-Clause license, as Go language.

Overview (`vendor/src/github.com/inconshreveable/mousetrap/README.md`)

mousetrap

mousetrap is a tiny library that answers a single question.

On a Windows machine, was the process invoked by someone double clicking on
the executable file while browsing in explorer?

Motivation

Windows developers unfamiliar with command line tools will often "double-click"
the executable for a tool. Because most CLI tools print the help and then exit
when invoked without arguments, this is often very frustrating for those users.

mousetrap provides a way to detect these invocations so that you can provide
more helpful behavior and instructions on how to run the CLI tool. To see what
this looks like, both from an organizational and a technical perspective, see
https://inconshreveable.com/09-09-2014/sweat-the-small-stuff/

The interface

The library exposes a single interface:

text

func StartedByExplorer() (bool)

Overview (`vendor/src/github.com/jmespath/go-jmespath/README.md`)

go-jmespath - A JMESPath implementation in Go

See http://jmespath.org for more info.

Overview (`vendor/src/github.com/kr/pty/README.md`)

pty

Pty is a Go package for using unix pseudo-terminals.

Install

text

go get github.com/kr/pty

Example

package main

import (
	"github.com/kr/pty"
	"io"
	"os"
	"os/exec"
)

func main() {
	c := exec.Command("grep", "--color=auto", "bar")
	f, err := pty.Start(c)
	if err != nil {
		panic(err)
	}

	go func() {
		f.Write([]byte("foo\n"))
		f.Write([]byte("bar\n"))
		f.Write([]byte("baz\n"))
		f.Write([]byte{4}) // EOT
	}()
	io.Copy(os.Stdout, f)
}

Overview (`vendor/src/github.com/mattn/go-shellwords/README.md`)

go-shellwords

Parse line as shell words.

Usage

args, err := shellwords.Parse("./foo --bar=baz")
// args should be ["./foo", "--bar=baz"]

os.Setenv("FOO", "bar")
p := shellwords.NewParser()
p.ParseEnv = true
args, err := p.Parse("./foo $FOO")
// args should be ["./foo", "bar"]

p := shellwords.NewParser()
p.ParseBacktick = true
args, err := p.Parse("./foo `echo $SHELL`")
// args should be ["./foo", "/bin/bash"]

shellwords.ParseBacktick = true
p := shellwords.NewParser()
args, err := p.Parse("./foo `echo $SHELL`")
// args should be ["./foo", "/bin/bash"]

Thanks

This is based on cpan module Parse::CommandLine.

License

under the MIT License: http://mattn.mit-license.org/2014

Author

Yasuhiro Matsumoto (a.k.a mattn)

Overview (`vendor/src/github.com/mattn/go-sqlite3/README.md`)

go-sqlite3

Description

sqlite3 driver conforming to the built-in database/sql interface

Installation

This package can be installed with the go get command:

text

go get github.com/mattn/go-sqlite3

Documentation

API documentation can be found here: http://godoc.org/github.com/mattn/go-sqlite3

Examples can be found under the ./_example directory

FAQ

Want to build go-sqlite3 with libsqlite3 on my linux.

Use go build --tags "libsqlite3 linux"
Want to build go-sqlite3 with icu extension.

Use go build --tags "icu"
Can't build go-sqlite3 on windows 64bit.

Probably, you are using go 1.0, go1.0 has a problem when it comes to compiling/linking on windows 64bit.
See: https://github.com/mattn/go-sqlite3/issues/27
Getting insert error while query is opened.

You can pass some arguments into the connection string, for example, a URI.
See: https://github.com/mattn/go-sqlite3/issues/39
Do you want cross compiling? mingw on Linux or Mac?

See: https://github.com/mattn/go-sqlite3/issues/106
See also: http://www.limitlessfx.com/cross-compile-golang-app-for-windows-from-linux.html
Want to get time.Time with current locale

Use loc=auto in SQLite3 filename schema like file:foo.db?loc=auto.

License

MIT: http://mattn.mit-license.org/2012

sqlite3-binding.c, sqlite3-binding.h, sqlite3ext.h

The -binding suffix was added to avoid build failures under gccgo.

In this repository, those files are amalgamation code that copied from SQLite3. The license of those codes are depend on the license of SQLite3.

Author

Yasuhiro Matsumoto (a.k.a mattn)

Overview (`vendor/src/github.com/miekg/dns/README.md`)

Alternative (more granular) approach to a DNS library

Less is more.

Complete and usable DNS library. All widely used Resource Records are
supported, including the DNSSEC types. It follows a lean and mean philosophy.
If there is stuff you should know as a DNS programmer there isn't a convenience
function for it. Server side and client side programming is supported, i.e. you
can build servers and resolvers with it.

We try to keep the "master" branch as sane as possible and at the bleeding edge
of standards, avoiding breaking changes wherever reasonable. We support the last
two versions of Go, currently: 1.4 and 1.5.

Goals

KISS;
Fast;
Small API, if its easy to code in Go, don't make a function for it.

Users

A not-so-up-to-date-list-that-may-be-actually-current:

Send pull request if you want to be listed here.

Features

UDP/TCP queries, IPv4 and IPv6;
RFC 1035 zone file parsing ($INCLUDE, $ORIGIN, $TTL and $GENERATE (for all record types) are supported;
Fast:
- Reply speed around ~ 80K qps (faster hardware results in more qps);
- Parsing RRs ~ 100K RR/s, that's 5M records in about 50 seconds;
Server side programming (mimicking the net/http package);
Client side programming;
DNSSEC: signing, validating and key generation for DSA, RSA and ECDSA;
EDNS0, NSID;
AXFR/IXFR;
TSIG, SIG(0);
DNS name compression;
Depends only on the standard library.

Have fun!

Miek Gieben - 2010-2012 - [email protected]

Building

Building is done with the go tool. If you have setup your GOPATH
correctly, the following should work:

text

go get github.com/miekg/dns
go build github.com/miekg/dns

Examples

A short "how to use the API" is at the beginning of doc.go (this also will show
when you call godoc github.com/miekg/dns).

Example programs can be found in the github.com/miekg/exdns repository.

Supported RFCs

all of them

103{4,5} - DNS standard
1348 - NSAP record (removed the record)
1982 - Serial Arithmetic
1876 - LOC record
1995 - IXFR
1996 - DNS notify
2136 - DNS Update (dynamic updates)
2181 - RRset definition - there is no RRset type though, just []RR
2537 - RSAMD5 DNS keys
2065 - DNSSEC (updated in later RFCs)
2671 - EDNS record
2782 - SRV record
2845 - TSIG record
2915 - NAPTR record
2929 - DNS IANA Considerations
3110 - RSASHA1 DNS keys
3225 - DO bit (DNSSEC OK)
340{1,2,3} - NAPTR record
3445 - Limiting the scope of (DNS)KEY
3597 - Unknown RRs
4025 - IPSECKEY
403{3,4,5} - DNSSEC + validation functions
4255 - SSHFP record
4343 - Case insensitivity
4408 - SPF record
4509 - SHA256 Hash in DS
4592 - Wildcards in the DNS
4635 - HMAC SHA TSIG
4701 - DHCID
4892 - id.server
5001 - NSID
5155 - NSEC3 record
5205 - HIP record
5702 - SHA2 in the DNS
5936 - AXFR
5966 - TCP implementation recommendations
6605 - ECDSA
6725 - IANA Registry Update
6742 - ILNP DNS
6840 - Clarifications and Implementation Notes for DNS Security
6844 - CAA record
6891 - EDNS0 update
6895 - DNS IANA considerations
6975 - Algorithm Understanding in DNSSEC
7043 - EUI48/EUI64 records
7314 - DNS (EDNS) EXPIRE Option
7553 - URI record
xxxx - EDNS0 DNS Update Lease (draft)

Loosely based upon

ldns
NSD
Net::DNS
GRONG

TODO

privatekey.Precompute() when signing?
Last remaining RRs: APL, ATMA, A6, NSAP and NXT.
Missing in parsing: ISDN, UNSPEC, NSAP and ATMA.
NSEC(3) cover/match/closest enclose.
Replies with TC bit are not parsed to the end.

Overview (`vendor/src/github.com/miekg/pkcs11/README.md`)

PKCS#11 [](https://travis-ci.org/miekg/pkcs11)

This is a Go implementation of the PKCS#11 API. It wraps the library closely, but uses Go idiom
were it makes sense. It has been tested with SoftHSM.

SoftHSM

Make it use a custom configuration file export SOFTHSM_CONF=$PWD/softhsm.conf

Then use softhsm to init it

text

softhsm --init-token --slot 0 --label test --pin 1234

Then use libsofthsm.so as the pkcs11 module:
text
```
p := pkcs11.New("/usr/lib/softhsm/libsofthsm.so")
```

Examples

A skeleton program would look somewhat like this (yes, pkcs#11 is verbose):

text

p := pkcs11.New("/usr/lib/softhsm/libsofthsm.so")
err := p.Initialize()
if err != nil {
    panic(err)
}

defer p.Destroy()
defer p.Finalize()

slots, err := p.GetSlotList(true)
if err != nil {
    panic(err)
}

session, err := p.OpenSession(slots[0], pkcs11.CKF_SERIAL_SESSION|pkcs11.CKF_RW_SESSION)
if err != nil {
    panic(err)
}
defer p.CloseSession(session)

err = p.Login(session, pkcs11.CKU_USER, "1234")
if err != nil {
    panic(err)
}
defer p.Logout(session)

p.DigestInit(session, []*pkcs11.Mechanism{pkcs11.NewMechanism(pkcs11.CKM_SHA_1, nil)})
hash, err := p.Digest(session, []byte("this is a string"))
if err != nil {
    panic(err)
}

for _, d := range hash {
        fmt.Printf("%x", d)
}
fmt.Println()

Further examples are included in the tests.

TODO

Fix/double check endian stuff, see types.go NewAttribute()
Look at the memory copying in fast functions (sign, hash etc)

Overview (`vendor/src/github.com/mistifyio/go-zfs/README.md`)

Go Wrapper for ZFS

Simple wrappers for ZFS command line tools.

Requirements

You need a working ZFS setup. To use on Ubuntu 14.04, setup ZFS:

text

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:zfs-native/stable
sudo apt-get update
sudo apt-get install ubuntu-zfs libzfs-dev

Developed using Go 1.3, but currently there isn't anything 1.3 specific. Don't use Ubuntu packages for Go, use http://golang.org/doc/install

Generally you need root privileges to use anything zfs related.

Status

This has been only been tested on Ubuntu 14.04

In the future, we hope to work directly with libzfs.

Hacking

The tests have decent examples for most functions.

//assuming a zpool named test
//error handling ommitted


f, err := zfs.CreateFilesystem("test/snapshot-test", nil)
ok(t, err)

s, err := f.Snapshot("test", nil)
ok(t, err)

// snapshot is named "test/snapshot-test@test"

c, err := s.Clone("test/clone-test", nil)

err := c.Destroy()
err := s.Destroy()
err := f.Destroy()

Contributing

See the contributing guidelines

Overview (`vendor/src/github.com/mreiferson/go-httpclient/README.md`)

go-httpclient

requires Go 1.1+ as of v0.4.0 the API has been completely re-written for Go 1.1 (for a Go
1.0.x compatible release see 1adef50)

Provides an HTTP Transport that implements the RoundTripper interface and
can be used as a built in replacement for the standard library's, providing:

connection timeouts
request timeouts

This is a thin wrapper around http.Transport that sets dial timeouts and uses
Go's internal timer scheduler to call the Go 1.1+ CancelRequest() API.

Example

transport := &httpclient.Transport{
    ConnectTimeout:        1*time.Second,
    RequestTimeout:        10*time.Second,
    ResponseHeaderTimeout: 5*time.Second,
}
defer transport.Close()

client := &http.Client{Transport: transport}
req, _ := http.NewRequest("GET", "http://127.0.0.1/test", nil)
resp, err := client.Do(req)
if err != nil {
    return err
}
defer resp.Body.Close()

Note: you will want to re-use a single client object rather than creating one for each request, otherwise you will end up leaking connections.

Reference Docs

For API docs see godoc.

Overview (`vendor/src/github.com/pborman/uuid/README.md`)

This project was automatically exported from code.google.com/p/go-uuid

uuid

The uuid package generates and inspects UUIDs based on RFC 412 and DCE 1.1: Authentication and Security Services.

Install

go get github.com/pborman/uuid

Documentation

Full go doc style documentation for the package can be viewed online without installing this package by using the GoDoc site here:
http://godoc.org/github.com/pborman/uuid

Overview (`vendor/src/github.com/philhofer/fwd/README.md`)

fwd

text

import "github.com/philhofer/fwd"

The fwd package provides a buffered reader
and writer. Each has methods that help improve
the encoding/decoding performance of some binary
protocols.

The fwd.Writer and fwd.Reader type provide similar
functionality to their counterparts in bufio, plus
a few extra utility methods that simplify read-ahead
and write-ahead. I wrote this package to improve serialization
performance for http://github.com/philhofer/msgp,
where it provided about a 2x speedup over bufio. However,
care must be taken to understand the semantics of the
extra methods provided by this package, as they allow
the user to access and manipulate the buffer memory
directly.

The extra methods for fwd.Reader are Peek, Skip
and Next. (*fwd.Reader).Peek, unlike (*bufio.Reader).Peek,
will re-allocate the read buffer in order to accommodate arbitrarily
large read-ahead. (*fwd.Reader).Skip skips the next n bytes
in the stream, and uses the io.Seeker interface if the underlying
stream implements it. (*fwd.Reader).Next returns a slice pointing
to the next n bytes in the read buffer (like Peek), but also
increments the read position. This allows users to process streams
in aribtrary block sizes without having to manage appropriately-sized
slices. Additionally, obviating the need to copy the data from the
buffer to another location in memory can improve performance dramatically
in CPU-bound applications.

fwd.Writer only has one extra method, which is (*fwd.Writer).Next, which
returns a slice pointing to the next n bytes of the writer, and increments
the write position by the length of the returned slice. This allows users
to write directly to the end of the buffer.

Constants

const (
    // DefaultReaderSize is the default size of the read buffer
    DefaultReaderSize = 2048
)

const (
    // DefaultWriterSize is the
    // default write buffer size.
    DefaultWriterSize = 2048
)

type Reader

type Reader struct {
    // contains filtered or unexported fields
}

Reader is a buffered look-ahead reader

func NewReader

func NewReader(r io.Reader) *Reader

NewReader returns a new *Reader that reads from 'r'

func NewReaderSize

func NewReaderSize(r io.Reader, n int) *Reader

NewReaderSize returns a new *Reader that
reads from 'r' and has a buffer size 'n'

func (\*Reader) BufferSize

func (r *Reader) BufferSize() int

BufferSize returns the total size of the buffer

func (\*Reader) Buffered

func (r *Reader) Buffered() int

Buffered returns the number of bytes currently in the buffer

func (\*Reader) Next

func (r *Reader) Next(n int) ([]byte, error)

Next returns the next 'n' bytes in the stream.
If the returned slice has a length less than 'n',
an error will also be returned.
Unlike Peek, Next advances the reader position.
The returned bytes point to the same
data as the buffer, so the slice is
only valid until the next reader method call.
An EOF is considered an unexpected error.

func (\*Reader) Peek

func (r *Reader) Peek(n int) ([]byte, error)

Peek returns the next 'n' buffered bytes,
reading from the underlying reader if necessary.
It will only return a slice shorter than 'n' bytes
if it also returns an error. Peek does not advance
the reader. EOF errors are not returned as
io.ErrUnexpectedEOF.

func (\*Reader) Read

func (r *Reader) Read(b []byte) (int, error)

Read implements io.Reader

func (\*Reader) ReadByte

func (r *Reader) ReadByte() (byte, error)

ReadByte implements io.ByteReader

func (\*Reader) ReadFull

func (r *Reader) ReadFull(b []byte) (int, error)

ReadFull attempts to read len(b) bytes into
'b'. It returns the number of bytes read into
'b', and an error if it does not return len(b).

func (\*Reader) Reset

func (r *Reader) Reset(rd io.Reader)

Reset resets the underlying reader
and the read buffer.

func (\*Reader) Skip

func (r *Reader) Skip(n int) (int, error)

Skip moves the reader forward 'n' bytes.
Returns the number of bytes skipped and any
errors encountered. It is analagous to Seek(n, 1).
If the underlying reader implements io.Seeker, then
that method will be used to skip forward.

If the reader encounters
an EOF before skipping 'n' bytes, it
returns io.ErrUnexpectedEOF. If the
underlying reader implements io.Seeker, then
those rules apply instead. (Many implementations
will not return io.EOF until the next call
to Read.)

func (\*Reader) WriteTo

func (r *Reader) WriteTo(w io.Writer) (int64, error)

WriteTo implements io.WriterTo

type Writer

type Writer struct {
    // contains filtered or unexported fields
}

Writer is a buffered writer

func NewWriter

func NewWriter(w io.Writer) *Writer

NewWriter returns a new writer
that writes to 'w' and has a buffer
that is DefaultWriterSize bytes.

func NewWriterSize

func NewWriterSize(w io.Writer, size int) *Writer

NewWriterSize returns a new writer
that writes to 'w' and has a buffer
that is 'size' bytes.

func (\*Writer) BufferSize

func (w *Writer) BufferSize() int

BufferSize returns the maximum size of the buffer.

func (\*Writer) Buffered

func (w *Writer) Buffered() int

Buffered returns the number of buffered bytes
in the reader.

func (\*Writer) Flush

func (w *Writer) Flush() error

Flush flushes any buffered bytes
to the underlying writer.

func (\*Writer) Next

func (w *Writer) Next(n int) ([]byte, error)

Next returns the next 'n' free bytes
in the write buffer, flushing the writer
as necessary. Next will return io.ErrShortBuffer
if 'n' is greater than the size of the write buffer.

func (\*Writer) ReadFrom

func (w *Writer) ReadFrom(r io.Reader) (int64, error)

ReadFrom implements io.ReaderFrom

func (\*Writer) Write

func (w *Writer) Write(p []byte) (int, error)

Write implements io.Writer

func (\*Writer) WriteByte

func (w *Writer) WriteByte(b byte) error

WriteByte implements io.ByteWriter

func (\*Writer) WriteString

func (w *Writer) WriteString(s string) (int, error)

WriteString is analagous to Write, but it takes a string.

Generated by godoc2md

Overview (`vendor/src/github.com/pivotal-golang/clock/README.md`)

Provides a Clock interface, useful for injecting time dependencies in tests.

Overview (`vendor/src/github.com/pkg/errors/README.md`)

errors [](https://travis-ci.org/pkg/errors) [](https://ci.appveyor.com/project/davecheney/errors/branch/master) [](http://godoc.org/github.com/pkg/errors) [](https://goreportcard.com/report/github.com/pkg/errors)

Package errors provides simple error handling primitives.

The traditional error handling idiom in Go is roughly akin to

if err != nil {
        return err
}

which applied recursively up the call stack results in error reports without context or debugging information. The errors package allows programmers to add context to the failure path in their code in a way that does not destroy the original value of the error.

Adding context to an error

The errors.Wrap function returns a new error that adds context to the original error. For example

_, err := ioutil.ReadAll(r)
if err != nil {
        return errors.Wrap(err, "read failed")
}

Retrieving the cause of an error

Using errors.Wrap constructs a stack of errors, adding context to the preceding error. Depending on the nature of the error it may be necessary to reverse the operation of errors.Wrap to retrieve the original error for inspection. Any error value which implements this interface can be inspected by errors.Cause.

type causer interface {
        Cause() error
}

errors.Cause will recursively retrieve the topmost error which does not implement causer, which is assumed to be the original cause. For example:

switch err := errors.Cause(err).(type) {
case *MyError:
        // handle specifically
default:
        // unknown error
}

Read the package documentation for more information.

Contributing

We welcome pull requests, bug fixes and issue reports. With that said, the bar for adding new symbols to this package is intentionally set high.

Before proposing a change, please discuss your change by raising an issue.

Licence

BSD-2-Clause

Overview (`vendor/src/github.com/prometheus/procfs/README.md`)

procfs

This procfs package provides functions to retrieve system, kernel and process
metrics from the pseudo-filesystem proc.

WARNING: This package is a work in progress. Its API may still break in
backwards-incompatible ways without warnings. Use it at your own risk.

Overview (`vendor/src/github.com/spf13/cobra/README.md`)

Cobra is both a library for creating powerful modern CLI applications as well as a program to generate applications and command files.

Many of the most widely used Go projects are built using Cobra including:

Overview

Cobra is a library providing a simple interface to create powerful modern CLI
interfaces similar to git & go tools.

Cobra is also an application that will generate your application scaffolding to rapidly
develop a Cobra-based application.

Cobra provides:

Easy subcommand-based CLIs: app server, app fetch, etc.
Fully POSIX-compliant flags (including short & long versions)
Nested subcommands
Global, local and cascading flags
Easy generation of applications & commands with cobra create appname & cobra add cmdname
Intelligent suggestions (app srver... did you mean app server?)
Automatic help generation for commands and flags
Automatic detailed help for app help [command]
Automatic help flag recognition of -h, --help, etc.
Automatically generated bash autocomplete for your application
Automatically generated man pages for your application
Command aliases so you can change things without breaking them
The flexibilty to define your own help, usage, etc.
Optional tight integration with viper for 12-factor apps

Cobra has an exceptionally clean interface and simple design without needless
constructors or initialization methods.

Applications built with Cobra commands are designed to be as user-friendly as
possible. Flags can be placed before or after the command (as long as a
confusing space isn’t provided). Both short and long flags can be used. A
command need not even be fully typed. Help is automatically generated and
available for the application or for a specific command using either the help
command or the --help flag.

Concepts

Cobra is built on a structure of commands, arguments & flags.

Commands represent actions, Args are things and Flags are modifiers for those actions.

The best applications will read like sentences when used. Users will know how
to use the application because they will natively understand how to use it.

The pattern to follow is
APPNAME VERB NOUN --ADJECTIVE.
or
APPNAME COMMAND ARG --FLAG

A few good real world examples may better illustrate this point.

In the following example, 'server' is a command, and 'port' is a flag:

text

> hugo server --port=1313

In this command we are telling Git to clone the url bare.

text

> git clone URL --bare

Commands

Command is the central point of the application. Each interaction that
the application supports will be contained in a Command. A command can
have children commands and optionally run an action.

In the example above, 'server' is the command.

A Command has the following structure:

type Command struct {
    Use string // The one-line usage message.
    Short string // The short description shown in the 'help' output.
    Long string // The long message shown in the 'help <this-command>' output.
    Run func(cmd *Command, args []string) // Run runs the command.
}

Flags

A Flag is a way to modify the behavior of a command. Cobra supports
fully POSIX-compliant flags as well as the Go flag package.
A Cobra command can define flags that persist through to children commands
and flags that are only available to that command.

In the example above, 'port' is the flag.

Flag functionality is provided by the pflag
library, a fork of the flag standard library
which maintains the same interface while adding POSIX compliance.

Usage

Cobra works by creating a set of commands and then organizing them into a tree.
The tree defines the structure of the application.

Once each command is defined with its corresponding flags, then the
tree is assigned to the commander which is finally executed.

Installing

Using Cobra is easy. First, use go get to install the latest version
of the library. This command will install the cobra generator executible
along with the library:

text

> go get -v github.com/spf13/cobra/cobra

Next, include Cobra in your application:

import "github.com/spf13/cobra"

Getting Started

While you are welcome to provide your own organization, typically a Cobra based
application will follow the following organizational structure.

text

▾ appName/
    ▾ cmd/
        add.go
        your.go
        commands.go
        here.go
      main.go

In a Cobra app, typically the main.go file is very bare. It serves, one purpose, to initialize Cobra.

package main

import "{pathToYourApp}/cmd"

func main() {
	if err := cmd.RootCmd.Execute(); err != nil {
		fmt.Println(err)
		os.Exit(-1)
	}
}

Using the Cobra Generator

Cobra provides its own program that will create your application and add any
commands you want. It's the easiest way to incorporate Cobra into your application.

cobra init

The cobra init [yourApp] command will create your initial application code
for you. It is a very powerful application that will populate your program with
the right structure so you can immediately enjoy all the benefits of Cobra. It
will also automatically apply the license you specify to your application.

Cobra init is pretty smart. You can provide it a full path, or simply a path
similar to what is expected in the import.

text

cobra init github.com/spf13/newAppName

cobra add

Once an application is initialized Cobra can create additional commands for you.
Let's say you created an app and you wanted the following commands for it:

app serve
app config
app config create

In your project directory (where your main.go file is) you would run the following:

text

cobra add serve
cobra add config
cobra add create -p 'configCmd'

Once you have run these three commands you would have an app structure that would look like:

text

▾ app/
    ▾ cmd/
        serve.go
        config.go
        create.go
      main.go

at this point you can run go run main.go and it would run your app. go run main.go serve, go run main.go config, go run main.go config create along
with go run main.go help serve, etc would all work.

Obviously you haven't added your own code to these yet, the commands are ready
for you to give them their tasks. Have fun.

Configuring the cobra generator

The cobra generator will be easier to use if you provide a simple configuration
file which will help you eliminate providing a bunch of repeated information in
flags over and over.

An example ~/.cobra.yaml file:

yaml

author: Steve Francia <[email protected]>
license: MIT

You can specify no license by setting license to none or you can specify
a custom license:

yaml

license:
  header: This file is part of {{ .appName }}.
  text: |
    {{ .copyright }}

    This is my license. There are many like it, but this one is mine.
    My license is my best friend. It is my life. I must master it as I must
    master my life.

Manually implementing Cobra

To manually implement cobra you need to create a bare main.go file and a RootCmd file.
You will optionally provide additional commands as you see fit.

Create the root command

The root command represents your binary itself.

Manually create rootCmd

Cobra doesn't require any special constructors. Simply create your commands.

Ideally you place this in app/cmd/root.go:

var RootCmd = &cobra.Command{
	Use:   "hugo",
	Short: "Hugo is a very fast static site generator",
	Long: `A Fast and Flexible Static Site Generator built with
                love by spf13 and friends in Go.
                Complete documentation is available at http://hugo.spf13.com`,
	Run: func(cmd *cobra.Command, args []string) {
		// Do Stuff Here
	},
}

You will additionally define flags and handle configuration in your init() function.

for example cmd/root.go:

func init() {
	cobra.OnInitialize(initConfig)
	RootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", "config file (default is $HOME/.cobra.yaml)")
	RootCmd.PersistentFlags().StringVarP(&projectBase, "projectbase", "b", "", "base project directory eg. github.com/spf13/")
	RootCmd.PersistentFlags().StringP("author", "a", "YOUR NAME", "Author name for copyright attribution")
	RootCmd.PersistentFlags().StringVarP(&userLicense, "license", "l", "", "Name of license for the project (can provide `licensetext` in config)")
	RootCmd.PersistentFlags().Bool("viper", true, "Use Viper for configuration")
	viper.BindPFlag("author", RootCmd.PersistentFlags().Lookup("author"))
	viper.BindPFlag("projectbase", RootCmd.PersistentFlags().Lookup("projectbase"))
	viper.BindPFlag("useViper", RootCmd.PersistentFlags().Lookup("viper"))
	viper.SetDefault("author", "NAME HERE <EMAIL ADDRESS>")
	viper.SetDefault("license", "apache")
}

Create your main.go

With the root command you need to have your main function execute it.
Execute should be run on the root for clarity, though it can be called on any command.

In a Cobra app, typically the main.go file is very bare. It serves, one purpose, to initialize Cobra.

package main

import "{pathToYourApp}/cmd"

func main() {
	if err := cmd.RootCmd.Execute(); err != nil {
		fmt.Println(err)
		os.Exit(-1)
	}
}

Create additional commands

Additional commands can be defined and typically are each given their own file
inside of the cmd/ directory.

If you wanted to create a version command you would create cmd/version.go and
populate it with the following:

package cmd

import (
	"github.com/spf13/cobra"
)

func init() {
	RootCmd.AddCommand(versionCmd)
}

var versionCmd = &cobra.Command{
	Use:   "version",
	Short: "Print the version number of Hugo",
	Long:  `All software has versions. This is Hugo's`,
	Run: func(cmd *cobra.Command, args []string) {
		fmt.Println("Hugo Static Site Generator v0.9 -- HEAD")
	},
}

Attach command to its parent

If you notice in the above example we attach the command to its parent. In
this case the parent is the rootCmd. In this example we are attaching it to the
root, but commands can be attached at any level.

RootCmd.AddCommand(versionCmd)

Remove a command from its parent

Removing a command is not a common action in simple programs, but it allows 3rd
parties to customize an existing command tree.

In this example, we remove the existing VersionCmd command of an existing
root command, and we replace it with our own version:

mainlib.RootCmd.RemoveCommand(mainlib.VersionCmd)
mainlib.RootCmd.AddCommand(versionCmd)

Working with Flags

Flags provide modifiers to control how the action command operates.

Assign flags to a command

Since the flags are defined and used in different locations, we need to
define a variable outside with the correct scope to assign the flag to
work with.

var Verbose bool
var Source string

There are two different approaches to assign a flag.

Persistent Flags

A flag can be 'persistent' meaning that this flag will be available to the
command it's assigned to as well as every command under that command. For
global flags, assign a flag as a persistent flag on the root.

RootCmd.PersistentFlags().BoolVarP(&Verbose, "verbose", "v", false, "verbose output")

Local Flags

A flag can also be assigned locally which will only apply to that specific command.

RootCmd.Flags().StringVarP(&Source, "source", "s", "", "Source directory to read from")

Positional Arguments

Validation of positional arguments can be specified using the Args field, which accepts
one of the following values:

NoArgs - the command will report an error if there are any positional args.
ArbitraryArgs - the command will accept any args.
OnlyValidArgs - the command will report an error if there are any positiona
args that are not in the ValidArgs list.
MinimumNArgs(int) - the command will report an error if there are not at
least N positional args.
MaximumNArgs(int) - the command will report an error if there are more than
N positional args.
ExactArgs(int) - the command will report an error if there are not
exactly N positional args.
RangeArgs(min, max) - the command will report an error if the number of args
is not between the minimum and maximum number of expected args.

By default, Args uses the following legacy behaviour:

root commands with no subcommands can take arbitrary arguments
root commands with subcommands will do subcommand validity checking
subcommands will always accept arbitrary arguments and do no subsubcommand validity checking

var HugoCmd = &cobra.Command{
    Use:   "hugo",
    Short: "Hugo is a very fast static site generator",
    ValidArgs: []string{"one", "two"}
    Args: cobra.OnlyValidArgs
    Run: func(cmd *cobra.Command, args []string) {
        // args will only have the values one, two
        // or the cmd.Execute() will fail.
    },
}

Example

In the example below, we have defined three commands. Two are at the top level
and one (cmdTimes) is a child of one of the top commands. In this case the root
is not executable meaning that a subcommand is required. This is accomplished
by not providing a 'Run' for the 'rootCmd'.

We have only defined one flag for a single command.

More documentation about flags is available at https://github.com/spf13/pflag

package main

import (
	"fmt"
	"strings"

	"github.com/spf13/cobra"
)

func main() {

	var echoTimes int

	var cmdPrint = &cobra.Command{
		Use:   "print [string to print]",
		Short: "Print anything to the screen",
		Long: `print is for printing anything back to the screen.
            For many years people have printed back to the screen.
            `,
		Run: func(cmd *cobra.Command, args []string) {
			fmt.Println("Print: " + strings.Join(args, " "))
		},
	}

	var cmdEcho = &cobra.Command{
		Use:   "echo [string to echo]",
		Short: "Echo anything to the screen",
		Long: `echo is for echoing anything back.
            Echo works a lot like print, except it has a child command.
            `,
		Run: func(cmd *cobra.Command, args []string) {
			fmt.Println("Print: " + strings.Join(args, " "))
		},
	}

	var cmdTimes = &cobra.Command{
		Use:   "times [# times] [string to echo]",
		Short: "Echo anything to the screen more times",
		Long: `echo things multiple times back to the user by providing
            a count and a string.`,
		Run: func(cmd *cobra.Command, args []string) {
			for i := 0; i < echoTimes; i++ {
				fmt.Println("Echo: " + strings.Join(args, " "))
			}
		},
	}

	cmdTimes.Flags().IntVarP(&echoTimes, "times", "t", 1, "times to echo the input")

	var rootCmd = &cobra.Command{Use: "app"}
	rootCmd.AddCommand(cmdPrint, cmdEcho)
	cmdEcho.AddCommand(cmdTimes)
	rootCmd.Execute()
}

For a more complete example of a larger application, please checkout Hugo.

The Help Command

Cobra automatically adds a help command to your application when you have subcommands.
This will be called when a user runs 'app help'. Additionally, help will also
support all other commands as input. Say, for instance, you have a command called
'create' without any additional configuration; Cobra will work when 'app help
create' is called. Every command will automatically have the '--help' flag added.

Example

The following output is automatically generated by Cobra. Nothing beyond the
command and flag definitions are needed.

text

> hugo help

hugo is the main command, used to build your Hugo site.

Hugo is a Fast and Flexible Static Site Generator
built with love by spf13 and friends in Go.

Complete documentation is available at http://gohugo.io/.

Usage:
  hugo [flags]
  hugo [command]

Available Commands:
  server          Hugo runs its own webserver to render the files
  version         Print the version number of Hugo
  config          Print the site configuration
  check           Check content in the source directory
  benchmark       Benchmark hugo by building a site a number of times.
  convert         Convert your content to different formats
  new             Create new content for your site
  list            Listing out various types of content
  undraft         Undraft changes the content's draft status from 'True' to 'False'
  genautocomplete Generate shell autocompletion script for Hugo
  gendoc          Generate Markdown documentation for the Hugo CLI.
  genman          Generate man page for Hugo
  import          Import your site from others.

Flags:
  -b, --baseURL="": hostname (and path) to the root, e.g. http://spf13.com/
  -D, --buildDrafts[=false]: include content marked as draft
  -F, --buildFuture[=false]: include content with publishdate in the future
      --cacheDir="": filesystem path to cache directory. Defaults: $TMPDIR/hugo_cache/
      --canonifyURLs[=false]: if true, all relative URLs will be canonicalized using baseURL
      --config="": config file (default is path/config.yaml|json|toml)
  -d, --destination="": filesystem path to write files to
      --disableRSS[=false]: Do not build RSS files
      --disableSitemap[=false]: Do not build Sitemap file
      --editor="": edit new content with this editor, if provided
      --ignoreCache[=false]: Ignores the cache directory for reading but still writes to it
      --log[=false]: Enable Logging
      --logFile="": Log File path (if set, logging enabled automatically)
      --noTimes[=false]: Don't sync modification time of files
      --pluralizeListTitles[=true]: Pluralize titles in lists using inflect
      --preserveTaxonomyNames[=false]: Preserve taxonomy names as written ("Gérard Depardieu" vs "gerard-depardieu")
  -s, --source="": filesystem path to read files relative from
      --stepAnalysis[=false]: display memory and timing of different steps of the program
  -t, --theme="": theme to use (located in /themes/THEMENAME/)
      --uglyURLs[=false]: if true, use /filename.html instead of /filename/
  -v, --verbose[=false]: verbose output
      --verboseLog[=false]: verbose logging
  -w, --watch[=false]: watch filesystem for changes and recreate as needed

Use "hugo [command] --help" for more information about a command.

Help is just a command like any other. There is no special logic or behavior
around it. In fact, you can provide your own if you want.

Defining your own help

You can provide your own Help command or your own template for the default command to use.

The default help command is

func (c *Command) initHelp() {
	if c.helpCommand == nil {
		c.helpCommand = &Command{
			Use:   "help [command]",
			Short: "Help about any command",
			Long: `Help provides help for any command in the application.
        Simply type ` + c.Name() + ` help [path to command] for full details.`,
			Run: c.HelpFunc(),
		}
	}
	c.AddCommand(c.helpCommand)
}

You can provide your own command, function or template through the following methods:

command.SetHelpCommand(cmd *Command)

command.SetHelpFunc(f func(*Command, []string))

command.SetHelpTemplate(s string)

The latter two will also apply to any children commands.

Usage

When the user provides an invalid flag or invalid command, Cobra responds by
showing the user the 'usage'.

Example

You may recognize this from the help above. That's because the default help
embeds the usage as part of its output.

text

Usage:
  hugo [flags]
  hugo [command]

Available Commands:
  server          Hugo runs its own webserver to render the files
  version         Print the version number of Hugo
  config          Print the site configuration
  check           Check content in the source directory
  benchmark       Benchmark hugo by building a site a number of times.
  convert         Convert your content to different formats
  new             Create new content for your site
  list            Listing out various types of content
  undraft         Undraft changes the content's draft status from 'True' to 'False'
  genautocomplete Generate shell autocompletion script for Hugo
  gendoc          Generate Markdown documentation for the Hugo CLI.
  genman          Generate man page for Hugo
  import          Import your site from others.

Flags:
  -b, --baseURL="": hostname (and path) to the root, e.g. http://spf13.com/
  -D, --buildDrafts[=false]: include content marked as draft
  -F, --buildFuture[=false]: include content with publishdate in the future
      --cacheDir="": filesystem path to cache directory. Defaults: $TMPDIR/hugo_cache/
      --canonifyURLs[=false]: if true, all relative URLs will be canonicalized using baseURL
      --config="": config file (default is path/config.yaml|json|toml)
  -d, --destination="": filesystem path to write files to
      --disableRSS[=false]: Do not build RSS files
      --disableSitemap[=false]: Do not build Sitemap file
      --editor="": edit new content with this editor, if provided
      --ignoreCache[=false]: Ignores the cache directory for reading but still writes to it
      --log[=false]: Enable Logging
      --logFile="": Log File path (if set, logging enabled automatically)
      --noTimes[=false]: Don't sync modification time of files
      --pluralizeListTitles[=true]: Pluralize titles in lists using inflect
      --preserveTaxonomyNames[=false]: Preserve taxonomy names as written ("Gérard Depardieu" vs "gerard-depardieu")
  -s, --source="": filesystem path to read files relative from
      --stepAnalysis[=false]: display memory and timing of different steps of the program
  -t, --theme="": theme to use (located in /themes/THEMENAME/)
      --uglyURLs[=false]: if true, use /filename.html instead of /filename/
  -v, --verbose[=false]: verbose output
      --verboseLog[=false]: verbose logging
  -w, --watch[=false]: watch filesystem for changes and recreate as needed

Defining your own usage

You can provide your own usage function or template for Cobra to use.

The default usage function is:

return func(c *Command) error {
	err := tmpl(c.Out(), c.UsageTemplate(), c)
	return err
}

Like help, the function and template are overridable through public methods:

command.SetUsageFunc(f func(*Command) error)

command.SetUsageTemplate(s string)

PreRun or PostRun Hooks

It is possible to run functions before or after the main Run function of your command. The PersistentPreRun and PreRun functions will be executed before Run. PersistentPostRun and PostRun will be executed after Run. The Persistent*Run functions will be inherrited by children if they do not declare their own. These function are run in the following order:

PersistentPreRun
PreRun
Run
PostRun
PersistentPostRun

An example of two commands which use all of these features is below. When the subcommand is executed, it will run the root command's PersistentPreRun but not the root command's PersistentPostRun:

text

/* Detailed source-code truncated for AI context efficiency. */

Alternative Error Handling

Cobra also has functions where the return signature is an error. This allows for errors to bubble up to the top,
providing a way to handle the errors in one location. The current list of functions that return an error is:

PersistentPreRunE
PreRunE
RunE
PostRunE
PersistentPostRunE

If you would like to silence the default error and usage output in favor of your own, you can set SilenceUsage
and SilenceErrors to false on the command. A child command respects these flags if they are set on the parent
command.

Example Usage using RunE:

package main

import (
	"errors"
	"log"

	"github.com/spf13/cobra"
)

func main() {
	var rootCmd = &cobra.Command{
		Use:   "hugo",
		Short: "Hugo is a very fast static site generator",
		Long: `A Fast and Flexible Static Site Generator built with
                love by spf13 and friends in Go.
                Complete documentation is available at http://hugo.spf13.com`,
		RunE: func(cmd *cobra.Command, args []string) error {
			// Do Stuff Here
			return errors.New("some random error")
		},
	}

	if err := rootCmd.Execute(); err != nil {
		log.Fatal(err)
	}
}

Suggestions when "unknown command" happens

Cobra will print automatic suggestions when "unknown command" errors happen. This allows Cobra to behave similarly to the git command when a typo happens. For example:

text

$ hugo srever
Error: unknown command "srever" for "hugo"

Did you mean this?
        server

Run 'hugo --help' for usage.

Suggestions are automatic based on every subcommand registered and use an implementation of Levenshtein distance. Every registered command that matches a minimum distance of 2 (ignoring case) will be displayed as a suggestion.

If you need to disable suggestions or tweak the string distance in your command, use:

command.DisableSuggestions = true

command.SuggestionsMinimumDistance = 1

You can also explicitly set names for which a given command will be suggested using the SuggestFor attribute. This allows suggestions for strings that are not close in terms of string distance, but makes sense in your set of commands and for some which you don't want aliases. Example:

text

$ kubectl remove
Error: unknown command "remove" for "kubectl"

Did you mean this?
        delete

Run 'kubectl help' for usage.

Generating Markdown-formatted documentation for your command

Cobra can generate a Markdown-formatted document based on the subcommands, flags, etc. A simple example of how to do this for your command can be found in Markdown Docs.

Generating man pages for your command

Cobra can generate a man page based on the subcommands, flags, etc. A simple example of how to do this for your command can be found in Man Docs.

Generating bash completions for your command

Cobra can generate a bash-completion file. If you add more information to your command, these completions can be amazingly powerful and flexible. Read more about it in Bash Completions.

Debugging

Cobra provides a ‘DebugFlags’ method on a command which, when called, will print
out everything Cobra knows about the flags for each command.

Example

command.DebugFlags()

Release Notes

0.9.0 June 17, 2014
- flags can appears anywhere in the args (provided they are unambiguous)
- --help prints usage screen for app or command
- Prefix matching for commands
- Cleaner looking help and usage output
- Extensive test suite
0.8.0 Nov 5, 2013
- Reworked interface to remove commander completely
- Command now primary structure
- No initialization needed
- Usage & Help templates & functions definable at any level
- Updated Readme
0.7.0 Sept 24, 2013
- Needs more eyes
- Test suite
- Support for automatic error messages
- Support for help command
- Support for printing to any io.Writer instead of os.Stderr
- Support for persistent flags which cascade down tree
- Ready for integration into Hugo
0.1.0 Sept 3, 2013
- Implement first draft

Extensions

Libraries for extending Cobra:

cmdns: Enables name spacing a command's immediate children. It provides an alternative way to structure subcommands, similar to heroku apps:create and ovrclk clusters:launch.

ToDo

Launch proper documentation site

Contributing

Fork it
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create new Pull Request

Contributors

Names in no particular order:

spf13,
eparis,
bep, and many more!

License

Cobra is released under the Apache 2.0 license. See LICENSE.txt

Overview (`vendor/src/github.com/spf13/pflag/README.md`)

Description

pflag is a drop-in replacement for Go's flag package, implementing
POSIX/GNU-style --flags.

pflag is compatible with the GNU extensions to the POSIX recommendations
for command-line options. For a more precise description, see the
"Command-line flag syntax" section below.

pflag is available under the same style of BSD license as the Go language,
which can be found in the LICENSE file.

Installation

pflag is available using the standard go get command.

Install by running:

text

go get github.com/spf13/pflag

Run tests by running:

text

go test github.com/spf13/pflag

Usage

pflag is a drop-in replacement of Go's native flag package. If you import
pflag under the name "flag" then all code should continue to function
with no changes.

import flag "github.com/spf13/pflag"

There is one exception to this: if you directly instantiate the Flag struct
there is one more field "Shorthand" that you will need to set.
Most code never instantiates this struct directly, and instead uses
functions such as String(), BoolVar(), and Var(), and is therefore
unaffected.

Define flags using flag.String(), Bool(), Int(), etc.

This declares an integer flag, -flagname, stored in the pointer ip, with type *int.

var ip *int = flag.Int("flagname", 1234, "help message for flagname")

If you like, you can bind the flag to a variable using the Var() functions.

var flagvar int
func init() {
    flag.IntVar(&flagvar, "flagname", 1234, "help message for flagname")
}

Or you can create custom flags that satisfy the Value interface (with
pointer receivers) and couple them to flag parsing by

flag.Var(&flagVal, "name", "help message for flagname")

For such flags, the default value is just the initial value of the variable.

After all flags are defined, call

flag.Parse()

to parse the command line into the defined flags.

Flags may then be used directly. If you're using the flags themselves,
they are all pointers; if you bind to variables, they're values.

fmt.Println("ip has value ", *ip)
fmt.Println("flagvar has value ", flagvar)

There are helpers function to get values later if you have the FlagSet but
it was difficult to keep up with all of the flag pointers in your code.
If you have a pflag.FlagSet with a flag called 'flagname' of type int you
can use GetInt() to get the int value. But notice that 'flagname' must exist
and it must be an int. GetString("flagname") will fail.

i, err := flagset.GetInt("flagname")

After parsing, the arguments after the flag are available as the
slice flag.Args() or individually as flag.Arg(i).
The arguments are indexed from 0 through flag.NArg()-1.

The pflag package also defines some new functions that are not in flag,
that give one-letter shorthands for flags. You can use these by appending
'P' to the name of any function that defines a flag.

var ip = flag.IntP("flagname", "f", 1234, "help message")
var flagvar bool
func init() {
    flag.BoolVarP("boolname", "b", true, "help message")
}
flag.VarP(&flagVar, "varname", "v", 1234, "help message")

Shorthand letters can be used with single dashes on the command line.
Boolean shorthand flags can be combined with other shorthand flags.

The default set of command-line flags is controlled by
top-level functions. The FlagSet type allows one to define
independent sets of flags, such as to implement subcommands
in a command-line interface. The methods of FlagSet are
analogous to the top-level functions for the command-line
flag set.

Setting no option default values for flags

After you create a flag it is possible to set the pflag.NoOptDefVal for
the given flag. Doing this changes the meaning of the flag slightly. If
a flag has a NoOptDefVal and the flag is set on the command line without
an option the flag will be set to the NoOptDefVal. For example given:

var ip = flag.IntP("flagname", "f", 1234, "help message")
flag.Lookup("flagname").NoOptDefVal = "4321"

Would result in something like

Parsed Arguments	Resulting Value
--flagname=1357	ip=1357
--flagname	ip=4321
[nothing]	ip=1234

Command line flag syntax

text

--flag    // boolean flags, or flags with no option default values
--flag x  // only on flags without a default value
--flag=x

Unlike the flag package, a single dash before an option means something
different than a double dash. Single dashes signify a series of shorthand
letters for flags. All but the last shorthand letter must be boolean flags
or a flag with a default value

text

// boolean or flags where the 'no option default value' is set
-f
-f=true
-abc
but
-b true is INVALID

// non-boolean and flags without a 'no option default value'
-n 1234
-n=1234
-n1234

// mixed
-abcs "hello"
-absd="hello"
-abcs1234

Flag parsing stops after the terminator "--". Unlike the flag package,
flags can be interspersed with arguments anywhere on the command line
before this terminator.

Integer flags accept 1234, 0664, 0x1234 and may be negative.
Boolean flags (in their long form) accept 1, 0, t, f, true, false,
TRUE, FALSE, True, False.
Duration flags accept any input valid for time.ParseDuration.

Mutating or "Normalizing" Flag names

It is possible to set a custom flag name 'normalization function.' It allows flag names to be mutated both when created in the code and when used on the command line to some 'normalized' form. The 'normalized' form is used for comparison. Two examples of using the custom normalization func follow.

Example #1: You want -, _, and . in flags to compare the same. aka --my-flag == --my_flag == --my.flag

func wordSepNormalizeFunc(f *pflag.FlagSet, name string) pflag.NormalizedName {
	from := []string{"-", "_"}
	to := "."
	for _, sep := range from {
		name = strings.Replace(name, sep, to, -1)
	}
	return pflag.NormalizedName(name)
}

myFlagSet.SetNormalizeFunc(wordSepNormalizeFunc)

Example #2: You want to alias two flags. aka --old-flag-name == --new-flag-name

func aliasNormalizeFunc(f *pflag.FlagSet, name string) pflag.NormalizedName {
	switch name {
	case "old-flag-name":
		name = "new-flag-name"
		break
	}
	return pflag.NormalizedName(name)
}

myFlagSet.SetNormalizeFunc(aliasNormalizeFunc)

Deprecating a flag or its shorthand

It is possible to deprecate a flag, or just its shorthand. Deprecating a flag/shorthand hides it from help text and prints a usage message when the deprecated flag/shorthand is used.

Example #1: You want to deprecate a flag named "badflag" as well as inform the users what flag they should use instead.

// deprecate a flag by specifying its name and a usage message
flags.MarkDeprecated("badflag", "please use --good-flag instead")

This hides "badflag" from help text, and prints Flag --badflag has been deprecated, please use --good-flag instead when "badflag" is used.

Example #2: You want to keep a flag name "noshorthandflag" but deprecate its shortname "n".

// deprecate a flag shorthand by specifying its flag name and a usage message
flags.MarkShorthandDeprecated("noshorthandflag", "please use --noshorthandflag only")

This hides the shortname "n" from help text, and prints Flag shorthand -n has been deprecated, please use --noshorthandflag only when the shorthand "n" is used.

Note that usage message is essential here, and it should not be empty.

Hidden flags

It is possible to mark a flag as hidden, meaning it will still function as normal, however will not show up in usage/help text.

Example: You have a flag named "secretFlag" that you need for internal use only and don't want it showing up in help text, or for its usage text to be available.

// hide a flag by specifying its name
flags.MarkHidden("secretFlag")

More info

You can see the full reference documentation of the pflag package
at godoc.org, or through go's standard documentation system by
running godoc -http=:6060 and browsing to
http://localhost:6060/pkg/github.com/ogier/pflag after
installation.

Overview (`vendor/src/github.com/vdemeester/shakers/README.md`)

Shakers

🐹 + 🐙 = 😽

A collection of go-check Checkers to ease the use of it.

Building and testing it

You need either docker, or go
and glide in order to build and test shakers.

Using Docker and Makefile

You need to run the test-unit target.

bash

$ make test-unit
docker build -t "shakers-dev:master" .
# […]
docker run --rm -it   "shakers-dev:master" ./script/make.sh test-unit
---> Making bundle: test-unit (in .)
+ go test -cover -coverprofile=cover.out .
ok      github.com/vdemeester/shakers   0.015s  coverage: 96.0% of statements

Test success

Using glide and `GO15VENDOREXPERIMENT`

Get the dependencies with glide up (or use go get but you have no garantuees over the version of the dependencies)
If you're using glide (and not standard go get) export GO15VENDOREXPERIMENT with export GO15VENDOREXPERIMENT=1
Run tests with go test .

Overview (`vendor/src/github.com/vishvananda/netlink/README.md`)

netlink - netlink library for go

The netlink package provides a simple netlink library for go. Netlink
is the interface a user-space program in linux uses to communicate with
the kernel. It can be used to add and remove interfaces, set ip addresses
and routes, and configure ipsec. Netlink communication requires elevated
privileges, so in most cases this code needs to be run as root. Since
low-level netlink messages are inscrutable at best, the library attempts
to provide an api that is loosely modeled on the CLI provied by iproute2.
Actions like ip link add will be accomplished via a similarly named
function like AddLink(). This library began its life as a fork of the
netlink functionality in
docker/libcontainer but was
heavily rewritten to improve testability, performance, and to add new
functionality like ipsec xfrm handling.

Local Build and Test

You can use go get command:

text

go get github.com/vishvananda/netlink

Testing dependencies:

text

go get github.com/vishvananda/netns

Testing (requires root):

text

sudo -E go test github.com/vishvananda/netlink

Examples

Add a new bridge and add eth1 into it:

package main

import (
    "net"
    "github.com/vishvananda/netlink"
)

func main() {
    la := netlink.NewLinkAttrs()
    la.Name = "foo"
    mybridge := &netlink.Bridge{la}}
    _ := netlink.LinkAdd(mybridge)
    eth1, _ := netlink.LinkByName("eth1")
    netlink.LinkSetMaster(eth1, mybridge)
}

Note NewLinkAttrs constructor, it sets default values in structure. For now
it sets only TxQLen to -1, so kernel will set default by itself. If you're
using simple initialization(LinkAttrs{Name: "foo"}) TxQLen will be set to
0 unless you specify it like LinkAttrs{Name: "foo", TxQLen: 1000}.

Add a new ip address to loopback:

package main

import (
    "net"
    "github.com/vishvananda/netlink"
)

func main() {
    lo, _ := netlink.LinkByName("lo")
    addr, _ := netlink.ParseAddr("169.254.169.254/32")
    netlink.AddrAdd(lo, addr)
}

Future Work

Many pieces of netlink are not yet fully supported in the high-level
interface. Aspects of virtually all of the high-level objects don't exist.
Many of the underlying primitives are there, so its a matter of putting
the right fields into the high-level objects and making sure that they
are serialized and deserialized correctly in the Add and List methods.

There are also a few pieces of low level netlink functionality that still
need to be implemented. Routing rules are not in place and some of the
more advanced link types. Hopefully there is decent structure and testing
in place to make these fairly straightforward to add.

Overview (`vendor/src/github.com/vishvananda/netns/README.md`)

netns - network namespaces in go

The netns package provides an ultra-simple interface for handling
network namespaces in go. Changing namespaces requires elevated
privileges, so in most cases this code needs to be run as root.

Local Build and Test

You can use go get command:

text

go get github.com/vishvananda/netns

Testing (requires root):

text

sudo -E go test github.com/vishvananda/netns

Example

package main

import (
    "net"
    "runtime"
    "github.com/vishvananada/netns"
)

func main() {
    // Lock the OS Thread so we don't accidentally switch namespaces
    runtime.LockOSThread()
    defer runtime.UnlockOSThread()

    // Save the current network namespace
    origns, _ := netns.Get()
    defer origns.Close()

    // Create a new network namespace
    newns, _ := netns.New()
    defer newns.Close()

    // Do something with tne network namespace
    ifaces, _ := net.Interfaces()
    fmt.Printf("Interfaces: %v\n", ifaces)

    // Switch back to the original namespace
    netns.Set(origns)
}

Overview (`vendor/src/golang.org/x/oauth2/README.md`)

OAuth2 for Go

oauth2 package contains a client implementation for OAuth 2.0 spec.

Installation

text

go get golang.org/x/oauth2

See godoc for further documentation and examples.

App Engine

In change 96e89be (March 2015) we removed the oauth2.Context2 type in favor
of the context.Context type from
the golang.org/x/net/context package

This means its no longer possible to use the "Classic App Engine"
appengine.Context type with the oauth2 package. (You're using
Classic App Engine if you import the package "appengine".)

To work around this, you may use the new "google.golang.org/appengine"
package. This package has almost the same API as the "appengine" package,
but it can be fetched with go get and used on "Managed VMs" and well as
Classic App Engine.

See the new appengine package's readme
for information on updating your app.

If you don't want to update your entire app to use the new App Engine packages,
you may use both sets of packages in parallel, using only the new packages
with the oauth2 package.

text

import (
	"golang.org/x/net/context"
	"golang.org/x/oauth2"
	"golang.org/x/oauth2/google"
	newappengine "google.golang.org/appengine"
	newurlfetch "google.golang.org/appengine/urlfetch"

	"appengine"
)

func handler(w http.ResponseWriter, r *http.Request) {
	var c appengine.Context = appengine.NewContext(r)
	c.Infof("Logging a message with the old package")

	var ctx context.Context = newappengine.NewContext(r)
	client := &http.Client{
		Transport: &oauth2.Transport{
			Source: google.AppEngineTokenSource(ctx, "scope"),
			Base:   &newurlfetch.Transport{Context: ctx},
		},
	}
	client.Get("...")
}

Overview (`vendor/src/github.com/cloudflare/cfssl/certdb/README.md`)

certdb usage

Using a database enables additional functionality for existing commands when a
db config is provided:

sign and gencert add a certificate to the certdb after signing it
serve enables database functionality for the sign and revoke endpoints

A database is required for the following:

revoke marks certificates revoked in the database with an optional reason
ocsprefresh refreshes the table of cached OCSP responses
ocspdump outputs cached OCSP responses in a concatenated base64-encoded format

Setup/Migration

This directory stores goose db migration scripts for various DB backends.
Currently supported:

SQLite in sqlite
PostgreSQL in pg

Get goose

text

go get https://bitbucket.org/liamstask/goose/

Use goose to start and terminate a SQLite DB

To start a SQLite DB using goose:

text

goose -path $GOPATH/src/github.com/cloudflare/cfssl/certdb/sqlite up'

To tear down a SQLite DB using goose

text

goose -path $GOPATH/src/github.com/cloudflare/cfssl/certdb/sqlite down

Use goose to start and terminate a PostgreSQL DB

To start a PostgreSQL using goose:

text

goose -path $GOPATH/src/github.com/cloudflare/cfssl/certdb/pg up

To tear down a PostgreSQL DB using goose

text

goose -path $GOPATH/src/github.com/cloudflare/cfssl/certdb/pg down

Note: the administration of PostgreSQL DB is not included. We assume
the databases being connected to are already created and access control
are properly handled.

CFSSL Configuration

Several cfssl commands take a -db-config flag. Create a file with a
JSON dictionary:

text

{"driver":"sqlite3","data_source":"certs.db"}

text

{"driver":"postgres","data_source":"postgres://user:password@host/db"}

Overview (`vendor/src/github.com/coreos/etcd/client/README.md`)

etcd/client

etcd/client is the Go client library for etcd.

Install

bash

go get github.com/coreos/etcd/client

Usage

package main

import (
	"log"
	"time"

	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
	"github.com/coreos/etcd/client"
)

func main() {
	cfg := client.Config{
		Endpoints:               []string{"http://127.0.0.1:2379"},
		Transport:               client.DefaultTransport,
		// set timeout per request to fail fast when the target endpoint is unavailable
		HeaderTimeoutPerRequest: time.Second,
	}
	c, err := client.New(cfg)
	if err != nil {
		log.Fatal(err)
	}
	kapi := client.NewKeysAPI(c)
	// set "/foo" key with "bar" value
	log.Print("Setting '/foo' key with 'bar' value")
	resp, err := kapi.Set(context.Background(), "/foo", "bar", nil)
	if err != nil {
		log.Fatal(err)
	} else {
		// print common key info
		log.Printf("Set is done. Metadata is %q\n", resp)
	}
	// get "/foo" key's value
	log.Print("Getting '/foo' key value")
	resp, err = kapi.Get(context.Background(), "/foo", nil)
	if err != nil {
		log.Fatal(err)
	} else {
		// print common key info
		log.Printf("Get is done. Metadata is %q\n", resp)
		// print value
		log.Printf("%q key has %q value\n", resp.Node.Key, resp.Node.Value)
	}
}

Error Handling

etcd client might return three types of errors.

context error

Each API call has its first parameter as context. A context can be canceled or have an attached deadline. If the context is canceled or reaches its deadline, the responding context error will be returned no matter what internal errors the API call has already encountered.

cluster error

Each API call tries to send request to the cluster endpoints one by one until it successfully gets a response. If a requests to an endpoint fails, due to exceeding per request timeout or connection issues, the error will be added into a list of errors. If all possible endpoints fail, a cluster error that includes all encountered errors will be returned.

response error

If the response gets from the cluster is invalid, a plain string error will be returned. For example, it might be a invalid JSON error.

Here is the example code to handle client errors:

cfg := client.Config{Endpoints: []string{"http://etcd1:2379","http://etcd2:2379","http://etcd3:2379"}}
c, err := client.New(cfg)
if err != nil {
	log.Fatal(err)
}

kapi := client.NewKeysAPI(c)
resp, err := kapi.Set(ctx, "test", "bar", nil)
if err != nil {
	if err == context.Canceled {
		// ctx is canceled by another routine
	} else if err == context.DeadlineExceeded {
		// ctx is attached with a deadline and it exceeded
	} else if cerr, ok := err.(*client.ClusterError); ok {
		// process (cerr.Errors)
	} else {
		// bad cluster endpoints, which are not etcd servers
	}
}

Caveat

etcd/client prefers to use the same endpoint as long as the endpoint continues to work well. This saves socket resources, and improves efficiency for both client and server side. This preference doesn't remove consistency from the data consumed by the client because data replicated to each etcd member has already passed through the consensus process.
etcd/client does round-robin rotation on other available endpoints if the preferred endpoint isn't functioning properly. For example, if the member that etcd/client connects to is hard killed, etcd/client will fail on the first attempt with the killed member, and succeed on the second attempt with another member. If it fails to talk to all available endpoints, it will return all errors happened.
Default etcd/client cannot handle the case that the remote server is SIGSTOPed now. TCP keepalive mechanism doesn't help in this scenario because operating system may still send TCP keep-alive packets. Over time we'd like to improve this functionality, but solving this issue isn't high priority because a real-life case in which a server is stopped, but the connection is kept alive, hasn't been brought to our attention.
etcd/client cannot detect whether the member in use is healthy when doing read requests. If the member is isolated from the cluster, etcd/client may retrieve outdated data. As a workaround, users could monitor experimental /health endpoint for member healthy information. We are improving it at #3265.

Overview (`vendor/src/github.com/coreos/pkg/capnslog/README.md`)

capnslog, the CoreOS logging package

There are far too many logging packages out there, with varying degrees of licenses, far too many features (colorization, all sorts of log frameworks) or are just a pain to use (lack of Fatalln()?).
capnslog provides a simple but consistent logging interface suitable for all kinds of projects.

Design Principles

`package main` is the place where logging gets turned on and routed

A library should not touch log options, only generate log entries. Libraries are silent until main lets them speak.

All log options are runtime-configurable.

Still the job of main to expose these configurations. main may delegate this to, say, a configuration webhook, but does so explicitly.

There is one log object per package. It is registered under its repository and package name.

main activates logging for its repository and any dependency repositories it would also like to have output in its logstream. main also dictates at which level each subpackage logs.

There is one output stream, and it is an `io.Writer` composed with a formatter.

Splitting streams is probably not the job of your program, but rather, your log aggregation framework. If you must split output streams, again, main configures this and you can write a very simple two-output struct that satisfies io.Writer.

Fancy colorful formatting and JSON output are beyond the scope of a basic logging framework -- they're application/log-collector dependant. These are, at best, provided as options, but more likely, provided by your application.

Log objects are an interface

An object knows best how to print itself. Log objects can collect more interesting metadata if they wish, however, because text isn't going away anytime soon, they must all be marshalable to text. The simplest log object is a string, which returns itself. If you wish to do more fancy tricks for printing your log objects, see also JSON output -- introspect and write a formatter which can handle your advanced log interface. Making strings is the only thing guaranteed.

Log levels have specific meanings:

Critical: Unrecoverable. Must fail.
Error: Data has been lost, a request has failed for a bad reason, or a required resource has been lost
Warning: (Hopefully) Temporary conditions that may cause errors, but may work fine. A replica disappearing (that may reconnect) is a warning.
Notice: Normal, but important (uncommon) log information.
Info: Normal, working log information, everything is fine, but helpful notices for auditing or common operations.
Debug: Everything is still fine, but even common operations may be logged, and less helpful but more quantity of notices.
Trace: Anything goes, from logging every function call as part of a common operation, to tracing execution of a query.

Overview (`vendor/src/github.com/docker/libnetwork/resolvconf/README.md`)

Package resolvconf provides utility code to query and update DNS configuration in /etc/resolv.conf

Overview (`vendor/src/github.com/docker/notary/tuf/README.md`)

GOTUF

This is still a work in progress but will shortly be a fully compliant
Go implementation of The Update Framework (TUF).

Where's the CLI

This repository provides a library only. The Notary project
from Docker should be considered the official CLI to be used with this implementation of TUF.

TODOs:

Add Targets to existing repo
Sign metadata files
Refactor TufRepo to take care of signing ~~and verification~~
Ensure consistent capitalization in naming (TUF___ vs Tuf___)
Make caching of metadata files smarter - PR #5
~~Add configuration for CLI commands. Order of configuration priority from most to least: flags, config file, defaults~~ Notary should be the official CLI
Reasses organization of data types. Possibly consolidate a few things into the data package but break up package into a few more distinct files
Comprehensive test cases
Delete files no longer in use
Fix up errors. Some have to be instantiated, others don't, the inconsistency is annoying.
Bump version numbers in meta files (could probably be done better)

Credits

This implementation was originally forked from flynn/go-tuf,
however in attempting to add delegations I found I was making such
significant changes that I could not maintain backwards compatibility
without the code becoming overly convoluted.

Some features such as pluggable verifiers have already been merged upstream to flynn/go-tuf
and we are in discussion with titanous about working to merge the 2 implementations.

This implementation retains the same 3 Clause BSD license present on
the original flynn implementation.

Overview (`vendor/src/github.com/google/certificate-transparency/go/README.md`)

This is the really early beginnings of a certificate transparency log
client written in Go, along with a log scanner tool.

You'll need go v1.1 or higher to compile.

Installation

This go code must be imported into your go workspace before you can
use it, which can be done with:

text

go get github.com/google/certificate-transparency/go/client
go get github.com/google/certificate-transparency/go/scanner
etc.

Building the binaries

To compile the log scanner run:

text

go build github.com/google/certificate-transparency/go/scanner/main/scanner.go

Contributing

When sending pull requests, please ensure that everything's been run
through gofmt beforehand so we can keep everything nice and
tidy.

Overview (`vendor/src/github.com/hashicorp/consul/api/README.md`)

Consul API client

This package provides the api package which attempts to
provide programmatic access to the full Consul API.

Currently, all of the Consul APIs included in version 0.3 are supported.

Documentation

The full documentation is available on Godoc

Usage

Below is an example of using the Consul client:

// Get a new client, with KV endpoints
client, _ := api.NewClient(api.DefaultConfig())
kv := client.KV()

// PUT a new KV pair
p := &api.KVPair{Key: "foo", Value: []byte("test")}
_, err := kv.Put(p, nil)
if err != nil {
    panic(err)
}

// Lookup the pair
pair, _, err := kv.Get("foo", nil)
if err != nil {
    panic(err)
}
fmt.Printf("KV: %v", pair)

Overview (`vendor/src/github.com/hashicorp/go-msgpack/codec/README.md`)

Codec

High Performance and Feature-Rich Idiomatic Go Library providing
encode/decode support for different serialization formats.

Supported Serialization formats are:

msgpack: [https://github.com/msgpack/msgpack]
binc: [http://github.com/ugorji/binc]

To install:

text

go get github.com/ugorji/go/codec

Online documentation: [http://godoc.org/github.com/ugorji/go/codec]

The idiomatic Go support is as seen in other encoding packages in
the standard library (ie json, xml, gob, etc).

Rich Feature Set includes:

Simple but extremely powerful and feature-rich API
Very High Performance.
Our extensive benchmarks show us outperforming Gob, Json and Bson by 2-4X.
This was achieved by taking extreme care on:
- managing allocation
- function frame size (important due to Go's use of split stacks),
- reflection use (and by-passing reflection for common types)
- recursion implications
- zero-copy mode (encoding/decoding to byte slice without using temp buffers)
Correct.
Care was taken to precisely handle corner cases like:
overflows, nil maps and slices, nil value in stream, etc.
Efficient zero-copying into temporary byte buffers
when encoding into or decoding from a byte slice.
Standard field renaming via tags
Encoding from any value
(struct, slice, map, primitives, pointers, interface{}, etc)
Decoding into pointer to any non-nil typed value
(struct, slice, map, int, float32, bool, string, reflect.Value, etc)
Supports extension functions to handle the encode/decode of custom types
Support Go 1.2 encoding.BinaryMarshaler/BinaryUnmarshaler
Schema-less decoding
(decode into a pointer to a nil interface{} as opposed to a typed non-nil value).
Includes Options to configure what specific map or slice type to use
when decoding an encoded list or map into a nil interface{}
Provides a RPC Server and Client Codec for net/rpc communication protocol.
Msgpack Specific:
- Provides extension functions to handle spec-defined extensions (binary, timestamp)
- Options to resolve ambiguities in handling raw bytes (as string or []byte)
  during schema-less decoding (decoding into a nil interface{})
- RPC Server/Client Codec for msgpack-rpc protocol defined at:
  https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
Fast Paths for some container types:
For some container types, we circumvent reflection and its associated overhead
and allocation costs, and encode/decode directly. These types are:
[]interface{}
[]int
[]string
map[interface{}]interface{}
map[int]interface{}
map[string]interface{}

Extension Support

Users can register a function to handle the encoding or decoding of
their custom types.

There are no restrictions on what the custom type can be. Some examples:

text

type BisSet   []int
type BitSet64 uint64
type UUID     string
type MyStructWithUnexportedFields struct { a int; b bool; c []int; }
type GifImage struct { ... }

As an illustration, MyStructWithUnexportedFields would normally be
encoded as an empty map because it has no exported fields, while UUID
would be encoded as a string. However, with extension support, you can
encode any of these however you like.

RPC

RPC Client and Server Codecs are implemented, so the codecs can be used
with the standard net/rpc package.

Usage

Typical usage model:

text

// create and configure Handle
var (
  bh codec.BincHandle
  mh codec.MsgpackHandle
)

mh.MapType = reflect.TypeOf(map[string]interface{}(nil))

// configure extensions
// e.g. for msgpack, define functions and enable Time support for tag 1
// mh.AddExt(reflect.TypeOf(time.Time{}), 1, myMsgpackTimeEncodeExtFn, myMsgpackTimeDecodeExtFn)

// create and use decoder/encoder
var (
  r io.Reader
  w io.Writer
  b []byte
  h = &bh // or mh to use msgpack
)

dec = codec.NewDecoder(r, h)
dec = codec.NewDecoderBytes(b, h)
err = dec.Decode(&v) 

enc = codec.NewEncoder(w, h)
enc = codec.NewEncoderBytes(&b, h)
err = enc.Encode(v)

//RPC Server
go func() {
    for {
        conn, err := listener.Accept()
        rpcCodec := codec.GoRpc.ServerCodec(conn, h)
        //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h)
        rpc.ServeCodec(rpcCodec)
    }
}()

//RPC Communication (client side)
conn, err = net.Dial("tcp", "localhost:5555")
rpcCodec := codec.GoRpc.ClientCodec(conn, h)
//OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
client := rpc.NewClientWithCodec(rpcCodec)

Representative Benchmark Results

A sample run of benchmark using "go test -bi -bench=. -benchmem":

text

/proc/cpuinfo: Intel(R) Core(TM) i7-2630QM CPU @ 2.00GHz (HT)

..............................................
BENCHMARK INIT: 2013-10-16 11:02:50.345970786 -0400 EDT
To run full benchmark comparing encodings (MsgPack, Binc, JSON, GOB, etc), use: "go test -bench=."
Benchmark: 
	Struct recursive Depth:             1
	ApproxDeepSize Of benchmark Struct: 4694 bytes
Benchmark One-Pass Run:
	 v-msgpack: len: 1600 bytes
	      bson: len: 3025 bytes
	   msgpack: len: 1560 bytes
	      binc: len: 1187 bytes
	       gob: len: 1972 bytes
	      json: len: 2538 bytes
..............................................
PASS
Benchmark__Msgpack____Encode	   50000	     54359 ns/op	   14953 B/op	      83 allocs/op
Benchmark__Msgpack____Decode	   10000	    106531 ns/op	   14990 B/op	     410 allocs/op
Benchmark__Binc_NoSym_Encode	   50000	     53956 ns/op	   14966 B/op	      83 allocs/op
Benchmark__Binc_NoSym_Decode	   10000	    103751 ns/op	   14529 B/op	     386 allocs/op
Benchmark__Binc_Sym___Encode	   50000	     65961 ns/op	   17130 B/op	      88 allocs/op
Benchmark__Binc_Sym___Decode	   10000	    106310 ns/op	   15857 B/op	     287 allocs/op
Benchmark__Gob________Encode	   10000	    135944 ns/op	   21189 B/op	     237 allocs/op
Benchmark__Gob________Decode	    5000	    405390 ns/op	   83460 B/op	    1841 allocs/op
Benchmark__Json_______Encode	   20000	     79412 ns/op	   13874 B/op	     102 allocs/op
Benchmark__Json_______Decode	   10000	    247979 ns/op	   14202 B/op	     493 allocs/op
Benchmark__Bson_______Encode	   10000	    121762 ns/op	   27814 B/op	     514 allocs/op
Benchmark__Bson_______Decode	   10000	    162126 ns/op	   16514 B/op	     789 allocs/op
Benchmark__VMsgpack___Encode	   50000	     69155 ns/op	   12370 B/op	     344 allocs/op
Benchmark__VMsgpack___Decode	   10000	    151609 ns/op	   20307 B/op	     571 allocs/op
ok  	ugorji.net/codec	30.827s

To run full benchmark suite (including against vmsgpack and bson),
see notes in ext_dep_test.go

Overview (`vendor/src/github.com/prometheus/client_golang/prometheus/README.md`)

Overview

This is the Prometheus telemetric
instrumentation client Go client library. It
enable authors to define process-space metrics for their servers and
expose them through a web service interface for extraction,
aggregation, and a whole slew of other post processing techniques.

Installing

text

$ go get github.com/prometheus/client_golang/prometheus

Example

package main

import (
	"net/http"

	"github.com/prometheus/client_golang/prometheus"
)

var (
	indexed = prometheus.NewCounter(prometheus.CounterOpts{
		Namespace: "my_company",
		Subsystem: "indexer",
		Name:      "documents_indexed",
		Help:      "The number of documents indexed.",
	})
	size = prometheus.NewGauge(prometheus.GaugeOpts{
		Namespace: "my_company",
		Subsystem: "storage",
		Name:      "documents_total_size_bytes",
		Help:      "The total size of all documents in the storage.",
	})
)

func main() {
	http.Handle("/metrics", prometheus.Handler())

	indexed.Inc()
	size.Set(5)

	http.ListenAndServe(":8080", nil)
}

func init() {
	prometheus.MustRegister(indexed)
	prometheus.MustRegister(size)
}

Documentation

Overview (`vendor/src/github.com/ugorji/go/codec/README.md`)

Codec

High Performance, Feature-Rich Idiomatic Go codec/encoding library for
binc, msgpack, cbor, json.

Supported Serialization formats are:

msgpack: https://github.com/msgpack/msgpack
binc: http://github.com/ugorji/binc
cbor: http://cbor.io http://tools.ietf.org/html/rfc7049
json: http://json.org http://tools.ietf.org/html/rfc7159
simple:

To install:

text

go get github.com/ugorji/go/codec

This package understands the unsafe tag, to allow using unsafe semantics:

When decoding into a struct, you need to read the field name as a string
so you can find the struct field it is mapped to.
Using unsafe will bypass the allocation and copying overhead of []byte->string conversion.

To use it, you must pass the unsafe tag during install:

text

go install -tags=unsafe github.com/ugorji/go/codec

Online documentation: http://godoc.org/github.com/ugorji/go/codec
Detailed Usage/How-to Primer: http://ugorji.net/blog/go-codec-primer

The idiomatic Go support is as seen in other encoding packages in
the standard library (ie json, xml, gob, etc).

Rich Feature Set includes:

Simple but extremely powerful and feature-rich API
Very High Performance.
Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X.
Multiple conversions:
Package coerces types where appropriate
e.g. decode an int in the stream into a float, etc.
Corner Cases:
Overflows, nil maps/slices, nil values in streams are handled correctly
Standard field renaming via tags
Support for omitting empty fields during an encoding
Encoding from any value and decoding into pointer to any value
(struct, slice, map, primitives, pointers, interface{}, etc)
Extensions to support efficient encoding/decoding of any named types
Support encoding.(Binary|Text)(M|Unm)arshaler interfaces
Decoding without a schema (into a interface{}).
Includes Options to configure what specific map or slice type to use
when decoding an encoded list or map into a nil interface{}
Encode a struct as an array, and decode struct from an array in the data stream
Comprehensive support for anonymous fields
Fast (no-reflection) encoding/decoding of common maps and slices
Code-generation for faster performance.
Support binary (e.g. messagepack, cbor) and text (e.g. json) formats
Support indefinite-length formats to enable true streaming
(for formats which support it e.g. json, cbor)
Support canonical encoding, where a value is ALWAYS encoded as same sequence of bytes.
This mostly applies to maps, where iteration order is non-deterministic.
NIL in data stream decoded as zero value
Never silently skip data when decoding.
User decides whether to return an error or silently skip data when keys or indexes
in the data stream do not map to fields in the struct.
Encode/Decode from/to chan types (for iterative streaming support)
Drop-in replacement for encoding/json. json: key in struct tag supported.
Provides a RPC Server and Client Codec for net/rpc communication protocol.
Handle unique idiosynchracies of codecs e.g.
- For messagepack, configure how ambiguities in handling raw bytes are resolved
- For messagepack, provide rpc server/client codec to support
  msgpack-rpc protocol defined at:
  https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md

Extension Support

Users can register a function to handle the encoding or decoding of
their custom types.

There are no restrictions on what the custom type can be. Some examples:

text

type BisSet   []int
type BitSet64 uint64
type UUID     string
type MyStructWithUnexportedFields struct { a int; b bool; c []int; }
type GifImage struct { ... }

RPC

RPC Client and Server Codecs are implemented, so the codecs can be used
with the standard net/rpc package.

Usage

Typical usage model:

text

// create and configure Handle
var (
  bh codec.BincHandle
  mh codec.MsgpackHandle
  ch codec.CborHandle
)

mh.MapType = reflect.TypeOf(map[string]interface{}(nil))

// configure extensions
// e.g. for msgpack, define functions and enable Time support for tag 1
// mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt)

// create and use decoder/encoder
var (
  r io.Reader
  w io.Writer
  b []byte
  h = &bh // or mh to use msgpack
)

dec = codec.NewDecoder(r, h)
dec = codec.NewDecoderBytes(b, h)
err = dec.Decode(&v)

enc = codec.NewEncoder(w, h)
enc = codec.NewEncoderBytes(&b, h)
err = enc.Encode(v)

//RPC Server
go func() {
    for {
        conn, err := listener.Accept()
        rpcCodec := codec.GoRpc.ServerCodec(conn, h)
        //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h)
        rpc.ServeCodec(rpcCodec)
    }
}()

//RPC Communication (client side)
conn, err = net.Dial("tcp", "localhost:5555")
rpcCodec := codec.GoRpc.ClientCodec(conn, h)
//OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
client := rpc.NewClientWithCodec(rpcCodec)

Overview (`vendor/src/github.com/docker/engine-api/types/versions/README.md`)

Legacy API type versions

This package includes types for legacy API versions. The stable version of the API types live in api/types/*.go.

Consider moving a type here when you need to keep backwards compatibility in the API. This legacy types are organized by the latest API version they appear in. For instance, types in the v1p19 package are valid for API versions below or equal 1.19. Types in the v1p20 package are valid for the API version 1.20, since the versions below that will use the legacy types in v1p19.

Package name conventions

The package name convention is to use v as a prefix for the version number and p(patch) as a separator. We use this nomenclature due to a few restrictions in the Go package name convention:

We cannot use . because it's interpreted by the language, think of v1.20.CallFunction.
We cannot use _ because golint complains about it. The code is actually valid, but it looks probably more weird: v1_20.CallFunction.

For instance, if you want to modify a type that was available in the version 1.21 of the API but it will have different fields in the version 1.22, you want to create a new package under api/types/versions/v1p21.

Overview (`vendor/src/github.com/vbatts/tar-split/tar/asm/README.md`)

asm

This library for assembly and disassembly of tar archives, facilitated by
github.com/vbatts/tar-split/tar/storage.

Concerns

For completely safe assembly/disassembly, there will need to be a Content
Addressable Storage (CAS) directory, that maps to a checksum in the
storage.Entity of storage.FileType.

This is due to the fact that tar archives can allow multiple records for the
same path, but the last one effectively wins. Even if the prior records had a
different payload.

In this way, when assembling an archive from relative paths, if the archive has
multiple entries for the same path, then all payloads read in from a relative
path would be identical.

Thoughts

Have a look-aside directory or storage. This way when a clobbering record is
encountered from the tar stream, then the payload of the prior/existing file is
stored to the CAS. This way the clobbering record's file payload can be
extracted, but we'll have preserved the payload needed to reassemble a precise
tar archive.

clobbered/path/to/file.[0-N]

alternatively

We could just not support tar streams that have clobbering file paths.
Appending records to the archive is not incredibly common, and doesn't happen
by default for most implementations. Not supporting them wouldn't be a
security concern either, as if it did occur, we would reassemble an archive
that doesn't validate signature/checksum, so it shouldn't be trusted anyway.

Otherwise, this will allow us to defer support for appended files as a FUTURE FEATURE.

Index (`docs/index.md`)

About Docker Engine

Develop, Ship and Run Any Application, Anywhere

Docker is a platform for developers and sysadmins
to develop, ship, and run applications. Docker lets you quickly assemble
applications from components and eliminates the friction that can come when
shipping code. Docker lets you get your code tested and deployed into production
as fast as possible.

Docker consists of:

The Docker Engine - our lightweight and powerful open source containerization
technology combined with a work flow for building and containerizing your
applications.
Docker Hub - our SaaS service for
sharing and managing your application stacks.

Why Docker?

Faster delivery of your applications

We want your environment to work better. Docker containers,
and the work flow that comes with them, help your developers,
sysadmins, QA folks, and release engineers work together to get your code
into production and make it useful. We've created a standard
container format that lets developers care about their applications
inside containers while sysadmins and operators can work on running the
container in your deployment. This separation of duties streamlines and
simplifies the management and deployment of code.
We make it easy to build new containers, enable rapid iteration of
your applications, and increase the visibility of changes. This
helps everyone in your organization understand how an application works
and how it is built.
Docker containers are lightweight and fast! Containers have
sub-second launch times, reducing the cycle
time of development, testing, and deployment.

Deploy and scale more easily

Docker containers run (almost) everywhere. You can deploy
containers on desktops, physical servers, virtual machines, into
data centers, and up to public and private clouds.
Since Docker runs on so many platforms, it's easy to move your
applications around. You can easily move an application from a
testing environment into the cloud and back whenever you need.
Docker's lightweight containers also make scaling up and
down fast and easy. You can quickly launch more containers when
needed and then shut them down easily when they're no longer needed.

Get higher density and run more workloads

Docker containers don't need a hypervisor, so you can pack more of
them onto your hosts. This means you get more value out of every
server and can potentially reduce what you spend on equipment and
licenses.

Faster deployment makes for easier management

As Docker speeds up your work flow, it gets easier to make lots
of small changes instead of huge, big bang updates. Smaller
changes mean reduced risk and more uptime.

About this guide

The Understanding Docker section will help you:

See how Docker works at a high level
Understand the architecture of Docker
Discover Docker's features;
See how Docker compares to virtual machines
See some common use cases.

Installation guides

The installation section will show you how to install Docker
on a variety of platforms.

Docker user guide

To learn about Docker in more detail and to answer questions about usage and
implementation, check out the Docker User Guide.

Release notes

A summary of the changes in each release in the current series can now be found
on the separate Release Notes page

Feature Deprecation Policy

As changes are made to Docker there may be times when existing features
will need to be removed or replaced with newer features. Before an existing
feature is removed it will be labeled as "deprecated" within the documentation
and will remain in Docker for, usually, at least 2 releases. After that time
it may be removed.

Users are expected to take note of the list of deprecated features each
release and plan their migration away from those features, and (if applicable)
towards the replacement features as soon as possible.

The complete list of deprecated features can be found on the
Deprecated Features page.

Licensing

Docker is licensed under the Apache License, Version 2.0. See
LICENSE for the full
license text.

Examples Index (`docs/examples/index.md`)

Dockerize an application

This section contains the following:

Extend Index (`docs/extend/index.md`)

Extending Docker Engine

Currently, you can extend Docker Engine by adding a plugin. This section contains the following topics:

Getstarted Index (`docs/getstarted/index.md`)

Get Started with Docker

This tutorial is a for non-technical users who are interested in learning more about Docker. By following these steps, you'll learn fundamental Docker features while working through some simple tasks.

Depending on how you got here, you may or may not have already downloaded Docker for your platform and installed it.

Got Docker?

If you haven't yet downloaded Docker for your platform or installed it, go to Get Docker.

Ready to start working with Docker?

If you have already downloaded and installed Docker, you are ready to run Docker commands! Go to Verify your installation.

What you'll learn and do

You'll learn how to:

install Docker software for your platform
run a software image in a container
browse for an image on Docker Hub
create your own image and run it in a container
create a Docker Hub account and an image repository
create an image of your own
push your image to Docker Hub for others to use

The getting started was user tested to reduce the chance of users having problems. For the best chance of success, follow the steps as written the first time before exploring on your own. It takes approximately 45 minutes to complete.

Flavors of Docker

This tutorial is designed as a getting started with Docker, and works the same whether you are using Docker for Mac, Docker for Windows, Docker on Linux, or Docker Toolbox (for older Mac and Windows systems).

If you are using Docker Toolbox, you can use the Docker Quickstart Terminal to run Docker commands in a pre-configured environment instead of opening a command line terminal.

If you are using Docker for Mac, Docker for Windows, or Docker on Linux, you will have Docker running in the background, and your standard command line terminal is already set up to run Docker commands.

How much command line savvy do I need?

The getting started tour uses Docker Engine CLI commands entered on the command line of a terminal window. You don't need to be a wizard at the command line, but you should be familiar with how to open your favorite shell or terminal, and run basic commands in that environment. It helps (but isn't required) to know how to navigate a directory tree, manipulate files, list running process, and so forth.

Go to the next page to install.

Installation Index (`docs/installation/index.md`)

Install Docker Engine

Docker Engine is supported on Linux, Cloud, Windows, and OS X. Installation instructions are available for the following:

On Linux

If your linux distribution is not listed above, don't give up yet. To try out Docker on a distribution that is not listed above, go here: Installation from binaries.

On Cloud

On OSX and Windows

The Docker Archives

Instructions for installing prior releases of Docker can be found in the following docker archives:
Docker v1.7, Docker v1.6, Docker v1.5, and Docker v1.4.

Where to go after installing

Reference Index (`docs/reference/index.md`)

Engine reference

Security Index (`docs/security/index.md`)

Secure Engine

This section discusses the security features you can configure and use within your Docker Engine installation.

You can configure Docker's trust features so that your users can push and pull trusted images. To learn how to do this, see Use trusted images in this section.
You can protect the Docker daemon socket and ensure only trusted Docker client connections. For more information, Protect the Docker daemon socket
You can use certificate-based client-server authentication to verify a Docker daemon has the rights to access images on a registry. For more information, see Using certificates for repository client verification.
You can configure secure computing mode (Seccomp) policies to secure system calls in a container. For more information, see Seccomp security profiles for Docker.
An AppArmor profile for Docker is installed with the official .deb packages. For information about this profile and overriding it, see AppArmor security profiles for Docker.

Swarm Index (`docs/swarm/index.md`)

Swarm mode overview

To use Docker Engine in swarm mode, install the Docker Engine v1.12.0-rc1 or
later from the Docker releases GitHub
repository. Alternatively, install
the latest Docker for Mac or Docker for Windows Beta.

Docker Engine 1.12 includes swarm mode for natively managing a cluster of
Docker Engines called a Swarm. Use the Docker CLI to create a swarm, deploy
application services to a swarm, and manage swarm behavior.

If you’re using a Docker version prior to v1.12.0-rc1, see Docker
Swarm.

Feature highlights

Cluster management integrated with Docker Engine: Use the Docker Engine
CLI to create a Swarm of Docker Engines where you can deploy application
services. You don't need additional orchestration software to create or manage
a Swarm.
Decentralized design: Instead of handling differentiation between node
roles at deployment time, the Docker Engine handles any specialization at
runtime. You can deploy both kinds of nodes, managers and workers, using the
Docker Engine. This means you can build an entire Swarm from a single disk
image.
Declarative service model: Docker Engine uses a declarative approach to
let you define the desired state of the various services in your application
stack. For example, you might describe an application comprised of a web front
end service with message queueing services and a database backend.
Scaling: For each service, you can declare the number of tasks you want to
run. When you scale up or down, the swarm manager automatically adapts by
adding or removing tasks to maintain the desired state.
Desired state reconciliation: The swarm manager node constantly monitors
the cluster state and reconciles any differences between the actual state your
expressed desired state. For example, if you set up a service to run 10
replicas of a container, and a worker machine hosting two of those replicas
crashes, the manager will create two new replicas to replace the ones that
crashed. The swarm manager assigns the new replicas to workers that are
running and available.
Multi-host networking: You can specify an overlay network for your
services. The swarm manager automatically assigns addresses to the containers
on the overlay network when it initializes or updates the application.
Service discovery: Swarm manager nodes assign each service in the swarm a
unique DNS name and load balances running containers. You can query every
container running in the swarm through a DNS server embedded in the swarm.
Load balancing: You can expose the ports for services to an
external load balancer. Internally, the swarm lets you specify how to distribute
service containers between nodes.
Secure by default: Each node in the swarm enforces TLS mutual
authentication and encryption to secure communications between itself and all
other nodes. You have the option to use self-signed root certificates or
certificates from a custom root CA.
Rolling updates: At rollout time you can apply service updates to nodes
incrementally. The swarm manager lets you control the delay between service
deployment to different sets of nodes. If anything goes wrong, you can
roll-back a task to a previous version of the service.

What's next?

Learn swarm mode key concepts.
Get started with the swarm mode tutorial.

Tutorials Index (`docs/tutorials/index.md`)

Learn by example

Userguide Index (`docs/userguide/index.md`)

Docker Engine user guide

This guide helps users learn how to use Docker Engine.

Introduction to Engine user guide

Learn by example

Work with images

Manage storage drivers

Configure networks

Work with the default network

Misc

Apply custom metadata

Userguide Intro (`docs/userguide/intro.md`)

Engine user guide

This guide takes you through the fundamentals of using Docker Engine and
integrating it into your environment. You'll learn how to use Engine to:

Dockerize your applications.
Run your own containers.
Build Docker images.
Share your Docker images with others.
And a whole lot more!

This guide is broken into major sections that take you through learning the basics of Docker Engine and the other Docker products that support it.

Dockerizing applications: A "Hello world"

How do I run applications inside containers?

Docker Engine offers a containerization platform to power your applications. To
learn how to Dockerize applications and run them:

Go to Dockerizing Applications.

Working with containers

How do I manage my containers?

Once you get a grip on running your applications in Docker containers, you'll learn how to manage those containers. To find out
about how to inspect, monitor and manage containers:

Go to Working with Containers.

Working with Docker images

How can I access, share and build my own images?

Once you've learnt how to use Docker it's time to take the next step and
learn how to build your own application images with Docker.

Go to Working with Docker Images.

Networking containers

Until now we've seen how to build individual applications inside Docker
containers. Now learn how to build whole application stacks with Docker
networking.

Go to Networking Containers.

Managing data in containers

Now we know how to link Docker containers together the next step is
learning how to manage data, volumes and mounts inside our containers.

Go to Managing Data in Containers.

Docker products that complement Engine

Often, one powerful technology spawns many other inventions that make that easier to get to, easier to use, and more powerful. These spawned things share one common characteristic: they augment the central technology. The following Docker products expand on the core Docker Engine functions.

Docker Hub

Docker Hub is the central hub for Docker. It hosts public Docker images
and provides services to help you build and manage your Docker
environment. To learn more:

Go to Using Docker Hub.

Docker Machine

Docker Machine helps you get Docker Engines up and running quickly. Machine
can set up hosts for Docker Engines on your computer, on cloud providers,
and/or in your data center, and then configure your Docker client to securely
talk to them.

Go to Docker Machine user guide.

Docker Compose

Docker Compose allows you to define an application's components -- their containers,
configuration, links and volumes -- in a single file. Then a single command
will set everything up and start your application running.

Go to Docker Compose user guide.

Docker Swarm

Docker Swarm pools several Docker Engines together and exposes them as a single
virtual Docker Engine. It serves the standard Docker API, so any tool that already
works with Docker can now transparently scale up to multiple hosts.

Go to Docker Swarm user guide.

Getting help

Docker homepage
Docker Hub
Docker blog
Docker documentation
Docker Getting Started Guide
Docker code on GitHub
Docker mailing
list
Docker on IRC: irc.freenode.net and channel #docker
Docker on Twitter
Get Docker help on
StackOverflow

Admin Logging Index (`docs/admin/logging/index.md`)

Logging Drivers

Admin Logging Overview (`docs/admin/logging/overview.md`)

Configure logging drivers

The container can have a different logging driver than the Docker daemon. Use
the --log-driver=VALUE with the docker run command to configure the
container's logging driver. If the --log-driver option is not set, docker
uses the default (json-file) logging driver. The following options are
supported:

Driver	Description
`none`	Disables any logging for the container. `docker logs` won't be available with this driver.
`json-file`	Default logging driver for Docker. Writes JSON messages to file.
`syslog`	Syslog logging driver for Docker. Writes log messages to syslog.
`journald`	Journald logging driver for Docker. Writes log messages to `journald`.
`gelf`	Graylog Extended Log Format (GELF) logging driver for Docker. Writes log messages to a GELF endpoint likeGraylog or Logstash.
`fluentd`	Fluentd logging driver for Docker. Writes log messages to `fluentd` (forward input).
`awslogs`	Amazon CloudWatch Logs logging driver for Docker. Writes log messages to Amazon CloudWatch Logs.
`splunk`	Splunk logging driver for Docker. Writes log messages to `splunk` using HTTP Event Collector.
`etwlogs`	ETW logging driver for Docker on Windows. Writes log messages as ETW events.
`gcplogs`	Google Cloud Logging driver for Docker. Writes log messages to Google Cloud Logging.

The docker logscommand is available only for the json-file and journald
logging drivers.

The labels and env options add additional attributes for use with logging
drivers that accept them. Each option takes a comma-separated list of keys. If
there is collision between label and env keys, the value of the env takes
precedence.

To use attributes, specify them when you start the Docker daemon. For example,
to manually start the daemon with the json-file driver, and include additional
attributes in the output, run the following command:

bash

$ docker daemon \
    --log-driver=json-file \
    --log-opt labels=foo \
    --log-opt env=foo,fizz

Then, run a container and specify values for the labels or env. For
example, you might use this:

bash

$ docker run -dit --label foo=bar -e fizz=buzz alpine sh

This adds additional fields to the log depending on the driver, e.g. for
json-file that looks like:

json

"attrs":{"fizz":"buzz","foo":"bar"}

json-file options

The following logging options are supported for the json-file logging driver:

bash

--log-opt max-size=[0-9+][k|m|g]
--log-opt max-file=[0-9+]
--log-opt labels=label1,label2
--log-opt env=env1,env2

Logs that reach max-size are rolled over. You can set the size in
kilobytes(k), megabytes(m), or gigabytes(g). eg --log-opt max-size=50m. If
max-size is not set, then logs are not rolled over.

max-file specifies the maximum number of files that a log is rolled over
before being discarded. eg --log-opt max-file=100. If max-size is not set,
then max-file is not honored.

If max-size and max-file are set, docker logs only returns the log lines
from the newest log file.

syslog options

The following logging options are supported for the syslog logging driver:

bash

--log-opt syslog-address=[tcp|udp|tcp+tls]://host:port
--log-opt syslog-address=unix://path
--log-opt syslog-address=unixgram://path
--log-opt syslog-facility=daemon
--log-opt syslog-tls-ca-cert=/etc/ca-certificates/custom/ca.pem
--log-opt syslog-tls-cert=/etc/ca-certificates/custom/cert.pem
--log-opt syslog-tls-key=/etc/ca-certificates/custom/key.pem
--log-opt syslog-tls-skip-verify=true
--log-opt tag="mailer"
--log-opt syslog-format=[rfc5424|rfc5424micro|rfc3164]
--log-opt env=ENV1,ENV2,ENV3
--log-opt labels=label1,label2,label3

syslog-address specifies the remote syslog server address where the driver
connects to. If not specified it defaults to the local unix socket of the
running system. If transport is either tcp or udp and port is not
specified it defaults to 514 The following example shows how to have the
syslog driver connect to a syslog remote server at 192.168.0.42 on port
123

bash

$ docker run --log-driver=syslog --log-opt syslog-address=tcp://192.168.0.42:123

The syslog-facility option configures the syslog facility. By default, the
system uses the daemon value. To override this behavior, you can provide an
integer of 0 to 23 or any of the following named facilities:

kern
user
mail
daemon
auth
syslog
lpr
news
uucp
cron
authpriv
ftp
local0
local1
local2
local3
local4
local5
local6
local7

syslog-tls-ca-cert specifies the absolute path to the trust certificates
signed by the CA. This option is ignored if the address protocol is not
tcp+tls.

syslog-tls-cert specifies the absolute path to the TLS certificate file. This
option is ignored if the address protocol is not tcp+tls.

syslog-tls-key specifies the absolute path to the TLS key file. This option
is ignored if the address protocol is not tcp+tls.

syslog-tls-skip-verify configures the TLS verification. This verification is
enabled by default, but it can be overriden by setting this option to true.
This option is ignored if the address protocol is not tcp+tls.

tag configures a string that is appended to the APP-NAME in the syslog
message. By default, Docker uses the first 12 characters of the container ID to
tag log messages. Refer to the log tag option documentation for
customizing the log tag format.

syslog-format specifies syslog message format to use when logging. If not
specified it defaults to the local unix syslog format without hostname
specification. Specify rfc3164 to perform logging in RFC-3164 compatible
format. Specify rfc5424 to perform logging in RFC-5424 compatible format.
Specify rfc5424micro to perform logging in RFC-5424 compatible format with
microsecond timestamp resolution.

env is a comma-separated list of keys of environment variables. Used for
advanced log tag options.

labels is a comma-separated list of keys of labels. Used for advanced log
tag options.

journald options

The journald logging driver stores the container id in the journal's
CONTAINER_ID field. For detailed information on working with this logging
driver, see the journald logging driver reference documentation.

GELF options

The GELF logging driver supports the following options:

bash

--log-opt gelf-address=udp://host:port
--log-opt tag="database"
--log-opt labels=label1,label2
--log-opt env=env1,env2
--log-opt gelf-compression-type=gzip
--log-opt gelf-compression-level=1

The gelf-address option specifies the remote GELF server address that the
driver connects to. Currently, only udp is supported as the transport and you
must specify a port value. The following example shows how to connect the
gelf driver to a GELF remote server at 192.168.0.42 on port 12201

bash

$ docker run -dit \
    --log-driver=gelf \
    --log-opt gelf-address=udp://192.168.0.42:12201 \
    alpine sh

By default, Docker uses the first 12 characters of the container ID to tag log
messages. Refer to the log tag option documentation for
customizing the log tag format.

The labels and env options are supported by the gelf logging
driver. It adds additional key on the extra fields, prefixed by an
underscore (_).

text

// […]
"_foo": "bar",
"_fizz": "buzz",
// […]

The gelf-compression-type option can be used to change how the GELF driver
compresses each log message. The accepted values are gzip, zlib and none.
gzip is chosen by default.

The gelf-compression-level option can be used to change the level of
compresssion when gzip or zlib is selected as gelf-compression-type.
Accepted value must be from from -1 to 9 (BestCompression). Higher levels
typically run slower but compress more. Default value is 1 (BestSpeed).

Fluentd options

You can use the --log-opt NAME=VALUE flag to specify these additional Fluentd
logging driver options.

fluentd-address: specify host:port to connect [localhost:24224]
tag: specify tag for fluentd message
fluentd-buffer-limit: specify the maximum size of the fluentd log buffer [8MB]
fluentd-retry-wait: initial delay before a connection retry (after which it increases exponentially) [1000ms]
fluentd-max-retries: maximum number of connection retries before abrupt failure of docker [1073741824]
fluentd-async-connect: whether to block on initial connection or not [false]

For example, to specify both additional options:

bash

$ docker run -dit \
    --log-driver=fluentd \
    --log-opt fluentd-address=localhost:24224 \
    --log-opt tag="docker.{{.Name}}" \
    alpine sh

If container cannot connect to the Fluentd daemon on the specified address and
fluentd-async-connect is not enabled, the container stops immediately.
For detailed information on working with this logging driver,
see the fluentd logging driver

Amazon CloudWatch Logs options

The Amazon CloudWatch Logs logging driver supports the following options:

bash

--log-opt awslogs-region=<aws_region>
--log-opt awslogs-group=<log_group_name>
--log-opt awslogs-stream=<log_stream_name>

For detailed information on working with this logging driver, see the awslogs
logging driver reference documentation.

Splunk options

The Splunk logging driver requires the following options:

bash

--log-opt splunk-token=<splunk_http_event_collector_token>
--log-opt splunk-url=https://your_splunk_instance:8088

For detailed information about working with this logging driver, see the
Splunk logging driver reference documentation.

ETW logging driver options

The etwlogs logging driver does not require any options to be specified. This
logging driver forwards each log message as an ETW event. An ETW listener
can then be created to listen for these events.

The ETW logging driver is only available on Windows. For detailed information
on working with this logging driver, see the ETW logging driver
reference documentation.

Google Cloud Logging options

The Google Cloud Logging driver supports the following options:

bash

--log-opt gcp-project=<gcp_projext>
--log-opt labels=<label1>,<label2>
--log-opt env=<envvar1>,<envvar2>
--log-opt log-cmd=true

For detailed information about working with this logging driver, see the
Google Cloud Logging driver. reference documentation.

Installation Cloud Index (`docs/installation/cloud/index.md`)

Install Engine in the cloud

Installation Cloud Overview (`docs/installation/cloud/overview.md`)

Choose how to install

You can install Docker Engine on any cloud platform that runs an operating system (OS) that Docker supports. This includes many flavors and versions of Linux, along with Mac and Windows.

You have two options for installing:

Manually install on the cloud (create cloud hosts, then install Docker Engine on them)
Use Docker Machine to provision cloud hosts

Manually install Docker Engine on a cloud host

To install on a cloud provider:

Create an account with the cloud provider, and read cloud provider documentation to understand their process for creating hosts.
Decide which OS you want to run on the cloud host.
Understand the Docker prerequisites and install process for the chosen OS. See Install Docker Engine for a list of supported systems and links to the install guides.
Create a host with a Docker supported OS, and install Docker per the instructions for that OS.

Example (AWS): Manual install on a cloud provider shows how to create an Amazon Web Services (AWS) EC2 instance, and install Docker Engine on it.

Use Docker Machine to provision cloud hosts

Docker Machine driver plugins are available for several popular cloud platforms, so you can use Machine to provision one or more Dockerized hosts on those platforms.

With Docker Machine, you can use the same interface to create cloud hosts with Docker Engine on them, each configured per the options you specify.

To do this, you use the docker-machine create command with the driver for the cloud provider, and provider-specific flags for account verification, security credentials, and other configuration details.

Example: Use Docker Machine to provision cloud hosts walks you through the steps to set up Docker Machine and provision a Dockerized host on Digital Ocean).

Where to go next

Example: Manual install on a cloud provider (AWS EC2)
Example: Use Docker Machine to provision cloud hosts (Digital Ocean)
For supported platforms, see Install Docker Engine.
To get started with Docker post-install, see Docker User Guide.

Installation Linux Index (`docs/installation/linux/index.md`)

Install Docker Engine on Linux

Docker Engine is supported on several Linux distributions. Installation instructions are available for the following:

If your linux distribution is not listed above, don't give up yet. To try out Docker on a distribution that is not listed above, go here: Installation from binaries.

Reference Api Index (`docs/reference/api/index.md`)

API Reference

Reference Commandline Index (`docs/reference/commandline/index.md`)

The Docker commands

This section contains reference information on using Docker's command line client. Each command has a reference page along with samples. If you are unfamiliar with the command line, you should start by reading about how to Use the Docker command line.

You start the Docker daemon with the command line. How you start the daemon affects your Docker containers. For that reason you should also make sure to read the dockerd reference page.

Docker management commands

Image commands

Container commands

Hub and registry commands

Network and connectivity commands

Shared data volume commands

Swarm node commands

Swarm swarm commands

Swarm service commands

Security Trust Index (`docs/security/trust/index.md`)

Use trusted images

The following topics are available:

Swarm Swarm Tutorial Index (`docs/swarm/swarm-tutorial/index.md`)

Getting started with swarm mode

This tutorial introduces you to the features of Docker Engine Swarm mode. You
may want to familiarize yourself with the key concepts
before you begin.

The tutorial guides you through the following activities:

initializing a cluster of Docker Engines in swarm mode
adding nodes to the swarm
deploying application services to the swarm
managing the swarm once you have everything running

This tutorial uses Docker Engine CLI commands entered on the command line of a
terminal window. You should be able to install Docker on networked machines and
be comfortable running commands in the shell of your choice.

If you’re brand new to Docker, see About Docker Engine.

Set up

To run this tutorial, you need the following:

three networked host machines
Docker Engine 1.12 or later installed
the IP address of the manager machine
open ports between the hosts

Three networked host machines

The tutorial uses three networked host machines as nodes in the swarm. These can
be virtual machines on your PC, in a data center, or on a cloud service
provider. This tutorial uses the following machine names:

manager1
worker1
worker2

Docker Engine 1.12 or later

You must install Docker Engine on each one of the host machines. To use swarm
mode, install the Docker Engine v1.12.0-rc1 or later from the Docker releases
GitHub repository. Alternatively,
install the latest Docker for Mac or Docker for Windows Beta.

Advisory: Some multi-node features may not work for Docker for Mac Beta and
Docker for Windows Beta. We're working on the multi-node features for GA.

Verify that the Docker Engine daemon is running on each of the machines.

The IP address of the manager machine

The IP address must be assigned to an a network interface available to the host
operating system. All nodes in the swarm must be able to access the manager at the IP address.

Tip: You can run ifconfig on Linux or Mac OS X to see a list of the
available network interfaces.

The tutorial uses manager1 : 192.168.99.100.

Open ports between the hosts

TCP port 2377 for cluster management communications
TCP and UDP port 7946 for communication among nodes
TCP and UDP port 4789 for overlay network traffic

Tip: Docker recommends that every node in the cluster be on the same layer
3 (IP) subnet with all traffic permitted between nodes.

What's next?

After you have set up your environment, you're ready to create a swarm.

Userguide Eng Image Index (`docs/userguide/eng-image/index.md`)

Work with images

Userguide Networking Index (`docs/userguide/networking/index.md`)

Docker networks feature overview

This sections explains how to use the Docker networks feature. This feature allows users to define their own networks and connect containers to them. Using this feature you can create a network on a single host or a network that spans across multiple hosts.

If you are already familiar with Docker's default bridge network, docker0 that network continues to be supported. It is created automatically in every installation. The default bridge network is also named bridge. To see a list of topics related to that network, read the articles listed in the Docker default bridge network.

Userguide Storagedriver Index (`docs/userguide/storagedriver/index.md`)

Docker storage drivers

Docker relies on driver technology to manage the storage and interactions associated with images and the containers that run them. This section contains the following pages:

If you are new to Docker containers make sure you read "Understand images, containers, and storage drivers" first. It explains key concepts and technologies that can help you when working with storage drivers.

Acknowledgment

The Docker storage driver material was created in large part by our guest author
Nigel Poulton with a bit of help from Docker's own Jérôme Petazzoni. In his
spare time Nigel creates IT training
videos, co-hosts the weekly
In Tech We Trust podcast, and lives it large
on Twitter.

Userguide Networking Default Network Index (`docs/userguide/networking/default_network/index.md`)

Docker default bridge network

With the introduction of the Docker networks feature, you can create your own
user-defined networks. The Docker default bridge is created when you install
Docker Engine. It is a bridge network and is also named bridge. The topics
in this section are related to interacting with that default bridge network.

Breaking Changes (`docs/breaking_changes.md`)

Breaking changes and incompatibilities

Every Engine release strives to be backward compatible with its predecessors.
In all cases, the policy is that feature removal is communicated two releases
in advance and documented as part of the deprecated features
page.

Unfortunately, Docker is a fast moving project, and newly introduced features
may sometime introduce breaking changes and/or incompatibilities. This page
documents these by Engine version.

Engine 1.12

Docker clients <= 1.9.2 used an invalid Host header when making request to the
daemon. Docker 1.12 is built using golang 1.6 which is now checking the validity
of the Host header and as such clients <= 1.9.2 can't talk anymore to the daemon.
An environment variable was added to overcome this issue.

Engine 1.10

There were two breaking changes in the 1.10 release.

Registry

Registry 2.3 includes improvements to the image manifest that have caused a
breaking change. Images pushed by Engine 1.10 to a Registry 2.3 cannot be
pulled by digest by older Engine versions. A docker pull that encounters this
situation returns the following error:

text

Error response from daemon: unsupported schema version 2 for tag TAGNAME

Docker Content Trust heavily relies on pull by digest. As a result, images
pushed from the Engine 1.10 CLI to a 2.3 Registry cannot be pulled by older
Engine CLIs (< 1.10) with Docker Content Trust enabled.

If you are using an older Registry version (< 2.3), this problem does not occur
with any version of the Engine CLI; push, pull, with and without content trust
work as you would expect.

Docker Content Trust

Engine older than the current 1.10 cannot pull images from repositories that
have enabled key delegation. Key delegation is a feature which requires a
manual action to enable.

Deprecated (`docs/deprecated.md`)

Deprecated Engine Features

The following list of features are deprecated in Engine.

Three argument form in `docker import`

Deprecated In Release: v0.6.7

Removed In Release: v1.12.0

The docker import command format 'file|URL|- [REPOSITORY [TAG]]' is deprecated since November 2013. It's no more supported.

Deprecated In Release: v1.11.0

Target For Removal In Release: v1.13

The docker login command is removing the ability to automatically register for an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated.

Separator (`:`) of `--security-opt` flag on `docker run`

Deprecated In Release: v1.11.0

Target For Removal In Release: v1.13

The flag --security-opt doesn't use the colon separator(:) anymore to divide keys and values, it uses the equal symbol(=) for consinstency with other similar flags, like --storage-opt.

`/containers/(id or name)/copy` endpoint

Deprecated In Release: v1.8

Removed In Release: v1.12.0

The endpoint /containers/(id or name)/copy is deprecated in favor of /containers/(id or name)/archive.

Ambiguous event fields in API

Deprecated In Release: v1.10.0

The fields ID, Status and From in the events API have been deprecated in favor of a more rich structure.
See the events API documentation for the new format.

`-f` flag on `docker tag`

Deprecated In Release: v1.10.0

Removed In Release: v1.12.0

To make tagging consistent across the various docker commands, the -f flag on the docker tag command is deprecated. It is not longer necessary to specify -f to move a tag from one image to another. Nor will docker generate an error if the -f flag is missing and the specified tag is already in use.

HostConfig at API container start

Deprecated In Release: v1.10.0

Removed In Release: v1.12.0

Passing an HostConfig to POST /containers/{name}/start is deprecated in favor of
defining it at container creation (POST /containers/create).

Docker ps 'before' and 'since' options

Deprecated In Release: v1.10.0

Removed In Release: v1.12.0

The docker ps --before and docker ps --since options are deprecated.
Use docker ps --filter=before=... and docker ps --filter=since=... instead.

Docker search 'automated' and 'stars' options

Deprecated in Release: v1.12.0

Target For Removal In Release: v1.14

The docker search --automated and docker search --stars options are deprecated.
Use docker search --filter=is-automated=... and docker search --filter=stars=... instead.

Driver Specific Log Tags

Deprecated In Release: v1.9.0

Removed In Release: v1.12.0

Log tags are now generated in a standard way across different logging drivers.
Because of which, the driver specific log tag options syslog-tag, gelf-tag and
fluentd-tag have been deprecated in favor of the generic tag option.

text

docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"

LXC built-in exec driver

Deprecated In Release: v1.8.0

Removed In Release: v1.10.0

The built-in LXC execution driver, the lxc-conf flag, and API fields have been removed.

Old Command Line Options

Deprecated In Release: v1.8.0

Removed In Release: v1.10.0

The flags -d and --daemon are deprecated in favor of the daemon subcommand:

text

docker daemon -H ...

The following single-dash (-opt) variant of certain command line options
are deprecated and replaced with double-dash options (--opt):

text

docker attach -nostdin
docker attach -sig-proxy
docker build -no-cache
docker build -rm
docker commit -author
docker commit -run
docker events -since
docker history -notrunc
docker images -notrunc
docker inspect -format
docker ps -beforeId
docker ps -notrunc
docker ps -sinceId
docker rm -link
docker run -cidfile
docker run -dns
docker run -entrypoint
docker run -expose
docker run -link
docker run -lxc-conf
docker run -n
docker run -privileged
docker run -volumes-from
docker search -notrunc
docker search -stars
docker search -t
docker search -trusted
docker tag -force

The following double-dash options are deprecated and have no replacement:

text

docker run --cpuset
docker run --networking
docker ps --since-id
docker ps --before-id
docker search --trusted

Interacting with V1 registries

Version 1.9 adds a flag (--disable-legacy-registry=false) which prevents the docker daemon from pull, push, and login operations against v1 registries. Though disabled by default, this signals the intent to deprecate the v1 protocol.

Docker Content Trust ENV passphrase variables name change

Deprecated In Release: v1.9.0

Removed In Release: v1.12.0

Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the Tagging key has been renamed to Repository key. Due to this renaming, we're also changing the corresponding environment variables

DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE is now named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE
DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE is now named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE

Faq (`docs/faq.md`)

Frequently Asked Questions (FAQ)

If you don't see your question here, feel free to submit new ones to
[email protected]. Or, you can fork the
repo and contribute them yourself by editing
the documentation sources.

How much does Engine cost?

Docker Engine is 100% free. It is open source, so you can use it without paying.

What open source license are you using?

We are using the Apache License Version 2.0, see it here:
https://github.com/docker/docker/blob/master/LICENSE

Does Docker run on Mac OS X or Windows?

Docker Engine currently runs only on Linux, but you can use VirtualBox to run
Engine in a virtual machine on your box, and get the best of both worlds. Check
out the Mac OS X and Microsoft
Windows installation guides. The small Linux
distribution boot2docker can be set up using the Docker Machine tool to be run
inside virtual machines on these two operating systems.

Note: if you are using a remote Docker Engine daemon on a VM through Docker
Machine, then do not type the sudo before the docker commands shown in
the documentation's examples.

How do containers compare to virtual machines?

They are complementary. VMs are best used to allocate chunks of hardware
resources. Containers operate at the process level, which makes them very
lightweight and perfect as a unit of software delivery.

What does Docker technology add to just plain LXC?

Docker technology is not a replacement for LXC. "LXC" refers to capabilities of
the Linux kernel (specifically namespaces and control groups) which allow
sandboxing processes from one another, and controlling their resource
allocations. On top of this low-level foundation of kernel features, Docker
offers a high-level tool with several powerful functionalities:

Portable deployment across machines. Docker defines a format for bundling
an application and all its dependencies into a single object which can be
transferred to any Docker-enabled machine, and executed there with the
guarantee that the execution environment exposed to the application will be the
same. LXC implements process sandboxing, which is an important pre-requisite
for portable deployment, but that alone is not enough for portable deployment.
If you sent me a copy of your application installed in a custom LXC
configuration, it would almost certainly not run on my machine the way it does
on yours, because it is tied to your machine's specific configuration:
networking, storage, logging, distro, etc. Docker defines an abstraction for
these machine-specific settings, so that the exact same Docker container can
run - unchanged - on many different machines, with many different
configurations.
Application-centric. Docker is optimized for the deployment of
applications, as opposed to machines. This is reflected in its API, user
interface, design philosophy and documentation. By contrast, the lxc helper
scripts focus on containers as lightweight machines - basically servers that
boot faster and need less RAM. We think there's more to containers than just
that.
Automatic build. Docker includes a tool for developers to automatically
assemble a container from their source
code, with full control over application
dependencies, build tools, packaging etc. They are free to use make, maven,
chef, puppet, salt, Debian packages, RPMs, source tarballs, or any
combination of the above, regardless of the configuration of the machines.
Versioning. Docker includes git-like capabilities for tracking successive
versions of a container, inspecting the diff between versions, committing new
versions, rolling back etc. The history also includes how a container was
assembled and by whom, so you get full traceability from the production server
all the way back to the upstream developer. Docker also implements incremental
uploads and downloads, similar to git pull, so new versions of a container
can be transferred by only sending diffs.
Component re-use. Any container can be used as a "base image" to create more specialized components. This can
be done manually or as part of an automated build. For example you can prepare
the ideal Python environment, and use it as a base for 10 different
applications. Your ideal PostgreSQL setup can be re-used for all your future
projects. And so on.
Sharing. Docker has access to a public registry on Docker Hub
where thousands of people have uploaded useful images: anything from Redis,
CouchDB, PostgreSQL to IRC bouncers to Rails app servers to Hadoop to base
images for various Linux distros. The
registry also
includes an official "standard library" of useful containers maintained by the
Docker team. The registry itself is open-source, so anyone can deploy their own
registry to store and transfer private containers, for internal server
deployments for example.
Tool ecosystem. Docker defines an API for automating and customizing the
creation and deployment of containers. There are a huge number of tools
integrating with Docker to extend its capabilities. PaaS-like deployment
(Dokku, Deis, Flynn), multi-node orchestration (Maestro, Salt, Mesos, Openstack
Nova), management dashboards (docker-ui, Openstack Horizon, Shipyard),
configuration management (Chef, Puppet), continuous integration (Jenkins,
Strider, Travis), etc. Docker is rapidly establishing itself as the standard
for container-based tooling.

What is different between a Docker container and a VM?

There's a great StackOverflow answer showing the differences.

Do I lose my data when the container exits?

Not at all! Any data that your application writes to disk gets preserved in its
container until you explicitly delete the container. The file system for the
container persists even after the container halts.

How far do Docker containers scale?

Some of the largest server farms in the world today are based on containers.
Large web deployments like Google and Twitter, and platform providers such as
Heroku and dotCloud all run on container technology, at a scale of hundreds of
thousands or even millions of containers running in parallel.

How do I connect Docker containers?

Currently the recommended way to connect containers is via the Docker network feature. You can see details of how to work with Docker networks here.

Also useful for more flexible service portability is the Ambassador linking
pattern.

How do I run more than one process in a Docker container?

Any capable process supervisor such as http://supervisord.org/, runit, s6, or daemontools can do the trick. Docker
will start up the process management daemon which will then fork to run
additional processes. As long as the processor manager daemon continues to run,
the container will continue to as well. You can see a more substantial example
that uses supervisord here.

What platforms does Docker run on?

Linux:

Ubuntu 12.04, 13.04 et al
Fedora 19/20+
RHEL 6.5+
CentOS 6+
Gentoo
ArchLinux
openSUSE 12.3+
CRUX 3.0+

Cloud:

Amazon EC2
Google Compute Engine
Microsoft Azure
Rackspace

How do I report a security issue with Docker?

You can learn about the project's security policy
here and report security issues to this
mailbox.

Why do I need to sign my commits to Docker with the DCO?

Please read our blog post on the introduction of the DCO.

When building an image, should I prefer system libraries or bundled ones?

This is a summary of a discussion on the docker-dev mailing list.

Virtually all programs depend on third-party libraries. Most frequently, they
will use dynamic linking and some kind of package dependency, so that when
multiple programs need the same library, it is installed only once.

Some programs, however, will bundle their third-party libraries, because they
rely on very specific versions of those libraries. For instance, Node.js bundles
OpenSSL; MongoDB bundles V8 and Boost (among others).

When creating a Docker image, is it better to use the bundled libraries, or
should you build those programs so that they use the default system libraries
instead?

The key point about system libraries is not about saving disk or memory space.
It is about security. All major distributions handle security seriously, by
having dedicated security teams, following up closely with published
vulnerabilities, and disclosing advisories themselves. (Look at the Debian
Security Information for an example of those
procedures.) Upstream developers, however, do not always implement similar
practices.

Before setting up a Docker image to compile a program from source, if you want
to use bundled libraries, you should check if the upstream authors provide a
convenient way to announce security vulnerabilities, and if they update their
bundled libraries in a timely manner. If they don't, you are exposing yourself
(and the users of your image) to security vulnerabilities.

Likewise, before using packages built by others, you should check if the
channels providing those packages implement similar security best practices.
Downloading and installing an "all-in-one" .deb or .rpm sounds great at first,
except if you have no way to figure out that it contains a copy of the OpenSSL
library vulnerable to the Heartbleed bug.

Why is `DEBIAN_FRONTEND=noninteractive` discouraged in Dockerfiles?

When building Docker images on Debian and Ubuntu you may have seen errors like:

text

unable to initialize frontend: Dialog

These errors don't stop the image from being built but inform you that the
installation process tried to open a dialog box, but was unable to. Generally,
these errors are safe to ignore.

Some people circumvent these errors by changing the DEBIAN_FRONTEND
environment variable inside the Dockerfile using:

text

ENV DEBIAN_FRONTEND=noninteractive

This prevents the installer from opening dialog boxes during installation which
stops the errors.

While this may sound like a good idea, it may have side effects. The
DEBIAN_FRONTEND environment variable will be inherited by all images and
containers built from your image, effectively changing their behavior. People
using those images will run into problems when installing software
interactively, because installers will not show any dialog boxes.

Because of this, and because setting DEBIAN_FRONTEND to noninteractive is
mainly a 'cosmetic' change, we discourage changing it.

If you really need to change its setting, make sure to change it back to its
default value
afterwards.

Why do I get `Connection reset by peer` when making a request to a service running in a container?

Typically, this message is returned if the service is already bound to your
localhost. As a result, requests coming to the container from outside are
dropped. To correct this problem, change the service's configuration on your
localhost so that the service accepts requests from all IPs. If you aren't sure
how to do this, check the documentation for your OS.

Why do I get `Cannot connect to the Docker daemon. Is the docker daemon running on this host?` when using docker-machine?

This error points out that the docker client cannot connect to the virtual machine.
This means that either the virtual machine that works underneath docker-machine
is not running or that the client doesn't correctly point at it.

To verify that the docker machine is running you can use the docker-machine ls
command and start it with docker-machine start if needed.

text

$ docker-machine ls
NAME             ACTIVE   DRIVER       STATE     URL   SWARM                   DOCKER    ERRORS
default          -        virtualbox   Stopped                                 Unknown

$ docker-machine start default

You have to tell Docker to talk to that machine. You can do this with the
docker-machine env command. For example,

text

$ eval "$(docker-machine env default)"
$ docker ps

Where can I find more answers?

You can find more answers on:

Looking for something else to read? Checkout the User Guide.

Migration (`docs/migration.md`)

Migrate to Engine 1.10

Starting from version 1.10 of Docker Engine, we completely change the way image
data is addressed on disk. Previously, every image and layer used a randomly
assigned UUID. In 1.10 we implemented a content addressable method using an ID,
based on a secure hash of the image and layer data.

The new method gives users more security, provides a built-in way to avoid ID
collisions and guarantee data integrity after pull, push, load, or save. It also
brings better sharing of layers by allowing many images to freely share their
layers even if they didn’t come from the same build.

Addressing images by their content also lets us more easily detect if something
has already been downloaded. Because we have separated images and layers, you
don’t have to pull the configurations for every image that was part of the
original build chain. We also don’t need to create layers for the build
instructions that didn’t modify the filesystem.

Content addressability is the foundation for the new distribution features. The
image pull and push code has been reworked to use a download/upload manager
concept that makes pushing and pulling images much more stable and mitigates any
parallel request issues. The download manager also brings retries on failed
downloads and better prioritization for concurrent downloads.

We are also introducing a new manifest format that is built on top of the
content addressable base. It directly references the content addressable image
configuration and layer checksums. The new manifest format also makes it
possible for a manifest list to be used for targeting multiple
architectures/platforms. Moving to the new manifest format will be completely
transparent.

Preparing for upgrade

To make your current images accessible to the new model we have to migrate them
to content addressable storage. This means calculating the secure checksums for
your current data.

All your current images, tags and containers are automatically migrated to the
new foundation the first time you start Docker Engine 1.10. Before loading your
container, the daemon will calculate all needed checksums for your current data,
and after it has completed, all your images and tags will have brand new secure
IDs.

While this is simple operation, calculating SHA256 checksums for your files
can take time if you have lots of image data. On average you should assume
that migrator can process data at a speed of 100MB/s. During this time your
Docker daemon won’t be ready to respond to requests.

Minimizing migration time

If you can accept this one time hit, then upgrading Docker Engine and restarting
the daemon will transparently migrate your images. However, if you want to
minimize the daemon’s downtime, a migration utility can be run while your old
daemon is still running.

This tool will find all your current images and calculate the checksums for
them. After you upgrade and restart the daemon, the checksum data of the
migrated images will already exist, freeing the daemon from that computation
work. If new images appeared between the migration and the upgrade, those will
be processed at time of upgrade to 1.10.

You can download the migration tool
here.

The migration tool can also be run as a Docker image. While running the migrator
image you need to expose your Docker data directory to the container. If you use
the default path then you would run:

text

$ docker run --rm -v /var/lib/docker:/var/lib/docker docker/v1.10-migrator

If you use the
devicemapper storage driver, you also need to pass the flag --privileged to
give the tool access to your storage devices.

Understanding Docker (`docs/understanding-docker.md`)

Docker Overview

Docker is an open platform for developing, shipping, and running applications.
Docker is designed to deliver your applications faster. With Docker you can
separate your applications from your infrastructure and treat your
infrastructure like a managed application. Docker helps you ship code faster,
test faster, deploy faster, and shorten the cycle between writing code and
running code.

Docker does this by combining kernel containerization features with workflows
and tooling that help you manage and deploy your applications.

What is the Docker platform?

At its core, Docker provides a way to run almost any application securely
isolated in a container. The isolation and security allow you to run many
containers simultaneously on your host. The lightweight nature of containers,
which run without the extra load of a hypervisor, means you can get more out of
your hardware.

Surrounding the container is tooling and a platform which can help you in
several ways:

Get your applications (and supporting components) into Docker containers
Distribute and ship those containers to your teams for further development
and testing
Deploy those applications to your production environment,
whether it is in a local data center or the Cloud

What is Docker Engine?

Docker Engine is a client-server application with these major components:

A server which is a type of long-running program called a daemon process.
A REST API which specifies interfaces that programs can use to talk to the daemon and instruct it what to do.
A command line interface (CLI) client.

The CLI imakes use of the Docker REST API to control or interact with the Docker daemon through scripting or direct CLI commands. Many other Docker applications make use of the underlying API and CLI.

The daemon creates and manages Docker objects. Docker objects include images, containers, networks, data volumes, and so forth.

Note: Docker is licensed under the open source Apache 2.0 license.

What can I use Docker for?

Faster delivery of your applications

Docker is perfect for helping you with the development lifecycle. Docker
allows your developers to develop on local containers that contain your
applications and services. It can then integrate into a continuous integration and
deployment workflow.

For example, your developers write code locally and share their development stack via
Docker with their colleagues. When they are ready, they push their code and the
stack they are developing onto a test environment and execute any required
tests. From the testing environment, you can then push the Docker images into
production and deploy your code.

Deploying and scaling more easily

Docker's container-based platform allows for highly portable workloads. Docker
containers can run on a developer's local host, on physical or virtual machines
in a data center, or in the Cloud.

Docker's portability and lightweight nature also make dynamically managing
workloads easy. You can use Docker to quickly scale up or tear down applications
and services. Docker's speed means that scaling can be near real time.

Achieving higher density and running more workloads

Docker is lightweight and fast. It provides a viable, cost-effective alternative
to hypervisor-based virtual machines. This is especially useful in high density
environments: for example, building your own Cloud or Platform-as-a-Service. But
it is also useful for small and medium deployments where you want to get more
out of the resources you have.

What is Docker's architecture?

Docker uses a client-server architecture. The Docker client talks to the
Docker daemon, which does the heavy lifting of building, running, and
distributing your Docker containers. Both the Docker client and the daemon can
run on the same system, or you can connect a Docker client to a remote Docker
daemon. The Docker client and daemon communicate via sockets or through a
RESTful API.

The Docker daemon

As shown in the diagram above, the Docker daemon runs on a host machine. The
user does not directly interact with the daemon, but instead through the Docker
client.

The Docker client

The Docker client, in the form of the docker binary, is the primary user
interface to Docker. It accepts commands from the user and communicates back and
forth with a Docker daemon.

Inside Docker

To understand Docker's internals, you need to know about three resources:

Docker images
Docker registries
Docker containers

Docker images

A Docker image is a read-only template. For example, an image could contain an Ubuntu
operating system with Apache and your web application installed. Images are used to create
Docker containers. Docker provides a simple way to build new images or update existing
images, or you can download Docker images that other people have already created.
Docker images are the build component of Docker.

Docker registries

Docker registries hold images. These are public or private stores from which you
upload or download images. The public Docker registry is provided with the
Docker Hub. It serves a huge collection of existing
images for your use. These can be images you create yourself or you can use
images that others have previously created. Docker registries are the
distribution component of Docker.
For more information, go to Docker Registry and
Docker Trusted Registry.

Docker containers

Docker containers are similar to a directory. A Docker container holds everything that
is needed for an application to run. Each container is created from a Docker
image. Docker containers can be run, started, stopped, moved, and deleted. Each
container is an isolated and secure application platform. Docker containers are the
run component of Docker.

How does a Docker image work?

We've already seen that Docker images are read-only templates from which Docker
containers are launched. Each image consists of a series of layers. Docker
makes use of union file systems to
combine these layers into a single image. Union file systems allow files and
directories of separate file systems, known as branches, to be transparently
overlaid, forming a single coherent file system.

One of the reasons Docker is so lightweight is because of these layers. When you
change a Docker image—for example, update an application to a new version— a new layer
gets built. Thus, rather than replacing the whole image or entirely
rebuilding, as you may do with a virtual machine, only that layer is added or
updated. Now you don't need to distribute a whole new image, just the update,
making distributing Docker images faster and simpler.

Every image starts from a base image, for example ubuntu, a base Ubuntu image,
or fedora, a base Fedora image. You can also use images of your own as the
basis for a new image, for example if you have a base Apache image you could use
this as the base of all your web application images.

Note: Docker Hub is a public registry and stores
images.

Docker images are then built from these base images using a simple, descriptive
set of steps we call instructions. Each instruction creates a new layer in our
image. Instructions include actions like:

Run a command
Add a file or directory
Create an environment variable
What process to run when launching a container from this image

These instructions are stored in a file called a Dockerfile. A Dockerfile is
a text based script that contains instructions and commands for building the image
from the base image. Docker reads this Dockerfile when you request a build of
an image, executes the instructions, and returns a final image.

How does a Docker registry work?

The Docker registry is the store for your Docker images. Once you build a Docker
image you can push it to a public registry such as Docker Hub
or to your own registry running behind your firewall.

Using the Docker client, you can search for already published images and then
pull them down to your Docker host to build containers from them.

Docker Hub provides both public and private storage
for images. Public storage is searchable and can be downloaded by anyone.
Private storage is excluded from search results and only you and your users can
pull images down and use them to build containers. You can sign up for a storage plan
here.

How does a container work?

A container consists of an operating system, user-added files, and meta-data. As
we've seen, each container is built from an image. That image tells Docker
what the container holds, what process to run when the container is launched, and
a variety of other configuration data. The Docker image is read-only. When
Docker runs a container from an image, it adds a read-write layer on top of the
image (using a union file system as we saw earlier) in which your application can
then run.

What happens when you run a container?

Either by using the docker binary or via the API, the Docker client tells the Docker
daemon to run a container.

text

$ docker run -i -t ubuntu /bin/bash

The Docker Engine client is launched using the docker binary with the run option
running a new container. The bare minimum the Docker client needs to tell the
Docker daemon to run the container is:

What Docker image to build the container from, for example, ubuntu
The command you want to run inside the container when it is launched,
for example,/bin/bash

So what happens under the hood when we run this command?

In order, Docker Engine does the following:

Pulls the ubuntu image: Docker Engine checks for the presence of the ubuntu
image. If the image already exists, then Docker Engine uses it for the new container.
If it doesn't exist locally on the host, then Docker Engine pulls it from
Docker Hub.
Creates a new container: Once Docker Engine has the image, it uses it to create a
container.
Allocates a filesystem and mounts a read-write layer: The container is created in
the file system and a read-write layer is added to the image.
Allocates a network / bridge interface: Creates a network interface that allows the
Docker container to talk to the local host.
Sets up an IP address: Finds and attaches an available IP address from a pool.
Executes a process that you specify: Runs your application, and;
Captures and provides application output: Connects and logs standard input, outputs
and errors for you to see how your application is running.

You now have a running container! Now you can manage your container, interact with
your application and then, when finished, stop and remove your container.

The underlying technology

Docker is written in Go and makes use of several kernel features to
deliver the functionality we've seen.

Namespaces

Docker takes advantage of a technology called namespaces to provide the
isolated workspace we call the container. When you run a container, Docker
creates a set of namespaces for that container.

This provides a layer of isolation: each aspect of a container runs in its own
namespace and does not have access outside it.

Some of the namespaces that Docker Engine uses on Linux are:

The pid namespace: Process isolation (PID: Process ID).
The net namespace: Managing network interfaces (NET:
Networking).
The ipc namespace: Managing access to IPC
resources (IPC: InterProcess Communication).
The mnt namespace: Managing mount-points (MNT: Mount).
The uts namespace: Isolating kernel and version identifiers. (UTS: Unix
Timesharing System).

Control groups

Docker Engine on Linux also makes use of another technology called cgroups or control groups.
A key to running applications in isolation is to have them only use the
resources you want. This ensures containers are good multi-tenant citizens on a
host. Control groups allow Docker Engine to share available hardware resources to
containers and, if required, set up limits and constraints. For example,
limiting the memory available to a specific container.

Union file systems

Union file systems, or UnionFS, are file systems that operate by creating layers,
making them very lightweight and fast. Docker Engine uses union file systems to provide
the building blocks for containers. Docker Engine can make use of several union file system variants
including: AUFS, btrfs, vfs, and DeviceMapper.

Container format

Docker Engine combines these components into a wrapper we call a container format. The
default container format is called libcontainer. In the future, Docker may
support other container formats, for example, by integrating with BSD Jails
or Solaris Zones.

Next steps

Read about Installing Docker Engine.
Learn about the Docker Engine User Guide.

Admin Ambassador Pattern Linking (`docs/admin/ambassador_pattern_linking.md`)

Link via an ambassador container

Rather than hardcoding network links between a service consumer and
provider, Docker encourages service portability, for example instead of:

text

(consumer) --> (redis)

Requiring you to restart the consumer to attach it to a different
redis service, you can add ambassadors:

text

(consumer) --> (redis-ambassador) --> (redis)

text

(consumer) --> (redis-ambassador) ---network---> (redis-ambassador) --> (redis)

When you need to rewire your consumer to talk to a different Redis
server, you can just restart the redis-ambassador container that the
consumer is connected to.

This pattern also allows you to transparently move the Redis server to a
different docker host from the consumer.

Using the svendowideit/ambassador container, the link wiring is
controlled entirely from the docker run parameters.

Two host example

Start actual Redis server on one Docker host

text

big-server $ docker run -d --name redis crosbymichael/redis

Then add an ambassador linked to the Redis server, mapping a port to the
outside world

text

big-server $ docker run -d --link redis:redis --name redis_ambassador -p 6379:6379 svendowideit/ambassador

On the other host, you can set up another ambassador setting environment
variables for each remote port we want to proxy to the big-server

text

client-server $ docker run -d --name redis_ambassador --expose 6379 -e REDIS_PORT_6379_TCP=tcp://192.168.1.52:6379 svendowideit/ambassador

Then on the client-server host, you can use a Redis client container
to talk to the remote Redis server, just by linking to the local Redis
ambassador.

text

client-server $ docker run -i -t --rm --link redis_ambassador:redis relateiq/redis-cli
redis 172.17.0.160:6379> ping
PONG

How it works

The following example shows what the svendowideit/ambassador container
does automatically (with a tiny amount of sed)

On the Docker host (192.168.1.52) that Redis will run on:

text

# start actual redis server
$ docker run -d --name redis crosbymichael/redis

# get a redis-cli container for connection testing
$ docker pull relateiq/redis-cli

# test the redis server by talking to it directly
$ docker run -t -i --rm --link redis:redis relateiq/redis-cli
redis 172.17.0.136:6379> ping
PONG
^D

# add redis ambassador
$ docker run -t -i --link redis:redis --name redis_ambassador -p 6379:6379 alpine:3.2 sh

In the redis_ambassador container, you can see the linked Redis
containers env:

text

/ # env
REDIS_PORT=tcp://172.17.0.136:6379
REDIS_PORT_6379_TCP_ADDR=172.17.0.136
REDIS_NAME=/redis_ambassador/redis
HOSTNAME=19d7adf4705e
SHLVL=1
HOME=/root
REDIS_PORT_6379_TCP_PORT=6379
REDIS_PORT_6379_TCP_PROTO=tcp
REDIS_PORT_6379_TCP=tcp://172.17.0.136:6379
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
/ # exit

This environment is used by the ambassador socat script to expose Redis
to the world (via the -p 6379:6379 port mapping):

text

$ docker rm redis_ambassador
$ CMD="apk update && apk add socat && sh"
$ docker run -t -i --link redis:redis --name redis_ambassador -p 6379:6379 alpine:3.2 sh -c "$CMD"
[...]
/ # socat -t 100000000 TCP4-LISTEN:6379,fork,reuseaddr TCP4:172.17.0.136:6379

Now ping the Redis server via the ambassador:

Now go to a different server:

text

$ CMD="apk update && apk add socat && sh"
$ docker run -t -i --expose 6379 --name redis_ambassador alpine:3.2 sh -c "$CMD"
[...]
/ # socat -t 100000000 TCP4-LISTEN:6379,fork,reuseaddr TCP4:192.168.1.52:6379

And get the redis-cli image so we can talk over the ambassador bridge.

text

$ docker pull relateiq/redis-cli
$ docker run -i -t --rm --link redis_ambassador:redis relateiq/redis-cli
redis 172.17.0.160:6379> ping
PONG

The svendowideit/ambassador Dockerfile

The svendowideit/ambassador image is based on the alpine:3.2 image with
socat installed. When you start the container, it uses a small sed
script to parse out the (possibly multiple) link environment variables
to set up the port forwarding. On the remote host, you need to set the
variable using the -e command line option.

text

--expose 1234 -e REDIS_PORT_1234_TCP=tcp://192.168.1.52:6379

Will forward the local 1234 port to the remote IP and port, in this
case 192.168.1.52:6379.

text

#
# do
#   docker build -t svendowideit/ambassador .
# then to run it (on the host that has the real backend on it)
#   docker run -t -i -link redis:redis -name redis_ambassador -p 6379:6379 svendowideit/ambassador
# on the remote host, you can set up another ambassador
#    docker run -t -i -name redis_ambassador -expose 6379 -e REDIS_PORT_6379_TCP=tcp://192.168.1.52:6379 svendowideit/ambassador sh
# you can read more about this process at https://docs.docker.com/articles/ambassador_pattern_linking/

# use alpine because its a minimal image with a package manager.
# prettymuch all that is needed is a container that has a functioning env and socat (or equivalent)
FROM	alpine:3.2
MAINTAINER	[email protected]

RUN apk update && \
	apk add socat && \
	rm -r /var/cache/

CMD	env | grep _TCP= | (sed 's/.*_PORT_\([0-9]*\)_TCP=tcp:\/\/\(.*\):\(.*\)/socat -t 100000000 TCP4-LISTEN:\1,fork,reuseaddr TCP4:\2:\3 \&/' && echo wait) | sh

Admin B2d Volume Resize (`docs/admin/b2d_volume_resize.md`)

Getting “no space left on device” errors with Boot2Docker?

If you're using Boot2Docker with a large number of images, or the images you're
working with are very large, your pulls might start failing with "no space left
on device" errors when the Boot2Docker volume fills up. There are two solutions
you can try.

Solution 1: Add the `DiskImage` property in boot2docker profile

The boot2docker command reads its configuration from the $BOOT2DOCKER_PROFILE if set, or $BOOT2DOCKER_DIR/profile or $HOME/.boot2docker/profile (on Windows this is %USERPROFILE%/.boot2docker/profile).

View the existing configuration, use the boot2docker config command.

text

$ boot2docker config
 # boot2docker profile filename: /Users/mary/.boot2docker/profile
 Init = false
 Verbose = false
 Driver = "virtualbox"
 Clobber = true
 ForceUpgradeDownload = false
 SSH = "ssh"
 SSHGen = "ssh-keygen"
 SSHKey = "/Users/mary/.ssh/id_boot2docker"
 VM = "boot2docker-vm"
 Dir = "/Users/mary/.boot2docker"
 ISOURL = "https://api.github.com/repos/boot2docker/boot2docker/releases"
 ISO = "/Users/mary/.boot2docker/boot2docker.iso"
 DiskSize = 20000
 Memory = 2048
 CPUs = 8
 SSHPort = 2022
 DockerPort = 0
 HostIP = "192.168.59.3"
 DHCPIP = "192.168.59.99"
 NetMask = [255, 255, 255, 0]
 LowerIP = "192.168.59.103"
 UpperIP = "192.168.59.254"
 DHCPEnabled = true
 Serial = false
 SerialFile = "/Users/mary/.boot2docker/boot2docker-vm.sock"
 Waittime = 300
 Retries = 75

The configuration shows you where boot2docker is looking for the profile file. It also output the settings that are in use.

Initialize a default file to customize using boot2docker config > ~/.boot2docker/profile command.
Add the following lines to $HOME/.boot2docker/profile:
text
```
# Disk image size in MB
 DiskSize = 50000
```
Run the following sequence of commands to restart Boot2Docker with the new settings.
text
```
$ boot2docker poweroff
 $ boot2docker destroy
 $ boot2docker init
 $ boot2docker up
```

Solution 2: Increase the size of boot2docker volume

This solution increases the volume size by first cloning it, then resizing it
using a disk partitioning tool. We recommend
GParted. The tool comes
as a bootable ISO, is a free download, and works well with VirtualBox.

Stop Boot2Docker

Issue the command to stop the Boot2Docker VM on the command line:

text

$ boot2docker stop

Clone the VMDK image to a VDI image

Boot2Docker ships with a VMDK image, which can't be resized by VirtualBox's
native tools. We will instead create a VDI volume and clone the VMDK volume to
it.

Using the command line VirtualBox tools, clone the VMDK image to a VDI image:

text

$ vboxmanage clonehd /full/path/to/boot2docker-hd.vmdk /full/path/to/<newVDIimage>.vdi --format VDI --variant Standard

Resize the VDI volume

Choose a size that will be appropriate for your needs. If you're spinning up a
lot of containers, or your containers are particularly large, larger will be
better:

text

$ vboxmanage modifyhd /full/path/to/<newVDIimage>.vdi --resize <size in MB>

Download a disk partitioning tool ISO

To resize the volume, we'll use GParted.
Once you've downloaded the tool, add the ISO to the Boot2Docker VM IDE bus.
You might need to create the bus before you can add the ISO.

Note:
It's important that you choose a partitioning tool that is available as an ISO so
that the Boot2Docker VM can be booted with it.

Add the new VDI image

In the settings for the Boot2Docker image in VirtualBox, remove the VMDK image
from the SATA controller and add the VDI image.

Verify the boot order

In the System settings for the Boot2Docker VM, make sure that CD/DVD is
at the top of the Boot Order list.
Boot to the disk partitioning ISO

Manually start the Boot2Docker VM in VirtualBox, and the disk partitioning ISO
should start up. Using GParted, choose the GParted Live (default settings)
option. Choose the default keyboard, language, and XWindows settings, and the
GParted tool will start up and display the VDI volume you created. Right click
on the VDI and choose Resize/Move.

Drag the slider representing the volume to the maximum available size.
Click Resize/Move followed by Apply.
Quit GParted and shut down the VM.
Remove the GParted ISO from the IDE controller for the Boot2Docker VM in
VirtualBox.
Start the Boot2Docker VM

Fire up the Boot2Docker VM manually in VirtualBox. The VM should log in
automatically, but if it doesn't, the credentials are docker/tcuser. Using
the df -h command, verify that your changes took effect.

You're done!

Admin Chef (`docs/admin/chef.md`)

Using Chef

Note:
Please note this is a community contributed installation path.

Requirements

To use this guide you'll need a working installation of
Chef. This cookbook supports a variety of
operating systems.

Installation

The cookbook is available on the Chef Supermarket and can be
installed using your favorite cookbook dependency manager.

The source can be found on
GitHub.

Usage

Add depends 'docker', '~> 2.0' to your cookbook's metadata.rb
Use resources shipped in cookbook in a recipe, the same way you'd
use core Chef resources (file, template, directory, package, etc).

ruby

docker_service 'default' do
  action [:create, :start]
end

docker_image 'busybox' do
  action :pull
end

docker_container 'an echo server' do
  repo 'busybox'
  port '1234:1234'
  command "nc -ll -p 1234 -e /bin/cat"
end

Getting Started

Here's a quick example of pulling the latest image and running a
container with exposed ports.

ruby

# Pull latest image
docker_image 'nginx' do
  tag 'latest'
  action :pull
end

# Run container exposing ports
docker_container 'my_nginx' do
  repo 'nginx'
  tag 'latest'
  port '80:80'
  binds [ '/some/local/files/:/etc/nginx/conf.d' ]
  host_name 'www'
  domain_name 'computers.biz'
  env 'FOO=bar'
  subscribes :redeploy, 'docker_image[nginx]'
end

Admin Configuring (`docs/admin/configuring.md`)

Configuring and running Docker on various distributions

After successfully installing Docker, the docker daemon runs with its default
configuration.

In a production environment, system administrators typically configure the
docker daemon to start and stop according to an organization's requirements. In most
cases, the system administrator configures a process manager such as SysVinit, Upstart,
or systemd to manage the docker daemon's start and stop.

Running the docker daemon directly

The docker daemon can be run directly using the dockerd command. By default it listens on
the Unix socket unix:///var/run/docker.sock

text

$ dockerd

INFO[0000] +job init_networkdriver()
INFO[0000] +job serveapi(unix:///var/run/docker.sock)
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
...
...

Configuring the docker daemon directly

If you're running the docker daemon directly by running docker daemon instead
of using a process manager, you can append the configuration options to the docker run
command directly. Other options can be passed to the docker daemon to configure it.

Some of the daemon's options are:

Flag	Description
`-D`, `--debug=false`	Enable or disable debug mode. By default, this is false.
`-H`,`--host=[]`	Daemon socket(s) to connect to.
`--tls=false`	Enable or disable TLS. By default, this is false.

Here is an example of running the docker daemon with configuration options:

text

$ dockerd -D --tls=true --tlscert=/var/docker/server.pem --tlskey=/var/docker/serverkey.pem -H tcp://192.168.59.3:2376

These options :

Enable -D (debug) mode
Set tls to true with the server certificate and key specified using --tlscert and --tlskey respectively
Listen for connections on tcp://192.168.59.3:2376

The command line reference has the complete list of daemon flags
with explanations.

Daemon debugging

As noted above, setting the log level of the daemon to "debug" or enabling debug mode
with -D allows the administrator or operator to gain much more knowledge about the
runtime activity of the daemon. If faced with a non-responsive daemon, the administrator
can force a full stack trace of all threads to be added to the daemon log by sending the
SIGUSR1 signal to the Docker daemon. A common way to send this signal is using the kill
command on Linux systems. For example, kill -USR1 <daemon-pid> sends the SIGUSR1
signal to the daemon process, causing the stack dump to be added to the daemon log.

Note: The log level setting of the daemon must be at least "info" level and above for
the stack trace to be saved to the logfile. By default the daemon's log level is set to
"info".

The daemon will continue operating after handling the SIGUSR1 signal and dumping the stack
traces to the log. The stack traces can be used to determine the state of all goroutines and
threads within the daemon.

Ubuntu

As of 14.04, Ubuntu uses Upstart as a process manager. By default, Upstart jobs
are located in /etc/init and the docker Upstart job can be found at /etc/init/docker.conf.

After successfully installing Docker for Ubuntu,
you can check the running status using Upstart in this way:

text

$ sudo status docker

docker start/running, process 989

Running Docker

You can start/stop/restart the docker daemon using

text

$ sudo start docker

$ sudo stop docker

$ sudo restart docker

Configuring Docker

The instructions below depict configuring Docker on a system that uses upstart
as the process manager. As of Ubuntu 15.04, Ubuntu uses systemd as its process
manager. For Ubuntu 15.04 and higher, refer to control and configure Docker with systemd.

You configure the docker daemon in the /etc/default/docker file on your
system. You do this by specifying values in a DOCKER_OPTS variable.

To configure Docker options:

Log into your host as a user with sudo or root privileges.
If you don't have one, create the /etc/default/docker file on your host. Depending on how
you installed Docker, you may already have this file.
Open the file with your favorite editor.
text
```
$ sudo vi /etc/default/docker
```
Add a DOCKER_OPTS variable with the following options. These options are appended to the
docker daemon's run command.

text

DOCKER_OPTS="-D --tls=true --tlscert=/var/docker/server.pem --tlskey=/var/docker/serverkey.pem -H tcp://192.168.59.3:2376"

These options :

Enable -D (debug) mode
Set tls to true with the server certificate and key specified using --tlscert and --tlskey respectively
Listen for connections on tcp://192.168.59.3:2376

The command line reference has the complete list of daemon flags
with explanations.

Save and close the file.
Restart the docker daemon.
text
```
$ sudo restart docker
```
Verify that the docker daemon is running as specified with the ps command.
text
```
$ ps aux | grep docker | grep -v grep
```

Logs

By default logs for Upstart jobs are located in /var/log/upstart and the logs for docker daemon
can be located at /var/log/upstart/docker.log

text

$ tail -f /var/log/upstart/docker.log
INFO[0000] Loading containers: done.
INFO[0000] Docker daemon commit=1b09a95-unsupported graphdriver=aufs version=1.11.0-dev
INFO[0000] +job acceptconnections()
INFO[0000] -job acceptconnections() = OK (0)
INFO[0000] Daemon has completed initialization

CentOS / Red Hat Enterprise Linux / Fedora

As of 7.x, CentOS and RHEL use systemd as the process manager. As of 21, Fedora uses
systemd as its process manager.

After successfully installing Docker for CentOS/Red Hat Enterprise Linux/Fedora, you can check the running status in this way:

text

$ sudo systemctl status docker

Running Docker

You can start/stop/restart the docker daemon using

text

$ sudo systemctl start docker

$ sudo systemctl stop docker

$ sudo systemctl restart docker

If you want Docker to start at boot, you should also:

text

$ sudo systemctl enable docker

Configuring Docker

For CentOS 7.x and RHEL 7.x you can control and configure Docker with systemd.

Previously, for CentOS 6.x and RHEL 6.x you would configure the docker daemon in
the /etc/sysconfig/docker file on your system. You would do this by specifying
values in a other_args variable. For a short time in CentOS 7.x and RHEL 7.x you
would specify values in a OPTIONS variable. This is no longer recommended in favor
of using systemd directly.

For this section, we will use CentOS 7.x as an example to configure the docker daemon.

To configure Docker options:

Log into your host as a user with sudo or root privileges.
Create the /etc/systemd/system/docker.service.d directory.
text
```
$ sudo mkdir /etc/systemd/system/docker.service.d
```
Create a /etc/systemd/system/docker.service.d/docker.conf file.

Open the file with your favorite editor.

text

$ sudo vi /etc/systemd/system/docker.service.d/docker.conf

Override the ExecStart configuration from your docker.service file to customize
the docker daemon. To modify the ExecStart configuration you have to specify
an empty configuration followed by a new one as follows:

text

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -D --tls=true --tlscert=/var/docker/server.pem --tlskey=/var/docker/serverkey.pem -H tcp://192.168.59.3:2376

These options :

Enable -D (debug) mode
Set tls to true with the server certificate and key specified using --tlscert and --tlskey respectively
Listen for connections on tcp://192.168.59.3:2376

The command line reference has the complete list of daemon flags
with explanations.

Save and close the file.
Flush changes.
text
```
$ sudo systemctl daemon-reload
```
Restart the docker daemon.
text
```
$ sudo systemctl restart docker
```
Verify that the docker daemon is running as specified with the ps command.
text
```
$ ps aux | grep docker | grep -v grep
```

Logs

systemd has its own logging system called the journal. The logs for the docker daemon can
be viewed using journalctl -u docker

text

$ sudo journalctl -u docker
May 06 00:22:05 localhost.localdomain systemd[1]: Starting Docker Application Container Engine...
May 06 00:22:05 localhost.localdomain docker[2495]: time="2015-05-06T00:22:05Z" level="info" msg="+job serveapi(unix:///var/run/docker.sock)"
May 06 00:22:05 localhost.localdomain docker[2495]: time="2015-05-06T00:22:05Z" level="info" msg="Listening for HTTP on unix (/var/run/docker.sock)"
May 06 00:22:06 localhost.localdomain docker[2495]: time="2015-05-06T00:22:06Z" level="info" msg="+job init_networkdriver()"
May 06 00:22:06 localhost.localdomain docker[2495]: time="2015-05-06T00:22:06Z" level="info" msg="-job init_networkdriver() = OK (0)"
May 06 00:22:06 localhost.localdomain docker[2495]: time="2015-05-06T00:22:06Z" level="info" msg="Loading containers: start."
May 06 00:22:06 localhost.localdomain docker[2495]: time="2015-05-06T00:22:06Z" level="info" msg="Loading containers: done."
May 06 00:22:06 localhost.localdomain docker[2495]: time="2015-05-06T00:22:06Z" level="info" msg="Docker daemon commit=1b09a95-unsupported graphdriver=aufs version=1.11.0-dev"
May 06 00:22:06 localhost.localdomain docker[2495]: time="2015-05-06T00:22:06Z" level="info" msg="+job acceptconnections()"
May 06 00:22:06 localhost.localdomain docker[2495]: time="2015-05-06T00:22:06Z" level="info" msg="-job acceptconnections() = OK (0)"

Note: Using and configuring journal is an advanced topic and is beyond the scope of this article.

Daemonless Containers

Starting with Docker 1.12 containers can run without Docker or containerd running. This allows the
Docker daemon to exit, be upgraded, or recover from a crash without affecting running containers
on the system. To enable this functionality you need to add the --live-restore flag when
launching dockerd. This will ensure that Docker does not kill containers on graceful shutdown or
on restart leaving the containers running.

While the Docker daemon is down logging will still be captured, however, it will be capped at the kernel's pipe buffer size before the buffer fills up, blocking the process.
Docker will need to be restarted to flush these buffers.
You can modify the kernel's buffer size by changing /proc/sys/fs/pipe-max-size.

Admin Dsc (`docs/admin/dsc.md`)

Using PowerShell DSC

Windows PowerShell Desired State Configuration (DSC) is a configuration
management tool that extends the existing functionality of Windows PowerShell.
DSC uses a declarative syntax to define the state in which a target should be
configured. More information about PowerShell DSC can be found at
http://technet.microsoft.com/en-us/library/dn249912.aspx.

Requirements

To use this guide you'll need a Windows host with PowerShell v4.0 or newer.

The included DSC configuration script also uses the official PPA so
only an Ubuntu target is supported. The Ubuntu target must already have the
required OMI Server and PowerShell DSC for Linux providers installed. More
information can be found at https://github.com/MSFTOSSMgmt/WPSDSCLinux.
The source repository listed below also includes PowerShell DSC for Linux
installation and init scripts along with more detailed installation information.

Installation

The DSC configuration example source is available in the following repository:
https://github.com/anweiss/DockerClientDSC. It can be cloned with:

text

$ git clone https://github.com/anweiss/DockerClientDSC.git

Usage

The DSC configuration utilizes a set of shell scripts to determine whether or
not the specified Docker components are configured on the target node(s). The
source repository also includes a script (RunDockerClientConfig.ps1) that can
be used to establish the required CIM session(s) and execute the
Set-DscConfiguration cmdlet.

More detailed usage information can be found at
https://github.com/anweiss/DockerClientDSC.

Install Docker

The Docker installation configuration is equivalent to running:

text

apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys\
36A1D7869245C8950F966E92D8576A8BA88D21E9
sh -c "echo deb https://apt.dockerproject.org/repo ubuntu-trusty main\
> /etc/apt/sources.list.d/docker.list"
apt-get update
apt-get install docker-engine

Ensure that your current working directory is set to the DockerClientDSC
source and load the DockerClient configuration into the current PowerShell
session

powershell

. .\DockerClient.ps1

Generate the required DSC configuration .mof file for the targeted node

powershell

DockerClient -Hostname "myhost"

A sample DSC configuration data file has also been included and can be modified
and used in conjunction with or in place of the Hostname parameter:

powershell

DockerClient -ConfigurationData .\DockerConfigData.psd1

Start the configuration application process on the targeted node

powershell

.\RunDockerClientConfig.ps1 -Hostname "myhost"

The RunDockerClientConfig.ps1 script can also parse a DSC configuration data
file and execute configurations against multiple nodes as such:

powershell

.\RunDockerClientConfig.ps1 -ConfigurationData .\DockerConfigData.psd1

Images

Image configuration is equivalent to running: docker pull [image] or
docker rmi -f [IMAGE].

Using the same steps defined above, execute DockerClient with the Image
parameter and apply the configuration:

powershell

DockerClient -Hostname "myhost" -Image "node"
.\RunDockerClientConfig.ps1 -Hostname "myhost"

You can also configure the host to pull multiple images:

powershell

DockerClient -Hostname "myhost" -Image "node","mongo"
.\RunDockerClientConfig.ps1 -Hostname "myhost"

To remove images, use a hashtable as follows:

powershell

DockerClient -Hostname "myhost" -Image @{Name="node"; Remove=$true}
.\RunDockerClientConfig.ps1 -Hostname $hostname

Containers

Container configuration is equivalent to running:

text

docker run -d --name="[containername]" -p '[port]' -e '[env]' --link '[link]'\
'[image]' '[command]'

text

docker rm -f [containername]

To create or remove containers, you can use the Container parameter with one
or more hashtables. The hashtable(s) passed to this parameter can have the
following properties:

Name (required)
Image (required unless Remove property is set to $true)
Port
Env
Link
Command
Remove

For example, create a hashtable with the settings for your container:

powershell

$webContainer = @{Name="web"; Image="anweiss/docker-platynem"; Port="80:80"}

Then, using the same steps defined above, execute
DockerClient with the -Image and -Container parameters:

powershell

DockerClient -Hostname "myhost" -Image node -Container $webContainer
.\RunDockerClientConfig.ps1 -Hostname "myhost"

Existing containers can also be removed as follows:

powershell

$containerToRemove = @{Name="web"; Remove=$true}
DockerClient -Hostname "myhost" -Container $containerToRemove
.\RunDockerClientConfig.ps1 -Hostname "myhost"

Here is a hashtable with all of the properties that can be used to create a
container:

powershell

$containerProps = @{Name="web"; Image="node:latest"; Port="80:80"; `
Env="PORT=80"; Link="db:db"; Command="grunt"}

Admin Formatting (`docs/admin/formatting.md`)

Formatting reference

Docker uses Go templates to allow users manipulate the output format
of certain commands and log drivers. Each command a driver provides a detailed
list of elements they support in their templates:

Template functions

Docker provides a set of basic functions to manipulate template elements.
This is the complete list of the available functions with examples:

Join

Join concatenates a list of strings to create a single string.
It puts a separator between each element in the list.

text

$ docker ps --format '{{join .Names " or "}}'

Json

Json encodes an element as a json string.

text

$ docker inspect --format '{{json .Mounts}}' container

Lower

Lower turns a string into its lower case representation.

text

$ docker inspect --format "{{lower .Name}}" container

Split

Split slices a string into a list of strings separated by a separator.

text

# docker inspect --format '{{split (join .Names "/") "/"}}' container

Title

Title capitalizes a string.

text

$ docker inspect --format "{{title .Name}}" container

Upper

Upper turms a string into its upper case representation.

text

$ docker inspect --format "{{upper .Name}}" container

Admin Host Integration (`docs/admin/host_integration.md`)

Automatically start containers

As of Docker 1.2,
restart policies are the
built-in Docker mechanism for restarting containers when they exit. If set,
restart policies will be used when the Docker daemon starts up, as typically
happens after a system boot. Restart policies will ensure that linked containers
are started in the correct order.

If restart policies don't suit your needs (i.e., you have non-Docker processes
that depend on Docker containers), you can use a process manager like
upstart,
systemd or
supervisor instead.

Using a process manager

Docker does not set any restart policies by default, but be aware that they will
conflict with most process managers. So don't set restart policies if you are
using a process manager.

When you have finished setting up your image and are happy with your
running container, you can then attach a process manager to manage it.
When you run docker start -a, Docker will automatically attach to the
running container, or start it if needed and forward all signals so that
the process manager can detect when a container stops and correctly
restart it.

Here are a few sample scripts for systemd and upstart to integrate with
Docker.

Examples

The examples below show configuration files for two popular process managers,
upstart and systemd. In these examples, we'll assume that we have already
created a container to run Redis with --name=redis_server. These files define
a new service that will be started after the docker daemon service has started.

upstart

text

description "Redis container"
author "Me"
start on filesystem and started docker
stop on runlevel [!2345]
respawn
script
  /usr/bin/docker start -a redis_server
end script

systemd

text

[Unit]
Description=Redis container
Requires=docker.service
After=docker.service

[Service]
Restart=always
ExecStart=/usr/bin/docker start -a redis_server
ExecStop=/usr/bin/docker stop -t 2 redis_server

[Install]
WantedBy=local.target

If you need to pass options to the redis container (such as --env),
then you'll need to use docker run rather than docker start. This will
create a new container every time the service is started, which will be stopped
and removed when the service is stopped.

text

[Service]
...
ExecStart=/usr/bin/docker run --env foo=bar --name redis_server redis
ExecStop=/usr/bin/docker stop -t 2 redis_server ; /usr/bin/docker rm -f redis_server
...

Admin Puppet (`docs/admin/puppet.md`)

Using Puppet

Note: Please note this is a community contributed installation path. The
only official installation is using the
Ubuntu installation
path. This version may sometimes be out of date.

Requirements

To use this guide you'll need a working installation of Puppet from
Puppet Labs .

The module also currently uses the official PPA so only works with
Ubuntu.

Installation

The module is available on the Puppet
Forge and can be
installed using the built-in module tool.

text

$ puppet module install garethr/docker

It can also be found on
GitHub if you would rather
download the source.

Usage

The module provides a puppet class for installing Docker and two defined
types for managing images and containers.

Installation

text

include 'docker'

Images

The next step is probably to install a Docker image. For this, we have a
defined type which can be used like so:

text

docker::image { 'ubuntu': }

This is equivalent to running:

text

$ docker pull ubuntu

Note that it will only be downloaded if an image of that name does not
already exist. This is downloading a large binary so on first run can
take a while. For that reason this define turns off the default 5 minute
timeout for the exec type. Note that you can also remove images you no
longer need with:

text

docker::image { 'ubuntu':
  ensure => 'absent',
}

Containers

Now you have an image where you can run commands within a container
managed by Docker.

text

docker::run { 'helloworld':
  image   => 'ubuntu',
  command => '/bin/sh -c "while true; do echo hello world; sleep 1; done"',
}

This is equivalent to running the following command, but under upstart:

text

$ docker run -d ubuntu /bin/sh -c "while true; do echo hello world; sleep 1; done"

Run also contains a number of optional parameters:

text

docker::run { 'helloworld':
  image        => 'ubuntu',
  command      => '/bin/sh -c "while true; do echo hello world; sleep 1; done"',
  ports        => ['4444', '4555'],
  volumes      => ['/var/lib/couchdb', '/var/log'],
  volumes_from => '6446ea52fbc9',
  memory_limit => 10485760, # bytes
  username     => 'example',
  hostname     => 'example.com',
  env          => ['FOO=BAR', 'FOO2=BAR2'],
  dns          => ['8.8.8.8', '8.8.4.4'],
}

Note:
The ports, env, dns and volumes attributes can be set with either a single
string or as above with an array of values.

Admin Registry Mirror (`docs/admin/registry_mirror.md`)

Run a local registry mirror

The original content was deprecated. An archived
version is available in
the 1.7 documentation. For information about configuring mirrors with the latest
Docker Registry version, please file a support request with the Distribution
project.

Admin Runmetrics (`docs/admin/runmetrics.md`)

Runtime metrics

Docker stats

You can use the docker stats command to live stream a container's
runtime metrics. The command supports CPU, memory usage, memory limit,
and network IO metrics.

The following is a sample output from the docker stats command

text

$ docker stats redis1 redis2
CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O
redis1              0.07%               796 KB / 64 MB        1.21%               788 B / 648 B       3.568 MB / 512 KB
redis2              0.07%               2.746 MB / 64 MB      4.29%               1.266 KB / 648 B    12.4 MB / 0 B

The docker stats reference page has
more details about the docker stats command.

Control groups

Linux Containers rely on control groups
which not only track groups of processes, but also expose metrics about
CPU, memory, and block I/O usage. You can access those metrics and
obtain network usage metrics as well. This is relevant for "pure" LXC
containers, as well as for Docker containers.

Control groups are exposed through a pseudo-filesystem. In recent
distros, you should find this filesystem under /sys/fs/cgroup. Under
that directory, you will see multiple sub-directories, called devices,
freezer, blkio, etc.; each sub-directory actually corresponds to a different
cgroup hierarchy.

On older systems, the control groups might be mounted on /cgroup, without
distinct hierarchies. In that case, instead of seeing the sub-directories,
you will see a bunch of files in that directory, and possibly some directories
corresponding to existing containers.

To figure out where your control groups are mounted, you can run:

text

$ grep cgroup /proc/mounts

Enumerating cgroups

You can look into /proc/cgroups to see the different control group subsystems
known to the system, the hierarchy they belong to, and how many groups they contain.

You can also look at /proc//cgroup to see which control groups a process
belongs to. The control group will be shown as a path relative to the root of
the hierarchy mountpoint; e.g., / means “this process has not been assigned into
a particular group”, while /lxc/pumpkin means that the process is likely to be
a member of a container named pumpkin.

Finding the cgroup for a given container

For each container, one cgroup will be created in each hierarchy. On
older systems with older versions of the LXC userland tools, the name of
the cgroup will be the name of the container. With more recent versions
of the LXC tools, the cgroup will be lxc/<container_name>.

For Docker containers using cgroups, the container name will be the full
ID or long ID of the container. If a container shows up as ae836c95b4c3
in docker ps, its long ID might be something like
ae836c95b4c3c9e9179e0e91015512da89fdec91612f63cebae57df9a5444c79. You can
look it up with docker inspect or docker ps --no-trunc.

Putting everything together to look at the memory metrics for a Docker
container, take a look at /sys/fs/cgroup/memory/docker/<longid>/.

Metrics from cgroups: memory, CPU, block I/O

For each subsystem (memory, CPU, and block I/O), you will find one or
more pseudo-files containing statistics.

Memory metrics: `memory.stat`

Memory metrics are found in the "memory" cgroup. Note that the memory
control group adds a little overhead, because it does very fine-grained
accounting of the memory usage on your host. Therefore, many distros
chose to not enable it by default. Generally, to enable it, all you have
to do is to add some kernel command-line parameters:
cgroup_enable=memory swapaccount=1.

The metrics are in the pseudo-file memory.stat.
Here is what it will look like:

text

cache 11492564992
rss 1930993664
mapped_file 306728960
pgpgin 406632648
pgpgout 403355412
swap 0
pgfault 728281223
pgmajfault 1724
inactive_anon 46608384
active_anon 1884520448
inactive_file 7003344896
active_file 4489052160
unevictable 32768
hierarchical_memory_limit 9223372036854775807
hierarchical_memsw_limit 9223372036854775807
total_cache 11492564992
total_rss 1930993664
total_mapped_file 306728960
total_pgpgin 406632648
total_pgpgout 403355412
total_swap 0
total_pgfault 728281223
total_pgmajfault 1724
total_inactive_anon 46608384
total_active_anon 1884520448
total_inactive_file 7003344896
total_active_file 4489052160
total_unevictable 32768

The first half (without the total_ prefix) contains statistics relevant
to the processes within the cgroup, excluding sub-cgroups. The second half
(with the total_ prefix) includes sub-cgroups as well.

Some metrics are "gauges", i.e., values that can increase or decrease
(e.g., swap, the amount of swap space used by the members of the cgroup).
Some others are "counters", i.e., values that can only go up, because
they represent occurrences of a specific event (e.g., pgfault, which
indicates the number of page faults which happened since the creation of
the cgroup; this number can never decrease).

cache:
the amount of memory used by the processes of this control group
that can be associated precisely with a block on a block device.
When you read from and write to files on disk, this amount will
increase. This will be the case if you use "conventional" I/O
(open, read,
write syscalls) as well as mapped files (with
mmap). It also accounts for the memory used by
tmpfs mounts, though the reasons are unclear.
rss:
the amount of memory that doesn't correspond to anything on disk:
stacks, heaps, and anonymous memory maps.
mapped_file:
indicates the amount of memory mapped by the processes in the
control group. It doesn't give you information about how much
memory is used; it rather tells you how it is used.
pgfault and pgmajfault:
indicate the number of times that a process of the cgroup triggered
a "page fault" and a "major fault", respectively. A page fault
happens when a process accesses a part of its virtual memory space
which is nonexistent or protected. The former can happen if the
process is buggy and tries to access an invalid address (it will
then be sent a SIGSEGV signal, typically
killing it with the famous Segmentation fault
message). The latter can happen when the process reads from a memory
zone which has been swapped out, or which corresponds to a mapped
file: in that case, the kernel will load the page from disk, and let
the CPU complete the memory access. It can also happen when the
process writes to a copy-on-write memory zone: likewise, the kernel
will preempt the process, duplicate the memory page, and resume the
write operation on the process` own copy of the page. "Major" faults
happen when the kernel actually has to read the data from disk. When
it just has to duplicate an existing page, or allocate an empty
page, it's a regular (or "minor") fault.
swap:
the amount of swap currently used by the processes in this cgroup.
active_anon and inactive_anon:
the amount of anonymous memory that has been identified has
respectively active and inactive by the kernel. "Anonymous"
memory is the memory that is not linked to disk pages. In other
words, that's the equivalent of the rss counter described above. In
fact, the very definition of the rss counter is active_anon +
inactive_anon - tmpfs (where tmpfs is the amount of memory
used up by tmpfs filesystems mounted by this
control group). Now, what's the difference between "active" and
"inactive"? Pages are initially "active"; and at regular intervals,
the kernel sweeps over the memory, and tags some pages as
"inactive". Whenever they are accessed again, they are immediately
retagged "active". When the kernel is almost out of memory, and time
comes to swap out to disk, the kernel will swap "inactive" pages.
active_file and inactive_file:
cache memory, with active and inactive similar to the anon
memory above. The exact formula is cache = active_file +
inactive_file + tmpfs. The exact rules used by the kernel
to move memory pages between active and inactive sets are different
from the ones used for anonymous memory, but the general principle
is the same. Note that when the kernel needs to reclaim memory, it
is cheaper to reclaim a clean (=non modified) page from this pool,
since it can be reclaimed immediately (while anonymous pages and
dirty/modified pages have to be written to disk first).
unevictable:
the amount of memory that cannot be reclaimed; generally, it will
account for memory that has been "locked" with mlock.
It is often used by crypto frameworks to make sure that
secret keys and other sensitive material never gets swapped out to
disk.
memory and memsw limits:
These are not really metrics, but a reminder of the limits applied
to this cgroup. The first one indicates the maximum amount of
physical memory that can be used by the processes of this control
group; the second one indicates the maximum amount of RAM+swap.

Accounting for memory in the page cache is very complex. If two
processes in different control groups both read the same file
(ultimately relying on the same blocks on disk), the corresponding
memory charge will be split between the control groups. It's nice, but
it also means that when a cgroup is terminated, it could increase the
memory usage of another cgroup, because they are not splitting the cost
anymore for those memory pages.

CPU metrics: `cpuacct.stat`

Now that we've covered memory metrics, everything else will look very
simple in comparison. CPU metrics will be found in the
cpuacct controller.

For each container, you will find a pseudo-file cpuacct.stat,
containing the CPU usage accumulated by the processes of the container,
broken down between user and system time. If you're not familiar
with the distinction, user is the time during which the processes were
in direct control of the CPU (i.e., executing process code), and system
is the time during which the CPU was executing system calls on behalf of
those processes.

Those times are expressed in ticks of 1/100th of a second. Actually,
they are expressed in "user jiffies". There are USER_HZ
"jiffies" per second, and on x86 systems,
USER_HZ is 100. This used to map exactly to the
number of scheduler "ticks" per second; but with the advent of higher
frequency scheduling, as well as tickless kernels, the number of kernel ticks
wasn't relevant anymore. It stuck around anyway, mainly for legacy and
compatibility reasons.

Block I/O metrics

Block I/O is accounted in the blkio controller.
Different metrics are scattered across different files. While you can
find in-depth details in the blkio-controller
file in the kernel documentation, here is a short list of the most
relevant ones:

blkio.sectors:
contain the number of 512-bytes sectors read and written by the
processes member of the cgroup, device by device. Reads and writes
are merged in a single counter.
blkio.io_service_bytes:
indicates the number of bytes read and written by the cgroup. It has
4 counters per device, because for each device, it differentiates
between synchronous vs. asynchronous I/O, and reads vs. writes.
blkio.io_serviced:
the number of I/O operations performed, regardless of their size. It
also has 4 counters per device.
blkio.io_queued:
indicates the number of I/O operations currently queued for this
cgroup. In other words, if the cgroup isn't doing any I/O, this will
be zero. Note that the opposite is not true. In other words, if
there is no I/O queued, it does not mean that the cgroup is idle
(I/O-wise). It could be doing purely synchronous reads on an
otherwise quiescent device, which is therefore able to handle them
immediately, without queuing. Also, while it is helpful to figure
out which cgroup is putting stress on the I/O subsystem, keep in
mind that it is a relative quantity. Even if a process group does
not perform more I/O, its queue size can increase just because the
device load increases because of other devices.

Network metrics

Network metrics are not exposed directly by control groups. There is a
good explanation for that: network interfaces exist within the context
of network namespaces. The kernel could probably accumulate metrics
about packets and bytes sent and received by a group of processes, but
those metrics wouldn't be very useful. You want per-interface metrics
(because traffic happening on the local lo
interface doesn't really count). But since processes in a single cgroup
can belong to multiple network namespaces, those metrics would be harder
to interpret: multiple network namespaces means multiple lo
interfaces, potentially multiple eth0
interfaces, etc.; so this is why there is no easy way to gather network
metrics with control groups.

Instead we can gather network metrics from other sources:

IPtables

IPtables (or rather, the netfilter framework for which iptables is just
an interface) can do some serious accounting.

For instance, you can setup a rule to account for the outbound HTTP
traffic on a web server:

text

$ iptables -I OUTPUT -p tcp --sport 80

There is no -j or -g flag,
so the rule will just count matched packets and go to the following
rule.

Later, you can check the values of the counters, with:

text

$ iptables -nxvL OUTPUT

Technically, -n is not required, but it will
prevent iptables from doing DNS reverse lookups, which are probably
useless in this scenario.

Counters include packets and bytes. If you want to setup metrics for
container traffic like this, you could execute a for
loop to add two iptables rules per
container IP address (one in each direction), in the FORWARD
chain. This will only meter traffic going through the NAT
layer; you will also have to add traffic going through the userland
proxy.

Then, you will need to check those counters on a regular basis. If you
happen to use collectd, there is a nice plugin
to automate iptables counters collection.

Interface-level counters

Since each container has a virtual Ethernet interface, you might want to
check directly the TX and RX counters of this interface. You will notice
that each container is associated to a virtual Ethernet interface in
your host, with a name like vethKk8Zqi. Figuring
out which interface corresponds to which container is, unfortunately,
difficult.

But for now, the best way is to check the metrics from within the
containers. To accomplish this, you can run an executable from the host
environment within the network namespace of a container using ip-netns
magic.

The ip-netns exec command will let you execute any
program (present in the host system) within any network namespace
visible to the current process. This means that your host will be able
to enter the network namespace of your containers, but your containers
won't be able to access the host, nor their sibling containers.
Containers will be able to “see” and affect their sub-containers,
though.

The exact format of the command is:

text

$ ip netns exec <nsname> <command...>

For example:

text

$ ip netns exec mycontainer netstat -i

ip netns finds the "mycontainer" container by
using namespaces pseudo-files. Each process belongs to one network
namespace, one PID namespace, one mnt namespace,
etc., and those namespaces are materialized under
/proc//ns/. For example, the network
namespace of PID 42 is materialized by the pseudo-file
/proc/42/ns/net.

When you run ip netns exec mycontainer ..., it
expects /var/run/netns/mycontainer to be one of
those pseudo-files. (Symlinks are accepted.)

In other words, to execute a command within the network namespace of a
container, we need to:

Find out the PID of any process within the container that we want to investigate;
Create a symlink from /var/run/netns/<somename> to /proc//ns/net
Execute ip netns exec <somename> ....

Please review Enumerating Cgroups to learn how to find
the cgroup of a process running in the container of which you want to
measure network usage. From there, you can examine the pseudo-file named
tasks, which contains the PIDs that are in the
control group (i.e., in the container). Pick any one of them.

Putting everything together, if the "short ID" of a container is held in
the environment variable $CID, then you can do this:

text

$ TASKS=/sys/fs/cgroup/devices/docker/$CID*/tasks
$ PID=$(head -n 1 $TASKS)
$ mkdir -p /var/run/netns
$ ln -sf /proc/$PID/ns/net /var/run/netns/$CID
$ ip netns exec $CID netstat -i

Tips for high-performance metric collection

Note that running a new process each time you want to update metrics is
(relatively) expensive. If you want to collect metrics at high
resolutions, and/or over a large number of containers (think 1000
containers on a single host), you do not want to fork a new process each
time.

Here is how to collect metrics from a single process. You will have to
write your metric collector in C (or any language that lets you do
low-level system calls). You need to use a special system call,
setns(), which lets the current process enter any
arbitrary namespace. It requires, however, an open file descriptor to
the namespace pseudo-file (remember: that's the pseudo-file in
/proc//ns/net).

However, there is a catch: you must not keep this file descriptor open.
If you do, when the last process of the control group exits, the
namespace will not be destroyed, and its network resources (like the
virtual interface of the container) will stay around for ever (or until
you close that file descriptor).

The right approach would be to keep track of the first PID of each
container, and re-open the namespace pseudo-file each time.

Collecting metrics when a container exits

Sometimes, you do not care about real time metric collection, but when a
container exits, you want to know how much CPU, memory, etc. it has
used.

Docker makes this difficult because it relies on lxc-start, which
carefully cleans up after itself, but it is still possible. It is
usually easier to collect metrics at regular intervals (e.g., every
minute, with the collectd LXC plugin) and rely on that instead.

But, if you'd still like to gather the stats when a container stops,
here is how:

For each container, start a collection process, and move it to the
control groups that you want to monitor by writing its PID to the tasks
file of the cgroup. The collection process should periodically re-read
the tasks file to check if it's the last process of the control group.
(If you also want to collect network statistics as explained in the
previous section, you should also move the process to the appropriate
network namespace.)

When the container exits, lxc-start will try to
delete the control groups. It will fail, since the control group is
still in use; but that's fine. You process should now detect that it is
the only one remaining in the group. Now is the right time to collect
all the metrics you need!

Finally, your process should move itself back to the root control group,
and remove the container control group. To remove a control group, just
rmdir its directory. It's counter-intuitive to
rmdir a directory as it still contains files; but
remember that this is a pseudo-filesystem, so usual rules don't apply.
After the cleanup is done, the collection process can exit safely.

Admin Systemd (`docs/admin/systemd.md`)

Control and configure Docker with systemd

Many Linux distributions use systemd to start the Docker daemon. This document
shows a few examples of how to customize Docker's settings.

Starting the Docker daemon

Once Docker is installed, you will need to start the Docker daemon.

text

$ sudo systemctl start docker
# or on older distributions, you may need to use
$ sudo service docker start

If you want Docker to start at boot, you should also:

text

$ sudo systemctl enable docker
# or on older distributions, you may need to use
$ sudo chkconfig docker on

Custom Docker daemon options

There are a number of ways to configure the daemon flags and environment variables
for your Docker daemon.

The recommended way is to use a systemd drop-in file (as described in
the systemd.unit
documentation). These are local files named <something>.conf in the
/etc/systemd/system/docker.service.d directory. This could also be
/etc/systemd/system/docker.service, which also works for overriding
the defaults from /lib/systemd/system/docker.service.

However, if you had previously used a package which had an
EnvironmentFile (often pointing to /etc/sysconfig/docker) then for
backwards compatibility, you drop a file with a .conf extension into
the /etc/systemd/system/docker.service.d directory including the
following:

text

[Service]
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// $OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $BLOCK_REGISTRY \
          $INSECURE_REGISTRY

To check if the docker.service uses an EnvironmentFile:

text

$ systemctl show docker | grep EnvironmentFile
EnvironmentFile=-/etc/sysconfig/docker (ignore_errors=yes)

Alternatively, find out where the service file is located:

text

$ systemctl show --property=FragmentPath docker
FragmentPath=/usr/lib/systemd/system/docker.service
$ grep EnvironmentFile /usr/lib/systemd/system/docker.service
EnvironmentFile=-/etc/sysconfig/docker

You can customize the Docker daemon options using override files as explained in the
HTTP Proxy example below. The files located in /usr/lib/systemd/system
or /lib/systemd/system contain the default options and should not be edited.

Runtime directory and storage driver

You may want to control the disk space used for Docker images, containers
and volumes by moving it to a separate partition.

In this example, we'll assume that your docker.service file looks something like:

text

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/docker daemon -H fd://
LimitNOFILE=1048576
LimitNPROC=1048576
TasksMax=1048576

[Install]
Also=docker.socket

This will allow us to add extra flags via a drop-in file (mentioned above) by
placing a file containing the following in the /etc/systemd/system/docker.service.d
directory:

text

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --graph="/mnt/docker-data" --storage-driver=overlay

You can also set other environment variables in this file, for example, the
HTTP_PROXY environment variables described below.

To modify the ExecStart configuration, specify an empty configuration followed
by a new configuration as follows:

text

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --bip=172.17.42.1/16

If you fail to specify an empty configuration, Docker reports an error such as:

text

docker.service has more than one ExecStart= setting, which is only allowed for Type=oneshot services. Refusing.

HTTP proxy

This example overrides the default docker.service file.

If you are behind an HTTP proxy server, for example in corporate settings,
you will need to add this configuration in the Docker systemd service file.

First, create a systemd drop-in directory for the docker service:

text

mkdir /etc/systemd/system/docker.service.d

Now create a file called /etc/systemd/system/docker.service.d/http-proxy.conf
that adds the HTTP_PROXY environment variable:

text

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:80/"

If you have internal Docker registries that you need to contact without
proxying you can specify them via the NO_PROXY environment variable:

text

Environment="HTTP_PROXY=http://proxy.example.com:80/" "NO_PROXY=localhost,127.0.0.1,docker-registry.somecorporation.com"

Flush changes:

text

$ sudo systemctl daemon-reload

Verify that the configuration has been loaded:

text

$ systemctl show --property=Environment docker
Environment=HTTP_PROXY=http://proxy.example.com:80/

Restart Docker:

text

$ sudo systemctl restart docker

Manually creating the systemd unit files

When installing the binary without a package, you may want
to integrate Docker with systemd. For this, simply install the two unit files
(service and socket) from the github
repository
to /etc/systemd/system.

Admin Using Supervisord (`docs/admin/using_supervisord.md`)

Using Supervisor with Docker

Note:

If you don't like sudo then see Giving non-root
access

Traditionally a Docker container runs a single process when it is launched, for
example an Apache daemon or a SSH server daemon. Often though you want to run
more than one process in a container. There are a number of ways you can
achieve this ranging from using a simple Bash script as the value of your
container's CMD instruction to installing a process management tool.

In this example you're going to make use of the process management tool,
Supervisor, to manage multiple processes in a
container. Using Supervisor allows you to better control, manage, and restart
the processes inside the container. To demonstrate this we're going to install
and manage both an SSH daemon and an Apache daemon.

Creating a Dockerfile

Let's start by creating a basic Dockerfile for our new image.

dockerfile

FROM ubuntu:16.04
MAINTAINER [email protected]

Installing Supervisor

You can now install the SSH and Apache daemons as well as Supervisor in the
container.

dockerfile

RUN apt-get update && apt-get install -y openssh-server apache2 supervisor
RUN mkdir -p /var/lock/apache2 /var/run/apache2 /var/run/sshd /var/log/supervisor

The first RUN instruction installs the openssh-server, apache2 and
supervisor (which provides the Supervisor daemon) packages. The next RUN
instruction creates four new directories that are needed to run the SSH daemon
and Supervisor.

Adding Supervisor's configuration file

Now let's add a configuration file for Supervisor. The default file is called
supervisord.conf and is located in /etc/supervisor/conf.d/.

dockerfile

COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf

Let's see what is inside the supervisord.conf file.

ini

[supervisord]
nodaemon=true

[program:sshd]
command=/usr/sbin/sshd -D

[program:apache2]
command=/bin/bash -c "source /etc/apache2/envvars && exec /usr/sbin/apache2 -DFOREGROUND"

The supervisord.conf configuration file contains directives that configure
Supervisor and the processes it manages. The first block [supervisord]
provides configuration for Supervisor itself. The nodaemon directive is used,
which tells Supervisor to run interactively rather than daemonize.

The next two blocks manage the services we wish to control. Each block controls
a separate process. The blocks contain a single directive, command, which
specifies what command to run to start each process.

Exposing ports and running Supervisor

Now let's finish the Dockerfile by exposing some required ports and
specifying the CMD instruction to start Supervisor when our container
launches.

dockerfile

EXPOSE 22 80
CMD ["/usr/bin/supervisord"]

These instructions tell Docker that ports 22 and 80 are exposed by the
container and that the /usr/bin/supervisord binary should be executed when
the container launches.

Building our image

Your completed Dockerfile now looks like this:

dockerfile

FROM ubuntu:16.04
MAINTAINER [email protected]

RUN apt-get update && apt-get install -y openssh-server apache2 supervisor
RUN mkdir -p /var/lock/apache2 /var/run/apache2 /var/run/sshd /var/log/supervisor

COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf

EXPOSE 22 80
CMD ["/usr/bin/supervisord"]

And your supervisord.conf file looks like this;

ini

[supervisord]
nodaemon=true

[program:sshd]
command=/usr/sbin/sshd -D

[program:apache2]
command=/bin/bash -c "source /etc/apache2/envvars && exec /usr/sbin/apache2 -DFOREGROUND"

You can now build the image using this command;

bash

$ docker build -t mysupervisord .

Running your Supervisor container

Once you have built your image you can launch a container from it.

bash

$ docker run -p 22 -p 80 -t -i mysupervisord
2013-11-25 18:53:22,312 CRIT Supervisor running as root (no user in config file)
2013-11-25 18:53:22,312 WARN Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
2013-11-25 18:53:22,342 INFO supervisord started with pid 1
2013-11-25 18:53:23,346 INFO spawned: 'sshd' with pid 6
2013-11-25 18:53:23,349 INFO spawned: 'apache2' with pid 7
...

You launched a new container interactively using the docker run command.
That container has run Supervisor and launched the SSH and Apache daemons with
it. We've specified the -p flag to expose ports 22 and 80. From here we can
now identify the exposed ports and connect to one or both of the SSH and Apache
daemons.

Examples Apt Cacher Ng (`docs/examples/apt-cacher-ng.md`)

Dockerizing an apt-cacher-ng service

Note:

If you don't like sudo then see Giving non-root
access.

If you're using OS X or docker via TCP then you shouldn't use
sudo.

When you have multiple Docker servers, or build unrelated Docker
containers which can't make use of the Docker build cache, it can be
useful to have a caching proxy for your packages. This container makes
the second download of any package almost instant.

Use the following Dockerfile:

text

#
# Build: docker build -t apt-cacher .
# Run: docker run -d -p 3142:3142 --name apt-cacher-run apt-cacher
#
# and then you can run containers with:
#   docker run -t -i --rm -e http_proxy http://dockerhost:3142/ debian bash
#
# Here, `dockerhost` is the IP address or FQDN of a host running the Docker daemon
# which acts as an APT proxy server.
FROM        ubuntu
MAINTAINER  [email protected]

VOLUME      ["/var/cache/apt-cacher-ng"]
RUN     apt-get update && apt-get install -y apt-cacher-ng

EXPOSE      3142
CMD     chmod 777 /var/cache/apt-cacher-ng && /etc/init.d/apt-cacher-ng start && tail -f /var/log/apt-cacher-ng/*

To build the image using:

text

$ docker build -t eg_apt_cacher_ng .

Then run it, mapping the exposed port to one on the host

text

$ docker run -d -p 3142:3142 --name test_apt_cacher_ng eg_apt_cacher_ng

To see the logfiles that are tailed in the default command, you can
use:

text

$ docker logs -f test_apt_cacher_ng

To get your Debian-based containers to use the proxy, you have following options

Add an apt Proxy setting
echo 'Acquire::http { Proxy "http://dockerhost:3142"; };' >> /etc/apt/conf.d/01proxy
Set an environment variable:
http_proxy=http://dockerhost:3142/
Change your sources.list entries to start with
http://dockerhost:3142/
Link Debian-based containers to the APT proxy container using --link
Create a custom network of an APT proxy container with Debian-based containers.

Option 1 injects the settings safely into your apt configuration in
a local version of a common base:

text

FROM ubuntu
RUN  echo 'Acquire::http { Proxy "http://dockerhost:3142"; };' >> /etc/apt/apt.conf.d/01proxy
RUN apt-get update && apt-get install -y vim git

# docker build -t my_ubuntu .

Option 2 is good for testing, but will break other HTTP clients
which obey http_proxy, such as curl, wget and others:

text

$ docker run --rm -t -i -e http_proxy=http://dockerhost:3142/ debian bash

Option 3 is the least portable, but there will be times when you
might need to do it and you can do it from your Dockerfile
too.

Option 4 links Debian-containers to the proxy server using following command:

text

$ docker run -i -t --link test_apt_cacher_ng:apt_proxy -e http_proxy=http://apt_proxy:3142/ debian bash

Option 5 creates a custom network of APT proxy server and Debian-based containers:

text

$ docker network create mynetwork
$ docker run -d -p 3142:3142 --net=mynetwork --name test_apt_cacher_ng eg_apt_cacher_ng
$ docker run --rm -it --net=mynetwork -e http_proxy=http://test_apt_cacher_ng:3142/ debian bash

Apt-cacher-ng has some tools that allow you to manage the repository,
and they can be used by leveraging the VOLUME
instruction, and the image we built to run the service:

text

$ docker run --rm -t -i --volumes-from test_apt_cacher_ng eg_apt_cacher_ng bash

$$ /usr/lib/apt-cacher-ng/distkill.pl
Scanning /var/cache/apt-cacher-ng, please wait...
Found distributions:
bla, taggedcount: 0
     1. precise-security (36 index files)
     2. wheezy (25 index files)
     3. precise-updates (36 index files)
     4. precise (36 index files)
     5. wheezy-updates (18 index files)

Found architectures:
     6. amd64 (36 index files)
     7. i386 (24 index files)

WARNING: The removal action may wipe out whole directories containing
         index files. Select d to see detailed list.

(Number nn: tag distribution or architecture nn; 0: exit; d: show details; r: remove tagged; q: quit): q

Finally, clean up after your test by stopping and removing the
container, and then removing the image.

text

$ docker stop test_apt_cacher_ng
$ docker rm test_apt_cacher_ng
$ docker rmi eg_apt_cacher_ng

Examples Couchbase (`docs/examples/couchbase.md`)

Dockerizing a Couchbase service

This example shows how to start a Couchbase server using Docker Compose, configure it using its REST API, and query it.

Couchbase is an open source, document-oriented NoSQL database for modern web, mobile, and IoT applications. It is designed for ease of development and Internet-scale performance.

Start Couchbase server

Couchbase Docker images are published at Docker Hub.

Start Couchbase server as:

text

docker run -d --name db -p 8091-8093:8091-8093 -p 11210:11210 couchbase

The purpose of each port exposed is explained at Couchbase Developer Portal - Network Configuration.

Logs can be seen as:

text

docker logs db
Starting Couchbase Server -- Web UI available at http://<ip>:8091

Note: The examples on this page assume that the Docker Host
is reachable on 192.168.99.100. Substitute 192.168.99.100 with
the actual IP address of your Docker Host. If you're running
Docker using Docker machine, you can obtain the IP address
of the Docker host using docker-machine ip <MACHINE-NAME>.

The logs show that Couchbase console can be accessed at http://192.168.99.100:8091. The default username is Administrator and the password is password.

Configure Couchbase Docker container

By default, Couchbase server needs to be configured using the console before it can be used. This can be simplified by configuring it using the REST API.

Configure memory for Data and Index service

Data, Query and Index are three different services that can be configured on a Couchbase instance. Each service has different operating needs. For example, Query is CPU intensive operation and so requires a faster processor. Index is disk heavy and so requires a faster solid state drive. Data needs to be read/written fast and so requires more memory.

Memory needs to be configured for Data and Index service only.

text

curl -v -X POST http://192.168.99.100:8091/pools/default -d memoryQuota=300 -d indexMemoryQuota=300
* Hostname was NOT found in DNS cache
*   Trying 192.168.99.100...
* Connected to 192.168.99.100 (192.168.99.100) port 8091 (#0)
> POST /pools/default HTTP/1.1
> User-Agent: curl/7.37.1
> Host: 192.168.99.100:8091
> Accept: */*
> Content-Length: 36
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 36 out of 36 bytes
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Basic realm="Couchbase Server Admin / REST"
* Server Couchbase Server is not blacklisted
< Server: Couchbase Server
< Pragma: no-cache
< Date: Wed, 25 Nov 2015 22:48:16 GMT
< Content-Length: 0
< Cache-Control: no-cache
<
* Connection #0 to host 192.168.99.100 left intact

The command shows an HTTP POST request to the REST endpoint /pools/default. The host is the IP address of the Docker machine. The port is the exposed port of Couchbase server. The memory and index quota for the server are passed in the request.

Configure Data, Query, and Index services

All three services, or only one of them, can be configured on each instance. This allows different Couchbase instances to use affinities and setup services accordingly. For example, if Docker host is running a machine with solid-state drive then only Data service can be started.

text

curl -v http://192.168.99.100:8091/node/controller/setupServices -d 'services=kv%2Cn1ql%2Cindex'
* Hostname was NOT found in DNS cache
*   Trying 192.168.99.100...
* Connected to 192.168.99.100 (192.168.99.100) port 8091 (#0)
> POST /node/controller/setupServices HTTP/1.1
> User-Agent: curl/7.37.1
> Host: 192.168.99.100:8091
> Accept: */*
> Content-Length: 26
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 26 out of 26 bytes
< HTTP/1.1 200 OK
* Server Couchbase Server is not blacklisted
< Server: Couchbase Server
< Pragma: no-cache
< Date: Wed, 25 Nov 2015 22:49:51 GMT
< Content-Length: 0
< Cache-Control: no-cache
<
* Connection #0 to host 192.168.99.100 left intact

The command shows an HTTP POST request to the REST endpoint /node/controller/setupServices. The command shows that all three services are configured for the Couchbase server. The Data service is identified by kv, Query service is identified by n1ql and Index service identified by index.

Setup credentials for the Couchbase server

Sets the username and password credentials that will subsequently be used for managing the Couchbase server.

text

curl -v -X POST http://192.168.99.100:8091/settings/web -d port=8091 -d username=Administrator -d password=password
* Hostname was NOT found in DNS cache
*   Trying 192.168.99.100...
* Connected to 192.168.99.100 (192.168.99.100) port 8091 (#0)
> POST /settings/web HTTP/1.1
> User-Agent: curl/7.37.1
> Host: 192.168.99.100:8091
> Accept: */*
> Content-Length: 50
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 50 out of 50 bytes
< HTTP/1.1 200 OK
* Server Couchbase Server is not blacklisted
< Server: Couchbase Server
< Pragma: no-cache
< Date: Wed, 25 Nov 2015 22:50:43 GMT
< Content-Type: application/json
< Content-Length: 44
< Cache-Control: no-cache
<
* Connection #0 to host 192.168.99.100 left intact
{"newBaseUri":"http://192.168.99.100:8091/"}

The command shows an HTTP POST request to the REST endpoint /settings/web. The user name and password credentials are passed in the request.

Install sample data

The Couchbase server can be easily load some sample data in the Couchbase instance.

text

curl -v -u Administrator:password -X POST http://192.168.99.100:8091/sampleBuckets/install -d '["travel-sample"]'
* Hostname was NOT found in DNS cache
*   Trying 192.168.99.100...
* Connected to 192.168.99.100 (192.168.99.100) port 8091 (#0)
* Server auth using Basic with user 'Administrator'
> POST /sampleBuckets/install HTTP/1.1
> Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==
> User-Agent: curl/7.37.1
> Host: 192.168.99.100:8091
> Accept: */*
> Content-Length: 17
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 17 out of 17 bytes
< HTTP/1.1 202 Accepted
* Server Couchbase Server is not blacklisted
< Server: Couchbase Server
< Pragma: no-cache
< Date: Wed, 25 Nov 2015 22:51:51 GMT
< Content-Type: application/json
< Content-Length: 2
< Cache-Control: no-cache
<
* Connection #0 to host 192.168.99.100 left intact
[]

The command shows an HTTP POST request to the REST endpoint /sampleBuckets/install. The name of the sample bucket is passed in the request.

Congratulations, you are now running a Couchbase container, fully configured using the REST API.

Query Couchbase using CBQ

CBQ, short for Couchbase Query, is a CLI tool that allows to create, read, update, and delete JSON documents on a Couchbase server. This tool is installed as part of the Couchbase Docker image.

Run CBQ tool:

text

docker run -it --link db:db couchbase cbq --engine http://db:8093
Couchbase query shell connected to http://db:8093/ . Type Ctrl-D to exit.
cbq>

--engine parameter to CBQ allows to specify the Couchbase server host and port running on the Docker host. For host, typically the host name or IP address of the host where Couchbase server is running is provided. In this case, the container name used when starting the container, db, can be used. 8093 port listens for all incoming queries.

Couchbase allows to query JSON documents using N1QL. N1QL is a comprehensive, declarative query language that brings SQL-like query capabilities to JSON documents.

Query the database by running a N1QL query:

text

cbq> select * from `travel-sample` limit 1;
{
    "requestID": "97816771-3c25-4a1d-9ea8-eb6ad8a51919",
    "signature": {
        "*": "*"
    },
    "results": [
        {
            "travel-sample": {
                "callsign": "MILE-AIR",
                "country": "United States",
                "iata": "Q5",
                "icao": "MLA",
                "id": 10,
                "name": "40-Mile Air",
                "type": "airline"
            }
        }
    ],
    "status": "success",
    "metrics": {
        "elapsedTime": "60.872423ms",
        "executionTime": "60.792258ms",
        "resultCount": 1,
        "resultSize": 300
    }
}

Couchbase Web Console

Couchbase Web Console is a console that allows to manage a Couchbase instance. It can be seen at:

http://192.168.99.100:8091/

Make sure to replace the IP address with the IP address of your Docker Machine or localhost if Docker is running locally.

Examples Couchdb Data Volumes (`docs/examples/couchdb_data_volumes.md`)

Dockerizing a CouchDB service

Note:

If you don't like sudo then see Giving non-root
access

Here's an example of using data volumes to share the same data between
two CouchDB containers. This could be used for hot upgrades, testing
different versions of CouchDB on the same data, etc.

Create first database

Note that we're marking /var/lib/couchdb as a data volume.

text

$ COUCH1=$(docker run -d -p 5984 -v /var/lib/couchdb shykes/couchdb:2013-05-03)

Add data to the first database

We're assuming your Docker host is reachable at localhost. If not,
replace localhost with the public IP of your Docker host.

text

$ HOST=localhost
$ URL="http://$HOST:$(docker port $COUCH1 5984 | grep -o '[1-9][0-9]*$')/_utils/"
$ echo "Navigate to $URL in your browser, and use the couch interface to add data"

Create second database

This time, we're requesting shared access to $COUCH1's volumes.

text

$ COUCH2=$(docker run -d -p 5984 --volumes-from $COUCH1 shykes/couchdb:2013-05-03)

Browse data on the second database

text

$ HOST=localhost
$ URL="http://$HOST:$(docker port $COUCH2 5984 | grep -o '[1-9][0-9]*$')/_utils/"
$ echo "Navigate to $URL in your browser. You should see the same data as in the first database"'!'

Congratulations, you are now running two Couchdb containers, completely
isolated from each other except for their data.

Examples Mongodb (`docs/examples/mongodb.md`)

Dockerizing MongoDB

Introduction

In this example, we are going to learn how to build a Docker image with
MongoDB pre-installed. We'll also see how to push that image to the
Docker Hub registry and share it with others!

Note: This guide will show the mechanics of building a MongoDB container, but
you will probably want to use the official image on Docker Hub

Using Docker and containers for deploying MongoDB
instances will bring several benefits, such as:

Easy to maintain, highly configurable MongoDB instances;
Ready to run and start working within milliseconds;
Based on globally accessible and shareable images.

Note:

If you do not like sudo, you might want to check out:
Giving non-root access.

Creating a Dockerfile for MongoDB

Let's create our Dockerfile and start building it:

text

$ nano Dockerfile

Although optional, it is handy to have comments at the beginning of a
Dockerfile explaining its purpose:

text

# Dockerizing MongoDB: Dockerfile for building MongoDB images
# Based on ubuntu:latest, installs MongoDB following the instructions from:
# http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/

Tip: Dockerfiles are flexible. However, they need to follow a certain
format. The first item to be defined is the name of an image, which becomes
the parent of your Dockerized MongoDB image.

We will build our image using the latest version of Ubuntu from the
Docker Hub Ubuntu repository.

text

# Format: FROM    repository[:version]
FROM       ubuntu:latest

Continuing, we will declare the MAINTAINER of the Dockerfile:

text

# Format: MAINTAINER Name 
MAINTAINER M.Y. Name <[email protected]>

Note: Although Ubuntu systems have MongoDB packages, they are likely to
be outdated. Therefore in this example, we will use the official MongoDB
packages.

We will begin with importing the MongoDB public GPG key. We will also create
a MongoDB repository file for the package manager.

text

# Installation:
# Import MongoDB public GPG key AND create a MongoDB list file
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
RUN echo "deb http://repo.mongodb.org/apt/ubuntu "$(lsb_release -sc)"/mongodb-org/3.0 multiverse" | tee /etc/apt/sources.list.d/mongodb-org-3.0.list

After this initial preparation we can update our packages and install MongoDB.

text

# Update apt-get sources AND install MongoDB
RUN apt-get update && apt-get install -y mongodb-org

Tip: You can install a specific version of MongoDB by using a list
of required packages with versions, e.g.:
text
RUN apt-get update && apt-get install -y mongodb-org=3.0.1 mongodb-org-server=3.0.1 mongodb-org-shell=3.0.1 mongodb-org-mongos=3.0.1 mongodb-org-tools=3.0.1

MongoDB requires a data directory. Let's create it as the final step of our
installation instructions.

text

# Create the MongoDB data directory
RUN mkdir -p /data/db

Lastly we set the ENTRYPOINT which will tell Docker to run mongod inside
the containers launched from our MongoDB image. And for ports, we will use
the EXPOSE instruction.

text

# Expose port 27017 from the container to the host
EXPOSE 27017

# Set usr/bin/mongod as the dockerized entry-point application
ENTRYPOINT ["/usr/bin/mongod"]

Now save the file and let's build our image.

Note:

The full version of this Dockerfile can be found here.

Building the MongoDB Docker image

With our Dockerfile, we can now build the MongoDB image using Docker. Unless
experimenting, it is always a good practice to tag Docker images by passing the
--tag option to docker build command.

text

# Format: docker build --tag/-t /<repository> .
# Example:
$ docker build --tag my/repo .

Once this command is issued, Docker will go through the Dockerfile and build
the image. The final image will be tagged my/repo.

Pushing the MongoDB image to Docker Hub

All Docker image repositories can be hosted and shared on
Docker Hub with the docker push command. For this,
you need to be logged-in.

text

# Log-in
$ docker login
Username:
..

# Push the image
# Format: docker push /<repository>
$ docker push my/repo
The push refers to a repository [my/repo] (len: 1)
Sending image list
Pushing repository my/repo (1 tags)
..

Using the MongoDB image

Using the MongoDB image we created, we can run one or more MongoDB instances
as daemon process(es).

text

# Basic way
# Usage: docker run --name <name for container> -d /<repository>
$ docker run -p 27017:27017 --name mongo_instance_001 -d my/repo

# Dockerized MongoDB, lean and mean!
# Usage: docker run --name <name for container> -d /<repository> --noprealloc --smallfiles
$ docker run -p 27017:27017 --name mongo_instance_001 -d my/repo --smallfiles

# Checking out the logs of a MongoDB container
# Usage: docker logs <name for container>
$ docker logs mongo_instance_001

# Playing with MongoDB
# Usage: mongo --port 
$ mongo --port 27017

# If using docker-machine
# Usage: mongo --port   --host 
$ mongo --port 27017 --host 192.168.59.103

Tip:
If you want to run two containers on the same engine, then you will need to map
the exposed port to two different ports on the host

text

# Start two containers and map the ports
$ docker run -p 28001:27017 --name mongo_instance_001 -d my/repo
$ docker run -p 28002:27017 --name mongo_instance_002 -d my/repo

# Now you can connect to each MongoDB instance on the two ports
$ mongo --port 28001
$ mongo --port 28002

Examples Postgresql Service (`docs/examples/postgresql_service.md`)

Dockerizing PostgreSQL

Note:

If you don't like sudo then see Giving non-root
access

Installing PostgreSQL on Docker

Assuming there is no Docker image that suits your needs on the Docker
Hub, you can create one yourself.

Start by creating a new Dockerfile:

Note:
This PostgreSQL setup is for development-only purposes. Refer to the
PostgreSQL documentation to fine-tune these settings so that it is
suitably secure.

text

#
# example Dockerfile for https://docs.docker.com/examples/postgresql_service/
#

FROM ubuntu
MAINTAINER [email protected]

# Add the PostgreSQL PGP key to verify their Debian packages.
# It should be the same key as https://www.postgresql.org/media/keys/ACCC4CF8.asc
RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8

# Add PostgreSQL's repository. It contains the most recent stable release
#     of PostgreSQL, ``9.3``.
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main" > /etc/apt/sources.list.d/pgdg.list

# Install ``python-software-properties``, ``software-properties-common`` and PostgreSQL 9.3
#  There are some warnings (in red) that show up during the build. You can hide
#  them by prefixing each apt-get statement with DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y python-software-properties software-properties-common postgresql-9.3 postgresql-client-9.3 postgresql-contrib-9.3

# Note: The official Debian and Ubuntu images automatically ``apt-get clean``
# after each ``apt-get``

# Run the rest of the commands as the ``postgres`` user created by the ``postgres-9.3`` package when it was ``apt-get installed``
USER postgres

# Create a PostgreSQL role named ``docker`` with ``docker`` as the password and
# then create a database `docker` owned by the ``docker`` role.
# Note: here we use ``&&\`` to run commands one after the other - the ``\``
#       allows the RUN command to span multiple lines.
RUN    /etc/init.d/postgresql start &&\
    psql --command "CREATE USER docker WITH SUPERUSER PASSWORD 'docker';" &&\
    createdb -O docker docker

# Adjust PostgreSQL configuration so that remote connections to the
# database are possible.
RUN echo "host all  all    0.0.0.0/0  md5" >> /etc/postgresql/9.3/main/pg_hba.conf

# And add ``listen_addresses`` to ``/etc/postgresql/9.3/main/postgresql.conf``
RUN echo "listen_addresses='*'" >> /etc/postgresql/9.3/main/postgresql.conf

# Expose the PostgreSQL port
EXPOSE 5432

# Add VOLUMEs to allow backup of config, logs and databases
VOLUME  ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]

# Set the default command to run when starting the container
CMD ["/usr/lib/postgresql/9.3/bin/postgres", "-D", "/var/lib/postgresql/9.3/main", "-c", "config_file=/etc/postgresql/9.3/main/postgresql.conf"]

Build an image from the Dockerfile assign it a name.

text

$ docker build -t eg_postgresql .

And run the PostgreSQL server container (in the foreground):

text

$ docker run --rm -P --name pg_test eg_postgresql

There are 2 ways to connect to the PostgreSQL server. We can use Link
Containers, or we can access it from our host
(or the network).

Note:
The --rm removes the container and its image when
the container exits successfully.

Using container linking

Containers can be linked to another container's ports directly using
-link remote_name:local_alias in the client's
docker run. This will set a number of environment
variables that can then be used to connect:

text

$ docker run --rm -t -i --link pg_test:pg eg_postgresql bash

postgres@7ef98b1b7243:/$ psql -h $PG_PORT_5432_TCP_ADDR -p $PG_PORT_5432_TCP_PORT -d docker -U docker --password

Connecting from your host system

Assuming you have the postgresql-client installed, you can use the
host-mapped port to test as well. You need to use docker ps
to find out what local host port the container is mapped to
first:

text

$ docker ps
CONTAINER ID        IMAGE                  COMMAND                CREATED             STATUS              PORTS                                      NAMES
5e24362f27f6        eg_postgresql:latest   /usr/lib/postgresql/   About an hour ago   Up About an hour    0.0.0.0:49153->5432/tcp                    pg_test
$ psql -h localhost -p 49153 -d docker -U docker --password

Testing the database

Once you have authenticated and have a docker =#
prompt, you can create a table and populate it.

text

psql (9.3.1)
Type "help" for help.

$ docker=# CREATE TABLE cities (
docker(#     name            varchar(80),
docker(#     location        point
docker(# );
CREATE TABLE
$ docker=# INSERT INTO cities VALUES ('San Francisco', '(-194.0, 53.0)');
INSERT 0 1
$ docker=# select * from cities;
     name      | location
---------------+-----------
 San Francisco | (-194,53)
(1 row)

Using the container volumes

You can use the defined volumes to inspect the PostgreSQL log files and
to backup your configuration and data:

text

$ docker run --rm --volumes-from pg_test -t -i busybox sh

/ # ls
bin      etc      lib      linuxrc  mnt      proc     run      sys      usr
dev      home     lib64    media    opt      root     sbin     tmp      var
/ # ls /etc/postgresql/9.3/main/
environment      pg_hba.conf      postgresql.conf
pg_ctl.conf      pg_ident.conf    start.conf
/tmp # ls /var/log
ldconfig    postgresql

Examples Running Redis Service (`docs/examples/running_redis_service.md`)

Dockerizing a Redis service

Very simple, no frills, Redis service attached to a web application
using a link.

Create a Docker container for Redis

Firstly, we create a Dockerfile for our new Redis
image.

text

FROM        ubuntu:14.04
RUN         apt-get update && apt-get install -y redis-server
EXPOSE      6379
ENTRYPOINT  ["/usr/bin/redis-server"]

Next we build an image from our Dockerfile.
Replace <your username> with your own user name.

text

$ docker build -t <your username>/redis .

Run the service

Use the image we've just created and name your container redis.

Running the service with -d runs the container in detached mode, leaving
the container running in the background.

Importantly, we're not exposing any ports on our container. Instead
we're going to use a container link to provide access to our Redis
database.

text

$ docker run --name redis -d <your username>/redis

Create your web application container

Next we can create a container for our application. We're going to use
the -link flag to create a link to the redis container we've just
created with an alias of db. This will create a secure tunnel to the
redis container and expose the Redis instance running inside that
container to only this container.

text

$ docker run --link redis:db -i -t ubuntu:14.04 /bin/bash

Once inside our freshly created container we need to install Redis to
get the redis-cli binary to test our connection.

text

$ sudo apt-get update
$ sudo apt-get install redis-server
$ sudo service redis-server stop

As we've used the --link redis:db option, Docker
has created some environment variables in our web application container.

text

$ env | grep DB_

# Should return something similar to this with your values
DB_NAME=/violet_wolf/db
DB_PORT_6379_TCP_PORT=6379
DB_PORT=tcp://172.17.0.33:6379
DB_PORT_6379_TCP=tcp://172.17.0.33:6379
DB_PORT_6379_TCP_ADDR=172.17.0.33
DB_PORT_6379_TCP_PROTO=tcp

We can see that we've got a small list of environment variables prefixed
with DB. The DB comes from the link alias specified when we launched
the container. Let's use the DB_PORT_6379_TCP_ADDR variable to connect to
our Redis container.

text

$ redis-cli -h $DB_PORT_6379_TCP_ADDR
$ redis 172.17.0.33:6379>
$ redis 172.17.0.33:6379> set docker awesome
OK
$ redis 172.17.0.33:6379> get docker
"awesome"
$ redis 172.17.0.33:6379> exit

We could easily use this or other environment variables in our web
application to make a connection to our redis
container.

Examples Running Riak Service (`docs/examples/running_riak_service.md`)

Dockerizing a Riak service

The goal of this example is to show you how to build a Docker image with
Riak pre-installed.

Creating a Dockerfile

Create an empty file called Dockerfile:

text

$ touch Dockerfile

Next, define the parent image you want to use to build your image on top
of. We'll use Ubuntu (tag:
trusty), which is available on Docker Hub:

text

# Riak
#
# VERSION       0.1.1

# Use the Ubuntu base image provided by dotCloud
FROM ubuntu:trusty
MAINTAINER Hector Castro [email protected]

After that, we install the curl which is used to download the repository setup
script and we download the setup script and run it.

text

# Install Riak repository before we do apt-get update, so that update happens
# in a single step
RUN apt-get install -q -y curl && \
    curl -fsSL https://packagecloud.io/install/repositories/basho/riak/script.deb | sudo bash

Then we install and setup a few dependencies:

supervisor is used manage the Riak processes
riak=2.0.5-1 is the Riak package coded to version 2.0.5

Install and setup project dependencies

RUN apt-get update &&
apt-get install -y supervisor riak=2.0.5-1

RUN mkdir -p /var/log/supervisor

RUN locale-gen en_US en_US.UTF-8

COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf

After that, we modify Riak's configuration:

text

# Configure Riak to accept connections from any host
RUN sed -i "s|listener.http.internal = 127.0.0.1:8098|listener.http.internal = 0.0.0.0:8098|" /etc/riak/riak.conf
RUN sed -i "s|listener.protobuf.internal = 127.0.0.1:8087|listener.protobuf.internal = 0.0.0.0:8087|" /etc/riak/riak.conf

Then, we expose the Riak Protocol Buffers and HTTP interfaces:

text

# Expose Riak Protocol Buffers and HTTP interfaces
EXPOSE 8087 8098

Finally, run supervisord so that Riak is started:

text

CMD ["/usr/bin/supervisord"]

Create a supervisord configuration file

Create an empty file called supervisord.conf. Make
sure it's at the same directory level as your Dockerfile:

text

touch supervisord.conf

Populate it with the following program definitions:

text

[supervisord]
nodaemon=true

[program:riak]
command=bash -c "/usr/sbin/riak console"
numprocs=1
autostart=true
autorestart=true
user=riak
environment=HOME="/var/lib/riak"
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s.log

Build the Docker image for Riak

Now you should be able to build a Docker image for Riak:

text

$ docker build -t "<yourname>/riak" .

Next steps

Riak is a distributed database. Many production deployments consist of
at least five nodes.
See the docker-riak project
details on how to deploy a Riak cluster using Docker and Pipework.

Examples Running Ssh Service (`docs/examples/running_ssh_service.md`)

Dockerizing an SSH daemon service

Build an `eg_sshd` image

The following Dockerfile sets up an SSHd service in a container that you
can use to connect to and inspect other container's volumes, or to get
quick access to a test container.

text

# sshd
#
# VERSION               0.0.2

FROM ubuntu:14.04
MAINTAINER Sven Dowideit <[email protected]>

RUN apt-get update && apt-get install -y openssh-server
RUN mkdir /var/run/sshd
RUN echo 'root:screencast' | chpasswd
RUN sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config

# SSH login fix. Otherwise user is kicked off after login
RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd

ENV NOTVISIBLE "in users profile"
RUN echo "export VISIBLE=now" >> /etc/profile

EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

Build the image using:

text

$ docker build -t eg_sshd .

Run a `test_sshd` container

Then run it. You can then use docker port to find out what host port
the container's port 22 is mapped to:

text

$ docker run -d -P --name test_sshd eg_sshd
$ docker port test_sshd 22
0.0.0.0:49154

And now you can ssh as root on the container's IP address (you can find it
with docker inspect) or on port 49154 of the Docker daemon's host IP address
(ip address or ifconfig can tell you that) or localhost if on the
Docker daemon host:

text

$ ssh [email protected] -p 49154
# The password is ``screencast``.
$$

Environment variables

Using the sshd daemon to spawn shells makes it complicated to pass environment
variables to the user's shell via the normal Docker mechanisms, as sshd scrubs
the environment before it starts the shell.

If you're setting values in the Dockerfile using ENV, you'll need to push them
to a shell initialization file like the /etc/profile example in the Dockerfile
above.

If you need to passdocker run -e ENV=value values, you will need to write a
short script to do the same before you start sshd -D and then replace the
CMD with that script.

Clean up

Finally, clean up after your test by stopping and removing the
container, and then removing the image.

text

$ docker stop test_sshd
$ docker rm test_sshd
$ docker rmi eg_sshd

Extend Plugin Api (`docs/extend/plugin_api.md`)

Docker Plugin API

Docker plugins are out-of-process extensions which add capabilities to the
Docker Engine.

This page is intended for people who want to develop their own Docker plugin.
If you just want to learn about or use Docker plugins, look
here.

What plugins are

A plugin is a process running on the same or a different host as the docker daemon,
which registers itself by placing a file on the same docker host in one of the plugin
directories described in Plugin discovery.

Plugins have human-readable names, which are short, lowercase strings. For
example, flocker or weave.

Plugins can run inside or outside containers. Currently running them outside
containers is recommended.

Plugin discovery

Docker discovers plugins by looking for them in the plugin directory whenever a
user or container tries to use one by name.

There are three types of files which can be put in the plugin directory.

.sock files are UNIX domain sockets.
.spec files are text files containing a URL, such as unix:///other.sock or tcp://localhost:8080.
.json files are text files containing a full json specification for the plugin.

Plugins with UNIX domain socket files must run on the same docker host, whereas
plugins with spec or json files can run on a different host if a remote URL is specified.

UNIX domain socket files must be located under /run/docker/plugins, whereas
spec files can be located either under /etc/docker/plugins or /usr/lib/docker/plugins.

The name of the file (excluding the extension) determines the plugin name.

For example, the flocker plugin might create a UNIX socket at
/run/docker/plugins/flocker.sock.

You can define each plugin into a separated subdirectory if you want to isolate definitions from each other.
For example, you can create the flocker socket under /run/docker/plugins/flocker/flocker.sock and only
mount /run/docker/plugins/flocker inside the flocker container.

Docker always searches for unix sockets in /run/docker/plugins first. It checks for spec or json files under
/etc/docker/plugins and /usr/lib/docker/plugins if the socket doesn't exist. The directory scan stops as
soon as it finds the first plugin definition with the given name.

JSON specification

This is the JSON format for a plugin:

json

{
  "Name": "plugin-example",
  "Addr": "https://example.com/docker/plugin",
  "TLSConfig": {
    "InsecureSkipVerify": false,
    "CAFile": "/usr/shared/docker/certs/example-ca.pem",
    "CertFile": "/usr/shared/docker/certs/example-cert.pem",
    "KeyFile": "/usr/shared/docker/certs/example-key.pem",
  }
}

The TLSConfig field is optional and TLS will only be verified if this configuration is present.

Plugin lifecycle

Plugins should be started before Docker, and stopped after Docker. For
example, when packaging a plugin for a platform which supports systemd, you
might use systemd dependencies to
manage startup and shutdown order.

When upgrading a plugin, you should first stop the Docker daemon, upgrade the
plugin, then start Docker again.

Plugin activation

When a plugin is first referred to -- either by a user referring to it by name
(e.g. docker run --volume-driver=foo) or a container already configured to
use a plugin being started -- Docker looks for the named plugin in the plugin
directory and activates it with a handshake. See Handshake API below.

Plugins are not activated automatically at Docker daemon startup. Rather,
they are activated only lazily, or on-demand, when they are needed.

Systemd socket activation

Plugins may also be socket activated by systemd. The official Plugins helpers
natively supports socket activation. In order for a plugin to be socket activated it needs
a service file and a socket file.

The service file (for example /lib/systemd/system/your-plugin.service):

text

[Unit]
Description=Your plugin
Before=docker.service
After=network.target your-plugin.socket
Requires=your-plugin.socket docker.service

[Service]
ExecStart=/usr/lib/docker/your-plugin

[Install]
WantedBy=multi-user.target

The socket file (for example /lib/systemd/system/your-plugin.socket):

text

[Unit]
Description=Your plugin

[Socket]
ListenStream=/run/docker/plugins/your-plugin.sock

[Install]
WantedBy=sockets.target

This will allow plugins to be actually started when the Docker daemon connects to
the sockets they're listening on (for instance the first time the daemon uses them
or if one of the plugin goes down accidentally).

API design

The Plugin API is RPC-style JSON over HTTP, much like webhooks.

Requests flow from the Docker daemon to the plugin. So the plugin needs to
implement an HTTP server and bind this to the UNIX socket mentioned in the
"plugin discovery" section.

All requests are HTTP POST requests.

The API is versioned via an Accept header, which currently is always set to
application/vnd.docker.plugins.v1+json.

Handshake API

Plugins are activated via the following "handshake" API call.

/Plugin.Activate

Request: empty body

Response:

text

{
    "Implements": ["VolumeDriver"]
}

Responds with a list of Docker subsystems which this plugin implements.
After activation, the plugin will then be sent events from this subsystem.

Possible values are:

Plugin retries

Attempts to call a method on a plugin are retried with an exponential backoff
for up to 30 seconds. This may help when packaging plugins as containers, since
it gives plugin containers a chance to start up before failing any user
containers which depend on them.

Plugins helpers

To ease plugins development, we're providing an sdk for each kind of plugins
currently supported by Docker at docker/go-plugins-helpers.

Extend Plugins (`docs/extend/plugins.md`)

Understand Engine plugins

You can extend the capabilities of the Docker Engine by loading third-party
plugins. This page explains the types of plugins and provides links to several
volume and network plugins for Docker.

Types of plugins

Plugins extend Docker's functionality. They come in specific types. For
example, a volume plugin might enable Docker
volumes to persist across multiple Docker hosts and a
network plugin might provide network plumbing.

Currently Docker supports authorization, volume and network driver plugins. In the future it
will support additional plugin types.

Installing a plugin

Follow the instructions in the plugin's documentation.

Finding a plugin

The sections below provide an inexhaustive overview of available plugins.

Network plugins

Plugin	Description
Contiv Networking	An open source network plugin to provide infrastructure and security policies for a multi-tenant micro services deployment, while providing an integration to physical network for non-container workload. Contiv Networking implements the remote driver and IPAM APIs available in Docker 1.9 onwards.
Kuryr Network Plugin	A network plugin is developed as part of the OpenStack Kuryr project and implements the Docker networking (libnetwork) remote driver API by utilizing Neutron, the OpenStack networking service. It includes an IPAM driver as well.
Weave Network Plugin	A network plugin that creates a virtual network that connects your Docker containers - across multiple hosts or clouds and enables automatic discovery of applications. Weave networks are resilient, partition tolerant, secure and work in partially connected networks, and other adverse environments - all configured with delightful simplicity.

Volume plugins

Plugin	Description
Azure File Storage plugin	Lets you mount Microsoft Azure File Storage shares to Docker containers as volumes using the SMB 3.0 protocol. Learn more.
Blockbridge plugin	A volume plugin that provides access to an extensible set of container-based persistent storage options. It supports single and multi-host Docker environments with features that include tenant isolation, automated provisioning, encryption, secure deletion, snapshots and QoS.
Contiv Volume Plugin	An open source volume plugin that provides multi-tenant, persistent, distributed storage with intent based consumption using ceph underneath.
Convoy plugin	A volume plugin for a variety of storage back-ends including device mapper and NFS. It's a simple standalone executable written in Go and provides the framework to support vendor-specific extensions such as snapshots, backups and restore.
DRBD plugin	A volume plugin that provides highly available storage replicated by DRBD. Data written to the docker volume is replicated in a cluster of DRBD nodes.
Flocker plugin	A volume plugin that provides multi-host portable volumes for Docker, enabling you to run databases and other stateful containers and move them around across a cluster of machines.
gce-docker plugin	A volume plugin able to attach, format and mount Google Compute persistent-disks.
GlusterFS plugin	A volume plugin that provides multi-host volumes management for Docker using GlusterFS.
Horcrux Volume Plugin	A volume plugin that allows on-demand, version controlled access to your data. Horcrux is an open-source plugin, written in Go, and supports SCP, Minio and Amazon S3.
IPFS Volume Plugin	An open source volume plugin that allows using an ipfs filesystem as a volume.
Keywhiz plugin	A plugin that provides credentials and secret management using Keywhiz as a central repository.
Local Persist Plugin	A volume plugin that extends the default `local` driver's functionality by allowing you specify a mountpoint anywhere on the host, which enables the files to always persist, even if the volume is removed via `docker volume rm`.
NetApp Plugin (nDVP)	A volume plugin that provides direct integration with the Docker ecosystem for the NetApp storage portfolio. The nDVP package supports the provisioning and management of storage resources from the storage platform to Docker hosts, with a robust framework for adding additional platforms in the future.
Netshare plugin	A volume plugin that provides volume management for NFS 3/4, AWS EFS and CIFS file systems.
OpenStorage Plugin	A cluster-aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few.
Quobyte Volume Plugin	A volume plugin that connects Docker to Quobyte's data center file system, a general-purpose scalable and fault-tolerant storage platform.
REX-Ray plugin	A volume plugin which is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC.
Virtuozzo Storage and Ploop plugin	A volume plugin with support for Virtuozzo Storage distributed cloud file system as well as ploop devices.
VMware vSphere Storage Plugin	Docker Volume Driver for vSphere enables customers to address persistent storage requirements for Docker containers in vSphere environments.

Authorization plugins

Plugin	Description
Twistlock AuthZ Broker	A basic extendable authorization plugin that runs directly on the host or inside a container. This plugin allows you to define user policies that it evaluates during authorization. Basic authorization is provided if Docker daemon is started with the --tlsverify flag (username is extracted from the certificate common name).

Troubleshooting a plugin

If you are having problems with Docker after loading a plugin, ask the authors
of the plugin for help. The Docker team may not be able to assist you.

Writing a plugin

If you are interested in writing a plugin for Docker, or seeing how they work
under the hood, see the docker plugins reference.

Extend Plugins Authorization (`docs/extend/plugins_authorization.md`)

Create an authorization plugin

Docker's out-of-the-box authorization model is all or nothing. Any user with
permission to access the Docker daemon can run any Docker client command. The
same is true for callers using Docker's remote API to contact the daemon. If you
require greater access control, you can create authorization plugins and add
them to your Docker daemon configuration. Using an authorization plugin, a
Docker administrator can configure granular access policies for managing access
to Docker daemon.

Anyone with the appropriate skills can develop an authorization plugin. These
skills, at their most basic, are knowledge of Docker, understanding of REST, and
sound programming knowledge. This document describes the architecture, state,
and methods information available to an authorization plugin developer.

Basic principles

Docker's plugin infrastructure enables
extending Docker by loading, removing and communicating with
third-party components using a generic API. The access authorization subsystem
was built using this mechanism.

Using this subsystem, you don't need to rebuild the Docker daemon to add an
authorization plugin. You can add a plugin to an installed Docker daemon. You do
need to restart the Docker daemon to add a new plugin.

An authorization plugin approves or denies requests to the Docker daemon based
on both the current authentication context and the command context. The
authentication context contains all user details and the authentication method.
The command context contains all the relevant request data.

Authorization plugins must follow the rules described in Docker Plugin API.
Each plugin must reside within directories described under the
Plugin discovery section.

Note: the abbreviations AuthZ and AuthN mean authorization and authentication
respectively.

Default user authorization mechanism

If TLS is enabled in the Docker daemon, the default user authorization flow extracts the user details from the certificate subject name.
That is, the User field is set to the client certificate subject common name, and the AuthenticationMethod field is set to TLS.

Basic architecture

You are responsible for registering your plugin as part of the Docker daemon
startup. You can install multiple plugins and chain them together. This chain
can be ordered. Each request to the daemon passes in order through the chain.
Only when all the plugins grant access to the resource, is the access granted.

When an HTTP request is made to the Docker daemon through the CLI or via the
remote API, the authentication subsystem passes the request to the installed
authentication plugin(s). The request contains the user (caller) and command
context. The plugin is responsible for deciding whether to allow or deny the
request.

The sequence diagrams below depict an allow and deny authorization flow:

Each request sent to the plugin includes the authenticated user, the HTTP
headers, and the request/response body. Only the user name and the
authentication method used are passed to the plugin. Most importantly, no user
credentials or tokens are passed. Finally, not all request/response bodies
are sent to the authorization plugin. Only those request/response bodies where
the Content-Type is either text/* or application/json are sent.

For commands that can potentially hijack the HTTP connection (HTTP Upgrade), such as exec, the authorization plugin is only called for the
initial HTTP requests. Once the plugin approves the command, authorization is
not applied to the rest of the flow. Specifically, the streaming data is not
passed to the authorization plugins. For commands that return chunked HTTP
response, such as logs and events, only the HTTP request is sent to the
authorization plugins.

During request/response processing, some authorization flows might
need to do additional queries to the Docker daemon. To complete such flows,
plugins can call the daemon API similar to a regular user. To enable these
additional queries, the plugin must provide the means for an administrator to
configure proper authentication and security policies.

Docker client flows

To enable and configure the authorization plugin, the plugin developer must
support the Docker client interactions detailed in this section.

Setting up Docker daemon

Enable the authorization plugin with a dedicated command line flag in the
--authorization-plugin=PLUGIN_ID format. The flag supplies a PLUGIN_ID
value. This value can be the plugin’s socket or a path to a specification file.

bash

$ docker daemon --authorization-plugin=plugin1 --authorization-plugin=plugin2,...

Docker's authorization subsystem supports multiple --authorization-plugin parameters.

Calling authorized command (allow)

bash

$ docker pull centos
...
f1b10cd84249: Pull complete
...

Calling unauthorized command (deny)

bash

$ docker pull centos
...
docker: Error response from daemon: authorization denied by plugin PLUGIN_NAME: volumes are not allowed.

Error from plugins

bash

$ docker pull centos
...
docker: Error response from daemon: plugin PLUGIN_NAME failed with error: AuthZPlugin.AuthZReq: Cannot connect to the Docker daemon. Is the docker daemon running on this host?.

API schema and implementation

In addition to Docker's standard plugin registration method, each plugin
should implement the following two methods:

/AuthzPlugin.AuthZReq This authorize request method is called before the Docker daemon processes the client request.
/AuthzPlugin.AuthZRes This authorize response method is called before the response is returned from Docker daemon to the client.

/AuthzPlugin.AuthZReq

Request:

json

{
    "User":              "The user identification",
    "UserAuthNMethod":   "The authentication method used",
    "RequestMethod":     "The HTTP method",
    "RequestURI":        "The HTTP request URI",
    "RequestBody":       "Byte array containing the raw HTTP request body",
    "RequestHeader":     "Byte array containing the raw HTTP request header as a map[string][]string "
}

Response:

json

{
    "Allow": "Determined whether the user is allowed or not",
    "Msg":   "The authorization message",
    "Err":   "The error message if things go wrong"
}

/AuthzPlugin.AuthZRes

Request:

json

{
    "User":              "The user identification",
    "UserAuthNMethod":   "The authentication method used",
    "RequestMethod":     "The HTTP method",
    "RequestURI":        "The HTTP request URI",
    "RequestBody":       "Byte array containing the raw HTTP request body",
    "RequestHeader":     "Byte array containing the raw HTTP request header as a map[string][]string",
    "ResponseBody":      "Byte array containing the raw HTTP response body",
    "ResponseHeader":    "Byte array containing the raw HTTP response header as a map[string][]string",
    "ResponseStatusCode":"Response status code"
}

Response:

json

{
   "Allow":              "Determined whether the user is allowed or not",
   "Msg":                "The authorization message",
   "Err":                "The error message if things go wrong"
}

Request authorization

Each plugin must support two request authorization messages formats, one from the daemon to the plugin and then from the plugin to the daemon. The tables below detail the content expected in each message.

Daemon -> Plugin

Name	Type	Description
User	string	The user identification
Authentication method	string	The authentication method used
Request method	enum	The HTTP method (GET/DELETE/POST)
Request URI	string	The HTTP request URI including API version (e.g., v.1.17/containers/json)
Request headers	map[string]string	Request headers as key value pairs (without the authorization header)
Request body	[]byte	Raw request body

Plugin -> Daemon

Name	Type	Description
Allow	bool	Boolean value indicating whether the request is allowed or denied
Msg	string	Authorization message (will be returned to the client in case the access is denied)
Err	string	Error message (will be returned to the client in case the plugin encounter an error. The string value supplied may appear in logs, so should not include confidential information)

Response authorization

The plugin must support two authorization messages formats, one from the daemon to the plugin and then from the plugin to the daemon. The tables below detail the content expected in each message.

Daemon -> Plugin

Name	Type	Description
User	string	The user identification
Authentication method	string	The authentication method used
Request method	string	The HTTP method (GET/DELETE/POST)
Request URI	string	The HTTP request URI including API version (e.g., v.1.17/containers/json)
Request headers	map[string]string	Request headers as key value pairs (without the authorization header)
Request body	[]byte	Raw request body
Response status code	int	Status code from the docker daemon
Response headers	map[string]string	Response headers as key value pairs
Response body	[]byte	Raw docker daemon response body

Plugin -> Daemon

Name	Type	Description
Allow	bool	Boolean value indicating whether the response is allowed or denied
Msg	string	Authorization message (will be returned to the client in case the access is denied)
Err	string	Error message (will be returned to the client in case the plugin encounter an error. The string value supplied may appear in logs, so should not include confidential information)

Extend Plugins Network (`docs/extend/plugins_network.md`)

Engine network driver plugins

Docker Engine network plugins enable Engine deployments to be extended to
support a wide range of networking technologies, such as VXLAN, IPVLAN, MACVLAN
or something completely different. Network driver plugins are supported via the
LibNetwork project. Each plugin is implemented as a "remote driver" for
LibNetwork, which shares plugin infrastructure with Engine. Effectively, network
driver plugins are activated in the same way as other plugins, and use the same
kind of protocol.

Using network driver plugins

The means of installing and running a network driver plugin depend on the
particular plugin. So, be sure to install your plugin according to the
instructions obtained from the plugin developer.

Once running however, network driver plugins are used just like the built-in
network drivers: by being mentioned as a driver in network-oriented Docker
commands. For example,

text

$ docker network create --driver weave mynet

Some network driver plugins are listed in plugins

The mynet network is now owned by weave, so subsequent commands
referring to that network will be sent to the plugin,

text

$ docker run --net=mynet busybox top

Write a network plugin

Network plugins implement the Docker plugin
API and the network plugin protocol

Network plugin protocol

The network driver protocol, in addition to the plugin activation call, is
documented as part of libnetwork:
https://github.com/docker/libnetwork/blob/master/docs/remote.md.

To interact with the Docker maintainers and other interested users, see the IRC channel #docker-network.

Docker networks feature overview
The LibNetwork project

Extend Plugins Volume (`docs/extend/plugins_volume.md`)

Write a volume plugin

Docker Engine volume plugins enable Engine deployments to be integrated with
external storage systems, such as Amazon EBS, and enable data volumes to persist
beyond the lifetime of a single Engine host. See the plugin
documentation for more information.

Changelog

1.12.0

Add Status field to VolumeDriver.Get response (#21006)
Add VolumeDriver.Capabilities to get capabilities of the volume driver(#22077)

1.10.0

Add VolumeDriver.Get which gets the details about the volume (#16534)
Add VolumeDriver.List which lists all volumes owned by the driver (#16534)

1.8.0

Initial support for volume driver plugins (#14659)

Command-line changes

A volume plugin makes use of the -vand --volume-driver flag on the docker run command. The -v flag accepts a volume name and the --volume-driver flag a driver type, for example:

text

$ docker run -ti -v volumename:/data --volume-driver=flocker   busybox sh

This command passes the volumename through to the volume plugin as a
user-given name for the volume. The volumename must not begin with a /.

By having the user specify a volumename, a plugin can associate the volume
with an external volume beyond the lifetime of a single container or container
host. This can be used, for example, to move a stateful container from one
server to another.

By specifying a volumedriver in conjunction with a volumename, users can use plugins such as Flocker to manage volumes external to a single host, such as those on EBS.

Create a VolumeDriver

The container creation endpoint (/containers/create) accepts a VolumeDriver
field of type string allowing to specify the name of the driver. It's default
value of "local" (the default driver for local volumes).

Volume plugin protocol

If a plugin registers itself as a VolumeDriver when activated, then it is
expected to provide writeable paths on the host filesystem for the Docker
daemon to provide to containers to consume.

The Docker daemon handles bind-mounting the provided paths into user
containers.

Note: Volume plugins should not write data to the /var/lib/docker/
directory, including /var/lib/docker/volumes. The /var/lib/docker/
directory is reserved for Docker.

/VolumeDriver.Create

Request:

json

{
    "Name": "volume_name",
    "Opts": {}
}

Instruct the plugin that the user wants to create a volume, given a user
specified volume name. The plugin does not need to actually manifest the
volume on the filesystem yet (until Mount is called).
Opts is a map of driver specific options passed through from the user request.

Response:

json

{
    "Err": ""
}

Respond with a string error if an error occurred.

/VolumeDriver.Remove

Request:

json

{
    "Name": "volume_name"
}

Delete the specified volume from disk. This request is issued when a user invokes docker rm -v to remove volumes associated with a container.

Response:

json

{
    "Err": ""
}

Respond with a string error if an error occurred.

/VolumeDriver.Mount

Request:

json

{
    "Name": "volume_name",
    "ID": "b87d7442095999a92b65b3d9691e697b61713829cc0ffd1bb72e4ccd51aa4d6c"
}

Docker requires the plugin to provide a volume, given a user specified volume
name. This is called once per container start. If the same volume_name is requested
more than once, the plugin may need to keep track of each new mount request and provision
at the first mount request and deprovision at the last corresponding unmount request.

ID is a unqiue ID for the caller that is requesting the mount.

Response:

json

{
    "Mountpoint": "/path/to/directory/on/host",
    "Err": ""
}

Respond with the path on the host filesystem where the volume has been made
available, and/or a string error if an error occurred.

/VolumeDriver.Path

Request:

json

{
    "Name": "volume_name"
}

Docker needs reminding of the path to the volume on the host.

Response:

json

{
    "Mountpoint": "/path/to/directory/on/host",
    "Err": ""
}

Respond with the path on the host filesystem where the volume has been made
available, and/or a string error if an error occurred. Mountpoint is optional,
however the plugin may be queried again later if one is not provided.

/VolumeDriver.Unmount

Request:

json

{
    "Name": "volume_name",
    "ID": "b87d7442095999a92b65b3d9691e697b61713829cc0ffd1bb72e4ccd51aa4d6c"
}

Indication that Docker no longer is using the named volume. This is called once
per container stop. Plugin may deduce that it is safe to deprovision it at
this point.

ID is a unqiue ID for the caller that is requesting the mount.

Response:

json

{
    "Err": ""
}

Respond with a string error if an error occurred.

/VolumeDriver.Get

Request:

json

{
    "Name": "volume_name"
}

Get the volume info.

Response:

json

{
  "Volume": {
    "Name": "volume_name",
    "Mountpoint": "/path/to/directory/on/host",
    "Status": {}
  },
  "Err": ""
}

Respond with a string error if an error occurred. Mountpoint and Status are
optional.

/VolumeDriver.List

Request:

json

{}

Get the list of volumes registered with the plugin.

Response:

json

{
  "Volumes": [
    {
      "Name": "volume_name",
      "Mountpoint": "/path/to/directory/on/host"
    }
  ],
  "Err": ""
}

Respond with a string error if an error occurred. Mountpoint is optional.

/VolumeDriver.Capabilities

Request:

json

{}

Get the list of capabilities the driver supports.
The driver is not required to implement this endpoint, however in such cases
the default values will be taken.

Response:

json

{
  "Capabilities": {
    "Scope": "global"
  }
}

Supported scopes are global and local. Any other value in Scope will be
ignored and assumed to be local. Scope allows cluster managers to handle the
volume differently, for instance with a scope of global, the cluster manager
knows it only needs to create the volume once instead of on every engine. More
capabilities may be added in the future.

Getstarted Last Page (`docs/getstarted/last_page.md`)

Learning more

This getting started provided very basic essentials for using Docker on Mac, Windows, and Linux. If you want to learn more with regard to end-to-end development, start with the full install instructions and feature overviews, then follow up with more advanced tutorials and user guides.

Depending on your interest, the Docker documentation contains a wealth of information. Here are some places to start:

If you are looking for	Where to find it
More about Docker for Mac, features, examples, FAQs, relationship to Docker Machine and Docker Toolbox, and how this fits in the Docker ecosystem	Getting Started with Docker for Mac
More about Docker for Windows, features, examples, FAQs, relationship to Docker Machine and Docker Toolbox, and how this fits in the Docker ecosystem	Getting Started with Docker for Windows
More about Docker Toolbox	Docker Toolbox Overview
More about Docker for Linux distributions	Install Docker Engine on Linux
More advanced tutorials on running containers, building your own images, networking containers, managing data for containers, and storing images on Docker Hub	Learn by example
Information about the Docker product line	The product explainer is a good place to start.
How to set up an automated build on Docker Hub	Docker Hub documentation
How to run a multi-container application with Compose	Docker Compose documentation

Getstarted Linux Install Help (`docs/getstarted/linux_install_help.md`)

Example: Install Docker on Ubuntu Linux

This installation procedure for users who are unfamiliar with package
managers, and just want to try out the Getting Started tutorial while running Docker on Linux. If you are comfortable with package managers, prefer not to use
curl, or have problems installing and want to troubleshoot, please use our
apt and yum repositories instead for your installation.

Log into your Ubuntu installation as a

Docker & Self-Hosted Deployment Verified OCI

Technical Documentation: aquasecurity/docker

Overview (`README.md`) Copy Section

Docker: the container engine [](https://github.com/docker/docker/releases/latest)

Security Disclosure Copy Chapter

Better than VMs Copy Chapter

Plays well with others Copy Chapter

Escape dependency hell Copy Chapter

Getting started

Usage examples

Under the hood Copy Chapter

Contributing to Docker [](https://godoc.org/github.com/docker/docker)

Getting the development builds

How the project is run

Talking to other Docker users and contributors Copy Section

Legal Copy Section

Licensing

Other Docker Related Projects

Awesome-Docker

Overview (`api/README.md`) Copy Section

Overview (`contrib/README.md`) Copy Section

Overview (`docs/README.md`) Copy Section

Docker Documentation

Understanding the documentation branches and processes Copy Chapter

Quickstart for documentation contributors Copy Section

Debugging and testing Copy Section

Style guide Copy Chapter

Publishing documentation (for Docker maintainers) Copy Chapter

Errors publishing using a Docker Machine VM Copy Section

Cherry-picking documentation changes to update an existing release. Copy Chapter

Caching and the docs Copy Section

Removing files from the docs.docker.com site Copy Chapter

Generate the man pages Copy Section

Overview (`experimental/README.md`) Copy Section

Docker Experimental Features

Install Docker experimental Copy Chapter

Get the Linux binary Copy Section

Build an experimental binary Copy Section

Current experimental features Copy Chapter

How to comment on an experimental feature Copy Chapter

Overview (`man/README.md`) Copy Section

Docker Documentation

Generating man pages from the Markdown files

Building the md2man image Copy Chapter

Utilizing the image Copy Chapter

Overview (`pkg/README.md`) Copy Section

Overview (`project/README.md`) Copy Section

Hacking on Docker

Guides Copy Chapter

Roadmap Copy Chapter

Build tools Copy Chapter

Overview (`cmd/dockerd/README.md`) Copy Section

Overview (`contrib/desktop-integration/README.md`) Copy Section

Desktop Integration

Examples

Overview (`contrib/docker-device-tool/README.md`) Copy Section

Docker device tool for devicemapper storage driver backend

Compile

Overview (`contrib/vagrant-docker/README.md`) Copy Section

Vagrant integration

Provisioners Copy Section

Providers Copy Section

Setting up Vagrant-docker with the Remote API Copy Chapter

Overview (`docs/static_files/README.md`) Copy Section

Static files dir

Overview (`hack/Jenkins/readme.md`) Copy Section

Overview (`hack/make/README.md`) Copy Section

Overview (`pkg/archive/README.md`) Copy Section

Overview (`pkg/discovery/README.md`) Copy Section

page_title: Docker discovery page_description: discovery page_keywords: docker, clustering, discovery Copy Chapter

Discovery

Backends Copy Chapter

Using etcd Copy Section

Using consul Copy Section

Using zookeeper Copy Section

Overview (`pkg/locker/README.md`) Copy Section

Locker

Usage Copy Chapter

Overview (`pkg/mflag/README.md`) Copy Section

Overview (`pkg/reexec/README.md`) Copy Section

Overview (`README.md`)

Security Disclosure

Better than VMs

Plays well with others

Escape dependency hell

Under the hood

Talking to other Docker users and contributors

Legal

Overview (`api/README.md`)

Overview (`contrib/README.md`)

Overview (`docs/README.md`)

Understanding the documentation branches and processes

Quickstart for documentation contributors

Debugging and testing

Style guide

Publishing documentation (for Docker maintainers)

Errors publishing using a Docker Machine VM

Cherry-picking documentation changes to update an existing release.

Caching and the docs

Removing files from the docs.docker.com site

Generate the man pages

Overview (`experimental/README.md`)

Install Docker experimental

Get the Linux binary

Build an experimental binary

Current experimental features

How to comment on an experimental feature

Overview (`man/README.md`)

Building the md2man image

Utilizing the image

Overview (`pkg/README.md`)

Overview (`project/README.md`)

Guides

Roadmap

Build tools

Overview (`cmd/dockerd/README.md`)

Overview (`contrib/desktop-integration/README.md`)

Overview (`contrib/docker-device-tool/README.md`)

Overview (`contrib/vagrant-docker/README.md`)

Provisioners

Providers

Setting up Vagrant-docker with the Remote API

Overview (`docs/static_files/README.md`)

Overview (`hack/Jenkins/readme.md`)

Overview (`hack/make/README.md`)

Overview (`pkg/archive/README.md`)

Overview (`pkg/discovery/README.md`)

page_title: Docker discovery page_description: discovery page_keywords: docker, clustering, discovery

Backends

Using etcd

Using consul

Using zookeeper

Overview (`pkg/locker/README.md`)

Usage

Overview (`pkg/mflag/README.md`)

Overview (`pkg/reexec/README.md`)

reexec

Overview (`pkg/signal/README.md`)

Overview (`pkg/stringid/README.md`)

Overview (`pkg/stringutils/README.md`)

Overview (`pkg/symlink/README.md`)

Overview (`pkg/sysinfo/README.md`)

Overview (`pkg/useragent/README.md`)

Overview (`contrib/syntax/nano/README.md`)

Single User Installation

System Wide Installation

Overview (`contrib/syntax/textmate/README.md`)

Install

Sublime Text

TextMate 2

Overview (`contrib/syntax/vim/README.md`)

Installation

Features

License

Overview (`daemon/graphdriver/devmapper/README.md`)

devicemapper - a storage backend based on Device Mapper

Theory of operation

Information on `docker info`

About the devicemapper options

Overview (`docs/reference/api/README.md`)