[](https://travis-ci.org/Sirupsen/logrus) [](https://godoc.org/github.com/Sirupsen/logrus)
Logrus is a structured logger for Go (golang), completely API compatible with
the standard library logger. [Godoc][godoc]. **Please note the Logrus API is not
yet stable (pre 1.0). Logrus itself is completely stable and has been used in
many large deployments. The core API is unlikely to change much but please
version control your Logrus to make sure you aren't fetching latest `master` on
every build.**
Nicely color-coded in development (when a TTY is attached, otherwise just
plain text):
With `log.SetFormatter(&log.JSONFormatter{})`, for easy parsing by logstash
or Splunk:
```json
{"animal":"walrus","level":"info","msg":"A group of walrus emerges from the
ocean","size":10,"time":"2014-03-10 19:57:38.562264131 -0400 EDT"}
{"level":"warning","msg":"The group's number increased tremendously!",
"number":122,"omg":true,"time":"2014-03-10 19:57:38.562471297 -0400 EDT"}
{"animal":"walrus","level":"info","msg":"A giant walrus appears!",
"size":10,"time":"2014-03-10 19:57:38.562500591 -0400 EDT"}
{"animal":"walrus","level":"info","msg":"Tremendously sized cow enters the ocean.",
"size":9,"time":"2014-03-10 19:57:38.562527896 -0400 EDT"}
{"level":"fatal","msg":"The ice breaks!","number":100,"omg":true,
"time":"2014-03-10 19:57:38.562543128 -0400 EDT"}
```
With the default `log.SetFormatter(&log.TextFormatter{})` when a TTY is not
attached, the output is compatible with the
[logfmt](http://godoc.org/github.com/kr/logfmt) format:
```text
time="2015-03-26T01:27:38-04:00" level=debug msg="Started observing beach" animal=walrus number=8
time="2015-03-26T01:27:38-04:00" level=info msg="A group of walrus emerges from the ocean" animal=walrus size=10
time="2015-03-26T01:27:38-04:00" level=warning msg="The group's number increased tremendously!" number=122 omg=true
time="2015-03-26T01:27:38-04:00" level=debug msg="Temperature changes" temperature=-4
time="2015-03-26T01:27:38-04:00" level=panic msg="It's over 9000!" animal=orca size=9009
time="2015-03-26T01:27:38-04:00" level=fatal msg="The ice breaks!" err=&{0x2082280c0 map[animal:orca size:9009] 2015-03-26 01:27:38.441574009 -0400 EDT panic It's over 9000!} number=100 omg=true
exit status 1
```
#### Example
The simplest way to use Logrus is simply the package-level exported logger:
```go
package main
import (
log "github.com/Sirupsen/logrus"
)
func main() {
log.WithFields(log.Fields{
"animal": "walrus",
}).Info("A walrus appears")
}
```
Note that it's completely api-compatible with the stdlib logger, so you can
replace your `log` imports everywhere with `log "github.com/Sirupsen/logrus"`
and you'll now have the flexibility of Logrus. You can customize it all you
want:
```go
package main
import (
"os"
log "github.com/Sirupsen/logrus"
)
func init() {
// Log as JSON instead of the default ASCII formatter.
log.SetFormatter(&log.JSONFormatter{})
// Output to stderr instead of stdout, could also be a file.
log.SetOutput(os.Stderr)
// Only log the warning severity or above.
log.SetLevel(log.WarnLevel)
}
func main() {
log.WithFields(log.Fields{
"animal": "walrus",
"size": 10,
}).Info("A group of walrus emerges from the ocean")
log.WithFields(log.Fields{
"omg": true,
"number": 122,
}).Warn("The group's number increased tremendously!")
log.WithFields(log.Fields{
"omg": true,
"number": 100,
}).Fatal("The ice breaks!")
// A common pattern is to re-use fields between logging statements by re-using
// the logrus.Entry returned from WithFields()
contextLogger := log.WithFields(log.Fields{
"common": "this is a common field",
"other": "I also should be logged always",
})
contextLogger.Info("I'll be logged with common and other field")
contextLogger.Info("Me too")
}
```
For more advanced usage such as logging to multiple locations from the same
application, you can also create an instance of the `logrus` Logger:
```go
package main
import (
"github.com/Sirupsen/logrus"
)
// Create a new instance of the logger. You can have any number of instances.
var log = logrus.New()
func main() {
// The API for setting attributes is a little different than the package level
// exported logger. See Godoc.
log.Out = os.Stderr
log.WithFields(logrus.Fields{
"animal": "walrus",
"size": 10,
}).Info("A group of walrus emerges from the ocean")
}
```
#### Fields
Logrus encourages careful, structured logging though logging fields instead of
long, unparseable error messages. For example, instead of: `log.Fatalf("Failed
to send event %s to topic %s with key %d")`, you should log the much more
discoverable:
```go
log.WithFields(log.Fields{
"event": event,
"topic": topic,
"key": key,
}).Fatal("Failed to send event")
```
We've found this API forces you to think about logging in a way that produces
much more useful logging messages. We've been in countless situations where just
a single added field to a log statement that was already there would've saved us
hours. The `WithFields` call is optional.
In general, with Logrus using any of the `printf`-family functions should be
seen as a hint you should add a field, however, you can still use the
`printf`-family functions with Logrus.
#### Hooks
You can add hooks for logging levels. For example to send errors to an exception
tracking service on `Error`, `Fatal` and `Panic`, info to StatsD or log to
multiple places simultaneously, e.g. syslog.
Logrus comes with [built-in hooks](hooks/). Add those, or your custom hook, in
`init`:
```go
import (
log "github.com/Sirupsen/logrus"
"gopkg.in/gemnasium/logrus-airbrake-hook.v2" // the package is named "aibrake"
logrus_syslog "github.com/Sirupsen/logrus/hooks/syslog"
"log/syslog"
)
func init() {
// Use the Airbrake hook to report errors that have Error severity or above to
// an exception tracker. You can create custom hooks, see the Hooks section.
log.AddHook(airbrake.NewHook(123, "xyz", "production"))
hook, err := logrus_syslog.NewSyslogHook("udp", "localhost:514", syslog.LOG_INFO, "")
if err != nil {
log.Error("Unable to connect to local syslog daemon")
} else {
log.AddHook(hook)
}
}
```
Note: Syslog hook also support connecting to local syslog (Ex. "/dev/log" or "/var/run/syslog" or "/var/run/log"). For the detail, please check the [syslog hook README](hooks/syslog/README.md).
| Hook | Description |
| ----- | ----------- |
| [Airbrake](https://github.com/gemnasium/logrus-airbrake-hook) | Send errors to the Airbrake API V3. Uses the official [`gobrake`](https://github.com/airbrake/gobrake) behind the scenes. |
| [Airbrake "legacy"](https://github.com/gemnasium/logrus-airbrake-legacy-hook) | Send errors to an exception tracking service compatible with the Airbrake API V2. Uses [`airbrake-go`](https://github.com/tobi/airbrake-go) behind the scenes. |
| [Papertrail](https://github.com/polds/logrus-papertrail-hook) | Send errors to the [Papertrail](https://papertrailapp.com) hosted logging service via UDP. |
| [Syslog](https://github.com/Sirupsen/logrus/blob/master/hooks/syslog/syslog.go) | Send errors to remote syslog server. Uses standard library `log/syslog` behind the scenes. |
| [Bugsnag](https://github.com/Shopify/logrus-bugsnag/blob/master/bugsnag.go) | Send errors to the Bugsnag exception tracking service. |
| [Sentry](https://github.com/evalphobia/logrus_sentry) | Send errors to the Sentry error logging and aggregation service. |
| [Hiprus](https://github.com/nubo/hiprus) | Send errors to a channel in hipchat. |
| [Logrusly](https://github.com/sebest/logrusly) | Send logs to [Loggly](https://www.loggly.com/) |
| [Slackrus](https://github.com/johntdyer/slackrus) | Hook for Slack chat. |
| [Journalhook](https://github.com/wercker/journalhook) | Hook for logging to `systemd-journald` |
| [Graylog](https://github.com/gemnasium/logrus-graylog-hook) | Hook for logging to [Graylog](http://graylog2.org/) |
| [Raygun](https://github.com/squirkle/logrus-raygun-hook) | Hook for logging to [Raygun.io](http://raygun.io/) |
| [LFShook](https://github.com/rifflock/lfshook) | Hook for logging to the local filesystem |
| [Honeybadger](https://github.com/agonzalezro/logrus_honeybadger) | Hook for sending exceptions to Honeybadger |
| [Mail](https://github.com/zbindenren/logrus_mail) | Hook for sending exceptions via mail |
| [Rollrus](https://github.com/heroku/rollrus) | Hook for sending errors to rollbar |
| [Fluentd](https://github.com/evalphobia/logrus_fluent) | Hook for logging to fluentd |
| [Mongodb](https://github.com/weekface/mgorus) | Hook for logging to mongodb |
| [InfluxDB](https://github.com/Abramovic/logrus_influxdb) | Hook for logging to influxdb |
| [Octokit](https://github.com/dorajistyle/logrus-octokit-hook) | Hook for logging to github via octokit |
| [DeferPanic](https://github.com/deferpanic/dp-logrus) | Hook for logging to DeferPanic |
| [Redis-Hook](https://github.com/rogierlommers/logrus-redis-hook) | Hook for logging to a ELK stack (through Redis) |
| [Amqp-Hook](https://github.com/vladoatanasov/logrus_amqp) | Hook for logging to Amqp broker (Like RabbitMQ) |
| [KafkaLogrus](https://github.com/goibibo/KafkaLogrus) | Hook for logging to kafka |
| [Typetalk](https://github.com/dragon3/logrus-typetalk-hook) | Hook for logging to [Typetalk](https://www.typetalk.in/) |
| [ElasticSearch](https://github.com/sohlich/elogrus) | Hook for logging to ElasticSearch|
#### Level logging
Logrus has six logging levels: Debug, Info, Warning, Error, Fatal and Panic.
```go
log.Debug("Useful debugging information.")
log.Info("Something noteworthy happened!")
log.Warn("You should probably take a look at this.")
log.Error("Something failed but I'm not quitting.")
// Calls os.Exit(1) after logging
log.Fatal("Bye.")
// Calls panic() after logging
log.Panic("I'm bailing.")
```
You can set the logging level on a `Logger`, then it will only log entries with
that severity or anything above it:
```go
// Will log anything that is info or above (warn, error, fatal, panic). Default.
log.SetLevel(log.InfoLevel)
```
It may be useful to set `log.Level = logrus.DebugLevel` in a debug or verbose
environment if your application has that.
#### Entries
Besides the fields added with `WithField` or `WithFields` some fields are
automatically added to all logging events:
1. `time`. The timestamp when the entry was created.
2. `msg`. The logging message passed to `{Info,Warn,Error,Fatal,Panic}` after
the `AddFields` call. E.g. `Failed to send event.`
3. `level`. The logging level. E.g. `info`.
#### Environments
Logrus has no notion of environment.
If you wish for hooks and formatters to only be used in specific environments,
you should handle that yourself. For example, if your application has a global
variable `Environment`, which is a string representation of the environment you
could do:
```go
import (
log "github.com/Sirupsen/logrus"
)
init() {
// do something here to set environment depending on an environment variable
// or command-line flag
if Environment == "production" {
log.SetFormatter(&log.JSONFormatter{})
} else {
// The TextFormatter is default, you don't actually have to do this.
log.SetFormatter(&log.TextFormatter{})
}
}
```
This configuration is how `logrus` was intended to be used, but JSON in
production is mostly only useful if you do log aggregation with tools like
Splunk or Logstash.
#### Formatters
The built-in logging formatters are:
* `logrus.TextFormatter`. Logs the event in colors if stdout is a tty, otherwise
without colors.
* *Note:* to force colored output when there is no TTY, set the `ForceColors`
field to `true`. To force no colored output even if there is a TTY set the
`DisableColors` field to `true`
* `logrus.JSONFormatter`. Logs fields as JSON.
* `logrus/formatters/logstash.LogstashFormatter`. Logs fields as [Logstash](http://logstash.net) Events.
```go
logrus.SetFormatter(&logstash.LogstashFormatter{Type: "application_name"})
```
Third party logging formatters:
* [`prefixed`](https://github.com/x-cray/logrus-prefixed-formatter). Displays log entry source along with alternative layout.
* [`zalgo`](https://github.com/aybabtme/logzalgo). Invoking the P͉̫o̳̼̊w̖͈̰͎e̬͔̭͂r͚̼̹̲ ̫͓͉̳͈ō̠͕͖̚f̝͍̠ ͕̲̞͖͑Z̖̫̤̫ͪa͉̬͈̗l͖͎g̳̥o̰̥̅!̣͔̲̻͊̄ ̙̘̦̹̦.
You can define your formatter by implementing the `Formatter` interface,
requiring a `Format` method. `Format` takes an `*Entry`. `entry.Data` is a
`Fields` type (`map[string]interface{}`) with all your fields as well as the
default ones (see Entries section above):
```go
type MyJSONFormatter struct {
}
log.SetFormatter(new(MyJSONFormatter))
func (f *MyJSONFormatter) Format(entry *Entry) ([]byte, error) {
// Note this doesn't include Time, Level and Message which are available on
// the Entry. Consult `godoc` on information about those fields or read the
// source of the official loggers.
serialized, err := json.Marshal(entry.Data)
if err != nil {
return nil, fmt.Errorf("Failed to marshal fields to JSON, %v", err)
}
return append(serialized, '\n'), nil
}
```
#### Logger as an `io.Writer`
Logrus can be transformed into an `io.Writer`. That writer is the end of an `io.Pipe` and it is your responsibility to close it.
```go
w := logger.Writer()
defer w.Close()
srv := http.Server{
// create a stdlib log.Logger that writes to
// logrus.Logger.
ErrorLog: log.New(w, "", 0),
}
```
Each line written to that writer will be printed the usual way, using formatters
and hooks. The level for those entries is `info`.
#### Rotation
Log rotation is not provided with Logrus. Log rotation should be done by an
external program (like `logrotate(8)`) that can compress and delete old log
entries. It should not be a feature of the application-level logger.
#### Tools
| Tool | Description |
| ---- | ----------- |
|[Logrus Mate](https://github.com/gogap/logrus_mate)|Logrus mate is a tool for Logrus to manage loggers, you can initial logger's level, hook and formatter by config file, the logger will generated with different config at different environment.|
#### Testing
Logrus has a built in facility for asserting the presence of log messages. This is implemented through the `test` hook and provides:
* decorators for existing logger (`test.NewLocal` and `test.NewGlobal`) which basically just add the `test` hook
* a test logger (`test.NewNullLogger`) that just records log messages (and does not output any):
```go
logger, hook := NewNullLogger()
logger.Error("Hello error")
assert.Equal(1, len(hook.Entries))
assert.Equal(logrus.ErrorLevel, hook.LastEntry().Level)
assert.Equal("Hello error", hook.LastEntry().Message)
hook.Reset()
assert.Nil(hook.LastEntry())
```
---
### Overview (`vendor/src/github.com/armon/go-metrics/README.md`)
go-metrics
==========
This library provides a `metrics` package which can be used to instrument code,
expose application metrics, and profile runtime performance in a flexible manner.
Sinks
=====
The `metrics` package makes use of a `MetricSink` interface to support delivery
to any type of backend. Currently the following sinks are provided:
* StatsiteSink : Sinks to a statsite instance (TCP)
* StatsdSink: Sinks to a statsd / statsite instance (UDP)
* InmemSink : Provides in-memory aggregation, can be used to export stats
* FanoutSink : Sinks to multiple sinks. Enables writing to multiple statsite instances for example.
* BlackholeSink : Sinks to nowhere
In addition to the sinks, the `InmemSignal` can be used to catch a signal,
and dump a formatted output of recent metrics. For example, when a process gets
a SIGUSR1, it can dump to stderr recent performance metrics for debugging.
Examples
========
Here is an example of using the package:
func SlowMethod() {
// Profiling the runtime of a method
defer metrics.MeasureSince([]string{"SlowMethod"}, time.Now())
}
// Configure a statsite sink as the global metrics sink
sink, _ := metrics.NewStatsiteSink("statsite:8125")
metrics.NewGlobal(metrics.DefaultConfig("service-name"), sink)
// Emit a Key/Value pair
metrics.EmitKey([]string{"questions", "meaning of life"}, 42)
Here is an example of setting up an signal handler:
// Setup the inmem sink and signal handler
inm := NewInmemSink(10*time.Second, time.Minute)
sig := DefaultInmemSignal(inm)
metrics.NewGlobal(metrics.DefaultConfig("service-name"), inm)
// Run some code
inm.SetGauge([]string{"foo"}, 42)
inm.EmitKey([]string{"bar"}, 30)
inm.IncrCounter([]string{"baz"}, 42)
inm.IncrCounter([]string{"baz"}, 1)
inm.IncrCounter([]string{"baz"}, 80)
inm.AddSample([]string{"method", "wow"}, 42)
inm.AddSample([]string{"method", "wow"}, 100)
inm.AddSample([]string{"method", "wow"}, 22)
....
When a signal comes in, output like the following will be dumped to stderr:
[2014-01-28 14:57:33.04 -0800 PST][G] 'foo': 42.000
[2014-01-28 14:57:33.04 -0800 PST][P] 'bar': 30.000
[2014-01-28 14:57:33.04 -0800 PST][C] 'baz': Count: 3 Min: 1.000 Mean: 41.000 Max: 80.000 Stddev: 39.509
[2014-01-28 14:57:33.04 -0800 PST][S] 'method.wow': Count: 3 Min: 22.000 Mean: 54.667 Max: 100.000 Stddev: 40.513
---
### Overview (`vendor/src/github.com/armon/go-radix/README.md`)
go-radix [](https://travis-ci.org/armon/go-radix)
=========
Provides the `radix` package that implements a [radix tree](http://en.wikipedia.org/wiki/Radix_tree).
The package only provides a single `Tree` implementation, optimized for sparse nodes.
As a radix tree, it provides the following:
* O(k) operations. In many cases, this can be faster than a hash table since
the hash function is an O(k) operation, and hash tables have very poor cache locality.
* Minimum / Maximum value lookups
* Ordered iteration
Documentation
=============
The full documentation is available on [Godoc](http://godoc.org/github.com/armon/go-radix).
Example
=======
Below is a simple example of usage
```go
// Create a tree
r := radix.New()
r.Insert("foo", 1)
r.Insert("bar", 2)
r.Insert("foobar", 2)
// Find the longest prefix match
m, _, _ := r.LongestPrefix("foozip")
if m != "foo" {
panic("should be foo")
}
```
---
### Overview (`vendor/src/github.com/boltdb/bolt/README.md`)
Bolt [](https://coveralls.io/r/boltdb/bolt?branch=master) [](https://godoc.org/github.com/boltdb/bolt)
====
Bolt is a pure Go key/value store inspired by [Howard Chu's][hyc_symas]
[LMDB project][lmdb]. The goal of the project is to provide a simple,
fast, and reliable database for projects that don't require a full database
server such as Postgres or MySQL.
Since Bolt is meant to be used as such a low-level piece of functionality,
simplicity is key. The API will be small and only focus on getting values
and setting values. That's it.
[hyc_symas]: https://twitter.com/hyc_symas
[lmdb]: http://symas.com/mdb/
## Project Status
Bolt is stable and the API is fixed. Full unit test coverage and randomized
black box testing are used to ensure database consistency and thread safety.
Bolt is currently in high-load production environments serving databases as
large as 1TB. Many companies such as Shopify and Heroku use Bolt-backed
services every day.
## Table of Contents
- [Getting Started](#getting-started)
- [Installing](#installing)
- [Opening a database](#opening-a-database)
- [Transactions](#transactions)
- [Read-write transactions](#read-write-transactions)
- [Read-only transactions](#read-only-transactions)
- [Batch read-write transactions](#batch-read-write-transactions)
- [Managing transactions manually](#managing-transactions-manually)
- [Using buckets](#using-buckets)
- [Using key/value pairs](#using-keyvalue-pairs)
- [Autoincrementing integer for the bucket](#autoincrementing-integer-for-the-bucket)
- [Iterating over keys](#iterating-over-keys)
- [Prefix scans](#prefix-scans)
- [Range scans](#range-scans)
- [ForEach()](#foreach)
- [Nested buckets](#nested-buckets)
- [Database backups](#database-backups)
- [Statistics](#statistics)
- [Read-Only Mode](#read-only-mode)
- [Mobile Use (iOS/Android)](#mobile-use-iosandroid)
- [Resources](#resources)
- [Comparison with other databases](#comparison-with-other-databases)
- [Postgres, MySQL, & other relational databases](#postgres-mysql--other-relational-databases)
- [LevelDB, RocksDB](#leveldb-rocksdb)
- [LMDB](#lmdb)
- [Caveats & Limitations](#caveats--limitations)
- [Reading the Source](#reading-the-source)
- [Other Projects Using Bolt](#other-projects-using-bolt)
## Getting Started
### Installing
To start using Bolt, install Go and run `go get`:
```sh
$ go get github.com/boltdb/bolt/...
```
This will retrieve the library and install the `bolt` command line utility into
your `$GOBIN` path.
### Opening a database
The top-level object in Bolt is a `DB`. It is represented as a single file on
your disk and represents a consistent snapshot of your data.
To open your database, simply use the `bolt.Open()` function:
```go
package main
import (
"log"
"github.com/boltdb/bolt"
)
func main() {
// Open the my.db data file in your current directory.
// It will be created if it doesn't exist.
db, err := bolt.Open("my.db", 0600, nil)
if err != nil {
log.Fatal(err)
}
defer db.Close()
...
}
```
Please note that Bolt obtains a file lock on the data file so multiple processes
cannot open the same database at the same time. Opening an already open Bolt
database will cause it to hang until the other process closes it. To prevent
an indefinite wait you can pass a timeout option to the `Open()` function:
```go
db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second})
```
### Transactions
Bolt allows only one read-write transaction at a time but allows as many
read-only transactions as you want at a time. Each transaction has a consistent
view of the data as it existed when the transaction started.
Individual transactions and all objects created from them (e.g. buckets, keys)
are not thread safe. To work with data in multiple goroutines you must start
a transaction for each one or use locking to ensure only one goroutine accesses
a transaction at a time. Creating transaction from the `DB` is thread safe.
Read-only transactions and read-write transactions should not depend on one
another and generally shouldn't be opened simultaneously in the same goroutine.
This can cause a deadlock as the read-write transaction needs to periodically
re-map the data file but it cannot do so while a read-only transaction is open.
#### Read-write transactions
To start a read-write transaction, you can use the `DB.Update()` function:
```go
err := db.Update(func(tx *bolt.Tx) error {
...
return nil
})
```
Inside the closure, you have a consistent view of the database. You commit the
transaction by returning `nil` at the end. You can also rollback the transaction
at any point by returning an error. All database operations are allowed inside
a read-write transaction.
Always check the return error as it will report any disk failures that can cause
your transaction to not complete. If you return an error within your closure
it will be passed through.
#### Read-only transactions
To start a read-only transaction, you can use the `DB.View()` function:
```go
err := db.View(func(tx *bolt.Tx) error {
...
return nil
})
```
You also get a consistent view of the database within this closure, however,
no mutating operations are allowed within a read-only transaction. You can only
retrieve buckets, retrieve values, and copy the database within a read-only
transaction.
#### Batch read-write transactions
Each `DB.Update()` waits for disk to commit the writes. This overhead
can be minimized by combining multiple updates with the `DB.Batch()`
function:
```go
err := db.Batch(func(tx *bolt.Tx) error {
...
return nil
})
```
Concurrent Batch calls are opportunistically combined into larger
transactions. Batch is only useful when there are multiple goroutines
calling it.
The trade-off is that `Batch` can call the given
function multiple times, if parts of the transaction fail. The
function must be idempotent and side effects must take effect only
after a successful return from `DB.Batch()`.
For example: don't display messages from inside the function, instead
set variables in the enclosing scope:
```go
var id uint64
err := db.Batch(func(tx *bolt.Tx) error {
// Find last key in bucket, decode as bigendian uint64, increment
// by one, encode back to []byte, and add new key.
...
id = newValue
return nil
})
if err != nil {
return ...
}
fmt.Println("Allocated ID %d", id)
```
#### Managing transactions manually
The `DB.View()` and `DB.Update()` functions are wrappers around the `DB.Begin()`
function. These helper functions will start the transaction, execute a function,
and then safely close your transaction if an error is returned. This is the
recommended way to use Bolt transactions.
However, sometimes you may want to manually start and end your transactions.
You can use the `Tx.Begin()` function directly but **please** be sure to close
the transaction.
```go
// Start a writable transaction.
tx, err := db.Begin(true)
if err != nil {
return err
}
defer tx.Rollback()
// Use the transaction...
_, err := tx.CreateBucket([]byte("MyBucket"))
if err != nil {
return err
}
// Commit the transaction and check for error.
if err := tx.Commit(); err != nil {
return err
}
```
The first argument to `DB.Begin()` is a boolean stating if the transaction
should be writable.
### Using buckets
Buckets are collections of key/value pairs within the database. All keys in a
bucket must be unique. You can create a bucket using the `DB.CreateBucket()`
function:
```go
db.Update(func(tx *bolt.Tx) error {
b, err := tx.CreateBucket([]byte("MyBucket"))
if err != nil {
return fmt.Errorf("create bucket: %s", err)
}
return nil
})
```
You can also create a bucket only if it doesn't exist by using the
`Tx.CreateBucketIfNotExists()` function. It's a common pattern to call this
function for all your top-level buckets after you open your database so you can
guarantee that they exist for future transactions.
To delete a bucket, simply call the `Tx.DeleteBucket()` function.
### Using key/value pairs
To save a key/value pair to a bucket, use the `Bucket.Put()` function:
```go
db.Update(func(tx *bolt.Tx) error {
b := tx.Bucket([]byte("MyBucket"))
err := b.Put([]byte("answer"), []byte("42"))
return err
})
```
This will set the value of the `"answer"` key to `"42"` in the `MyBucket`
bucket. To retrieve this value, we can use the `Bucket.Get()` function:
```go
db.View(func(tx *bolt.Tx) error {
b := tx.Bucket([]byte("MyBucket"))
v := b.Get([]byte("answer"))
fmt.Printf("The answer is: %s\n", v)
return nil
})
```
The `Get()` function does not return an error because its operation is
guaranteed to work (unless there is some kind of system failure). If the key
exists then it will return its byte slice value. If it doesn't exist then it
will return `nil`. It's important to note that you can have a zero-length value
set to a key which is different than the key not existing.
Use the `Bucket.Delete()` function to delete a key from the bucket.
Please note that values returned from `Get()` are only valid while the
transaction is open. If you need to use a value outside of the transaction
then you must use `copy()` to copy it to another byte slice.
### Autoincrementing integer for the bucket
By using the `NextSequence()` function, you can let Bolt determine a sequence
which can be used as the unique identifier for your key/value pairs. See the
example below.
```go
// CreateUser saves u to the store. The new user ID is set on u once the data is persisted.
func (s *Store) CreateUser(u *User) error {
return s.db.Update(func(tx *bolt.Tx) error {
// Retrieve the users bucket.
// This should be created when the DB is first opened.
b := tx.Bucket([]byte("users"))
// Generate ID for the user.
// This returns an error only if the Tx is closed or not writeable.
// That can't happen in an Update() call so I ignore the error check.
id, _ = b.NextSequence()
u.ID = int(id)
// Marshal user data into bytes.
buf, err := json.Marshal(u)
if err != nil {
return err
}
// Persist bytes to users bucket.
return b.Put(itob(u.ID), buf)
})
}
// itob returns an 8-byte big endian representation of v.
func itob(v int) []byte {
b := make([]byte, 8)
binary.BigEndian.PutUint64(b, uint64(v))
return b
}
type User struct {
ID int
...
}
```
### Iterating over keys
Bolt stores its keys in byte-sorted order within a bucket. This makes sequential
iteration over these keys extremely fast. To iterate over keys we'll use a
`Cursor`:
```go
db.View(func(tx *bolt.Tx) error {
// Assume bucket exists and has keys
b := tx.Bucket([]byte("MyBucket"))
c := b.Cursor()
for k, v := c.First(); k != nil; k, v = c.Next() {
fmt.Printf("key=%s, value=%s\n", k, v)
}
return nil
})
```
The cursor allows you to move to a specific point in the list of keys and move
forward or backward through the keys one at a time.
The following functions are available on the cursor:
```
First() Move to the first key.
Last() Move to the last key.
Seek() Move to a specific key.
Next() Move to the next key.
Prev() Move to the previous key.
```
Each of those functions has a return signature of `(key []byte, value []byte)`.
When you have iterated to the end of the cursor then `Next()` will return a
`nil` key. You must seek to a position using `First()`, `Last()`, or `Seek()`
before calling `Next()` or `Prev()`. If you do not seek to a position then
these functions will return a `nil` key.
During iteration, if the key is non-`nil` but the value is `nil`, that means
the key refers to a bucket rather than a value. Use `Bucket.Bucket()` to
access the sub-bucket.
#### Prefix scans
To iterate over a key prefix, you can combine `Seek()` and `bytes.HasPrefix()`:
```go
db.View(func(tx *bolt.Tx) error {
// Assume bucket exists and has keys
c := tx.Bucket([]byte("MyBucket")).Cursor()
prefix := []byte("1234")
for k, v := c.Seek(prefix); bytes.HasPrefix(k, prefix); k, v = c.Next() {
fmt.Printf("key=%s, value=%s\n", k, v)
}
return nil
})
```
#### Range scans
Another common use case is scanning over a range such as a time range. If you
use a sortable time encoding such as RFC3339 then you can query a specific
date range like this:
```go
db.View(func(tx *bolt.Tx) error {
// Assume our events bucket exists and has RFC3339 encoded time keys.
c := tx.Bucket([]byte("Events")).Cursor()
// Our time range spans the 90's decade.
min := []byte("1990-01-01T00:00:00Z")
max := []byte("2000-01-01T00:00:00Z")
// Iterate over the 90's.
for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() {
fmt.Printf("%s: %s\n", k, v)
}
return nil
})
```
Note that, while RFC3339 is sortable, the Golang implementation of RFC3339Nano does not use a fixed number of digits after the decimal point and is therefore not sortable.
#### ForEach()
You can also use the function `ForEach()` if you know you'll be iterating over
all the keys in a bucket:
```go
db.View(func(tx *bolt.Tx) error {
// Assume bucket exists and has keys
b := tx.Bucket([]byte("MyBucket"))
b.ForEach(func(k, v []byte) error {
fmt.Printf("key=%s, value=%s\n", k, v)
return nil
})
return nil
})
```
### Nested buckets
You can also store a bucket in a key to create nested buckets. The API is the
same as the bucket management API on the `DB` object:
```go
func (*Bucket) CreateBucket(key []byte) (*Bucket, error)
func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error)
func (*Bucket) DeleteBucket(key []byte) error
```
### Database backups
Bolt is a single file so it's easy to backup. You can use the `Tx.WriteTo()`
function to write a consistent view of the database to a writer. If you call
this from a read-only transaction, it will perform a hot backup and not block
your other database reads and writes.
By default, it will use a regular file handle which will utilize the operating
system's page cache. See the [`Tx`](https://godoc.org/github.com/boltdb/bolt#Tx)
documentation for information about optimizing for larger-than-RAM datasets.
One common use case is to backup over HTTP so you can use tools like `cURL` to
do database backups:
```go
func BackupHandleFunc(w http.ResponseWriter, req *http.Request) {
err := db.View(func(tx *bolt.Tx) error {
w.Header().Set("Content-Type", "application/octet-stream")
w.Header().Set("Content-Disposition", `attachment; filename="my.db"`)
w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size())))
_, err := tx.WriteTo(w)
return err
})
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
```
Then you can backup using this command:
```sh
$ curl http://localhost/backup > my.db
```
Or you can open your browser to `http://localhost/backup` and it will download
automatically.
If you want to backup to another file you can use the `Tx.CopyFile()` helper
function.
### Statistics
The database keeps a running count of many of the internal operations it
performs so you can better understand what's going on. By grabbing a snapshot
of these stats at two points in time we can see what operations were performed
in that time range.
For example, we could start a goroutine to log stats every 10 seconds:
```go
go func() {
// Grab the initial stats.
prev := db.Stats()
for {
// Wait for 10s.
time.Sleep(10 * time.Second)
// Grab the current stats and diff them.
stats := db.Stats()
diff := stats.Sub(&prev)
// Encode stats to JSON and print to STDERR.
json.NewEncoder(os.Stderr).Encode(diff)
// Save stats for the next loop.
prev = stats
}
}()
```
It's also useful to pipe these stats to a service such as statsd for monitoring
or to provide an HTTP endpoint that will perform a fixed-length sample.
### Read-Only Mode
Sometimes it is useful to create a shared, read-only Bolt database. To this,
set the `Options.ReadOnly` flag when opening your database. Read-only mode
uses a shared lock to allow multiple processes to read from the database but
it will block any processes from opening the database in read-write mode.
```go
db, err := bolt.Open("my.db", 0666, &bolt.Options{ReadOnly: true})
if err != nil {
log.Fatal(err)
}
```
### Mobile Use (iOS/Android)
Bolt is able to run on mobile devices by leveraging the binding feature of the
[gomobile](https://github.com/golang/mobile) tool. Create a struct that will
contain your database logic and a reference to a `*bolt.DB` with a initializing
contstructor that takes in a filepath where the database file will be stored.
Neither Android nor iOS require extra permissions or cleanup from using this method.
```go
func NewBoltDB(filepath string) *BoltDB {
db, err := bolt.Open(filepath+"/demo.db", 0600, nil)
if err != nil {
log.Fatal(err)
}
return &BoltDB{db}
}
type BoltDB struct {
db *bolt.DB
...
}
func (b *BoltDB) Path() string {
return b.db.Path()
}
func (b *BoltDB) Close() {
b.db.Close()
}
```
Database logic should be defined as methods on this wrapper struct.
To initialize this struct from the native language (both platforms now sync
their local storage to the cloud. These snippets disable that functionality for the
database file):
#### Android
```java
String path;
if (android.os.Build.VERSION.SDK_INT >=android.os.Build.VERSION_CODES.LOLLIPOP){
path = getNoBackupFilesDir().getAbsolutePath();
} else{
path = getFilesDir().getAbsolutePath();
}
Boltmobiledemo.BoltDB boltDB = Boltmobiledemo.NewBoltDB(path)
```
#### iOS
```objc
- (void)demo {
NSString* path = [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory,
NSUserDomainMask,
YES) objectAtIndex:0];
GoBoltmobiledemoBoltDB * demo = GoBoltmobiledemoNewBoltDB(path);
[self addSkipBackupAttributeToItemAtPath:demo.path];
//Some DB Logic would go here
[demo close];
}
- (BOOL)addSkipBackupAttributeToItemAtPath:(NSString *) filePathString
{
NSURL* URL= [NSURL fileURLWithPath: filePathString];
assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]);
NSError *error = nil;
BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES]
forKey: NSURLIsExcludedFromBackupKey error: &error];
if(!success){
NSLog(@"Error excluding %@ from backup %@", [URL lastPathComponent], error);
}
return success;
}
```
## Resources
For more information on getting started with Bolt, check out the following articles:
* [Intro to BoltDB: Painless Performant Persistence](http://npf.io/2014/07/intro-to-boltdb-painless-performant-persistence/) by [Nate Finch](https://github.com/natefinch).
* [Bolt -- an embedded key/value database for Go](https://www.progville.com/go/bolt-embedded-db-golang/) by Progville
## Comparison with other databases
### Postgres, MySQL, & other relational databases
Relational databases structure data into rows and are only accessible through
the use of SQL. This approach provides flexibility in how you store and query
your data but also incurs overhead in parsing and planning SQL statements. Bolt
accesses all data by a byte slice key. This makes Bolt fast to read and write
data by key but provides no built-in support for joining values together.
Most relational databases (with the exception of SQLite) are standalone servers
that run separately from your application. This gives your systems
flexibility to connect multiple application servers to a single database
server but also adds overhead in serializing and transporting data over the
network. Bolt runs as a library included in your application so all data access
has to go through your application's process. This brings data closer to your
application but limits multi-process access to the data.
### LevelDB, RocksDB
LevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Bolt in that
they are libraries bundled into the application, however, their underlying
structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes
random writes by using a write ahead log and multi-tiered, sorted files called
SSTables. Bolt uses a B+tree internally and only a single file. Both approaches
have trade-offs.
If you require a high random write throughput (>10,000 w/sec) or you need to use
spinning disks then LevelDB could be a good choice. If your application is
read-heavy or does a lot of range scans then Bolt could be a good choice.
One other important consideration is that LevelDB does not have transactions.
It supports batch writing of key/values pairs and it supports read snapshots
but it will not give you the ability to do a compare-and-swap operation safely.
Bolt supports fully serializable ACID transactions.
### LMDB
Bolt was originally a port of LMDB so it is architecturally similar. Both use
a B+tree, have ACID semantics with fully serializable transactions, and support
lock-free MVCC using a single writer and multiple readers.
The two projects have somewhat diverged. LMDB heavily focuses on raw performance
while Bolt has focused on simplicity and ease of use. For example, LMDB allows
several unsafe actions such as direct writes for the sake of performance. Bolt
opts to disallow actions which can leave the database in a corrupted state. The
only exception to this in Bolt is `DB.NoSync`.
There are also a few differences in API. LMDB requires a maximum mmap size when
opening an `mdb_env` whereas Bolt will handle incremental mmap resizing
automatically. LMDB overloads the getter and setter functions with multiple
flags whereas Bolt splits these specialized cases into their own functions.
## Caveats & Limitations
It's important to pick the right tool for the job and Bolt is no exception.
Here are a few things to note when evaluating and using Bolt:
* Bolt is good for read intensive workloads. Sequential write performance is
also fast but random writes can be slow. You can use `DB.Batch()` or add a
write-ahead log to help mitigate this issue.
* Bolt uses a B+tree internally so there can be a lot of random page access.
SSDs provide a significant performance boost over spinning disks.
* Try to avoid long running read transactions. Bolt uses copy-on-write so
old pages cannot be reclaimed while an old transaction is using them.
* Byte slices returned from Bolt are only valid during a transaction. Once the
transaction has been committed or rolled back then the memory they point to
can be reused by a new page or can be unmapped from virtual memory and you'll
see an `unexpected fault address` panic when accessing it.
* Be careful when using `Bucket.FillPercent`. Setting a high fill percent for
buckets that have random inserts will cause your database to have very poor
page utilization.
* Use larger buckets in general. Smaller buckets causes poor page utilization
once they become larger than the page size (typically 4KB).
* Bulk loading a lot of random writes into a new bucket can be slow as the
page will not split until the transaction is committed. Randomly inserting
more than 100,000 key/value pairs into a single new bucket in a single
transaction is not advised.
* Bolt uses a memory-mapped file so the underlying operating system handles the
caching of the data. Typically, the OS will cache as much of the file as it
can in memory and will release memory as needed to other processes. This means
that Bolt can show very high memory usage when working with large databases.
However, this is expected and the OS will release memory as needed. Bolt can
handle databases much larger than the available physical RAM, provided its
memory-map fits in the process virtual address space. It may be problematic
on 32-bits systems.
* The data structures in the Bolt database are memory mapped so the data file
will be endian specific. This means that you cannot copy a Bolt file from a
little endian machine to a big endian machine and have it work. For most
users this is not a concern since most modern CPUs are little endian.
* Because of the way pages are laid out on disk, Bolt cannot truncate data files
and return free pages back to the disk. Instead, Bolt maintains a free list
of unused pages within its data file. These free pages can be reused by later
transactions. This works well for many use cases as databases generally tend
to grow. However, it's important to note that deleting large chunks of data
will not allow you to reclaim that space on disk.
For more information on page allocation, [see this comment][page-allocation].
[page-allocation]: https://github.com/boltdb/bolt/issues/308#issuecomment-74811638
## Reading the Source
Bolt is a relatively small code base (<3KLOC) for an embedded, serializable,
transactional key/value database so it can be a good starting point for people
interested in how databases work.
The best places to start are the main entry points into Bolt:
- `Open()` - Initializes the reference to the database. It's responsible for
creating the database if it doesn't exist, obtaining an exclusive lock on the
file, reading the meta pages, & memory-mapping the file.
- `DB.Begin()` - Starts a read-only or read-write transaction depending on the
value of the `writable` argument. This requires briefly obtaining the "meta"
lock to keep track of open transactions. Only one read-write transaction can
exist at a time so the "rwlock" is acquired during the life of a read-write
transaction.
- `Bucket.Put()` - Writes a key/value pair into a bucket. After validating the
arguments, a cursor is used to traverse the B+tree to the page and position
where they key & value will be written. Once the position is found, the bucket
materializes the underlying page and the page's parent pages into memory as
"nodes". These nodes are where mutations occur during read-write transactions.
These changes get flushed to disk during commit.
- `Bucket.Get()` - Retrieves a key/value pair from a bucket. This uses a cursor
to move to the page & position of a key/value pair. During a read-only
transaction, the key and value data is returned as a direct reference to the
underlying mmap file so there's no allocation overhead. For read-write
transactions, this data may reference the mmap file or one of the in-memory
node values.
- `Cursor` - This object is simply for traversing the B+tree of on-disk pages
or in-memory nodes. It can seek to a specific key, move to the first or last
value, or it can move forward or backward. The cursor handles the movement up
and down the B+tree transparently to the end user.
- `Tx.Commit()` - Converts the in-memory dirty nodes and the list of free pages
into pages to be written to disk. Writing to disk then occurs in two phases.
First, the dirty pages are written to disk and an `fsync()` occurs. Second, a
new meta page with an incremented transaction ID is written and another
`fsync()` occurs. This two phase write ensures that partially written data
pages are ignored in the event of a crash since the meta page pointing to them
is never written. Partially written meta pages are invalidated because they
are written with a checksum.
If you have additional notes that could be helpful for others, please submit
them via pull request.
## Other Projects Using Bolt
Below is a list of public, open source projects that use Bolt:
* [Operation Go: A Routine Mission](http://gocode.io) - An online programming game for Golang using Bolt for user accounts and a leaderboard.
* [Bazil](https://bazil.org/) - A file system that lets your data reside where it is most convenient for it to reside.
* [DVID](https://github.com/janelia-flyem/dvid) - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb.
* [Skybox Analytics](https://github.com/skybox/skybox) - A standalone funnel analysis tool for web analytics.
* [Scuttlebutt](https://github.com/benbjohnson/scuttlebutt) - Uses Bolt to store and process all Twitter mentions of GitHub projects.
* [Wiki](https://github.com/peterhellberg/wiki) - A tiny wiki using Goji, BoltDB and Blackfriday.
* [ChainStore](https://github.com/pressly/chainstore) - Simple key-value interface to a variety of storage engines organized as a chain of operations.
* [MetricBase](https://github.com/msiebuhr/MetricBase) - Single-binary version of Graphite.
* [Gitchain](https://github.com/gitchain/gitchain) - Decentralized, peer-to-peer Git repositories aka "Git meets Bitcoin".
* [event-shuttle](https://github.com/sclasen/event-shuttle) - A Unix system service to collect and reliably deliver messages to Kafka.
* [ipxed](https://github.com/kelseyhightower/ipxed) - Web interface and api for ipxed.
* [BoltStore](https://github.com/yosssi/boltstore) - Session store using Bolt.
* [photosite/session](https://godoc.org/bitbucket.org/kardianos/photosite/session) - Sessions for a photo viewing site.
* [LedisDB](https://github.com/siddontang/ledisdb) - A high performance NoSQL, using Bolt as optional storage.
* [ipLocator](https://github.com/AndreasBriese/ipLocator) - A fast ip-geo-location-server using bolt with bloom filters.
* [cayley](https://github.com/google/cayley) - Cayley is an open-source graph database using Bolt as optional backend.
* [bleve](http://www.blevesearch.com/) - A pure Go search engine similar to ElasticSearch that uses Bolt as the default storage backend.
* [tentacool](https://github.com/optiflows/tentacool) - REST api server to manage system stuff (IP, DNS, Gateway...) on a linux server.
* [SkyDB](https://github.com/skydb/sky) - Behavioral analytics database.
* [Seaweed File System](https://github.com/chrislusf/seaweedfs) - Highly scalable distributed key~file system with O(1) disk read.
* [InfluxDB](https://influxdata.com) - Scalable datastore for metrics, events, and real-time analytics.
* [Freehold](http://tshannon.bitbucket.org/freehold/) - An open, secure, and lightweight platform for your files and data.
* [Prometheus Annotation Server](https://github.com/oliver006/prom_annotation_server) - Annotation server for PromDash & Prometheus service monitoring system.
* [Consul](https://github.com/hashicorp/consul) - Consul is service discovery and configuration made easy. Distributed, highly available, and datacenter-aware.
* [Kala](https://github.com/ajvb/kala) - Kala is a modern job scheduler optimized to run on a single node. It is persistent, JSON over HTTP API, ISO 8601 duration notation, and dependent jobs.
* [drive](https://github.com/odeke-em/drive) - drive is an unofficial Google Drive command line client for \*NIX operating systems.
* [stow](https://github.com/djherbis/stow) - a persistence manager for objects
backed by boltdb.
* [buckets](https://github.com/joyrexus/buckets) - a bolt wrapper streamlining
simple tx and key scans.
* [mbuckets](https://github.com/abhigupta912/mbuckets) - A Bolt wrapper that allows easy operations on multi level (nested) buckets.
* [Request Baskets](https://github.com/darklynx/request-baskets) - A web service to collect arbitrary HTTP requests and inspect them via REST API or simple web UI, similar to [RequestBin](http://requestb.in/) service
* [Go Report Card](https://goreportcard.com/) - Go code quality report cards as a (free and open source) service.
* [Boltdb Boilerplate](https://github.com/bobintornado/boltdb-boilerplate) - Boilerplate wrapper around bolt aiming to make simple calls one-liners.
* [lru](https://github.com/crowdriff/lru) - Easy to use Bolt-backed Least-Recently-Used (LRU) read-through cache with chainable remote stores.
* [Storm](https://github.com/asdine/storm) - A simple ORM around BoltDB.
* [GoWebApp](https://github.com/josephspurrier/gowebapp) - A basic MVC web application in Go using BoltDB.
* [SimpleBolt](https://github.com/xyproto/simplebolt) - A simple way to use BoltDB. Deals mainly with strings.
* [Algernon](https://github.com/xyproto/algernon) - A HTTP/2 web server with built-in support for Lua. Uses BoltDB as the default database backend.
If you are using Bolt in a project please send a pull request to add it to the list.
---
### Overview (`vendor/src/github.com/deckarep/golang-set/README.md`)
[](https://travis-ci.org/deckarep/golang-set)
[](http://godoc.org/github.com/deckarep/golang-set)
## golang-set
The missing set collection for the Go language. Until Go has sets built-in...use this.
Coming from Python one of the things I miss is the superbly wonderful set collection. This is my attempt to mimic the primary features of the set from Python.
You can of course argue that there is no need for a set in Go, otherwise the creators would have added one to the standard library. To those I say simply ignore this repository
and carry-on and to the rest that find this useful please contribute in helping me make it better by:
* Helping to make more idiomatic improvements to the code.
* Helping to increase the performance of it. ~~(So far, no attempt has been made, but since it uses a map internally, I expect it to be mostly performant.)~~
* Helping to make the unit-tests more robust and kick-ass.
* Helping to fill in the [documentation.](http://godoc.org/github.com/deckarep/golang-set)
* Simply offering feedback and suggestions. (Positive, constructive feedback is appreciated.)
I have to give some credit for helping seed the idea with this post on [stackoverflow.](http://programmers.stackexchange.com/questions/177428/sets-data-structure-in-golang)
*Update* - as of 3/9/2014, you can use a compile-time generic version of this package in the [gen](http://clipperhouse.github.io/gen/) framework. This framework allows you to use the golang-set in a completely generic and type-safe way by allowing you to generate a supporting .go file based on your custom types.
## Features (as of 9/22/2014)
* a CartesionProduct() method has been added with unit-tests: [Read more about the cartesion product](http://en.wikipedia.org/wiki/Cartesian_product)
## Features (as of 9/15/2014)
* a PowerSet() method has been added with unit-tests: [Read more about the Power set](http://en.wikipedia.org/wiki/Power_set)
## Features (as of 4/22/2014)
* One common interface to both implementations
* Two set implementations to choose from
* a thread-safe implementation designed for concurrent use
* a non-thread-safe implementation designed for performance
* 75 benchmarks for both implementations
* 35 unit tests for both implementations
* 14 concurrent tests for the thread-safe implementation
Please see the unit test file for additional usage examples. The Python set documentation will also do a better job than I can of explaining how a set typically [works.](http://docs.python.org/2/library/sets.html) Please keep in mind
however that the Python set is a built-in type and supports additional features and syntax that make it awesome.
## Examples but not exhaustive:
```
/* Detailed source-code truncated for AI context efficiency. */
```
Thanks!
-Ralph
[](https://bitdeli.com/free "Bitdeli Badge")
[](https://github.com/igrigorik/ga-beacon)
---
### Overview (`vendor/src/github.com/docker/distribution/README.md`)
# Distribution
The Docker toolset to pack, ship, store, and deliver content.
This repository's main product is the Docker Registry 2.0 implementation
for storing and distributing Docker images. It supersedes the
[docker/docker-registry](https://github.com/docker/docker-registry)
project with a new API design, focused around security and performance.
[](https://circleci.com/gh/docker/distribution/tree/master)
[](https://godoc.org/github.com/docker/distribution)
This repository contains the following components:
|**Component** |Description |
|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **registry** | An implementation of the [Docker Registry HTTP API V2](docs/spec/api.md) for use with docker 1.6+. |
| **libraries** | A rich set of libraries for interacting with distribution components. Please see [godoc](https://godoc.org/github.com/docker/distribution) for details. **Note**: These libraries are **unstable**. |
| **specifications** | _Distribution_ related specifications are available in [docs/spec](docs/spec) |
| **documentation** | Docker's full documentation set is available at [docs.docker.com](https://docs.docker.com). This repository [contains the subset](docs/index.md) related just to the registry. |
### How does this integrate with Docker engine?
This project should provide an implementation to a V2 API for use in the [Docker
core project](https://github.com/docker/docker). The API should be embeddable
and simplify the process of securely pulling and pushing content from `docker`
daemons.
### What are the long term goals of the Distribution project?
The _Distribution_ project has the further long term goal of providing a
secure tool chain for distributing content. The specifications, APIs and tools
should be as useful with Docker as they are without.
Our goal is to design a professional grade and extensible content distribution
system that allow users to:
* Enjoy an efficient, secured and reliable way to store, manage, package and
exchange content
* Hack/roll their own on top of healthy open-source components
* Implement their own home made solution through good specs, and solid
extensions mechanism.
## More about Registry 2.0
The new registry implementation provides the following benefits:
- faster push and pull
- new, more efficient implementation
- simplified deployment
- pluggable storage backend
- webhook notifications
For information on upcoming functionality, please see [ROADMAP.md](ROADMAP.md).
### Who needs to deploy a registry?
By default, Docker users pull images from Docker's public registry instance.
[Installing Docker](https://docs.docker.com/engine/installation/) gives users this
ability. Users can also push images to a repository on Docker's public registry,
if they have a [Docker Hub](https://hub.docker.com/) account.
For some users and even companies, this default behavior is sufficient. For
others, it is not.
For example, users with their own software products may want to maintain a
registry for private, company images. Also, you may wish to deploy your own
image repository for images used to test or in continuous integration. For these
use cases and others, [deploying your own registry instance](docs/deploying.md)
may be the better choice.
### Migration to Registry 2.0
For those who have previously deployed their own registry based on the Registry
1.0 implementation and wish to deploy a Registry 2.0 while retaining images,
data migration is required. A tool to assist with migration efforts has been
created. For more information see [docker/migrator]
(https://github.com/docker/migrator).
## Contribute
Please see [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to contribute
issues, fixes, and patches to this project. If you are contributing code, see
the instructions for [building a development environment](docs/building.md).
## Support
If any issues are encountered while using the _Distribution_ project, several
avenues are available for support:
| IRC | #docker-distribution on FreeNode |
|---|---|
| Issue Tracker | github.com/docker/distribution/issues |
| Google Groups | https://groups.google.com/a/dockerproject.org/forum/#!forum/distribution |
| Mailing List | docker@dockerproject.org |
--- ### Tutorials Index (`docs/tutorials/index.md`) # Learn by example * [Hello world in a container](dockerizing.md) * [Run a simple application](usingdocker.md) * [Build your own images](dockerimages.md) * [Network containers](networkingcontainers.md) * [Manage data in containers](dockervolumes.md) * [Store images on Docker Hub](dockerrepos.md) --- ### Userguide Index (`docs/userguide/index.md`) # Docker Engine user guide This guide helps users learn how to use Docker Engine. - [Introduction to Engine user guide](intro.md) ## Learn by example - [Hello world in a container](../tutorials/dockerizing.md) - [Build your own images](../tutorials/dockerimages.md) - [Network containers](../tutorials/networkingcontainers.md) - [Run a simple application](../tutorials/usingdocker.md) - [Manage data in containers](../tutorials/dockervolumes.md) - [Store images on Docker Hub](../tutorials/dockerrepos.md) ## Work with images - [Best practices for writing Dockerfiles](eng-image/dockerfile_best-practices.md) - [Create a base image](eng-image/baseimages.md) - [Image management](eng-image/image_management.md) ## Manage storage drivers - [Understand images, containers, and storage drivers](storagedriver/imagesandcontainers.md) - [Select a storage driver](storagedriver/selectadriver.md) - [AUFS storage in practice](storagedriver/aufs-driver.md) - [Btrfs storage in practice](storagedriver/btrfs-driver.md) - [Device Mapper storage in practice](storagedriver/device-mapper-driver.md) - [OverlayFS storage in practice](storagedriver/overlayfs-driver.md) - [ZFS storage in practice](storagedriver/zfs-driver.md) ## Configure networks - [Understand Docker container networks](networking/dockernetworks.md) - [Embedded DNS server in user-defined networks](networking/configure-dns.md) - [Get started with multi-host networking](networking/get-started-overlay.md) - [Work with network commands](networking/work-with-networks.md) ### Work with the default network - [Understand container communication](networking/default_network/container-communication.md) - [Legacy container links](networking/default_network/dockerlinks.md) - [Binding container ports to the host](networking/default_network/binding.md) - [Build your own bridge](networking/default_network/build-bridges.md) - [Configure container DNS](networking/default_network/configure-dns.md) - [Customize the docker0 bridge](networking/default_network/custom-docker0.md) - [IPv6 with Docker](networking/default_network/ipv6.md) ## Misc - [Apply custom metadata](labels-custom-metadata.md) --- ### Userguide Intro (`docs/userguide/intro.md`) # Engine user guide This guide takes you through the fundamentals of using Docker Engine and integrating it into your environment. You'll learn how to use Engine to: * Dockerize your applications. * Run your own containers. * Build Docker images. * Share your Docker images with others. * And a whole lot more! This guide is broken into major sections that take you through learning the basics of Docker Engine and the other Docker products that support it. ## Dockerizing applications: A "Hello world" *How do I run applications inside containers?* Docker Engine offers a containerization platform to power your applications. To learn how to Dockerize applications and run them: Go to [Dockerizing Applications](../tutorials/dockerizing.md). ## Working with containers *How do I manage my containers?* Once you get a grip on running your applications in Docker containers, you'll learn how to manage those containers. To find out about how to inspect, monitor and manage containers: Go to [Working with Containers](../tutorials/usingdocker.md). ## Working with Docker images *How can I access, share and build my own images?* Once you've learnt how to use Docker it's time to take the next step and learn how to build your own application images with Docker. Go to [Working with Docker Images](../tutorials/dockerimages.md). ## Networking containers Until now we've seen how to build individual applications inside Docker containers. Now learn how to build whole application stacks with Docker networking. Go to [Networking Containers](../tutorials/networkingcontainers.md). ## Managing data in containers Now we know how to link Docker containers together the next step is learning how to manage data, volumes and mounts inside our containers. Go to [Managing Data in Containers](../tutorials/dockervolumes.md). ## Docker products that complement Engine Often, one powerful technology spawns many other inventions that make that easier to get to, easier to use, and more powerful. These spawned things share one common characteristic: they augment the central technology. The following Docker products expand on the core Docker Engine functions. ### Docker Hub Docker Hub is the central hub for Docker. It hosts public Docker images and provides services to help you build and manage your Docker environment. To learn more: Go to [Using Docker Hub](https://docs.docker.com/docker-hub). ### Docker Machine Docker Machine helps you get Docker Engines up and running quickly. Machine can set up hosts for Docker Engines on your computer, on cloud providers, and/or in your data center, and then configure your Docker client to securely talk to them. Go to [Docker Machine user guide](https://docs.docker.com/machine/). ### Docker Compose Docker Compose allows you to define an application's components -- their containers, configuration, links and volumes -- in a single file. Then a single command will set everything up and start your application running. Go to [Docker Compose user guide](https://docs.docker.com/compose/). ### Docker Swarm Docker Swarm pools several Docker Engines together and exposes them as a single virtual Docker Engine. It serves the standard Docker API, so any tool that already works with Docker can now transparently scale up to multiple hosts. Go to [Docker Swarm user guide](https://docs.docker.com/swarm/). ## Getting help * [Docker homepage](https://www.docker.com/) * [Docker Hub](https://hub.docker.com) * [Docker blog](https://blog.docker.com/) * [Docker documentation](https://docs.docker.com/) * [Docker Getting Started Guide](https://docs.docker.com/mac/started/) * [Docker code on GitHub](https://github.com/docker/docker) * [Docker mailing list](https://groups.google.com/forum/#!forum/docker-user) * Docker on IRC: irc.freenode.net and channel #docker * [Docker on Twitter](https://twitter.com/docker) * Get [Docker help](https://stackoverflow.com/search?q=docker) on StackOverflow --- ### Admin Logging Index (`docs/admin/logging/index.md`) # Logging Drivers * [Configuring logging drivers](overview.md) * [Configuring log tags](log_tags.md) * [Fluentd logging driver](fluentd.md) * [Journald logging driver](journald.md) * [Amazon CloudWatch Logs logging driver](awslogs.md) * [Splunk logging driver](splunk.md) * [ETW logging driver](etwlogs.md) --- ### Admin Logging Overview (`docs/admin/logging/overview.md`) # Configure logging drivers The container can have a different logging driver than the Docker daemon. Use the `--log-driver=VALUE` with the `docker run` command to configure the container's logging driver. If the `--log-driver` option is not set, docker uses the default (`json-file`) logging driver. The following options are supported: | Driver | Description | |-------------|-------------------------------------------------------------------------------------------------------------------------------| | `none` | Disables any logging for the container. `docker logs` won't be available with this driver. | | `json-file` | Default logging driver for Docker. Writes JSON messages to file. | | `syslog` | Syslog logging driver for Docker. Writes log messages to syslog. | | `journald` | Journald logging driver for Docker. Writes log messages to `journald`. | | `gelf` | Graylog Extended Log Format (GELF) logging driver for Docker. Writes log messages to a GELF endpoint likeGraylog or Logstash. | | `fluentd` | Fluentd logging driver for Docker. Writes log messages to `fluentd` (forward input). | | `awslogs` | Amazon CloudWatch Logs logging driver for Docker. Writes log messages to Amazon CloudWatch Logs. | | `splunk` | Splunk logging driver for Docker. Writes log messages to `splunk` using HTTP Event Collector. | | `etwlogs` | ETW logging driver for Docker on Windows. Writes log messages as ETW events. | | `gcplogs` | Google Cloud Logging driver for Docker. Writes log messages to Google Cloud Logging. | The `docker logs`command is available only for the `json-file` and `journald` logging drivers. The `labels` and `env` options add additional attributes for use with logging drivers that accept them. Each option takes a comma-separated list of keys. If there is collision between `label` and `env` keys, the value of the `env` takes precedence. To use attributes, specify them when you start the Docker daemon. For example, to manually start the daemon with the `json-file` driver, and include additional attributes in the output, run the following command: ```bash $ docker daemon \ --log-driver=json-file \ --log-opt labels=foo \ --log-opt env=foo,fizz ``` Then, run a container and specify values for the `labels` or `env`. For example, you might use this: ```bash $ docker run -dit --label foo=bar -e fizz=buzz alpine sh ``` This adds additional fields to the log depending on the driver, e.g. for `json-file` that looks like: ```json "attrs":{"fizz":"buzz","foo":"bar"} ``` ## json-file options The following logging options are supported for the `json-file` logging driver: ```bash --log-opt max-size=[0-9+][k|m|g] --log-opt max-file=[0-9+] --log-opt labels=label1,label2 --log-opt env=env1,env2 ``` Logs that reach `max-size` are rolled over. You can set the size in kilobytes(k), megabytes(m), or gigabytes(g). eg `--log-opt max-size=50m`. If `max-size` is not set, then logs are not rolled over. `max-file` specifies the maximum number of files that a log is rolled over before being discarded. eg `--log-opt max-file=100`. If `max-size` is not set, then `max-file` is not honored. If `max-size` and `max-file` are set, `docker logs` only returns the log lines from the newest log file. ## syslog options The following logging options are supported for the `syslog` logging driver: ```bash --log-opt syslog-address=[tcp|udp|tcp+tls]://host:port --log-opt syslog-address=unix://path --log-opt syslog-address=unixgram://path --log-opt syslog-facility=daemon --log-opt syslog-tls-ca-cert=/etc/ca-certificates/custom/ca.pem --log-opt syslog-tls-cert=/etc/ca-certificates/custom/cert.pem --log-opt syslog-tls-key=/etc/ca-certificates/custom/key.pem --log-opt syslog-tls-skip-verify=true --log-opt tag="mailer" --log-opt syslog-format=[rfc5424|rfc5424micro|rfc3164] --log-opt env=ENV1,ENV2,ENV3 --log-opt labels=label1,label2,label3 ``` `syslog-address` specifies the remote syslog server address where the driver connects to. If not specified it defaults to the local unix socket of the running system. If transport is either `tcp` or `udp` and `port` is not specified it defaults to `514` The following example shows how to have the `syslog` driver connect to a `syslog` remote server at `192.168.0.42` on port `123` ```bash $ docker run --log-driver=syslog --log-opt syslog-address=tcp://192.168.0.42:123 ``` The `syslog-facility` option configures the syslog facility. By default, the system uses the `daemon` value. To override this behavior, you can provide an integer of 0 to 23 or any of the following named facilities: * `kern` * `user` * `mail` * `daemon` * `auth` * `syslog` * `lpr` * `news` * `uucp` * `cron` * `authpriv` * `ftp` * `local0` * `local1` * `local2` * `local3` * `local4` * `local5` * `local6` * `local7` `syslog-tls-ca-cert` specifies the absolute path to the trust certificates signed by the CA. This option is ignored if the address protocol is not `tcp+tls`. `syslog-tls-cert` specifies the absolute path to the TLS certificate file. This option is ignored if the address protocol is not `tcp+tls`. `syslog-tls-key` specifies the absolute path to the TLS key file. This option is ignored if the address protocol is not `tcp+tls`. `syslog-tls-skip-verify` configures the TLS verification. This verification is enabled by default, but it can be overriden by setting this option to `true`. This option is ignored if the address protocol is not `tcp+tls`. `tag` configures a string that is appended to the APP-NAME in the syslog message. By default, Docker uses the first 12 characters of the container ID to tag log messages. Refer to the [log tag option documentation](log_tags.md) for customizing the log tag format. `syslog-format` specifies syslog message format to use when logging. If not specified it defaults to the local unix syslog format without hostname specification. Specify rfc3164 to perform logging in RFC-3164 compatible format. Specify rfc5424 to perform logging in RFC-5424 compatible format. Specify rfc5424micro to perform logging in RFC-5424 compatible format with microsecond timestamp resolution. `env` is a comma-separated list of keys of environment variables. Used for advanced [log tag options](log_tags.md). `labels` is a comma-separated list of keys of labels. Used for advanced [log tag options](log_tags.md). ## journald options The `journald` logging driver stores the container id in the journal's `CONTAINER_ID` field. For detailed information on working with this logging driver, see [the journald logging driver](journald.md) reference documentation. ## GELF options The GELF logging driver supports the following options: ```bash --log-opt gelf-address=udp://host:port --log-opt tag="database" --log-opt labels=label1,label2 --log-opt env=env1,env2 --log-opt gelf-compression-type=gzip --log-opt gelf-compression-level=1 ``` The `gelf-address` option specifies the remote GELF server address that the driver connects to. Currently, only `udp` is supported as the transport and you must specify a `port` value. The following example shows how to connect the `gelf` driver to a GELF remote server at `192.168.0.42` on port `12201` ```bash $ docker run -dit \ --log-driver=gelf \ --log-opt gelf-address=udp://192.168.0.42:12201 \ alpine sh ``` By default, Docker uses the first 12 characters of the container ID to tag log messages. Refer to the [log tag option documentation](log_tags.md) for customizing the log tag format. The `labels` and `env` options are supported by the gelf logging driver. It adds additional key on the `extra` fields, prefixed by an underscore (`_`). // […] "_foo": "bar", "_fizz": "buzz", // […] The `gelf-compression-type` option can be used to change how the GELF driver compresses each log message. The accepted values are `gzip`, `zlib` and `none`. `gzip` is chosen by default. The `gelf-compression-level` option can be used to change the level of compresssion when `gzip` or `zlib` is selected as `gelf-compression-type`. Accepted value must be from from -1 to 9 (BestCompression). Higher levels typically run slower but compress more. Default value is 1 (BestSpeed). ## Fluentd options You can use the `--log-opt NAME=VALUE` flag to specify these additional Fluentd logging driver options. - `fluentd-address`: specify `host:port` to connect [localhost:24224] - `tag`: specify tag for `fluentd` message - `fluentd-buffer-limit`: specify the maximum size of the fluentd log buffer [8MB] - `fluentd-retry-wait`: initial delay before a connection retry (after which it increases exponentially) [1000ms] - `fluentd-max-retries`: maximum number of connection retries before abrupt failure of docker [1073741824] - `fluentd-async-connect`: whether to block on initial connection or not [false] For example, to specify both additional options: ```bash $ docker run -dit \ --log-driver=fluentd \ --log-opt fluentd-address=localhost:24224 \ --log-opt tag="docker.{{.Name}}" \ alpine sh ``` If container cannot connect to the Fluentd daemon on the specified address and `fluentd-async-connect` is not enabled, the container stops immediately. For detailed information on working with this logging driver, see [the fluentd logging driver](fluentd.md) ## Amazon CloudWatch Logs options The Amazon CloudWatch Logs logging driver supports the following options: ```bash --log-opt awslogs-region=
--- ### Userguide Eng Image Index (`docs/userguide/eng-image/index.md`) # Work with images * [Create a base image](baseimages.md) * [Best practices for writing Dockerfiles](dockerfile_best-practices.md) * [Image management](image_management.md) --- ### Userguide Networking Index (`docs/userguide/networking/index.md`) # Docker networks feature overview This sections explains how to use the Docker networks feature. This feature allows users to define their own networks and connect containers to them. Using this feature you can create a network on a single host or a network that spans across multiple hosts. - [Understand Docker container networks](dockernetworks.md) - [Work with network commands](work-with-networks.md) - [Get started with multi-host networking](get-started-overlay.md) If you are already familiar with Docker's default bridge network, `docker0` that network continues to be supported. It is created automatically in every installation. The default bridge network is also named `bridge`. To see a list of topics related to that network, read the articles listed in the [Docker default bridge network](default_network/index.md). --- ### Userguide Storagedriver Index (`docs/userguide/storagedriver/index.md`) # Docker storage drivers Docker relies on driver technology to manage the storage and interactions associated with images and the containers that run them. This section contains the following pages: * [Understand images, containers, and storage drivers](imagesandcontainers.md) * [Select a storage driver](selectadriver.md) * [AUFS storage driver in practice](aufs-driver.md) * [Btrfs storage driver in practice](btrfs-driver.md) * [Device Mapper storage driver in practice](device-mapper-driver.md) * [OverlayFS in practice](overlayfs-driver.md) * [ZFS storage in practice](zfs-driver.md) If you are new to Docker containers make sure you read ["Understand images, containers, and storage drivers"](imagesandcontainers.md) first. It explains key concepts and technologies that can help you when working with storage drivers. ### Acknowledgment The Docker storage driver material was created in large part by our guest author Nigel Poulton with a bit of help from Docker's own Jérôme Petazzoni. In his spare time Nigel creates [IT training videos](http://www.pluralsight.com/author/nigel-poulton), co-hosts the weekly [In Tech We Trust podcast](http://intechwetrustpodcast.com/), and lives it large on [Twitter](https://twitter.com/nigelpoulton). --- ### Userguide Networking Default Network Index (`docs/userguide/networking/default_network/index.md`) # Docker default bridge network With the introduction of the Docker networks feature, you can create your own user-defined networks. The Docker default bridge is created when you install Docker Engine. It is a `bridge` network and is also named `bridge`. The topics in this section are related to interacting with that default bridge network. - [Understand container communication](container-communication.md) - [Legacy container links](dockerlinks.md) - [Binding container ports to the host](binding.md) - [Build your own bridge](build-bridges.md) - [Configure container DNS](configure-dns.md) - [Customize the docker0 bridge](custom-docker0.md) - [IPv6 with Docker](ipv6.md) --- ### Breaking Changes (`docs/breaking_changes.md`) # Breaking changes and incompatibilities Every Engine release strives to be backward compatible with its predecessors. In all cases, the policy is that feature removal is communicated two releases in advance and documented as part of the [deprecated features](deprecated.md) page. Unfortunately, Docker is a fast moving project, and newly introduced features may sometime introduce breaking changes and/or incompatibilities. This page documents these by Engine version. # Engine 1.12 Docker clients <= 1.9.2 used an invalid Host header when making request to the daemon. Docker 1.12 is built using golang 1.6 which is now checking the validity of the Host header and as such clients <= 1.9.2 can't talk anymore to the daemon. [An environment variable was added to overcome this issue.](reference/commandline/dockerd.md#miscellaneous-options) # Engine 1.10 There were two breaking changes in the 1.10 release. ## Registry Registry 2.3 includes improvements to the image manifest that have caused a breaking change. Images pushed by Engine 1.10 to a Registry 2.3 cannot be pulled by digest by older Engine versions. A `docker pull` that encounters this situation returns the following error: ``` Error response from daemon: unsupported schema version 2 for tag TAGNAME ``` Docker Content Trust heavily relies on pull by digest. As a result, images pushed from the Engine 1.10 CLI to a 2.3 Registry cannot be pulled by older Engine CLIs (< 1.10) with Docker Content Trust enabled. If you are using an older Registry version (< 2.3), this problem does not occur with any version of the Engine CLI; push, pull, with and without content trust work as you would expect. ## Docker Content Trust Engine older than the current 1.10 cannot pull images from repositories that have enabled key delegation. Key delegation is a feature which requires a manual action to enable. --- ### Deprecated (`docs/deprecated.md`) # Deprecated Engine Features The following list of features are deprecated in Engine. ### Three argument form in `docker import` **Deprecated In Release: [v0.6.7](https://github.com/docker/docker/releases/tag/v0.6.7)** **Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** The `docker import` command format 'file|URL|- [REPOSITORY [TAG]]' is deprecated since November 2013. It's no more supported. ### `-e` and `--email` flags on `docker login` **Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** **Target For Removal In Release: v1.13** The docker login command is removing the ability to automatically register for an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated. ### Separator (`:`) of `--security-opt` flag on `docker run` **Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** **Target For Removal In Release: v1.13** The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide keys and values, it uses the equal symbol(`=`) for consinstency with other similar flags, like `--storage-opt`. ### `/containers/(id or name)/copy` endpoint **Deprecated In Release: v1.8** **Removed In Release: v1.12.0** The endpoint `/containers/(id or name)/copy` is deprecated in favor of `/containers/(id or name)/archive`. ### Ambiguous event fields in API **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** The fields `ID`, `Status` and `From` in the events API have been deprecated in favor of a more rich structure. See the events API documentation for the new format. ### `-f` flag on `docker tag` **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** **Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** To make tagging consistent across the various `docker` commands, the `-f` flag on the `docker tag` command is deprecated. It is not longer necessary to specify `-f` to move a tag from one image to another. Nor will `docker` generate an error if the `-f` flag is missing and the specified tag is already in use. ### HostConfig at API container start **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** **Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** Passing an `HostConfig` to `POST /containers/{name}/start` is deprecated in favor of defining it at container creation (`POST /containers/create`). ### Docker ps 'before' and 'since' options **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** **Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** The `docker ps --before` and `docker ps --since` options are deprecated. Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. ### Docker search 'automated' and 'stars' options **Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** **Target For Removal In Release: v1.14** The `docker search --automated` and `docker search --stars` options are deprecated. Use `docker search --filter=is-automated=...` and `docker search --filter=stars=...` instead. ### Driver Specific Log Tags **Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** **Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** Log tags are now generated in a standard way across different logging drivers. Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` and `fluentd-tag` have been deprecated in favor of the generic `tag` option. docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" ### LXC built-in exec driver **Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** **Removed In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** The built-in LXC execution driver, the lxc-conf flag, and API fields have been removed. ### Old Command Line Options **Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** **Removed In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** The flags `-d` and `--daemon` are deprecated in favor of the `daemon` subcommand: docker daemon -H ... The following single-dash (`-opt`) variant of certain command line options are deprecated and replaced with double-dash options (`--opt`): docker attach -nostdin docker attach -sig-proxy docker build -no-cache docker build -rm docker commit -author docker commit -run docker events -since docker history -notrunc docker images -notrunc docker inspect -format docker ps -beforeId docker ps -notrunc docker ps -sinceId docker rm -link docker run -cidfile docker run -dns docker run -entrypoint docker run -expose docker run -link docker run -lxc-conf docker run -n docker run -privileged docker run -volumes-from docker search -notrunc docker search -stars docker search -t docker search -trusted docker tag -force The following double-dash options are deprecated and have no replacement: docker run --cpuset docker run --networking docker ps --since-id docker ps --before-id docker search --trusted ### Interacting with V1 registries Version 1.9 adds a flag (`--disable-legacy-registry=false`) which prevents the docker daemon from `pull`, `push`, and `login` operations against v1 registries. Though disabled by default, this signals the intent to deprecate the v1 protocol. ### Docker Content Trust ENV passphrase variables name change **Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** **Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the Tagging key has been renamed to Repository key. Due to this renaming, we're also changing the corresponding environment variables - DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE is now named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE - DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE is now named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE --- ### Faq (`docs/faq.md`) # Frequently Asked Questions (FAQ) If you don't see your question here, feel free to submit new ones to
 |
 |
7. Verify the boot order
In the **System** settings for the Boot2Docker VM, make sure that **CD/DVD** is
at the top of the **Boot Order** list.
8. Boot to the disk partitioning ISO
Manually start the Boot2Docker VM in VirtualBox, and the disk partitioning ISO
should start up. Using GParted, choose the **GParted Live (default settings)**
option. Choose the default keyboard, language, and XWindows settings, and the
GParted tool will start up and display the VDI volume you created. Right click
on the VDI and choose **Resize/Move**.
9. Drag the slider representing the volume to the maximum available size.
10. Click **Resize/Move** followed by **Apply**.
11. Quit GParted and shut down the VM.
12. Remove the GParted ISO from the IDE controller for the Boot2Docker VM in
VirtualBox.
13. Start the Boot2Docker VM
Fire up the Boot2Docker VM manually in VirtualBox. The VM should log in
automatically, but if it doesn't, the credentials are `docker/tcuser`. Using
the `df -h` command, verify that your changes took effect.
You're done!
---
### Admin Chef (`docs/admin/chef.md`)
# Using Chef
> **Note**:
> Please note this is a community contributed installation path.
## Requirements
To use this guide you'll need a working installation of
[Chef](https://www.chef.io/). This cookbook supports a variety of
operating systems.
## Installation
The cookbook is available on the [Chef Supermarket](https://supermarket.chef.io/cookbooks/docker) and can be
installed using your favorite cookbook dependency manager.
The source can be found on
[GitHub](https://github.com/someara/chef-docker).
Usage
-----
- Add ```depends 'docker', '~> 2.0'``` to your cookbook's metadata.rb
- Use resources shipped in cookbook in a recipe, the same way you'd
use core Chef resources (file, template, directory, package, etc).
```ruby
docker_service 'default' do
action [:create, :start]
end
docker_image 'busybox' do
action :pull
end
docker_container 'an echo server' do
repo 'busybox'
port '1234:1234'
command "nc -ll -p 1234 -e /bin/cat"
end
```
## Getting Started
Here's a quick example of pulling the latest image and running a
container with exposed ports.
```ruby
# Pull latest image
docker_image 'nginx' do
tag 'latest'
action :pull
end
# Run container exposing ports
docker_container 'my_nginx' do
repo 'nginx'
tag 'latest'
port '80:80'
binds [ '/some/local/files/:/etc/nginx/conf.d' ]
host_name 'www'
domain_name 'computers.biz'
env 'FOO=bar'
subscribes :redeploy, 'docker_image[nginx]'
end
```
---
### Admin Configuring (`docs/admin/configuring.md`)
# Configuring and running Docker on various distributions
After successfully installing Docker, the `docker` daemon runs with its default
configuration.
In a production environment, system administrators typically configure the
`docker` daemon to start and stop according to an organization's requirements. In most
cases, the system administrator configures a process manager such as `SysVinit`, `Upstart`,
or `systemd` to manage the `docker` daemon's start and stop.
### Running the docker daemon directly
The `docker` daemon can be run directly using the `dockerd` command. By default it listens on
the Unix socket `unix:///var/run/docker.sock`
$ dockerd
INFO[0000] +job init_networkdriver()
INFO[0000] +job serveapi(unix:///var/run/docker.sock)
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
...
...
### Configuring the docker daemon directly
If you're running the `docker` daemon directly by running `docker daemon` instead
of using a process manager, you can append the configuration options to the `docker` run
command directly. Other options can be passed to the `docker` daemon to configure it.
Some of the daemon's options are:
| Flag | Description |
|-----------------------|-----------------------------------------------------------|
| `-D`, `--debug=false` | Enable or disable debug mode. By default, this is false. |
| `-H`,`--host=[]` | Daemon socket(s) to connect to. |
| `--tls=false` | Enable or disable TLS. By default, this is false. |
Here is an example of running the `docker` daemon with configuration options:
$ dockerd -D --tls=true --tlscert=/var/docker/server.pem --tlskey=/var/docker/serverkey.pem -H tcp://192.168.59.3:2376
These options :
- Enable `-D` (debug) mode
- Set `tls` to true with the server certificate and key specified using `--tlscert` and `--tlskey` respectively
- Listen for connections on `tcp://192.168.59.3:2376`
The command line reference has the [complete list of daemon flags](../reference/commandline/dockerd.md)
with explanations.
### Daemon debugging
As noted above, setting the log level of the daemon to "debug" or enabling debug mode
with `-D` allows the administrator or operator to gain much more knowledge about the
runtime activity of the daemon. If faced with a non-responsive daemon, the administrator
can force a full stack trace of all threads to be added to the daemon log by sending the
`SIGUSR1` signal to the Docker daemon. A common way to send this signal is using the `kill`
command on Linux systems. For example, `kill -USR1 | If you are looking for | Where to find it |
|---|---|
| More about Docker for Mac, features, examples, FAQs, relationship to Docker Machine and Docker Toolbox, and how this fits in the Docker ecosystem | [Getting Started with Docker for Mac](/docker-for-mac/index.md) |
| More about Docker for Windows, features, examples, FAQs, relationship to Docker Machine and Docker Toolbox, and how this fits in the Docker ecosystem | [Getting Started with Docker for Windows](/docker-for-windows/index.md) |
| More about Docker Toolbox | [Docker Toolbox Overview](/toolbox/overview.md) |
| More about Docker for Linux distributions | [Install Docker Engine on Linux](/engine/installation/linux/index.md) |
| More advanced tutorials on running containers, building your own images, networking containers, managing data for containers, and storing images on Docker Hub | [Learn by example](/engine/tutorials/index.md) |
| Information about the Docker product line | The product explainer is a good place to start. |
| How to set up an automated build on Docker Hub | Docker Hub documentation |
| How to run a multi-container application with Compose | [Docker Compose documentation](/compose/overview.md) |