telegraf Documentation

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.

17,684 stars Go Go 1 file ¡ ~32,121 tokens #golang#hacktoberfest#influxdb#json#kafka#logs#metrics#modbus
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy telegraf on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name telegraf -p 3306:3306 --restart always docker.io/zookeeper:3.5.6
Configured Services & Ports
⚙ī¸ clickhouse
image: docker.io/yandex/clickhouse-server:${CLICKHOUSE_VERSION:-latest}
🔌 Ports: 3306:3306, 8123:8123, 8443:8443, 9000:9000, 9009:9009
⚙ī¸ zookeeper
image: docker.io/zookeeper:3.5.6
🔌 Ports: 2181:2181
docker-compose.yml
yaml (docker-compose.yml) 
version: '3'

services:
  clickhouse:
# choose `:latest` after resolve https://github.com/ClickHouse/ClickHouse/issues/13057
    image: docker.io/yandex/clickhouse-server:${CLICKHOUSE_VERSION:-latest}
    volumes:
      - ./init_schema.sql:/docker-entrypoint-initdb.d/init_schema.sql
      - ./test_dictionary.xml:/etc/clickhouse-server/01-test_dictionary.xml
      - ./zookeeper.xml:/etc/clickhouse-server/config.d/00-zookeeper.xml
      - ./tls_settings.xml:/etc/clickhouse-server/config.d/01-tls_settings.xml
      # please comment text_log.xml when CLICKHOUSE_VERSION = 19.16
      - ./text_log.xml:/etc/clickhouse-server/config.d/02-text_log.xml
      - ./part_log.xml:/etc/clickhouse-server/config.d/03-part_log.xml
      - ./mysql_port.xml:/etc/clickhouse-server/config.d/04-mysql_port.xml
      - ./dhparam.pem:/etc/clickhouse-server/dhparam.pem
      - ../../../../testutil/pki/serverkey.pem:/etc/clickhouse-server/server.key
      - ../../../../testutil/pki/servercert.pem:/etc/clickhouse-server/server.crt
    ports:
      - 3306:3306
      - 8123:8123
      - 8443:8443
      - 9000:9000
      - 9009:9009
  zookeeper:
    image: docker.io/zookeeper:3.5.6
    volumes:
    - /var/lib/zookeeper
    ports:
      - 2181:2181

1. Project Overview & Quickstart (influxdata/telegraf)

File: README.md

Telegraf




Telegraf is an agent for collecting, processing, aggregating, and writing
metrics, logs, and other arbitrary data.

  • Offers a comprehensive suite of over 300 plugins, covering a wide range of
    functionalities including system monitoring, cloud services, and message
    passing
  • Enables the integration of user-defined code to collect, transform, and
    transmit data efficiently
  • Compiles into a standalone static binary without any external dependencies,
    ensuring a streamlined deployment process
  • Utilizes TOML for configuration, providing a user-friendly and unambiguous
    setup experience
  • Developed with contributions from a diverse community of over 1,200
    contributors

Users can choose plugins from a wide range of topics, including but not limited
to:

🔨 Installation

For binary builds, Docker images, RPM & DEB packages, and other builds of
Telegraf, please see the install guide.

See the releases documentation for details on versioning
and when releases are made.

đŸ’ģ Usage

Users define a TOML configuration with the plugins and settings they wish to
use, then pass that configuration to Telegraf. The Telegraf agent then
collects data from inputs at each interval and sends data to outputs at each
flush interval.

For a basic walkthrough see quick start.

📖 Documentation

For a full list of documentation including tutorials, reference and other
material, start with the /docs directory.

Additionally, each plugin has its own README that includes details about how to
configure, use, and sometimes debug or troubleshoot. Look under the
/plugins directory for specific plugins.

Here are some commonly used documents:

❤ī¸ Contribute

We love our community of over 1,200 contributors! Many of the plugins included
in Telegraf were originally contributed by community members. Check out
our contributing guide if you are interested in helping out.
Also, join us on our Community Slack or
Community Forums if you have questions or
comments for our engineering teams.

If you are completely new to Telegraf and InfluxDB, you can also enroll for free
at InfluxDB university to take courses
to learn more.

ℹī¸ Support


Please use the Community Slack or
Community Forums if you have questions or
comments for our engineering teams. GitHub issues are limited to actual issues
and feature requests only.

📜 License

File: agent/README.md

Agent

For a complete list of configuration options and details about the agent, please
see the configuration document's agent section.

File: docs/developers/CODE_STYLE.md

Code Style

Code is required to be formatted using gofmt, this covers most code style
requirements. It is also highly recommended to use goimports to
automatically order imports.

Please try to keep lines length under 80 characters, the exact number of
characters is not strict but it generally helps with readability.

File: docs/developers/DEBUG.md

Debug

The following describes how to use the delve debugger with telegraf
during development. Delve has many, very well documented subcommands and
options.

CLI

To run telegraf manually, users can run:

bash
go run ./cmd/telegraf --config config.toml

To attach delve with a similar config users can run the following. Note the
additional -- to specify flags passed to telegraf. Additional flags need to
go after this double dash:

bash
$ dlv debug ./cmd/telegraf -- --config config.toml
Type 'help' for list of commands.
(dlv)

At this point a user could set breakpoints and continue execution.

Visual Studio Code

Visual Studio Code's go language extension includes the ability to easily
make use of delve for debugging. Check out this full tutorial from
the go extension's wiki.

A basic config is all that is required along with additional arguments to tell
Telegraf where the config is located:

json
{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Launch Package",
            "type": "go",
            "request": "launch",
            "mode": "auto",
            "program": "${fileDirname}",
            "args": ["--config", "/path/to/config"]
        }
    ]
}

GoLand

JetBrains' GoLand also includes full featured debugging options.

The following is an example debug config to run Telegraf with a config:

xml
<component name="ProjectRunConfigurationManager">
  <configuration default="false" name="build &amp; run" type="GoApplicationRunConfiguration" factoryName="Go Application">
    <module name="telegraf" />
    <working_directory value="$PROJECT_DIR$" />
    
    <kind value="DIRECTORY" />
    
    <directory value="$PROJECT_DIR$/cmd/telegraf" />
    <filePath value="$PROJECT_DIR$" />
    <method v="2" />
  </configuration>
</component>

File: docs/developers/DEPRECATION.md

Deprecation

Deprecation is the primary tool for making changes in Telegraf. A deprecation
indicates that the community should move away from using a feature, and
documents that the feature will be removed in the next major update (2.0).

Key to deprecation is that the feature remains in Telegraf and the behavior is
not changed.

We do not have a strict definition of a breaking change. All code changes
change behavior, the decision to deprecate or make the change immediately is
decided based on the impact.

Deprecate plugins

Add an entry to the plugins deprecation list (e.g. in
plugins/inputs/deprecations.go). Include the deprecation version and any
replacement, e.g.

golang
"logparser": {
    Since:  "1.15.0",
    Notice: "use 'inputs.tail' with 'grok' data format instead",
  },

The entry can contain an optional RemovalIn field specifying the planned
version for removal of the plugin.

Also add the deprecation warning to the plugin's README:

markdown
# Logparser Input Plugin

### **Deprecated in 1.10**: Please use the [tail][] plugin along with the
`grok` [data format][].

[tail]: /plugins/inputs/tail/README.md
[data formats]: /docs/DATA_FORMATS_INPUT.md

Telegraf will automatically check if a deprecated plugin is configured and print
a warning

text
2022-01-26T20:08:15Z W! DeprecationWarning: Plugin "inputs.logparser" deprecated since version 1.15.0 and will be removed in 2.0.0: use 'inputs.tail' with 'grok' data format instead

Deprecate options

Mark the option as deprecated in the sample config, include the deprecation
version and any replacement.

toml
## Broker to publish to.
  ##   deprecated in 1.7; use the brokers option
  # url = "amqp://localhost:5672/influxdb"

In the plugins configuration struct, add a deprecated tag to the option:

go
type AMQP struct {
    URL       string `toml:"url" deprecated:"1.7.0;use 'brokers' instead"`
    Precision string `toml:"precision" deprecated:"1.2.0;option is ignored"`
}

The deprecated tag has the format <since version>[;removal version];<notice>
where the removal version is optional. The specified deprecation info will
automatically displayed by Telegraf if the option is used in the config

text
2022-01-26T20:08:15Z W! DeprecationWarning: Option "url" of plugin "outputs.amqp" deprecated since version 1.7.0 and will be removed in 2.0.0: use 'brokers' instead

Option value

In the case a specific option value is being deprecated, the method
models.PrintOptionValueDeprecationNotice needs to be called in the plugin's
Init method.

Deprecate metrics

In the README document the metric as deprecated. If there is a replacement
field, tag, or measurement then mention it.

markdown
- system
  - fields:
    - uptime_format (string, deprecated in 1.10: use `uptime` field)

Add filtering to the sample config, leave it commented out.

toml
[[inputs.system]]
  ## Uncomment to remove deprecated metrics.
  # fieldexclude = ["uptime_format"]

File: docs/developers/LOGGING.md

Logging

Plugin Logging

You can access the Logger for a plugin by defining a field named Log. This
Logger is configured internally with the plugin name and alias so they do not
need to be specified for each log call.

go
type MyPlugin struct {
    Log telegraf.Logger `toml:"-"`
}

You can then use this Logger in the plugin. Use the method corresponding to
the log level of the message.

go
p.Log.Errorf("Unable to write to file: %v", err)

Agent Logging

In other sections of the code it is required to add the log level and module
manually:

go
log.Printf("E! [agent] Error writing to %s: %v", output.LogName(), err)

When to Log

Log a message if an error occurs but the plugin can continue working. For
example if the plugin handles several servers and only one of them has a fatal
error, it can be logged as an error.

Use logging judiciously for debug purposes. Since Telegraf does not currently
support setting the log level on a per module basis, it is especially important
to not over do it with debug logging.

If the plugin is listening on a socket, log a message with the address of the socket:

go
p.log.InfoF("Listening on %s://%s", protocol, l.Addr())

When not to Log

Don't use logging to emit performance data or other meta data about the plugin,
instead use the internal plugin and the selfstats package.

Don't log fatal errors in the plugin that require the plugin to return, instead
return them from the function and Telegraf will handle the logging.

Don't log for static configuration errors, check for them in a plugin Init()
function and return an error there.

Don't log a warning every time a plugin is called for situations that are
normal on some systems.

Log Level

The log level is indicated by a single character at the start of the log
message. Adding this prefix is not required when using the Plugin Logger.

  • D! Debug
  • I! Info
  • W! Warning
  • E! Error

Style

Log messages should be capitalized and be a single line.

If it includes data received from another system or process, such as the text
of an error message, the text should be quoted with %q.

Use the %v format for the Go error type instead of %s to ensure a nil error
is printed.

File: docs/developers/METRIC_FORMAT_CHANGES.md

Metric Format Changes

When making changes to an existing input plugin, care must be taken not to
change the metric format in ways that will cause trouble for existing users.
This document helps developers understand how to make metric format changes
safely.

Changes can cause incompatibilities

If the metric format changes, data collected in the new format can be
incompatible with data in the old format. Database queries designed around the
old format may not work with the new format. This can cause application failures.

Some metric format changes don't cause incompatibilities. Also, some unsafe
changes are necessary. How do you know what changes are safe and what to do if
your change isn't safe?

Guidelines

The main guideline is just to keep compatibility in mind when making changes.
Often developers are focused on making a change that fixes their particular
problem and they forget that many people use the existing code and will upgrade.
When you're coding, keep existing users and applications in mind.

Renaming, removing, reusing

Database queries refer to the metric and its tags and fields by name. Any
Telegraf code change that changes those names has the potential to break an
existing query. Similarly, removing tags or fields can break queries.

Changing the meaning of an existing tag value or field value or reusing an
existing one in a new way isn't safe. Although queries that use these tags/field
may not break, they will not work as they did before the change.

Adding a field doesn't break existing queries. Queries that select all fields
and/or tags (like "select * from") will return an extra series but this is often
useful.

Performance and storage

Time series databases can store large amounts of data but many of them don't
perform well on high cardinality data. If a metric format change includes a new
tag that holds high cardinality data, database performance could be reduced
enough to cause existing applications not to work as they previously did.
Metric format changes that dramatically increase the number of tags or fields of
a metric can increase database storage requirements unexpectedly. Both of these
types of changes are unsafe.

Make unsafe changes opt-in

If your change has the potential to seriously affect existing users, the change
must be opt-in. To do this, add a plugin configuration setting that lets the
user select the metric format. Make the setting's default value select the old
metric format. When new users add the plugin they can choose the new format and
get its benefits. When existing users upgrade, their config files won't have the
new setting so the default will ensure that there is no change.

When adding a setting, avoid using a boolean and consider instead a string or
int for future flexibility. A boolean can only handle two formats but a string
can handle many. For example, compare use_new_format=true and
features=["enable_foo_fields"]; the latter is much easier to extend and still
very descriptive.

If you want to encourage existing users to use the new format you can log a
warning once on startup when the old format is selected. The warning should
tell users in a gentle way that they can upgrade to a better metric format.
If it doesn't make sense to maintain multiple metric formats forever, you can
change the default on a major release or even remove the old format completely.
See [[Deprecation]] for details.

Utility

Changes should be useful to many or most users. A change that is only useful for
a small number of users may not accepted, even if it's off by default.

Summary table

delete rename add
metric unsafe unsafe safe
tag unsafe unsafe be careful with cardinality
field unsafe unsafe ok as long as it's useful for existing users and is worth the added space

References

InfluxDB Documentation: "Schema and data layout"

File: docs/developers/PACKAGING.md

Packaging

Building the packages for Telegraf is automated using Make. Just running
make will build a Telegraf binary for the operating system and architecture
you are using (if it is supported). If you need to build a different package
then you can run make package which will build all the supported packages. You
will most likely only want a subset, you can define a subset of packages to be
built by overriding the include_packages variable like so

shell
make package include_packages="amd64.deb"

You can also build all packages for a specific architecture like so

shell
make package include_packages="$(make amd64)"

The packaging steps require certain tools to be setup before hand to work.
These dependencies are listed in the ci.docker file which you can find in the
scripts directory. Therefore it is recommended to use Docker to build the
artifacts, see more details below.

Go Version

Telegraf will be built using the latest version of Go whenever possible.

Update CI image

Incrementing the version is maintained by the core Telegraf team because it
requires access to an internal docker repository that hosts the docker CI
images. When a new version is released, the following process is followed:

  1. Within the Makefile, .circleci\config.yml, and scripts/ci.docker files
    update the Go versions to the new version number
  2. Run make ci, this requires quay.io internal permissions
  3. The files scripts\installgo_linux.sh, scripts\installgo_mac.sh, and
    scripts\installgo_windows.sh need to be updated as well with the new Go
    version and SHA
  4. Create a pull request with these new changes, and verify the CI passes and
    uses the new docker image

See the previous PRs
as examples.

Access to quay.io

A member of the team needs to invite you to the quay.io organization.
To push new images, the user needs to do the following:

  1. Create a password if the user logged in using Google authentication
  2. Download an encrypted username/password from the quay.io user page
  3. Run docker login quay.io and enter in the encrypted username and password
    from the previous step

Package using Docker

This packaging method uses the CI images, and is very similar to how the
official packages are created on release. This is the recommended method for
building the rpm/deb as it is less system dependent.

Pull the CI images from quay, the version corresponds to the version of Go
that is used to build the binary:

shell
docker pull quay.io/influxdb/telegraf-ci:1.9.7

Start a shell in the container:

shell
docker run -ti quay.io/influxdb/telegraf-ci:1.9.7 /bin/bash

From within the container:

  1. go get -d github.com/influxdata/telegraf
  2. cd /go/src/github.com/influxdata/telegraf
  3. git checkout release-1.10
    • Replace tag release-1.10 with the version of Telegraf you would like to
      build
  4. git reset --hard 1.10.2
  5. make deps
  6. make package include_packages="amd64.deb"
    • Change include_packages to change what package you want, run make help
      to see possible values

From the host system, copy the build artifacts out of the container:

shell
docker cp romantic_ptolemy:/go/src/github.com/influxdata/telegraf/build/telegraf-1.10.2-1.x86_64.rpm .

File: docs/developers/PROFILING.md

Profiling

This article describes how to collect performance traces and memory profiles
from Telegraf. If you are submitting this for an issue, please include the
version.txt generated below.

Use the --pprof-addr option to enable the profiler, the easiest way to do
this may be to add this line to /etc/default/telegraf:

shell
TELEGRAF_OPTS="--pprof-addr localhost:6060"

Restart Telegraf to activate the profile address.

Trace Profile

Collect a trace during the time where the performance issue is occurring. This
example collects a 10 second trace and runs for 10 seconds:

shell
curl 'http://localhost:6060/debug/pprof/trace?seconds=10' > trace.bin
telegraf --version > version.txt
go env GOOS GOARCH >> version.txt

The trace.bin and version.txt files can be sent in for analysis or, if
desired, you can analyze the trace with:

shell
go tool trace trace.bin

Memory Profile

Collect a heap memory profile:

shell
curl 'http://localhost:6060/debug/pprof/heap' > mem.prof
telegraf --version > version.txt
go env GOOS GOARCH >> version.txt

Analyze:

shell
$ go tool pprof mem.prof
(pprof) top5

CPU Profile

Collect a 30s CPU profile:

shell
curl 'http://localhost:6060/debug/pprof/profile' > cpu.prof
telegraf --version > version.txt
go env GOOS GOARCH >> version.txt

Analyze:

shell
go tool pprof cpu.prof
(pprof) top5

File: docs/developers/REVIEWS.md

Reviews

Pull-requests require two approvals before being merged. Expect several rounds
of back and forth on reviews, non-trivial changes are rarely accepted on the
first pass. It might take some time until you see a first review so please be
patient.

All pull requests should follow the style and best practices in the
CONTRIBUTING.md document.

Process

The review process is roughly structured as follows:

  1. Submit a pull request.
    Please check that you signed the CLA
    (and Corporate CLA if you are
    contributing code on as an employee of your company). Provide a short
    description of your submission and reference issues that you potentially
    close. Make sure the CI tests are all green and there are no linter-issues.
  2. Get feedback from a first reviewer and a ready for final review tag.
    Please constructively work with the reviewer to get your code into a
    mergeable state (see also below).
  3. Get a final review by one of the InfluxData maintainers. Please fix any issue
    raised.
  4. Wait for the pull-request to be merged. It might take some time until your PR
    gets merged, depending on the release cycle and the type of your pull-request
    (bugfix, enhancement of existing code, new plugin, etc). Remember, it might
    be necessary to rebase your code before merge to resolve conflicts.

Please read the review comments carefully, fix the related part of the code
and/or respond in case there is anything unclear. Maintainers will add the
waiting for response tag to PRs to make it clear we are waiting on the
submitter for updates.

[!IMPORTANT]
If there is no activity on a pull request or the contributor does not respond
after the tag was added, our bot will automatically close the PR after two
weeks! If you expect a longer period of inactivity or you want to abandon a
pull request, please let us know.

In case you still want to continue with the PR, please drop a note in the PR
so we can reopen it.

Reviewing Plugin Code

  • Avoid variables scoped to the package. Everything should be scoped to the
    plugin struct, since multiple instances of the same plugin are allowed and
    package-level variables will cause race conditions.
  • SampleConfig must match the readme, but not include the plugin name.
  • structs should include toml tags for fields that are expected to be editable
    from the config. eg toml:"command" (snake_case)
  • plugins that want to log should declare the Telegraf logger, not use the log
    package. eg:
go
Log telegraf.Logger `toml:"-"`

(in tests, you can do myPlugin.Log = testutil.Logger{})

  • Initialization and config checking should be done on the Init() error
    function, not in the Connect, Gather, or Start functions.
  • Init() error should not contain connections to external services. If
    anything fails in Init, Telegraf will consider it a configuration error and
    refuse to start.
  • plugins should avoid synchronization code if they are not starting goroutines.
    Plugin functions are never called in parallel.
  • avoid goroutines when you don't need them and removing them would simplify the
    code
  • errors should almost always be checked.
  • avoid boolean fields when a string or enumerated type would be better for
    future extension. Lots of boolean fields also make the code difficult to
    maintain.
  • use config.Duration instead of internal.Duration
  • compose tls.ClientConfig as opposed to specifying all the TLS fields manually
  • http.Client should be declared once on Init() error and reused, (or better
    yet, on the package if there's no client-specific configuration).
    http.Client has built-in concurrency protection and reuses connections
    transparently when possible.
  • avoid doing network calls in loops where possible, as this has a large
    performance cost. This isn't always possible to avoid.
  • when processing batches of records with multiple network requests (some
    outputs that need to partition writes do this), return an error when you want
    the whole batch to be retried, log the error when you want the batch to
    continue without the record
  • consider using the StreamingProcessor interface instead of the (legacy)
    Processor interface
  • avoid network calls in processors when at all possible. If it's necessary,
    it's possible, but complicated (see processor.reversedns).
  • avoid dependencies when:
    • they require cgo
    • they pull in massive projects instead of small libraries
    • they could be replaced by a simple http call
    • they seem unnecessary, superfluous, or gratuitous
  • consider adding build tags if plugins have OS-specific considerations
  • use the right logger log levels so that Telegraf is normally quiet eg
    plugin.Log.Debugf() only shows up when running Telegraf with --debug
  • consistent field types: dynamically setting the type of a field should be
    strongly avoided as it causes problems that are difficult to solve later,
    made worse by having to worry about backwards compatibility in future changes.
    For example, if an numeric value comes from a string field and it is not clear
    if the field can sometimes be a float, the author should pick either a float
    or an int, and parse that field consistently every time. Better to sometimes
    truncate a float, or to always store ints as floats, rather than changing the
    field type, which causes downstream problems with output databases.
  • backwards compatibility: We work hard not to break existing configurations
    during new changes. Upgrading Telegraf should be a seamless transition.
    Possible tools to make this transition smooth are:
    • enumerable type fields that allow you to customize behavior
      (avoid boolean feature flags)
    • version fields that can be used to opt in to newer changed behavior without
      breaking old (see inputs.mysql for example)
    • a new version of the plugin if it has changed significantly
      (e.g. outputs.influxdb and outputs.influxdb_v2)
    • Logger and README deprecation warnings
    • changing the default value of a field can be okay, but will affect users who
      have not specified the field and should be approached cautiously.
    • The general rule here is "don't surprise me": users should not be caught
      off-guard by unexpected or breaking changes.

Linting

Each pull request will have the appropriate linters checking the files for any
common mistakes using the Super Linter github action. If it is
failing you can click on the action and read the logs to figure out the issue.
You can also run the github action locally for faster
feedback. You can find more information on each of the linters in the super
linter readme.

Testing

Sufficient unit tests must be created. New plugins must always contain
some unit tests. Bug fixes and enhancements should include new tests, but
they can be allowed if the reviewer thinks it would not be worth the effort.

Table Driven Tests are
encouraged to reduce boiler plate in unit tests.

The stretchr/testify library should be
used for assertions within the tests when possible, with preference towards
github.com/stretchr/testify/require.

Primarily use the require package to avoid cascading errors:

go
assert.Equal(t, lhs, rhs) # avoid
require.Equal(t, lhs, rhs) # good

Configuration

The config file is the primary interface and should be carefully scrutinized.

Ensure the example configuration and README
match with the current standards.

READMEs should:

  • be spaces, not tabs
  • be indented consistently, matching other READMEs
  • have two # for comments
  • have one # for optional options with the value being the default
  • include all appropriate types as a list for enumerable field types
  • include a useful example, avoiding "example", "test", etc.
  • include tips for any common problems
  • include example output from the plugin if the plugin outputs data

Metric Schema

Telegraf metrics are heavily based on InfluxDB points, but have some
extensions to support other outputs and metadata.

New metrics must follow the recommended schema design.
Each metric should be evaluated for series cardinality, proper use of tags vs
fields, and should use existing patterns for encoding metrics.

Metrics use snake_case naming style.

Enumerations

Generally enumeration data should be encoded as a tag. In some cases it may
be desirable to also include the data as an integer field:

shell
net_response,result=success result_code=0i

Histograms

Use tags for each range with the le tag, and +Inf for the values out of
range. This format is inspired by the Prometheus project:

shell
cpu,le=0.0 usage_idle_bucket=0i 1486998330000000000
cpu,le=50.0 usage_idle_bucket=2i 1486998330000000000
cpu,le=100.0 usage_idle_bucket=2i 1486998330000000000
cpu,le=+Inf usage_idle_bucket=2i 1486998330000000000

Lists

Lists are tricky, but the general technique is to encode using a tag, creating
one series be item in the list.

Counters

Counters retrieved from other projects often are in one of two styles,
monotonically increasing without reset and reset on each interval. No attempt
should be made to switch between these two styles but if given the option it
is preferred to use the non-resetting variant. This style is more resilient in
the face of downtime and does not contain a fixed time element.

Source tag

When metrics are gathered from another host, the metric schema should have a tag
named "source" that contains the other host's name. See this feature
request for details.

The metric schema doesn't need to have a tag for the host running
telegraf. Telegraf agent code can add a tag named "host" and by default
containing the hostname reported by the kernel. This can be configured through
the "hostname" and "omit_hostname" agent settings.

Go Best Practices

In general code should follow best practice describe in Code Review
Comments.

Networking

All network operations should have appropriate timeouts. The ability to
cancel the option, preferably using a context, is desirable but not always
worth the implementation complexity.

Channels

Channels should be used in judiciously as they often complicate the design and
can easily be used improperly. Only use them when they are needed.

2. Official Technical Reference & Guides (influxdata/docs)

File: README.md

What is this?

This repository contains the image documentation for each of the Docker Official Images. See docker-library/official-images for more information about the program in general.

All Markdown files here are run through tianon's fork of markdownfmt, and verified as formatted correctly via GitHub Actions.

Table of Contents

  1. What is this?
    1. Table of Contents
  2. How do I update an image's docs
  3. How do I add a new image's docs
  4. Files related to an image's docs
    1. folder ``
    2. README.md
    3. content.md
    4. get-help.md
    5. github-repo
    6. license.md
    7. logo.png
    8. maintainer.md
    9. README-short.txt
    10. stack.yml
  5. Files for main Docs repo
    1. update.sh
    2. markdownfmt.sh and ymlfmt.sh
    3. .template-helpers/generate-dockerfile-links-partial.sh
    4. .template-helpers/
  6. Scripts unrelated to templates
    1. generate-repo-stub-readme.sh
    2. push.pl and push.sh
  7. Issues and Contributing

How do I update an image's docs

Edit the content.md for an image; not the README.md as it's auto-generated from the contents of the other files in that repo. To see the changes to the README.md, run ./update.sh myimage from the repo root, but do not add the README.md changes to your pull request. See also markdownfmt.sh point below.

After opening your Pull Request the changes will be checked by an automated markdownfmt.sh before it can be merged. A common issue is incorrect spacing such as with two lines missing an empty line between them (double-spaced).

How do I add a new image's docs

  • Create a folder for my image: mkdir myimage
  • Create a README-short.txt (required, 100 char max)
  • Create a content.md (required)
  • Create a license.md (required)
  • Create a maintainer.md (required)
  • Create a github-repo (required)
  • Add a logo.png (recommended)

Optionally:

  • Run ./markdownfmt.sh -l myimage to list any files that are non-compliant to tianon/markdownfmt.
    Any files in the list will result in a failed build during continuous integration.
    • run ./markdownfmt.sh -d myimage to see a diff of changes required to pass.
  • Run ./update.sh myimage to generate myimage/README.md for manual review of the generated copy.
    Note: do not actually commit the README.md file; it is automatically generated/committed before being uploaded to Docker Hub.

folder ``

This is where all the partial (e.g. content.md) and generated files (e.g. README.md) for a given image reside, (e.g. golang/). It must match the name of the image used in docker-library/official-images.

`README.md`

This file is generated using update.sh. Do not commit or edit this file; it is regenerated periodically by a bot.

`content.md`

This file contains the main content of your image's long description. The basic parts you should have are a "What Is" section and a "How To" section. The following is a basic layout:

markdown
# What is XYZ?

// about what the contained software is

%%LOGO%%

# How to use this image

// descriptions and examples of common use cases for the image
// make use of subsections as necessary

`get-help.md`

This file is an optional override of the default get-help.md. This is the content of the "Where to get help" part of the "Quick reference" at the top of the generated README. We recommend linking to the best places for community support like forums, chat rooms, or mailing lists.

`github-repo`

This file should contain the URL to the GitHub repository for the Dockerfiles that become the images. The file should be in a single line ending in a newline with no extraneous whitespace. Only one GitHub repo per image repository is supported. It is used in generating links. Here is an example for golang:

text
https://github.com/docker-library/golang

`license.md`

This file should contain a link to the license for the main software in the image. Here is an example for golang:

markdown
View [license information](http://golang.org/LICENSE) for the software contained in this image.

`logo.png`

Logo for the contained software. While there are not hard rules on formatting, most existing logos are square or landscape and stay within a few hundred pixels of width. Alternatively, a logo.svg can be used instead, but only one logo file will apply. To use it within content.md, put %%LOGO%% as shown above in the basic content.md layout.

The image is automatically scaled to a 120 pixel square for the top of the Docker Hub page and Hub search results.

`maintainer.md`

This file should contain a link to the maintainers of the Dockerfile.

`README-short.txt`

This is the short description for the Docker Hub, limited to 100 characters in a single line.

Go (golang) is a general purpose, higher-level, imperative programming language.

`stack.yml`

This optional file contains a small, working Compose file for Docker Swarm showing off how to use the image. To use the stack.yml, add %%STACK%% to the content.md and this will embed the YAML along with a link to directly try it in Play with Docker.

The file must work via docker stack deploy since that is how Play with Docker will launch it, but it is helpful for users to try locally if it works for docker-compose as well. Other official images may be referenced within the YAML to demonstrate the functionality of the image, but no images external to the Docker Official Images program may be referenced.

Files for main Docs repo

`update.sh`

This is the main script used to generate the README.md files for each image. The generated file is committed along with the files used to generate it. Accepted arguments are which image(s) you want to update or no arguments to update all of them.

This script assumes bashbrew is in your PATH (for scraping relevant tag information from the library manifest file for each repository).

`markdownfmt.sh` and `ymlfmt.sh`

These two scripts are for verifying the formatting of Markdown (.md) and YAML (.yml) files, respectively. markdownfmt.sh uses the tianon/markdownfmt image and ymlfmt.sh uses the tianon/ymlfmt image.

This script is used by update.sh to create the "Supported tags and respective Dockerfile links" section of each generated README.md from the information in the official-images library/ manifests.

`.template-helpers/`

The scripts and Markdown files in here are used in building an image's README.md file in combination with its individual files.

Scripts unrelated to templates

`generate-repo-stub-readme.sh`

This is used to generate a simple README.md to put in the image's repo. We use this in Git repositories within https://github.com/docker-library to simplify our maintenance, but it is not required for anyone else. The only argument is the name of the image (or repo), like golang and it then outputs the readme to standard out.

`push.pl` and `push.sh`

These are used by us to push the actual content of the READMEs to the Docker Hub as special access is required to modify the Hub description contents. The Dockerfile is used to create a suitable environment for push.pl.

Issues and Contributing

If you would like to make a new Official Image, be sure to follow the guidelines.

Feel free to make a pull request for fixes and improvements to current documentation. For questions or problems on this repo come talk to us via the #docker-library IRC channel on Libera.Chat or open up an issue.

File: adminer/README.md

Quick reference

Quick reference (cont.)

Adminer

What is Adminer?

Adminer (formerly phpMinAdmin) is a full-featured database management tool written in PHP. Conversely to phpMyAdmin, it consist of a single file ready to deploy to the target server. Adminer is available for MySQL, PostgreSQL, SQLite, MS SQL, Oracle, Firebird, SimpleDB, Elasticsearch and MongoDB.

adminer.org

How to use this image

Standalone

console
$ docker run --link some_database:db -p 8080:8080 adminer

Then you can hit http://localhost:8080 or http://host-ip:8080 in your browser.

FastCGI

If you are already running a FastCGI capable web server you might prefer running Adminer via FastCGI:

console
$ docker run --link some_database:db -p 9000:9000 adminer:fastcgi

Then point your web server to port 9000 of the container.

Note: This exposes the FastCGI socket to the Internet. Make sure to add proper firewall rules or use a private Docker network instead to prevent a direct access.

... via [`docker stack deploy`](https://docs.docker.com/engine/reference/commandline/stack_deploy/) or [`docker-compose`](https://github.com/docker/compose)

Example stack.yml for adminer:

yaml
# Use root/example as user/password credentials

version: '3.1'

services:

  adminer:
    image: adminer
    restart: always
    ports:
      - 8080:8080

  db:
    image: mysql:5.6
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: example

Run docker stack deploy -c stack.yml adminer (or docker-compose -f stack.yml up), wait for it to initialize completely, and visit http://swarm-ip:8080, http://localhost:8080, or http://host-ip:8080 (as appropriate).

Loading plugins

This image bundles all official Adminer plugins. You can find the list of plugins on GitHub: https://github.com/vrana/adminer/tree/master/plugins.

To load plugins you can pass a list of filenames in ADMINER_PLUGINS:

console
$ docker run --link some_database:db -p 8080:8080 -e ADMINER_PLUGINS='tables-filter tinymce' adminer

If a plugin requires parameters to work correctly instead of adding the plugin to ADMINER_PLUGINS, you need to add a custom file to the container:

console
$ docker run --link some_database:db -p 8080:8080 -e ADMINER_PLUGINS='login-servers' adminer
Unable to load plugin file "login-servers", because it has required parameters: servers
Create a file "/var/www/html/plugins-enabled/login-servers.php" with the following contents to load the plugin:

<?php
require_once('plugins/login-servers.php');

/** Set supported servers
    * @param array array($domain) or array($domain => $description) or array($category => array())
    * @param string
    */
return new AdminerLoginServers(
    $servers = ???,
    $driver = 'server'
);

To load a custom plugin you can add PHP scripts that return the instance of the plugin object to /var/www/html/plugins-enabled/.

Choosing a design

The image bundles all the designs that are available in the source package of adminer. You can find the list of designs on GitHub: https://github.com/vrana/adminer/tree/master/designs.

To use a bundled design you can pass its name in ADMINER_DESIGN:

console
$ docker run --link some_database:db -p 8080:8080 -e ADMINER_DESIGN='nette' adminer

To use a custom design you can add a file called /var/www/html/adminer.css.

Usage with external server

You can specify the default host with the ADMINER_DEFAULT_SERVER environment variable. This is useful if you are connecting to an external server or a docker container named something other than the default db.

console
docker run -p 8080:8080 -e ADMINER_DEFAULT_SERVER=mysql adminer

Supported Drivers

While Adminer supports a wide range of database drivers this image only supports the following out of the box:

  • MySQL
  • PostgreSQL
  • SQLite
  • SimpleDB
  • Elasticsearch

To add support for the other drivers you will need to install the following PHP extensions on top of this image:

  • pdo_dblib (MS SQL)
  • oci8 (Oracle)
  • interbase (Firebird)
  • mongodb (MongoDB)

License

View license information for the software contained in this image.

As with all Docker images, these likely also contain other software which may be under other licenses (such as Bash, etc from the base distribution, along with any direct or indirect dependencies of the primary software being contained).

Some additional license information which was able to be auto-detected might be found in the repo-info repository's adminer/ directory.

As for any pre-built image usage, it is the image user's responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within.

File: aerospike/README.md

Quick reference

Quick reference (cont.)

Aerospike Database Docker Images

What is Aerospike?

Aerospike is a distributed NoSQL database purposefully designed for high performance web scale applications. Aerospike supports key-value and document data models, and has multiple data types including List, Map, HyperLogLog, GeoJSON, and Blob. Aerospike's patented hybrid memory architecture delivers predictable high performance at scale and high data density per node.

Getting Started

Aerospike Enterprise Edition requires a feature key file to start and to ungate certain features in the database, such as compression. Enterprise customers can use their production or development keys.

Anyone can sign up to get an evaluation feature key file for a full-featured, single-node Aerospike Enterprise Edition.

Aerospike Community Edition supports the same developer APIs as Aerospike Enterprise Edition, and differs in ease of operation and enterprise features. See the product matrix for more.

Running an Aerospike EE node with a feature key file in a mapped directory

console
docker run -d -v DIR:/opt/aerospike/etc/ -e "FEATURE_KEY_FILE=/opt/aerospike/etc/features.conf" --name aerospike -p 3000-3002:3000-3002 aerospike:ee-[version]

Above, DIR is a directory on your machine where you drop your feature key file. Make sure Docker Desktop has file sharing permission to bind mount it into Docker containers.

Running an Aerospike EE node with a feature key file in a mapped directory

console
FEATKEY=$(base64 ~/Desktop/evaluation-features.conf)
docker run -d -e "FEATURES=$FEATKEY" -e "FEATURE_KEY_FILE=env-b64:FEATURES" --name aerospike -p 3000-3002:3000-3002 aerospike:ee-[version]

Running an Aerospike CE node

console
docker run -d --name aerospike -p 3000-3002:3000-3002 aerospike:ce-[version]

Advanced Configuration

The Aerospike Docker image has a default configuration file template that can be populated with individual configuration parameters, as we did before with FEATURE_KEY_FILE. Alternatively, it can be replaced with a custom configuration file.

The following sections describe both advanced options.

Injecting configuration parameters

You can inject parameters into the configuration template using container-side environment variables with the -e flag.

For example, to set the default namespace name to demo:

console
docker run -d --name aerospike -e "NAMESPACE=demo" -p 3000-3002:3000-3002 -v /my/dir:/opt/aerospike/etc/ -e "FEATURE_KEY_FILE=/opt/aerospike/etc/features.conf" aerospike:ee-[version]

Injecting configuration parameters into the configuration template isn't compatible with using a custom configuration file. You can use one or the other.

List of template variables

  • FEATURE_KEY_FILE - the feature_key_file is only required for the EE image. Default: /etc/aerospike/features.conf
  • SERVICE_THREADS - the service_threads. Default: Number of vCPUs
  • LOGFILE - the file param of the logging context. Default: /dev/null, do not log to file, log to stdout
  • SERVICE_ADDRESS - the bind address of the networking.service subcontext. Default: any
  • SERVICE_PORT - the port of the networking.service subcontext. Default: 3000
  • HB_ADDRESS - the networking.heartbeat address for cross cluster mesh. Default: any
  • HB_PORT - the port for networking.heartbeat communications. Default: 3002
  • FABRIC_ADDRESS - the address of the networking.fabric subcontext. Default: any
  • FABRIC_PORT - the port of the networking.fabric subcontext. Default: 3001

The single preconfigured namespace is in-memory with filesystem persistence

  • NAMESPACE - the name of the namespace. Default: test
  • REPL_FACTOR - the namespace replication-factor. Default: 2
  • MEM_GB - the namespace memory-size. Default: 1, the unit is always G (GB)
  • DEFAULT_TTL - the namespace default-ttl. Default: 30d
  • STORAGE_GB - the namespace persistence file size. Default: 4, the unit is always G (GB)
  • NSUP_PERIOD - the namespace nsup-period. Default: 120 , nsup-period in seconds

Using a custom configuration file

You can override the use of the configuration file template by providing your own aerospike.conf, as described in Configuring Aerospike Database.

You should first -v map a local directory, which Docker will bind mount. Next, drop your aerospike.conf file into this directory. Finally, use the --config-file option to tell Aerospike where in the container the configuration file is (the default path is /etc/aerospike/aerospike.conf). Remember that the feature key file is required, so use feature-key-file in your config file to point to a mounted path (such as /opt/aerospike/etc/feature.conf).

For example:

console
docker run -d -v /opt/aerospike/etc/:/opt/aerospike/etc/ --name aerospike-p 3000-3002:3000-3002 aerospike:ee-[version] --config-file /opt/aerospike/etc/aerospike.conf

Persistent Data Directory

With Docker, the files within the container are not persisted past the life of the container. To persist data, you will want to mount a directory from the host to the container's /opt/aerospike/data using the -v option:

For example:

console
docker run -d  -v /opt/aerospike/data:/opt/aerospike/data  -v /opt/aerospike/etc:/opt/aerospike/etc/ --name aerospike -p 3000-3002:3000-3002 -e "FEATURE_KEY_FILE=/opt/aerospike/etc/features.conf" aerospike:ee-[version]

The example above uses the configuration template, where the single defined namespace is in-memory with file-based persistence. Just mounting the predefined /opt/aerospike/data directory enables the data to be persisted on the host.

Alternatively, a custom configuration file is used with the parameter file set to be a file in the mounted /opt/aerospike/data, such as in the following config snippet:

text
namespace test {
    # :
    storage-engine device {
        file /opt/aerospike/data/test.dat
        filesize 4G
        data-in-memory true
    }
}

In this example we also mount the data directory in a similar way, using a custom configuration file.

console
docker run -d -v /opt/aerospike/data:/opt/aerospike/data -v /opt/aerospike/etc/:/opt/aerospike/etc/ --name aerospike -p 3000-3002:3000-3002 aerospike:ee-[version] --config-file /opt/aerospike/etc/aerospike.conf

Block Storage

Docker provides an ability to expose a host's block devices to a running container. The --device option can be used to map a host block device within a container.

Update the storage-engine device section of the namespace in the custom aerospike configuration file.

text
namespace test {
    # :
    storage-engine device {
        device /dev/xvdc
            write-block-size 128k
    }
}

Now to map a host drive /dev/sdc to /dev/xvdc on a container

console
docker run -d --device '/dev/sdc:/dev/xvdc' -v /opt/aerospike/etc/:/opt/aerospike/etc/ --name aerospike -p 3000-3002:3000-3002 aerospike:ee-[version] --config-file /opt/aerospike/etc/aerospike.conf

Persistent Lua Cache

Upon restart, your lua cache will become emptied. To persist the cache, you will want to mount a directory from the host to the container's /opt/aerospike/usr/udf/lua using the -v option:

sh
docker run -d -v /opt/aerospike/lua:/opt/aerospike/usr/udf/lua -v /opt/aerospike/data:/opt/aerospike/data --name aerospike -p 3000-3002:3000-3002 -e "FEATURE_KEY_FILE=/opt/etc/aerospike/features.conf" aerospike:ee-[version]

Clustering

Developers using the Aerospike EE single-node evaluation, and most others using Docker Desktop on their machine for development, will not need to configure the node for clustering. If you're interested in using clustering and have a feature key file without a single node limit, read the following sections.

Configuring the node's access address

In order for the Aerospike node to properly broadcast its address to the cluster and applications, the access-address configuration parameter needs to be set in the configuration file. If it is not set, then the IP address within the container will be used, which is not accessible to other nodes.

text
network {
        service {
            address any                  # Listening IP Address
            port 3000                    # Listening Port
            access-address 192.168.1.100 # IP Address used by cluster nodes and applications
        }

Mesh Clustering

Mesh networking requires setting up links between each node in the cluster. This can be achieved in two ways:

  1. Add a configuration for each node in the cluster, as defined in Network Heartbeat Configuration.
  2. Use asinfo to send the tip command, to make the node aware of another node, as defined in tip command in asinfo.

For more, see How do I get a 2 nodes Aerospike cluster running quickly in Docker without editing a single file?

Image Versions

These images are based on debian:strech-slim.

ee-[version]

These tags are for Aerospike EE images, and will require a feature key file, such as the one you get with the single-node EE evaluation, or one associated with a commercial enterprise license agreement.

ce-[version]

These tags are for Aerospike CE images, and do not require a feature key file to start. As mentioned above, the developer API for both is the same, but the editions differ in operational features.

Reporting Issues

If you have any problems with or questions about this image, please post on the Aerospike discussion forum.

Enterprise customers are welcome to participate in the community forum, but can also report issues through the enterprise support system.

Aerospike EE evaluation users can open an issue in aerospike/aerospike-server-enterprise.docker.

Aerospike CE users can open an issue in aerospike/aerospike-server.docker.

License

Copyright 2019-2021 Aerospike, Inc.

If you are using a feature key file you received as part of your commercial enterprise license, you are operating under the Aerospike Master License Agreement.

If you are using the Aerospike Database EE evaluation feature key file, you are operating under the Aerospike Evaluation License Agreement.

If you are using Aerospike Database CE refer to the license information in the aerospike/aerospike-server repository.

As with all Docker images, these likely also contain other software which may be under other licenses (such as Bash, etc from the base distribution, along with any direct or indirect dependencies of the primary software being contained).

Some additional license information which was able to be auto-detected might be found in the repo-info repository's aerospike/ directory.

As for any pre-built image usage, it is the image user's responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within.

File: almalinux/README.md

Quick reference

Quick reference (cont.)

AlmaLinux OS

AlmaLinux OS is an Open Source and forever-free enterprise Linux distribution, governed and driven by the community, focused on long-term stability and a robust production-grade platform. AlmaLinux OS is 1:1 binary compatible with RHELÂŽ 8 and it was founded by the team behind the well-established CloudLinux OS.

About this image

Platform image

The default (platform) image is a general-purpose image with a full DNF stack and basic tools like find, tar, vi, etc.

The almalinux:latest tag will always point to the latest stable release of the default image. Major releases and minor releases are also tagged with their version (e.g. almalinux:8 or almalinux:8.4).

Minimal image

The minimal image is a stripped-down image that uses the microdnf package manager and contains a very limited package set. It is designed for applications that come with their own dependencies bundled (e.g. NodeJS, Python).

The almalinux:minimal tag always points to the most recent version of the minimal image. Tags for major (e.g. almalinux:8-minimal) and minor (e.g. almalinux:8.4-minimal) releases are also available.

Upgrade policy

All images for supported releases will be updated monthly or as needed for security fixes.

How It's Made

The rootfs tarballs for this image are built using the livemedia-creator tool. The build script and kickstart files are available in the AlmaLinux/docker-images git repository.

License

View license information for the software contained in this image.

As with all Docker images, these likely also contain other software which may be under other licenses (such as Bash, etc from the base distribution, along with any direct or indirect dependencies of the primary software being contained).

Some additional license information which was able to be auto-detected might be found in the repo-info repository's almalinux/ directory.

As for any pre-built image usage, it is the image user's responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within.

File: alpine/README.md

Quick reference

Quick reference (cont.)

What is Alpine Linux?

Alpine Linux is a Linux distribution built around musl libc and BusyBox. The image is only 5 MB in size and has access to a package repository that is much more complete than other BusyBox based images. This makes Alpine Linux a great image base for utilities and even production applications. Read more about Alpine Linux here and you can see how their mantra fits in right at home with Docker images.

How to use this image

Usage

Use like you would any other base image:

dockerfile
FROM alpine:3.14
RUN apk add --no-cache mysql-client
ENTRYPOINT ["mysql"]

This example has a virtual image size of only 36.8MB. Compare that to our good friend Ubuntu:

dockerfile
FROM ubuntu:20.04
RUN apt-get update \
    && apt-get install -y --no-install-recommends mysql-client \
    && rm -rf /var/lib/apt/lists/*
ENTRYPOINT ["mysql"]

This yields us a virtual image size of about 145MB image.

License

View license information for the software contained in this image.

As with all Docker images, these likely also contain other software which may be under other licenses (such as Bash, etc from the base distribution, along with any direct or indirect dependencies of the primary software being contained).

Some additional license information which was able to be auto-detected might be found in the repo-info repository's alpine/ directory.

As for any pre-built image usage, it is the image user's responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within.

File: alt/README.md

Quick reference

Quick reference (cont.)

  • Where to file issues:
    First of all, please, make sure that the developers are not aware of the bug in ALT bugzilla.

For create new bug, please use ALT's bugzilla page (choose Official image as component and include details about image problems in the description) or GitHub.

ALT Linux

This image serves as the official ALT image for the ALT Linux.

ALT Linux Team

ALT Linux Team is an international community numbering over 200 free software developers.

Sisyphus

Sisyphus is a daily updated repository which serves as a basis for all ALT distributions. Sisyphus integrity supported by the ALT Linux Team, the original software packaging technology, and apt-get (apt-rpm) utility allows users to easily update the system and be aware of any news on the free software. This daily updated repository contains the latest software with all its benefits and pitfalls (latent sometimes).

The name Sisyphus comes from the Greek mythology. Uninterrupted efforts on improvement of technologies set in the repository make ALT Linux Team similar to tireless Sisyphus continuously rolling stones up the hill.

About Platform 9

As said above, Sisyphus is a frequently updated repository mainly designed for developers. Stable distributions by ALT are the best solution for those users preferring stability and predictability of the system to its functionality (primarily new and corporate users). Such stable distributions are based on stable Sisyphus repository slices called platforms. Platform 9 (p9) was created in June, 2019, and will be supported till December, 2022.

About this image

The alt:latest tag will always point the latest stable release (which is, at the time of this writing, alt:p9).

How It's Made

This image was created with the help of mkimage and hasher tools.

Mkimage is a tool used to create images with Linux kernel from the set Sisyphus-like repository. Mkimage uses a set of Makefiles and shell hooks.

Hasher is a tool used for safe and repeatable packaging in clean and controllable environment. This is achieved through creation of a minimum build environment in chroot, installation of all necessary packaging dependencies there, and packet assembly in the new environment. Every package requires a new build environment.

License

View license information for the software contained in this image.

As with all Docker images, these likely also contain other software which may be under other licenses (such as Bash, etc from the base distribution, along with any direct or indirect dependencies of the primary software being contained).

Some additional license information which was able to be auto-detected might be found in the repo-info repository's alt/ directory.

As for any pre-built image usage, it is the image user's responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within.

File: amazoncorretto/README.md

Quick reference

Quick reference (cont.)

What is Amazon Corretto?

Corretto is a binary distribution of the Open Java Development Kit (OpenJDK) with long-term support from Amazon. Corretto is certified using the Java Technical Compatibility Kit (TCK) to ensure it meets the Java SE standard and is available on Linux, Windows, and macOS. It includes patches from Amazon that have proven useful in running our own services.

Why should I use Corretto?

Corretto is a reliable binary distribution of OpenJDK with the assurance of long-term support provided at no cost to you. Amazon runs Corretto internally on thousands of production services. Every modification we make to Corretto fixes or mitigates a problem we found running OpenJDK. Amazon also plans to apply urgent fixes (including security) when they are available and ready to use, outside of the regular quarterly cycle.

How is Corretto different from OpenJDK?

Corretto is a distribution of Open JDK with patches included by Amazon that are not yet integrated in the corresponding OpenJDK update projects. We focus on patches that improve performance or stability in OpenJDK, chosen based on Amazon's observations running large services.

What kinds of patches does Amazon intend to include in Corretto?

Patches will include security fixes, performance enhancements (e.g., speeding up frequently used functions), garbage collection scheduling, and preventing out-of-memory situations, as well as improved monitoring, reporting, and thread management.

Is there any cost associated with using Corretto?

Corretto is distributed by Amazon under an Open Source license at no cost to you. It is licensed under the terms of the GNU Public License version 2 with the Class Path Exception (GPLv2 with CPE). Amazon does not charge for its use or distribution.

What is included in Corretto's long-term support?

Amazon will provide security updates for Corretto 8 until at least June 2023. Updates are planned to be released quarterly. Corretto 11, corresponding to OpenJDK 11, will be available during the first half of 2019. Amazon will support Corretto 11 with quarterly updates until at least August 2024.

Can I use Corretto as a drop-in replacement for other JDKs?

Corretto is designed as a drop-in replacement for all Java SE distributions unless you are using features (e.g., Java Flight Recorder) not available in OpenJDK. Once Corretto binaries are installed on a host and correctly invoked to run your Java applications (e.g., using the alternatives command on Linux), existing command-line options, tuning parameters, monitoring, and anything else in place will continue to work as before.

Why does security scanner show that a docker image has a CVE?

If a security scanner reports that an amazoncorretto image includes a CVE, the first recommended action is to pull an updated version of this image.

If no updated image is available, run the appropriate command to update packages for the platform, ie. run "apk -U upgrade" for Alpine or "yum update -y --security" for AmazonLinux in your Dockerfiles or systems to resolve the issue immediately.

If no updated package is available, please treat this as a potential security issue and follow these instructions or email AWS security directly at [email protected].

It is the responsibility of the base docker image supplier to provide timely security updates to images and packages. The amazoncorretto images are automatically rebuilt when a new base image is made available, but we do not make changes to our Dockerfiles to pull in one-off package updates. If a new base image has not yet been made generally available by a base docker image maintainer, please contact that maintainer to request that the issue be addressed.

Note that there are multiple reasons why a CVE may appear to be present in a docker image, as explained in the docker library FAQs.

Image Variants

The amazoncorretto images come in many flavors, each designed for a specific use case.

`amazoncorretto:`

This is the defacto image. If you are unsure about what your needs are, you probably want to use this one. It is designed to be used both as a throw away container (mount your source code and start the container to start your app), as well as the base to build other images off of.

`amazoncorretto:-alpine`

This image is based on the popular Alpine Linux project, available in the alpine official image. Alpine Linux is much smaller than most distribution base images (~5MB), and thus leads to much slimmer images in general.

This variant is useful when final image size being as small as possible is your primary concern. The main caveat to note is that it does use musl libc instead of glibc and friends, so software will often run into issues depending on the depth of their libc requirements/assumptions. See this Hacker News comment thread for more discussion of the issues that might arise and some pro/con comparisons of using Alpine-based images.

To minimize image size, it's uncommon for additional related tools (such as git or bash) to be included in Alpine-based images. Using this image as a base, add the things you need in your own Dockerfile (see the alpine image description for examples of how to install packages if you are unfamiliar).

License

Amazon Corretto is released under the same open source license as OpenJDK, which is licensed under the GNU Public License version 2 with the Class Path Exception (GPLv2 with CPE).

As with all Docker images, these likely also contain other software which may be under other licenses (such as Bash, etc from the base distribution, along with any direct or indirect dependencies of the primary software being contained).

Some additional license information which was able to be auto-detected might be found in the repo-info repository's amazoncorretto/ directory.

As for any pre-built image usage, it is the image user's responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within.

File: amazonlinux/README.md

Quick reference

Quick reference (cont.)

What is Amazon Linux?

Amazon Linux is provided by Amazon Web Services (AWS). It is designed to provide a stable, secure, and high-performance execution environment for applications running on Amazon EC2. The full distribution includes packages that enable easy integration with AWS, including launch configuration tools and many popular AWS libraries and tools. AWS provides ongoing security and maintenance updates to all instances running Amazon Linux.

The Amazon Linux container image contains a minimal set of packages. To install additional packages, use yum.

AWS provides two versions of Amazon Linux: Amazon Linux 2 and Amazon Linux AMI.

Please note, that Amazon Linux 2022 is currently in the Preview mode and is not recommended for production workloads. Review the Amazon Linux 2022 documentation to learn more.

For information on security updates for Amazon Linux, please refer to Amazon Linux 2 Security Advisories and Amazon Linux AMI Security Advisories. Note that Docker Hub's vulnerability scanning for Amazon Linux is currently based on RPM versions, which does not reflect the state of backported patches for vulnerabilities.

Where can I run Amazon Linux container images?

You can run Amazon Linux container images in any Docker based environment. Examples include, your laptop, in AWS EC2 instances, and ECS clusters.

How is Amazon Linux 2 different from Amazon Linux AMI?

There are three major differences in Amazon Linux 2 from its predecessors:

  1. it is available as a VM image for on-premises development and testing
  2. it includes systemd service and systems manager as opposed to System V init system and also includes new version of compiler and build tools
  3. it gives you the ability to install additional software packages through Extras mechanism without impacting the underlying LTS stability

Is Amazon Linux 2 build an official LTS build?

Yes. Amazon Linux 2 end of support date (End of Life, or EOL) has been extended by two years from 2023-06-30 to 2025-06-30 to provide customers with ample time to migrate to the next version.

What packages are available in the Amazon Linux containers?

Amazon Linux Docker container images contain a subset of the packages in the images for use on EC2 and as VMs in on-premises scenarios. The container images can be configured to use any of the full set of packages in images for EC2 and on-premises use. The Amazon Linux 2 container images comes with Extras included.

What is an Amazon Linux 2 Extra?

Extras is a new mechanism introduced in Amazon Linux 2 to enable the consumption of the newest versions of application software in a fully supported manner on a stable Amazon Linux 2 base. Extras help alleviate the compromise between stability of the OS and freshness of available software. For example, now you can install newer versions of Python being rest assured that the underlying operating system is stable. Examples of Extras include nginx, PostgreSQL, MariaDB, Go, and Rust.

How do Amazon Linux 2 Extras work?

Extras introduces the notion of topics to select software bundles. Each topic contains all the dependencies required for the software to install and function on Amazon Linux 2. For example, Rust is an Extras topic in the curated list provide by Amazon. It provides the toolchain and runtimes for Rust, the systems programming language. This topic includes the cmake build system for Rust, cargo - the rust package manager, and the LLVM based compiler toolchain for Rust. The packages associated with each topic are consumed via the well-known yum installation process.

How are Amazon Linux 2 Extras topics different from the packages available in yum repositories?

yum is a utility for package management of RPM packages. The base image of Amazon Linux 2 (LTS) includes access to repositories that already contain stable versions of popular packages that can be installed with yum. These packages are part of the long term support for Amazon Linux 2.

However, if you need a new software package or a newer version of an existing software package that is not included in the base Amazon Linux 2 image, Extras provide a way to install those packages in a supported manner. Extras is essentially a simplified mechanism to point yum to AWS curated sets of packages for a selected topic.

How do I install a software package from Extras repository in Amazon Linux 2?

Available packages can be listed with the amazon-linux-extras command. Packages can be installed with the amazon-linux-extras install command. Example: amazon-linux-extras install rust1

Will updates be available for Amazon Linux containers?

Similar to the Amazon Linux images for AWS EC2 and on-premises use, Amazon Linux container images will get ongoing updates from Amazon in the form of security updates, bug fix updates, and other enhancements. Security bulletins for Amazon Linux are available at https://alas.aws.amazon.com/

What support is available for Amazon Linux outside AWS?

With the availability of Amazon Linux 2, are there any changes to the existing version of Amazon Linux AMI?

With the availability of Amazon Linux 2, we are announcing that 2018.03 release of Amazon Linux AMI container image will be the last release for the current generation of Amazon Linux. Going forward, AWS will provide newer versions only for Amazon Linux 2.

Will AWS support the current version of Amazon Linux going forward?

Yes; in order to avoid any disruption to your existing applications and to facilitate migration to Amazon Linux 2, AWS will provide regular security updates for Amazon Linux 2018.03 AMI and container image for 2 years after the final LTS build is announced. You can also use all your existing support channels such as AWS Premium Support and Amazon Linux Discussion Forum to continue to submit support requests.

Is Amazon Linux 2 backward compatible with Amazon Linux AMI?

Due to the inclusion of new components in Amazon Linux 2 such as systemd, your applications running on the current version of Amazon Linux may require additional changes to run on Amazon Linux 2.

License

Amazon Linux is available under the GNU General Public License, version 2.0. Individual software packages are available under their own licenses; run rpm -qi [package name] or check /usr/share/doc/[package name]-* and /usr/share/licenses/[package name]-* for details.

As with all Docker images, these likely also contain other software which may be under other licenses (such as Bash, etc from the base distribution, along with any direct or indirect dependencies of the primary software being contained).

Some additional license information which was able to be auto-detected might be found in the repo-info repository's amazonlinux/ directory.

As for any pre-built image usage, it is the image user's responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within.

File: api-firewall/README.md

Quick reference

Quick reference (cont.)

What is API Firewall?

Wallarm API Firewall is an open-source light-weighted proxy designed to protect REST API endpoints in cloud-native environments by hardening based on a strict OpenAPI/Swagger schema validation. Wallarm API Firewall relies on a positive security model allowing calls that match a predefined API specification for requests and responses, while rejecting everything else.

The key features of API Firewall are:

  • Protect REST API endpoints by blocking requests and responses that do not match the OAS/Swagger schema
  • Discover Shadow API endpoints
  • If using OAuth 2.0 protocol-based authentication, validate access tokens
  • Quick and easy deployment and configuration
  • Customization of request and response processing modes, response codes and log format

Use cases

  • Block abnormal requests and responses that do not match the OpenAPI 3.0 specification (if running API Firewall in the blocking mode)
  • Discover Shadow APIs and undocumented endpoints (if running API Firewall in the logging mode)
  • Log abnormal requests and responses that do not match the OpenAPI 3.0 specification (if running API Firewall in the logging mode)

API schema validation and positive security model

When starting API Firewall, you should provide the OpenAPI 3.0 specification of the application to be protected with API Firewall. The started API Firewall will operate as a reverse proxy and validate whether requests and responses match the schema defined in the specification.

The traffic that does not match the schema will be logged using the STDOUT and STDERR Docker services or blocked (depending on the configured API Firewall operation mode). If operating in the logging mode and detecting the traffic on endpoints that are not included in the specification, API Firewall also logs these endpoints as the shadow ones (except for endpoints returning the code 404).

Provided API schema should be described using the OpenAPI 3.0 specification in the YAML or JSON file (.yaml, .yml, .json file extensions).

By allowing you to set the traffic requirements with the OpenAPI 3.0 specification, Wallarm API Firewall relies on a positive security model.

Technical characteristics

API Firewall works as a reverse proxy with a built-in OpenAPI 3.0 request and response validator. The validator is written in Go and optimized for extreme performance and near-zero added latency.

Starting API Firewall

To download, install, and start Wallarm API Firewall on Docker, see the instructions.

Demos

You can try API Firewall by running the demo environment that deploys an example application protected with Wallarm API Firewall. There are two available demo environments:

Performance

When creating API Firewall, we prioritized speed and efficiency to ensure that our customers would have the fastest APIs possible. Our latest tests demonstrate that the average time required for API Firewall to process one request is 1.339 ms:

console
$ ab -c 200 -n 10000 -p ./large.json -T application/json http://127.0.0.1:8282/test/signup

Document Path:          /test/signup
Document Length:        20 bytes

Concurrency Level:      200
Time taken for tests:   0.769 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      2150000 bytes
Total body sent:        283770000
HTML transferred:       200000 bytes
Requests per second:    13005.81 [#/sec] (mean)
Time per request:       15.378 [ms] (mean)
Time per request:       0.077 [ms] (mean, across all concurrent requests)
Transfer rate:          2730.71 [Kbytes/sec] received
                        360415.95 kb/s sent
                        363146.67 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    5   1.6      5      12
Processing:     2   10   5.4      9      59
Waiting:        2    8   5.2      7      56
Total:          3   15   5.7     14      68

Percentage of the requests served within a certain time (ms)
  50%     14
  66%     15
  75%     16
  80%     17
  90%     18
  95%     23
  98%     36
  99%     44
 100%     68 (longest request)

These performance results are not the only ones we have got during API Firewall testing. Other results along with the methods used to improve API Firewall performance are described in this Wallarm's blog article.

License

View license information for the software contained in this image.

As with all Docker images, these likely also contain other software which may be under other licenses (such as Bash, etc from the base distribution, along with any direct or indirect dependencies of the primary software being contained).

Some additional license information which was able to be auto-detected might be found in the repo-info repository's api-firewall/ directory.

As for any pre-built image usage, it is the image user's responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within.

File: arangodb/README.md

Quick reference

Quick reference (cont.)

What is ArangoDB?

ArangoDB is a multi-model, open-source database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions. Use ACID transactions if you require them. Scale horizontally and vertically with a few mouse clicks.

ArangoDB runs everywhere: On-prem, in the cloud and also on ArangoDB's Cloud Service Oasis.

The supported data models can be mixed in queries and allow ArangoDB to be the aggregation point for your data.

arangodb.com

Key Features in ArangoDB

Multi-Model Documents, graphs and key-value pairs — model your data as you see fit for your application.

Joins Conveniently join what belongs together for flexible ad-hoc querying, less data redundancy.

Transactions Easy application development keeping your data consistent and safe. No hassle in your client.

Joins and Transactions are key features for flexible, secure data designs, widely used in RDBMSs that you won't want to miss in NoSQL products. You decide how and when to use Joins and strong consistency guarantees, keeping all the power for scaling and performance as choice.

Furthermore, ArangoDB offers a microservice framework called Foxx to build your own Rest API with a few lines of code.

ArangoDB Documentation

How to use this image

Start an ArangoDB instance

In order to start an ArangoDB instance, run:

console
unix> docker run -e ARANGO_RANDOM_ROOT_PASSWORD=1 -d --name arangodb-instance arangodb

Will create and launch the arangodb docker instance as background process. The Identifier of the process is printed. By default ArangoDB listen on port 8529 for request and the image includes EXPOSE 8529. If you link an application container it is automatically available in the linked container. See the following examples.

In order to get the IP arango listens on, run:

console
unix> docker inspect --format '{{ .NetworkSettings.IPAddress }}' arangodb-instance

Using the instance

In order to use the running instance from an application, link the container:

console
unix> docker run -e ARANGO_RANDOM_ROOT_PASSWORD=1 --name my-app --link arangodb-instance:db-link arangodb

This will use the instance with the name arangodb-instance and link it into the application container. The application container will contain environment variables

text
DB_LINK_PORT_8529_TCP=tcp://172.17.0.17:8529
DB_LINK_PORT_8529_TCP_ADDR=172.17.0.17
DB_LINK_PORT_8529_TCP_PORT=8529
DB_LINK_PORT_8529_TCP_PROTO=tcp
DB_LINK_NAME=/naughty_ardinghelli/db-link

These can be used to access the database.

Exposing the port to the outside world

If you want to expose the port to the outside world, run:

console
unix> docker run -e ARANGO_RANDOM_ROOT_PASSWORD=1 -p 8529:8529 -d arangodb

ArangoDB listen on port 8529 for request and the image includes EXPOSE 8529. The -p 8529:8529 exposes this port on the host.

Choosing an authentication method

The ArangoDB image provides several authentication methods which can be specified via environment variables (-e) when using docker run

  1. ARANGO_RANDOM_ROOT_PASSWORD=1

    Generate a random root password when starting. The password will be printed to stdout (may be inspected later using docker logs)

  2. ARANGO_NO_AUTH=1

    Disable authentication. Useful for testing.

    WARNING Doing so in production will expose all your data. Make sure that ArangoDB is not directly accessible from the internet!

  3. ARANGO_ROOT_PASSWORD=somepassword

    Specify your own root password.

Note: this way of specifying logins only applies to single server installations. With clusters you have to provision the users via the root user with empty password once the system is up.

Command line options

You can pass arguments to the ArangoDB server by appending them to the end of the Docker command:

console
unix> docker run -e ARANGO_RANDOM_ROOT_PASSWORD=1 arangodb --help

The entrypoint script starts the arangod binary by default and forwards your arguments.

You may also start other binaries, such as the ArangoShell:

console
unix> docker run -it arangodb arangosh --server.database myDB ...

Note that you need to set up networking for containers if arangod runs in one container and you want to access it with arangosh running in another container. It is easier to execute it in the same container instead. Use docker ps to find out the container ID / name of a running container:

console
unix> docker ps
CONTAINER ID   IMAGE     COMMAND                 CREATED      STATUS      PORTS                   NAMES
1234567890ab   arangodb  "/entrypoint.sh aranâ€Ļ"  2 hours ago  Up 2 hours  0.0.0.0:8529->8529/tcp  jolly_joker

Then use docker exec and the ID / name to run something inside of the existing container:

console
unix> docker exec -it jolly_joker arangosh

See more information about Configuration

Limiting resource utilization

arangod checks the following environment variables, which can be used to restrict how much memory and how many CPU cores it should use:

  • ARANGODB_OVERRIDE_DETECTED_TOTAL_MEMORY (introduced in v3.6.3)

    This variable can be used to override the automatic detection of the total amount of RAM present on the system. One can specify a decimal number (in bytes). Furthermore, if G or g is appended, the value is multiplied by 2^30. If M or m is appended, the value is multiplied by 2^20. If K or k is appended, the value is multiplied by 2^10. That is, 64G means 64 gigabytes.

    The total amount of RAM detected is logged as an INFO message at server start. If the variable is set, the overridden value is shown. Various default sizes are calculated based on this value (e.g. RocksDB buffer cache size).

    Setting this option can in particular be useful in two cases:

    1. If arangod is running in a container and its cgroup has a RAM limitation, then one should specify this limitation in this environment variable, since it is currently not automatically detected.
    2. If arangod is running alongside other services on the same machine and thus sharing the RAM with them, one should limit the amount of memory using this environment variable.

  • ARANGODB_OVERRIDE_DETECTED_NUMBER_OF_CORES (introduced in v3.7.1)

    This variable can be used to override the automatic detection of the number of CPU cores present on the system.

    The number of CPU cores detected is logged as an INFO message at server start. If the variable is set, the overridden value is shown. Various default values for threading are calculated based on this value.

    Setting this option is useful if arangod is running in a container or alongside other services on the same machine and shall not use all available CPUs.

Persistent Data

ArangoDB supports two different storage engines from version 3.2 to 3.6. You can choose them while instantiating the container with the environment variable ARANGO_STORAGE_ENGINE. With mmfiles you choose the classic storage engine (not available in 3.7 and later), rocksdb will choose the storage engine based on RocksDB. The default choice is rocksdb from version 3.4 on.

ArangoDB uses the volume /var/lib/arangodb3 as database directory to store the collection data and the volume /var/lib/arangodb3-apps as apps directory to store any extensions. These directories are marked as docker volumes.

See docker inspect --format "{{ .Config.Volumes}}" arangodb for all volumes.

A good explanation about persistence and docker container can be found here: Docker In-depth: Volumes, Why Docker Data Containers are Good

Using host directories

You can map the container's volumes to a directory on the host, so that the data is kept between runs of the container. This path /tmp/arangodb is in general not the correct place to store you persistent files - it is just an example!

console
unix> mkdir /tmp/arangodb
unix> docker run -e ARANGO_RANDOM_ROOT_PASSWORD=1 -p 8529:8529 -d \
          -v /tmp/arangodb:/var/lib/arangodb3 \
          arangodb

This will use the /tmp/arangodb directory of the host as database directory for ArangoDB inside the container.

Using a data container

Alternatively you can create a container holding the data.

console
unix> docker create --name arangodb-persist arangodb true

And use this data container in your ArangoDB container.

console
unix> docker run -e ARANGO_RANDOM_ROOT_PASSWORD=1 --volumes-from arangodb-persist -p 8529:8529 arangodb

If want to save a few bytes you can alternatively use busybox or alpine for creating the volume only containers. Please note that you need to provide the used volumes in this case. For example

console
unix> docker run -d --name arangodb-persist -v /var/lib/arangodb3 busybox true

Using as a base image

If you are using the image as a base image please make sure to wrap any CMD in the exec form. Otherwise the default entrypoint will not do its bootstrapping work.

When deriving the image, you can control the instantiation via putting files into /docker-entrypoint-initdb.d/.

  • *.sh - files ending with .sh will be run as a bash shellscript.
  • *.js - files will be executed with arangosh. You can specify additional arangosh arguments via the ARANGOSH_ARGS environment variable.
  • dumps/ - in this directory you can place subdirectories containing database dumps generated using arangodump. They can be restored using arangorestore.

License

ArangoDB itself is licensed under the Apache License, but it contains software of third parties under their respective licenses.

As with all Docker images, these likely also contain other software which may be under other licenses (such as Bash, etc from the base distribution, along with any direct or indirect dependencies of the primary software being contained).

Some additional license information which was able to be auto-detected might be found in the repo-info repository's arangodb/ directory.

As for any pre-built image usage, it is the image user's responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within.

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#golang Browse golang catalog → #hacktoberfest Browse hacktoberfest catalog → #influxdb Browse influxdb catalog → #json Browse json catalog → #kafka Browse kafka catalog → #logs Browse logs catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →