Docker Documentation

Technical Documentation: docker/Docker

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

The Moby Project

Moby is an open-source project created by Docker to enable and accelerate software containerization.

It provides a "Lego set" of toolkit components, the framework for assembling them into custom container-based systems, and a place for all container enthusiasts and professionals to experiment and exchange ideas.
Components include container build tools, a container registry, orchestration tools, a runtime and more, and these can be used as building blocks in conjunction with other tools and projects.

Principles

Moby is an open project guided by strong principles, aiming to be modular, flexible and without too strong an opinion on user experience.
It is open to the community to help set its direction.

Modular: the project includes lots of components that have well-defined functions and APIs that work together.
Batteries included but swappable: Moby includes enough components to build fully featured container systems, but its modular architecture ensures that most of the components can be swapped by different implementations.
Usable security: Moby provides secure defaults without compromising usability.
Developer focused: The APIs are intended to be functional and useful to build powerful tools.
They are not necessarily intended as end user tools but as components aimed at developers.
Documentation and UX is aimed at developers not end users.

Audience

The Moby Project is intended for engineers, integrators and enthusiasts looking to modify, hack, fix, experiment, invent and build systems based on containers.
It is not for people looking for a commercially supported system, but for people who want to work and learn with open source code.

Relationship with Docker

The components and tools in the Moby Project are initially the open source components that Docker and the community have built for the Docker Project.
New projects can be added if they fit with the community goals. Docker is committed to using Moby as the upstream for the Docker Product.
However, other projects are also encouraged to use Moby as an upstream, and to reuse the components in diverse ways, and all these uses will be treated in the same way. External maintainers and contributors are welcomed.

The Moby project is not intended as a location for support or feature requests for Docker products, but as a place for contributors to work on open source code, fix bugs, and make the code more useful.
The releases are supported by the maintainers, community and users, on a best efforts basis only. For customers who want enterprise or commercial support, Docker Desktop and Mirantis Container Runtime are the appropriate products for these use cases.

Go modules

[!IMPORTANT]
Starting with Docker v29 (released November 2025), the Go module github.com/docker/docker is deprecated and won't be updated.

The supported public Go modules are:

Module	Description
`github.com/moby/moby/client`	Go client for the Docker Engine API
`github.com/moby/moby/api`	API types shared between client and server

The root module github.com/moby/moby/v2 is the codebase for building container engines based on Moby (such as Docker Engine).
It produces binaries only - it is not intended to be imported as a Go library and has no API stability guarantees.

Release tags

Docker Engine releases are tagged with a docker- prefix (e.g. docker-v29.0.0 for Docker Engine 29.0.0).

These tags are only used to build the Docker Engine binary from the root module - they must not be consumed via go get.

The client and api modules are versioned independently with their own tags (e.g. client/v1.x.x, api/v1.x.x).

Migrating from `github.com/docker/docker`

Replace the old import paths:

diff

- import "github.com/docker/docker/client"
+ import "github.com/moby/moby/client"

- import "github.com/docker/docker/api/types"
+ import "github.com/moby/moby/api/types"

Note that v29 includes many breaking API changes (option structs, renamed methods, moved types).
See the v29.0.0 release notes for the full list of Go SDK changes.

Legal

Brought to you courtesy of our legal counsel. For more context,
please see the NOTICE document in this repo.

Use and transfer of Moby may be subject to certain restrictions by the
United States and other governments.

It is your responsibility to ensure that your use and/or transfer does not
violate applicable laws.

For more information, please see https://www.bis.doc.gov

Licensing

Moby is licensed under the Apache License, Version 2.0. See
LICENSE for the full
license text.

Overview (`api/README.md`)

Engine API

The Engine API is an HTTP API used by the command-line client to communicate with the daemon. It can also be used by third-party software to control the daemon.

It consists of various components in this repository:

api/swagger.yaml A Swagger definition of the API.
api/types/ Types shared by both the client and server, representing various objects, options, responses, etc. Most are written manually, but some are automatically generated from the Swagger definition. See #27919 for progress on this.
client/ The Go client used by the command-line client. It can also be used by third-party Go programs.
daemon/ The daemon, which serves the API.

Swagger definition

The API is defined by the Swagger definition in api/swagger.yaml. This definition can be used to:

Automatically generate documentation.
Automatically generate the Go server and client. (A work-in-progress.)
Provide a machine readable version of the API for introspecting what it can do, automatically generating clients for other languages, etc.

Updating the API documentation

The API documentation is generated entirely from api/swagger.yaml. If you make updates to the API, edit this file to represent the change in the documentation.
Documentation for each API version can be found in the docs directory, which also provides a CHANGELOG.md.

The file is split into two main sections:

definitions, which defines re-usable objects used in requests and responses
paths, which defines the API endpoints (and some inline objects which don't need to be reusable)

To make an edit, first look for the endpoint you want to edit under paths, then make the required edits. Endpoints may reference reusable objects with $ref, which can be found in the definitions section.

There is hopefully enough example material in the file for you to copy a similar pattern from elsewhere in the file (e.g. adding new fields or endpoints), but for the full reference, see the Swagger specification.

swagger.yaml is validated by hack/validate/swagger to ensure it is a valid Swagger definition. This is useful when making edits to ensure you are doing the right thing.

Viewing the API documentation

When you make edits to swagger.yaml, you may want to check the generated API documentation to ensure it renders correctly.

Run make swagger-docs and a preview will be running at http://localhost:9000. Some of the styling may be incorrect, but you'll be able to ensure that it is generating the correct documentation.

The production documentation is generated by vendoring swagger.yaml into docker/docs.

Overview (`client/README.md`)

Go client for the Docker Engine API

The docker command uses this package to communicate with the daemon. It can
also be used by your own Go applications to do anything the command-line
interface does; running containers, pulling or pushing images, etc.

For example, to list all containers (the equivalent of docker ps --all):

package main

import (
	"context"
	"fmt"

	"github.com/moby/moby/client"
)

func main() {
	// Create a new client with "client.FromEnv" (configuring the client
	// from commonly used environment variables such as DOCKER_HOST and
	// DOCKER_API_VERSION) and set a custom User-Agent.
	//
	// API-version negotiation is enabled by default to allow downgrading
	// the API version when connecting with an older daemon version.
	apiClient, err := client.New(
		client.FromEnv,
		client.WithUserAgent("my-application/1.0.0"),
	)
	if err != nil {
		panic(err)
	}
	defer apiClient.Close()

	// List all containers (both stopped and running).
	result, err := apiClient.ContainerList(context.Background(), client.ContainerListOptions{
		All: true,
	})
	if err != nil {
		panic(err)
	}

	// Print each container's ID, status and the image it was created from.
	fmt.Printf("%s  %-22s  %s\n", "ID", "STATUS", "IMAGE")
	for _, ctr := range result.Items {
		fmt.Printf("%s  %-22s  %s\n", ctr.ID, ctr.Status, ctr.Image)
	}
}

Full documentation is available on pkg.go.dev.

Overview (`contrib/README.md`)

The contrib directory contains scripts, images, and other helpful things
which are not part of the core docker distribution. Please note that they
could be out of date, since they do not receive the same attention as the
rest of the repository.

Overview (`hack/README.md`)

About

This directory contains a collection of scripts used to build and manage this
repository. If there are any issues regarding the intention of a particular
script (or even part of a certain script), please reach out to us.
It may help us either refine our current scripts, or add on new ones
that are appropriate for a given use case.

DinD (dind.sh)

DinD is a wrapper script which allows Docker to be run inside a Docker
container. DinD requires the container to
be run with privileged mode enabled.

Generate Authors (generate-authors.sh)

Generates AUTHORS; a file with all the names and corresponding emails of
individual contributors. AUTHORS can be found in the home directory of
this repository.

Make

There are two make files, each with different extensions. Neither are supposed
to be called directly; only invoke make. Both scripts run inside a Docker
container.

make.ps1

The Windows native build script that uses PowerShell semantics; it is limited
unlike hack\make.sh since it does not provide support for the full set of
operations provided by the Linux counterpart, make.sh. However, make.ps1
does provide support for local Windows development and Windows to Windows CI.
More information is found within make.ps1 by the author, @jhowardmsft

make.sh

Referenced via make test when running tests on a local machine,
or directly referenced when running tests inside a Docker development container.
When running on a local machine, make test to run all tests found in
test, test-unit, test-integration, and test-docker-py on
your local machine. The default timeout is set in make.sh to 60 minutes
(${TIMEOUT:=60m}), since it currently takes up to an hour to run
all of the tests.
When running inside a Docker development container, hack/make.sh does
not have a single target that runs all the tests. You need to provide a
single command line with multiple targets that performs the same thing.
An example referenced from Run targets inside a development container: root@5f8630b873fe:/go/src/github.com/moby/moby# hack/make.sh dynbinary binary test-unit test-integration test-docker-py
For more information related to testing outside the scope of this README,
refer to
Run tests and test documentation

Vendor (vendor.sh)

A shell script that is a wrapper around go mod vendor.

Overview (`man/README.md`)

Docker Engine Documentation

The man pages for Docker Engine are generated from the markdown sources and tooling in this directory.

Generate the man pages

Run make from within this directory.
A Go toolchain is required.
The generated man pages will be placed in manN subdirectories, where N is the manual section number.

Install the man pages

Run make install from within this directory.
The make variables prefix, mandir, INSTALL, INSTALL_DATA and DESTDIR
are supported for customizing the installation.

Add a new man page

Create a new Markdown file in this directory with a filename TITLE.SECTION.md,
where TITLE is the man page title and SECTION is the section number.
The Makefile will pick it up automatically.

The Makefile ignores Markdown files that do not match the glob *.*.md,
allowing non-manpage documentation (like this README file) to coexist.

Overview (`pkg/README.md`)

pkg/ is a collection of utility packages used by the Moby project without being specific to its internals.

Utility packages are kept separate from the moby core codebase to keep it as small and concise as possible.
If some utilities grow larger and their APIs stabilize, they may be moved to their own repository under the
Moby organization, to facilitate re-use by other projects. However that is not the priority.

The directory pkg is named after the same directory in the camlistore project. Since Brad is a core
Go maintainer, we thought it made sense to copy his methods for organizing Go code :) Thanks Brad!

Because utility packages are small and neatly separated from the rest of the codebase, they are a good
place to start for aspiring maintainers and contributors. Get in touch if you want to help maintain them!

Overview (`project/README.md`)

Hacking on Docker

The project/ directory holds information and tools for everyone involved in the process of creating and
distributing Docker, specifically:

Guides

If you're a contributor or aspiring contributor, you should read CONTRIBUTING.md.

If you're a maintainer or aspiring maintainer, you should read MAINTAINERS.

If you're a packager or aspiring packager, you should read PACKAGERS.md.

If you're a maintainer in charge of a release, you should read RELEASE-PROCESS.md.

Roadmap

A high-level roadmap is available at ROADMAP.md.

Build tools

hack/make.sh is the primary build tool for docker. It is used for compiling the official binary,
running the test suite, and pushing releases.

Overview (`api/docs/README.md`)

API Documentation

This directory contains versioned documents for each version of the API
specification supported by this module. While this module provides support
for older API versions, support should be considered "best-effort", especially
for very old versions. Users are recommended to use the latest API versions,
and only rely on older API versions for compatibility with older clients.

Newer API versions tend to be backward-compatible with older versions,
with some exceptions where features were deprecated. For an overview
of changes for each version, refer to CHANGELOG.md.

The latest version of the API specification can be found at the root directory
of this module which may contain unreleased changes.

For API version v1.24, documentation is only available in markdown
format, for later versions Swagger (OpenAPI) v2.0
specifications can be found in this directory. The Moby project itself
primarily uses these swagger files to produce the API documentation;
while we attempt to make these files match the actual implementation,
the OpenAPI 2.0 specification has limitations that prevent us from
expressing all options provided. There may be discrepancies (for which
we welcome contributions). If you find bugs, or discrepancies, please
open a ticket (or pull request).

Overview (`contrib/otel/README.md`)

Sample stack for testing OTEL functionality with moby

To easily test the OTEL functionality present in moby, you can spin up a small demo compose stack that includes:

an OTEL collector container;
a Jaeger container to visualize traces;
an alternative Aspire Dashboard container to visualize traces;

The OTEL collector is configured to export Traces to both the Jaeger and Aspire containers.

The contrib/otel directory contains the compose file with the services configured, along with a basic configuration file for the OTEL collector.

How can I use it?

Export the env var used to override the OTLP endpoint:
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 (if running in a devcontainer or in other ways, you might have to change how you pass this env var to the daemon);
Start the moby engine you want to get traces from (make sure it gets the env var declared above);
Start the otel compose stack by running docker compose up -d in the contrib/otel/ directory;
Make some calls to the engine using the Docker CLI to send some traces to the endpoint;
Browse Jaeger at http://localhost:16686 or the Aspire Dashboard at http://localhost:18888/traces;
To see some traces from the engine, select dockerd in the top left dropdown

Note: The precise steps may vary based on how you're working on the codebase (building a binary and executing natively, running/debugging in a devcontainer, etc... )

Cleanup?

Simply run docker compose down in the contrib/otel/ directory.

You can also run unset OTEL_EXPORTER_OTLP_ENDPOINT to get rid of the OTLP env var from your environment

Overview (`contrib/wireshark/README.md`)

Wireshark Dissectors for NetworkDB

memberlist.lua is a Wireshark plugin
which registers a memberlist protocol
that can dissect the memberlist TCP and UDP protocols.
The memberlist protocol can be configured to dissect user data
as the protocol named in the memberlist.userdata_dissector preference.

moby-networkdb.lua is a Wireshark plugin which registers
a protocol named networkdbgossip
that dissects NetworkDB gossip messages.
As node-to-node communications for NetworkDB
are transported as memberlist user messages,
the memberlist protocol dissector must be configured
to invoke the networkdbgossip protocol as a sub-dissector.
(Read: set the preference memberlist.userdata_dissector:networkdbgossip)

Installation

Install Wireshark 4.5 or newer.

Wireshark 4.4 has an incomplete msgpack protocol dissector
that is unable to properly decode memberlist messages.
As of 2025-06-30 Wireshark 4.5 has yet to be released.
A nightly build may be required.

Install the plugins

Configure Wireshark/Tshark to load memberlist.lua and moby-networkdb.lua.
Refer to the Wireshark Lua manual for further instruction.

Configure the ProtoBuf protocol

NetworkDB messages are serialized to protobuf,
which is not self-describing.
The ProtoBuf Wireshark protocol needs to be supplied with
the protobuf IDL definitions of the messages
in order to dissect them.

Clone moby/moby for the NetworkDB IDL definitions.
Clone protocolbuffers/protobuf for the protobuf "standard library" IDL.
Configure the ProtoBuf protocol (Preferences -> Protocols -> ProtoBuf)
as follows:
- ✅ Load .proto files on startup. (protobuf.reload_protos)
- ✅ Dissect Protobuf fields as Wireshark fields. (protobuf.pbf_as_hf)
- Add entries to the Protobuf Search Paths table (uat:protobuf.search_paths):
  - path/to/protocolbuffers/protobuf/src
  - path/to/moby/moby/vendor
- path/to/moby/moby
- path/to/moby/moby/libnetwork/networkdb (✅ Load all files)
- path/to/moby/moby/libnetwork/drivers/overlay (✅ Load all files)

Note that it is not sufficient to just grab the .proto files from the repos.
The directory structure is necessary for the definitions to load properly.

Configure the Memberlist protocol

Configure memberlist to dissect user data messages as NetworkDB gossip.

In Preferences -> Protocols -> MEMBERLIST,
set the User Data Dissector to networkdbgossip.
(memberlist.userdata_dissector)
Optional: set Memberlist TCP+UDP port(s) (memberlist.ports) as needed.
E.g. a value such as 7946,10000-10999 would be useful
for analyzing packet captures from NetworkDB unit tests.

Usage Notes

Encryption

The memberlist protocol dissector supports decryption
of encrypted memberlist messages
when provided with a file containing the encryption keys used.
In Preferences -> Protocols -> MEMBERLIST,
set the Encryption Key Logfile Path
(memberlist.keylog)
to a file containing the encryption keys.

The logfile should list the encryption keys
as hexadecimal strings, delimited by newlines.

dockerd may be configured to write the NetworkDB encryption keys to a logfile
by setting the environment variable NETWORKDBKEYLOGFILE
to the path where the file should reside.

Known Issues

The NetworkDB protocol may fail to load with an error when Wireshark is first started:

text

moby-networkdb.lua:4: bad argument #1 to 'new'
(Field_new: a field with this name must exist)

This is due to a known issue in Wireshark.

Workaround: reload Lua plugins after Wireshark has been initialized.

Menu: Analyze -> Reload Lua Plugins, or
Keyboard shortcut: Ctrl-Shift-L (Windows/Linux), ⇧⌘L (macOS)

Limitations

Only memberlist encryption version 1 (AES-GCM 128, no padding) is supported,
not version 0 (AES-GCM 128 using PKCS#7 padding).
Labelled messages cannot currently be decrypted.

Overview (`docs/api/README.md`)

The API documentation has moved

The API is now implemented in a separate module, and the documentation
has moved to api/docs.

Overview (`docs/contributing/README.md`)

Get set up for Moby development

Overview (`hack/make/README.md`)

This directory holds scripts called by make.sh in the parent directory.

Each script is named after the bundle it creates.
They should not be called directly - instead, pass it as argument to make.sh, for example:

text

./hack/make.sh binary ubuntu

# Or to run all default bundles:
./hack/make.sh

To add a bundle:

Create a shell-compatible file here
Add it to $DEFAULT_BUNDLES in make.sh

Overview (`pkg/useragent/README.md`)

This package provides helper functions to pack version information into a single User-Agent header.

Overview (`pkg/plugins/pluginrpc-gen/README.md`)

Plugin RPC Generator

Generates go code from a Go interface definition for proxying between the plugin
API and the subsystem being extended.

Usage

Given an interface definition:

type volumeDriver interface {
	Create(name string, opts opts) (err error)
	Remove(name string) (err error)
	Path(name string) (mountpoint string, err error)
	Mount(name string) (mountpoint string, err error)
	Unmount(name string) (err error)
}

Note: All function options and return values must be named in the definition.

Run the generator:

bash

$ pluginrpc-gen --type volumeDriver --name VolumeDriver -i volume/drivers/extpoint.go -o volume/drivers/proxy.go

Where:

--type is the name of the interface to use
--name is the subsystem that the plugin "Implements"
-i is the input file containing the interface definition
-o is the output file where the generated code should go

Note: The generated code will use the same package name as the one defined in the input file

Optionally, you can skip functions on the interface that should not be
implemented in the generated proxy code by passing in the function name to --skip.
This flag can be specified multiple times.

You can also add build tags that should be prepended to the generated code by
supplying --tag. This flag can be specified multiple times.

Annotations

pluginrpc-gen supports annotations to customize the behavior of the generated code. These annotations are added as comments directly above the interface methods.

Supported Annotations

pluginrpc-gen:timeout-type=<value>
- Specifies the timeout type to use for the method in the generated proxy code.
- The <value> can be:
  - short: Uses the shortTimeout constant (default: 1 minute).
  - long: Uses the longTimeout constant (default: 2 minutes).

Usage

To use the pluginrpc-gen:timeout-type annotation, place it directly above the method definition in the interface.
To use the timeout value annotations, place them directly above the interface definition.

Example

type volumeDriver interface {
    // pluginrpc-gen:timeout-type=long
    Create(name string, opts map[string]string) error

    // pluginrpc-gen:timeout-type=short
    Remove(name string) error
}

Default Behavior

If no pluginrpc-gen:timeout-type annotation is provided, the shortTimeout value is used by default.

Known issues

go-generate

You can also use this with go-generate, which is pretty awesome.
To do so, place the code at the top of the file which contains the interface
definition (i.e., the input file):

//go:generate pluginrpc-gen -i $GOFILE -o proxy.go -type volumeDriver -name VolumeDriver

Then cd to the package dir and run go generate

Note: the pluginrpc-gen binary must be within your $PATH

Overview (`vendor/cloud.google.com/go/README.md`)

Google Cloud Client Libraries for Go

Go packages for Google Cloud Platform services.

Installation

bash

go get cloud.google.com/go/firestore@latest # Replace firestore with the package you want to use.

NOTE: Some of these packages are under development, and may occasionally
make backwards-incompatible changes.

Supported APIs

For an updated list of all of our released APIs please see our
reference docs.

[Go Versions Supported](#supported-versions)

Our libraries are compatible with the two most recent major Go
releases, the same policy the Go
programming language follows. This means the currently supported versions are:

Go 1.24
Go 1.25

Authentication

By default, each client library will use Application Default Credentials
(ADC) to automatically configure the credentials used in calling the API endpoint.
When using the libraries in a Google Cloud Platform environment such as Compute
Engine, Kubernetes Engine, or App Engine, no additional authentication steps are
necessary. See Authentication methods at Google
and Authenticate for using client libraries
for more information.

client, err := storage.NewClient(ctx)

For applications running elsewhere, such as your local development environment,
you can use the gcloud auth application-default login command from the
Google Cloud CLI to set user credentials in
your local filesystem. Application Default Credentials will automatically detect
these credentials. See Set up ADC for a local development
environment
for more information.

Alternately, you may need to provide an explicit path to your credentials. To authenticate
using a service account
key file, either set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path
to your key file, or programmatically pass
option.WithCredentialsFile
to the NewClient function of the desired package. For example:

client, err := storage.NewClient(ctx, option.WithCredentialsFile("path/to/keyfile.json"))

You can exert even more control over authentication by using the
credentials package to
create an auth.Credentials.
Then pass option.WithAuthCredentials
to the NewClient function:

creds, err := credentials.DetectDefault(&credentials.DetectOptions{...})
...
client, err := storage.NewClient(ctx, option.WithAuthCredentials(creds))

Contributing

Contributions are welcome. Please, see the
CONTRIBUTING
document for details.

Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms.
See Contributor Code of Conduct
for more information.

Links

Overview (`vendor/code.cloudfoundry.org/clock/README.md`)

Clock

Provides a Clock interface, useful for injecting time dependencies in
tests.

[!NOTE]

This repository should be imported as code.cloudfoundry.org/clock.

Contributing

See the Contributing.md for more
information on how to contribute.

Working Group Charter

This repository is maintained by App Runtime
Platform
under Diego area.

[!IMPORTANT]

Content in this file is managed by the CI task
sync-readme
and is generated by CI following a convention.

Overview (`vendor/dario.cat/mergo/README.md`)

Mergo

A helper to merge structs and maps in Golang. Useful for configuration default values, avoiding messy if-statements.

Mergo merges same-type structs and maps by setting default values in zero-value fields. Mergo won't merge unexported (private) fields. It will do recursively any exported one. It also won't merge structs inside maps (because they are not addressable using Go reflection).

Also a lovely comune (municipality) in the Province of Ancona in the Italian region of Marche.

Status

Mergo is stable and frozen, ready for production. Check a short list of the projects using at large scale it here.

No new features are accepted. They will be considered for a future v2 that improves the implementation and fixes bugs for corner cases.

Important notes

1.0.0

In 1.0.0 Mergo moves to a vanity URL dario.cat/mergo. No more v1 versions will be released.

If the vanity URL is causing issues in your project due to a dependency pulling Mergo - it isn't a direct dependency in your project - it is recommended to use replace to pin the version to the last one with the old import URL:

text

replace github.com/imdario/mergo => github.com/imdario/mergo v0.3.16

0.3.9

Please keep in mind that a problematic PR broke 0.3.9. I reverted it in 0.3.10, and I consider it stable but not bug-free. Also, this version adds support for go modules.

Keep in mind that in 0.3.2, Mergo changed Merge()and Map() signatures to support transformers. I added an optional/variadic argument so that it won't break the existing code.

If you were using Mergo before April 6th, 2015, please check your project works as intended after updating your local copy with go get -u dario.cat/mergo. I apologize for any issue caused by its previous behavior and any future bug that Mergo could cause in existing projects after the change (release 0.2.0).

Donations

If Mergo is useful to you, consider buying me a coffee, a beer, or making a monthly donation to allow me to keep building great free software. :heart_eyes:

Mergo in the wild

Mergo is used by thousands of projects, including:

Install

text

go get dario.cat/mergo

// use in your .go code
import (
    "dario.cat/mergo"
)

Usage

You can only merge same-type structs with exported fields initialized as zero value of their type and same-types maps. Mergo won't merge unexported (private) fields but will do recursively any exported one. It won't merge empty structs value as they are zero values too. Also, maps will be merged recursively except for structs inside maps (because they are not addressable using Go reflection).

if err := mergo.Merge(&dst, src); err != nil {
    // ...
}

Also, you can merge overwriting values using the transformer WithOverride.

if err := mergo.Merge(&dst, src, mergo.WithOverride); err != nil {
    // ...
}

If you need to override pointers, so the source pointer's value is assigned to the destination's pointer, you must use WithoutDereference:

package main

import (
	"fmt"

	"dario.cat/mergo"
)

type Foo struct {
	A *string
	B int64
}

func main() {
	first := "first"
	second := "second"
	src := Foo{
		A: &first,
		B: 2,
	}

	dest := Foo{
		A: &second,
		B: 1,
	}

	mergo.Merge(&dest, src, mergo.WithOverride, mergo.WithoutDereference)
}

Additionally, you can map a map[string]interface{} to a struct (and otherwise, from struct to map), following the same restrictions as in Merge(). Keys are capitalized to find each corresponding exported field.

if err := mergo.Map(&dst, srcMap); err != nil {
    // ...
}

Warning: if you map a struct to map, it won't do it recursively. Don't expect Mergo to map struct members of your struct as map[string]interface{}. They will be just assigned as values.

Here is a nice example:

package main

import (
	"fmt"
	"dario.cat/mergo"
)

type Foo struct {
	A string
	B int64
}

func main() {
	src := Foo{
		A: "one",
		B: 2,
	}
	dest := Foo{
		A: "two",
	}
	mergo.Merge(&dest, src)
	fmt.Println(dest)
	// Will print
	// {two 2}
}

Transformers

Transformers allow to merge specific types differently than in the default behavior. In other words, now you can customize how some types are merged. For example, time.Time is a struct; it doesn't have zero value but IsZero can return true because it has fields with zero value. How can we merge a non-zero time.Time?

package main

import (
	"fmt"
	"dario.cat/mergo"
    "reflect"
    "time"
)

type timeTransformer struct {
}

func (t timeTransformer) Transformer(typ reflect.Type) func(dst, src reflect.Value) error {
	if typ == reflect.TypeOf(time.Time{}) {
		return func(dst, src reflect.Value) error {
			if dst.CanSet() {
				isZero := dst.MethodByName("IsZero")
				result := isZero.Call([]reflect.Value{})
				if result[0].Bool() {
					dst.Set(src)
				}
			}
			return nil
		}
	}
	return nil
}

type Snapshot struct {
	Time time.Time
	// ...
}

func main() {
	src := Snapshot{time.Now()}
	dest := Snapshot{}
	mergo.Merge(&dest, src, mergo.WithTransformers(timeTransformer{}))
	fmt.Println(dest)
	// Will print
	// { 2018-01-12 01:15:00 +0000 UTC m=+0.000000001 }
}

Contact me

If I can help you, you have an idea or you are using Mergo in your projects, don't hesitate to drop me a line (or a pull request): @im_dario

About

Written by Dario Castañé.

License

BSD 3-Clause license, as Go language.

Overview (`vendor/go.etcd.io/bbolt/README.md`)

bbolt

bbolt is a fork of Ben Johnson's Bolt key/value
store. The purpose of this fork is to provide the Go community with an active
maintenance and development target for Bolt; the goal is improved reliability
and stability. bbolt includes bug fixes, performance enhancements, and features
not found in Bolt while preserving backwards compatibility with the Bolt API.

Bolt is a pure Go key/value store inspired by Howard Chu's
LMDB project. The goal of the project is to provide a simple,
fast, and reliable database for projects that don't require a full database
server such as Postgres or MySQL.

Since Bolt is meant to be used as such a low-level piece of functionality,
simplicity is key. The API will be small and only focus on getting values
and setting values. That's it.

Project Status

Bolt is stable, the API is fixed, and the file format is fixed. Full unit
test coverage and randomized black box testing are used to ensure database
consistency and thread safety. Bolt is currently used in high-load production
environments serving databases as large as 1TB. Many companies such as
Shopify and Heroku use Bolt-backed services every day.

Project versioning

bbolt uses semantic versioning.
API should not change between patch and minor releases.
New minor versions may add additional features to the API.

Getting Started
Resources
Comparison with other databases
Caveats & Limitations
Reading the Source
Known Issues
Other Projects Using Bolt

Getting Started

Installing

To start using bbolt, install Go and run go get:

$ go get go.etcd.io/bbolt@latest

This will retrieve the library and update your go.mod and go.sum files.

To run the command line utility, execute:

$ go run go.etcd.io/bbolt/cmd/bbolt@latest

Run go install to install the bbolt command line utility into
your $GOBIN path, which defaults to $GOPATH/bin or $HOME/go/bin if the
GOPATH environment variable is not set.

$ go install go.etcd.io/bbolt/cmd/bbolt@latest

Importing bbolt

To use bbolt as an embedded key-value store, import as:

import bolt "go.etcd.io/bbolt"

db, err := bolt.Open(path, 0600, nil)
if err != nil {
  return err
}
defer db.Close()

Opening a database

The top-level object in Bolt is a DB. It is represented as a single file on
your disk and represents a consistent snapshot of your data.

To open your database, simply use the bolt.Open() function:

package main

import (
	"log"

	bolt "go.etcd.io/bbolt"
)

func main() {
	// Open the my.db data file in your current directory.
	// It will be created if it doesn't exist.
	db, err := bolt.Open("my.db", 0600, nil)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	...
}

Please note that Bolt obtains a file lock on the data file so multiple processes
cannot open the same database at the same time. Opening an already open Bolt
database will cause it to hang until the other process closes it. To prevent
an indefinite wait you can pass a timeout option to the Open() function:

db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second})

Transactions

Bolt allows only one read-write transaction at a time but allows as many
read-only transactions as you want at a time. Each transaction has a consistent
view of the data as it existed when the transaction started.

Individual transactions and all objects created from them (e.g. buckets, keys)
are not thread safe. To work with data in multiple goroutines you must start
a transaction for each one or use locking to ensure only one goroutine accesses
a transaction at a time. Creating transaction from the DB is thread safe.

Transactions should not depend on one another and generally shouldn't be opened
simultaneously in the same goroutine. This can cause a deadlock as the read-write
transaction needs to periodically re-map the data file but it cannot do so while
any read-only transaction is open. Even a nested read-only transaction can cause
a deadlock, as the child transaction can block the parent transaction from releasing
its resources.

Read-write transactions

To start a read-write transaction, you can use the DB.Update() function:

err := db.Update(func(tx *bolt.Tx) error {
	...
	return nil
})

Inside the closure, you have a consistent view of the database. You commit the
transaction by returning nil at the end. You can also rollback the transaction
at any point by returning an error. All database operations are allowed inside
a read-write transaction.

Always check the return error as it will report any disk failures that can cause
your transaction to not complete. If you return an error within your closure
it will be passed through.

Read-only transactions

To start a read-only transaction, you can use the DB.View() function:

err := db.View(func(tx *bolt.Tx) error {
	...
	return nil
})

You also get a consistent view of the database within this closure, however,
no mutating operations are allowed within a read-only transaction. You can only
retrieve buckets, retrieve values, and copy the database within a read-only
transaction.

Batch read-write transactions

Each DB.Update() waits for disk to commit the writes. This overhead
can be minimized by combining multiple updates with the DB.Batch()
function:

err := db.Batch(func(tx *bolt.Tx) error {
	...
	return nil
})

Concurrent Batch calls are opportunistically combined into larger
transactions. Batch is only useful when there are multiple goroutines
calling it.

The trade-off is that Batch can call the given
function multiple times, if parts of the transaction fail. The
function must be idempotent and side effects must take effect only
after a successful return from DB.Batch().

For example: don't display messages from inside the function, instead
set variables in the enclosing scope:

var id uint64
err := db.Batch(func(tx *bolt.Tx) error {
	// Find last key in bucket, decode as bigendian uint64, increment
	// by one, encode back to []byte, and add new key.
	...
	id = newValue
	return nil
})
if err != nil {
	return ...
}
fmt.Println("Allocated ID %d", id)

Managing transactions manually

The DB.View() and DB.Update() functions are wrappers around the DB.Begin()
function. These helper functions will start the transaction, execute a function,
and then safely close your transaction if an error is returned. This is the
recommended way to use Bolt transactions.

However, sometimes you may want to manually start and end your transactions.
You can use the DB.Begin() function directly but please be sure to close
the transaction.

// Start a writable transaction.
tx, err := db.Begin(true)
if err != nil {
    return err
}
defer tx.Rollback()

// Use the transaction...
_, err := tx.CreateBucket([]byte("MyBucket"))
if err != nil {
    return err
}

// Commit the transaction and check for error.
if err := tx.Commit(); err != nil {
    return err
}

The first argument to DB.Begin() is a boolean stating if the transaction
should be writable.

Using buckets

Buckets are collections of key/value pairs within the database. All keys in a
bucket must be unique. You can create a bucket using the Tx.CreateBucket()
function:

db.Update(func(tx *bolt.Tx) error {
	b, err := tx.CreateBucket([]byte("MyBucket"))
	if err != nil {
		return fmt.Errorf("create bucket: %s", err)
	}
	return nil
})

You can retrieve an existing bucket using the Tx.Bucket() function:

db.Update(func(tx *bolt.Tx) error {
	b := tx.Bucket([]byte("MyBucket"))
	if b == nil {
		return errors.New("bucket does not exist")
	}
	return nil
})

You can also create a bucket only if it doesn't exist by using the
Tx.CreateBucketIfNotExists() function. It's a common pattern to call this
function for all your top-level buckets after you open your database so you can
guarantee that they exist for future transactions.

To delete a bucket, simply call the Tx.DeleteBucket() function.

You can also iterate over all existing top-level buckets with Tx.ForEach():

db.View(func(tx *bolt.Tx) error {
	tx.ForEach(func(name []byte, b *bolt.Bucket) error {
		fmt.Println(string(name))
		return nil
	})
	return nil
})

Using key/value pairs

To save a key/value pair to a bucket, use the Bucket.Put() function:

db.Update(func(tx *bolt.Tx) error {
	b := tx.Bucket([]byte("MyBucket"))
	err := b.Put([]byte("answer"), []byte("42"))
	return err
})

This will set the value of the "answer" key to "42" in the MyBucket
bucket. To retrieve this value, we can use the Bucket.Get() function:

db.View(func(tx *bolt.Tx) error {
	b := tx.Bucket([]byte("MyBucket"))
	v := b.Get([]byte("answer"))
	fmt.Printf("The answer is: %s\n", v)
	return nil
})

The Get() function does not return an error because its operation is
guaranteed to work (unless there is some kind of system failure). If the key
exists then it will return its byte slice value. If it doesn't exist then it
will return nil. It's important to note that you can have a zero-length value
set to a key which is different than the key not existing.

Use the Bucket.Delete() function to delete a key from the bucket:

db.Update(func (tx *bolt.Tx) error {
    b := tx.Bucket([]byte("MyBucket"))
    err := b.Delete([]byte("answer"))
    return err
})

This will delete the key answers from the bucket MyBucket.

Please note that values returned from Get() are only valid while the
transaction is open. If you need to use a value outside of the transaction
then you must use copy() to copy it to another byte slice.

Autoincrementing integer for the bucket

By using the NextSequence() function, you can let Bolt determine a sequence
which can be used as the unique identifier for your key/value pairs. See the
example below.

// CreateUser saves u to the store. The new user ID is set on u once the data is persisted.
func (s *Store) CreateUser(u *User) error {
    return s.db.Update(func(tx *bolt.Tx) error {
        // Retrieve the users bucket.
        // This should be created when the DB is first opened.
        b := tx.Bucket([]byte("users"))

        // Generate ID for the user.
        // This returns an error only if the Tx is closed or not writeable.
        // That can't happen in an Update() call so I ignore the error check.
        id, _ := b.NextSequence()
        u.ID = int(id)

        // Marshal user data into bytes.
        buf, err := json.Marshal(u)
        if err != nil {
            return err
        }

        // Persist bytes to users bucket.
        return b.Put(itob(u.ID), buf)
    })
}

// itob returns an 8-byte big endian representation of v.
func itob(v int) []byte {
    b := make([]byte, 8)
    binary.BigEndian.PutUint64(b, uint64(v))
    return b
}

type User struct {
    ID int
    ...
}

Iterating over keys

Bolt stores its keys in byte-sorted order within a bucket. This makes sequential
iteration over these keys extremely fast. To iterate over keys we'll use a
Cursor:

db.View(func(tx *bolt.Tx) error {
	// Assume bucket exists and has keys
	b := tx.Bucket([]byte("MyBucket"))

	c := b.Cursor()

	for k, v := c.First(); k != nil; k, v = c.Next() {
		fmt.Printf("key=%s, value=%s\n", k, v)
	}

	return nil
})

The cursor allows you to move to a specific point in the list of keys and move
forward or backward through the keys one at a time.

The following functions are available on the cursor:

text

First()  Move to the first key.
Last()   Move to the last key.
Seek()   Move to a specific key.
Next()   Move to the next key.
Prev()   Move to the previous key.

Each of those functions has a return signature of (key []byte, value []byte).
You must seek to a position using First(), Last(), or Seek() before calling
Next() or Prev(). If you do not seek to a position then these functions will
return a nil key.

When you have iterated to the end of the cursor, then Next() will return a
nil key and the cursor still points to the last element if present. When you
have iterated to the beginning of the cursor, then Prev() will return a nil
key and the cursor still points to the first element if present.

If you remove key/value pairs during iteration, the cursor may automatically
move to the next position if present in current node each time removing a key.
When you call c.Next() after removing a key, it may skip one key/value pair.
Refer to pull/611 to get more detailed info.

During iteration, if the key is non-nil but the value is nil, that means
the key refers to a bucket rather than a value. Use Bucket.Bucket() to
access the sub-bucket.

Prefix scans

To iterate over a key prefix, you can combine Seek() and bytes.HasPrefix():

db.View(func(tx *bolt.Tx) error {
	// Assume bucket exists and has keys
	c := tx.Bucket([]byte("MyBucket")).Cursor()

	prefix := []byte("1234")
	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
		fmt.Printf("key=%s, value=%s\n", k, v)
	}

	return nil
})

Range scans

Another common use case is scanning over a range such as a time range. If you
use a sortable time encoding such as RFC3339 then you can query a specific
date range like this:

db.View(func(tx *bolt.Tx) error {
	// Assume our events bucket exists and has RFC3339 encoded time keys.
	c := tx.Bucket([]byte("Events")).Cursor()

	// Our time range spans the 90's decade.
	min := []byte("1990-01-01T00:00:00Z")
	max := []byte("2000-01-01T00:00:00Z")

	// Iterate over the 90's.
	for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() {
		fmt.Printf("%s: %s\n", k, v)
	}

	return nil
})

Note that, while RFC3339 is sortable, the Golang implementation of RFC3339Nano does not use a fixed number of digits after the decimal point and is therefore not sortable.

ForEach()

You can also use the function ForEach() if you know you'll be iterating over
all the keys in a bucket:

db.View(func(tx *bolt.Tx) error {
	// Assume bucket exists and has keys
	b := tx.Bucket([]byte("MyBucket"))

	b.ForEach(func(k, v []byte) error {
		fmt.Printf("key=%s, value=%s\n", k, v)
		return nil
	})
	return nil
})

Please note that keys and values in ForEach() are only valid while
the transaction is open. If you need to use a key or value outside of
the transaction, you must use copy() to copy it to another byte
slice.

Nested buckets

You can also store a bucket in a key to create nested buckets. The API is the
same as the bucket management API on the DB object:

func (*Bucket) CreateBucket(key []byte) (*Bucket, error)
func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error)
func (*Bucket) DeleteBucket(key []byte) error

Say you had a multi-tenant application where the root level bucket was the account bucket. Inside of this bucket was a sequence of accounts which themselves are buckets. And inside the sequence bucket you could have many buckets pertaining to the Account itself (Users, Notes, etc) isolating the information into logical groupings.

// createUser creates a new user in the given account.
func createUser(accountID int, u *User) error {
    // Start the transaction.
    tx, err := db.Begin(true)
    if err != nil {
        return err
    }
    defer tx.Rollback()

    // Retrieve the root bucket for the account.
    // Assume this has already been created when the account was set up.
    root := tx.Bucket([]byte(strconv.FormatUint(accountID, 10)))

    // Setup the users bucket.
    bkt, err := root.CreateBucketIfNotExists([]byte("USERS"))
    if err != nil {
        return err
    }

    // Generate an ID for the new user.
    userID, err := bkt.NextSequence()
    if err != nil {
        return err
    }
    u.ID = userID

    // Marshal and save the encoded user.
    if buf, err := json.Marshal(u); err != nil {
        return err
    } else if err := bkt.Put([]byte(strconv.FormatUint(u.ID, 10)), buf); err != nil {
        return err
    }

    // Commit the transaction.
    if err := tx.Commit(); err != nil {
        return err
    }

    return nil
}

Database backups

Bolt is a single file so it's easy to backup. You can use the Tx.WriteTo()
function to write a consistent view of the database to a writer. If you call
this from a read-only transaction, it will perform a hot backup and not block
your other database reads and writes.

By default, it will use a regular file handle which will utilize the operating
system's page cache. See the Tx
documentation for information about optimizing for larger-than-RAM datasets.

One common use case is to backup over HTTP so you can use tools like cURL to
do database backups:

func BackupHandleFunc(w http.ResponseWriter, req *http.Request) {
	err := db.View(func(tx *bolt.Tx) error {
		w.Header().Set("Content-Type", "application/octet-stream")
		w.Header().Set("Content-Disposition", `attachment; filename="my.db"`)
		w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size())))
		_, err := tx.WriteTo(w)
		return err
	})
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
	}
}

Then you can backup using this command:

$ curl http://localhost/backup > my.db

Or you can open your browser to http://localhost/backup and it will download
automatically.

If you want to backup to another file you can use the Tx.CopyFile() helper
function.

Statistics

The database keeps a running count of many of the internal operations it
performs so you can better understand what's going on. By grabbing a snapshot
of these stats at two points in time we can see what operations were performed
in that time range.

For example, we could start a goroutine to log stats every 10 seconds:

go func() {
	// Grab the initial stats.
	prev := db.Stats()

	for {
		// Wait for 10s.
		time.Sleep(10 * time.Second)

		// Grab the current stats and diff them.
		stats := db.Stats()
		diff := stats.Sub(&prev)

		// Encode stats to JSON and print to STDERR.
		json.NewEncoder(os.Stderr).Encode(diff)

		// Save stats for the next loop.
		prev = stats
	}
}()

It's also useful to pipe these stats to a service such as statsd for monitoring
or to provide an HTTP endpoint that will perform a fixed-length sample.

Read-Only Mode

Sometimes it is useful to create a shared, read-only Bolt database. To this,
set the Options.ReadOnly flag when opening your database. Read-only mode
uses a shared lock to allow multiple processes to read from the database but
it will block any processes from opening the database in read-write mode.

db, err := bolt.Open("my.db", 0600, &bolt.Options{ReadOnly: true})
if err != nil {
	log.Fatal(err)
}

Mobile Use (iOS/Android)

Bolt is able to run on mobile devices by leveraging the binding feature of the
gomobile tool. Create a struct that will
contain your database logic and a reference to a *bolt.DB with a initializing
constructor that takes in a filepath where the database file will be stored.
Neither Android nor iOS require extra permissions or cleanup from using this method.

func NewBoltDB(filepath string) *BoltDB {
	db, err := bolt.Open(filepath+"/demo.db", 0600, nil)
	if err != nil {
		log.Fatal(err)
	}

	return &BoltDB{db}
}

type BoltDB struct {
	db *bolt.DB
	...
}

func (b *BoltDB) Path() string {
	return b.db.Path()
}

func (b *BoltDB) Close() {
	b.db.Close()
}

Database logic should be defined as methods on this wrapper struct.

To initialize this struct from the native language (both platforms now sync
their local storage to the cloud. These snippets disable that functionality for the
database file):

Android

java

String path;
if (android.os.Build.VERSION.SDK_INT >=android.os.Build.VERSION_CODES.LOLLIPOP){
    path = getNoBackupFilesDir().getAbsolutePath();
} else{
    path = getFilesDir().getAbsolutePath();
}
Boltmobiledemo.BoltDB boltDB = Boltmobiledemo.NewBoltDB(path)

iOS

objc

- (void)demo {
    NSString* path = [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory,
                                                          NSUserDomainMask,
                                                          YES) objectAtIndex:0];
	GoBoltmobiledemoBoltDB * demo = GoBoltmobiledemoNewBoltDB(path);
	[self addSkipBackupAttributeToItemAtPath:demo.path];
	//Some DB Logic would go here
	[demo close];
}

- (BOOL)addSkipBackupAttributeToItemAtPath:(NSString *) filePathString
{
    NSURL* URL= [NSURL fileURLWithPath: filePathString];
    assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]);

    NSError *error = nil;
    BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES]
                                  forKey: NSURLIsExcludedFromBackupKey error: &error];
    if(!success){
        NSLog(@"Error excluding %@ from backup %@", [URL lastPathComponent], error);
    }
    return success;
}

Resources

For more information on getting started with Bolt, check out the following articles:

Intro to BoltDB: Painless Performant Persistence by Nate Finch.
Bolt -- an embedded key/value database for Go by Progville

Comparison with other databases

Postgres, MySQL, & other relational databases

Relational databases structure data into rows and are only accessible through
the use of SQL. This approach provides flexibility in how you store and query
your data but also incurs overhead in parsing and planning SQL statements. Bolt
accesses all data by a byte slice key. This makes Bolt fast to read and write
data by key but provides no built-in support for joining values together.

Most relational databases (with the exception of SQLite) are standalone servers
that run separately from your application. This gives your systems
flexibility to connect multiple application servers to a single database
server but also adds overhead in serializing and transporting data over the
network. Bolt runs as a library included in your application so all data access
has to go through your application's process. This brings data closer to your
application but limits multi-process access to the data.

LevelDB, RocksDB

LevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Bolt in that
they are libraries bundled into the application, however, their underlying
structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes
random writes by using a write ahead log and multi-tiered, sorted files called
SSTables. Bolt uses a B+tree internally and only a single file. Both approaches
have trade-offs.

If you require a high random write throughput (>10,000 w/sec) or you need to use
spinning disks then LevelDB could be a good choice. If your application is
read-heavy or does a lot of range scans then Bolt could be a good choice.

One other important consideration is that LevelDB does not have transactions.
It supports batch writing of key/values pairs and it supports read snapshots
but it will not give you the ability to do a compare-and-swap operation safely.
Bolt supports fully serializable ACID transactions.

LMDB

Bolt was originally a port of LMDB so it is architecturally similar. Both use
a B+tree, have ACID semantics with fully serializable transactions, and support
lock-free MVCC using a single writer and multiple readers.

The two projects have somewhat diverged. LMDB heavily focuses on raw performance
while Bolt has focused on simplicity and ease of use. For example, LMDB allows
several unsafe actions such as direct writes for the sake of performance. Bolt
opts to disallow actions which can leave the database in a corrupted state. The
only exception to this in Bolt is DB.NoSync.

There are also a few differences in API. LMDB requires a maximum mmap size when
opening an mdb_env whereas Bolt will handle incremental mmap resizing
automatically. LMDB overloads the getter and setter functions with multiple
flags whereas Bolt splits these specialized cases into their own functions.

Caveats & Limitations

It's important to pick the right tool for the job and Bolt is no exception.
Here are a few things to note when evaluating and using Bolt:

Bolt is good for read intensive workloads. Sequential write performance is
also fast but random writes can be slow. You can use DB.Batch() or add a
write-ahead log to help mitigate this issue.
Bolt uses a B+tree internally so there can be a lot of random page access.
SSDs provide a significant performance boost over spinning disks.
Try to avoid long running read transactions. Bolt uses copy-on-write so
old pages cannot be reclaimed while an old transaction is using them.
Byte slices returned from Bolt are only valid during a transaction. Once the
transaction has been committed or rolled back then the memory they point to
can be reused by a new page or can be unmapped from virtual memory and you'll
see an unexpected fault address panic when accessing it.
Bolt uses an exclusive write lock on the database file so it cannot be
shared by multiple processes.
Be careful when using Bucket.FillPercent. Setting a high fill percent for
buckets that have random inserts will cause your database to have very poor
page utilization.
Use larger buckets in general. Smaller buckets causes poor page utilization
once they become larger than the page size (typically 4KB).
Bulk loading a lot of random writes into a new bucket can be slow as the
page will not split until the transaction is committed. Randomly inserting
more than 100,000 key/value pairs into a single new bucket in a single
transaction is not advised.
Bolt uses a memory-mapped file so the underlying operating system handles the
caching of the data. Typically, the OS will cache as much of the file as it
can in memory and will release memory as needed to other processes. This means
that Bolt can show very high memory usage when working with large databases.
However, this is expected and the OS will release memory as needed. Bolt can
handle databases much larger than the available physical RAM, provided its
memory-map fits in the process virtual address space. It may be problematic
on 32-bits systems.
The data structures in the Bolt database are memory mapped so the data file
will be endian specific. This means that you cannot copy a Bolt file from a
little endian machine to a big endian machine and have it work. For most
users this is not a concern since most modern CPUs are little endian.
Because of the way pages are laid out on disk, Bolt cannot truncate data files
and return free pages back to the disk. Instead, Bolt maintains a free list
of unused pages within its data file. These free pages can be reused by later
transactions. This works well for many use cases as databases generally tend
to grow. However, it's important to note that deleting large chunks of data
will not allow you to reclaim that space on disk.

For more information on page allocation, see this comment.
Removing key/values pairs in a bucket during iteration on the bucket using
cursor may not work properly. Each time when removing a key/value pair, the
cursor may automatically move to the next position if present. When users
call c.Next() after removing a key, it may skip one key/value pair.
Refer to https://github.com/etcd-io/bbolt/pull/611 for more detailed info.
Bolt db can be corrupted during the initialization phase due to abrupt power failure.
- Please note: This issue can only be reproduced during the very first initialization phase, when there is
  no existing data in bolt database.
- In normal production environment, it is difficult to reproduce this. Once the database file has been initialized, it can no longer occur.
- Please refer to this issue for more details: https://github.com/etcd-io/etcd/issues/16596.

Reading the Source

Bolt is a relatively small code base (<5KLOC) for an embedded, serializable,
transactional key/value database so it can be a good starting point for people
interested in how databases work.

The best places to start are the main entry points into Bolt:

Open() - Initializes the reference to the database. It's responsible for
creating the database if it doesn't exist, obtaining an exclusive lock on the
file, reading the meta pages, & memory-mapping the file.
DB.Begin() - Starts a read-only or read-write transaction depending on the
value of the writable argument. This requires briefly obtaining the "meta"
lock to keep track of open transactions. Only one read-write transaction can
exist at a time so the "rwlock" is acquired during the life of a read-write
transaction.
Bucket.Put() - Writes a key/value pair into a bucket. After validating the
arguments, a cursor is used to traverse the B+tree to the page and position
where the key & value will be written. Once the position is found, the bucket
materializes the underlying page and the page's parent pages into memory as
"nodes". These nodes are where mutations occur during read-write transactions.
These changes get flushed to disk during commit.
Bucket.Get() - Retrieves a key/value pair from a bucket. This uses a cursor
to move to the page & position of a key/value pair. During a read-only
transaction, the key and value data is returned as a direct reference to the
underlying mmap file so there's no allocation overhead. For read-write
transactions, this data may reference the mmap file or one of the in-memory
node values.
Cursor - This object is simply for traversing the B+tree of on-disk pages
or in-memory nodes. It can seek to a specific key, move to the first or last
value, or it can move forward or backward. The cursor handles the movement up
and down the B+tree transparently to the end user.
Tx.Commit() - Converts the in-memory dirty nodes and the list of free pages
into pages to be written to disk. Writing to disk then occurs in two phases.
First, the dirty pages are written to disk and an fsync() occurs. Second, a
new meta page with an incremented transaction ID is written and another
fsync() occurs. This two phase write ensures that partially written data
pages are ignored in the event of a crash since the meta page pointing to them
is never written. Partially written meta pages are invalidated because they
are written with a checksum.

If you have additional notes that could be helpful for others, please submit
them via pull request.

Known Issues

bbolt might run into data corruption issue on Linux when the feature
ext4: fast commit, which was introduced in
linux kernel version v5.10, is enabled. The fixes to the issue are included in
stable LTS patchlevels 5.10.94+ and 5.15.17+ (ftruncate tracking), plus
5.15.27+ (ineligible-commit fallback). Linux 5.17 includes these fixes as
well, but 5.17 is not an LTS release. Please refer to links below,
Please also refer to the discussion in https://github.com/etcd-io/bbolt/issues/562.
Writing a value with a length of 0 will always result in reading back an empty []byte{} value.
Please refer to issues/726#issuecomment-2061694802.

Other Projects Using Bolt

Below is a list of public, open source projects that use Bolt:

Algernon - A HTTP/2 web server with built-in support for Lua. Uses BoltDB as the default database backend.
Bazil - A file system that lets your data reside where it is most convenient for it to reside.
bolter - Command-line app for viewing BoltDB file in your terminal.
boltcli - the redis-cli for boltdb with Lua script support.
BoltHold - An embeddable NoSQL store for Go types built on BoltDB
BoltStore - Session store using Bolt.
Boltdb Boilerplate - Boilerplate wrapper around bolt aiming to make simple calls one-liners.
BoltDbWeb - A web based GUI for BoltDB files.
BoltDB Viewer - A BoltDB Viewer Can run on Windows、Linux、Android system.
bleve - A pure Go search engine similar to ElasticSearch that uses Bolt as the default storage backend.
bstore - Database library storing Go values, with referential/unique/nonzero constraints, indices, automatic schema management with struct tags, and a query API.
btcwallet - A bitcoin wallet.
buckets - a bolt wrapper streamlining
simple tx and key scans.
Buildkit - concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
cayley - Cayley is an open-source graph database using Bolt as optional backend.
ChainStore - Simple key-value interface to a variety of storage engines organized as a chain of operations.
🌰 Chestnut - Chestnut is encrypted storage for Go.
Consul - Consul is service discovery and configuration made easy. Distributed, highly available, and datacenter-aware.
Containerd - An open and reliable container runtime
DVID - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb.
dcrwallet - A wallet for the Decred cryptocurrency.
drive - drive is an unofficial Google Drive command line client for *NIX operating systems.
event-shuttle - A Unix system service to collect and reliably deliver messages to Kafka.
Freehold - An open, secure, and lightweight platform for your files and data.
Go Report Card - Go code quality report cards as a (free and open source) service.
GoWebApp - A basic MVC web application in Go using BoltDB.
GoShort - GoShort is a URL shortener written in Golang and BoltDB for persistent key/value storage and for routing it's using high performent HTTPRouter.
gopherpit - A web service to manage Go remote import paths with custom domains
gokv - Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more)
goraphdb - A graph database provides Cypher query, fluent builder and management UI.
Gitchain - Decentralized, peer-to-peer Git repositories aka "Git meets Bitcoin".
InfluxDB - Scalable datastore for metrics, events, and real-time analytics.
ipLocator - A fast ip-geo-location-server using bolt with bloom filters.
ipxed - Web interface and api for ipxed.
Ironsmith - A simple, script-driven continuous integration (build - > test -> release) tool, with no external dependencies
Kala - Kala is a modern job scheduler optimized to run on a single node. It is persistent, JSON over HTTP API, ISO 8601 duration notation, and dependent jobs.
Key Value Access Language (KVAL) - A proposed grammar for key-value datastores offering a bbolt binding.
LedisDB - A high performance NoSQL, using Bolt as optional storage.
lru - Easy to use Bolt-backed Least-Recently-Used (LRU) read-through cache with chainable remote stores.
mbuckets - A Bolt wrapper that allows easy operations on multi level (nested) buckets.
MetricBase - Single-binary version of Graphite.
MuLiFS - Music Library Filesystem creates a filesystem to organise your music files.
NATS - NATS Streaming uses bbolt for message and metadata storage.
Portainer - A lightweight service delivery platform for containerized applications that can be used to manage Docker, Swarm, Kubernetes and ACI environments.
Prometheus Annotation Server - Annotation server for PromDash & Prometheus service monitoring system.
Rain - BitTorrent client and library.
reef-pi - reef-pi is an award winning, modular, DIY reef tank controller using easy to learn electronics based on a Raspberry Pi.
Request Baskets - A web service to collect arbitrary HTTP requests and inspect them via REST API or simple web UI, similar to RequestBin service
Seaweed File System - Highly scalable distributed key~file system with O(1) disk read.
stow - a persistence manager for objects
backed by boltdb.
Storm - Simple and powerful ORM for BoltDB.
SimpleBolt - A simple way to use BoltDB. Deals mainly with strings.
Skybox Analytics - A standalone funnel analysis tool for web analytics.
Scuttlebutt - Uses Bolt to store and process all Twitter mentions of GitHub projects.
tentacool - REST api server to manage system stuff (IP, DNS, Gateway...) on a linux server.
torrent - Full-featured BitTorrent client package and utilities in Go. BoltDB is a storage backend in development.
Wiki - A tiny wiki using Goji, BoltDB and Blackfriday.

If you are using Bolt in a project please send a pull request to add it to the list.

Overview (`vendor/go.opentelemetry.io/otel/README.md`)

OpenTelemetry-Go

OpenTelemetry-Go is the Go implementation of OpenTelemetry.
It provides a set of APIs to directly measure performance and behavior of your software and send this data to observability platforms.

Project Status

Signal	Status
Traces	Stable
Metrics	Stable
Logs	Stable

Progress and status specific to this repository is tracked in our
project boards
and
milestones.

Project versioning information and stability guarantees can be found in the
versioning documentation.

Compatibility

OpenTelemetry-Go ensures compatibility with the current supported versions of
the Go language:

Each major Go release is supported until there are two newer major releases.
For example, Go 1.5 was supported until the Go 1.7 release, and Go 1.6 was supported until the Go 1.8 release.

For versions of Go that are no longer supported upstream, opentelemetry-go will
stop ensuring compatibility with these versions in the following manner:

A minor release of opentelemetry-go will be made to add support for the new
supported release of Go.
The following minor release of opentelemetry-go will remove compatibility
testing for the oldest (now archived upstream) version of Go. This, and
future, releases of opentelemetry-go may include features only supported by
the currently supported versions of Go.

Currently, this project supports the following environments.

OS	Go Version	Architecture
Ubuntu	1.27	amd64
Ubuntu	1.26	amd64
Ubuntu	1.27	386
Ubuntu	1.26	386
Ubuntu	1.27	arm64
Ubuntu	1.26	arm64
macOS	1.27	amd64
macOS	1.26	amd64
macOS	1.27	arm64
macOS	1.26	arm64
Windows	1.27	amd64
Windows	1.26	amd64
Windows	1.27	386
Windows	1.26	386

While this project should work for other systems, no compatibility guarantees
are made for those systems currently.

Getting Started

You can find a getting started guide on opentelemetry.io.

OpenTelemetry's goal is to provide a single set of APIs to capture distributed
traces and metrics from your application and send them to an observability
platform. This project allows you to do just that for applications written in
Go. There are two steps to this process: instrument your application, and
configure an exporter.

Instrumentation

To start capturing distributed traces and metric events from your application
it first needs to be instrumented. The easiest way to do this is by using an
instrumentation library for your code. Be sure to check out the officially
supported instrumentation
libraries.

If you need to extend the telemetry an instrumentation library provides or want
to build your own instrumentation for your application directly you will need
to use the
Go otel
package. The examples
are a good way to see some practical uses of this process.

Export

Now that your application is instrumented to collect telemetry, it needs an
export pipeline to send that telemetry to an observability platform.

All officially supported exporters for the OpenTelemetry project are contained in the exporters directory.

Exporter	Logs	Metrics	Traces
OTLP	✓	✓	✓
Prometheus		✓
stdout	✓	✓	✓
Zipkin			✓

Contributing

See the contributing documentation.

Overview (`vendor/go.uber.org/multierr/README.md`)

multierr [![GoDoc][doc-img]][doc] [![Build Status][ci-img]][ci] [![Coverage Status][cov-img]][cov]

multierr allows combining one or more Go errors together.

Features

Idiomatic:
multierr follows best practices in Go, and keeps your code idiomatic.
- It keeps the underlying error type hidden,
  allowing you to deal in error values exclusively.
- It provides APIs to safely append into an error from a defer statement.
Performant:
multierr is optimized for performance:
- It avoids allocations where possible.
- It utilizes slice resizing semantics to optimize common cases
  like appending into the same error object from a loop.
Interoperable:
multierr interoperates with the Go standard library's error APIs seamlessly:
- The errors.Is and errors.As functions just work.
Lightweight:
multierr comes with virtually no dependencies.

Installation

bash

go get -u go.uber.org/multierr@latest

Status

Stable: No breaking changes will be made before 2.0.

Released under the MIT License.

Overview (`vendor/go.uber.org/zap/README.md`)

:zap: zap

Installation

go get -u go.uber.org/zap

Note that zap only supports the two most recent minor versions of Go.

Quick Start

In contexts where performance is nice, but not critical, use the
SugaredLogger. It's 4-10x faster than other structured logging
packages and includes both structured and printf-style APIs.

logger, _ := zap.NewProduction()
defer logger.Sync() // flushes buffer, if any
sugar := logger.Sugar()
sugar.Infow("failed to fetch URL",
  // Structured context as loosely typed key-value pairs.
  "url", url,
  "attempt", 3,
  "backoff", time.Second,
)
sugar.Infof("Failed to fetch URL: %s", url)

When performance and type safety are critical, use the Logger. It's even
faster than the SugaredLogger and allocates far less, but it only supports
structured logging.

logger, _ := zap.NewProduction()
defer logger.Sync()
logger.Info("failed to fetch URL",
  // Structured context as strongly typed Field values.
  zap.String("url", url),
  zap.Int("attempt", 3),
  zap.Duration("backoff", time.Second),
)

See the documentation and FAQ for more details.

Performance

For applications that log in the hot path, reflection-based serialization and
string formatting are prohibitively expensive — they're CPU-intensive
and make many small allocations. Put differently, using encoding/json and
fmt.Fprintf to log tons of interface{}s makes your application slow.

Zap takes a different approach. It includes a reflection-free, zero-allocation
JSON encoder, and the base Logger strives to avoid serialization overhead
and allocations wherever possible. By building the high-level SugaredLogger
on that foundation, zap lets users choose when they need to count every
allocation and when they'd prefer a more familiar, loosely typed API.

As measured by its own benchmarking suite, not only is zap more performant
than comparable structured logging packages — it's also faster than the
standard library. Like all benchmarks, take these with a grain of salt.¹

Log a message and 10 fields:

Package	Time	Time % to zap	Objects Allocated
:zap: zap	656 ns/op	+0%	5 allocs/op
:zap: zap (sugared)	935 ns/op	+43%	10 allocs/op
zerolog	380 ns/op	-42%	1 allocs/op
go-kit	2249 ns/op	+243%	57 allocs/op
slog (LogAttrs)	2479 ns/op	+278%	40 allocs/op
slog	2481 ns/op	+278%	42 allocs/op
apex/log	9591 ns/op	+1362%	63 allocs/op
log15	11393 ns/op	+1637%	75 allocs/op
logrus	11654 ns/op	+1677%	79 allocs/op

Log a message with a logger that already has 10 fields of context:

Package	Time	Time % to zap	Objects Allocated
:zap: zap	67 ns/op	+0%	0 allocs/op
:zap: zap (sugared)	84 ns/op	+25%	1 allocs/op
zerolog	35 ns/op	-48%	0 allocs/op
slog	193 ns/op	+188%	0 allocs/op
slog (LogAttrs)	200 ns/op	+199%	0 allocs/op
go-kit	2460 ns/op	+3572%	56 allocs/op
log15	9038 ns/op	+13390%	70 allocs/op
apex/log	9068 ns/op	+13434%	53 allocs/op
logrus	10521 ns/op	+15603%	68 allocs/op

Log a static string, without any context or printf-style templating:

Package	Time	Time % to zap	Objects Allocated
:zap: zap	63 ns/op	+0%	0 allocs/op
:zap: zap (sugared)	81 ns/op	+29%	1 allocs/op
zerolog	32 ns/op	-49%	0 allocs/op
standard library	124 ns/op	+97%	1 allocs/op
slog	196 ns/op	+211%	0 allocs/op
slog (LogAttrs)	200 ns/op	+217%	0 allocs/op
go-kit	213 ns/op	+238%	9 allocs/op
apex/log	771 ns/op	+1124%	5 allocs/op
logrus	1439 ns/op	+2184%	23 allocs/op
log15	2069 ns/op	+3184%	20 allocs/op

Development Status: Stable

All APIs are finalized, and no breaking changes will be made in the 1.x series
of releases. Users of semver-aware dependency management systems should pin
zap to ^1.

Contributing

We encourage and support an active, healthy community of contributors —
including you! Details are in the contribution guide and
the code of conduct. The zap maintainers keep an eye on
issues and pull requests, but you can also report any negative conduct to
[email protected]. That email list is a private, safe space; even the zap
maintainers don't have access, so don't hesitate to hold us to a high
standard.

Released under the MIT License.

¹ In particular, keep in mind that we may be
benchmarking against slightly older versions of other packages. Versions are
pinned in the benchmarks/go.mod file. ↩

Overview (`vendor/google.golang.org/grpc/README.md`)

gRPC-Go

The Go implementation of gRPC: A high performance, open source, general
RPC framework that puts mobile and HTTP/2 first. For more information see the
Go gRPC docs, or jump directly into the quick start.

Prerequisites

Go: any one of the two latest major releases.

Installation

Simply add the following import to your code, and then go [build|run|test]
will automatically fetch the necessary dependencies:

import "google.golang.org/grpc"

Note: If you are trying to access grpc-go from China, see the
FAQ below.

Learn more

Go gRPC docs, which include a quick start and API
reference among other resources
Low-level technical docs from this repository
Performance benchmark
Examples
Contribution guidelines

FAQ

I/O Timeout Errors

The golang.org domain may be blocked from some countries. go get usually
produces an error like the following when this happens:

console

$ go get -u google.golang.org/grpc
package google.golang.org/grpc: unrecognized import path "google.golang.org/grpc" (https fetch: Get https://google.golang.org/grpc?go-get=1: dial tcp 216.239.37.1:443: i/o timeout)

To build Go code, there are several options:

Set up a VPN and access google.golang.org through that.
With Go module support: it is possible to use the replace feature of go mod to create aliases for golang.org packages. In your project's directory:
sh
```
go mod edit -replace=google.golang.org/grpc=github.com/grpc/grpc-go@latest
go mod tidy
go mod vendor
go build -mod=vendor
```
Again, this will need to be done for all transitive dependencies hosted on
golang.org as well. For details, refer to golang/go issue
#28652.

Compiling error, undefined: grpc.SupportPackageIsVersion

Please update to the latest version of gRPC-Go using
go get google.golang.org/grpc.

How to turn on logging

The default logger is controlled by environment variables. Turn everything on
like this:

console

$ export GRPC_GO_LOG_VERBOSITY_LEVEL=99
$ export GRPC_GO_LOG_SEVERITY_LEVEL=info

The RPC failed with error `"code = Unavailable desc = transport is closing"`

This error means the connection the RPC is using was closed, and there are many
possible reasons, including:

mis-configured transport credentials, connection failed on handshaking
bytes disrupted, possibly by a proxy in between
server shutdown
Keepalive parameters caused connection shutdown, for example if you have
configured your server to terminate connections regularly to trigger DNS
lookups.
If this is the case, you may want to increase your
MaxConnectionAgeGrace,
to allow longer RPC calls to finish.

It can be tricky to debug this because the error happens on the client side but
the root cause of the connection being closed is on the server side. Turn on
logging on both client and server, and see if there are any transport
errors.

Overview (`vendor/pgregory.net/rapid/README.md`)

rapid [![PkgGoDev][godev-img]][godev] [![CI][ci-img]][ci]

Rapid is a Go library for property-based testing.

Rapid checks that properties you define hold for a large number
of automatically generated test cases. If a failure is found, rapid
automatically minimizes the failing test case before presenting it.

Features

Imperative Go API with type-safe data generation using generics
Data generation biased to explore "small" values and edge cases more thoroughly
Fully automatic minimization of failing test cases
Persistence and automatic re-running of minimized failing test cases
Support for state machine ("stateful" or "model-based") testing
No dependencies outside the Go standard library

If you like rapid but are looking for a Rust alternative, check out
chaos_theory.

Examples

Here is what a trivial test using rapid looks like (playground):

package rapid_test

import (
	"slices"
	"testing"

	"pgregory.net/rapid"
)

func TestSortStrings(t *testing.T) {
	rapid.Check(t, func(t *rapid.T) {
		s := rapid.SliceOf(rapid.String()).Draw(t, "s")
		slices.Sort(s)
		if !slices.IsSorted(s) {
			t.Fatalf("unsorted after sort: %v", s)
		}
	})
}

More complete examples:

ParseDate function test:
source code, playground
Queue state machine test:
source code, playground

Comparison

Rapid aims to bring to Go the power and convenience
Hypothesis brings to Python.

Compared to testing.F.Fuzz, rapid shines
in generating complex structured data, including state machine tests, but lacks
coverage-guided feedback and mutations. Note that with
MakeFuzz, any rapid test
can be used as a fuzz target for the standard fuzzer.

Compared to gopter, rapid
provides a much simpler API (queue test in rapid vs
gopter),
is much smarter about data generation and is able to minimize failing test cases
fully automatically, without any user code.

As for testing/quick, it lacks both
convenient data generation facilities and any form of test case minimization, which
are two main things to look for in a property-based testing library.

FAQ

What is property-based testing?

Suppose we've written arithmetic functions add, subtract and multiply
and want to test them. Traditional testing approach is example-based —
we come up with example inputs and outputs, and verify that the system behavior
matches the examples:

func TestArithmetic_Example(t *testing.T) {
	t.Run("add", func(t *testing.T) {
		examples := [][3]int{
			{0, 0, 0},
			{0, 1, 1},
			{2, 2, 4},
			// ...
		}
		for _, e := range examples {
			if add(e[0], e[1]) != e[2] {
				t.Fatalf("add(%v, %v) != %v", e[0], e[1], e[2])
			}
		}
	})
	t.Run("subtract", func(t *testing.T) { /* ... */ })
	t.Run("multiply", func(t *testing.T) { /* ... */ })
}

In comparison, with property-based testing we define higher-level properties
that should hold for arbitrary input. Each time we run a property-based test,
properties are checked on a new set of pseudo-random data:

func TestArithmetic_Property(t *testing.T) {
	rapid.Check(t, func(t *rapid.T) {
		var (
			a = rapid.Int().Draw(t, "a")
			b = rapid.Int().Draw(t, "b")
			c = rapid.Int().Draw(t, "c")
		)
		if add(a, 0) != a {
			t.Fatalf("add() does not have 0 as identity")
		}
		if add(a, b) != add(b, a) {
			t.Fatalf("add() is not commutative")
		}
		if add(a, add(b, c)) != add(add(a, b), c) {
			t.Fatalf("add() is not associative")
		}
		if multiply(a, add(b, c)) != add(multiply(a, b), multiply(a, c)) {
			t.Fatalf("multiply() is not distributive over add()")
		}
		// ...
	})
}

Property-based tests are more powerful and concise than example-based ones —
and are also much more fun to write. As an additional benefit, coming up with
general properties of the system often improves the design of the system itself.

What properties should I test?

As you've seen from the examples above, it depends on the system you are testing.
Usually a good place to start is to put yourself in the shoes of your user
and ask what are the properties the user will rely on (often unknowingly or
implicitly) when building on top of your system. That said, here are some
broadly applicable and often encountered properties to keep in mind:

function does not panic on valid input data
behavior of two algorithms or data structures is identical
all variants of the decode(encode(x)) == x roundtrip

How does rapid work?

At its core, rapid does a fairly simple thing: generates pseudo-random data
based on the specification you provide, and check properties that you define
on the generated data.

Checking is easy: you simply write if statements and call something like
t.Fatalf when things look wrong.

Generating is a bit more involved. When you construct a Generator, nothing
happens: Generator is just a specification of how to Draw the data you
want. When you call Draw, rapid will take some bytes from its internal
random bitstream, use them to construct the value based on the Generator
specification, and track how the random bytes used correspond to the value
(and its subparts). This knowledge about the structure of the values being
generated, as well as their relationship with the parts of the bitstream
allows rapid to intelligently and automatically minify any failure found.

What about fuzzing?

Property-based testing focuses on quick feedback loop: checking the properties
on a small but diverse set of pseudo-random inputs in a fractions of a second.

In comparison, fuzzing focuses on slow, often multi-day, brute force input
generation that maximizes the coverage.

Both approaches are useful. Property-based tests are used alongside regular
example-based tests during development, and fuzzing is used to search for edge
cases and security vulnerabilities. With
MakeFuzz, any rapid test
can be used as a fuzz target.

Usage

Just run go test as usual, it will pick up also all rapid tests.

There are a number of optional flags to influence rapid behavior, run
go test -args -h and look at the flags with the -rapid. prefix. You can
then pass such flags as usual. For example:

go test -rapid.checks=10_000

Every flag with the -rapid. prefix also has a matching RAPID_ environment
variable (replace the dot with an underscore and uppercase the name). For
instance, RAPID_CHECKS=10 go test sets the default for -rapid.checks=10,
and an explicit flag still overrides the environment default when both are
present.

Status

Rapid is stable: tests using rapid should continue to work with all future
rapid releases with the same major version. Possible exceptions to this rule
are API changes that replace the concrete type of parameter with an interface
type, or other similar mostly non-breaking changes.

License

Rapid is licensed under the Mozilla Public License Version 2.0.

Overview (`vendor/resenje.org/singleflight/README.md`)

Singleflight

Package singleflight provides a duplicate function call suppression
mechanism similar to golang.org/x/sync/singleflight but with:

support for context cancelation. The context passed to the callback function is a context that preserves all values
from the passed context but is cancelled by the singleflight only when all awaiting caller's contexts are cancelled.
support for generics.

Installation

Run go get resenje.org/singleflight from command line.

Overview (`vendor/sigs.k8s.io/json/README.md`)

sigs.k8s.io/json

Introduction

This library is a subproject of sig-api-machinery.
It provides case-sensitive, integer-preserving JSON unmarshaling functions based on encoding/json Unmarshal().

Compatibility

The UnmarshalCaseSensitivePreserveInts() function behaves like encoding/json#Unmarshal() with the following differences:

JSON object keys are treated case-sensitively.
Object keys must exactly match json tag names (for tagged struct fields)
or struct field names (for untagged struct fields).
JSON integers are unmarshaled into interface{} fields as an int64 instead of a
float64 when possible, falling back to float64 on any parse or overflow error.
Syntax errors do not return an encoding/json *SyntaxError error.
Instead, they return an error which can be passed to SyntaxErrorOffset() to obtain an offset.

Additional capabilities

The UnmarshalStrict() function decodes identically to UnmarshalCaseSensitivePreserveInts(),
and also returns non-fatal strict errors encountered while decoding:

Duplicate fields encountered
Unknown fields encountered

Community, discussion, contribution, and support

You can reach the maintainers of this project via the
sig-api-machinery mailing list / channels.

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

Overview (`vendor/sigs.k8s.io/yaml/README.md`)

YAML marshaling and unmarshaling support for Go

kubernetes-sigs/yaml is a permanent fork of ghodss/yaml.

Introduction

A wrapper around go-yaml designed to enable a better way of handling YAML when marshaling to and from structs.

In short, this library first converts YAML to JSON using go-yaml and then uses json.Marshal and json.Unmarshal to convert to or from the struct. This means that it effectively reuses the JSON struct tags as well as the custom JSON methods MarshalJSON and UnmarshalJSON unlike go-yaml. For a detailed overview of the rationale behind this method, see this blog post.

Compatibility

This package uses go-yaml and therefore supports everything go-yaml supports.

Caveats

Caveat #1: When using yaml.Marshal and yaml.Unmarshal, binary data should NOT be preceded with the !!binary YAML tag. If you do, go-yaml will convert the binary data from base64 to native binary data, which is not compatible with JSON. You can still use binary in your YAML files though - just store them without the !!binary tag and decode the base64 in your code (e.g. in the custom JSON methods MarshalJSON and UnmarshalJSON). This also has the benefit that your YAML and your JSON binary data will be decoded exactly the same way. As an example:

text

BAD:
	exampleKey: !!binary gIGC

GOOD:
	exampleKey: gIGC
... and decode the base64 data in your code.

Caveat #2: When using YAMLToJSON directly, maps with keys that are maps will result in an error since this is not supported by JSON. This error will occur in Unmarshal as well since you can't unmarshal map keys anyways since struct fields can't be keys.

Installation and usage

To install, run:

text

$ go get sigs.k8s.io/yaml

And import using:

text

import "sigs.k8s.io/yaml"

Usage is very similar to the JSON library:

package main

import (
	"fmt"

	"sigs.k8s.io/yaml"
)

type Person struct {
	Name string `json:"name"` // Affects YAML field names too.
	Age  int    `json:"age"`
}

func main() {
	// Marshal a Person struct to YAML.
	p := Person{"John", 30}
	y, err := yaml.Marshal(p)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(string(y))
	/* Output:
	age: 30
	name: John
	*/

	// Unmarshal the YAML back into a Person struct.
	var p2 Person
	err = yaml.Unmarshal(y, &p2)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(p2)
	/* Output:
	{John 30}
	*/
}

yaml.YAMLToJSON and yaml.JSONToYAML methods are also available:

package main

import (
	"fmt"

	"sigs.k8s.io/yaml"
)

func main() {
	j := []byte(`{"name": "John", "age": 30}`)
	y, err := yaml.JSONToYAML(j)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(string(y))
	/* Output:
	age: 30
	name: John
	*/
	j2, err := yaml.YAMLToJSON(y)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(string(j2))
	/* Output:
	{"age":30,"name":"John"}
	*/
}

Overview (`daemon/libnetwork/cmd/diagnostic/README.md`)

keywords: network, troubleshooting, debug title: Debug overlay or swarm networking issues

WARNING
This tool can change the internal state of the libnetwork API, be really mindful
on its use and read carefully the following guide. Improper use of it will damage
or permanently destroy the network configuration.

Docker CE 17.12 and higher introduce a network debugging tool designed to help
debug issues with overlay networks and swarm services running on Linux hosts.
When enabled, a network diagnostic server listens on the specified port and
provides diagnostic information. The network debugging tool should only be
started to debug specific issues, and should not be left running all the time.

Information about networks is stored in the database, which can be examined using
the API. Currently the database contains information about the overlay network
as well as the service discovery data.

The Docker API exposes endpoints to query and control the network debugging
tool. CLI integration is provided as a preview, but the implementation is not
yet considered stable and commands and options may change without notice.

The tool is available into 2 forms:

client only: dockereng/network-diagnostic:onlyclient
docker in docker version: dockereng/network-diagnostic:17.12-dind
The latter allows to use the tool with a cluster running an engine older than 17.12

Enable the diagnostic server

The tool currently only works on Docker hosts running on Linux. To enable it on a node
follow the step below.

Set the network-diagnostic-port to a port which is free on the Docker
host, in the /etc/docker/daemon.json configuration file.
json
```
“network-diagnostic-port”: <port>
```
Get the process ID (PID) of the dockerd process. It is the second field in
the output, and is typically a number from 2 to 6 digits long.
bash
```
$ ps aux |grep dockerd | grep -v grep
```
Reload the Docker configuration without restarting Docker, by sending the
HUP signal to the PID you found in the previous step.
bash
```
kill -HUP <pid-of-dockerd>
```

If systemd is used the command systemctl reload docker will be enough

A message like the following will appear in the Docker host logs:

none

Starting the diagnostic server listening on <port> for commands

Disable the diagnostic tool

Repeat these steps for each node participating in the swarm.

Remove the network-diagnostic-port key from the /etc/docker/daemon.json
configuration file.
Get the process ID (PID) of the dockerd process. It is the second field in
the output, and is typically a number from 2 to 6 digits long.
bash
```
$ ps aux |grep dockerd | grep -v grep
```
Reload the Docker configuration without restarting Docker, by sending the
HUP signal to the PID you found in the previous step.
bash
```
kill -HUP <pid-of-dockerd>
```

A message like the following will appear in the Docker host logs:

none

Disabling the diagnostic server

Access the diagnostic tool's API

The network diagnostic tool exposes its own RESTful API. To access the API,
send a HTTP request to the port where the tool is listening. The following
commands assume the tool is listening on port 2000.

Examples are not given for every endpoint.

Get help

bash

$ curl localhost:2000/help

OK
/updateentry
/getentry
/gettable
/leavenetwork
/createentry
/help
/clusterpeers
/ready
/joinnetwork
/deleteentry
/networkpeers
/
/join

Join or leave the network database cluster

bash

$ curl localhost:2000/join?members=ip1,ip2,...

bash

$ curl localhost:2000/leave?members=ip1,ip2,...

ip1, ip2, ... are the swarm node ips (usually one is enough)

Join or leave a network

bash

$ curl localhost:2000/joinnetwork?nid=<network id>

bash

$ curl localhost:2000/leavenetwork?nid=<network id>

network id can be retrieved on the manager with docker network ls --no-trunc and has
to be the full length identifier

List cluster peers

bash

$ curl localhost:2000/clusterpeers

List nodes connected to a given network

bash

$ curl localhost:2000/networkpeers?nid=<network id>

network id can be retrieved on the manager with docker network ls --no-trunc and has
to be the full length identifier

Dump database tables

The tables are called endpoint_table and overlay_peer_table.
The overlay_peer_table contains all the overlay forwarding information
The endpoint_table contains all the service discovery information

bash

$ curl localhost:2000/gettable?nid=<network id>&tname=<table name>

Interact with a specific database table

The tables are called endpoint_table and overlay_peer_table.

bash

$ curl localhost:2000/<method>?nid=<network id>&tname=<table name>&key=<key>[&value=<value>]

Note:
operations on tables have node ownership, this means that are going to remain persistent till
the node that inserted them is part of the cluster

Access the diagnostic tool's CLI

The CLI is provided as a preview and is not yet stable. Commands or options may
change at any time.

The CLI executable is called diagnosticClient and is made available using a
standalone container.

docker run --net host dockereng/network-diagnostic:onlyclient -v -net <full network id> -t sd

The following flags are supported:

Flag	Description
-t	Table one of `sd` or `overlay`.
-ip	The IP address to query. Defaults to 127.0.0.1.
-net	The target network ID.
-port	The target port. (default port is 2000)
-a	Join/leave network
-v	Enable verbose output.

NOTE
By default the tool won't try to join the network. This is following the intent to not change
the state on which the node is when the diagnostic client is run. This means that it is safe
to run the diagnosticClient against a running daemon because it will just dump the current state.
When using instead the diagnosticClient in the containerized version the flag -a MUST be passed
to avoid retrieving empty results. On the other side using the -a flag against a loaded daemon
will have the undesirable side effect to leave the network and so cutting down the data path for
that daemon.

Container version of the diagnostic tool

The CLI is provided as a container with a 17.12 engine that needs to run using privileged mode.
NOTE
Remember that table operations have ownership, so any create entry will be persistent till
the diagnostic container is part of the swarm.

Make sure that the node where the diagnostic client will run is not part of the swarm, if so do docker swarm leave -f

To run the container, use a command like the following:

bash

$ docker container run --name net-diagnostic -d --privileged --network host dockereng/network-diagnostic:17.12-dind

Connect to the container using docker exec -it <container-ID> sh,
and start the server using the following command:
bash
```
$ kill -HUP 1
```
Join the diagnostic container to the swarm, then run the diagnostic CLI within the container.
bash
```
$ ./diagnosticClient <flags>...
```
When finished debugging, leave the swarm and stop the container.

Examples

The following commands dump the service discovery table and verify node
ownership.

NOTE
Remember to use the full network ID, you can easily find that with docker network ls --no-trunc

Service discovery and load balancer:

bash

$ diagnostiClient -t sd -v -net n8a8ie6tb3wr2e260vxj8ncy4 -a

Overlay network:

bash

$ diagnostiClient -port 2001 -t overlay -v -net n8a8ie6tb3wr2e260vxj8ncy4 -a

Overview (`daemon/libnetwork/cmd/ssd/README.md`)

Docker Swarm Service Driller(ssd)

ssd is a troubleshooting utility for Docker swarm networks.

control-plane and datapath consistency check on a node

ssd checks for the consistency between docker network control-plane (from the docker daemon in-memory state) and kernel data path programming. Currently the tool checks only for the consistency of the Load balancer (implemented using IPVS).

In a three node swarm cluster ssd status for a overlay network ov2 which has three services running, each replicated to 3 instances.

bash

vagrant@net-1:~/code/go/src/github.com/docker/docker-e2e/tests$ docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/run/docker/netns:/var/run/docker/netns --privileged --net=host sanimej/ssd ov2
Verifying LB programming for containers on network ov2
Verifying container /s2.3.ltrdwef0iqf90rqauw3ehcs56...
service s2... OK
service s3... OK
service s1... OK
Verifying container /s3.3.nyhwvdvnocb4wftyhb8dr4fj8...
service s2... OK
service s3... OK
service s1... OK
Verifying container /s1.3.wwx5tuxhnvoz5vrb8ohphby0r...
service s2... OK
service s3... OK
service s1... OK
Verifying LB programming for containers on network ingress
Verifying container Ingress...
service web... OK

ssd checks the required iptables programming to direct an incoming packet with the : to the right :

control-plane consistency check across nodes in a cluster

Docker networking uses a gossip protocol to synchronize networking state across nodes in a cluster. ssd's gossip-consistency command verifies if the state maintained by all the nodes are consistent.

bash

In a three node cluster with services running on an overlay network ov2 ssd consistency-checker shows 

vagrant@net-1:~/code/go/src/github.com/docker/docker-e2e/tests$ docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/run/docker/netns:/var/run/docker/netns --privileged sanimej/ssd ov2 gossip-consistency
Node id: sjfp0ca8f43rvnab6v7f21gq0 gossip hash c57d89094dbb574a37930393278dc282

Node id: bg228r3q9095grj4wxkqs80oe gossip hash c57d89094dbb574a37930393278dc282

Node id: 6jylcraipcv2pxdricqe77j5q gossip hash c57d89094dbb574a37930393278dc282

This is hash digest of the control-plane state for the network ov2 from all the cluster nodes. If the values have a mismatch docker network inspect --verbose on the individual nodes can help in identifying what the specific difference is.

Overview (`extpoints/storage/kv/v0/README.md`)

Persistent record storage

Store opaque key/value records without depending on host-local paths.
Declare storagekv.Point.Dependency() and resolve a namespace during extension initialization:

kv, err := storagekv.GetKV(resolver, "org.my.extension")
if err != nil {
    return err
}

Use the namespace-bound helper for record operations:

err = kv.Create(ctx, "jobs/123", data)
data, err = kv.Get(ctx, "jobs/123")
err = kv.Update(ctx, "jobs/123", updatedData)
err = kv.Delete(ctx, "jobs/123")
err = kv.DeletePrefix(ctx, "jobs/")
page, err := kv.List(ctx, storagekv.ListOptions{Prefix: "jobs/"})

Record operations

Keys are logical strings, not paths; consumers own value encoding and schema versions.
Create requires an absent key; Update requires an existing key; Delete tolerates absence.
DeletePrefix atomically and durably removes all matching records; a nonempty prefix is required, and no matches (including a missing namespace) succeeds.
Individual writes are durable and atomic per record.
DeletePrefix is the only multi-record atomic operation; there is no concurrent-update protection.
Listing is paginated, not a snapshot; continue with NextCursor using the same prefix.
The helper needs no Close; records survive provider restarts and extension removal.

Executable consumers register the generated protogen.ClientPoint with their SDK client-point registry.
See storage.go for the full contract and size limits.

Namespace and trust model

Namespaces partition records for cooperative use; they are not an authorization boundary.
All trusted installed extensions that can reach this provider can access any namespace.
v0 performs no identity-based authorization, and namespace names do not authenticate ownership.
Any future identity enforcement belongs in host-resolved identity and per-caller storage wiring.
Identity enforcement must still allow explicit namespace sharing.

Persistence and lifecycle

Records are not purged automatically when an extension is absent or uninstalled.
Each extension owns cleanup of its individual records.
v0 stores no namespace ownership metadata and defines no garbage-collection policy; policy is deferred to v1.
Do not delete the shared database to remove one extension's data.
Corruption of the shared database affects all namespaces, and storage provides no automatic repair or replacement.
Storage does not automatically compact the database or reclaim file space.

Overview (`vendor/cloud.google.com/go/auth/README.md`)

Google Auth Library for Go

Install

bash

go get cloud.google.com/go/auth@latest

Usage

The most common way this library is used is transitively, by default, from any
of our Go client libraries.

Notable use-cases

To create a credential directly please see examples in the
credentials
package.
To create a authenticated HTTP client please see examples in the
httptransport
package.
To create a authenticated gRPC connection please see examples in the
grpctransport
package.
To create an ID token please see examples in the
idtoken
package.

Contributing

Contributions are welcome. Please, see the
CONTRIBUTING
document for details.

Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms.
See Contributor Code of Conduct
for more information.

Overview (`vendor/cloud.google.com/go/logging/README.md`)

Cloud Logging API

Go Client Library for Cloud Logging API.

Install

bash

go get cloud.google.com/go/logging

Stability

The stability of this module is indicated by SemVer.

However, a v1+ module may have breaking changes in two scenarios:

Packages with alpha or beta in the import path
The GoDoc has an explicit stability disclaimer (for example, for an experimental feature).

Which package to use?

Generated client library surfaces can be found in packages whose import path
ends in .../apivXXX. The XXX could be something like 1 or 2 in the case
of a stable service backend or may be like 1beta2 or 2beta in the case of a
more experimental service backend. Because of this fact, a given module can have
multiple clients for different service backends. In these cases it is generally
recommended to use clients with stable service backends, with import suffixes like
apiv1, unless you need to use features that are only present in a beta backend
or there is not yet a stable backend available.

Google Cloud Samples

To browse ready to use code samples check Google Cloud Samples.

Go Version Support

See the Go Versions Supported
section in the root directory's README.

Authorization

See the Authorization
section in the root directory's README.

Contributing

Contributions are welcome. Please, see the CONTRIBUTING
document for details.

Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms. See
Contributor Code of Conduct
for more information.

Overview (`vendor/cloud.google.com/go/longrunning/README.md`)

Long Running Operations API

Go Client Library for Long Running Operations API.

Install

bash

go get cloud.google.com/go/longrunning

Stability

The stability of this module is indicated by SemVer.

However, a v1+ module may have breaking changes in two scenarios:

Packages with alpha or beta in the import path
The GoDoc has an explicit stability disclaimer (for example, for an experimental feature).

Which package to use?

Google Cloud Samples

To browse ready to use code samples check Google Cloud Samples.

Go Version Support

See the Go Versions Supported
section in the root directory's README.

Authorization

See the Authorization
section in the root directory's README.

Contributing

Contributions are welcome. Please, see the CONTRIBUTING
document for details.

Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms. See
Contributor Code of Conduct
for more information.

Overview (`vendor/github.com/Azure/go-ansiterm/README.md`)

go-ansiterm

This is a cross platform Ansi Terminal Emulation library. It reads a stream of Ansi characters and produces the appropriate function calls. The results of the function calls are platform dependent.

For example the parser might receive "ESC, [, A" as a stream of three characters. This is the code for Cursor Up (http://www.vt100.net/docs/vt510-rm/CUU). The parser then calls the cursor up function (CUU()) on an event handler. The event handler determines what platform specific work must be done to cause the cursor to move up one position.

The parser (parser.go) is a partial implementation of this state machine (http://vt100.net/emu/vt500_parser.png). There are also two event handler implementations, one for tests (test_event_handler.go) to validate that the expected events are being produced and called, the other is a Windows implementation (winterm/win_event_handler.go).

See parser_test.go for examples exercising the state machine and generating appropriate function calls.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Overview (`vendor/github.com/Microsoft/go-winio/README.md`)

go-winio [](https://github.com/microsoft/go-winio/actions/workflows/ci.yml)

This repository contains utilities for efficiently performing Win32 IO operations in
Go. Currently, this is focused on accessing named pipes and other file handles, and
for using named pipes as a net transport.

This code relies on IO completion ports to avoid blocking IO on system threads, allowing Go
to reuse the thread to schedule another goroutine. This limits support to Windows Vista and
newer operating systems. This is similar to the implementation of network sockets in Go's net
package.

Please see the LICENSE file for licensing information.

Contributing

This project welcomes contributions and suggestions.
Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that
you have the right to, and actually do, grant us the rights to use your contribution.
For details, visit Microsoft CLA.

When you submit a pull request, a CLA-bot will automatically determine whether you need to
provide a CLA and decorate the PR appropriately (e.g., label, comment).
Simply follow the instructions provided by the bot.
You will only need to do this once across all repos using our CLA.

Additionally, the pull request pipeline requires the following steps to be performed before
mergining.

Code Sign-Off

We require that contributors sign their commits using git commit --signoff
to certify they either authored the work themselves or otherwise have permission to use it in this project.

A range of commits can be signed off using git rebase --signoff.

Please see the developer certificate for more info,
as well as to make sure that you can attest to the rules listed.
Our CI uses the DCO Github app to ensure that all commits in a given PR are signed-off.

Linting

Code must pass a linting stage, which uses golangci-lint.
The linting settings are stored in .golangci.yaml, and can be run
automatically with VSCode by adding the following to your workspace or folder settings:

json

"go.lintTool": "golangci-lint",
    "go.lintOnSave": "package",

Additional editor integrations options are also available.

Alternatively, golangci-lint can be installed locally and run from the repo root:

shell

# use . or specify a path to only lint a package
# to show all lint errors, use flags "--max-issues-per-linter=0 --max-same-issues=0"
> golangci-lint run ./...

Go Generate

The pipeline checks that auto-generated code, via go generate, are up to date.

This can be done for the entire repo:

shell

> go generate ./...

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct.
For more information see the Code of Conduct FAQ or
contact [email protected] with any additional questions or comments.

Special Thanks

Thanks to natefinch for the inspiration for this library.
See npipe for another named pipe implementation.

Overview (`vendor/github.com/Microsoft/hcsshim/README.md`)

hcsshim

This package contains the Golang interface for using the Windows Host Compute Service (HCS) to launch and manage Windows Containers. It also contains other helpers and functions for managing Windows Containers such as the Golang interface for the Host Network Service (HNS), as well as code for the guest agent (commonly referred to as the GCS or Guest Compute Service in the codebase) used to support running Linux Hyper-V containers.

It is primarily used in the Moby and Containerd projects, but it can be freely used by other projects as well.

Building

While this repository can be used as a library of sorts to call the HCS apis, there are a couple binaries built out of the repository as well. The main ones being the Linux guest agent, and an implementation of the runtime v2 containerd shim api.

Linux Hyper-V Container Guest Agent

To build the Linux guest agent itself all that's needed is to set your GOOS to "Linux" and build out of ./cmd/gcs.

powershell

C:\> $env:GOOS="linux"
C:\> go build .\cmd\gcs\

or on a Linux machine

> go build ./cmd/gcs

If you want it to be packaged inside of a rootfs to boot with alongside all of the other tools then you'll need to provide a rootfs that it can be packaged inside of. An easy way is to export the rootfs of a container.

docker pull busybox
docker run --name base_image_container busybox
docker export base_image_container | gzip > base.tar.gz
BASE=./base.tar.gz
make all

If the build is successful, in the ./out folder you should see:

> ls ./out/
delta.tar.gz  initrd.img  rootfs.tar.gz

Containerd Shim

For info on the Runtime V2 API.

Contrary to the typical Linux architecture of shim -> runc, the runhcs shim is used both to launch and manage the lifetime of containers.

powershell

C:\> $env:GOOS="windows"
C:\> go build .\cmd\containerd-shim-runhcs-v1

Then place the binary in the same directory that Containerd is located at in your environment.
A default Containerd configuration file can be generated by running:

powershell

.\containerd.exe config default | Out-File "C:\Program Files\containerd\config.toml" -Encoding ascii

This config file will already have the shim set as the default runtime for cri interactions.

To trial using the shim out with ctr.exe:

powershell

C:\> ctr.exe run --runtime io.containerd.runhcs.v1 --rm mcr.microsoft.com/windows/nanoserver:2004 windows-test cmd /c "echo Hello World!"

Containerd Shim V2

The V2 shims are the rewrite of the Windows containerd shim. The V1 shim
(containerd-shim-runhcs-v1) is a single, monolithic
binary that handles LCOW (Linux Containers on Windows), Hyper-V WCOW (Windows Containers on Windows), process-isolated
WCOW and host-process containers. In the V2 model that monolith is split into focused,
per-platform shims, each backed 1:1 by a sandbox.

V2 shims are used with containerd in the same way as the V1 shim, but the
API surface they expose is different. The V1 shim implemented only the containerd
Task API,
and used it to manage both the sandbox lifecycle and the container/process (task)
lifecycle through a single service. Each V2 shim instead splits these responsibilities
across the two APIs that containerd now provides for this purpose: the
Sandbox API
is used to manage the sandbox, while the
Task API
is used to manage containers and processes running inside it. Internally each V2 shim
implements these as separate sandbox and task services, alongside an auxiliary
shimdiag service used for diagnostics.

All three shims honor the same CRI pod model. A task annotated with
"io.kubernetes.cri.container-type": "sandbox" is treated as the pause/infra container
that creates the pod; sibling workload tasks set "io.kubernetes.cri.container-type": "container" and reference their pause via "io.kubernetes.cri.sandbox-id". What a
"sandbox" physically corresponds to depends on the shim, and is described in each
subsection below.

containerd-shim-lcow-v2

Purpose: Runs Linux Containers on Windows (LCOW) — a Linux utility VM hosting Linux
containers.
Sandbox: The Linux UVM. Each shim instance is backed 1:1 by a single UVM. This shim supports running
multiple pods in the same UVM, so a single shim instance may host more than one
CRI pods.
Tasks: Linux containers and processes running inside the UVM, identified via the
same CRI annotations described above.
Implementation: ./cmd/containerd-shim-lcow-v2.
Build Tag: lcow
Platform requirement: Windows Server 2025 (build 26100) or later.

containerd-shim-wcow-v2

Purpose: Runs Hyper-V isolated Windows containers (WCOW) — a Windows utility VM hosting
Process and/or Host Process Containers inside it.
Sandbox: The Windows utility VM (UVM). Each shim instance is backed 1:1 by a single UVM.
Tasks: the Windows containers and processes running inside the UVM, identified
via the standard CRI annotations described above.
Implementation: coming soon.
Build Tag: wcow

containerd-shim-process-v2

Purpose: Runs Process-isolated Windows Server containers and Host
Process Containers — workloads that execute directly on the host with no utility VM.
Sandbox: A pause container. The pause container is a minimal, long-lived
container that owns the pod's shared resources (such as the network namespace) and
keeps them alive while sibling workload containers are started, stopped or replaced.
This is the standard Kubernetes pod model: the pause container is the sandbox that
the rest of the pod attaches to.
Tasks: the actual workload containers belonging to the pod, linked back to the
pause via the io.kubernetes.cri.sandbox-id annotation.
Implementation: coming soon.
Build Tag: process

Building

The V2 shim sources are guarded by the build tags as mentioned above, so the tag must be passed
to go build. For example the LCOW shim has lcow tag-

powershell

C:\> $env:GOOS="windows"
C:\> go build -tags lcow .\cmd\containerd-shim-lcow-v2

Place the resulting containerd-shim-lcow-v2.exe in the same directory as containerd.exe,
the same as for the V1 shim.

Running unit tests

The shim's unit tests (and the rest of the tagged packages) are run with the
shim specific build tag:

powershell

C:\> go test -tags lcow ./...

Running parity tests

The repository ships parity tests under ./test/parity that feed
identical inputs through the legacy V1 and the new V2 pipelines and assert that the
resulting HCS ComputeSystem documents are equivalent. They live in the test Go
module and are also built with the build tag:

powershell

C:\> cd test
C:\> go test -tags lcow ./parity/...

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit Microsoft CLA.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

We require that contributors sign their commits
to certify they either authored the work themselves or otherwise have permission to use it in this project.

We also require that contributors sign their commits using using git commit --signoff
to certify they either authored the work themselves or otherwise have permission to use it in this project.
A range of commits can be signed off using git rebase --signoff.

Linting

Code must pass a linting stage, which uses golangci-lint.
Since ./test is a separate Go module, the linter is run from both the root and the
test directories. Additionally, the linter is run with GOOS set to both windows and
linux.

The linting settings are stored in .golangci.yaml, and can be run
automatically with VSCode by adding the following to your workspace or folder settings:

json

"go.lintTool": "golangci-lint",
    "go.lintOnSave": "package",

Additional editor integrations options are also available.

Alternatively, golangci-lint can be installed and run locally:

shell

# use . or specify a path to only lint a package
# to show all lint errors, use flags "--max-issues-per-linter=0 --max-same-issues=0"
> golangci-lint run

To run across the entire repo for both GOOS=windows and linux:

powershell

> foreach ( $goos in ('windows', 'linux') ) {
    foreach ( $repo in ('.', 'test') ) {
        pwsh -Command "cd $repo && go env -w GOOS=$goos && golangci-lint.exe run --verbose"
    }
}

Go Generate

The pipeline checks that auto-generated code, via go generate, are up to date.
Similar to the linting stage, go generate is run in both the root and test Go modules.

This can be done via:

shell

> go generate ./...
> cd test && go generate ./...

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct.
For more information see the Code of Conduct FAQ or
contact [email protected] with any additional questions or comments.

Dependencies

This project requires Golang 1.18 or newer to build.

For system requirements to run this project, see the Microsoft docs on Windows Container requirements.

Reporting Security Issues

Security issues and bugs should be reported privately, via email, to the Microsoft Security
Response Center (MSRC) at [email protected]. You should
receive a response within 24 hours. If for some reason you do not, please follow up via
email to ensure we received your original message. Further information, including the
MSRC PGP key, can be found in
the Security TechCenter.

For additional details, see Report a Computer Security Vulnerability on Technet

Overview (`vendor/github.com/RackSec/srslog/README.md`)

srslog

Go has a syslog package in the standard library, but it has the following
shortcomings:

It doesn't have TLS support
According to bradfitz on the Go team, it is no longer being maintained.

I agree that it doesn't need to be in the standard library. So, I've
followed Brad's suggestion and have made a separate project to handle syslog.

This code was taken directly from the Go project as a base to start from.

However, this does have TLS support.

Usage

Basic usage retains the same interface as the original syslog package. We
only added to the interface where required to support new functionality.

Switch from the standard library:

text

import(
    //"log/syslog"
    syslog "github.com/RackSec/srslog"
)

You can still use it for local syslog:

text

w, err := syslog.Dial("", "", syslog.LOG_ERR, "testtag")

Or to unencrypted UDP:

text

w, err := syslog.Dial("udp", "192.168.0.50:514", syslog.LOG_ERR, "testtag")

Or to unencrypted TCP:

text

w, err := syslog.Dial("tcp", "192.168.0.51:514", syslog.LOG_ERR, "testtag")

But now you can also send messages via TLS-encrypted TCP:

text

w, err := syslog.DialWithTLSCertPath("tcp+tls", "192.168.0.52:514", syslog.LOG_ERR, "testtag", "/path/to/servercert.pem")

And if you need more control over your TLS configuration :

text

pool := x509.NewCertPool()
serverCert, err := ioutil.ReadFile("/path/to/servercert.pem")
if err != nil {
    return nil, err
}
pool.AppendCertsFromPEM(serverCert)
config := tls.Config{
    RootCAs: pool,
}

w, err := DialWithTLSConfig(network, raddr, priority, tag, &config)

(Note that in both TLS cases, this uses a self-signed certificate, where the
remote syslog server has the keypair and the client has only the public key.)

And then to write log messages, continue like so:

text

if err != nil {
    log.Fatal("failed to connect to syslog:", err)
}
defer w.Close()

w.Alert("this is an alert")
w.Crit("this is critical")
w.Err("this is an error")
w.Warning("this is a warning")
w.Notice("this is a notice")
w.Info("this is info")
w.Debug("this is debug")
w.Write([]byte("these are some bytes"))

If you need further control over connection attempts, you can use the DialWithCustomDialer
function. To continue with the DialWithTLSConfig example:

text

netDialer := &net.Dialer{Timeout: time.Second*5} // easy timeouts
realNetwork := "tcp" // real network, other vars your dail func can close over
dial := func(network, addr string) (net.Conn, error) {
    // cannot use "network" here as it'll simply be "custom" which will fail
    return tls.DialWithDialer(netDialer, realNetwork, addr, &config)
}

w, err := DialWithCustomDialer("custom", "192.168.0.52:514", syslog.LOG_ERR, "testtag", dial)

Your custom dial func can set timeouts, proxy connections, and do whatever else it needs before returning a net.Conn.

Generating TLS Certificates

We've provided a script that you can use to generate a self-signed keypair:

text

pip install cryptography
python script/gen-certs.py

That outputs the public key and private key to standard out. Put those into
.pem files. (And don't put them into any source control. The certificate in
the test directory is used by the unit tests, and please do not actually use
it anywhere else.)

Running Tests

Run the tests as usual:

text

go test

But we've also provided a test coverage script that will show you which
lines of code are not covered:

text

script/coverage --html

That will open a new browser tab showing coverage information.

License

This project uses the New BSD License, the same as the Go project itself.

Code of Conduct

Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms.

Overview (`vendor/github.com/agext/levenshtein/README.md`)

A Go package for calculating the Levenshtein distance between two strings

This package implements distance and similarity metrics for strings, based on the Levenshtein measure, in Go.

Project Status

v1.2.3 Stable: Guaranteed no breaking changes to the API in future v1.x releases. Probably safe to use in production, though provided on "AS IS" basis.

This package is being actively maintained. If you encounter any problems or have any suggestions for improvement, please open an issue. Pull requests are welcome.

Overview

The Levenshtein Distance between two strings is the minimum total cost of edits that would convert the first string into the second. The allowed edit operations are insertions, deletions, and substitutions, all at character (one UTF-8 code point) level. Each operation has a default cost of 1, but each can be assigned its own cost equal to or greater than 0.

A Distance of 0 means the two strings are identical, and the higher the value the more different the strings. Since in practice we are interested in finding if the two strings are "close enough", it often does not make sense to continue the calculation once the result is mathematically guaranteed to exceed a desired threshold. Providing this value to the Distance function allows it to take a shortcut and return a lower bound instead of an exact cost when the threshold is exceeded.

The Similarity function calculates the distance, then converts it into a normalized metric within the range 0..1, with 1 meaning the strings are identical, and 0 that they have nothing in common. A minimum similarity threshold can be provided to speed up the calculation of the metric for strings that are far too dissimilar for the purpose at hand. All values under this threshold are rounded down to 0.

The Match function provides a similarity metric, with the same range and meaning as Similarity, but with a bonus for string pairs that share a common prefix and have a similarity above a "bonus threshold". It uses the same method as proposed by Winkler for the Jaro distance, and the reasoning behind it is that these string pairs are very likely spelling variations or errors, and they are more closely linked than the edit distance alone would suggest.

The underlying Calculate function is also exported, to allow the building of other derivative metrics, if needed.

Installation

text

go get github.com/agext/levenshtein

License

Package levenshtein is released under the Apache 2.0 license. See the LICENSE file for details.

Overview (`vendor/github.com/anchore/go-struct-converter/README.md`)

Go `struct` Converter

A library for converting between Go structs.

chain := converter.NewChain(V1{}, V2{}, V3{})

chain.Convert(myV1struct, &myV3struct)

Details

At its core, this library provides a Convert function, which automatically
handles converting fields with the same name, and "convertable"
types. Some examples are:

string -> string
string -> *string
int -> string
string -> []string

The automatic conversions are implemented when there is an obvious way
to convert between the types. A lot more automatic conversions happen
-- see the converter tests for a more comprehensive
list of what is currently supported.

Not everything can be handled automatically, however, so there is also
a ConvertFrom interface any struct in the graph can implement to
perform custom conversion, similar to how the stdlib MarshalJSON and
UnmarshalJSON would be implemented.

Additionally, and maybe most importantly, there is a converter.Chain available,
which orchestrates conversions between multiple versions of structs. This could
be thought of similar to database migrations: given a starting struct and a target
struct, the chain.Convert function iterates through every intermediary migration
in order to arrive at the target struct.

Basic Usage

To illustrate usage we'll start with a few basic structs, some of which
implement the ConvertFrom interface due to breaking changes:

// --------- V1 struct definition below ---------

type V1 struct {
  Name     string
  OldField string
}

// --------- V2 struct definition below ---------

type V2 struct {
  Name     string
  NewField string // this was a renamed field
}

func (to *V2) ConvertFrom(from interface{}) error {
  if from, ok := from.(V1); ok { // forward migration
    to.NewField = from.OldField
  }
  return nil
}

// --------- V3 struct definition below ---------

type V3 struct {
  Name       []string
  FinalField []string // this field was renamed and the type was changed
}

func (to *V3) ConvertFrom(from interface{}) error {
  if from, ok := from.(V2); ok { // forward migration
    to.FinalField = []string{from.NewField}
  }
  return nil
}

Given these type definitions, we can easily set up a conversion chain
like this:

chain := converter.NewChain(V1{}, V2{}, V3{})

This chain can then be used to convert from an older version to a newer
version. This is because our ConvertFrom definitions are only handling
forward migrations.

This chain can be used to convert from a V1 struct to a V3 struct easily,
like this:

v1 := // somehow get a populated v1 struct
v3 := V3{}
chain.Convert(v1, &v3)

Since we've defined our chain as V1 → V2 → V3, the chain will execute
conversions to all intermediary structs (V2, in this case) and ultimately end
when we've populated the v3 instance.

Note we haven't needed to define any conversions on the Name field of any structs
since this one is convertible between structs: string → string → []string.

Backwards Migrations

If we wanted to also provide backwards migrations, we could also easily add a case
to the ConvertFrom methods. The whole set of structs would look something like this:

// --------- V1 struct definition below ---------

type V1 struct {
  Name     string
  OldField string
}

func (to *V1) ConvertFrom(from interface{}) error {
  if from, ok := from.(V2); ok { // backward migration
    to.OldField = from.NewField
  }
  return nil
}

// --------- V2 struct definition below ---------

type V2 struct {
  Name     string
  NewField string
}

func (to *V2) ConvertFrom(from interface{}) error {
  if from, ok := from.(V1); ok { // forward migration
    to.NewField = from.OldField
  }
  if from, ok := from.(V3); ok { // backward migration
    to.NewField = from.FinalField[0]
  }
  return nil
}

// --------- V3 struct definition below ---------

type V3 struct {
  Name       []string
  FinalField []string
}

func (to *V3) ConvertFrom(from interface{}) error {
  if from, ok := from.(V2); ok { // forward migration
    to.FinalField = []string{from.NewField}
  }
  return nil
}

At this point we could convert in either direction, for example a
V3 struct could convert to a V1 struct, with the caveat that there
may be data loss, as might need to happen due to changes in the data shapes.

Contributing

If you would like to contribute to this repository, please see the
CONTRIBUTING.md.

Overview (`vendor/github.com/armon/circbuf/README.md`)

circbuf

This repository provides the circbuf package. This provides a Buffer object
which is a circular (or ring) buffer. It has a fixed size, but can be written
to infinitely. Only the last size bytes are ever retained. The buffer implements
the io.Writer interface.

Documentation

Full documentation can be found on Godoc

Usage

The circbuf package is very easy to use:

buf, _ := NewBuffer(6)
buf.Write([]byte("hello world"))

if string(buf.Bytes()) != " world" {
    panic("should only have last 6 bytes!")
}

Overview (`vendor/github.com/asaskevich/govalidator/README.md`)

govalidator

A package of validators and sanitizers for strings, structs and collections. Based on validator.js.

Installation

Make sure that Go is installed on your computer.
Type the following command in your terminal:

text

go get github.com/asaskevich/govalidator

or you can get specified release of the package with gopkg.in:

text

go get gopkg.in/asaskevich/govalidator.v10

After it the package is ready to use.

Import package in your project

Add following line in your *.go file:

import "github.com/asaskevich/govalidator"

If you are unhappy to use long govalidator, you can do something like this:

import (
  valid "github.com/asaskevich/govalidator"
)

Activate behavior to require all fields have a validation tag by default

SetFieldsRequiredByDefault causes validation to fail when struct fields do not include validations or are not explicitly marked as exempt (using valid:"-" or valid:"email,optional"). A good place to activate this is a package init function or the main() function.

SetNilPtrAllowedByRequired causes validation to pass when struct fields marked by required are set to nil. This is disabled by default for consistency, but some packages that need to be able to determine between nil and zero value state can use this. If disabled, both nil and zero values cause validation errors.

import "github.com/asaskevich/govalidator"

func init() {
  govalidator.SetFieldsRequiredByDefault(true)
}

Here's some code to explain it:

// this struct definition will fail govalidator.ValidateStruct() (and the field values do not matter):
type exampleStruct struct {
  Name  string ``
  Email string `valid:"email"`
}

// this, however, will only fail when Email is empty or an invalid email address:
type exampleStruct2 struct {
  Name  string `valid:"-"`
  Email string `valid:"email"`
}

// lastly, this will only fail when Email is an invalid email address but not when it's empty:
type exampleStruct2 struct {
  Name  string `valid:"-"`
  Email string `valid:"email,optional"`
}

Recent breaking changes (see [#123](https://github.com/asaskevich/govalidator/pull/123))

Custom validator function signature

A context was added as the second parameter, for structs this is the object being validated – this makes dependent validation possible.

import "github.com/asaskevich/govalidator"

// old signature
func(i interface{}) bool

// new signature
func(i interface{}, o interface{}) bool

Adding a custom validator

This was changed to prevent data races when accessing custom validators.

import "github.com/asaskevich/govalidator"

// before
govalidator.CustomTypeTagMap["customByteArrayValidator"] = func(i interface{}, o interface{}) bool {
  // ...
}

// after
govalidator.CustomTypeTagMap.Set("customByteArrayValidator", func(i interface{}, o interface{}) bool {
  // ...
})

List of functions:

text

/* Detailed source-code truncated for AI context efficiency. */

Examples

IsURL

println(govalidator.IsURL(`http://user@pass:domain.com/path/page`))

IsType

println(govalidator.IsType("Bob", "string"))
println(govalidator.IsType(1, "int"))
i := 1
println(govalidator.IsType(&i, "*int"))

IsType can be used through the tag type which is essential for map validation:

type User	struct {
  Name string      `valid:"type(string)"`
  Age  int         `valid:"type(int)"`
  Meta interface{} `valid:"type(string)"`
}
result, err := govalidator.ValidateStruct(User{"Bob", 20, "meta"})
if err != nil {
	println("error: " + err.Error())
}
println(result)

ToString

type User struct {
	FirstName string
	LastName string
}

str := govalidator.ToString(&User{"John", "Juan"})
println(str)

Each, Map, Filter, Count for slices

Each iterates over the slice/array and calls Iterator for every item

data := []interface{}{1, 2, 3, 4, 5}
var fn govalidator.Iterator = func(value interface{}, index int) {
	println(value.(int))
}
govalidator.Each(data, fn)

data := []interface{}{1, 2, 3, 4, 5}
var fn govalidator.ResultIterator = func(value interface{}, index int) interface{} {
	return value.(int) * 3
}
_ = govalidator.Map(data, fn) // result = []interface{}{1, 6, 9, 12, 15}

data := []interface{}{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
var fn govalidator.ConditionIterator = func(value interface{}, index int) bool {
	return value.(int)%2 == 0
}
_ = govalidator.Filter(data, fn) // result = []interface{}{2, 4, 6, 8, 10}
_ = govalidator.Count(data, fn) // result = 5

ValidateStruct [#2](https://github.com/asaskevich/govalidator/pull/2)

If you want to validate structs, you can use tag valid for any field in your structure. All validators used with this field in one tag are separated by comma. If you want to skip validation, place - in your tag. If you need a validator that is not on the list below, you can add it like this:

govalidator.TagMap["duck"] = govalidator.Validator(func(str string) bool {
	return str == "duck"
})

For completely custom validators (interface-based), see below.

Here is a list of available validators for struct fields (validator - used function):

text

/* Detailed source-code truncated for AI context efficiency. */

Validators with parameters

"range(min|max)": Range,
"length(min|max)": ByteLength,
"runelength(min|max)": RuneLength,
"stringlength(min|max)": StringLength,
"matches(pattern)": StringMatches,
"in(string1|string2|...|stringN)": IsIn,
"rsapub(keylength)" : IsRsaPub,
"minstringlength(int): MinStringLength,
"maxstringlength(int): MaxStringLength,

Validators with parameters for any type

"type(type)": IsType,

And here is small example of usage:

type Post struct {
	Title    string `valid:"alphanum,required"`
	Message  string `valid:"duck,ascii"`
	Message2 string `valid:"animal(dog)"`
	AuthorIP string `valid:"ipv4"`
	Date     string `valid:"-"`
}
post := &Post{
	Title:   "My Example Post",
	Message: "duck",
	Message2: "dog",
	AuthorIP: "123.234.54.3",
}

// Add your own struct validation tags
govalidator.TagMap["duck"] = govalidator.Validator(func(str string) bool {
	return str == "duck"
})

// Add your own struct validation tags with parameter
govalidator.ParamTagMap["animal"] = govalidator.ParamValidator(func(str string, params ...string) bool {
    species := params[0]
    return str == species
})
govalidator.ParamTagRegexMap["animal"] = regexp.MustCompile("^animal\\((\\w+)\\)$")

result, err := govalidator.ValidateStruct(post)
if err != nil {
	println("error: " + err.Error())
}
println(result)

ValidateMap [#2](https://github.com/asaskevich/govalidator/pull/338)

If you want to validate maps, you can use the map to be validated and a validation map that contain the same tags used in ValidateStruct, both maps have to be in the form map[string]interface{}

So here is small example of usage:

var mapTemplate = map[string]interface{}{
	"name":"required,alpha",
	"family":"required,alpha",
	"email":"required,email",
	"cell-phone":"numeric",
	"address":map[string]interface{}{
		"line1":"required,alphanum",
		"line2":"alphanum",
		"postal-code":"numeric",
	},
}

var inputMap = map[string]interface{}{
	"name":"Bob",
	"family":"Smith",
	"email":"[email protected]",
	"address":map[string]interface{}{
		"line1":"",
		"line2":"",
		"postal-code":"",
	},
}

result, err := govalidator.ValidateMap(inputMap, mapTemplate)
if err != nil {
	println("error: " + err.Error())
}
println(result)

WhiteList

// Remove all characters from string ignoring characters between "a" and "z"
println(govalidator.WhiteList("a3a43a5a4a3a2a23a4a5a4a3a4", "a-z") == "aaaaaaaaaaaa")

Custom validation functions

Custom validation using your own domain specific validators is also available - here's an example of how to use it:

text

/* Detailed source-code truncated for AI context efficiency. */

Loop over Error()

By default .Error() returns all errors in a single String. To access each error you can do this:

if err != nil {
    errs := err.(govalidator.Errors).Errors()
    for _, e := range errs {
      fmt.Println(e.Error())
    }
  }

Custom error messages

Custom error messages are supported via annotations by adding the ~ separator - here's an example of how to use it:

type Ticket struct {
  Id        int64     `json:"id"`
  FirstName string    `json:"firstname" valid:"required~First name is blank"`
}

Notes

Documentation is available here: godoc.org.
Full information about code coverage is also available here: govalidator on gocover.io.

Support

If you do have a contribution to the package, feel free to create a Pull Request or an Issue.

What to contribute

If you don't know what to do, there are some features and functions that need to be done

Refactor code
Edit docs and README: spellcheck, grammar and typo check
Create actual list of contributors and projects that currently using this package
Resolve issues and bugs
Update actual list of functions
Update list of validators that available for ValidateStruct and add new
Implement new validators: IsFQDN, IsIMEI, IsPostalCode, IsISIN, IsISRC etc
Implement validation by maps
Implement fuzzing testing
Implement some struct/map/array utilities
Implement map/array validation
Implement benchmarking
Implement batch of examples
Look at forks for new features and fixes

Advice

Feel free to create what you want, but keep in mind when you implement new features:

Code must be clear and readable, names of variables/constants clearly describes what they are doing
Public functions must be documented and described in source file and added to README.md to the list of available functions
There are must be unit-tests for any new functions and improvements

Credits

Contributors

This project exists thanks to all the people who contribute. [Contribute].

Special thanks to [contributors](https://github.com/asaskevich/govalidator/graphs/contributors)

Backers

Thank you to all our backers! 🙏 [Become a backer]

License

Overview (`vendor/github.com/aws/smithy-go/README.md`)

Smithy Go

Smithy code generators for Go, and
the runtime package the generated code depends on.

[!WARNING]
All interfaces in this repository are subject to change. The client code
generator in particular powers
aws-sdk-go-v2, but arbitrary client
generation is in an early stage of development:

Generated clients are missing certain features that were originally
implemented SDK-side (e.g. retries)

There may be bugs

The public APIs of generated clients may be unstable

If you are interested in using the client code generators, we encourage you
to experiment and share any feedback with us in an issue.

This repository contains two major components:

Codegen (codegen/): A Java/Gradle-based Smithy build
plugin that generates Go client, server, and shape code from Smithy
models.
Runtime (this module, github.com/aws/smithy-go): The Go packages
that code generated by the plugins above depends on.

[!NOTE]
This repository does not contain any generated clients, such as for S3 or
other AWS services. Rather, these are the tools that facilitate generating
those clients (and non-AWS Smithy clients) from a Smithy model.

Getting started

If this is your first time using Smithy, follow the
Smithy Quickstart guide to learn the
basics and create a simple Smithy model.

The smithy-go runtime requires a minimum version of Go 1.24.

Codegen

To generate a Go client from a Smithy model, apply the go-codegen plugin in
your smithy-build.json:

json

{
  "version": "1.0",
  "sources": [
    "models"
  ],
  "maven": {
    "dependencies": [
      "software.amazon.smithy.go:smithy-go-codegen:[0.1.0,2.0)"
    ]
  },
  "plugins": {
    "go-codegen": {
      "service": "example.weather#Weather",
      "module": "github.com/example/weather",
      "generateGoMod": true,
      "goDirective": "1.24"
    }
  }
}

This repository implements the following Smithy build plugins:

ID	GAV prefix	Description
`go-codegen`	`software.amazon.smithy.go:smithy-go-codegen`	Go client code generation for Smithy models.
`go-server-codegen`	`software.amazon.smithy.go:smithy-go-codegen`	Go server code generation for Smithy models (work-in-progress, undocumented).
`go-shape-codegen`	`software.amazon.smithy.go:smithy-go-codegen`	Go shape code generation (types only) for Smithy models (work-in-progress, undocumented).

`go-codegen` settings

GoSettings
contains all of the settings enabled from smithy-build.json. The up-to-date
list of top-level properties enabled for go-codegen can be found in
GoSettings::from().

Setting	Type	Required	Description
`service`	string	yes	The Shape ID of the service for which to generate the client.
`module`	string	yes	Name of the module in `generated.json` (and `go.mod` if `generateGoMod` is enabled) and `doc.go`.
`generateGoMod`	boolean		Whether to generate a default `go.mod` file. The default value is `false`.
`goDirective`	string		Go directive of the module. The default value is the minimum supported Go version.

Supported protocols

The protocol a generated client uses is configured by the Protocol field on
the client's Options. The SDK configures a default based on the protocol
traits applied to the modeled service.

Each protocol is implemented as its own package under
transport/http/protocol. Serialization for these
protocols is handled by internal codecs under protocol/internal (e.g.
protocol/internal/json, protocol/internal/cbor,
protocol/internal/xml); these largely supersede the top-level
encoding/ packages of the same name; see
Encoding below.

Protocol	Package	Notes
`smithy.protocols#rpcv2Cbor`	`protocol/rpcv2`
`aws.protocols#restJson1`	`protocol/restjson1`
`aws.protocols#restXml`	`protocol/restxml`
`aws.protocols#awsJson1_0`	`protocol/awsjson`	`awsjson.New10`
`aws.protocols#awsJson1_1`	`protocol/awsjson`	`awsjson.New11`
`aws.protocols#awsQuery`	`protocol/awsquery`
`aws.protocols#ec2Query`	`protocol/ec2query`

These packages are client-only; smithy-go does not currently implement
server-side protocol support.

Building and testing the codegen plugins is done via Gradle from the
codegen/ directory:

bash

# Build and test codegen
cd codegen && ./gradlew build

# Publish to local Maven for downstream use
cd codegen && ./gradlew publishToMavenLocal

See codegen/README.md for local setup instructions.

Runtime packages

All generated clients depend on this module (github.com/aws/smithy-go) at
runtime. It also includes a few standalone submodules published separately.

Core

middleware: The middleware stack that drives the operation
pipeline.
transport/http: HTTP request/response types and
middleware.
auth: Auth identity and scheme interfaces.
- auth/bearer: Bearer token auth.
endpoints: Endpoint resolution types.
context: Context helpers.
logging: Logging interfaces.
tracing: Tracing interfaces.
metrics: Metrics interfaces.

Encoding

Wire format encoders/decoders under encoding/.

[!NOTE]
Most of these packages are effectively legacy. The protocol
implementations under transport/http/protocol
(used by current client codegen) have their own internal codecs under
protocol/internal/* and generally do not build on these packages.
The encoding/json, encoding/xml, and encoding/cbor packages otherwise
remain in use by document/json/document/cbor
and eventstream, and by existing generated SDK code that
predates the newer protocol implementations.

encoding/json: JSON encoding/decoding.
encoding/xml: XML encoding/decoding.
encoding/cbor: CBOR encoding/decoding (used by
rpcv2Cbor).
encoding/httpbinding: HTTP binding serde helpers.

Document

document: Smithy document type abstraction.
- document/json: JSON document codec.

Utilities

container: Generic container types.
io: I/O helpers.
ptr: Pointer-to/from-value helpers.
time: Smithy timestamp format helpers.
rand: UUID/randomness helpers.
testing: Test assertion helpers for generated protocol tests.

Standalone modules

These are published as separate Go modules, each with its own go.mod.

aws-http-auth: AWS SigV4/SigV4A HTTP signing.
tracing/smithyoteltracing: OpenTelemetry
tracing adapter.
metrics/smithyotelmetrics: OpenTelemetry
metrics adapter.

Development

See CONTRIBUTING for more information on contributing to
this project, including the changelog process for runtime changes.

License

This project is licensed under the Apache-2.0 License.

Overview (`vendor/github.com/bits-and-blooms/bitset/README.md`)

bitset

Go language library to map between non-negative integers and boolean values

This library is part of the awesome go collection. It is used in production by several important systems.

Description

Package bitset implements bitsets, a mapping between non-negative integers and boolean values.
It should be more efficient than map[uint] bool.

It provides methods for setting, clearing, flipping, and testing individual integers.

But it also provides set intersection, union, difference, complement, and symmetric operations, as well as tests to check whether any, all, or no bits are set, and querying a bitset's current length and number of positive bits.

BitSets are expanded to the size of the largest set bit; the memory allocation is approximately Max bits, where Max is the largest set bit. BitSets are never shrunk automatically, but Shrink and Compact methods are available. On creation, a hint can be given for the number of bits that will be used.

Many of the methods, including Set, Clear, and Flip, return a BitSet pointer, which allows for chaining.

Example use:

package main

import (
	"fmt"
	"math/rand"

	"github.com/bits-and-blooms/bitset"
)

func main() {
	fmt.Printf("Hello from BitSet!\n")
	var b bitset.BitSet
	// play some Go Fish
	for i := 0; i < 100; i++ {
		card1 := uint(rand.Intn(52))
		card2 := uint(rand.Intn(52))
		b.Set(card1)
		if b.Test(card2) {
			fmt.Println("Go Fish!")
		}
		b.Clear(card1)
	}

	// Chaining
	b.Set(10).Set(11)

	for i, e := b.NextSet(0); e; i, e = b.NextSet(i + 1) {
		fmt.Println("The following bit is set:", i)
	}
	if b.Intersection(bitset.New(100).Set(10)).Count() == 1 {
		fmt.Println("Intersection works.")
	} else {
		fmt.Println("Intersection doesn't work???")
	}
}

If you have Go 1.23 or better, you can iterate over the set bits like so:

for i := range b.EachSet() {}

Package documentation is at: https://pkg.go.dev/github.com/bits-and-blooms/bitset?tab=doc

Serialization

You may serialize a bitset safely and portably to a stream
of bytes as follows:

const length = 9585
	const oneEvery = 97
	bs := bitset.New(length)
	// Add some bits
	for i := uint(0); i < length; i += oneEvery {
		bs = bs.Set(i)
	}

	var buf bytes.Buffer
	n, err := bs.WriteTo(&buf)
	if err != nil {
		// failure
	}
	// Here n == buf.Len()

You can later deserialize the result as follows:

// Read back from buf
	bs = bitset.New()
	n, err = bs.ReadFrom(&buf)
	if err != nil {
		// error
	}
	// n is the number of bytes read

The ReadFrom function attempts to read the data into the existing
BitSet instance, to minimize memory allocations.

Performance tip:
When reading and writing to a file or a network connection, you may get better performance by
wrapping your streams with bufio instances.

E.g.,

f, err := os.Create("myfile")
	w := bufio.NewWriter(f)

f, err := os.Open("myfile")
	r := bufio.NewReader(f)

Memory Usage

The memory usage of a bitset using N bits is at least N/8 bytes. The number of bits in a bitset is at least as large as one plus the greatest bit index you have accessed. Thus it is possible to run out of memory while using a bitset. If you have lots of bits, you might prefer compressed bitsets, like the Roaring bitmaps and its Go implementation.

The roaring library allows you to go back and forth between compressed Roaring bitmaps and the conventional bitset instances:

mybitset := roaringbitmap.ToBitSet()
			newroaringbitmap := roaring.FromBitSet(mybitset)

Goroutine safety

In general, it's not safe to access the same BitSet using different goroutines--they are unsynchronized for performance.

Should you want to access a BitSet from more than one goroutine, you should provide synchronization. Typically this is done by using channels to pass the *BitSet around (in Go style; so there is only ever one owner), or by using sync.Mutex to serialize operations on BitSets.

Installation

bash

go get github.com/bits-and-blooms/bitset

Users

Databases & Storage

milvus-io/milvus — popular open-source vector database
cubefs/cubefs — CNCF distributed filesystem (used at JD.com)
sourcenetwork/defradb — decentralized document database
siglens/siglens — open-source observability/log analytics platform
ByteStorage/FlyDB — high-performance KV store

Search & Indexing

blevesearch/bleve — full-text search library for Go
blevesearch/vellum — FST library used internally by Bleve
blugelabs/bluge — modern Go search library (successor to Bleve)
geange/lucene-go — Lucene port to Go

Blockchain & Cryptography

consensys/gnark — zk-SNARK proving system
consensys/gnark-crypto — elliptic curve cryptography for ZK proofs
NethermindEth/juno — StarkNet full node
onflow/cadence — Flow blockchain smart contract language runtime
cosmos/cosmos-sdk — Cosmos blockchain SDK (slashing module)
babylonlabs-io/babylon — Bitcoin staking protocol
bnb-chain/greenfield — BNB Chain decentralized storage

Infrastructure & Networking

moby/swarmkit — Docker Swarm orchestration (ID management)
coredhcp/coredhcp — pluggable DHCP server (bitmap IP allocation)
Scalingo/sand — private networking for containers (IP allocation)
AliyunContainerService/terway — Alibaba Cloud K8s CNI
pomerium/pomerium — identity-aware access proxy

Distributed Systems & Messaging

apache/pulsar-client-go — official Apache Pulsar Go client
d7y.io/dragonfly — CNCF P2P file distribution system
RoaringBitmap/roaring — compressed bitmap library (interops with bitset)
zhenghaoz/gorse — Go recommendation engine

Machine Learning / Vector Search

fogfish/hnsw — HNSW approximate nearest-neighbor graph
semafind/semadb — vector database

Developer Tools & Security

bearer/bearer — security code scanner (AST traversal)
zrepl/zrepl — ZFS replication tool
NVIDIA/dcgm-exporter — NVIDIA GPU Prometheus exporter
NVIDIA/go-dcgm — NVIDIA datacenter GPU management bindings
gohugoio/hugo — the Hugo static site generator
godoctor/godoctor — Go refactoring tool (dataflow analysis)

Contributing

If you wish to contribute to this project, please branch and issue a pull request against master ("GitHub Flow")

Running all tests

Before committing the code, please check if it passes tests, has adequate coverage, etc.

bash

go test
go test -cover

semver for golang [](https://travis-ci.org/blang/semver) [](https://godoc.org/github.com/blang/semver) [](https://coveralls.io/r/blang/semver?branch=master)

semver is a Semantic Versioning library written in golang. It fully covers spec version 2.0.0.

Usage

bash

$ go get github.com/blang/semver

Note: Always vendor your dependencies or fix on a specific version tag.

import github.com/blang/semver
v1, err := semver.Make("1.0.0-beta")
v2, err := semver.Make("2.0.0-beta")
v1.Compare(v2)

Also check the GoDocs.

Why should I use this lib?

Fully spec compatible
No reflection
No regex
Fully tested (Coverage >99%)
Readable parsing/validation errors
Fast (See Benchmarks)
Only Stdlib
Uses values instead of pointers
Many features, see below

Features

Parsing and validation at all levels
Comparator-like comparisons
Compare Helper Methods
InPlace manipulation
Ranges >=1.0.0 <2.0.0 || >=3.0.0 !3.0.1-beta.1
Wildcards >=1.x, <=2.5.x
Sortable (implements sort.Interface)
database/sql compatible (sql.Scanner/Valuer)
encoding/json compatible (json.Marshaler/Unmarshaler)

Ranges

A Range is a set of conditions which specify which versions satisfy the range.

A condition is composed of an operator and a version. The supported operators are:

<1.0.0 Less than 1.0.0
<=1.0.0 Less than or equal to 1.0.0
>1.0.0 Greater than 1.0.0
>=1.0.0 Greater than or equal to 1.0.0
1.0.0, =1.0.0, ==1.0.0 Equal to 1.0.0
!1.0.0, !=1.0.0 Not equal to 1.0.0. Excludes version 1.0.0.

Note that spaces between the operator and the version will be gracefully tolerated.

A Range can link multiple Ranges separated by space:

Ranges can be linked by logical AND:

>1.0.0 <2.0.0 would match between both ranges, so 1.1.1 and 1.8.7 but not 1.0.0 or 2.0.0
>1.0.0 <3.0.0 !2.0.3-beta.2 would match every version between 1.0.0 and 3.0.0 except 2.0.3-beta.2

Ranges can also be linked by logical OR:

<2.0.0 || >=3.0.0 would match 1.x.x and 3.x.x but not 2.x.x

AND has a higher precedence than OR. It's not possible to use brackets.

Ranges can be combined by both AND and OR

>1.0.0 <2.0.0 || >3.0.0 !4.2.1 would match 1.2.3, 1.9.9, 3.1.1, but not 4.2.1, 2.1.1

Range usage:

text

v, err := semver.Parse("1.2.3")
range, err := semver.ParseRange(">1.0.0 <2.0.0 || >=3.0.0")
if range(v) {
    //valid
}

Example

Have a look at full examples in examples/main.go

import github.com/blang/semver

v, err := semver.Make("0.0.1-alpha.preview+123.github")
fmt.Printf("Major: %d\n", v.Major)
fmt.Printf("Minor: %d\n", v.Minor)
fmt.Printf("Patch: %d\n", v.Patch)
fmt.Printf("Pre: %s\n", v.Pre)
fmt.Printf("Build: %s\n", v.Build)

// Prerelease versions array
if len(v.Pre) > 0 {
    fmt.Println("Prerelease versions:")
    for i, pre := range v.Pre {
        fmt.Printf("%d: %q\n", i, pre)
    }
}

// Build meta data array
if len(v.Build) > 0 {
    fmt.Println("Build meta data:")
    for i, build := range v.Build {
        fmt.Printf("%d: %q\n", i, build)
    }
}

v001, err := semver.Make("0.0.1")
// Compare using helpers: v.GT(v2), v.LT, v.GTE, v.LTE
v001.GT(v) == true
v.LT(v001) == true
v.GTE(v) == true
v.LTE(v) == true

// Or use v.Compare(v2) for comparisons (-1, 0, 1):
v001.Compare(v) == 1
v.Compare(v001) == -1
v.Compare(v) == 0

// Manipulate Version in place:
v.Pre[0], err = semver.NewPRVersion("beta")
if err != nil {
    fmt.Printf("Error parsing pre release version: %q", err)
}

fmt.Println("\nValidate versions:")
v.Build[0] = "?"

err = v.Validate()
if err != nil {
    fmt.Printf("Validation failed: %s\n", err)
}

Benchmarks

text

BenchmarkParseSimple-4           5000000    390    ns/op    48 B/op   1 allocs/op
BenchmarkParseComplex-4          1000000   1813    ns/op   256 B/op   7 allocs/op
BenchmarkParseAverage-4          1000000   1171    ns/op   163 B/op   4 allocs/op
BenchmarkStringSimple-4         20000000    119    ns/op    16 B/op   1 allocs/op
BenchmarkStringLarger-4         10000000    206    ns/op    32 B/op   2 allocs/op
BenchmarkStringComplex-4         5000000    324    ns/op    80 B/op   3 allocs/op
BenchmarkStringAverage-4         5000000    273    ns/op    53 B/op   2 allocs/op
BenchmarkValidateSimple-4      200000000      9.33 ns/op     0 B/op   0 allocs/op
BenchmarkValidateComplex-4       3000000    469    ns/op     0 B/op   0 allocs/op
BenchmarkValidateAverage-4       5000000    256    ns/op     0 B/op   0 allocs/op
BenchmarkCompareSimple-4       100000000     11.8  ns/op     0 B/op   0 allocs/op
BenchmarkCompareComplex-4       50000000     30.8  ns/op     0 B/op   0 allocs/op
BenchmarkCompareAverage-4       30000000     41.5  ns/op     0 B/op   0 allocs/op
BenchmarkSort-4                  3000000    419    ns/op   256 B/op   2 allocs/op
BenchmarkRangeParseSimple-4      2000000    850    ns/op   192 B/op   5 allocs/op
BenchmarkRangeParseAverage-4     1000000   1677    ns/op   400 B/op  10 allocs/op
BenchmarkRangeParseComplex-4      300000   5214    ns/op  1440 B/op  30 allocs/op
BenchmarkRangeMatchSimple-4     50000000     25.6  ns/op     0 B/op   0 allocs/op
BenchmarkRangeMatchAverage-4    30000000     56.4  ns/op     0 B/op   0 allocs/op
BenchmarkRangeMatchComplex-4    10000000    153    ns/op     0 B/op   0 allocs/op

See benchmark cases at semver_test.go

Motivation

I simply couldn't find any lib supporting the full spec. Others were just wrong or used reflection and regex which i don't like.

Contribution

Feel free to make a pull request. For bigger changes create a issue first to discuss about it.

License

See LICENSE file.

Overview (`vendor/github.com/cilium/ebpf/README.md`)

eBPF

ebpf-go is a pure Go library that provides utilities for loading, compiling, and
debugging eBPF programs. It has minimal external dependencies and is intended to
be used in long running processes.

See ebpf.io for complementary projects from the wider eBPF
ecosystem.

Getting Started

Please take a look at our Getting Started guide.

Contributions are highly encouraged, as they highlight certain use cases of
eBPF and the library, and help shape the future of the project.

Getting Help

The community actively monitors our GitHub Discussions page.
Please search for existing threads before starting a new one. Refrain from
opening issues on the bug tracker if you're just starting out or if you're not
sure if something is a bug in the library code.

Alternatively, join the
#ebpf-go channel on Slack if you
have other questions regarding the project. Note that this channel is ephemeral
and has its history erased past a certain point, which is less helpful for
others running into the same problem later.

Packages

This library includes the following packages:

asm contains a basic
assembler, allowing you to write eBPF assembly instructions directly
within your Go code. (You don't need to use this if you prefer to write your eBPF program in C.)
cmd/bpf2go allows
compiling and embedding eBPF programs written in C within Go code. As well as
compiling the C code, it auto-generates Go code for loading and manipulating
the eBPF program and map objects.
link allows attaching eBPF
to various hooks
perf allows reading from a
PERF_EVENT_ARRAY
ringbuf allows reading from a
BPF_MAP_TYPE_RINGBUF map
features implements the equivalent
of bpftool feature probe for discovering BPF-related kernel features using native Go.
rlimit provides a convenient API to lift
the RLIMIT_MEMLOCK constraint on kernels before 5.11.
btf allows reading the BPF Type Format.
pin provides APIs for working with pinned objects on bpffs.

Requirements

A version of Go that is supported by
upstream
Linux (amd64, arm64): CI is run against kernel.org LTS releases. >= 4.4 should work but EOL'ed
versions are not supported.
Windows (amd64): CI is run against Windows Server 2022. Only the latest eBPF for Windows
release is supported.
Other architectures are best effort. 32bit arches are not supported.

License

MIT

eBPF Gopher

The eBPF honeygopher is based on the Go gopher designed by Renee French.

Overview (`vendor/github.com/containerd/console/README.md`)

console

Golang package for dealing with consoles. Light on deps and a simple API.

Modifying the current process

current := console.Current()
defer current.Reset()

if err := current.SetRaw(); err != nil {
}
ws, err := current.Size()
current.Resize(ws)

Project details

console is a containerd sub-project, licensed under the Apache 2.0 license.
As a containerd sub-project, you will find the:

information in our containerd/project repository.

Overview (`vendor/github.com/containerd/errdefs/README.md`)

errdefs

A Go package for defining and checking common containerd errors.

Project details

errdefs is a containerd sub-project, licensed under the Apache 2.0 license.
As a containerd sub-project, you will find the:

information in our containerd/project repository.

Overview (`vendor/github.com/containerd/fifo/readme.md`)

fifo

Go package for handling fifos in a sane way.

text

// OpenFifo opens a fifo. Returns io.ReadWriteCloser.
// Context can be used to cancel this function until open(2) has not returned.
// Accepted flags:
// - syscall.O_CREAT - create new fifo if one doesn't exist
// - syscall.O_RDONLY - open fifo only from reader side
// - syscall.O_WRONLY - open fifo only from writer side
// - syscall.O_RDWR - open fifo from both sides, never block on syscall level
// - syscall.O_NONBLOCK - return io.ReadWriteCloser even if other side of the
//     fifo isn't open. read/write will be connected after the actual fifo is
//     open or after fifo is closed.
func OpenFifo(ctx context.Context, fn string, flag int, perm os.FileMode) (io.ReadWriteCloser, error)


// Read from a fifo to a byte array.
func (f *fifo) Read(b []byte) (int, error)


// Write from byte array to a fifo.
func (f *fifo) Write(b []byte) (int, error)


// Close the fifo. Next reads/writes will error. This method can also be used
// before open(2) has returned and fifo was never opened.
func (f *fifo) Close() error

Project details

The fifo is a containerd sub-project, licensed under the Apache 2.0 license.
As a containerd sub-project, you will find the:

information in our containerd/project repository.

Overview (`vendor/github.com/containerd/go-cni/README.md`)

go-cni

A generic CNI library to provide APIs for CNI plugin interactions. The library provides APIs to:

Load CNI network config from different sources
Setup networks for container namespace
Remove networks from container namespace
Query status of CNI network plugin initialization
Check verifies the network is still in desired state

go-cni aims to support plugins that implement the Container Network Interface.

Usage

text

/* Detailed source-code truncated for AI context efficiency. */

Project details

The go-cni is a containerd sub-project, licensed under the Apache 2.0 license.
As a containerd sub-project, you will find the:

information in our containerd/project repository.

Overview (`vendor/github.com/containerd/go-runc/README.md`)

go-runc

This is a module for consuming the runc binary in your Go applications.
It tries to expose all the settings and features of the runc CLI.

Project details

go-runc is a containerd sub-project, licensed under the Apache 2.0 license.
As a containerd sub-project, you will find the:

information in our containerd/project repository.

Overview (`vendor/github.com/containerd/log/README.md`)

log

A Go package providing a common logging interface across containerd repositories and a way for clients to use and configure logging in containerd packages.

This package is not intended to be used as a standalone logging package outside of the containerd ecosystem and is intended as an interface wrapper around a logging implementation.
In the future this package may be replaced with a common go logging interface.

Project details

log is a containerd sub-project, licensed under the Apache 2.0 license.
As a containerd sub-project, you will find the:

information in our containerd/project repository.

Overview (`vendor/github.com/containerd/otelttrpc/README.md`)

ttrpc OpenTelemetry Instrumentation

This golang package implements OpenTelemetry instrumentation support for
ttrpc. It can be used to automatically generate OpenTelemetry trace spans
for RPC methods called on the ttrpc client side and served on the ttrpc
server side.

Usage

Instrumentation is provided by two interceptors, one to enable instrumentation
for unary clients and another for enabling instrumentation for unary servers.
These interceptors can be passed as ttrpc.ClientOpts and ttrpc.ServerOpt to
ttrpc during client and server creation with code like this:

golang

import (
       "github.com/containerd/ttrpc"
       "github.com/containerd/otelttrpc"
   )

   // on the client side
   ...
   client := ttrpc.NewClient(
       conn,
       ttrpc.UnaryClientInterceptor(
           otelttrpc.UnaryClientInterceptor(),
       ),
   )

   // and on the server side
   ...
   server, err := ttrpc.NewServer(
       ttrpc.WithUnaryServerInterceptor(
           otelttrpc.UnaryServerInterceptor(),
       ),
   )

Once enabled, the interceptors generate trace Spans for all called and served
unary method calls. If the rest of the code is properly set up to collect and
export tracing data to opentelemetry, these spans should show up as part of
the collected traces.

For a more complete example see the sample client
and the sample server code.

Limitations

Currently only unary client and unary server methods can be instrumented.
Support for streaming interfaces is yet to be implemented.

Project details

otelttrpc is a containerd sub-project, licensed under the Apache 2.0 license.
As a containerd sub-project, you will find the:

information in our containerd/project repository.

Overview (`vendor/github.com/containerd/platforms/README.md`)

platforms

A Go package for formatting, normalizing and matching container platforms.

This package is based on the Open Containers Image Spec definition of a platform.

Platform Specifier

While the OCI platform specifications provide a tool for components to
specify structured information, user input typically doesn't need the full
context and much can be inferred. To solve this problem, this package introduces
"specifiers". A specifier has the format
<os>|<arch>|<os>/<arch>[/<variant>]. The user can provide either the
operating system or the architecture or both.

An example of a common specifier is linux/amd64. If the host has a default
runtime that matches this, the user can simply provide the component that
matters. For example, if an image provides amd64 and arm64 support, the
operating system, linux can be inferred, so they only have to provide
arm64 or amd64. Similar behavior is implemented for operating systems,
where the architecture may be known but a runtime may support images from
different operating systems.

Project details

platforms is a containerd sub-project, licensed under the Apache 2.0 license.
As a containerd sub-project, you will find the:

information in our containerd/project repository.

Overview (`vendor/github.com/containerd/plugin/README.md`)

plugin

A Go package providing a common plugin interface across containerd repositories.

This package is intended to be imported by the main containerd repository as well as plugin implementations.
By sharing a common implementations, plugins can register themselves without needing to import the main containerd repository.
This plugin is intended to provide an interface and common functionality, but is not intended to define plugin types used by containerd.
Plugins should copy plugin type strings to avoid creating unintended depdenencies.

Project details

plugin is a containerd sub-project, licensed under the Apache 2.0 license.
As a containerd sub-project, you will find the:

information in our containerd/project repository.

Overview (`vendor/github.com/containerd/ttrpc/README.md`)

ttrpc

GRPC for low-memory environments.

The existing grpc-go project requires a lot of memory overhead for importing
packages and at runtime. While this is great for many services with low density
requirements, this can be a problem when running a large number of services on
a single machine or on a machine with a small amount of memory.

Using the same GRPC definitions, this project reduces the binary size and
protocol overhead required. We do this by eliding the net/http, net/http2
and grpc package used by grpc replacing it with a lightweight framing
protocol. The result are smaller binaries that use less resident memory with
the same ease of use as GRPC.

Please note that while this project supports generating either end of the
protocol, the generated service definitions will be incompatible with regular
GRPC services, as they do not speak the same protocol.

Protocol

See the protocol specification.

Usage

It's recommended to use protobuild
to build the protobufs for this project, but this will work with protoc
directly, if required.

Differences from GRPC

The protocol stack has been replaced with a lighter protocol that doesn't
require http, http2 and tls.
The client and server interface are identical whereas in GRPC there is a
client and server interface that are different.
The Go stdlib context package is used instead.

Status

TODO:

Add testing under concurrent load to ensure
Verify connection error handling

Project details

ttrpc is a containerd sub-project, licensed under the Apache 2.0 license.
As a containerd sub-project, you will find the:

information in our containerd/project repository.

Overview (`vendor/github.com/cpuguy83/tar2go/README.md`)

tar2go

tar2go implements are go fs.FS for tar files.

Tars are not indexed so by themselves don't really have support for random access.
When a request to open/stat a file is made tar2go will scan through the tar, indexing each entry along the way, until the file is found in the tar.
A tar file is only ever scanned 1 time and scanning is done lazily (as needed to index the requested entry).

tar2go does not support modifying a tar file, however there is support for modifying the in-memory representation of the tar which will show up in the fs.FS.
You can also write a new tar file with requested modifications.

Usage

f, _ := os.Open(p)
  defer f.Close()
  
  // Entrypoint into this library
  idx := NewIndex(f)
  
  // Get the `fs.FS` implementation
  goFS := idx.FS()
  // Do stuff with your fs
  // ...
  
  
  // Add or replace a file in the index
  _ := idx.Replace("foo", strings.NewReader("random stuff")
  data, _ := fs.ReadFile(goFS, "foo")
  if string(data) != "random stuff") {
    panic("unexpected data")
  }
  
  // Delete a file in the index
  _ := idx.Replace("foo", nil)
  if _, err := fs.ReadFile(goFS, "foo"); !errors.Is(err, fs.ErrNotExist) {
    panic(err)
  }
  
  // Create a new tar with updated content
  // First we need to create an `io.Writer`, which is where the updated tar stream will be written to.
  f, _ := os.CreateTemp("", "updated")
  idx.Update(f, func(name string, rdr ReaderAtSized) (ReaderAtSized, bool, error) {
    // Update calls this function for every file in the tar
    // The returned `ReaderAtSized` is used instead of the content passed in (rdr).
    // To make no changes just return the same rdr back.
    // Return true for the bool value if the content is changed.
  })

Overview (`vendor/github.com/creack/pty/README.md`)

pty

Pty is a Go package for using unix pseudo-terminals.

Install

go get github.com/creack/pty

Examples

Note that those examples are for demonstration purpose only, to showcase how to use the library. They are not meant to be used in any kind of production environment. If you want to set deadlines to work and Close() interrupting Read() on the returned *os.File, you will need to call syscall.SetNonblock manually.

Command

package main

import (
	"io"
	"os"
	"os/exec"

	"github.com/creack/pty"
)

func main() {
	c := exec.Command("grep", "--color=auto", "bar")
	f, err := pty.Start(c)
	if err != nil {
		panic(err)
	}

	go func() {
		f.Write([]byte("foo\n"))
		f.Write([]byte("bar\n"))
		f.Write([]byte("baz\n"))
		f.Write([]byte{4}) // EOT
	}()
	io.Copy(os.Stdout, f)
}

Shell

text

/* Detailed source-code truncated for AI context efficiency. */

Overview (`vendor/github.com/cyphar/filepath-securejoin/README.md`)

`filepath-securejoin`

Old API

This library was originally just an implementation of SecureJoin which was
intended to be included in the Go standard library as a safer
filepath.Join that would restrict the path lookup to be inside a root
directory.

The implementation was based on code that existed in several container
runtimes. Unfortunately, this API is fundamentally unsafe against attackers
that can modify path components after SecureJoin returns and before the
caller uses the path, allowing for some fairly trivial TOCTOU attacks.

SecureJoin (and SecureJoinVFS) are still provided by this library to
support legacy users, but new users are strongly suggested to avoid using
SecureJoin and instead use the new api or switch to
libpathrs.

With the above limitations in mind, this library guarantees the following:

If no error is set, the resulting string must be a child path of
root and will not contain any symlink path components (they will all be
expanded).
When expanding symlinks, all symlink path components must be resolved
relative to the provided root. In particular, this can be considered a
userspace implementation of how chroot(2) operates on file paths. Note that
these symlinks will not be expanded lexically (filepath.Clean is not
called on the input before processing).
Non-existent path components are unaffected by SecureJoin (similar to
filepath.EvalSymlinks's semantics).
The returned path will always be filepath.Cleaned and thus not contain any
.. components.

A (trivial) implementation of this function on GNU/Linux systems could be done
with the following (note that this requires root privileges and is far more
opaque than the implementation in this library, and also requires that
readlink is inside the root path and is trustworthy):

package securejoin

import (
	"os/exec"
	"path/filepath"
)

func SecureJoin(root, unsafePath string) (string, error) {
	unsafePath = string(filepath.Separator) + unsafePath
	cmd := exec.Command("chroot", root,
		"readlink", "--canonicalize-missing", "--no-newline", unsafePath)
	output, err := cmd.CombinedOutput()
	if err != nil {
		return "", err
	}
	expanded := string(output)
	return filepath.Join(root, expanded), nil
}

New API

While we recommend users switch to libpathrs as soon as it has a
stable release, some methods implemented by libpathrs have been ported to this
library to ease the transition. These APIs are only supported on Linux.

These APIs are implemented such that filepath-securejoin will
opportunistically use certain newer kernel APIs that make these operations far
more secure. In particular:

All of the lookup operations will use openat2 on new enough
kernels (Linux 5.6 or later) to restrict lookups through magic-links and
bind-mounts (for certain operations) and to make use of RESOLVE_IN_ROOT to
efficiently resolve symlinks within a rootfs.
The APIs provide hardening against a malicious /proc mount to either detect
or avoid being tricked by a /proc that is not legitimate. This is done
using openat2 for all users, and privileged users will also be
further protected by using fsopen and open_tree
(Linux 5.2 or later).

`OpenInRoot`

func OpenInRoot(root, unsafePath string) (*os.File, error)
func OpenatInRoot(root *os.File, unsafePath string) (*os.File, error)
func Reopen(handle *os.File, flags int) (*os.File, error)

OpenInRoot is a much safer version of

path, err := securejoin.SecureJoin(root, unsafePath)
file, err := os.OpenFile(path, unix.O_PATH|unix.O_CLOEXEC)

that protects against various race attacks that could lead to serious security
issues, depending on the application. Note that the returned *os.File is an
O_PATH file descriptor, which is quite restricted. Callers will probably need
to use Reopen to get a more usable handle (this split is done to provide
useful features like PTY spawning and to avoid users accidentally opening bad
inodes that could cause a DoS).

Callers need to be careful in how they use the returned *os.File. Usually it
is only safe to operate on the handle directly, and it is very easy to create a
security issue. libpathrs provides far more helpers to make using
these handles safer -- there is currently no plan to port them to
filepath-securejoin.

OpenatInRoot is like OpenInRoot except that the root is provided using an
*os.File. This allows you to ensure that multiple OpenatInRoot (or
MkdirAllHandle) calls are operating on the same rootfs.

NOTE: Unlike SecureJoin, OpenInRoot will error out as soon as it hits
a dangling symlink or non-existent path. This is in contrast to SecureJoin
which treated non-existent components as though they were real directories,
and would allow for partial resolution of dangling symlinks. These behaviours
are at odds with how Linux treats non-existent paths and dangling symlinks,
and so these are no longer allowed.

`MkdirAll`

func MkdirAll(root, unsafePath string, mode int) error
func MkdirAllHandle(root *os.File, unsafePath string, mode int) (*os.File, error)

MkdirAll is a much safer version of

path, err := securejoin.SecureJoin(root, unsafePath)
err = os.MkdirAll(path, mode)

that protects against the same kinds of races that OpenInRoot protects
against.

MkdirAllHandle is like MkdirAll except that the root is provided using an
*os.File (the reason for this is the same as with OpenatInRoot) and an
*os.File of the final created directory is returned (this directory is
guaranteed to be effectively identical to the directory created by
MkdirAllHandle, which is not possible to ensure by just using OpenatInRoot
after MkdirAll).

NOTE: Unlike SecureJoin, MkdirAll will error out as soon as it hits
a dangling symlink or non-existent path. This is in contrast to SecureJoin
which treated non-existent components as though they were real directories,
and would allow for partial resolution of dangling symlinks. These behaviours
are at odds with how Linux treats non-existent paths and dangling symlinks,
and so these are no longer allowed. This means that MkdirAll will not
create non-existent directories referenced by a dangling symlink.

License

SPDX-License-Identifier: BSD-3-Clause AND MPL-2.0

Some of the code in this project is derived from Go, and is licensed under a
BSD 3-clause license (available in LICENSE.BSD). Other files (many of which
are derived from libpathrs) are licensed under the Mozilla Public
License version 2.0 (available in LICENSE.MPL-2.0). If you are using the
"New API" described above, you are probably using code from files
released under this license.

Every source file in this project has a copyright header describing its
license. Please check the license headers of each file to see what license
applies to it.

See COPYING.md for some more details.

Overview (`vendor/github.com/digitorus/pkcs7/README.md`)

pkcs7

pkcs7 implements parsing and creating signed and enveloped messages.

package main

import (
	"bytes"
	"crypto/rsa"
	"crypto/x509"
	"encoding/pem"
	"fmt"
	"os"

    "go.mozilla.org/pkcs7"
)

func SignAndDetach(content []byte, cert *x509.Certificate, privkey *rsa.PrivateKey) (signed []byte, err error) {
	toBeSigned, err := NewSignedData(content)
	if err != nil {
		err = fmt.Errorf("Cannot initialize signed data: %s", err)
		return
	}
	if err = toBeSigned.AddSigner(cert, privkey, SignerInfoConfig{}); err != nil {
		err = fmt.Errorf("Cannot add signer: %s", err)
		return
	}

	// Detach signature, omit if you want an embedded signature
	toBeSigned.Detach()

	signed, err = toBeSigned.Finish()
	if err != nil {
		err = fmt.Errorf("Cannot finish signing data: %s", err)
		return
	}

	// Verify the signature
	pem.Encode(os.Stdout, &pem.Block{Type: "PKCS7", Bytes: signed})
	p7, err := pkcs7.Parse(signed)
	if err != nil {
		err = fmt.Errorf("Cannot parse our signed data: %s", err)
		return
	}

	// since the signature was detached, reattach the content here
	p7.Content = content

	if bytes.Compare(content, p7.Content) != 0 {
		err = fmt.Errorf("Our content was not in the parsed data:\n\tExpected: %s\n\tActual: %s", content, p7.Content)
		return
	}
	if err = p7.Verify(); err != nil {
		err = fmt.Errorf("Cannot verify our signed data: %s", err)
		return
	}

	return signed, nil
}

Credits

This is a fork of fullsailor/pkcs7

Overview (`vendor/github.com/digitorus/timestamp/README.md`)

RFC3161 Time-Stamp Protocol (TSP) package for Go

Time-Stamp Protocol (TSP) package for Go

General

The package timestamp implements the Time-Stamp Protocol (TSP) as specified in RFC3161 (Internet X.509 Public Key Infrastructure Time-Stamp Protocol (TSP)).

Overview (`vendor/github.com/dimchansky/utfbom/README.md`)

utfbom [](https://godoc.org/github.com/dimchansky/utfbom) [](https://opensource.org/licenses/Apache-2.0) [](https://travis-ci.org/dimchansky/utfbom) [](https://goreportcard.com/report/github.com/dimchansky/utfbom) [](https://coveralls.io/github/dimchansky/utfbom?branch=master)

The package utfbom implements the detection of the BOM (Unicode Byte Order Mark) and removing as necessary. It can also return the encoding detected by the BOM.

Installation

text

go get -u github.com/dimchansky/utfbom

Example

package main

import (
	"bytes"
	"fmt"
	"io/ioutil"

	"github.com/dimchansky/utfbom"
)

func main() {
	trySkip([]byte("\xEF\xBB\xBFhello"))
	trySkip([]byte("hello"))
}

func trySkip(byteData []byte) {
	fmt.Println("Input:", byteData)

	// just skip BOM
	output, err := ioutil.ReadAll(utfbom.SkipOnly(bytes.NewReader(byteData)))
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println("ReadAll with BOM skipping", output)

	// skip BOM and detect encoding
	sr, enc := utfbom.Skip(bytes.NewReader(byteData))
	fmt.Printf("Detected encoding: %s\n", enc)
	output, err = ioutil.ReadAll(sr)
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println("ReadAll with BOM detection and skipping", output)
	fmt.Println()
}

Output:

text

$ go run main.go
Input: [239 187 191 104 101 108 108 111]
ReadAll with BOM skipping [104 101 108 108 111]
Detected encoding: UTF8
ReadAll with BOM detection and skipping [104 101 108 108 111]

Input: [104 101 108 108 111]
ReadAll with BOM skipping [104 101 108 108 111]
Detected encoding: Unknown
ReadAll with BOM detection and skipping [104 101 108 108 111]

Overview (`vendor/github.com/distribution/reference/README.md`)

Distribution reference

Go library to handle references to container images.

This repository contains a library for handling references to container images held in container registries. Please see godoc for details.

Contribution

Please see CONTRIBUTING.md for details on how to contribute
issues, fixes, and patches to this project.

Communication

For async communication and long running discussions please use issues and pull requests on the github repo.
This will be the best place to discuss design and implementation.

For sync communication we have a #distribution channel in the CNCF Slack
that everyone is welcome to join and chat about development.

Licenses

The distribution codebase is released under the Apache 2.0 license.

Overview (`vendor/github.com/docker/distribution/README.md`)

Distribution

The Docker toolset to pack, ship, store, and deliver content.

This repository provides the Docker Registry 2.0 implementation
for storing and distributing Docker images. It supersedes the
docker/docker-registry
project with a new API design, focused around security and performance.

This repository contains the following components:

Component	Description
registry	An implementation of the Docker Registry HTTP API V2 for use with docker 1.6+.
libraries	A rich set of libraries for interacting with distribution components. Please see godoc for details. Note: These libraries are unstable.
specifications	Distribution related specifications are available in docs/spec
documentation	Docker's full documentation set is available at docs.docker.com. This repository contains the subset related just to the registry.

How does this integrate with Docker engine?

This project should provide an implementation to a V2 API for use in the Docker
core project. The API should be embeddable
and simplify the process of securely pulling and pushing content from docker
daemons.

What are the long term goals of the Distribution project?

The Distribution project has the further long term goal of providing a
secure tool chain for distributing content. The specifications, APIs and tools
should be as useful with Docker as they are without.

Our goal is to design a professional grade and extensible content distribution
system that allow users to:

Enjoy an efficient, secured and reliable way to store, manage, package and
exchange content
Hack/roll their own on top of healthy open-source components
Implement their own home made solution through good specs, and solid
extensions mechanism.

More about Registry 2.0

The new registry implementation provides the following benefits:

faster push and pull
new, more efficient implementation
simplified deployment
pluggable storage backend
webhook notifications

For information on upcoming functionality, please see ROADMAP.md.

Who needs to deploy a registry?

By default, Docker users pull images from Docker's public registry instance.
Installing Docker gives users this
ability. Users can also push images to a repository on Docker's public registry,
if they have a Docker Hub account.

For some users and even companies, this default behavior is sufficient. For
others, it is not.

For example, users with their own software products may want to maintain a
registry for private, company images. Also, you may wish to deploy your own
image repository for images used to test or in continuous integration. For these
use cases and others, deploying your own registry instance
may be the better choice.

Migration to Registry 2.0

For those who have previously deployed their own registry based on the Registry
1.0 implementation and wish to deploy a Registry 2.0 while retaining images,
data migration is required. A tool to assist with migration efforts has been
created. For more information see docker/migrator.

Contribute

Please see CONTRIBUTING.md for details on how to contribute
issues, fixes, and patches to this project. If you are contributing code, see
the instructions for building a development environment.

Support

If any issues are encountered while using the Distribution project, several
avenues are available for support:

IRC	#docker-distribution on FreeNode
Issue Tracker	github.com/docker/distribution/issues
Google Groups	https://groups.google.com/a/dockerproject.org/forum/#!forum/distribution
Mailing List	[email protected]

License

This project is distributed under Apache License, Version 2.0.

Overview (`vendor/github.com/docker/go-events/README.md`)

Docker Events Package

The Docker events package implements a composable event distribution package
for Go.

Originally created to implement the notifications in Docker Registry
2,
we've found the pattern to be useful in other applications. This package is
most of the same code with slightly updated interfaces. Much of the internals
have been made available.

Usage

The events package centers around a Sink type. Events are written with
calls to Sink.Write(event Event). Sinks can be wired up in various
configurations to achieve interesting behavior.

The canonical example is that employed by the
docker/distribution/notifications
package. Let's say we have a type httpSink where we'd like to queue
notifications. As a rule, it should send a single http request and return an
error if it fails:

func (h *httpSink) Write(event Event) error {
	p, err := json.Marshal(event)
	if err != nil {
		return err
	}
	body := bytes.NewReader(p)
	resp, err := h.client.Post(h.url, "application/json", body)
	if err != nil {
		return err
	}
	defer resp.Body.Close()
	
	if resp.Status != 200 {
		return errors.New("unexpected status")
	}

	return nil
}

// implement (*httpSink).Close()

With just that, we can start using components from this package. One can call
(*httpSink).Write to send events as the body of a post request to a
configured URL.

Retries

HTTP can be unreliable. The first feature we'd like is to have some retry:

hs := newHTTPSink(/*...*/)
retry := NewRetryingSink(hs, NewBreaker(5, time.Second))

We now have a sink that will retry events against the httpSink until they
succeed. The retry will backoff for one second after 5 consecutive failures
using the breaker strategy.

Queues

This isn't quite enough. We we want a sink that doesn't block while we are
waiting for events to be sent. Let's add a Queue:

queue := NewQueue(retry)

Now, we have an unbounded queue that will work through all events sent with
(*Queue).Write. Events can be added asynchronously to the queue without
blocking the current execution path. This is ideal for use in an http request.

Broadcast

It usually turns out that you want to send to more than one listener. We can
use Broadcaster to support this:

var broadcast = NewBroadcaster() // make it available somewhere in your application.
broadcast.Add(queue) // add your queue!
broadcast.Add(queue2) // and another!

With the above, we can now call broadcast.Write in our http handlers and have
all the events distributed to each queue. Because the events are queued, not
listener blocks another.

Extending

For the most part, the above is sufficient for a lot of applications. However,
extending the above functionality can be done implementing your own Sink. The
behavior and semantics of the sink can be completely dependent on the
application requirements. The interface is provided below for reference:

type Sink {
	Write(Event) error
	Close() error
}

Application behavior can be controlled by how Write behaves. The examples
above are designed to queue the message and return as quickly as possible.
Other implementations may block until the event is committed to durable
storage.

Copyright and license

Overview (`vendor/github.com/docker/go-metrics/README.md`)

go-metrics

This package is small wrapper around the prometheus go client to help enforce convention and best practices for metrics collection in Docker projects.

Best Practices

This packages is meant to be used for collecting metrics. It is not meant to be
used as a replacement for the prometheus client but to help enforce consistent
naming across metrics collected. If you have not already read the prometheus
best practices around naming and labels you can read the naming section in the
prometheus documentation.

The following are a few conventions that help naming metrics in your project.

Namespace and Subsystem

This package provides you with a namespace type that allows you to specify the
same namespace and subsystem for your metrics.

ns := metrics.NewNamespace("engine", "daemon", metrics.Labels{
        "version": daemon.Version,
        "commit":  daemon.GitCommit,
})

In the example above we are creating metrics for the daemon package.
engine would be the namespace in this example where daemon is the subsystem
or package where we are collecting the metrics.

A namespace also allows you to attach constant labels to the metrics such as
the git commit and version that it is collecting.

Declaring your Metrics

Try to keep all your metric declarations in one file. This makes it easy for
others to see what constant labels are defined on the namespace and what
labels are defined on the metrics when they are created.

Use labels instead of multiple metrics

Labels allow you to define one metric such as the time it takes to perform a
certain action on an object. If we wanted to collect timings on various container
actions such as create, start, and delete then we can define one metric called
container_actions and use labels to specify the type of action.

containerActions = ns.NewLabeledTimer("container_actions", "The number of milliseconds it takes to process each container action", "action")

The last parameter is the label name or key. When adding a data point to the
metric you can use the WithValues function to specify the action that you
are collecting for.

containerActions.WithValues("create").UpdateSince(start)

Always use a unit

The metric name should describe what you are measuring but you also need to
provide the unit that it is being measured with. For a timer, the standard
unit is seconds and a counter's standard unit is a total. For gauges you must
provide the unit. This package provides a standard set of units for use within
your projects.

Nanoseconds Unit = "nanoseconds"
Seconds     Unit = "seconds"
Bytes       Unit = "bytes"
Total       Unit = "total"

If you need to use a unit but it is not defined in the package please open
a PR to add it but first try to see if one of the existing units work for
your metric, i.e., seconds or nanoseconds vs adding milliseconds.

HTTP Metrics

To instrument a http handler, you can wrap the code like this:

namespace := metrics.NewNamespace("api_server", "http", metrics.Labels{"handler": "your_http_handler_name"})
httpMetrics := namespace.NewDefaultHttpMetrics()
metrics.Register(namespace)
instrumentedHandler = metrics.InstrumentHandler(httpMetrics, unInstrumentedHandler)

Note: The handler label must be provided when a new namespace is created.

Additional Metrics

Additional metrics are also defined here that are not available in the prometheus client.
If you need a custom metrics and it is generic enough to be used by multiple projects, define it here.

Copyright and license

Overview (`vendor/github.com/docker/go-units/README.md`)

Introduction

go-units is a library to transform human friendly measurements into machine friendly values.

Usage

See the docs in godoc for examples and documentation.

Copyright and license

go-units is licensed under the Apache License, Version 2.0.
See LICENSE for the full text of the license.

Overview (`vendor/github.com/felixge/httpsnoop/README.md`)

httpsnoop

Package httpsnoop provides an easy way to capture http related metrics (i.e.
response time, bytes written, and http status code) from your application's
http.Handlers.

Doing this requires non-trivial wrapping of the http.ResponseWriter interface,
which is also exposed for users interested in a more low-level API.

Usage Example

// myH is your app's http handler, perhaps a http.ServeMux or similar.
var myH http.Handler
// wrappedH wraps myH in order to log every request.
wrappedH := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	m := httpsnoop.CaptureMetrics(myH, w, r)
	log.Printf(
		"%s %s (code=%d dt=%s written=%d)",
		r.Method,
		r.URL,
		m.Code,
		m.Duration,
		m.Written,
	)
})
http.ListenAndServe(":8080", wrappedH)

Why this package exists

Instrumenting an application's http.Handler is surprisingly difficult.

However if you google for e.g. "capture ResponseWriter status code" you'll find
lots of advise and code examples that suggest it to be a fairly trivial
undertaking. Unfortunately everything I've seen so far has a high chance of
breaking your application.

The main problem is that a http.ResponseWriter often implements additional
interfaces such as http.Flusher, http.CloseNotifier, http.Hijacker, http.Pusher, and
io.ReaderFrom. So the naive approach of just wrapping http.ResponseWriter
in your own struct that also implements the http.ResponseWriter interface
will hide the additional interfaces mentioned above. This has a high change of
introducing subtle bugs into any non-trivial application.

Another approach I've seen people take is to return a struct that implements
all of the interfaces above. However, that's also problematic, because it's
difficult to fake some of these interfaces behaviors when the underlying
http.ResponseWriter doesn't have an implementation. It's also dangerous,
because an application may choose to operate differently, merely because it
detects the presence of these additional interfaces.

This package solves this problem by checking which additional interfaces a
http.ResponseWriter implements, returning a wrapped version implementing the
exact same set of interfaces.

Additionally this package properly handles edge cases such as WriteHeader not
being called, or called more than once, as well as concurrent calls to
http.ResponseWriter methods, and even calls happening after the wrapped
ServeHTTP has already returned.

Unfortunately this package is not perfect either. It's possible that it is
still missing some interfaces provided by the go core (let me know if you find
one), and it won't work for applications adding their own interfaces into the
mix. You can however use httpsnoop.Unwrap(w) to access the underlying
http.ResponseWriter and type-assert the result to its other interfaces.

However, hopefully the explanation above has sufficiently scared you of rolling
your own solution to this problem. httpsnoop may still break your application,
but at least it tries to avoid it as much as possible.

Anyway, the real problem here is that smuggling additional interfaces inside
http.ResponseWriter is a problematic design choice, but it probably goes as
deep as the Go language specification itself. But that's okay, I still prefer
Go over the alternatives ;).

Performance

text

BenchmarkBaseline-8      	   20000	     94912 ns/op
BenchmarkCaptureMetrics-8	   20000	     95461 ns/op

As you can see, using CaptureMetrics on a vanilla http.Handler introduces an
overhead of ~500 ns per http request on my machine. However, the margin of
error appears to be larger than that, therefor it should be reasonable to
assume that the overhead introduced by CaptureMetrics is absolutely
negligible.

License

MIT

Overview (`vendor/github.com/fsnotify/fsnotify/README.md`)

fsnotify is a Go library to provide cross-platform filesystem notifications on
Windows, Linux, macOS, BSD, and illumos.

Go 1.23 or newer is required; the full documentation is at
https://pkg.go.dev/github.com/fsnotify/fsnotify

Platform support:

Backend	OS	Status
inotify	Linux	Supported
kqueue	BSD, macOS	Supported
ReadDirectoryChangesW	Windows	Supported (excluding `Chmod` operations)
FEN	illumos	Supported
fanotify	Linux 5.9+	Not yet
FSEvents	macOS	Needs support in x/sys/unix
USN Journals	Windows	Needs support in x/sys/windows
Polling	All	Not yet

Linux and illumos should include Android and Solaris, but these are currently
untested.

Usage

A basic example:

package main

import (
    "log"

    "github.com/fsnotify/fsnotify"
)

func main() {
    // Create new watcher.
    watcher, err := fsnotify.NewWatcher()
    if err != nil {
        log.Fatal(err)
    }
    defer watcher.Close()

    // Start listening for events.
    go func() {
        for {
            select {
            case event, ok := <-watcher.Events:
                if !ok {
                    return
                }
                log.Println("event:", event)
                if event.Has(fsnotify.Write) {
                    log.Println("modified file:", event.Name)
                }
            case err, ok := <-watcher.Errors:
                if !ok {
                    return
                }
                log.Println("error:", err)
            }
        }
    }()

    // Add a path.
    err = watcher.Add("/tmp")
    if err != nil {
        log.Fatal(err)
    }

    // Block main goroutine forever.
    <-make(chan struct{})
}

Some more examples can be found in cmd/fsnotify, which can be
run with:

text

% go run ./cmd/fsnotify

Further detailed documentation can be found in godoc:
https://pkg.go.dev/github.com/fsnotify/fsnotify

FAQ

Will a file still be watched when it's moved to another directory?

No, not unless you are watching the location it was moved to.

Are subdirectories watched?

No, you must add watches for any directory you want to watch (a recursive
watcher is on the roadmap: #18).

Do I have to watch the Error and Event channels in a goroutine?

Yes. You can read both channels in the same goroutine using select (you don't
need a separate goroutine for both channels; see the example).

Why don't notifications work with NFS, SMB, FUSE, /proc, or /sys?

fsnotify requires support from underlying OS to work. The current NFS and SMB
protocols does not provide network level support for file notifications, and
neither do the /proc and /sys virtual filesystems.

This could be fixed with a polling watcher (#9), but it's not yet implemented.

Why do I get many Chmod events?

Some programs may generate a lot of attribute changes; for example Spotlight on
macOS, anti-virus programs, backup applications, and some others are known to do
this. As a rule, it's typically best to ignore Chmod events. They're often not
useful, and tend to cause problems.

Spotlight indexing on macOS can result in multiple events (see #15). A
temporary workaround is to add your folder(s) to the Spotlight Privacy
settings until we have a native FSEvents implementation (see #11).

Watching a file doesn't work well

Watching individual files (rather than directories) is generally not recommended
as many programs (especially editors) update files atomically: it will write to
a temporary file which is then moved to a destination, overwriting the original
(or some variant thereof). The watcher on the original file is now lost, as that
no longer exists.

The upshot of this is that a power failure or crash won't leave a half-written
file.

Watch the parent directory and use Event.Name to filter out files you're not
interested in. There is an example of this in cmd/fsnotify/file.go.

Platform-specific notes

Linux

When a file is removed a REMOVE event won't be emitted until all file
descriptors are closed; it will emit a CHMOD instead:

text

fp := os.Open("file")
os.Remove("file")        // CHMOD
fp.Close()               // REMOVE

This is the event that inotify sends, so not much can be changed about this.

The fs.inotify.max_user_watches sysctl variable specifies the upper limit for
the number of watches per user, and fs.inotify.max_user_instances specifies
the maximum number of inotify instances per user. Every Watcher you create is an
"instance", and every path you add is a "watch". Reaching the limit will result
in a "no space left on device" or "too many open files" error.

These are also exposed in /proc as /proc/sys/fs/inotify/max_user_watches and
/proc/sys/fs/inotify/max_user_instances. The default values differ per distro
and available memory.

To increase them you can use sysctl or write the value to proc file:

text

sysctl fs.inotify.max_user_watches=200000
sysctl fs.inotify.max_user_instances=256

To make the changes persist on reboot edit /etc/sysctl.conf or
/usr/lib/sysctl.d/50-default.conf (details differ per Linux distro; check your
distro's documentation):

text

fs.inotify.max_user_watches=200000
fs.inotify.max_user_instances=256

Windows

Recursive watching is not currently enabled through fsnotify's public API
(see the FAQ "Are subdirectories watched?" above). The notes below
describe Windows backend behavior observed when recursive watching is
enabled internally (for example, in fsnotify's own tests). They are kept
here as a reference for maintainers and contributors who encounter the
behavior, since the recursive code path still exists in the backend.

When recursive watching is enabled and you watch a directory, you may
receive a Write event for an intermediate directory whenever a child
entry inside it is created, renamed, or removed. For example, with a
recursive watch on /a and a new file /a/b/c, you will receive
Create /a/b/c and may also receive Write /a/b.

This happens because, on NTFS-backed volumes, modifying the entries of a
directory updates that directory's last-write time, and the Windows
backend requests FILE_NOTIFY_CHANGE_LAST_WRITE to support Write events
on files. The same Write filter therefore picks up the directory's
metadata update.

kqueue has the same "directory Write = directory contents changed"
semantics, so portable code that treats Write on a directory as
"something inside it changed" works on Windows and BSD/macOS, but not on
Linux (inotify uses Write only for file-content changes). If you only
care about file content, filter out Write events whose path refers to a
directory.

Whether the directory Write is actually delivered alongside the child
events is not guaranteed: it depends on ReadDirectoryChangesW buffering,
NTFS metadata update timing, and event coalescing, none of which fsnotify
controls.

kqueue (macOS, all BSD systems)

kqueue requires opening a file descriptor for every file that's being watched;
so if you're watching a directory with five files then that's six file
descriptors. You will run in to your system's "max open files" limit faster on
these platforms.

The sysctl variables kern.maxfiles and kern.maxfilesperproc can be used to
control the maximum number of open files.

Overview (`vendor/github.com/go-logr/logr/README.md`)

A minimal logging API for Go

logr offers an(other) opinion on how Go programs and libraries can do logging
without becoming coupled to a particular logging implementation. This is not
an implementation of logging - it is an API. In fact it is two APIs with two
different sets of users.

The Logger type is intended for application and library authors. It provides
a relatively small API which can be used everywhere you want to emit logs. It
defers the actual act of writing logs (to files, to stdout, or whatever) to the
LogSink interface.

The LogSink interface is intended for logging library implementers. It is a
pure interface which can be implemented by logging frameworks to provide the actual logging
functionality.

This decoupling allows application and library developers to write code in
terms of logr.Logger (which has very low dependency fan-out) while the
implementation of logging is managed "up stack" (e.g. in or near main().)
Application developers can then switch out implementations as necessary.

Many people assert that libraries should not be logging, and as such efforts
like this are pointless. Those people are welcome to convince the authors of
the tens-of-thousands of libraries that DO write logs that they are all
wrong. In the meantime, logr takes a more practical approach.

Typical usage

Somewhere, early in an application's life, it will make a decision about which
logging library (implementation) it actually wants to use. Something like:

text

func main() {
        // ... other setup code ...

        // Create the "root" logger.  We have chosen the "logimpl" implementation,
        // which takes some initial parameters and returns a logr.Logger.
        logger := logimpl.New(param1, param2)

        // ... other setup code ...

Most apps will call into other libraries, create structures to govern the flow,
etc. The logr.Logger object can be passed to these other libraries, stored
in structs, or even used as a package-global variable, if needed. For example:

text

app := createTheAppObject(logger)
    app.Run()

Outside of this early setup, no other packages need to know about the choice of
implementation. They write logs in terms of the logr.Logger that they
received:

text

type appObject struct {
        // ... other fields ...
        logger logr.Logger
        // ... other fields ...
    }

    func (app *appObject) Run() {
        app.logger.Info("starting up", "timestamp", time.Now())

        // ... app code ...

Background

If the Go standard library had defined an interface for logging, this project
probably would not be needed. Alas, here we are.

When the Go developers started developing such an interface with
slog, they adopted some of the
logr design but also left out some parts and changed others:

Feature	logr	slog
High-level API	`Logger` (passed by value)	`Logger` (passed by pointer)
Low-level API	`LogSink`	`Handler`
Stack unwinding	done by `LogSink`	done by `Logger`
Skipping helper functions	`WithCallDepth`, `WithCallStackHelper`	not supported by Logger
Generating a value for logging on demand	`Marshaler`	`LogValuer`
Log levels	>= 0, higher meaning "less important"	positive and negative, with 0 for "info" and higher meaning "more important"
Error log entries	always logged, don't have a verbosity level	normal log entries with level >= `LevelError`
Passing logger via context	`NewContext`, `FromContext`	no API
Adding a name to a logger	`WithName`	no API
Modify verbosity of log entries in a call chain	`V`	no API
Grouping of key/value pairs	not supported	`WithGroup`, `GroupValue`
Pass context for extracting additional values	no API	API variants like `InfoCtx`

The high-level slog API is explicitly meant to be one of many different APIs
that can be layered on top of a shared slog.Handler. logr is one such
alternative API, with interoperability provided by
some conversion functions.

Inspiration

Before you consider this package, please read this blog post by the
inimitable Dave Cheney. We really appreciate what
he has to say, and it largely aligns with our own experiences.

Differences from Dave's ideas

The main differences are:

Dave basically proposes doing away with the notion of a logging API in favor
of fmt.Printf(). We disagree, especially when you consider things like output
locations, timestamps, file and line decorations, and structured logging. This
package restricts the logging API to just 2 types of logs: info and error.

Info logs are things you want to tell the user which are not errors. Error
logs are, well, errors. If your code receives an error from a subordinate
function call and is logging that error and not returning it, use error
logs.

Verbosity-levels on info logs. This gives developers a chance to indicate
arbitrary grades of importance for info logs, without assigning names with
semantic meaning such as "warning", "trace", and "debug." Superficially this
may feel very similar, but the primary difference is the lack of semantics.
Because verbosity is a numerical value, it's safe to assume that an app running
with higher verbosity means more (and less important) logs will be generated.

Implementations (non-exhaustive)

There are implementations for the following logging libraries:

a function (can bridge to non-structured libraries): funcr
a testing.T (for use in Go tests, with JSON-like output): testr
github.com/google/glog: glogr
k8s.io/klog (for Kubernetes): klogr
a testing.T (with klog-like text output): ktesting
go.uber.org/zap: zapr
log (the Go standard library logger): stdr
github.com/sirupsen/logrus: logrusr
github.com/wojas/genericr: genericr (makes it easy to implement your own backend)
logfmt (Heroku style logging): logfmtr
github.com/rs/zerolog: zerologr
github.com/go-kit/log: gokitlogr (also compatible with github.com/go-kit/kit/log since v0.12.0)
bytes.Buffer (writing to a buffer): bufrlogr (useful for ensuring values were logged, like during testing)

slog interoperability

Interoperability goes both ways, using the logr.Logger API with a slog.Handler
and using the slog.Logger API with a logr.LogSink. FromSlogHandler and
ToSlogHandler convert between a logr.Logger and a slog.Handler.
As usual, slog.New can be used to wrap such a slog.Handler in the high-level
slog API.

Using a `logr.LogSink` as backend for slog

Ideally, a logr sink implementation should support both logr and slog by
implementing both the normal logr interface(s) and SlogSink. Because
of a conflict in the parameters of the common Enabled method, it is not
possible to implement both slog.Handler and logr.Sink in the same
type.

If both are supported, log calls can go from the high-level APIs to the backend
without the need to convert parameters. FromSlogHandler and ToSlogHandler can
convert back and forth without adding additional wrappers, with one exception:
when Logger.V was used to adjust the verbosity for a slog.Handler, then
ToSlogHandler has to use a wrapper which adjusts the verbosity for future
log calls.

Such an implementation should also support values that implement specific
interfaces from both packages for logging (logr.Marshaler, slog.LogValuer,
slog.GroupValue). logr does not convert those.

Not supporting slog has several drawbacks:

Recording source code locations works correctly if the handler gets called
through slog.Logger, but may be wrong in other cases. That's because a
logr.Sink does its own stack unwinding instead of using the program counter
provided by the high-level API.
slog levels <= 0 can be mapped to logr levels by negating the level without a
loss of information. But all slog levels > 0 (e.g. slog.LevelWarning as
used by slog.Logger.Warn) must be mapped to 0 before calling the sink
because logr does not support "more important than info" levels.
The slog group concept is supported by prefixing each key in a key/value
pair with the group names, separated by a dot. For structured output like
JSON it would be better to group the key/value pairs inside an object.
Special slog values and interfaces don't work as expected.
The overhead is likely to be higher.

These drawbacks are severe enough that applications using a mixture of slog and
logr should switch to a different backend.

Using a `slog.Handler` as backend for logr

Using a plain slog.Handler without support for logr works better than the
other direction:

All logr verbosity levels can be mapped 1:1 to their corresponding slog level
by negating them.
Stack unwinding is done by the SlogSink and the resulting program
counter is passed to the slog.Handler.
Names added via Logger.WithName are gathered and recorded in an additional
attribute with logger as key and the names separated by slash as value.
Logger.Error is turned into a log record with slog.LevelError as level
and an additional attribute with err as key, if an error was provided.

The main drawback is that logr.Marshaler will not be supported. Types should
ideally support both logr.Marshaler and slog.Valuer. If compatibility
with logr implementations without slog support is not important, then
slog.Valuer is sufficient.

Context support for slog

Storing a logger in a context.Context is not supported by
slog. NewContextWithSlogLogger and FromContextAsSlogLogger can be
used to fill this gap. They store and retrieve a slog.Logger pointer
under the same context key that is also used by NewContext and
FromContext for logr.Logger value.

When NewContextWithSlogLogger is followed by FromContext, the latter will
automatically convert the slog.Logger to a
logr.Logger. FromContextAsSlogLogger does the same for the other direction.

With this approach, binaries which use either slog or logr are as efficient as
possible with no unnecessary allocations. This is also why the API stores a
slog.Logger pointer: when storing a slog.Handler, creating a slog.Logger
on retrieval would need to allocate one.

The downside is that switching back and forth needs more allocations. Because
logr is the API that is already in use by different packages, in particular
Kubernetes, the recommendation is to use the logr.Logger API in code which
uses contextual logging.

An alternative to adding values to a logger and storing that logger in the
context is to store the values in the context and to configure a logging
backend to extract those values when emitting log entries. This only works when
log calls are passed the context, which is not supported by the logr API.

With the slog API, it is possible, but not
required. https://github.com/veqryn/slog-context is a package for slog which
provides additional support code for this approach. It also contains wrappers
for the context functions in logr, so developers who prefer to not use the logr
APIs directly can use those instead and the resulting code will still be
interoperable with logr.

FAQ

Conceptual

Why structured logging?

Structured logs are more easily queryable: Since you've got
key-value pairs, it's much easier to query your structured logs for
particular values by filtering on the contents of a particular key --
think searching request logs for error codes, Kubernetes reconcilers for
the name and namespace of the reconciled object, etc.
Structured logging makes it easier to have cross-referenceable logs:
Similarly to searchability, if you maintain conventions around your
keys, it becomes easy to gather all log lines related to a particular
concept.
Structured logs allow better dimensions of filtering: if you have
structure to your logs, you've got more precise control over how much
information is logged -- you might choose in a particular configuration
to log certain keys but not others, only log lines where a certain key
matches a certain value, etc., instead of just having v-levels and names
to key off of.
Structured logs better represent structured data: sometimes, the
data that you want to log is inherently structured (think tuple-link
objects.) Structured logs allow you to preserve that structure when
outputting.

Why V-levels?

V-levels give operators an easy way to control the chattiness of log
operations. V-levels provide a way for a given package to distinguish
the relative importance or verbosity of a given log message. Then, if
a particular logger or package is logging too many messages, the user
of the package can simply change the v-levels for that library.

Why not named levels, like Info/Warning/Error?

Read Dave Cheney's post. Then read Differences
from Dave's ideas.

Why not allow format strings, too?

Format strings negate many of the benefits of structured logs:

They're not easily searchable without resorting to fuzzy searching,
regular expressions, etc.
They don't store structured data well, since contents are flattened into
a string.
They're not cross-referenceable.
They don't compress easily, since the message is not constant.

(Unless you turn positional parameters into key-value pairs with numerical
keys, at which point you've gotten key-value logging with meaningless
keys.)

Practical

Why key-value pairs, and not a map?

Key-value pairs are much easier to optimize, especially around
allocations. Zap (a structured logger that inspired logr's interface) has
performance measurements
that show this quite nicely.

While the interface ends up being a little less obvious, you get
potentially better performance, plus avoid making users type
map[string]string{} every time they want to log.

What if my V-levels differ between libraries?

That's fine. Control your V-levels on a per-logger basis, and use the
WithName method to pass different loggers to different libraries.

Generally, you should take care to ensure that you have relatively
consistent V-levels within a given logger, however, as this makes deciding
on what verbosity of logs to request easier.

But I really want to use a format string!

That's not actually a question. Assuming your question is "how do
I convert my mental model of logging with format strings to logging with
constant messages":

Figure out what the error actually is, as you'd write in a TL;DR style,
and use that as a message.
For every place you'd write a format specifier, look to the word before
it, and add that as a key value pair.

For instance, consider the following examples (all taken from spots in the
Kubernetes codebase):

klog.V(4).Infof("Client is returning errors: code %v, error %v", responseCode, err) becomes logger.Error(err, "client returned an error", "code", responseCode)
klog.V(4).Infof("Got a Retry-After %ds response for attempt %d to %v", seconds, retries, url) becomes logger.V(4).Info("got a retry-after response when requesting url", "attempt", retries, "after seconds", seconds, "url", url)

If you really must use a format string, use it in a key's value, and
call fmt.Sprintf yourself. For instance: log.Printf("unable to reflect over type %T") becomes logger.Info("unable to reflect over type", "type", fmt.Sprintf("%T")). In general though, the cases where
this is necessary should be few and far between.

How do I choose my V-levels?

This is basically the only hard constraint: increase V-levels to denote
more verbose or more debug-y logs.

Otherwise, you can start out with 0 as "you always want to see this",
1 as "common logging that you might possibly want to turn off", and
10 as "I would like to performance-test your log collection stack."

Then gradually choose levels in between as you need them, working your way
down from 10 (for debug and trace style logs) and up from 1 (for chattier
info-type logs). For reference, slog pre-defines -4 for debug logs
(corresponds to 4 in logr), which matches what is
recommended for Kubernetes.

How do I choose my keys?

Keys are fairly flexible, and can hold more or less any string
value. For best compatibility with implementations and consistency
with existing code in other projects, there are a few conventions you
should consider.

Make your keys human-readable.
Constant keys are generally a good idea.
Be consistent across your codebase.
Keys should naturally match parts of the message string.
Use lower case for simple keys and
lowerCamelCase for
more complex ones. Kubernetes is one example of a project that has
adopted that
convention.

While key names are mostly unrestricted (and spaces are acceptable),
it's generally a good idea to stick to printable ascii characters, or at
least match the general character set of your log lines.

Why should keys be constant values?

The point of structured logging is to make later log processing easier. Your
keys are, effectively, the schema of each log message. If you use different
keys across instances of the same log line, you will make your structured logs
much harder to use. Sprintf() is for values, not for keys!

Why is this not a pure interface?

The Logger type is implemented as a struct in order to allow the Go compiler to
optimize things like high-V Info logs that are not triggered. Not all of
these implementations are implemented yet, but this structure was suggested as
a way to ensure they can be implemented. All of the real work is behind the
LogSink interface.

Overview (`vendor/github.com/go-logr/stdr/README.md`)

Minimal Go logging using logr and Go's standard library

This package implements the logr interface
in terms of Go's standard log package(https://pkg.go.dev/log).

Overview (`vendor/github.com/go-openapi/analysis/README.md`)

analysis

Top language Commits since latest release

A foundational library to analyze, diff, flatten, merge, and fix OAI specification documents for easier reasoning about the content.

Announcements

2025-12-19 : new community chat on discord
- a new discord community channel is available to be notified of changes and support users

You may join the discord community by clicking the invite link on the discord badge (also above).

Status

API is stable.

Import this library in your project

cmd

go get github.com/go-openapi/analysis

What's inside

An analyzer providing methods to walk the functional content of a specification
A spec flattener producing a self-contained document bundle, while preserving $refs
A spec differ ("diff") to compare two specs and report structural and compatibility changes
A spec merger ("mixin") to merge several spec documents into a primary spec
A spec "fixer" ensuring that response descriptions are non empty

FAQ

Does this library support OpenAPI 3?

No.
This package currently only supports OpenAPI 2.0 (aka Swagger 2.0).
There is no plan to make it evolve toward supporting OpenAPI 3.x.
This discussion thread relates the full story.

Change log

See https://github.com/go-openapi/analysis/releases

Licensing

This library ships under the SPDX-License-Identifier: Apache-2.0.

Cutting a new release

Maintainers can cut a new release by either:

running this workflow
or pushing a semver tag
- signed tags are preferred
- The tag message is prepended to release notes

Overview (`vendor/github.com/go-openapi/errors/README.md`)

errors

Top language Commits since latest release

Shared errors and error interface used throughout the various libraries found in the go-openapi toolkit.

Announcements

2025-12-19 : new community chat on discord
- a new discord community channel is available to be notified of changes and support users

You may join the discord community by clicking the invite link on the discord badge (also above).

Status

API is stable.

Import this library in your project

cmd

go get github.com/go-openapi/errors

Basic usage

const url = "https://www.example.com/#"

errGeneric := New(401,"onvalid argument: %s", url)

errNotFound := NotFound("resource not found: %s", url)

errNotImplemented := NotImplemented("method: %s", url)

Change log

See https://github.com/go-openapi/errors/releases

Licensing

This library ships under the SPDX-License-Identifier: Apache-2.0.

Cutting a new release

Maintainers can cut a new release by either:

running this workflow
or pushing a semver tag
- signed tags are preferred
- The tag message is prepended to release notes

Overview (`vendor/github.com/go-openapi/jsonpointer/README.md`)

jsonpointer

Top language Commits since latest release

An implementation of JSON Pointer for golang, which supports go struct.

Announcements

2026-07-07 : landing v1.0.0
- stable API pledge
2026-06-29 : reinsourced external dependency to swag (v0.24.0)
- module github.com/go-openapi/swag/jsonname is source directly here, so we no longer have any external dependency
- jsonname was never really used by any other package, so it makes sense to deprecate it away from the swag family
  and retrofit its functionality here. jsonpointer no longer get external dependencies, besides test dependencies.
2026-04-15 : added support for trailing "-" for arrays (v0.23.0)
- this brings full support of RFC6901
- this is supported for types relying on the reflection-based implemented
- API semantics remain essentially unaltered. Exception: Pointer.Set(document any,value any) (document any, err error)
  can only perform a best-effort to mutate the input document in place. In the case of adding elements to an array with a
  trailing "-", either pass a mutable array (*[]T) as the input document, or use the returned updated document instead.
- types that implement the JSONSetable interface may not implement the mutation implied by the trailing "-"
2026-04-15 : added support for optional alternate JSON name providers
- for struct support the defaults might not suit all situations: there are known limitations
  when it comes to handle untagged fields or embedded types.
- the default name provider in use is not fully aligned with go JSON stdlib
- exposed an option (or global setting) to change the provider that resolves a struct into json keys
- the default behavior is not altered

Status

API is stable and feature-complete.

The project continues to receive regular updates, bug fixes and hygiene maintenance (CI, linting, etc).

Import this library in your project

cmd

go get github.com/go-openapi/jsonpointer

Basic usage

Retrieving a value

import (
    "github.com/go-openapi/jsonpointer"
  )


  var doc any

  ...

	pointer, err := jsonpointer.New("/foo/1")
	if err != nil {
		... // error: e.g. invalid JSON pointer specification
	}

	value, kind, err := pointer.Get(doc)
	if err != nil {
		... // error: e.g. key not found, index out of bounds, etc.
	}

  ...

Setting a value

...
  var doc any
  ...
	pointer, err := jsonpointer.New("/foo/1")
	if err != nil {
		... // error: e.g. invalid JSON pointer specification
  }

	doc, err = p.Set(doc, "value")
	if err != nil {
		... // error: e.g. key not found, index out of bounds, etc.
	}

Change log

See https://github.com/go-openapi/jsonpointer/releases

References

https://tools.ietf.org/html/draft-ietf-appsawg-json-pointer-07

also known as RFC6901.

Licensing

This library ships under the SPDX-License-Identifier: Apache-2.0.

See the license NOTICE, which recalls the licensing terms of all the pieces of software
on top of which it has been built.

Limitations

RFC6901 is now fully supported, including trailing "-" semantics for arrays (for Set operations).
Default behavior: JSON name detection in go structs
- Unlike go standard marshaling, untagged fields do not default to the go field name and are ignored.
- anonymous fields are not traversed if untagged
- the above limitations may be overcome by calling UseGoNameProvider() at initialization time.
- alternatively, users may inject the desired custom behavior for naming fields as an option.

Cutting a new release

Maintainers can cut a new release by either:

running this workflow
or pushing a semver tag
- signed tags are preferred
- The tag message is prepended to release notes

Overview (`vendor/github.com/go-openapi/jsonreference/README.md`)

jsonreference

Top language Commits since latest release

An implementation of JSON Reference for golang.

Announcements

2026-07-07 : landing v1.0.0
- stable API pledge

Status

API is stable.

Import this library in your project

cmd

go get github.com/go-openapi/jsonreference

Dependencies

https://github.com/go-openapi/jsonpointer

Basic usage

// Creating a new reference
ref, err := jsonreference.New("http://example.com/doc.json#/definitions/Pet")

// Fragment-only reference
fragRef := jsonreference.MustCreateRef("#/definitions/Pet")

// Resolving references
parent, _ := jsonreference.New("http://example.com/base.json")
child, _ := jsonreference.New("#/definitions/Pet")
resolved, _ := parent.Inherits(child)
// Result: "http://example.com/base.json#/definitions/Pet"

Change log

See https://github.com/go-openapi/jsonreference/releases

References

Licensing

This library ships under the SPDX-License-Identifier: Apache-2.0.

See the license NOTICE, which recalls the licensing terms of all the pieces of software
on top of which it has been built.

Cutting a new release

Maintainers can cut a new release by either:

running this workflow
or pushing a semver tag
- signed tags are preferred
- The tag message is prepended to release notes

Overview (`vendor/github.com/go-openapi/loads/README.md`)

Loads OAI specs

Top language Commits since latest release

Loads OAI v2 API specification documents from local or remote locations.

Supports JSON and YAML documents.

Announcements

2025-12-19 : new community chat on discord
- a new discord community channel is available to be notified of changes and support users

You may join the discord community by clicking the invite link on the discord badge (also above).

Status

API is stable.

Import this library in your project

cmd

go get github.com/go-openapi/loads

Basic usage

import (
	  "github.com/go-openapi/loads"
  )

  ...

	// loads a YAML spec from a http file
	doc, err := loads.Spec(ts.URL)
  
  ...

  // retrieves the object model for the API specification
  spec := doc.Spec()

  ...

Security

This library does not enforce a security policy of its own: it reads whatever the configured
loader is allowed to read.

This is deliberate — like go-openapi/swag/loading, it is a base utility,
and sanitizing or containing untrusted input is the caller's responsibility,
just as sanitizing a file name before passing it to os.ReadFile is not that function's job.

When a spec — its path or its $ref contents — may come from an untrusted source, confine
loading explicitly (e.g. loading.WithRoot for local files and a restricted
loading.WithHTTPClient for remote URLs, passed via loads.WithLoadingOptions).

For the common case, the pre-baked loads.SpecRestricted / loads.JSONSpecRestricted loaders
bundle a trusted root with a network-restricted client (loads.RestrictedHTTPClient) and apply
the confinement to $ref resolution as well:

doc, err := loads.SpecRestricted(path, trustedRoot)

To harden the package-level default in one call — so even callers that rely on the global
loader (including cross-package $ref resolution via spec.PathLoader) are confined, with no
unconfined fallback left — use loads.SetRestrictedLoaders at startup:

loads.SetRestrictedLoaders(trustedRoot)

Note that loads.AddLoader only prepends to the default chain, leaving the unconfined loader
reachable; use loads.SetLoaders / loads.SetRestrictedLoaders to replace it.

See the Security section of the package documentation for the threat model and
runnable examples. For the project's vulnerability reporting policy, see SECURITY.md.

Change log

See https://github.com/go-openapi/loads/releases

Licensing

This library ships under the SPDX-License-Identifier: Apache-2.0.

Cutting a new release

Maintainers can cut a new release by either:

running this workflow
or pushing a semver tag
- signed tags are preferred
- The tag message is prepended to release notes

Overview (`vendor/github.com/go-openapi/runtime/README.md`)

runtime

[![Doc][doc-badge]][doc-url] [![GoDoc][godoc-badge]][godoc-url] [![Discord Channel][discord-badge]][discord-url] [![go version][goversion-badge]][goversion-url] ![Top language][top-badge] ![Commits since latest release][commits-badge]

A runtime for go OpenAPI toolkit.

The runtime component for use in code generation or as untyped usage.

Announcements

Complete documentation as github pages

Changes to the API surface in v0.30.0:

utility package header has now moved to github.com/go-openapi/runtime/server-middleware/negotiate/header

A shim is provided to support existing programs, with a deprecation notice.

Changes in semantics in v0.30.0:

Function negotiate.NegotiateContentType (available as an alias for backward compatibility as middleware.NegotiateContentType
now performs a full match considering MIME parameters.

The previous behavior (matching in order of appearance after stripping parameters) may be enabled explicitly with
option negotiate.WithIgnoreParameters.

2026-05-07 : exposed UI and Spec middleware as a separate, dependency-free module.

Newly available package: github.com/go-openapi/runtime/server-middleware/docui that now holds our
UI and spec serve middleware.

A shim is available in github.com/go-openapi/runtime/middleware to bridge the older UI options to the new ones,
with a deprecation notice.

Methods that were unduly exported and purely used to manipulate options (e.g. SwaggerUIOpts.EnsureDefaults) have been
removed. New options in docui should be used instead.

Users may reuse this middleware to serve a Redoc, Rapidoc or SwaggerUI documentation without
importing the complete go-openapi scaffolding.

2026-05-05 : exposed content negotiation methods as a separate, dependency-free module

Users may reuse these utilities to support content-negotiation without extra dependencies.

Newly available module: github.com/go-openapi/runtime/server-middleware

Newly available packages: github.com/go-openapi/runtime/server-middleware/negotiate and
github.com/go-openapi/runtime/server-middleware/mediatype.

Status

API is stable.

Import this library in your project

cmd

go get github.com/go-openapi/runtime

Change log

See https://github.com/go-openapi/runtime/releases

For v0.29.0 release see release notes.
From that release onwards, changes are tracked in the github release notes.

What coming next?

Moving forward, we want to :

fix a few known issues with some file upload requests (e.g. #286)
[] continue narrowing down the scope of dependencies:
- split middleware and other useful utilities as a separate dependency-free module
- yaml support in an independent module (v2)
- introduce more up-to-date support for opentelemetry as a separate module that evolves
  independently from the main package (to avoid breaking changes, the existing API
  will remain maintained, but evolve at a slower pace than opentelemetry). (v2)
[] publish proper documentation and examples

Licensing

This library ships under the SPDX-License-Identifier: Apache-2.0.

See the license NOTICE, which recalls the licensing terms of all the pieces of software
on top of which it has been built.

Cutting a new release

Maintainers can cut a new release by either:

running this workflow
or pushing a semver tag
- signed tags are preferred
- The tag message is prepended to release notes

Overview (`vendor/github.com/go-openapi/spec/README.md`)

spec

Top language Commits since latest release

The object model for OpenAPI v2 specification documents.

Announcements

2025-12-19 : new community chat on discord
- a new discord community channel is available to be notified of changes and support users

You may join the discord community by clicking the invite link on the discord badge (also above).

Status

API is stable.

Import this library in your project

cmd

go get github.com/go-openapi/spec

FAQ

What does this do?

This package knows how to marshal and unmarshal Swagger API specifications into a golang object model

It knows how to resolve $ref and expand them to make a single root document

How does it play with the rest of the go-openapi packages ?

This package is at the core of the go-openapi suite of packages and code generator

There is a spec loading package to fetch specs as JSON or YAML from local or remote locations

There is a spec validation package built on top of it

There is a spec analysis package built on top of it, to analyze, flatten, fix and merge spec documents

Does this library support OpenAPI 3?

No.
This package currently only supports OpenAPI 2.0 (aka Swagger 2.0).
There is no plan to make it evolve toward supporting OpenAPI 3.x.
This discussion thread relates the full story.

An early attempt to support Swagger 3 may be found at: https://github.com/go-openapi/spec3

Does the unmarshaling support YAML?

Not directly. The exposed types know only how to unmarshal from JSON.

In order to load a YAML document as a Swagger spec, you need to use the loaders provided by
github.com/go-openapi/loads

Take a look at the example there: https://pkg.go.dev/github.com/go-openapi/loads#example-Spec

See also https://github.com/go-openapi/spec/issues/164

How can I validate a spec?

Validation is provided by the validate package

Why do we have an ID field for Schema which is not part of the swagger spec?

We found jsonschema compatibility more important: since id in jsonschema influences
how $ref are resolved.
This id does not conflict with any property named id.

See also https://github.com/go-openapi/spec/issues/23

Change log

See https://github.com/go-openapi/spec/releases

References

https://github.com/OAI/OpenAPI-Specification/blob/main/versions/2.0.md

Licensing

This library ships under the SPDX-License-Identifier: Apache-2.0.

Cutting a new release

Maintainers can cut a new release by either:

running this workflow
or pushing a semver tag
- signed tags are preferred
- The tag message is prepended to release notes

Overview (`vendor/github.com/go-openapi/strfmt/README.md`)

strfmt

Top language Commits since latest release

Golang support for string formats defined by JSON Schema and OpenAPI.

Announcements

2026-03-07 : v0.26.0 dropped dependency to the mongodb driver
- mongodb users can still use this package without any change
- however, we have frozen the back-compatible support for mongodb driver at v2.5.0
- users who want to keep-up with future evolutions (possibly incompatible) of this driver
  can do so by adding a blank import in their program: import _ "github.com/go-openapi/strfmt/enable/mongodb".
  This will switch the behavior to the actual driver, which remains regularly updated as an independent module.

Status

API is stable.

Import this library in your project

cmd

go get github.com/go-openapi/strfmt

This package exposes a registry of data types to support string formats in the go-openapi toolkit.

strfmt represents a well known string format such as hostname or email.

This package provides a few extra formats such as credit card (US), color, etc.

Format types can serialize and deserialize JSON or from a SQL database.

BSON is also supported (MongoDB).

Supported formats

go-openapi/strfmt follows the swagger 2.0 specification with the following formats
defined here.

It also provides convenient extensions to go-openapi users.

JSON-schema draft 4 formats
- date-time
- email
- hostname
- ipv4
- ipv6
- uri
swagger 2.0 format extensions
- binary
- byte (e.g. base64 encoded string)
- date (e.g. "1970-01-01")
- password
go-openapi custom format extensions
- bsonobjectid (BSON objectID)
- creditcard
- duration (e.g. "3 weeks", "1ms") (aka "duration-human")
- hexcolor (e.g. "#FFFFFF")
- isbn, isbn10, isbn13
- mac (e.g "01:02:03:04:05:06")
- rgbcolor (e.g. "rgb(100,100,100)")
- ssn
- uuid, uuid3, uuid4, uuid5, uuid7
- cidr (e.g. "192.0.2.1/24", "2001:db8:a0b:12f0::1/32")
- ulid (e.g. "00000PP9HGSBSSDZ1JTEXBJ0PW", spec)
JSON-schema draft 2020 formats
- duration-iso8601 B(e.g. "P2W")

NOTE: as the name stands for, this package is intended to support string formatting only.
It does not provide validation for numerical values with swagger format extension for JSON types "number" or
"integer" (e.g. float, double, int32...).

Durations

We have 2 very different definitions of the "duration" format: the "human-readable" duration that used to be just "duration",
and the new "duration-iso8601". There is no "dual" parser that accepts both formats: types are specialized.

To clarify the situation, a new alias for the duration format is introduced "duration-human" (e.g. "1 ms"), as opposed to
"duration-iso8601".

The Default format registry wires "duration-human" as the default mapping for "duration"
(preexisting behavior, no breaking change - aligned with Swagger 2.0 which did not define "duration").

A new JSONSchema2020 registry wires "duration-iso8601" as the default mapping for "duration".

Type conversion

All types defined here are stringers and may be converted to strings with .String().
Note that most types defined by this package may be converted directly to string like string(Email{}).

Date and DateTime may be converted directly to time.Time like time.Time(Time{}).
Similarly, you can convert Duration to time.Duration as in time.Duration(Duration{})

Using pointers

The conv subpackage provides helpers to convert the types to and from pointers, just like go-openapi/swag does
with primitive types.

Format types

Types defined in strfmt expose marshaling and validation capabilities.

List of defined types:

Base64
CreditCard
Date
DateTime
Duration
DurationISO8601 and ISODuration[P ISODurationPolicy] (for optional behavior)
Email
HexColor
Hostname
IPv4
IPv6
CIDR
ISBN
ISBN10
ISBN13
MAC
ObjectId
Password
RGBColor
SSN
URI
UUID
UUID3
UUID4
UUID5
UUID7
ULID

Database support

All format types implement the database/sql interfaces sql.Scanner and driver.Valuer,
so they work out of the box with Go's standard database/sql package and any SQL driver.

All format types also implement BSON marshaling/unmarshaling for use with MongoDB.
By default, a built-in minimal codec is used (compatible with mongo-driver v2.5.0).
For full driver support, add import _ "github.com/go-openapi/strfmt/enable/mongodb".

MySQL / MariaDB caveat for DateTime:
The go-sql-driver/mysql driver has hard-coded handling for time.Time but does not
intercept type redefinitions like strfmt.DateTime. As a result, DateTime.Value() sends
an RFC 3339 string (e.g. "2024-06-15T12:30:45.123Z") that MySQL/MariaDB rejects for
DATETIME columns.

Workaround: set strfmt.MarshalFormat to a MySQL-compatible format such as
strfmt.ISO8601LocalTime and normalize to UTC before marshaling:
go
strfmt.MarshalFormat = strfmt.ISO8601LocalTime
strfmt.NormalizeTimeForMarshal = func(t time.Time) time.Time { return t.UTC() }
See #174 for details.

Integration tests for MongoDB, MariaDB, and PostgreSQL run in CI to verify database roundtrip
compatibility for all format types. See internal/testintegration/.

Change log

See https://github.com/go-openapi/strfmt/releases

References

https://github.com/OAI/OpenAPI-Specification/blob/main/versions/2.0.md

Licensing

This library ships under the SPDX-License-Identifier: Apache-2.0.

Cutting a new release

Maintainers can cut a new release by either:

running this workflow
or pushing a semver tag
- signed tags are preferred
- The tag message is prepended to release notes

Overview (`vendor/github.com/go-openapi/swag/README.md`)

Swag

Top language Commits since latest release

A bunch of helper functions for go-openapi and go-swagger projects.

You may also use it standalone for your projects.

NOTE
swag is one of the foundational building blocks of the go-openapi initiative.

Most repositories in github.com/go-openapi/... depend on it in some way.
And so does our CLI tool github.com/go-swagger/go-swagger,
as well as the code generated by this tool.

Contents
Dependencies
Change log
Licensing
Note to contributors
Roadmap

Announcements

2025-12-19 : new community chat on discord
- a new discord community channel is available to be notified of changes and support users

You may join the discord community by clicking the invite link on the discord badge (also above).

Status

API is stable.

Import this library in your project

cmd

go get github.com/go-openapi/swag/{module}

Or for backward compatibility:

cmd

go get github.com/go-openapi/swag

go-openapi/swag exposes a collection of relatively independent modules.

Moving forward, no additional feature will be added to the swag API directly at the root package level,
which remains there for backward-compatibility purposes. All exported top-level features are now deprecated.

Child modules will continue to evolve and some new ones may be added in the future.

Module	Content	Main features
`cmdutils`	utilities to work with CLIs
`conv`	type conversion utilities	convert between values and pointers for any typesconvert from string to builtin types (wraps `strconv`)require `./typeutils` (test dependency)
`fileutils`	file utilities
`jsonname`	JSON utilities (deprecated)	infer JSON names from `go` propertiesuse `github.com/go-openapi/jsonpointer/jsonname` instead
`jsonutils`	JSON utilities	fast json concatenationread and write JSON from and to dynamic `go` data structures~~require `github.com/mailru/easyjson`~~
`loading`	file loading	load from file or httprequire `./yamlutils`
`mangling`	safe name generation	name mangling for `go`
`netutils`	networking utilities	host, port from address
`pools`	utilities to work with sync.Pools
`stringutils`	`string` utilities	search in slice (with case-insensitive)split/join query parameters as arrays
`typeutils`	`go` types utilities	check the zero value for any typesafe check for a nil value
`yamlutils`	YAML utilities	converting YAML to JSONloading YAML into a dynamic YAML documentmaintaining the original order of keys in YAML objectsrequire `./jsonutils`~require `github.com/mailru/easyjson`~require `go.yaml.in/yaml/v3`

Dependencies

The root module github.com/go-openapi/swag at the repo level maintains a few
dependencies outside of the standard library.

YAML utilities depend on go.yaml.in/yaml/v3
JSON utilities depend on their registered adapter module:
- by default, only the standard library is used
- github.com/mailru/easyjson is now only a dependency for module
  github.com/go-openapi/swag/jsonutils/adapters/easyjson/json,
  for users willing to import that module.
- integration tests and benchmarks use all the dependencies are published as their own module
other dependencies are test dependencies drawn from github.com/stretchr/testify

Usage

How to explicitly register a dependency at runtime?

The following would maintain how JSON utilities proposed by swag used work, up to v0.24.1.

import (
  "github.com/go-openapi/swag/jsonutils/adapters"
  easyjson "github.com/go-openapi/swag/jsonutils/adapters/easyjson/json"
)

func init() {
 easyjson.Register(adapters.Registry)
}

Subsequent calls to jsonutils.ReadJSON() or jsonutils.WriteJSON() will switch to easyjson
whenever the passed data structures implement the easyjson.Unmarshaler or easyjson.Marshaler respectively,
or fallback to the standard library.

For more details, you may also look at our
integration tests.

Note to contributors

All kinds of contributions are welcome.

This repo is a go mono-repo. See docs.

More general guidelines are available here.

Roadmap

See the current TODO list

Change log

See https://github.com/go-openapi/swag/releases

For pre-v0.26.0 releases, see release notes.

What coming next?

Moving forward, we want to :

provide an implementation of the JSON adapter based on encoding/json/v2, for go1.25 builds.
provide similar implementations for goccy/go-json and jsoniterator/go, and perhaps some other
similar libraries may be interesting too.

Licensing

This library ships under the SPDX-License-Identifier: Apache-2.0.

Cutting a new release

Maintainers can cut a new release by either:

running this workflow
or pushing a semver tag
- signed tags are preferred
- The tag message is prepended to release notes

Overview (`vendor/github.com/go-openapi/validate/README.md`)

validate

Top language Commits since latest release

A validator for OpenAPI v2 specifications and JSON schema draft 4.

Announcements

2025-12-19 : new community chat on discord
- a new discord community channel is available to be notified of changes and support users

You may join the discord community by clicking the invite link on the discord badge (also above).

Status

API is stable.

Import this library in your project

cmd

go get github.com/go-openapi/validate

This package provides helpers to validate Swagger 2.0. specification (aka OpenAPI 2.0).

A validator for Swagger specifications
A validator for JSON schemas draft4
Helper functions to validate individual values (used by code generated by go-swagger).
- Required, RequiredNumber, RequiredString
- ReadOnly
- UniqueItems, MaxItems, MinItems
- Enum, EnumCase
- Pattern, MinLength, MaxLength
- Minimum, Maximum, MultipleOf
- FormatOf

FAQ

Does this library support OpenAPI 3?

No.
This package currently only supports OpenAPI 2.0 (aka Swagger 2.0).
There is no plan to make it evolve toward supporting OpenAPI 3.x.
This discussion thread relates the full story.

An early attempt to support Swagger 3 may be found at: https://github.com/go-openapi/spec3

Change log

See https://github.com/go-openapi/validate/releases

References

https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md

Licensing

This library ships under the SPDX-License-Identifier: Apache-2.0.

Cutting a new release

Maintainers can cut a new release by either:

running this workflow
or pushing a semver tag
- signed tags are preferred
- The tag message is prepended to release notes

Overview (`vendor/github.com/gofrs/flock/README.md`)

flock

flock implements a thread-safe file lock.

It also includes a non-blocking TryLock() function to allow locking without blocking execution.

Installation

bash

go get -u github.com/gofrs/flock

Usage

import "github.com/gofrs/flock"

fileLock := flock.New("/var/lock/go-lock.lock")

locked, err := fileLock.TryLock()

if err != nil {
	// handle locking error
}

if locked {
	// do work
	fileLock.Unlock()
}

For more detailed usage information take a look at the package API docs on
GoDoc.

License

flock is released under the BSD 3-Clause License. See the LICENSE file for more details.

Project History

This project was originally github.com/theckman/go-flock, it was transferred to Gofrs by the original author Tim Heckman .

Overview (`vendor/github.com/gohugoio/hashstructure/README.md`)

hashstructure [](https://godoc.org/github.com/mitchellh/hashstructure)

hashstructure is a Go library for creating a unique hash value
for arbitrary values in Go.

This can be used to key values in a hash (for use in a map, set, etc.)
that are complex. The most common use case is comparing two values without
sending data across the network, caching values locally (de-dup), and so on.

Features

Hash any arbitrary Go value, including complex types.
Tag a struct field to ignore it and not affect the hash value.
Tag a slice type struct field to treat it as a set where ordering
doesn't affect the hash code but the field itself is still taken into
account to create the hash value.
Optionally, specify a custom hash function to optimize for speed, collision
avoidance for your data set, etc.
Optionally, hash the output of .String() on structs that implement fmt.Stringer,
allowing effective hashing of time.Time
Optionally, override the hashing process by implementing Hashable.

Installation

Standard go get:

text

$ go get github.com/gohugoio/hashstructure

Usage & Example

For usage and examples see the Godoc.

A quick code example is shown below:

type ComplexStruct struct {
    Name     string
    Age      uint
    Metadata map[string]interface{}
}

v := ComplexStruct{
    Name: "mitchellh",
    Age:  64,
    Metadata: map[string]interface{}{
        "car":      true,
        "location": "California",
        "siblings": []string{"Bob", "John"},
    },
}

hash, err := hashstructure.Hash(v, nil)
if err != nil {
    panic(err)
}

fmt.Printf("%d", hash)
// Output:
// 2307517237273902113

Overview (`vendor/github.com/google/btree/README.md`)

BTree implementation for Go

This package provides an in-memory B-Tree implementation for Go, useful as
an ordered, mutable data structure.

The API is based off of the wonderful
http://godoc.org/github.com/petar/GoLLRB/llrb, and is meant to allow btree to
act as a drop-in replacement for gollrb trees.

See http://godoc.org/github.com/google/btree for documentation.

Overview (`vendor/github.com/google/certificate-transparency-go/README.md`)

Certificate Transparency: Go Code

This repository holds Go code related to
Certificate Transparency (CT). The
repository requires Go version 1.24.

Repository Structure
Trillian CT Personality
Working on the Code
- Running Codebase Checks
- Rebuilding Generated Code

Support

Slack: https://transparency-dev.slack.com/ (invitation)

Repository Structure

The main parts of the repository are:

Encoding libraries:
- asn1/ and x509/ are forks of the upstream Go encoding/asn1 and
  crypto/x509 libraries. We maintain separate forks of these packages
  because CT is intended to act as an observatory of certificates across the
  ecosystem; as such, we need to be able to process somewhat-malformed
  certificates that the stricter upstream code would (correctly) reject.
  Our x509 fork also includes code for working with the
  pre-certificates defined in RFC 6962.
- tls holds a library for processing TLS-encoded data as described in
  RFC 5246.
- x509util/ provides additional utilities for dealing with
  x509.Certificates.
CT client libraries:
- The top-level ct package (in .) holds types and utilities for working
  with CT data structures defined in
  RFC 6962.
- client/ and jsonclient/ hold libraries that allow access to CT Logs
  via HTTP entrypoints described in
  section 4 of RFC 6962.
- dnsclient/ has a library that allows access to CT Logs over
  DNS.
- scanner/ holds a library for scanning the entire contents of an existing
  CT Log.
CT Personality for Trillian:
- trillian/ holds code that allows a Certificate Transparency Log to be
  run using a Trillian Log as its back-end -- see
  below.
Command line tools:
- ./client/ctclient allows interaction with a CT Log.
- ./ctutil/sctcheck allows SCTs (signed certificate timestamps) from a CT
  Log to be verified.
- ./scanner/scanlog allows an existing CT Log to be scanned for certificates
  of interest; please be polite when running this tool against a Log.
- ./x509util/certcheck allows display and verification of certificates
- ./x509util/crlcheck allows display and verification of certificate
  revocation lists (CRLs).
Other libraries related to CT:
- ctutil/ holds utility functions for validating and verifying CT data
  structures.
- loglist3/ has a library for reading
  v3 JSON lists of CT Logs.

Trillian CT Personality

The trillian/ subdirectory holds code and scripts for running a CT Log based
on the Trillian general transparency Log,
and is documented separately.

Working on the Code

Developers who want to make changes to the codebase need some additional
dependencies and tools, described in the following sections.

Running Codebase Checks

The scripts/presubmit.sh script runs various tools
and tests over the codebase; please ensure this script passes before sending
pull requests for review.

bash

# Install golangci-lint
go install github.com/golangci/golangci-lint/v2/cmd/[email protected]

# Run code generation, build, test and linters
./scripts/presubmit.sh

# Run build, test and linters but skip code generation
./scripts/presubmit.sh  --no-generate

# Or just run the linters alone:
golangci-lint run

Rebuilding Generated Code

Some of the CT Go code is autogenerated from other files:

Protocol buffer message
definitions are converted to .pb.go implementations.
A mock implementation of the Trillian gRPC API (in trillian/mockclient) is
created with GoMock.
Some enums have string-conversion methods (satisfying the fmt.Stringer
interface) created using the
stringer tool (go get golang.org/x/tools/cmd/stringer).

Re-generating mock or protobuffer files is only needed if you're changing
the original files. The recommended way to do this is by using the Docker
image used by the Cloud Build:

shell

docker build -f ./integration/Dockerfile -t ctgo-builder .
docker run -it --mount type=bind,src="$(pwd)",target=/src ctgo-builder /bin/bash -c "cd /src; ./scripts/install_deps.sh; go generate -x ./..."

These commands first create a docker image from the Dockerfile in this repo, and
then launch a container based on this image with the local directory mounted. The
correct versions of the tools are determined using the go.mod file in this repo,
and these tools are installed. Finally, all of the generated files are regenerated
and Docker exits.

Alternatively, you can install the prerequisites locally:

a series of tools, using go install to ensure that the versions are
compatible and tested:

text

cd $(go list -f '{{ .Dir }}' github.com/google/certificate-transparency-go); \
go install github.com/golang/mock/mockgen; \
go install google.golang.org/protobuf/proto; \
go install google.golang.org/protobuf/cmd/protoc-gen-go; \
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc; \
go install github.com/pseudomuto/protoc-gen-doc/cmd/protoc-gen-doc; \
go install golang.org/x/tools/cmd/stringer

protoc tool: you'll need version 3.20.1 installed, and PATH updated to include its bin/ directory.

and run the following:

bash

go generate -x ./...  # hunts for //go:generate comments and runs them

Overview (`vendor/github.com/google/s2a-go/README.md`)

Secure Session Agent Client Libraries

The Secure Session Agent is a service that enables a workload to offload select
operations from the mTLS handshake and protects a workload's private key
material from exfiltration. Specifically, the workload asks the Secure Session
Agent for the TLS configuration to use during the handshake, to perform private
key operations, and to validate the peer certificate chain. The Secure Session
Agent's client libraries enable applications to communicate with the Secure
Session Agent during the TLS handshake, and to encrypt traffic to the peer
after the TLS handshake is complete.

This repository contains the source code for the Secure Session Agent's Go
client libraries, which allow gRPC and HTTP Go applications to use the Secure Session
Agent.

Overview (`vendor/github.com/google/uuid/README.md`)

uuid

The uuid package generates and inspects UUIDs based on
RFC 4122
and DCE 1.1: Authentication and Security Services.

This package is based on the github.com/pborman/uuid package (previously named
code.google.com/p/go-uuid). It differs from these earlier packages in that
a UUID is a 16 byte array rather than a byte slice. One loss due to this
change is the ability to represent an invalid UUID (vs a NIL UUID).

Install

go get github.com/google/uuid

Documentation

Full go doc style documentation for the package can be viewed online without
installing this package by using the GoDoc site here:
http://pkg.go.dev/github.com/google/uuid

Overview (`vendor/github.com/gorilla/mux/README.md`)

gorilla/mux

Package gorilla/mux implements a request router and dispatcher for matching incoming requests to
their respective handler.

The name mux stands for "HTTP request multiplexer". Like the standard http.ServeMux, mux.Router matches incoming requests against a list of registered routes and calls a handler for the route that matches the URL or other conditions. The main features are:

It implements the http.Handler interface so it is compatible with the standard http.ServeMux.
Requests can be matched based on URL host, path, path prefix, schemes, header and query values, HTTP methods or using custom matchers.
URL hosts, paths and query values can have variables with an optional regular expression.
Registered URLs can be built, or "reversed", which helps maintaining references to resources.
Routes can be used as subrouters: nested routes are only tested if the parent route matches. This is useful to define groups of routes that share common conditions like a host, a path prefix or other repeated attributes. As a bonus, this optimizes request matching.

Install
Examples
Matching Routes
Static Files
Serving Single Page Applications (e.g. React, Vue, Ember.js, etc.)
Registered URLs
Walking Routes
Graceful Shutdown
Middleware
Handling CORS Requests
Testing Handlers
Full Example

Install

With a correctly configured Go toolchain:

go get -u github.com/gorilla/mux

Examples

Let's start registering a couple of URL paths and handlers:

func main() {
    r := mux.NewRouter()
    r.HandleFunc("/", HomeHandler)
    r.HandleFunc("/products", ProductsHandler)
    r.HandleFunc("/articles", ArticlesHandler)
    http.Handle("/", r)
}

Here we register three routes mapping URL paths to handlers. This is equivalent to how http.HandleFunc() works: if an incoming request URL matches one of the paths, the corresponding handler is called passing (http.ResponseWriter, *http.Request) as parameters.

Paths can have variables. They are defined using the format {name} or {name:pattern}. If a regular expression pattern is not defined, the matched variable will be anything until the next slash. For example:

r := mux.NewRouter()
r.HandleFunc("/products/{key}", ProductHandler)
r.HandleFunc("/articles/{category}/", ArticlesCategoryHandler)
r.HandleFunc("/articles/{category}/{id:[0-9]+}", ArticleHandler)

The names are used to create a map of route variables which can be retrieved calling mux.Vars():

func ArticlesCategoryHandler(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    w.WriteHeader(http.StatusOK)
    fmt.Fprintf(w, "Category: %v\n", vars["category"])
}

And this is all you need to know about the basic usage. More advanced options are explained below.

Matching Routes

Routes can also be restricted to a domain or subdomain. Just define a host pattern to be matched. They can also have variables:

r := mux.NewRouter()
// Only matches if domain is "www.example.com".
r.Host("www.example.com")
// Matches a dynamic subdomain.
r.Host("{subdomain:[a-z]+}.example.com")

There are several other matchers that can be added. To match path prefixes:

r.PathPrefix("/products/")

...or HTTP methods:

r.Methods("GET", "POST")

...or URL schemes:

r.Schemes("https")

...or header values:

r.Headers("X-Requested-With", "XMLHttpRequest")

...or query values:

r.Queries("key", "value")

...or to use a custom matcher function:

r.MatcherFunc(func(r *http.Request, rm *RouteMatch) bool {
    return r.ProtoMajor == 0
})

...and finally, it is possible to combine several matchers in a single route:

r.HandleFunc("/products", ProductsHandler).
  Host("www.example.com").
  Methods("GET").
  Schemes("http")

Routes are tested in the order they were added to the router. If two routes match, the first one wins:

r := mux.NewRouter()
r.HandleFunc("/specific", specificHandler)
r.PathPrefix("/").Handler(catchAllHandler)

Setting the same matching conditions again and again can be boring, so we have a way to group several routes that share the same requirements. We call it "subrouting".

For example, let's say we have several URLs that should only match when the host is www.example.com. Create a route for that host and get a "subrouter" from it:

r := mux.NewRouter()
s := r.Host("www.example.com").Subrouter()

Then register routes in the subrouter:

s.HandleFunc("/products/", ProductsHandler)
s.HandleFunc("/products/{key}", ProductHandler)
s.HandleFunc("/articles/{category}/{id:[0-9]+}", ArticleHandler)

The three URL paths we registered above will only be tested if the domain is www.example.com, because the subrouter is tested first. This is not only convenient, but also optimizes request matching. You can create subrouters combining any attribute matchers accepted by a route.

Subrouters can be used to create domain or path "namespaces": you define subrouters in a central place and then parts of the app can register its paths relatively to a given subrouter.

There's one more thing about subroutes. When a subrouter has a path prefix, the inner routes use it as base for their paths:

r := mux.NewRouter()
s := r.PathPrefix("/products").Subrouter()
// "/products/"
s.HandleFunc("/", ProductsHandler)
// "/products/{key}/"
s.HandleFunc("/{key}/", ProductHandler)
// "/products/{key}/details"
s.HandleFunc("/{key}/details", ProductDetailsHandler)

Static Files

Note that the path provided to PathPrefix() represents a "wildcard": calling
PathPrefix("/static/").Handler(...) means that the handler will be passed any
request that matches "/static/*". This makes it easy to serve static files with mux:

func main() {
    var dir string

    flag.StringVar(&dir, "dir", ".", "the directory to serve files from. Defaults to the current dir")
    flag.Parse()
    r := mux.NewRouter()

    // This will serve files under http://localhost:8000/static/<filename>
    r.PathPrefix("/static/").Handler(http.StripPrefix("/static/", http.FileServer(http.Dir(dir))))

    srv := &http.Server{
        Handler:      r,
        Addr:         "127.0.0.1:8000",
        // Good practice: enforce timeouts for servers you create!
        WriteTimeout: 15 * time.Second,
        ReadTimeout:  15 * time.Second,
    }

    log.Fatal(srv.ListenAndServe())
}

Serving Single Page Applications

Most of the time it makes sense to serve your SPA on a separate web server from your API,
but sometimes it's desirable to serve them both from one place. It's possible to write a simple
handler for serving your SPA (for use with React Router's BrowserRouter for example), and leverage
mux's powerful routing for your API endpoints.

text

/* Detailed source-code truncated for AI context efficiency. */

Registered URLs

Now let's see how to build registered URLs.

Routes can be named. All routes that define a name can have their URLs built, or "reversed". We define a name calling Name() on a route. For example:

r := mux.NewRouter()
r.HandleFunc("/articles/{category}/{id:[0-9]+}", ArticleHandler).
  Name("article")

To build a URL, get the route and call the URL() method, passing a sequence of key/value pairs for the route variables. For the previous route, we would do:

url, err := r.Get("article").URL("category", "technology", "id", "42")

...and the result will be a url.URL with the following path:

text

"/articles/technology/42"

This also works for host and query value variables:

r := mux.NewRouter()
r.Host("{subdomain}.example.com").
  Path("/articles/{category}/{id:[0-9]+}").
  Queries("filter", "{filter}").
  HandlerFunc(ArticleHandler).
  Name("article")

// url.String() will be "http://news.example.com/articles/technology/42?filter=gorilla"
url, err := r.Get("article").URL("subdomain", "news",
                                 "category", "technology",
                                 "id", "42",
                                 "filter", "gorilla")

All variables defined in the route are required, and their values must conform to the corresponding patterns. These requirements guarantee that a generated URL will always match a registered route -- the only exception is for explicitly defined "build-only" routes which never match.

Regex support also exists for matching Headers within a route. For example, we could do:

r.HeadersRegexp("Content-Type", "application/(text|json)")

...and the route will match both requests with a Content-Type of application/json as well as application/text

There's also a way to build only the URL host or path for a route: use the methods URLHost() or URLPath() instead. For the previous route, we would do:

// "http://news.example.com/"
host, err := r.Get("article").URLHost("subdomain", "news")

// "/articles/technology/42"
path, err := r.Get("article").URLPath("category", "technology", "id", "42")

And if you use subrouters, host and path defined separately can be built as well:

r := mux.NewRouter()
s := r.Host("{subdomain}.example.com").Subrouter()
s.Path("/articles/{category}/{id:[0-9]+}").
  HandlerFunc(ArticleHandler).
  Name("article")

// "http://news.example.com/articles/technology/42"
url, err := r.Get("article").URL("subdomain", "news",
                                 "category", "technology",
                                 "id", "42")

To find all the required variables for a given route when calling URL(), the method GetVarNames() is available:

r := mux.NewRouter()
r.Host("{domain}").
    Path("/{group}/{item_id}").
    Queries("some_data1", "{some_data1}").
    Queries("some_data2", "{some_data2}").
    Name("article")

// Will print [domain group item_id some_data1 some_data2] <nil>
fmt.Println(r.Get("article").GetVarNames())

Walking Routes

The Walk function on mux.Router can be used to visit all of the routes that are registered on a router. For example,
the following prints all of the registered routes:

package main

import (
	"fmt"
	"net/http"
	"strings"

	"github.com/gorilla/mux"
)

func handler(w http.ResponseWriter, r *http.Request) {
	return
}

func main() {
	r := mux.NewRouter()
	r.HandleFunc("/", handler)
	r.HandleFunc("/products", handler).Methods("POST")
	r.HandleFunc("/articles", handler).Methods("GET")
	r.HandleFunc("/articles/{id}", handler).Methods("GET", "PUT")
	r.HandleFunc("/authors", handler).Queries("surname", "{surname}")
	err := r.Walk(func(route *mux.Route, router *mux.Router, ancestors []*mux.Route) error {
		pathTemplate, err := route.GetPathTemplate()
		if err == nil {
			fmt.Println("ROUTE:", pathTemplate)
		}
		pathRegexp, err := route.GetPathRegexp()
		if err == nil {
			fmt.Println("Path regexp:", pathRegexp)
		}
		queriesTemplates, err := route.GetQueriesTemplates()
		if err == nil {
			fmt.Println("Queries templates:", strings.Join(queriesTemplates, ","))
		}
		queriesRegexps, err := route.GetQueriesRegexp()
		if err == nil {
			fmt.Println("Queries regexps:", strings.Join(queriesRegexps, ","))
		}
		methods, err := route.GetMethods()
		if err == nil {
			fmt.Println("Methods:", strings.Join(methods, ","))
		}
		fmt.Println()
		return nil
	})

	if err != nil {
		fmt.Println(err)
	}

	http.Handle("/", r)
}

Graceful Shutdown

Go 1.8 introduced the ability to gracefully shutdown a *http.Server. Here's how to do that alongside mux:

text

/* Detailed source-code truncated for AI context efficiency. */

Middleware

Mux supports the addition of middlewares to a Router, which are executed in the order they are added if a match is found, including its subrouters.
Middlewares are (typically) small pieces of code which take one request, do something with it, and pass it down to another middleware or the final handler. Some common use cases for middleware are request logging, header manipulation, or ResponseWriter hijacking.

Mux middlewares are defined using the de facto standard type:

type MiddlewareFunc func(http.Handler) http.Handler

Typically, the returned handler is a closure which does something with the http.ResponseWriter and http.Request passed to it, and then calls the handler passed as parameter to the MiddlewareFunc. This takes advantage of closures being able access variables from the context where they are created, while retaining the signature enforced by the receivers.

A very basic middleware which logs the URI of the request being handled could be written as:

func loggingMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Do stuff here
        log.Println(r.RequestURI)
        // Call the next handler, which can be another middleware in the chain, or the final handler.
        next.ServeHTTP(w, r)
    })
}

Middlewares can be added to a router using Router.Use():

r := mux.NewRouter()
r.HandleFunc("/", handler)
r.Use(loggingMiddleware)

A more complex authentication middleware, which maps session token to users, could be written as:

// Define our struct
type authenticationMiddleware struct {
	tokenUsers map[string]string
}

// Initialize it somewhere
func (amw *authenticationMiddleware) Populate() {
	amw.tokenUsers["00000000"] = "user0"
	amw.tokenUsers["aaaaaaaa"] = "userA"
	amw.tokenUsers["05f717e5"] = "randomUser"
	amw.tokenUsers["deadbeef"] = "user0"
}

// Middleware function, which will be called for each request
func (amw *authenticationMiddleware) Middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        token := r.Header.Get("X-Session-Token")

        if user, found := amw.tokenUsers[token]; found {
        	// We found the token in our map
        	log.Printf("Authenticated user %s\n", user)
        	// Pass down the request to the next middleware (or final handler)
        	next.ServeHTTP(w, r)
        } else {
        	// Write an error and stop the handler chain
        	http.Error(w, "Forbidden", http.StatusForbidden)
        }
    })
}

r := mux.NewRouter()
r.HandleFunc("/", handler)

amw := authenticationMiddleware{tokenUsers: make(map[string]string)}
amw.Populate()

r.Use(amw.Middleware)

Note: The handler chain will be stopped if your middleware doesn't call next.ServeHTTP() with the corresponding parameters. This can be used to abort a request if the middleware writer wants to. Middlewares should write to ResponseWriter if they are going to terminate the request, and they should not write to ResponseWriter if they are not going to terminate it.

Handling CORS Requests

CORSMethodMiddleware intends to make it easier to strictly set the Access-Control-Allow-Methods response header.

You will still need to use your own CORS handler to set the other CORS headers such as Access-Control-Allow-Origin
The middleware will set the Access-Control-Allow-Methods header to all the method matchers (e.g. r.Methods(http.MethodGet, http.MethodPut, http.MethodOptions) -> Access-Control-Allow-Methods: GET,PUT,OPTIONS) on a route
If you do not specify any methods, then:

Important: there must be an OPTIONS method matcher for the middleware to set the headers.

Here is an example of using CORSMethodMiddleware along with a custom OPTIONS handler to set all the required CORS headers:

package main

import (
	"net/http"
	"github.com/gorilla/mux"
)

func main() {
    r := mux.NewRouter()

    // IMPORTANT: you must specify an OPTIONS method matcher for the middleware to set CORS headers
    r.HandleFunc("/foo", fooHandler).Methods(http.MethodGet, http.MethodPut, http.MethodPatch, http.MethodOptions)
    r.Use(mux.CORSMethodMiddleware(r))
    
    http.ListenAndServe(":8080", r)
}

func fooHandler(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Access-Control-Allow-Origin", "*")
    if r.Method == http.MethodOptions {
        return
    }

    w.Write([]byte("foo"))
}

And an request to /foo using something like:

bash

curl localhost:8080/foo -v

Would look like:

bash

*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /foo HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.59.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Access-Control-Allow-Methods: GET,PUT,PATCH,OPTIONS
< Access-Control-Allow-Origin: *
< Date: Fri, 28 Jun 2019 20:13:30 GMT
< Content-Length: 3
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host localhost left intact
foo

Testing Handlers

Testing handlers in a Go web application is straightforward, and mux doesn't complicate this any further. Given two files: endpoints.go and endpoints_test.go, here's how we'd test an application using mux.

First, our simple HTTP handler:

// endpoints.go
package main

func HealthCheckHandler(w http.ResponseWriter, r *http.Request) {
    // A very simple health check.
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusOK)

    // In the future we could report back on the status of our DB, or our cache
    // (e.g. Redis) by performing a simple PING, and include them in the response.
    io.WriteString(w, `{"alive": true}`)
}

func main() {
    r := mux.NewRouter()
    r.HandleFunc("/health", HealthCheckHandler)

    log.Fatal(http.ListenAndServe("localhost:8080", r))
}

Our test code:

// endpoints_test.go
package main

import (
    "net/http"
    "net/http/httptest"
    "testing"
)

func TestHealthCheckHandler(t *testing.T) {
    // Create a request to pass to our handler. We don't have any query parameters for now, so we'll
    // pass 'nil' as the third parameter.
    req, err := http.NewRequest("GET", "/health", nil)
    if err != nil {
        t.Fatal(err)
    }

    // We create a ResponseRecorder (which satisfies http.ResponseWriter) to record the response.
    rr := httptest.NewRecorder()
    handler := http.HandlerFunc(HealthCheckHandler)

    // Our handlers satisfy http.Handler, so we can call their ServeHTTP method
    // directly and pass in our Request and ResponseRecorder.
    handler.ServeHTTP(rr, req)

    // Check the status code is what we expect.
    if status := rr.Code; status != http.StatusOK {
        t.Errorf("handler returned wrong status code: got %v want %v",
            status, http.StatusOK)
    }

    // Check the response body is what we expect.
    expected := `{"alive": true}`
    if rr.Body.String() != expected {
        t.Errorf("handler returned unexpected body: got %v want %v",
            rr.Body.String(), expected)
    }
}

In the case that our routes have variables, we can pass those in the request. We could write
table-driven tests to test multiple
possible route variables as needed.

// endpoints.go
func main() {
    r := mux.NewRouter()
    // A route with a route variable:
    r.HandleFunc("/metrics/{type}", MetricsHandler)

    log.Fatal(http.ListenAndServe("localhost:8080", r))
}

Our test file, with a table-driven test of routeVariables:

// endpoints_test.go
func TestMetricsHandler(t *testing.T) {
    tt := []struct{
        routeVariable string
        shouldPass bool
    }{
        {"goroutines", true},
        {"heap", true},
        {"counters", true},
        {"queries", true},
        {"adhadaeqm3k", false},
    }

    for _, tc := range tt {
        path := fmt.Sprintf("/metrics/%s", tc.routeVariable)
        req, err := http.NewRequest("GET", path, nil)
        if err != nil {
            t.Fatal(err)
        }

        rr := httptest.NewRecorder()
	
	// To add the vars to the context, 
	// we need to create a router through which we can pass the request.
	router := mux.NewRouter()
        router.HandleFunc("/metrics/{type}", MetricsHandler)
        router.ServeHTTP(rr, req)

        // In this case, our MetricsHandler returns a non-200 response
        // for a route variable it doesn't know about.
        if rr.Code == http.StatusOK && !tc.shouldPass {
            t.Errorf("handler should have failed on routeVariable %s: got %v want %v",
                tc.routeVariable, rr.Code, http.StatusOK)
        }
    }
}

Full Example

Here's a complete, runnable example of a small mux based server:

package main

import (
    "net/http"
    "log"
    "github.com/gorilla/mux"
)

func YourHandler(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("Gorilla!\n"))
}

func main() {
    r := mux.NewRouter()
    // Routes consist of a path and a handler function.
    r.HandleFunc("/", YourHandler)

    // Bind to a port and pass our router in
    log.Fatal(http.ListenAndServe(":8000", r))
}

License

BSD licensed. See the LICENSE file for details.

Overview (`vendor/github.com/grpc-ecosystem/go-grpc-prometheus/README.md`)

Go gRPC Interceptors for Prometheus monitoring

Prometheus monitoring for your gRPC Go servers and clients.

A sister implementation for gRPC Java (same metrics, same semantics) is in grpc-ecosystem/java-grpc-prometheus.

Interceptors

gRPC Go recently acquired support for Interceptors, i.e. middleware that is executed
by a gRPC Server before the request is passed onto the user's application logic. It is a perfect way to implement
common patterns: auth, logging and... monitoring.

To use Interceptors in chains, please see go-grpc-middleware.

Usage

There are two types of interceptors: client-side and server-side. This package provides monitoring Interceptors for both.

Server-side

import "github.com/grpc-ecosystem/go-grpc-prometheus"
...
    // Initialize your gRPC server's interceptor.
    myServer := grpc.NewServer(
        grpc.StreamInterceptor(grpc_prometheus.StreamServerInterceptor),
        grpc.UnaryInterceptor(grpc_prometheus.UnaryServerInterceptor),
    )
    // Register your gRPC service implementations.
    myservice.RegisterMyServiceServer(s.server, &myServiceImpl{})
    // After all your registrations, make sure all of the Prometheus metrics are initialized.
    grpc_prometheus.Register(myServer)
    // Register Prometheus metrics handler.    
    http.Handle("/metrics", promhttp.Handler())
...

Client-side

import "github.com/grpc-ecosystem/go-grpc-prometheus"
...
   clientConn, err = grpc.Dial(
       address,
		   grpc.WithUnaryInterceptor(grpc_prometheus.UnaryClientInterceptor),
		   grpc.WithStreamInterceptor(grpc_prometheus.StreamClientInterceptor)
   )
   client = pb_testproto.NewTestServiceClient(clientConn)
   resp, err := client.PingEmpty(s.ctx, &myservice.Request{Msg: "hello"})
...

Metrics

Labels

All server-side metrics start with grpc_server as Prometheus subsystem name. All client-side metrics start with grpc_client. Both of them have mirror-concepts. Similarly all methods
contain the same rich labels:

grpc_service - the gRPC service name, which is the combination of protobuf package and
the grpc_service section name. E.g. for package = mwitkow.testproto and
service TestService the label will be grpc_service="mwitkow.testproto.TestService"
grpc_method - the name of the method called on the gRPC service. E.g.
grpc_method="Ping"
grpc_type - the gRPC type of request.
Differentiating between the two is important especially for latency measurements.
- unary is single request, single response RPC
- client_stream is a multi-request, single response RPC
- server_stream is a single request, multi-response RPC
- bidi_stream is a multi-request, multi-response RPC

Additionally for completed RPCs, the following labels are used:

grpc_code - the human-readable gRPC status code.
The list of all statuses is to long, but here are some common ones:
- OK - means the RPC was successful
- IllegalArgument - RPC contained bad values
- Internal - server-side error not disclosed to the clients

Counters

The counters and their up to date documentation is in server_reporter.go and client_reporter.go
the respective Prometheus handler (usually /metrics).

For the purpose of this documentation we will only discuss grpc_server metrics. The grpc_client ones contain mirror concepts.

For simplicity, let's assume we're tracking a single server-side RPC call of mwitkow.testproto.TestService,
calling the method PingList. The call succeeds and returns 20 messages in the stream.

First, immediately after the server receives the call it will increment the
grpc_server_started_total and start the handling time clock (if histograms are enabled).

jsoniq

grpc_server_started_total{grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 1

Then the user logic gets invoked. It receives one message from the client containing the request
(it's a server_stream):

jsoniq

grpc_server_msg_received_total{grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 1

The user logic may return an error, or send multiple messages back to the client. In this case, on
each of the 20 messages sent back, a counter will be incremented:

jsoniq

grpc_server_msg_sent_total{grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 20

After the call completes, its status (OK or other gRPC status code)
and the relevant call labels increment the grpc_server_handled_total counter.

jsoniq

grpc_server_handled_total{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 1

Histograms

Prometheus histograms are a great way
to measure latency distributions of your RPCs. However, since it is bad practice to have metrics
of high cardinality
the latency monitoring metrics are disabled by default. To enable them please call the following
in your server initialization code:

jsoniq

grpc_prometheus.EnableHandlingTimeHistogram()

After the call completes, its handling time will be recorded in a Prometheus histogram
variable grpc_server_handling_seconds. The histogram variable contains three sub-metrics:

grpc_server_handling_seconds_count - the count of all completed RPCs by status and method
grpc_server_handling_seconds_sum - cumulative time of RPCs by status and method, useful for
calculating average handling times
grpc_server_handling_seconds_bucket - contains the counts of RPCs by status and method in respective
handling-time buckets. These buckets can be used by Prometheus to estimate SLAs (see here)

The counter values will look as follows:

text

/* Detailed source-code truncated for AI context efficiency. */

Useful query examples

Prometheus philosophy is to provide raw metrics to the monitoring system, and
let the aggregations be handled there. The verbosity of above metrics make it possible to have that
flexibility. Here's a couple of useful monitoring queries:

request inbound rate

jsoniq

sum(rate(grpc_server_started_total{job="foo"}[1m])) by (grpc_service)

For job="foo" (common label to differentiate between Prometheus monitoring targets), calculate the
rate of requests per second (1 minute window) for each gRPC grpc_service that the job has. Please note
how the grpc_method is being omitted here: all methods of a given gRPC service will be summed together.

unary request error rate

jsoniq

sum(rate(grpc_server_handled_total{job="foo",grpc_type="unary",grpc_code!="OK"}[1m])) by (grpc_service)

For job="foo", calculate the per-grpc_service rate of unary (1:1) RPCs that failed, i.e. the
ones that didn't finish with OK code.

unary request error percentage

jsoniq

sum(rate(grpc_server_handled_total{job="foo",grpc_type="unary",grpc_code!="OK"}[1m])) by (grpc_service)
 / 
sum(rate(grpc_server_started_total{job="foo",grpc_type="unary"}[1m])) by (grpc_service)
 * 100.0

For job="foo", calculate the percentage of failed requests by service. It's easy to notice that
this is a combination of the two above examples. This is an example of a query you would like to
alert on in your system for SLA violations, e.g.
"no more than 1% requests should fail".

average response stream size

jsoniq

sum(rate(grpc_server_msg_sent_total{job="foo",grpc_type="server_stream"}[10m])) by (grpc_service)
 /
sum(rate(grpc_server_started_total{job="foo",grpc_type="server_stream"}[10m])) by (grpc_service)

For job="foo" what is the grpc_service-wide 10m average of messages returned for all server_stream RPCs. This allows you to track the stream sizes returned by your system, e.g. allows
you to track when clients started to send "wide" queries that ret
Note the divisor is the number of started RPCs, in order to account for in-flight requests.

99%-tile latency of unary requests

jsoniq

histogram_quantile(0.99, 
  sum(rate(grpc_server_handling_seconds_bucket{job="foo",grpc_type="unary"}[5m])) by (grpc_service,le)
)

For job="foo", returns an 99%-tile quantile estimation
of the handling time of RPCs per service. Please note the 5m rate, this means that the quantile
estimation will take samples in a rolling 5m window. When combined with other quantiles
(e.g. 50%, 90%), this query gives you tremendous insight into the responsiveness of your system
(e.g. impact of caching).

percentage of slow unary queries (>250ms)

jsoniq

100.0 - (
sum(rate(grpc_server_handling_seconds_bucket{job="foo",grpc_type="unary",le="0.25"}[5m])) by (grpc_service)
 / 
sum(rate(grpc_server_handling_seconds_count{job="foo",grpc_type="unary"}[5m])) by (grpc_service)
) * 100.0

For job="foo" calculate the by-grpc_service fraction of slow requests that took longer than 0.25
seconds. This query is relatively complex, since the Prometheus aggregations use le (less or equal)
buckets, meaning that counting "fast" requests fractions is easier. However, simple maths helps.
This is an example of a query you would like to alert on in your system for SLA violations,
e.g. "less than 1% of requests are slower than 250ms".

Status

This code has been used since August 2015 as the basis for monitoring of production gRPC micro services at Improbable.

License

go-grpc-prometheus is released under the Apache 2.0 license. See the LICENSE file for details.

Overview (`vendor/github.com/hashicorp/errwrap/README.md`)

errwrap

errwrap is a package for Go that formalizes the pattern of wrapping errors
and checking if an error contains another error.

There is a common pattern in Go of taking a returned error value and
then wrapping it (such as with fmt.Errorf) before returning it. The problem
with this pattern is that you completely lose the original error structure.

Arguably the correct approach is that you should make a custom structure
implementing the error interface, and have the original error as a field
on that structure, such as this example.
This is a good approach, but you have to know the entire chain of possible
rewrapping that happens, when you might just care about one.

errwrap formalizes this pattern (it doesn't matter what approach you use
above) by giving a single interface for wrapping errors, checking if a specific
error is wrapped, and extracting that error.

Installation and Docs

Install using go get github.com/hashicorp/errwrap.

Full documentation is available at
http://godoc.org/github.com/hashicorp/errwrap

Usage

Basic Usage

Below is a very basic example of its usage:

// A function that always returns an error, but wraps it, like a real
// function might.
func tryOpen() error {
	_, err := os.Open("/i/dont/exist")
	if err != nil {
		return errwrap.Wrapf("Doesn't exist: {{err}}", err)
	}

	return nil
}

func main() {
	err := tryOpen()

	// We can use the Contains helpers to check if an error contains
	// another error. It is safe to do this with a nil error, or with
	// an error that doesn't even use the errwrap package.
	if errwrap.Contains(err, "does not exist") {
		// Do something
	}
	if errwrap.ContainsType(err, new(os.PathError)) {
		// Do something
	}

	// Or we can use the associated `Get` functions to just extract
	// a specific error. This would return nil if that specific error doesn't
	// exist.
	perr := errwrap.GetType(err, new(os.PathError))
}

Custom Types

If you're already making custom types that properly wrap errors, then
you can get all the functionality of errwraps.Contains and such by
implementing the Wrapper interface with just one function. Example:

type AppError {
  Code ErrorCode
  Err  error
}

func (e *AppError) WrappedErrors() []error {
  return []error{e.Err}
}

Now this works:

err := &AppError{Err: fmt.Errorf("an error")}
if errwrap.ContainsType(err, fmt.Errorf("")) {
	// This will work!
}

Overview (`vendor/github.com/hashicorp/go-immutable-radix/README.md`)

go-immutable-radix [](https://circleci.com/gh/hashicorp/go-immutable-radix/tree/master)

Provides the iradix package that implements an immutable radix tree.
The package only provides a single Tree implementation, optimized for sparse nodes.

As a radix tree, it provides the following:

O(k) operations. In many cases, this can be faster than a hash table since
the hash function is an O(k) operation, and hash tables have very poor cache locality.
Minimum / Maximum value lookups
Ordered iteration

A tree supports using a transaction to batch multiple updates (insert, delete)
in a more efficient manner than performing each operation one at a time.

For a mutable variant, see go-radix.

Documentation

The full documentation is available on Godoc.

Example

Below is a simple example of usage

// Create a tree
r := iradix.New()
r, _, _ = r.Insert([]byte("foo"), 1)
r, _, _ = r.Insert([]byte("bar"), 2)
r, _, _ = r.Insert([]byte("foobar"), 2)

// Find the longest prefix match
m, _, _ := r.Root().LongestPrefix([]byte("foozip"))
if string(m) != "foo" {
    panic("should be foo")
}

Here is an example of performing a range scan of the keys.

// Create a tree
r := iradix.New()
r, _, _ = r.Insert([]byte("001"), 1)
r, _, _ = r.Insert([]byte("002"), 2)
r, _, _ = r.Insert([]byte("005"), 5)
r, _, _ = r.Insert([]byte("010"), 10)
r, _, _ = r.Insert([]byte("100"), 10)

// Range scan over the keys that sort lexicographically between [003, 050)
it := r.Root().Iterator()
it.SeekLowerBound([]byte("003"))
for key, _, ok := it.Next(); ok; key, _, ok = it.Next() {
  if key >= "050" {
      break
  }
  fmt.Println(key)
}
// Output:
//  005
//  010

Overview (`vendor/github.com/hashicorp/go-memdb/README.md`)

go-memdb [](https://circleci.com/gh/hashicorp/go-memdb/tree/master)

Provides the memdb package that implements a simple in-memory database
built on immutable radix trees. The database provides Atomicity, Consistency
and Isolation from ACID. Being that it is in-memory, it does not provide durability.
The database is instantiated with a schema that specifies the tables and indices
that exist and allows transactions to be executed.

The database provides the following:

Multi-Version Concurrency Control (MVCC) - By leveraging immutable radix trees
the database is able to support any number of concurrent readers without locking,
and allows a writer to make progress.
Transaction Support - The database allows for rich transactions, in which multiple
objects are inserted, updated or deleted. The transactions can span multiple tables,
and are applied atomically. The database provides atomicity and isolation in ACID
terminology, such that until commit the updates are not visible.
Rich Indexing - Tables can support any number of indexes, which can be simple like
a single field index, or more advanced compound field indexes. Certain types like
UUID can be efficiently compressed from strings into byte indexes for reduced
storage requirements.
Watches - Callers can populate a watch set as part of a query, which can be used to
detect when a modification has been made to the database which affects the query
results. This lets callers easily watch for changes in the database in a very general
way.

For the underlying immutable radix trees, see go-immutable-radix.

Documentation

The full documentation is available on Godoc.

Example

Below is a simple example of usage

text

/* Detailed source-code truncated for AI context efficiency. */

Overview (`vendor/github.com/hashicorp/go-metrics/README.md`)

go-metrics

This library provides a metrics package which can be used to instrument code,
expose application metrics, and profile runtime performance in a flexible manner.

Current API:

Sinks

The metrics package makes use of a MetricSink interface to support delivery
to any type of backend. Currently the following sinks are provided:

StatsiteSink : Sinks to a statsite instance (TCP)
StatsdSink: Sinks to a StatsD / statsite instance (UDP)
PrometheusSink: Sinks to a Prometheus metrics endpoint (exposed via HTTP for scrapes)
InmemSink : Provides in-memory aggregation, can be used to export stats
FanoutSink : Sinks to multiple sinks. Enables writing to multiple statsite instances for example.
BlackholeSink : Sinks to nowhere

In addition to the sinks, the InmemSignal can be used to catch a signal,
and dump a formatted output of recent metrics. For example, when a process gets
a SIGUSR1, it can dump to stderr recent performance metrics for debugging.

Labels

Most metrics do have an equivalent ending with WithLabels, such methods
allow to push metrics with labels and use some features of underlying Sinks
(ex: translated into Prometheus labels).

Since some of these labels may increase the cardinality of metrics, the
library allows filtering labels using a allow/block list filtering system
which is global to all metrics.

If Config.AllowedLabels is not nil, then only labels specified in this value will be sent to underlying Sink, otherwise, all labels are sent by default.
If Config.BlockedLabels is not nil, any label specified in this value will not be sent to underlying Sinks.

By default, both Config.AllowedLabels and Config.BlockedLabels are nil, meaning that
no tags are filtered at all, but it allows a user to globally block some tags with high
cardinality at the application level.

Examples

Here is an example of using the package:

func SlowMethod() {
    // Profiling the runtime of a method
    defer metrics.MeasureSince([]string{"SlowMethod"}, time.Now())
}

// Configure a statsite sink as the global metrics sink
sink, _ := metrics.NewStatsiteSink("statsite:8125")
metrics.NewGlobal(metrics.DefaultConfig("service-name"), sink)

// Emit a Key/Value pair
metrics.EmitKey([]string{"questions", "meaning of life"}, 42)

Here is an example of setting up a signal handler:

// Setup the inmem sink and signal handler
inm := metrics.NewInmemSink(10*time.Second, time.Minute)
sig := metrics.DefaultInmemSignal(inm)
metrics.NewGlobal(metrics.DefaultConfig("service-name"), inm)

// Run some code
inm.SetGauge([]string{"foo"}, 42)
inm.EmitKey([]string{"bar"}, 30)

inm.IncrCounter([]string{"baz"}, 42)
inm.IncrCounter([]string{"baz"}, 1)
inm.IncrCounter([]string{"baz"}, 80)

inm.AddSample([]string{"method", "wow"}, 42)
inm.AddSample([]string{"method", "wow"}, 100)
inm.AddSample([]string{"method", "wow"}, 22)

....

When a signal comes in, output like the following will be dumped to stderr:

text

[2014-01-28 14:57:33.04 -0800 PST][G] 'foo': 42.000
[2014-01-28 14:57:33.04 -0800 PST][P] 'bar': 30.000
[2014-01-28 14:57:33.04 -0800 PST][C] 'baz': Count: 3 Min: 1.000 Mean: 41.000 Max: 80.000 Stddev: 39.509
[2014-01-28 14:57:33.04 -0800 PST][S] 'method.wow': Count: 3 Min: 22.000 Mean: 54.667 Max: 100.000 Stddev: 40.513

Overview (`vendor/github.com/hashicorp/go-multierror/README.md`)

go-multierror

go-multierror is a package for Go that provides a mechanism for
representing a list of error values as a single error.

This allows a function in Go to return an error that might actually
be a list of errors. If the caller knows this, they can unwrap the
list and access the errors. If the caller doesn't know, the error
formats to a nice human-readable format.

go-multierror is fully compatible with the Go standard library
errors package, including the
functions As, Is, and Unwrap. This provides a standardized approach
for introspecting on error values.

Installation and Docs

Install using go get github.com/hashicorp/go-multierror.

Full documentation is available at
https://pkg.go.dev/github.com/hashicorp/go-multierror

Requires go version 1.13 or newer

go-multierror requires go version 1.13 or newer. Go 1.13 introduced
error wrapping, which
this library takes advantage of.

If you need to use an earlier version of go, you can use the
v1.0.0
tag, which doesn't rely on features in go 1.13.

If you see compile errors that look like the below, it's likely that
you're on an older version of go:

text

/go/src/github.com/hashicorp/go-multierror/multierror.go:112:9: undefined: errors.As
/go/src/github.com/hashicorp/go-multierror/multierror.go:117:9: undefined: errors.Is

Usage

go-multierror is easy to use and purposely built to be unobtrusive in
existing Go applications/libraries that may not be aware of it.

Building a list of errors

The Append function is used to create a list of errors. This function
behaves a lot like the Go built-in append function: it doesn't matter
if the first argument is nil, a multierror.Error, or any other error,
the function behaves as you would expect.

var result error

if err := step1(); err != nil {
	result = multierror.Append(result, err)
}
if err := step2(); err != nil {
	result = multierror.Append(result, err)
}

return result

Customizing the formatting of the errors

By specifying a custom ErrorFormat, you can customize the format
of the Error() string function:

var result *multierror.Error

// ... accumulate errors here, maybe using Append

if result != nil {
	result.ErrorFormat = func([]error) string {
		return "errors!"
	}
}

Accessing the list of errors

multierror.Error implements error so if the caller doesn't know about
multierror, it will work just fine. But if you're aware a multierror might
be returned, you can use type switches to access the list of errors:

if err := something(); err != nil {
	if merr, ok := err.(*multierror.Error); ok {
		// Use merr.Errors
	}
}

You can also use the standard errors.Unwrap
function. This will continue to unwrap into subsequent errors until none exist.

Extracting an error

The standard library errors.As
function can be used directly with a multierror to extract a specific error:

// Assume err is a multierror value
err := somefunc()

// We want to know if "err" has a "RichErrorType" in it and extract it.
var errRich RichErrorType
if errors.As(err, &errRich) {
	// It has it, and now errRich is populated.
}

Checking for an exact error value

Some errors are returned as exact errors such as the ErrNotExist
error in the os package. You can check if this error is present by using
the standard errors.Is function.

// Assume err is a multierror value
err := somefunc()
if errors.Is(err, os.ErrNotExist) {
	// err contains os.ErrNotExist
}

Returning a multierror only if there are errors

If you build a multierror.Error, you can use the ErrorOrNil function
to return an error implementation only if there are errors to return:

var result *multierror.Error

// ... accumulate errors here

// Return the `error` only if errors were added to the multierror, otherwise
// return nil since there are no errors.
return result.ErrorOrNil()

Overview (`vendor/github.com/hashicorp/go-sockaddr/README.md`)

go-sockaddr

`sockaddr` Library

Socket address convenience functions for Go. go-sockaddr is a convenience
library that makes doing the right thing with IP addresses easy. go-sockaddr
is loosely modeled after the UNIX sockaddr_t and creates a union of the family
of sockaddr_t types (see below for an ascii diagram). Library documentation
is available
at
https://godoc.org/github.com/hashicorp/go-sockaddr.
The primary intent of the library was to make it possible to define heuristics
for selecting the correct IP addresses when a configuration is evaluated at
runtime. See
the
docs,
template package,
tests,
and
CLI utility
for details and hints as to how to use this library.

For example, with this library it is possible to find an IP address that:

is attached to a default route
(GetDefaultInterfaces())
is contained within a CIDR block (IfByNetwork())
is an RFC1918 address
(IfByRFC("1918"))
is ordered
(OrderedIfAddrBy(args) where
args includes, but is not limited
to,
AscIfType,
AscNetworkSize)
excludes all IPv6 addresses
(IfByType("^(IPv4)$"))
is larger than a /32
(IfByMaskSize(32))
is not on a down interface
(ExcludeIfs("flags", "down"))
preferences an IPv6 address over an IPv4 address
(SortIfByType() +
ReverseIfAddrs()); and
excludes any IP in RFC6890 address
(IfByRFC("6890"))

Or any combination or variation therein.

There are also a few simple helper functions such as GetPublicIP and
GetPrivateIP which both return strings and select the first public or private
IP address on the default interface, respectively. Similarly, there is also a
helper function called GetInterfaceIP which returns the first usable IP
address on the named interface.

`sockaddr` CLI

Given the possible complexity of the sockaddr library, there is a CLI utility
that accompanies the library, also
called
sockaddr.
The
sockaddr
utility exposes nearly all of the functionality of the library and can be used
either as an administrative tool or testing tool. To install
the
sockaddr,
run:

text

$ go install github.com/hashicorp/go-sockaddr/cmd/sockaddr@latest

If you're familiar with UNIX's sockaddr struct's, the following diagram
mapping the C sockaddr (top) to go-sockaddr structs (bottom) and
interfaces will be helpful:

text

+-------------------------------------------------------+
|                                                       |
|                        sockaddr                       |
|                        SockAddr                       |
|                                                       |
| +--------------+ +----------------------------------+ |
| | sockaddr_un  | |                                  | |
| | SockAddrUnix | |           sockaddr_in{,6}        | |
| +--------------+ |                IPAddr            | |
|                  |                                  | |
|                  | +-------------+ +--------------+ | |
|                  | | sockaddr_in | | sockaddr_in6 | | |
|                  | |   IPv4Addr  | |   IPv6Addr   | | |
|                  | +-------------+ +--------------+ | |
|                  |                                  | |
|                  +----------------------------------+ |
|                                                       |
+-------------------------------------------------------+

Inspiration and Design

There were many subtle inspirations that led to this design, but the most direct
inspiration for the filtering syntax was
OpenBSD's
pf.conf(5) firewall
syntax that lets you select the first IP address on a given named interface.
The original problem stemmed from:

needing to create immutable images using Packer that
ran the Consul process (Consul can only use one IP
address at a time);
images that may or may not have multiple interfaces or IP addresses at
runtime; and
we didn't want to rely on configuration management to render out the correct
IP address if the VM image was being used in an auto-scaling group.

Instead we needed some way to codify a heuristic that would correctly select the
right IP address but the input parameters were not known when the image was
created.

Overview (`vendor/github.com/hashicorp/memberlist/README.md`)

memberlist [](https://godoc.org/github.com/hashicorp/memberlist)

memberlist is a Go library that manages cluster
membership and member failure detection using a gossip based protocol.

The use cases for such a library are far-reaching: all distributed systems
require membership, and memberlist is a re-usable solution to managing
cluster membership and node failure detection.

memberlist is eventually consistent but converges quickly on average.
The speed at which it converges can be heavily tuned via various knobs
on the protocol. Node failures are detected and network partitions are partially
tolerated by attempting to communicate to potentially dead nodes through
multiple routes.

Building

If you wish to build memberlist you'll need Go version 1.2+ installed.

Please check your installation with:

text

go version

Usage

Memberlist is surprisingly simple to use. An example is shown below:

/* Create the initial memberlist from a safe configuration.
   Please reference the godoc for other default config types.
   http://godoc.org/github.com/hashicorp/memberlist#Config
*/
list, err := memberlist.Create(memberlist.DefaultLocalConfig())
if err != nil {
	panic("Failed to create memberlist: " + err.Error())
}

// Join an existing cluster by specifying at least one known member.
n, err := list.Join([]string{"1.2.3.4"})
if err != nil {
	panic("Failed to join cluster: " + err.Error())
}

// Ask for members of the cluster
for _, member := range list.Members() {
	fmt.Printf("Member: %s %s\n", member.Name, member.Addr)
}

// Continue doing whatever you need, memberlist will maintain membership
// information in the background. Delegates can be used for receiving
// events when members join or leave.

The most difficult part of memberlist is configuring it since it has many
available knobs in order to tune state propagation delay and convergence times.
Memberlist provides a default configuration that offers a good starting point,
but errs on the side of caution, choosing values that are optimized for
higher convergence at the cost of higher bandwidth usage.

For complete documentation, see the associated Godoc.

Protocol

memberlist is based on "SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol". However, we extend the protocol in a number of ways:

Several extensions are made to increase propagation speed and
convergence rate.
Another set of extensions, that we call Lifeguard, are made to make memberlist more robust in the presence of slow message processing (due to factors such as CPU starvation, and network delay or loss).

For details on all of these extensions, please read our paper "Lifeguard : SWIM-ing with Situational Awareness", along with the memberlist source. We welcome any questions related
to the protocol on our issue tracker.

Metrics Emission and Compatibility

This library can emit metrics using either github.com/armon/go-metrics or github.com/hashicorp/go-metrics. Choosing between the libraries is controlled via build tags.

Build Tags

armonmetrics - Using this tag will cause metrics to be routed to armon/go-metrics
hashicorpmetrics - Using this tag will cause all metrics to be routed to hashicorp/go-metrics

If no build tag is specified, the default behavior is to use armon/go-metrics.

Deprecating armon/go-metrics

Emitting metrics to armon/go-metrics is officially deprecated. Usage of armon/go-metrics will remain the default until mid-2025 with opt-in support continuing to the end of 2025.

Migration
To migrate an application currently using the older armon/go-metrics to instead use hashicorp/go-metrics the following should be done.

Upgrade libraries using armon/go-metrics to consume hashicorp/go-metrics/compat instead. This should involve only changing import statements. All repositories in the hashicorp namespace
Update an applications library dependencies to those that have the compatibility layer configured.
Update the application to use hashicorp/go-metrics for configuring metrics export instead of armon/go-metrics
- Replace all application imports of github.com/armon/go-metrics with github.com/hashicorp/go-metrics
- Instrument your build system to build with the hashicorpmetrics tag.

Eventually once the default behavior changes to use hashicorp/go-metrics by default (mid-2025), you can drop the hashicorpmetrics build tag.

Overview (`vendor/github.com/hiddeco/sshsig/README.md`)

sshsig

This Go library implements the SSHSIG wire protocol, and
can be used to sign and verify messages using SSH keys.

Compared to other implementations, this library does all the following:

Accepts an io.Reader as input for signing and verifying messages.
Performs simple public key fingerprint and namespace mismatch checks in
Verify. Malicious input will still fail signature verification, but this
provides more useful error messages.
Properly uses ssh-sha2-512 as signature algorithm when signing with an RSA
private key, as described in the protocol.
Does not accept a Sign operation without a namespace as specified in the
protocol.
Allows Verify operations to be performed without a namespace, ensuring
compatibility with loose implementations.
Provides Armor and Unarmor functions to encode/decode the signature
to/from an (armored) PEM format.

For more information about the use of this library, see the Go Reference.

Acknowledgements

There are several other implementations of the SSHSIG protocol in Go, from
which this library has borrowed ideas:

go-sshsig by Paul Tagliamonte
Sigstore Rekor from the Sigstore project

Overview (`vendor/github.com/inconshreveable/mousetrap/README.md`)

mousetrap

mousetrap is a tiny library that answers a single question.

On a Windows machine, was the process invoked by someone double clicking on
the executable file while browsing in explorer?

Motivation

Windows developers unfamiliar with command line tools will often "double-click"
the executable for a tool. Because most CLI tools print the help and then exit
when invoked without arguments, this is often very frustrating for those users.

mousetrap provides a way to detect these invocations so that you can provide
more helpful behavior and instructions on how to run the CLI tool. To see what
this looks like, both from an organizational and a technical perspective, see
https://inconshreveable.com/09-09-2014/sweat-the-small-stuff/

The interface

The library exposes a single interface:

text

func StartedByExplorer() (bool)

Overview (`vendor/github.com/ishidawataru/sctp/README.md`)

Stream Control Transmission Protocol (SCTP)

Examples

See example/sctp.go

$ cd example
$ go build
$ # run example SCTP server
$ ./example -server -port 1000 -ip 10.10.0.1,10.20.0.1
$ # run example SCTP client
$ ./example -port 1000 -ip 10.10.0.1,10.20.0.1

Overview (`vendor/github.com/klauspost/compress/README.md`)

compress

This package provides various compression algorithms.

zstandard compression and decompression in pure Go.
S2 is a high performance replacement for Snappy.
Optimized deflate packages which can be used as a dropin replacement for gzip, zip and zlib.
snappy is a drop-in replacement for github.com/golang/snappy offering better compression and concurrent streams.
lzw is a drop-in replacement for compress/lzw with 1.4-4x faster decompression and 1.1-2.7x faster compression, depending on the data.
huff0 and FSE implementations for raw entropy encoding.
Xpress decompression of the Microsoft XPRESS (MS-XCA) plain LZ77 and LZ77+Huffman formats (the LZ77+Huffman variant is the one used in WIM images and Windows Compact OS / WOF data).
gzhttp Provides client and server wrappers for handling gzipped/zstd HTTP requests efficiently.
pgzip is a separate package that provides a very fast parallel gzip implementation.

package usage

Use go get github.com/klauspost/compress@latest to add it to your project.

This package will support the current Go version and 2 versions back.

Use the nounsafe tag to disable all use of the "unsafe" package.
Use the noasm tag to disable all assembly across packages.

Use the links above for more information on each.

changelog

Jul 1st, 2026 1.19.0
- zstd: Add true concurrent stream encodingin https://github.com/klauspost/compress/pull/1136
- zstd: arm64 decoder asm by @lizthegrey in https://github.com/klauspost/compress/pull/1160
- flate: Add inflate checkpoints in https://github.com/klauspost/compress/pull/1154
- zstd: avoid unused BuildDict encoder allocation by @snissn in https://github.com/klauspost/compress/pull/1147
- snappy/s2: Limit length of varint in decodedLen by @eustas in https://github.com/klauspost/compress/pull/1148
- gzhttp: match qvalue parameter case-insensitively (RFC 7231) by @z9z in https://github.com/klauspost/compress/pull/1149
- zip: add NameDecoder callback for legacy encoding rewrite by @SAY-5 in https://github.com/klauspost/compress/pull/1150
- huff0: Allow building tables from histogram in https://github.com/klauspost/compress/pull/1155
- huff0: Allow building table from oversized histogram in https://github.com/klauspost/compress/pull/1156
- s2sx: Clean symlink targets in https://github.com/klauspost/compress/pull/1163
Feb 9th, 2026 1.18.4
- gzhttp: Add zstandard to server handler wrapper https://github.com/klauspost/compress/pull/1121
- zstd: Add ResetWithOptions to encoder/decoder https://github.com/klauspost/compress/pull/1122
- gzhttp: preserve qvalue when extra parameters follow in Accept-Encoding by @analytically in https://github.com/klauspost/compress/pull/1116
Jan 16th, 2026 1.18.3
- Downstream CVE-2025-61728. See golang/go#77102.
Dec 1st, 2025 - 1.18.2
- flate: Fix invalid encoding on level 9 with single value input in https://github.com/klauspost/compress/pull/1115
- flate: reduce stateless allocations by @RXamzin in https://github.com/klauspost/compress/pull/1106
Oct 20, 2025 - 1.18.1 - RETRACTED
- zstd: Add simple zstd EncodeTo/DecodeTo functions https://github.com/klauspost/compress/pull/1079
- zstd: Fix incorrect buffer size in dictionary encodes https://github.com/klauspost/compress/pull/1059
- s2: check for cap, not len of buffer in EncodeBetter/Best by @vdarulis in https://github.com/klauspost/compress/pull/1080
- zlib: Avoiding extra allocation in zlib.reader.Reset by @travelpolicy in https://github.com/klauspost/compress/pull/1086
- gzhttp: remove redundant err check in zstdReader by @ryanfowler in https://github.com/klauspost/compress/pull/1090
- flate: Faster load+store https://github.com/klauspost/compress/pull/1104
- flate: Simplify matchlen https://github.com/klauspost/compress/pull/1101
- flate: Use exact sizes for huffman tables https://github.com/klauspost/compress/pull/1103
Feb 19th, 2025 - 1.18.0
- Add unsafe little endian loaders https://github.com/klauspost/compress/pull/1036
- fix: check r.err != nil but return a nil value error err by @alingse in https://github.com/klauspost/compress/pull/1028
- flate: Simplify L4-6 loading https://github.com/klauspost/compress/pull/1043
- flate: Simplify matchlen (remove asm) https://github.com/klauspost/compress/pull/1045
- s2: Improve small block compression speed w/o asm https://github.com/klauspost/compress/pull/1048
- flate: Fix matchlen L5+L6 https://github.com/klauspost/compress/pull/1049
- flate: Cleanup & reduce casts https://github.com/klauspost/compress/pull/1050
  
  See changes to v1.17.x
Oct 11th, 2024 - 1.17.11
- zstd: Fix extra CRC written with multiple Close calls https://github.com/klauspost/compress/pull/1017
- s2: Don't use stack for index tables https://github.com/klauspost/compress/pull/1014
- gzhttp: No content-type on no body response code by @juliens in https://github.com/klauspost/compress/pull/1011
- gzhttp: Do not set the content-type when response has no body by @kevinpollet in https://github.com/klauspost/compress/pull/1013
Sep 23rd, 2024 - 1.17.10
- gzhttp: Add TransportAlwaysDecompress option. https://github.com/klauspost/compress/pull/978
- gzhttp: Add supported decompress request body by @mirecl in https://github.com/klauspost/compress/pull/1002
- s2: Add EncodeBuffer buffer recycling callback https://github.com/klauspost/compress/pull/982
- zstd: Improve memory usage on small streaming encodes https://github.com/klauspost/compress/pull/1007
- flate: read data written with partial flush by @vajexal in https://github.com/klauspost/compress/pull/996
Jun 12th, 2024 - 1.17.9
- s2: Reduce ReadFrom temporary allocations https://github.com/klauspost/compress/pull/949
- flate, zstd: Shave some bytes off amd64 matchLen by @greatroar in https://github.com/klauspost/compress/pull/963
- Upgrade zip/zlib to 1.22.4 upstream https://github.com/klauspost/compress/pull/970 https://github.com/klauspost/compress/pull/971
- zstd: BuildDict fails with RLE table https://github.com/klauspost/compress/pull/951
Apr 9th, 2024 - 1.17.8
- zstd: Reject blocks where reserved values are not 0 https://github.com/klauspost/compress/pull/885
- zstd: Add RLE detection+encoding https://github.com/klauspost/compress/pull/938
Feb 21st, 2024 - 1.17.7
- s2: Add AsyncFlush method: Complete the block without flushing by @Jille in https://github.com/klauspost/compress/pull/927
- s2: Fix literal+repeat exceeds dst crash https://github.com/klauspost/compress/pull/930
Feb 5th, 2024 - 1.17.6
- zstd: Fix incorrect repeat coding in best mode https://github.com/klauspost/compress/pull/923
- s2: Fix DecodeConcurrent deadlock on errors https://github.com/klauspost/compress/pull/925
Jan 26th, 2024 - v1.17.5
- flate: Fix reset with dictionary on custom window encodes https://github.com/klauspost/compress/pull/912
- zstd: Add Frame header encoding and stripping https://github.com/klauspost/compress/pull/908
- zstd: Limit better/best default window to 8MB https://github.com/klauspost/compress/pull/913
- zstd: Speed improvements by @greatroar in https://github.com/klauspost/compress/pull/896 https://github.com/klauspost/compress/pull/910
- s2: Fix callbacks for skippable blocks and disallow 0xfe (Padding) by @Jille in https://github.com/klauspost/compress/pull/916 https://github.com/klauspost/compress/pull/917
  https://github.com/klauspost/compress/pull/919 https://github.com/klauspost/compress/pull/918
Dec 1st, 2023 - v1.17.4
- huff0: Speed up symbol counting by @greatroar in https://github.com/klauspost/compress/pull/887
- huff0: Remove byteReader by @greatroar in https://github.com/klauspost/compress/pull/886
- gzhttp: Allow overriding decompression on transport https://github.com/klauspost/compress/pull/892
- gzhttp: Clamp compression level https://github.com/klauspost/compress/pull/890
- gzip: Error out if reserved bits are set https://github.com/klauspost/compress/pull/891
Nov 15th, 2023 - v1.17.3
- fse: Fix max header size https://github.com/klauspost/compress/pull/881
- zstd: Improve better/best compression https://github.com/klauspost/compress/pull/877
- gzhttp: Fix missing content type on Close https://github.com/klauspost/compress/pull/883
Oct 22nd, 2023 - v1.17.2
- zstd: Fix rare CORRUPTION output in "best" mode. See https://github.com/klauspost/compress/pull/876
Oct 14th, 2023 - v1.17.1
- s2: Fix S2 "best" dictionary wrong encoding https://github.com/klauspost/compress/pull/871
- flate: Reduce allocations in decompressor and minor code improvements by @fakefloordiv in https://github.com/klauspost/compress/pull/869
- s2: Fix EstimateBlockSize on 6&7 length input https://github.com/klauspost/compress/pull/867
Sept 19th, 2023 - v1.17.0
- Add experimental dictionary builder https://github.com/klauspost/compress/pull/853
- Add xerial snappy read/writer https://github.com/klauspost/compress/pull/838
- flate: Add limited window compression https://github.com/klauspost/compress/pull/843
- s2: Do 2 overlapping match checks https://github.com/klauspost/compress/pull/839
- flate: Add amd64 assembly matchlen https://github.com/klauspost/compress/pull/837
- gzip: Copy bufio.Reader on Reset by @thatguystone in https://github.com/klauspost/compress/pull/860
See changes to v1.16.x
July 1st, 2023 - v1.16.7
- zstd: Fix default level first dictionary encode https://github.com/klauspost/compress/pull/829
- s2: add GetBufferCapacity() method by @GiedriusS in https://github.com/klauspost/compress/pull/832
June 13, 2023 - v1.16.6
- zstd: correctly ignore WithEncoderPadding(1) by @ianlancetaylor in https://github.com/klauspost/compress/pull/806
- zstd: Add amd64 match length assembly https://github.com/klauspost/compress/pull/824
- gzhttp: Handle informational headers by @rtribotte in https://github.com/klauspost/compress/pull/815
- s2: Improve Better compression slightly https://github.com/klauspost/compress/pull/663
Apr 16, 2023 - v1.16.5
- zstd: readByte needs to use io.ReadFull by @jnoxon in https://github.com/klauspost/compress/pull/802
- gzip: Fix WriterTo after initial read https://github.com/klauspost/compress/pull/804
Apr 5, 2023 - v1.16.4
- zstd: Improve zstd best efficiency by @greatroar and @klauspost in https://github.com/klauspost/compress/pull/784
- zstd: Respect WithAllLitEntropyCompression https://github.com/klauspost/compress/pull/792
- zstd: Fix amd64 not always detecting corrupt data https://github.com/klauspost/compress/pull/785
- zstd: Various minor improvements by @greatroar in https://github.com/klauspost/compress/pull/788 https://github.com/klauspost/compress/pull/794 https://github.com/klauspost/compress/pull/795
- s2: Fix huge block overflow https://github.com/klauspost/compress/pull/779
- s2: Allow CustomEncoder fallback https://github.com/klauspost/compress/pull/780
- gzhttp: Support ResponseWriter Unwrap() in gzhttp handler by @jgimenez in https://github.com/klauspost/compress/pull/799
Mar 13, 2023 - v1.16.1
- zstd: Speed up + improve best encoder by @greatroar in https://github.com/klauspost/compress/pull/776
- gzhttp: Add optional BREACH mitigation. https://github.com/klauspost/compress/pull/762 https://github.com/klauspost/compress/pull/768 https://github.com/klauspost/compress/pull/769 https://github.com/klauspost/compress/pull/770 https://github.com/klauspost/compress/pull/767
- s2: Add Intel LZ4s converter https://github.com/klauspost/compress/pull/766
- zstd: Minor bug fixes https://github.com/klauspost/compress/pull/771 https://github.com/klauspost/compress/pull/772 https://github.com/klauspost/compress/pull/773
- huff0: Speed up compress1xDo by @greatroar in https://github.com/klauspost/compress/pull/774
Feb 26, 2023 - v1.16.0
- s2: Add Dictionary support. https://github.com/klauspost/compress/pull/685
- s2: Add Compression Size Estimate. https://github.com/klauspost/compress/pull/752
- s2: Add support for custom stream encoder. https://github.com/klauspost/compress/pull/755
- s2: Add LZ4 block converter. https://github.com/klauspost/compress/pull/748
- s2: Support io.ReaderAt in ReadSeeker. https://github.com/klauspost/compress/pull/747
- s2c/s2sx: Use concurrent decoding. https://github.com/klauspost/compress/pull/746
See changes to v1.15.x
Jan 21st, 2023 (v1.15.15)
- deflate: Improve level 7-9 https://github.com/klauspost/compress/pull/739
- zstd: Add delta encoding support by @greatroar in https://github.com/klauspost/compress/pull/728
- zstd: Various speed improvements by @greatroar https://github.com/klauspost/compress/pull/741 https://github.com/klauspost/compress/pull/734 https://github.com/klauspost/compress/pull/736 https://github.com/klauspost/compress/pull/744 https://github.com/klauspost/compress/pull/743 https://github.com/klauspost/compress/pull/745
- gzhttp: Add SuffixETag() and DropETag() options to prevent ETag collisions on compressed responses by @willbicks in https://github.com/klauspost/compress/pull/740
Jan 3rd, 2023 (v1.15.14)
- flate: Improve speed in big stateless blocks https://github.com/klauspost/compress/pull/718
- zstd: Minor speed tweaks by @greatroar in https://github.com/klauspost/compress/pull/716 https://github.com/klauspost/compress/pull/720
- export NoGzipResponseWriter for custom ResponseWriter wrappers by @harshavardhana in https://github.com/klauspost/compress/pull/722
- s2: Add example for indexing and existing stream https://github.com/klauspost/compress/pull/723
Dec 11, 2022 (v1.15.13)
- zstd: Add MaxEncodedSize to encoder https://github.com/klauspost/compress/pull/691
- zstd: Various tweaks and improvements https://github.com/klauspost/compress/pull/693 https://github.com/klauspost/compress/pull/695 https://github.com/klauspost/compress/pull/696 https://github.com/klauspost/compress/pull/701 https://github.com/klauspost/compress/pull/702 https://github.com/klauspost/compress/pull/703 https://github.com/klauspost/compress/pull/704 https://github.com/klauspost/compress/pull/705 https://github.com/klauspost/compress/pull/706 https://github.com/klauspost/compress/pull/707 https://github.com/klauspost/compress/pull/708
Oct 26, 2022 (v1.15.12)
- zstd: Tweak decoder allocs. https://github.com/klauspost/compress/pull/680
- gzhttp: Always delete HeaderNoCompression https://github.com/klauspost/compress/pull/683
Sept 26, 2022 (v1.15.11)
- flate: Improve level 1-3 compression https://github.com/klauspost/compress/pull/678
- zstd: Improve "best" compression by @nightwolfz in https://github.com/klauspost/compress/pull/677
- zstd: Fix+reduce decompression allocations https://github.com/klauspost/compress/pull/668
- zstd: Fix non-effective noescape tag https://github.com/klauspost/compress/pull/667
Sept 16, 2022 (v1.15.10)
- zstd: Add WithDecodeAllCapLimit https://github.com/klauspost/compress/pull/649
- Add Go 1.19 - deprecate Go 1.16 https://github.com/klauspost/compress/pull/651
- flate: Improve level 5+6 compression https://github.com/klauspost/compress/pull/656
- zstd: Improve "better" compression https://github.com/klauspost/compress/pull/657
- s2: Improve "best" compression https://github.com/klauspost/compress/pull/658
- s2: Improve "better" compression. https://github.com/klauspost/compress/pull/635
- s2: Slightly faster non-assembly decompression https://github.com/klauspost/compress/pull/646
- Use arrays for constant size copies https://github.com/klauspost/compress/pull/659
July 21, 2022 (v1.15.9)
- zstd: Fix decoder crash on amd64 (no BMI) on invalid input https://github.com/klauspost/compress/pull/645
- zstd: Disable decoder extended memory copies (amd64) due to possible crashes https://github.com/klauspost/compress/pull/644
- zstd: Allow single segments up to "max decoded size" https://github.com/klauspost/compress/pull/643
July 13, 2022 (v1.15.8)
- gzip: fix stack exhaustion bug in Reader.Read https://github.com/klauspost/compress/pull/641
- s2: Add Index header trim/restore https://github.com/klauspost/compress/pull/638
- zstd: Optimize seqdeq amd64 asm by @greatroar in https://github.com/klauspost/compress/pull/636
- zstd: Improve decoder memcopy https://github.com/klauspost/compress/pull/637
- huff0: Pass a single bitReader pointer to asm by @greatroar in https://github.com/klauspost/compress/pull/634
- zstd: Branchless getBits for amd64 w/o BMI2 by @greatroar in https://github.com/klauspost/compress/pull/640
- gzhttp: Remove header before writing https://github.com/klauspost/compress/pull/639
June 29, 2022 (v1.15.7)
- s2: Fix absolute forward seeks https://github.com/klauspost/compress/pull/633
- zip: Merge upstream https://github.com/klauspost/compress/pull/631
- zip: Re-add zip64 fix https://github.com/klauspost/compress/pull/624
- zstd: translate fseDecoder.buildDtable into asm by @WojciechMula in https://github.com/klauspost/compress/pull/598
- flate: Faster histograms https://github.com/klauspost/compress/pull/620
- deflate: Use compound hcode https://github.com/klauspost/compress/pull/622
June 3, 2022 (v1.15.6)
- s2: Improve coding for long, close matches https://github.com/klauspost/compress/pull/613
- s2c: Add Snappy/S2 stream recompression https://github.com/klauspost/compress/pull/611
- zstd: Always use configured block size https://github.com/klauspost/compress/pull/605
- zstd: Fix incorrect hash table placement for dict encoding in default https://github.com/klauspost/compress/pull/606
- zstd: Apply default config to ZipDecompressor without options https://github.com/klauspost/compress/pull/608
- gzhttp: Exclude more common archive formats https://github.com/klauspost/compress/pull/612
- s2: Add ReaderIgnoreCRC https://github.com/klauspost/compress/pull/609
- s2: Remove sanity load on index creation https://github.com/klauspost/compress/pull/607
- snappy: Use dedicated function for scoring https://github.com/klauspost/compress/pull/614
- s2c+s2d: Use official snappy framed extension https://github.com/klauspost/compress/pull/610
May 25, 2022 (v1.15.5)
- s2: Add concurrent stream decompression https://github.com/klauspost/compress/pull/602
- s2: Fix final emit oob read crash on amd64 https://github.com/klauspost/compress/pull/601
- huff0: asm implementation of Decompress1X by @WojciechMula https://github.com/klauspost/compress/pull/596
- zstd: Use 1 less goroutine for stream decoding https://github.com/klauspost/compress/pull/588
- zstd: Copy literal in 16 byte blocks when possible https://github.com/klauspost/compress/pull/592
- zstd: Speed up when WithDecoderLowmem(false) https://github.com/klauspost/compress/pull/599
- zstd: faster next state update in BMI2 version of decode by @WojciechMula in https://github.com/klauspost/compress/pull/593
- huff0: Do not check max size when reading table. https://github.com/klauspost/compress/pull/586
- flate: Inplace hashing for level 7-9 https://github.com/klauspost/compress/pull/590
May 11, 2022 (v1.15.4)
- huff0: decompress directly into output by @WojciechMula in #577
- inflate: Keep dict on stack #581
- zstd: Faster decoding memcopy in asm #583
- zstd: Fix ignored crc #580
May 5, 2022 (v1.15.3)
- zstd: Allow to ignore checksum checking by @WojciechMula #572
- s2: Fix incorrect seek for io.SeekEnd in #575
Apr 26, 2022 (v1.15.2)
- zstd: Add x86-64 assembly for decompression on streams and blocks. Contributed by @WojciechMula. Typically 2x faster. #528 #531 #545 #537
- zstd: Add options to ZipDecompressor and fixes #539
- s2: Use sorted search for index #555
- Minimum version is Go 1.16, added CI test on 1.18.
Mar 11, 2022 (v1.15.1)
- huff0: Add x86 assembly of Decode4X by @WojciechMula in #512
- zstd: Reuse zip decoders in #514
- zstd: Detect extra block data and report as corrupted in #520
- zstd: Handle zero sized frame content size stricter in #521
- zstd: Add stricter block size checks in #523
Mar 3, 2022 (v1.15.0)
- zstd: Refactor decoder #498
- zstd: Add stream encoding without goroutines #505
- huff0: Prevent single blocks exceeding 16 bits by @klauspost in#507
- flate: Inline literal emission #509
- gzhttp: Add zstd to transport #400
- gzhttp: Make content-type optional #510

Both compression and decompression now supports "synchronous" stream operations. This means that whenever "concurrency" is set to 1, they will operate without spawning goroutines.

Stream decompression is now faster on asynchronous, since the goroutine allocation much more effectively splits the workload. On typical streams this will typically use 2 cores fully for decompression. When a stream has finished decoding no goroutines will be left over, so decoders can now safely be pooled and still be garbage collected.

While the release has been extensively tested, it is recommended to testing when upgrading.

text

See changes to v1.14.x

Feb 22, 2022 (v1.14.4)
- flate: Fix rare huffman only (-2) corruption. #503
- zip: Update deprecated CreateHeaderRaw to correctly call CreateRaw by @saracen in #502
- zip: don't read data descriptor early by @saracen in #501 #501
- huff0: Use static decompression buffer up to 30% faster #499 #500
Feb 17, 2022 (v1.14.3)
- flate: Improve fastest levels compression speed ~10% more throughput. #482 #489 #490 #491 #494 #478
- flate: Faster decompression speed, ~5-10%. #483
- s2: Faster compression with Go v1.18 and amd64 microarch level 3+. #484 #486
Jan 25, 2022 (v1.14.2)
- zstd: improve header decoder by @dsnet #476
- zstd: Add bigger default blocks #469
- zstd: Remove unused decompression buffer #470
- zstd: Fix logically dead code by @ningmingxiao #472
- flate: Improve level 7-9 #471 #473
- zstd: Add noasm tag for xxhash #475
Jan 11, 2022 (v1.14.1)
- s2: Add stream index in #462
- flate: Speed and efficiency improvements in #439 #461 #455 #452 #458
- zstd: Performance improvement in #420 #456 #437 #467 #468
- zstd: add arm64 xxhash assembly in #464
- Add garbled for binaries for s2 in #445
See changes to v1.13.x
Aug 30, 2021 (v1.13.5)
- gz/zlib/flate: Alias stdlib errors #425
- s2: Add block support to commandline tools #413
- zstd: pooledZipWriter should return Writers to the same pool #426
- Removed golang/snappy as external dependency for tests #421
Aug 12, 2021 (v1.13.4)
- Add snappy replacement package.
- zstd: Fix incorrect encoding in "best" mode #415
Aug 3, 2021 (v1.13.3)
- zstd: Improve Best compression #404
- zstd: Fix WriteTo error forwarding #411
- gzhttp: Return http.HandlerFunc instead of http.Handler. Unlikely breaking change. #406
- s2sx: Fix max size error #399
- zstd: Add optional stream content size on reset #401
- zstd: use SpeedBestCompression for level >= 10 #410
Jun 14, 2021 (v1.13.1)
- s2: Add full Snappy output support #396
- zstd: Add configurable Decoder window size #394
- gzhttp: Add header to skip compression #389
- s2: Improve speed with bigger output margin #395
Jun 3, 2021 (v1.13.0)
- Added gzhttp which allows wrapping HTTP servers and clients with GZIP compressors.
- zstd: Detect short invalid signatures #382
- zstd: Spawn decoder goroutine only if needed. #380
See changes to v1.12.x
May 25, 2021 (v1.12.3)
- deflate: Better/faster Huffman encoding #374
- deflate: Allocate less for history. #375
- zstd: Forward read errors #373
Apr 27, 2021 (v1.12.2)
- zstd: Improve better/best compression #360 #364 #365
- zstd: Add helpers to compress/decompress zstd inside zip files #363
- deflate: Improve level 5+6 compression #367
- s2: Improve better/best compression #358 #359
- s2: Load after checking src limit on amd64. #362
- s2sx: Limit max executable size #368
Apr 14, 2021 (v1.12.1)
- snappy package removed. Upstream added as dependency.
- s2: Better compression in "best" mode #353
- s2sx: Add stdin input and detect pre-compressed from signature #352
- s2c/s2d: Add http as possible input #348
- s2c/s2d/s2sx: Always truncate when writing files #352
- zstd: Reduce memory usage further when using WithLowerEncoderMem #346
- s2: Fix potential problem with amd64 assembly and profilers #349
See changes to v1.11.x
Mar 26, 2021 (v1.11.13)
- zstd: Big speedup on small dictionary encodes #344 #345
- zstd: Add WithLowerEncoderMem encoder option #336
- deflate: Improve entropy compression #338
- s2: Clean up and minor performance improvement in best #341
Mar 5, 2021 (v1.11.12)
- s2: Add s2sx binary that creates self extracting archives.
- s2: Speed up decompression on non-assembly platforms #328
Mar 1, 2021 (v1.11.9)
- s2: Add ARM64 decompression assembly. Around 2x output speed. #324
- s2: Improve "better" speed and efficiency. #325
- s2: Fix binaries.
Feb 25, 2021 (v1.11.8)
- s2: Fixed occasional out-of-bounds write on amd64. Upgrade recommended.
- s2: Add AMD64 assembly for better mode. 25-50% faster. #315
- s2: Less upfront decoder allocation. #322
- zstd: Faster "compression" of incompressible data. #314
- zip: Fix zip64 headers. #313
Jan 14, 2021 (v1.11.7)
- Use Bytes() interface to get bytes across packages. #309
- s2: Add 'best' compression option. #310
- s2: Add ReaderMaxBlockSize, changes s2.NewReader signature to include varargs. #311
- s2: Fix crash on small better buffers. #308
- s2: Clean up decoder. #312
Jan 7, 2021 (v1.11.6)
- zstd: Make decoder allocations smaller #306
- zstd: Free Decoder resources when Reset is called with a nil io.Reader #305
Dec 20, 2020 (v1.11.4)
- zstd: Add Best compression mode #304
- Add header decoder #299
- s2: Add uncompressed stream option #297
- Simplify/speed up small blocks with known max size. #300
- zstd: Always reset literal dict encoder #303
Nov 15, 2020 (v1.11.3)
- inflate: 10-15% faster decompression #293
- zstd: Tweak DecodeAll default allocation #295
Oct 11, 2020 (v1.11.2)
- s2: Fix out of bounds read in "better" block compression #291
Oct 1, 2020 (v1.11.1)
- zstd: Set allLitEntropy true in default configuration #286
Sept 8, 2020 (v1.11.0)
- zstd: Add experimental compression dictionaries #281
- zstd: Fix mixed Write and ReadFrom calls #282
- inflate/gz: Limit variable shifts, ~5% faster decompression #274
See changes to v1.10.x
July 8, 2020 (v1.10.11)
- zstd: Fix extra block when compressing with ReadFrom. #278
- huff0: Also populate compression table when reading decoding table. #275
June 23, 2020 (v1.10.10)
- zstd: Skip entropy compression in fastest mode when no matches. #270
June 16, 2020 (v1.10.9):
- zstd: API change for specifying dictionaries. See #268
- zip: update CreateHeaderRaw to handle zip64 fields. #266
- Fuzzit tests removed. The service has been purchased and is no longer available.
June 5, 2020 (v1.10.8):
- 1.15x faster zstd block decompression. #265
June 1, 2020 (v1.10.7):
- Added zstd decompression dictionary support
- Increase zstd decompression speed up to 1.19x. #259
- Remove internal reset call in zstd compression and reduce allocations. #263
May 21, 2020: (v1.10.6)
- zstd: Reduce allocations while decoding. #258, #252
- zstd: Stricter decompression checks.
April 12, 2020: (v1.10.5)
- s2-commands: Flush output when receiving SIGINT. #239
Apr 8, 2020: (v1.10.4)
- zstd: Minor/special case optimizations. #251, #250, #249, #247
Mar 11, 2020: (v1.10.3)
- s2: Use S2 encoder in pure Go mode for Snappy output as well. #245
- s2: Fix pure Go block encoder. #244
- zstd: Added "better compression" mode. #240
- zstd: Improve speed of fastest compression mode by 5-10% #241
- zstd: Skip creating encoders when not needed. #238
Feb 27, 2020: (v1.10.2)
- Close to 50% speedup in inflate (gzip/zip decompression). #236 #234 #232
- Reduce deflate level 1-6 memory usage up to 59%. #227
Feb 18, 2020: (v1.10.1)
- Fix zstd crash when resetting multiple times without sending data. #226
- deflate: Fix dictionary use on level 1-6. #224
- Remove deflate writer reference when closing. #224
Feb 4, 2020: (v1.10.0)
- Add optional dictionary to stateless deflate. Breaking change, send nil for previous behaviour. #216
- Fix buffer overflow on repeated small block deflate. #218
- Allow copying content from an existing ZIP file without decompressing+compressing. #214
- Added S2 AMD64 assembler and various optimizations. Stream speed >10GB/s. #186
See changes prior to v1.10.0
Jan 20,2020 (v1.9.8) Optimize gzip/deflate with better size estimates and faster table generation. #207 by luyu6056, #206.
Jan 11, 2020: S2 Encode/Decode will use provided buffer if capacity is big enough. #204
Jan 5, 2020: (v1.9.7) Fix another zstd regression in v1.9.5 - v1.9.6 removed.
Jan 4, 2020: (v1.9.6) Regression in v1.9.5 fixed causing corrupt zstd encodes in rare cases.
Jan 4, 2020: Faster IO in s2c + s2d commandline tools compression/decompression. #192
Dec 29, 2019: Removed v1.9.5 since fuzz tests showed a compatibility problem with the reference zstandard decoder.
Dec 29, 2019: (v1.9.5) zstd: 10-20% faster block compression. #199
Dec 29, 2019: zip package updated with latest Go features
Dec 29, 2019: zstd: Single segment flag condintions tweaked. #197
Dec 18, 2019: s2: Faster compression when ReadFrom is used. #198
Dec 10, 2019: s2: Fix repeat length output when just above at 16MB limit.
Dec 10, 2019: zstd: Add function to get decoder as io.ReadCloser. #191
Dec 3, 2019: (v1.9.4) S2: limit max repeat length. #188
Dec 3, 2019: Add WithNoEntropyCompression to zstd #187
Dec 3, 2019: Reduce memory use for tests. Check for leaked goroutines.
Nov 28, 2019 (v1.9.3) Less allocations in stateless deflate.
Nov 28, 2019: 5-20% Faster huff0 decode. Impacts zstd as well. #184
Nov 12, 2019 (v1.9.2) Added Stateless Compression for gzip/deflate.
Nov 12, 2019: Fixed zstd decompression of large single blocks. #180
Nov 11, 2019: Set default s2c block size to 4MB.
Nov 11, 2019: Reduce inflate memory use by 1KB.
Nov 10, 2019: Less allocations in deflate bit writer.
Nov 10, 2019: Fix inconsistent error returned by zstd decoder.
Oct 28, 2019 (v1.9.1) ztsd: Fix crash when compressing blocks. #174
Oct 24, 2019 (v1.9.0) zstd: Fix rare data corruption #173
Oct 24, 2019 zstd: Fix huff0 out of buffer write #171 and always return errors #172
Oct 10, 2019: Big deflate rewrite, 30-40% faster with better compression #105

See changes prior to v1.9.0
Oct 10, 2019: (v1.8.6) zstd: Allow partial reads to get flushed data. #169
Oct 3, 2019: Fix inconsistent results on broken zstd streams.
Sep 25, 2019: Added -rm (remove source files) and -q (no output except errors) to s2c and s2d commands
Sep 16, 2019: (v1.8.4) Add s2c and s2d commandline tools.
Sep 10, 2019: (v1.8.3) Fix s2 decoder Skip.
Sep 7, 2019: zstd: Added WithWindowSize, contributed by ianwilkes.
Sep 5, 2019: (v1.8.2) Add WithZeroFrames which adds full zero payload block encoding option.
Sep 5, 2019: Lazy initialization of zstandard predefined en/decoder tables.
Aug 26, 2019: (v1.8.1) S2: 1-2% compression increase in "better" compression mode.
Aug 26, 2019: zstd: Check maximum size of Huffman 1X compressed literals while decoding.
Aug 24, 2019: (v1.8.0) Added S2 compression, a high performance replacement for Snappy.
Aug 21, 2019: (v1.7.6) Fixed minor issues found by fuzzer. One could lead to zstd not decompressing.
Aug 18, 2019: Add fuzzit continuous fuzzing.
Aug 14, 2019: zstd: Skip incompressible data 2x faster. #147
Aug 4, 2019 (v1.7.5): Better literal compression. #146
Aug 4, 2019: Faster zstd compression. #143 #144
Aug 4, 2019: Faster zstd decompression. #145 #143 #142
July 15, 2019 (v1.7.4): Fix double EOF block in rare cases on zstd encoder.
July 15, 2019 (v1.7.3): Minor speedup/compression increase in default zstd encoder.
July 14, 2019: zstd decoder: Fix decompression error on multiple uses with mixed content.
July 7, 2019 (v1.7.2): Snappy update, zstd decoder potential race fix.
June 17, 2019: zstd decompression bugfix.
June 17, 2019: fix 32 bit builds.
June 17, 2019: Easier use in modules (less dependencies).
June 9, 2019: New stronger "default" zstd compression mode. Matches zstd default compression ratio.
June 5, 2019: 20-40% throughput in zstandard compression and better compression.
June 5, 2019: deflate/gzip compression: Reduce memory usage of lower compression levels.
June 2, 2019: Added zstandard compression!
May 25, 2019: deflate/gzip: 10% faster bit writer, mostly visible in lower levels.
Apr 22, 2019: zstd decompression added.
Aug 1, 2018: Added huff0 README.
Jul 8, 2018: Added Performance Update 2018 below.
Jun 23, 2018: Merged Go 1.11 inflate optimizations. Go 1.9 is now required. Backwards compatible version tagged with v1.3.0.
Apr 2, 2018: Added huff0 en/decoder. Experimental for now, API may change.
Mar 4, 2018: Added FSE Entropy en/decoder. Experimental for now, API may change.
Nov 3, 2017: Add compression Estimate function.
May 28, 2017: Reduce allocations when resetting decoder.
Apr 02, 2017: Change back to official crc32, since changes were merged in Go 1.7.
Jan 14, 2017: Reduce stack pressure due to array copies. See Issue #18625.
Oct 25, 2016: Level 2-4 have been rewritten and now offers significantly better performance than before.
Oct 20, 2016: Port zlib changes from Go 1.7 to fix zlib writer issue. Please update.
Oct 16, 2016: Go 1.7 changes merged. Apples to apples this package is a few percent faster, but has a significantly better balance between speed and compression per level.
Mar 24, 2016: Always attempt Huffman encoding on level 4-7. This improves base 64 encoded data compression.
Mar 24, 2016: Small speedup for level 1-3.
Feb 19, 2016: Faster bit writer, level -2 is 15% faster, level 1 is 4% faster.
Feb 19, 2016: Handle small payloads faster in level 1-3.
Feb 19, 2016: Added faster level 2 + 3 compression modes.
Feb 19, 2016: Rebalanced compression levels, so there is a more even progression in terms of compression. New default level is 5.
Feb 14, 2016: Snappy: Merge upstream changes.
Feb 14, 2016: Snappy: Fix aggressive skipping.
Feb 14, 2016: Snappy: Update benchmark.
Feb 13, 2016: Deflate: Fixed assembler problem that could lead to sub-optimal compression.
Feb 12, 2016: Snappy: Added AMD64 SSE 4.2 optimizations to matching, which makes easy to compress material run faster. Typical speedup is around 25%.
Feb 9, 2016: Added Snappy package fork. This version is 5-7% faster, much more on hard to compress content.
Jan 30, 2016: Optimize level 1 to 3 by not considering static dictionary or storing uncompressed. ~4-5% speedup.
Jan 16, 2016: Optimization on deflate level 1,2,3 compression.
Jan 8 2016: Merge CL 18317: fix reading, writing of zip64 archives.
Dec 8 2015: Make level 1 and -2 deterministic even if write size differs.
Dec 8 2015: Split encoding functions, so hashing and matching can potentially be inlined. 1-3% faster on AMD64. 5% faster on other platforms.
Dec 8 2015: Fixed rare one byte out-of bounds read. Please update!
Nov 23 2015: Optimization on token writer. ~2-4% faster. Contributed by @dsnet.
Nov 20 2015: Small optimization to bit writer on 64 bit systems.
Nov 17 2015: Fixed out-of-bound errors if the underlying Writer returned an error. See #15.
Nov 12 2015: Added io.WriterTo support to gzip/inflate.
Nov 11 2015: Merged CL 16669: archive/zip: enable overriding (de)compressors per file
Oct 15 2015: Added skipping on uncompressible data. Random data speed up >5x.

deflate usage

The packages are drop-in replacements for standard library deflate, gzip, zip, and zlib. Simply replace the import path to use them:

Typical speed is about 2x of the standard library packages.

old import	new import	Documentation
`compress/gzip`	`github.com/klauspost/compress/gzip`	gzip
`compress/zlib`	`github.com/klauspost/compress/zlib`	zlib
`archive/zip`	`github.com/klauspost/compress/zip`	zip
`compress/flate`	`github.com/klauspost/compress/flate`	flate

You may also be interested in pgzip, which is a drop-in replacement for gzip, which support multithreaded compression on big files and the optimized crc32 package used by these packages.

The packages implement the same API as the standard library, so you can use the original godoc documentation: gzip, zip, zlib, flate.

Currently there is only minor speedup on decompression (mostly CRC32 calculation).

Memory usage is typically 1MB for a Writer. stdlib is in the same range.
If you expect to have a lot of concurrently allocated Writers consider using
the stateless compression described below.

For compression performance, see: this spreadsheet.

To disable all assembly add -tags=noasm. This works across all packages.

Stateless compression

This package offers stateless compression as a special option for gzip/deflate.
It will do compression but without maintaining any state between Write calls.

This means there will be no memory kept between Write calls, but compression and speed will be suboptimal.

This is only relevant in cases where you expect to run many thousands of compressors concurrently,
but with very little activity. This is not intended for regular web servers serving individual requests.

Because of this, the size of actual Write calls will affect output size.

In gzip, specify level -3 / gzip.StatelessCompression to enable.

For direct deflate use, NewStatelessWriter and StatelessDeflate are available. See documentation

A bufio.Writer can of course be used to control write sizes. For example, to use a 4KB buffer:

// replace 'ioutil.Discard' with your output.
	gzw, err := gzip.NewWriterLevel(ioutil.Discard, gzip.StatelessCompression)
	if err != nil {
		return err
	}
	defer gzw.Close()

	w := bufio.NewWriterSize(gzw, 4096)
	defer w.Flush()
	
	// Write to 'w'

This will only use up to 4KB in memory when the writer is idle.

Compression is almost always worse than the fastest compression level
and each write will allocate (a little) memory.

Other packages

Here are other packages of good quality and pure Go (no cgo wrappers or autoconverted code):

github.com/pierrec/lz4 - strong multithreaded LZ4 compression.
github.com/cosnicolaou/pbzip2 - multithreaded bzip2 decompression.
github.com/dsnet/compress - brotli decompression, bzip2 writer.
github.com/ronanh/intcomp - Integer compression.
github.com/spenczar/fpc - Float compression.
github.com/minio/zipindex - External ZIP directory index.
github.com/ybirader/pzip - Fast concurrent zip archiver and extractor.

license

This code is licensed under the same conditions as the original Go code. See LICENSE file.

Overview (`vendor/github.com/mdlayher/socket/README.md`)

socket [](https://github.com/mdlayher/socket/actions) [](https://pkg.go.dev/github.com/mdlayher/socket)

Package socket provides a low-level network connection type which integrates
with Go's runtime network poller to provide asynchronous I/O and deadline
support. MIT Licensed.

This package focuses on UNIX-like operating systems which make use of BSD
sockets system call APIs. It is meant to be used as a foundation for the
creation of operating system-specific socket packages, for socket families such
as Linux's AF_NETLINK, AF_PACKET, or AF_VSOCK. This package should not be
used directly in end user applications.

Any use of package socket should be guarded by build tags, as one would also
use when importing the syscall or golang.org/x/sys packages.

Ecosystem

Over time, an ecosystem of Go packages has developed around package socket.
Many of these packages provide access to a specific socket family, such as
AF_NETLINK or AF_PACKET, and act as building blocks for further packages.

To have your package included in this diagram, please send a pull request!

text

/* Detailed source-code truncated for AI context efficiency. */

Stability

See the CHANGELOG file for a description of changes between
releases.

This package only supports the two most recent major versions of Go, mirroring
Go's own release policy. Older versions of Go may lack critical features and bug
fixes which are necessary for this package to function correctly.

Overview (`vendor/github.com/mdlayher/vsock/README.md`)

vsock [](https://github.com/mdlayher/vsock/actions) [](https://pkg.go.dev/github.com/mdlayher/vsock) [](https://goreportcard.com/report/github.com/mdlayher/vsock)

Package vsock provides access to Linux VM sockets (AF_VSOCK) for
communication between a hypervisor and its virtual machines. MIT Licensed.

For more information about VM sockets, see my blog about
Linux VM sockets in Go or
the QEMU wiki page on virtio-vsock.

Stability

See the CHANGELOG file for a description of changes between
releases.

This package has a stable v1 API and any future breaking changes will prompt
the release of a new major version. Features and bug fixes will continue to
occur in the v1.x.x series.

Overview (`vendor/github.com/miekg/dns/README.md`)

DNS version 2 is now available at https://codeberg.org/miekg/dns. This version should be 2x faster across
the board. Further optimizations welcome.

The version here will only see specific fixes and nothing else. At some point this repo will be archived.

Alternative (more granular) approach to a DNS library

Less is more.

Complete and usable DNS library. All Resource Records are supported, including the DNSSEC types.
It follows a lean and mean philosophy. If there is stuff you should know as a DNS programmer there
isn't a convenience function for it. Server side and client side programming is supported, i.e. you
can build servers and resolvers with it.

We try to keep the "master" branch as sane as possible and at the bleeding edge of standards,
avoiding breaking changes wherever reasonable. We support the last two versions of Go.

Goals

KISS;
Fast;
Small API. If it's easy to code in Go, don't make a function for it.

Users

A not-so-up-to-date-list-that-may-be-actually-current:

Send pull request if you want to be listed here.

Features

UDP/TCP queries, IPv4 and IPv6
RFC 1035 zone file parsing ($INCLUDE, $ORIGIN, $TTL and $GENERATE (for all record types) are supported
Fast
Server side programming (mimicking the net/http package)
Client side programming
DNSSEC: signing, validating and key generation for DSA, RSA, ECDSA and Ed25519
EDNS0, NSID, Cookies
AXFR/IXFR
TSIG, SIG(0)
DNS over TLS (DoT): encrypted connection between client and server over TCP
DNS name compression

Have fun!

Miek Gieben - 2010-2012 - [email protected]
DNS Authors 2012-

Building

This library uses Go modules and uses semantic versioning. Building is done with the go tool, so
the following should work:

text

go get github.com/miekg/dns
go build github.com/miekg/dns

Examples

A short "how to use the API" is at the beginning of doc.go (this also will show when you call godoc github.com/miekg/dns).

Example programs can be found in the github.com/miekg/exdns repository.

Supported RFCs

all of them

103{4,5} - DNS standard
1183 - ISDN, X25 and other deprecated records
1348 - NSAP record (removed the record)
1982 - Serial Arithmetic
1876 - LOC record
1995 - IXFR
1996 - DNS notify
2136 - DNS Update (dynamic updates)
2181 - RRset definition - there is no RRset type though, just []RR
2537 - RSAMD5 DNS keys
2065 - DNSSEC (updated in later RFCs)
2671 - EDNS record
2782 - SRV record
2845 - TSIG record
2915 - NAPTR record
2929 - DNS IANA Considerations
3110 - RSASHA1 DNS keys
3123 - APL record
3225 - DO bit (DNSSEC OK)
340{1,2,3} - NAPTR record
3445 - Limiting the scope of (DNS)KEY
3596 - AAAA record
3597 - Unknown RRs
4025 - A Method for Storing IPsec Keying Material in DNS
403{3,4,5} - DNSSEC + validation functions
4255 - SSHFP record
4343 - Case insensitivity
4408 - SPF record
4509 - SHA256 Hash in DS
4592 - Wildcards in the DNS
4635 - HMAC SHA TSIG
4701 - DHCID
4892 - id.server
5001 - NSID
5155 - NSEC3 record
5205 - HIP record
5702 - SHA2 in the DNS
5936 - AXFR
5966 - TCP implementation recommendations
6605 - ECDSA
6725 - IANA Registry Update
6742 - ILNP DNS
6840 - Clarifications and Implementation Notes for DNS Security
6844 - CAA record
6891 - EDNS0 update
6895 - DNS IANA considerations
6944 - DNSSEC DNSKEY Algorithm Status
6975 - Algorithm Understanding in DNSSEC
7043 - EUI48/EUI64 records
7314 - DNS (EDNS) EXPIRE Option
7477 - CSYNC RR
7828 - edns-tcp-keepalive EDNS0 Option
7553 - URI record
7858 - DNS over TLS: Initiation and Performance Considerations
7871 - EDNS0 Client Subnet
7873 - Domain Name System (DNS) Cookies
8080 - EdDSA for DNSSEC
8490 - DNS Stateful Operations
8499 - DNS Terminology
8659 - DNS Certification Authority Authorization (CAA) Resource Record
8777 - DNS Reverse IP Automatic Multicast Tunneling (AMT) Discovery
8914 - Extended DNS Errors
8976 - Message Digest for DNS Zones (ZONEMD RR)
9460 - Service Binding and Parameter Specification via the DNS
9461 - Service Binding Mapping for DNS Servers
9462 - Discovery of Designated Resolvers
9460 - SVCB and HTTPS Records
9567 - DNS Error Reporting
9606 - DNS Resolver Information
9660 - DNS Zone Version (ZONEVERSION) Option
Draft - Compact Denial of Existence in DNSSEC

Loosely Based Upon

ldns - https://nlnetlabs.nl/projects/ldns/about/
NSD - https://nlnetlabs.nl/projects/nsd/about/
Net::DNS - http://www.net-dns.org/
GRONG - https://github.com/bortzmeyer/grong

Overview (`vendor/github.com/mitchellh/copystructure/README.md`)

copystructure

copystructure is a Go library for deep copying values in Go.

This allows you to copy Go values that may contain reference values
such as maps, slices, or pointers, and copy their data as well instead
of just their references.

Installation

Standard go get:

text

$ go get github.com/mitchellh/copystructure

Usage & Example

For usage and examples see the Godoc.

The Copy function has examples associated with it there.

Overview (`vendor/github.com/mitchellh/reflectwalk/README.md`)

reflectwalk

reflectwalk is a Go library for "walking" a value in Go using reflection,
in the same way a directory tree can be "walked" on the filesystem. Walking
a complex structure can allow you to do manipulations on unknown structures
such as those decoded from JSON.

Overview (`vendor/github.com/moby/extensions/README.md`)

Extensions

[!WARNING]
This project is still a work in progress.

Moby extensions provide a common model for extending engine behavior.
They are intended to replace the separate plugin systems currently used by network, volume, and logging drivers.

An extension can run inside the Host process or as a separate process.
In both cases, it implements the same Go interface.

See the authoring guide for examples and setup instructions.
For details about the architecture, wire protocol, current limitations, and discovery security, see the design guide.

How it works

Extensions are built around Points.
A Point is an interface that describes a capability, such as running a workload.
Providers implement Points, and consumers call them.
The Host registers these components, connects consumers to providers, and manages their lifecycle.

For example, a daemon can expose its workload runtime as a Point.
An extension can then use that capability without depending on the daemon's internal runtime implementation.
The same mechanism is used for daemon-owned capabilities and communication between extensions.

To add an extension:

Define the contract.

A Point assigns a namespaced, versioned identity to a Go interface and its message types.
Declare what the extension provides and depends on.

Providers implement Points.
Consumers declare their dependencies using Point.Dependency().
Register the extension with the Host.

The daemon or application decides which extensions to load and supplies any generated wiring required for cross-process calls.
Merely importing an extension package does not activate it.
Resolve dependencies during initialization.

In Declaration.Init, consumers resolve their declared dependencies using Point.Single, Point.All, or Point.ByExtension.
The resolved implementation is not automatically assigned to a field.
Call the Point interface.

Consumers retain and call the resolved interface rather than a concrete Host or daemon type.
Calls go directly to an in-process provider or through generated gRPC code when the provider runs in another process.

The Host initializes providers before their consumers and shuts them down in the opposite order.
A provider must be ready for use when its Init method returns.

Keep the Host returned by host.New and call Shutdown when the application exits so that lifecycle resources are released.

Breaking changes require a new Point version.
The .v0 suffix identifies an experimental contract.
For fan-out calls, provider order is unspecified unless the Point defines its own ordering rules.

Running extensions

A Point defines an interface, not where its implementation runs.
In-process consumers call the Go implementation directly.
Cross-process consumers use generated gRPC wiring.

Because the same contract may cross a process boundary, Point interfaces and messages should not contain Host types, internal backend structures, callbacks, or channels.

The mobyextgen generator creates the required wire code:

ServerPoint exposes a Point implementation over gRPC.
ClientPoint presents the remote implementation through the same typed Go interface.

The exact setup depends on where the consumer and provider run:

Register in-process extensions with host.WithExtensions.
When the Host calls a provider in a launched process, configure the Host with host.WithClientProviders and the generated ClientPoint.
The provider registers its generated ServerPoint with the SDK.
When a launched consumer calls a Host provider, configure the Host with host.WithDependencyProviders and the generated ServerPoint.
In the consumer, pass the generated ClientPoint to Server.Depends.

The last case exposes an existing Host provider to the launched process.
The consumer must still declare the dependency itself.

A dependency callback currently exposes one effective provider for each Point.
Do not retain and call that dependency during Shutdown, because the SDK closes the connection before shutdown begins.

For launched processes, the ready\n handshake only confirms that the listener is ready.
It does not mean that the extension's Init method has completed.

See the authoring guide for complete examples covering code generation, SDK registration, dependency resolution, and deployment.

Publishing services

Implementing a Point makes it available within the extension system, but does not automatically expose it to external callers.

Publishing a Point requires three things:

The extension offers it using servicev0.Offer.
Host policy permits it through host.WithProviderPolicy.
The generated server wiring is registered.

For an in-process provider, supply the server wiring through host.WithPointServers.
A provider running in a separate process registers its ServerPoint with the SDK.

An offer is only a request to publish a Point; it does not grant access or register the transport.
If no provider policy is configured, external publication is disabled, while providers remain available to internal consumers.

Published services are responsible for enforcing their own access control.

See the publication examples for complete configurations and generated client usage.

Glossary

Point

A named and versioned Go interface, together with its message types.
It describes a capability independently of its implementation or location.

Extension

A component that can provide Points, depend on other Points or extensions, and perform initialization and shutdown work.
It may run inside the Host or in a separate process.

Host

The framework component embedded in a daemon or application.
It registers extensions, resolves their dependencies, and manages their lifecycle.

Provider

An extension that implements a Point.

Consumer

Code that resolves and calls a Point.
An extension may be both a consumer and a provider.

Dependency

A declared requirement for a Point provider or another extension.
The Host uses dependencies to verify that requirements are available and to determine initialization order.

Host capability

Functionality owned by the daemon or application and exposed as a Point through an in-process extension.

ClientPoint / ServerPoint

Generated code used to call a Point across a process boundary.
ClientPoint presents the Point's Go interface to the consumer, while ServerPoint forwards incoming calls to the implementation.

Offer

A request from an extension to make one or more of its Points available to external callers.

Publication

The Host exposing an offered Point to external callers after applying its policy and registering the required server wiring.

License

Licensed under the Apache License, Version 2.0.

Overview (`vendor/github.com/moby/ipvs/README.md`)

ipvs - networking for containers

ipvs provides a native Go implementation for communicating with IPVS kernel module using a netlink socket.

Using ipvs

import (
	"log"

	"github.com/moby/ipvs"
)

func main() {
	handle, err := ipvs.New("")
	if err != nil {
		log.Fatalf("ipvs.New: %s", err)
	}
	svcs, err := handle.GetServices()
	if err != nil {
		log.Fatalf("handle.GetServices: %s", err)
	}
}

Contributing

Want to hack on ipvs? Docker's contributions guidelines apply.

Copyright and license

Overview (`vendor/github.com/moby/locker/README.md`)

Locker

locker provides a mechanism for creating finer-grained locking to help
free up more global locks to handle other tasks.

The implementation looks close to a sync.Mutex, however, the user must provide a
reference to use to refer to the underlying lock when locking and unlocking,
and unlock may generate an error.

If a lock with a given name does not exist when Lock is called, one is
created.
Lock references are automatically cleaned up on Unlock if nothing else is
waiting for the lock.

Usage

package important

import (
	"sync"
	"time"

	"github.com/moby/locker"
)

type important struct {
	locks *locker.Locker
	data  map[string]interface{}
	mu    sync.Mutex
}

func (i *important) Get(name string) interface{} {
	i.locks.Lock(name)
	defer i.locks.Unlock(name)
	return i.data[name]
}

func (i *important) Create(name string, data interface{}) {
	i.locks.Lock(name)
	defer i.locks.Unlock(name)

	i.createImportant(data)

	i.mu.Lock()
	i.data[name] = data
	i.mu.Unlock()
}

func (i *important) createImportant(data interface{}) {
	time.Sleep(10 * time.Second)
}

For functions dealing with a given name, always lock at the beginning of the
function (or before doing anything with the underlying state), this ensures any
other function that is dealing with the same name will block.

When needing to modify the underlying data, use the global lock to ensure nothing
else is modifying it at the same time.
Since name lock is already in place, no reads will occur while the modification
is being performed.

Overview (`vendor/github.com/moby/policy-helpers/README.md`)

Work in progress policy helpers for Moby & BuildKit.

Overview (`vendor/github.com/moby/term/README.md`)

term - utilities for dealing with terminals

term provides structures and helper functions to work with terminal (state, sizes).

Using term

package main

import (
	"log"
	"os"

	"github.com/moby/term"
)

func main() {
	fd := os.Stdin.Fd()
	if term.IsTerminal(fd) {
		ws, err := term.GetWinsize(fd)
		if err != nil {
			log.Fatalf("term.GetWinsize: %s", err)
		}
		log.Printf("%d:%d\n", ws.Height, ws.Width)
	}
}

Contributing

Want to hack on term? Docker's contributions guidelines apply.

Copyright and license

Overview (`vendor/github.com/morikuni/aec/README.md`)

aec

Go wrapper for ANSI escape code.

Install

bash

go get github.com/morikuni/aec

Features

ANSI escape codes depend on terminal environment.
Some of these features may not work.
Check supported Font-Style/Font-Color features with checkansi.

Wikipedia for more detail.

Cursor

Up(n)
Down(n)
Right(n)
Left(n)
NextLine(n)
PreviousLine(n)
Column(col)
Position(row, col)
Save
Restore
Hide
Show
Report

Erase

EraseDisplay(mode)
EraseLine(mode)

Scroll

ScrollUp(n)
ScrollDown(n)

Font Style

Bold
Faint
Italic
Underline
BlinkSlow
BlinkRapid
Inverse
Conceal
CrossOut
Frame
Encircle
Overline

Font Color

Foreground color.

DefaultF
BlackF
RedF
GreenF
YellowF
BlueF
MagentaF
CyanF
WhiteF
LightBlackF
LightRedF
LightGreenF
LightYellowF
LightBlueF
LightMagentaF
LightCyanF
LightWhiteF
Color3BitF(color)
Color8BitF(color)
FullColorF(r, g, b)

Background color.

DefaultB
BlackB
RedB
GreenB
YellowB
BlueB
MagentaB
CyanB
WhiteB
LightBlackB
LightRedB
LightGreenB
LightYellowB
LightBlueB
LightMagentaB
LightCyanB
LightWhiteB
Color3BitB(color)
Color8BitB(color)
FullColorB(r, g, b)

Color Converter

24bit RGB color to ANSI color.

NewRGB3Bit(r, g, b)
NewRGB8Bit(r, g, b)

Builder

To mix these features.

custom := aec.EmptyBuilder.Right(2).RGB8BitF(128, 255, 64).RedB().ANSI
custom.Apply("Hello World")

Usage

Create ANSI by aec.XXX().With(aec.YYY()) or aec.EmptyBuilder.XXX().YYY().ANSI
Print ANSI by fmt.Print(ansi, "some string", aec.Reset) or fmt.Print(ansi.Apply("some string"))

aec.Reset should be added when using font style or font color features.

Example

Simple progressbar.

package main

import (
	"fmt"
	"strings"
	"time"

	"github.com/morikuni/aec"
)

func main() {
	const n = 20
	builder := aec.EmptyBuilder

	up2 := aec.Up(2)
	col := aec.Column(n + 2)
	bar := aec.Color8BitF(aec.NewRGB8Bit(64, 255, 64))
	label := builder.LightRedF().Underline().With(col).Right(1).ANSI

	// for up2
	fmt.Println()
	fmt.Println()

	for i := 0; i <= n; i++ {
		fmt.Print(up2)
		fmt.Println(label.Apply(fmt.Sprint(i, "/", n)))
		fmt.Print("[")
		fmt.Print(bar.Apply(strings.Repeat("=", i)))
		fmt.Println(col.Apply("]"))
		time.Sleep(100 * time.Millisecond)
	}
}

License

MIT

Overview (`vendor/github.com/opencontainers/cgroups/README.md`)

OCI Project Template

Useful boilerplate and organizational information for all OCI projects.

Overview (`vendor/github.com/opencontainers/go-digest/README.md`)

go-digest

Common digest package used across the container ecosystem.

Please see the godoc for more information.

What is a digest?

A digest is just a hash.

The most common use case for a digest is to create a content identifier for use in Content Addressable Storage systems:

id := digest.FromBytes([]byte("my content"))

In the example above, the id can be used to uniquely identify the byte slice "my content".
This allows two disparate applications to agree on a verifiable identifier without having to trust one another.

An identifying digest can be verified, as follows:

if id != digest.FromBytes([]byte("my content")) {
  return errors.New("the content has changed!")
}

A Verifier type can be used to handle cases where an io.Reader makes more sense:

rd := getContent()
verifier := id.Verifier()
io.Copy(verifier, rd)

if !verifier.Verified() {
  return errors.New("the content has changed!")
}

Using Merkle DAGs, this can power a rich, safe, content distribution system.

Usage

While the godoc is considered the best resource, a few important items need to be called out when using this package.

Make sure to import the hash implementations into your application or the package will panic.
You should have something like the following in the main (or other entrypoint) of your application:
go
```
import (
    _ "crypto/sha256"
    _ "crypto/sha512"
)
```
This may seem inconvenient but it allows you replace the hash
implementations with others, such as https://github.com/stevvooe/resumable.
Even though digest.Digest may be assemblable as a string, always verify your input with digest.Parse or use Digest.Validate when accepting untrusted input.
While there are measures to avoid common problems, this will ensure you have valid digests in the rest of your application.
While alternative encodings of hash values (digests) are possible (for example, base64), this package deals exclusively with hex-encoded digests.

Stability

The Go API, at this stage, is considered stable, unless otherwise noted.

As always, before using a package export, read the godoc.

Contributing

This package is considered fairly complete.
It has been in production in thousands (millions?) of deployments and is fairly battle-hardened.
New additions will be met with skepticism.
If you think there is a missing feature, please file a bug clearly describing the problem and the alternatives you tried before submitting a PR.

Code of Conduct

Participation in the OpenContainers community is governed by OpenContainer's Code of Conduct.

Security

If you find an issue, please follow the security protocol to report it.

Copyright and license

Copyright © 2019, 2020 OCI Contributors
Copyright © 2016 Docker, Inc.
All rights reserved, except as follows.
Code is released under the Apache 2.0 license.
This README.md file and the CONTRIBUTING.md file are licensed under the Creative Commons Attribution 4.0 International License under the terms and conditions set forth in the file LICENSE.docs.
You may obtain a duplicate copy of the same license, titled CC BY-SA 4.0, at http://creativecommons.org/licenses/by-sa/4.0/.

Overview (`vendor/github.com/package-url/packageurl-go/README.md`)

packageurl-go

Go implementation of the package url spec.

Install

text

go get -u github.com/package-url/packageurl-go

Versioning

The versions will follow the spec. So if the spec is released at 1.0. Then all versions in the 1.x.y will follow the 1.x spec.

Usage

Create from parts

package main

import (
	"fmt"

	"github.com/package-url/packageurl-go"
)

func main() {
	instance := packageurl.NewPackageURL("test", "ok", "name", "version", nil, "")
	fmt.Printf("%s", instance.ToString())
}

Parse from string

package main

import (
	"fmt"

	"github.com/package-url/packageurl-go"
)

func main() {
	instance, err := packageurl.FromString("test:ok/name@version")
	if err != nil {
		panic(err)
	}
	fmt.Printf("%#v", instance)
}

Test

Testing using the normal go test command. Using make test will pull the test fixtures shared between all package-url projects and then execute the tests.

text

curl -Ls https://raw.githubusercontent.com/package-url/purl-spec/master/test-suite-data.json -o testdata/test-suite-data.json
go test -v -cover ./...
=== RUN   TestFromStringExamples
--- PASS: TestFromStringExamples (0.00s)
=== RUN   TestToStringExamples
--- PASS: TestToStringExamples (0.00s)
=== RUN   TestStringer
--- PASS: TestStringer (0.00s)
=== RUN   TestQualifiersMapConversion
--- PASS: TestQualifiersMapConversion (0.00s)
PASS
        github.com/package-url/packageurl-go    coverage: 90.7% of statements
ok      github.com/package-url/packageurl-go    0.004s  coverage: 90.7% of statements

Fuzzing

Fuzzing is done with standard Go fuzzing, introduced in Go 1.18.

Fuzz tests check for inputs that cause FromString to panic.

Using make fuzz will run fuzz tests for one minute.

To run fuzz tests longer:

text

go test -fuzztime=60m -fuzz .

Or omit -fuzztime entirely to run indefinitely.

Overview (`vendor/github.com/pelletier/go-toml/README.md`)

go-toml

Go library for the TOML format.

This library supports TOML version
v1.0.0-rc.3

Development status

ℹ️ Consider go-toml v2!

The next version of go-toml is in active development, and
nearing completion.

Though technically in beta, v2 is already more tested, fixes bugs,
and much faster. If you only need reading and writing TOML documents
(majority of cases), those features are implemented and the API unlikely to
change.

The remaining features will be added shortly. While pull-requests are welcome on
v1, no active development is expected on it. When v2.0.0 is released, v1 will be
deprecated.

👉 go-toml v2

Features

Go-toml provides the following features for using data parsed from TOML documents:

Load TOML documents from files and string data
Easily navigate TOML structure using Tree
Marshaling and unmarshaling to and from data structures
Line & co

Docker & Self-Hosted Deployment Verified OCI

Technical Documentation: docker/Docker

Overview (`README.md`) Copy Section

The Moby Project

Principles Copy Chapter

Audience Copy Chapter

Relationship with Docker Copy Chapter

Go modules Copy Chapter

Release tags Copy Section

Migrating from `github.com/docker/docker` Copy Section

Legal

Licensing

Overview (`api/README.md`) Copy Section

Engine API

Swagger definition Copy Chapter

Updating the API documentation Copy Chapter

Viewing the API documentation Copy Chapter

Overview (`client/README.md`) Copy Section

Go client for the Docker Engine API

Overview (`contrib/README.md`) Copy Section

Overview (`hack/README.md`) Copy Section

About Copy Chapter

DinD (dind.sh) Copy Chapter

Generate Authors (generate-authors.sh) Copy Chapter

Make Copy Chapter

make.ps1 Copy Section

make.sh Copy Section

Vendor (vendor.sh) Copy Chapter

Overview (`man/README.md`) Copy Section

Docker Engine Documentation

Generate the man pages Copy Chapter

Install the man pages Copy Chapter

Add a new man page Copy Chapter

Overview (`pkg/README.md`) Copy Section

Overview (`project/README.md`) Copy Section

Hacking on Docker

Guides Copy Chapter

Roadmap Copy Chapter

Build tools Copy Chapter

Overview (`api/docs/README.md`) Copy Section

API Documentation

Overview (`contrib/otel/README.md`) Copy Section

Sample stack for testing OTEL functionality with moby

How can I use it? Copy Chapter

Cleanup? Copy Chapter

Overview (`contrib/wireshark/README.md`) Copy Section

Wireshark Dissectors for NetworkDB

Installation Copy Chapter

Install Wireshark 4.5 or newer. Copy Section

Install the plugins Copy Section

Configure the ProtoBuf protocol Copy Section

Configure the Memberlist protocol Copy Section

Usage Notes Copy Chapter

Encryption Copy Section

Known Issues Copy Section

Limitations Copy Section

Overview (`docs/api/README.md`) Copy Section

The API documentation has moved

Overview (`docs/contributing/README.md`) Copy Section

Get set up for Moby development Copy Section

Overview (`hack/make/README.md`) Copy Section

Overview (`pkg/useragent/README.md`) Copy Section

Overview (`pkg/plugins/pluginrpc-gen/README.md`) Copy Section

Plugin RPC Generator

Usage Copy Chapter

Annotations Copy Chapter

Supported Annotations Copy Section

Usage Copy Section

Example

Default Behavior Copy Section

Known issues Copy Chapter

go-generate Copy Chapter

Overview (`vendor/cloud.google.com/go/README.md`) Copy Section

Google Cloud Client Libraries for Go

Installation Copy Chapter

Supported APIs Copy Chapter

[Go Versions Supported](#supported-versions) Copy Chapter

Authentication Copy Chapter

Contributing Copy Chapter

Links Copy Chapter

Overview (`README.md`)

Principles

Audience

Relationship with Docker

Go modules

Release tags

Migrating from `github.com/docker/docker`

Overview (`api/README.md`)

Swagger definition

Updating the API documentation

Viewing the API documentation

Overview (`client/README.md`)

Overview (`contrib/README.md`)

Overview (`hack/README.md`)

About

DinD (dind.sh)

Generate Authors (generate-authors.sh)

Make

make.ps1

make.sh

Vendor (vendor.sh)

Overview (`man/README.md`)

Generate the man pages

Install the man pages

Add a new man page

Overview (`pkg/README.md`)

Overview (`project/README.md`)

Guides

Roadmap

Build tools

Overview (`api/docs/README.md`)

Overview (`contrib/otel/README.md`)

How can I use it?

Cleanup?

Overview (`contrib/wireshark/README.md`)

Installation

Install Wireshark 4.5 or newer.

Install the plugins

Configure the ProtoBuf protocol

Configure the Memberlist protocol

Usage Notes

Encryption

Known Issues

Limitations

Overview (`docs/api/README.md`)

Overview (`docs/contributing/README.md`)

Get set up for Moby development

Overview (`hack/make/README.md`)

Overview (`pkg/useragent/README.md`)

Overview (`pkg/plugins/pluginrpc-gen/README.md`)

Usage

Annotations

Supported Annotations

Usage

Default Behavior

Known issues

go-generate

Overview (`vendor/cloud.google.com/go/README.md`)

Installation

Supported APIs

[Go Versions Supported](#supported-versions)

Authentication

Contributing

Links

Overview (`vendor/code.cloudfoundry.org/clock/README.md`)

Overview (`vendor/dario.cat/mergo/README.md`)

Status

Important notes

Donations

Mergo in the wild

Install

Usage

Transformers

Contact me

About

License

Overview (`vendor/go.etcd.io/bbolt/README.md`)

Project Status

Project versioning

Table of Contents