DISTRIBUTION Documentation

Technical Documentation: docker/DISTRIBUTION

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

The toolset to pack, ship, store, and deliver content.

This repository's main product is the Open Source Registry implementation
for storing and distributing container images and other content using the
OCI Distribution Specification.
The goal of this project is to provide a simple, secure, and scalable base
for building a large scale registry solution or running a simple private registry.
It is a core library for many registry operators including Docker Hub, GitHub Container Registry,
GitLab Container Registry and DigitalOcean Container Registry, as well as the CNCF Harbor
Project, and VMware Harbor Registry.

This repository contains the following components:

How does this integrate with Docker, containerd, and other OCI client? Copy Section

Clients implement against the OCI specification and communicate with the
registry using HTTP. This project contains a client implementation which
is currently in use by Docker, however, it is deprecated for the
implementation in containerd
and will not support new features.

What are the long term goals of the Distribution project? Copy Section

The Distribution project has the further long term goal of providing a
secure tool chain for distributing content. The specifications, APIs and tools
should be as useful with Docker as they are without.

Our goal is to design a professional grade and extensible content distribution
system that allow users to:

• Enjoy an efficient, secured and reliable way to store, manage, package and
  exchange content
• Hack/roll their own on top of healthy open-source components
• Implement their own home made solution through good specs, and solid
  extensions mechanism.

Contribution Copy Chapter

Please see CONTRIBUTING.md for details on how to contribute
issues, fixes, and patches to this project. If you are contributing code, see
the instructions for building a development environment.

Communication Copy Chapter

For async communication and long running discussions please use issues and pull requests on the github repo.
This will be the best place to discuss design and implementation.

For sync communication we have a #distribution channel in the CNCF Slack
that everyone is welcome to join and chat about development.

Licenses Copy Chapter

The distribution codebase is released under the Apache 2.0 license.
The README.md file, and files in the "docs" folder are licensed under the
Creative Commons Attribution 4.0 International License. You may obtain a
copy of the license, titled CC-BY-4.0, at http://creativecommons.org/licenses/by/4.0/.

Overview (`releases/README.md`) Copy Section

Registry Release Checklist Copy Chapter

10. Compile release notes detailing features added since the last release.

Add release template file to releases/ directory. The template is defined
by containerd's release tool. Name the file using the version, for rc add
an -rc suffix.
See https://github.com/containerd/release-tool

20. Update the .mailmap files.

21. Update the version file: https://github.com/distribution/distribution/blob/master/version/version.go

22. Create a signed tag.

Choose a tag for the next release, distribution uses semantic versioning
and expects tags to be formatted as vx.y.z[-rc.n]. Run the release tool using
the release template file and tag to generate the release notes for the tag
and Github release. To create the tag, you will need PGP installed and a PGP
key which has been added to your Github account. The comment for the tag will
be the generated release notes, always compare with previous tags to ensure
the output is expected and consistent.
Run git tag --cleanup=whitespace -s vx.y.z[-rc.n] -F release-notes to create
tag and git -v vx.y.z[-rc.n] to verify tag, check comment and correct commit
hash.

50. Push the signed tag

51. Create a new release.
    In the case of a release candidate, tick the pre-release checkbox. Use
    the generate release notes from the release tool

52. Update the registry binary in the distribution library image repo by running the update script and opening a pull request.

53. Update the official image. Add the new version in the official images repo by appending a new version to the registry/registry file with the git hash pointed to by the signed tag. Update the major version to point to the latest version and the minor version to point to new patch release if necessary.
    e.g. to release 2.3.1

2.3.1 (new)

2.3.0 -> 2.3.0 can be removed

2 -> 2.3.1

2.3 -> 2.3.1

90. Build a new distribution/registry image on Docker Hub by adding a new automated build with the new tag and re-building the images.

Overview (`project/hooks/README.md`) Copy Section

Git Hooks

To enforce valid and properly-formatted code, there is CI in place which runs gofmt, golint, and go vet against code in the repository.

As an aid to prevent committing invalid code in the first place, a git pre-commit hook has been added to the repository, found in pre-commit. As it is impossible to automatically add linked hooks to a git repository, this hook should be linked into your .git/hooks/pre-commit, which can be done by running the configure-hooks.sh script in this directory. This script is the preferred method of configuring hooks, as it will be updated as more are added.

Overview (`docs/themes/hugo-geekdoc/README.md`) Copy Section

Geekdoc

Geekdoc is a simple Hugo theme for documentations. It is intentionally designed as a fast and lean theme and may not fit the requirements of complex projects. If a more feature-complete theme is required there are a lot of good alternatives out there. You can find a demo and the full documentation at https://geekdocs.de.

Build and release process Copy Chapter

This theme is subject to a CI driven build and release process common for software development. During the release build, all necessary assets are automatically built by webpack and bundled in a release tarball. You can download the latest release from the GitHub release page.

Due to the fact that webpack and npm scripts are used as pre-processors, the theme cannot be used from the main branch by default. If you want to use the theme from a cloned branch instead of a release tarball you'll need to install webpack locally and run the build script once to create all required assets.

# install required packages from package.json
npm install

# run the build script to build required assets
npm run build

# build release tarball
npm run pack

See the Getting Started Guide for details about the different setup options.

Contributors Copy Chapter

Special thanks to all contributors. If you would like to contribute, please see the instructions.

Geekdoc is inspired and partially based on the hugo-book theme, thanks Alex Shpak for your work.

License Copy Chapter

This project is licensed under the MIT License - see the LICENSE file for details.

The used SVG icons and generated icon fonts are licensed under the license of the respective icon pack:

• Font Awesome: CC BY 4.0 License
• IcoMoon Free Pack: GPL/CC BY 4.0
• Material Icons: Apache License 2.0

Overview (`vendor/cel.dev/expr/README.md`) Copy Section

Common Expression Language

The Common Expression Language (CEL) implements common semantics for expression
evaluation, enabling different applications to more easily interoperate.

Key Applications

• Security policy: organizations have complex infrastructure and need common
  tooling to reason about the system as a whole
• Protocols: expressions are a useful data type and require interoperability
  across programming languages and platforms.

Guiding philosophy:

1. Keep it small & fast.
   • CEL evaluates in linear time, is mutation free, and not Turing-complete.
     This limitation is a feature of the language design, which allows the
     implementation to evaluate orders of magnitude faster than equivalently
     sandboxed JavaScript.
2. Make it extensible.
   • CEL is designed to be embedded in applications, and allows for
     extensibility via its context which allows for functions and data to be
     provided by the software that embeds it.
3. Developer-friendly.
   • The language is approachable to developers. The initial spec was based
     on the experience of developing Firebase Rules and usability testing
     many prior iterations.
   • The library itself and accompanying toolings should be easy to adopt by
     teams that seek to integrate CEL into their platforms.

The required components of a system that supports CEL are:

• The textual representation of an expression as written by a developer. It is
  of similar syntax to expressions in C/C++/Java/JavaScript
• A representation of the program's abstract syntax tree (AST).
• A compiler library that converts the textual representation to the binary
  representation. This can be done ahead of time (in the control plane) or
  just before evaluation (in the data plane).
• A context containing one or more typed variables, often protobuf messages.
  Most use-cases will use attribute_context.proto
• An evaluator library that takes the binary format in the context and
  produces a result, usually a Boolean.

For use cases which require persistence or cross-process communcation, it is
highly recommended to serialize the type-checked expression as a protocol
buffer. The CEL team will maintains canonical protocol buffers for ASTs and
will keep these versions identical and wire-compatible in perpetuity:

• CEL canonical
• CEL v1alpha1

Example of boolean conditions and object construction:

// Condition
account.balance >= transaction.withdrawal
    || (account.overdraftProtection
    && account.overdraftLimit >= transaction.withdrawal  - account.balance)

// Object construction
common.GeoPoint{ latitude: 10.0, longitude: -5.5 }

For more detail, see:

• Introduction
• Language Definition

Released under the Apache License.

Overview (`vendor/go.opentelemetry.io/otel/README.md`) Copy Section

OpenTelemetry-Go

OpenTelemetry-Go is the Go implementation of OpenTelemetry.
It provides a set of APIs to directly measure performance and behavior of your software and send this data to observability platforms.

Project Status Copy Chapter

Progress and status specific to this repository is tracked in our
project boards
and
milestones.

Project versioning information and stability guarantees can be found in the
versioning documentation.

Compatibility Copy Section

OpenTelemetry-Go ensures compatibility with the current supported versions of
the Go language:

Each major Go release is supported until there are two newer major releases.
For example, Go 1.5 was supported until the Go 1.7 release, and Go 1.6 was supported until the Go 1.8 release.

For versions of Go that are no longer supported upstream, opentelemetry-go will
stop ensuring compatibility with these versions in the following manner:

• A minor release of opentelemetry-go will be made to add support for the new
  supported release of Go.
• The following minor release of opentelemetry-go will remove compatibility
  testing for the oldest (now archived upstream) version of Go. This, and
  future, releases of opentelemetry-go may include features only supported by
  the currently supported versions of Go.

Currently, this project supports the following environments.

While this project should work for other systems, no compatibility guarantees
are made for those systems currently.

Getting Started Copy Chapter

You can find a getting started guide on opentelemetry.io.

OpenTelemetry's goal is to provide a single set of APIs to capture distributed
traces and metrics from your application and send them to an observability
platform. This project allows you to do just that for applications written in
Go. There are two steps to this process: instrument your application, and
configure an exporter.

Instrumentation Copy Section

To start capturing distributed traces and metric events from your application
it first needs to be instrumented. The easiest way to do this is by using an
instrumentation library for your code. Be sure to check out the officially
supported instrumentation
libraries.

If you need to extend the telemetry an instrumentation library provides or want
to build your own instrumentation for your application directly you will need
to use the
Go otel
package. The examples
are a good way to see some practical uses of this process.

Export Copy Section

Now that your application is instrumented to collect telemetry, it needs an
export pipeline to send that telemetry to an observability platform.

All officially supported exporters for the OpenTelemetry project are contained in the exporters directory.

Contributing Copy Chapter

See the contributing documentation.

Emeritus Copy Section

• Alex Kats, Triager

For more information about the emeritus role, see the
community repository.

Overview (`vendor/google.golang.org/grpc/README.md`) Copy Section

gRPC-Go

The Go implementation of gRPC: A high performance, open source, general
RPC framework that puts mobile and HTTP/2 first. For more information see the
Go gRPC docs, or jump directly into the quick start.

Prerequisites Copy Chapter

• Go: any one of the two latest major releases.

Installation Copy Chapter

Simply add the following import to your code, and then go [build|run|test]
will automatically fetch the necessary dependencies:

import "google.golang.org/grpc"

Note: If you are trying to access grpc-go from China, see the
FAQ below.

Learn more Copy Chapter

• Go gRPC docs, which include a quick start and API
  reference among other resources
• Low-level technical docs from this repository
• Performance benchmark
• Examples
• Contribution guidelines

FAQ Copy Chapter

I/O Timeout Errors Copy Section

The golang.org domain may be blocked from some countries. go get usually
produces an error like the following when this happens:

$ go get -u google.golang.org/grpc
package google.golang.org/grpc: unrecognized import path "google.golang.org/grpc" (https fetch: Get https://google.golang.org/grpc?go-get=1: dial tcp 216.239.37.1:443: i/o timeout)

To build Go code, there are several options:

• Set up a VPN and access google.golang.org through that.

• With Go module support: it is possible to use the replace feature of go mod to create aliases for golang.org packages. In your project's directory:

  sh Copy

  go mod edit -replace=google.golang.org/grpc=github.com/grpc/grpc-go@latest
  go mod tidy
  go mod vendor
  go build -mod=vendor

  Again, this will need to be done for all transitive dependencies hosted on
  golang.org as well. For details, refer to golang/go issue
  #28652.

Compiling error, undefined: grpc.SupportPackageIsVersion Copy Section

Please update to the latest version of gRPC-Go using
go get google.golang.org/grpc.

How to turn on logging Copy Section

The default logger is controlled by environment variables. Turn everything on
like this:

$ export GRPC_GO_LOG_VERBOSITY_LEVEL=99
$ export GRPC_GO_LOG_SEVERITY_LEVEL=info

The RPC failed with error `"code = Unavailable desc = transport is closing"` Copy Section

This error means the connection the RPC is using was closed, and there are many
possible reasons, including:

1. mis-configured transport credentials, connection failed on handshaking
2. bytes disrupted, possibly by a proxy in between
3. server shutdown
4. Keepalive parameters caused connection shutdown, for example if you have
   configured your server to terminate connections regularly to trigger DNS
   lookups.
   If this is the case, you may want to increase your
   MaxConnectionAgeGrace,
   to allow longer RPC calls to finish.

It can be tricky to debug this because the error happens on the client side but
the root cause of the connection being closed is on the server side. Turn on
logging on both client and server, and see if there are any transport
errors.

Overview (`vendor/cloud.google.com/go/auth/README.md`) Copy Section

Google Auth Library for Go

Install Copy Chapter

go get cloud.google.com/go/auth@latest

Usage Copy Chapter

The most common way this library is used is transitively, by default, from any
of our Go client libraries.

Notable use-cases Copy Section

• To create a credential directly please see examples in the
  credentials
  package.
• To create a authenticated HTTP client please see examples in the
  httptransport
  package.
• To create a authenticated gRPC connection please see examples in the
  grpctransport
  package.
• To create an ID token please see examples in the
  idtoken
  package.

Contributing Copy Chapter

Contributions are welcome. Please, see the
CONTRIBUTING
document for details.

Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms.
See Contributor Code of Conduct
for more information.

Overview (`vendor/cloud.google.com/go/iam/README.md`) Copy Section

IAM API

Go Client Library for IAM API.

Install Copy Chapter

go get cloud.google.com/go/iam

Stability Copy Chapter

The stability of this module is indicated by SemVer.

However, a v1+ module may have breaking changes in two scenarios:

• Packages with alpha or beta in the import path
• The GoDoc has an explicit stability disclaimer (for example, for an experimental feature).

Go Version Support Copy Chapter

See the Go Versions Supported
section in the root directory's README.

Authorization Copy Chapter

See the Authorization
section in the root directory's README.

Contributing Copy Chapter

Contributions are welcome. Please, see the CONTRIBUTING
document for details.

Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms. See
Contributor Code of Conduct
for more information.

Overview (`vendor/cloud.google.com/go/internal/README.md`) Copy Section

Internal

This directory contains internal code for cloud.google.com/go packages.

.repo-metadata-full.json Copy Chapter

.repo-metadata-full.json contains metadata about the packages in this repo. It
is generated by internal/gapicgen/generator. It's processed by external tools
to build lists of all of the packages.

Don't make breaking changes to the format without consulting with the external
tools.

One day, we may want to create individual .repo-metadata.json files next to
each package, which is the pattern followed by some other languages. External
tools would then talk to pkg.go.dev or some other service to get the overall
list of packages and use the .repo-metadata.json files to get the additional
metadata required. For now, .repo-metadata-full.json includes everything.

Updating OwlBot SHA Copy Section

You may want to manually update the which version of the post-processor will be
used -- to do this you need to update the SHA in the OwlBot lock file.

See the postprocessor/README for detailed
instructions.

Note: OwlBot will eventually open a pull request to update this value if it
discovers a new version of the container.

Overview (`vendor/cloud.google.com/go/storage/README.md`) Copy Section

Cloud Storage [](https://pkg.go.dev/cloud.google.com/go/storage) Copy Chapter

• About Cloud Storage
• API documentation
• Go client documentation
• Complete sample programs

Example Usage Copy Section

First create a storage.Client to use throughout your application:

client, err := storage.NewClient(ctx)
if err != nil {
	log.Fatal(err)
}

// Read the object1 from bucket.
rc, err := client.Bucket("bucket").Object("object1").NewReader(ctx)
if err != nil {
	log.Fatal(err)
}
defer rc.Close()
body, err := io.ReadAll(rc)
if err != nil {
	log.Fatal(err)
}

Overview (`vendor/github.com/AdaLogics/go-fuzz-headers/README.md`) Copy Section

go-fuzz-headers

This repository contains various helper functions for go fuzzing. It is mostly used in combination with go-fuzz, but compatibility with fuzzing in the standard library will also be supported. Any coverage guided fuzzing engine that provides an array or slice of bytes can be used with go-fuzz-headers.

Usage Copy Chapter

Using go-fuzz-headers is easy. First create a new consumer with the bytes provided by the fuzzing engine:

import (
	fuzz "github.com/AdaLogics/go-fuzz-headers"
)
data := []byte{'R', 'a', 'n', 'd', 'o', 'm'}
f := fuzz.NewConsumer(data)

This creates a Consumer that consumes the bytes of the input as it uses them to fuzz different types.

After that, f can be used to easily create fuzzed instances of different types. Below are some examples:

Structs Copy Section

One of the most useful features of go-fuzz-headers is its ability to fill structs with the data provided by the fuzzing engine. This is done with a single line:

type Person struct {
    Name string
    Age  int
}
p := Person{}
// Fill p with values based on the data provided by the fuzzing engine:
err := f.GenerateStruct(&p)

This includes nested structs too. In this example, the fuzz Consumer will also insert values in p.BestFriend:

type PersonI struct {
    Name       string
    Age        int
    BestFriend PersonII
}
type PersonII struct {
    Name string
    Age  int
}
p := PersonI{}
err := f.GenerateStruct(&p)

If the consumer should insert values for unexported fields as well as exported, this can be enabled with:

f.AllowUnexportedFields()

...and disabled with:

f.DisallowUnexportedFields()

Other types: Copy Section

Other useful APIs:

createdString, err := f.GetString() // Gets a string
createdInt, err := f.GetInt() // Gets an integer
createdByte, err := f.GetByte() // Gets a byte
createdBytes, err := f.GetBytes() // Gets a byte slice
createdBool, err := f.GetBool() // Gets a boolean
err := f.FuzzMap(target_map) // Fills a map
createdTarBytes, err := f.TarBytes() // Gets bytes of a valid tar archive
err := f.CreateFiles(inThisDir) // Fills inThisDir with files
createdString, err := f.GetStringFrom("anyCharInThisString", ofThisLength) // Gets a string that consists of chars from "anyCharInThisString" and has the exact length "ofThisLength"

Most APIs are added as they are needed.

Projects that use go-fuzz-headers Copy Chapter

• runC
• Istio
• Vitess
• Containerd

Feel free to add your own project to the list, if you use go-fuzz-headers to fuzz it.

Status Copy Chapter

The project is under development and will be updated regularly.

References Copy Chapter

go-fuzz-headers' approach to fuzzing structs is strongly inspired by gofuzz.

Overview (`vendor/github.com/cyphar/filepath-securejoin/README.md`) Copy Section

`filepath-securejoin` Copy Chapter

An implementation of SecureJoin, a candidate for inclusion in the Go
standard library. The purpose of this function is to be a "secure"
alternative to filepath.Join, and in particular it provides certain
guarantees that are not provided by filepath.Join.

NOTE: This code is only safe if you are not at risk of other processes
modifying path components after you've used SecureJoin. If it is possible
for a malicious process to modify path components of the resolved path, then
you will be vulnerable to some fairly trivial TOCTOU race conditions. There
are some Linux kernel patches I'm working on which might allow for a better
solution.

In addition, with a slightly modified API it might be possible to use
O_PATH and verify that the opened path is actually the resolved one -- but
I have not done that yet. I might add it in the future as a helper function
to help users verify the path (we can't just return /proc/self/fd/<foo>
because that doesn't always work transparently for all users).

This is the function prototype:

func SecureJoin(root, unsafePath string) (string, error)

This library guarantees the following:

• If no error is set, the resulting string must be a child path of
  root and will not contain any symlink path components (they will all be
  expanded).

• When expanding symlinks, all symlink path components must be resolved
  relative to the provided root. In particular, this can be considered a
  userspace implementation of how chroot(2) operates on file paths. Note that
  these symlinks will not be expanded lexically (filepath.Clean is not
  called on the input before processing).

• Non-existent path components are unaffected by SecureJoin (similar to
  filepath.EvalSymlinks's semantics).

• The returned path will always be filepath.Cleaned and thus not contain any
  .. components.

A (trivial) implementation of this function on GNU/Linux systems could be done
with the following (note that this requires root privileges and is far more
opaque than the implementation in this library, and also requires that
readlink is inside the root path):

package securejoin

import (
	"os/exec"
	"path/filepath"
)

func SecureJoin(root, unsafePath string) (string, error) {
	unsafePath = string(filepath.Separator) + unsafePath
	cmd := exec.Command("chroot", root,
		"readlink", "--canonicalize-missing", "--no-newline", unsafePath)
	output, err := cmd.CombinedOutput()
	if err != nil {
		return "", err
	}
	expanded := string(output)
	return filepath.Join(root, expanded), nil
}

License Copy Section

The license of this project is the same as Go, which is a BSD 3-clause license
available in the LICENSE file.

Overview (`vendor/github.com/distribution/reference/README.md`) Copy Section

Distribution reference

Go library to handle references to container images.

This repository contains a library for handling references to container images held in container registries. Please see godoc for details.

Contribution Copy Chapter

Please see CONTRIBUTING.md for details on how to contribute
issues, fixes, and patches to this project.

Communication Copy Chapter

For async communication and long running discussions please use issues and pull requests on the github repo.
This will be the best place to discuss design and implementation.

For sync communication we have a #distribution channel in the CNCF Slack
that everyone is welcome to join and chat about development.

Licenses Copy Chapter

The distribution codebase is released under the Apache 2.0 license.

Overview (`vendor/github.com/docker/go-events/README.md`) Copy Section

Docker Events Package

The Docker events package implements a composable event distribution package
for Go.

Originally created to implement the notifications in Docker Registry
2,
we've found the pattern to be useful in other applications. This package is
most of the same code with slightly updated interfaces. Much of the internals
have been made available.

Usage Copy Chapter

The events package centers around a Sink type. Events are written with
calls to Sink.Write(event Event). Sinks can be wired up in various
configurations to achieve interesting behavior.

The canonical example is that employed by the
docker/distribution/notifications
package. Let's say we have a type httpSink where we'd like to queue
notifications. As a rule, it should send a single http request and return an
error if it fails:

func (h *httpSink) Write(event Event) error {
	p, err := json.Marshal(event)
	if err != nil {
		return err
	}
	body := bytes.NewReader(p)
	resp, err := h.client.Post(h.url, "application/json", body)
	if err != nil {
		return err
	}
	defer resp.Body.Close()
	
	if resp.Status != 200 {
		return errors.New("unexpected status")
	}

	return nil
}

// implement (*httpSink).Close()

With just that, we can start using components from this package. One can call
(*httpSink).Write to send events as the body of a post request to a
configured URL.

Retries Copy Section

HTTP can be unreliable. The first feature we'd like is to have some retry:

hs := newHTTPSink(/*...*/)
retry := NewRetryingSink(hs, NewBreaker(5, time.Second))

We now have a sink that will retry events against the httpSink until they
succeed. The retry will backoff for one second after 5 consecutive failures
using the breaker strategy.

Queues Copy Section

This isn't quite enough. We we want a sink that doesn't block while we are
waiting for events to be sent. Let's add a Queue:

queue := NewQueue(retry)

Now, we have an unbounded queue that will work through all events sent with
(*Queue).Write. Events can be added asynchronously to the queue without
blocking the current execution path. This is ideal for use in an http request.

Broadcast Copy Section

It usually turns out that you want to send to more than one listener. We can
use Broadcaster to support this:

var broadcast = NewBroadcaster() // make it available somewhere in your application.
broadcast.Add(queue) // add your queue!
broadcast.Add(queue2) // and another!

With the above, we can now call broadcast.Write in our http handlers and have
all the events distributed to each queue. Because the events are queued, not
listener blocks another.

Extending Copy Section

For the most part, the above is sufficient for a lot of applications. However,
extending the above functionality can be done implementing your own Sink. The
behavior and semantics of the sink can be completely dependent on the
application requirements. The interface is provided below for reference:

type Sink {
	Write(Event) error
	Close() error
}

Application behavior can be controlled by how Write behaves. The examples
above are designed to queue the message and return as quickly as possible.
Other implementations may block until the event is committed to durable
storage.

Copyright and license Copy Chapter

Copyright © 2016 Docker, Inc. go-events is licensed under the Apache License,
Version 2.0. See LICENSE for the full license text.

Overview (`vendor/github.com/docker/go-metrics/README.md`) Copy Section

go-metrics

This package is small wrapper around the prometheus go client to help enforce convention and best practices for metrics collection in Docker projects.

Best Practices Copy Chapter

This packages is meant to be used for collecting metrics. It is not meant to be
used as a replacement for the prometheus client but to help enforce consistent
naming across metrics collected. If you have not already read the prometheus
best practices around naming and labels you can read the naming section in the
prometheus documentation.

The following are a few conventions that help naming metrics in your project.

Namespace and Subsystem Copy Section

This package provides you with a namespace type that allows you to specify the
same namespace and subsystem for your metrics.

ns := metrics.NewNamespace("engine", "daemon", metrics.Labels{
        "version": daemon.Version,
        "commit":  daemon.GitCommit,
})

In the example above we are creating metrics for the daemon package.
engine would be the namespace in this example where daemon is the subsystem
or package where we are collecting the metrics.

A namespace also allows you to attach constant labels to the metrics such as
the git commit and version that it is collecting.

Declaring your Metrics Copy Section

Try to keep all your metric declarations in one file. This makes it easy for
others to see what constant labels are defined on the namespace and what
labels are defined on the metrics when they are created.

Use labels instead of multiple metrics Copy Section

Labels allow you to define one metric such as the time it takes to perform a
certain action on an object. If we wanted to collect timings on various container
actions such as create, start, and delete then we can define one metric called
container_actions and use labels to specify the type of action.

containerActions = ns.NewLabeledTimer("container_actions", "The number of milliseconds it takes to process each container action", "action")

The last parameter is the label name or key. When adding a data point to the
metric you can use the WithValues function to specify the action that you
are collecting for.

containerActions.WithValues("create").UpdateSince(start)

Always use a unit Copy Section

The metric name should describe what you are measuring but you also need to
provide the unit that it is being measured with. For a timer, the standard
unit is seconds and a counter's standard unit is a total. For gauges you must
provide the unit. This package provides a standard set of units for use within
your projects.

Nanoseconds Unit = "nanoseconds"
Seconds     Unit = "seconds"
Bytes       Unit = "bytes"
Total       Unit = "total"

If you need to use a unit but it is not defined in the package please open
a PR to add it but first try to see if one of the existing units work for
your metric, i.e., seconds or nanoseconds vs adding milliseconds.

HTTP Metrics Copy Chapter

To instrument a http handler, you can wrap the code like this:

namespace := metrics.NewNamespace("api_server", "http", metrics.Labels{"handler": "your_http_handler_name"})
httpMetrics := namespace.NewDefaultHttpMetrics()
metrics.Register(namespace)
instrumentedHandler = metrics.InstrumentHandler(httpMetrics, unInstrumentedHandler)

Note: The handler label must be provided when a new namespace is created.

Additional Metrics Copy Chapter

Additional metrics are also defined here that are not available in the prometheus client.
If you need a custom metrics and it is generic enough to be used by multiple projects, define it here.

Copyright and license Copy Chapter

Copyright © 2016 Docker, Inc. All rights reserved, except as follows. Code is released under the Apache 2.0 license.

Overview (`vendor/github.com/felixge/httpsnoop/README.md`) Copy Section

httpsnoop

Package httpsnoop provides an easy way to capture http related metrics (i.e.
response time, bytes written, and http status code) from your application's
http.Handlers.

Doing this requires non-trivial wrapping of the http.ResponseWriter interface,
which is also exposed for users interested in a more low-level API.

Usage Example Copy Chapter

// myH is your app's http handler, perhaps a http.ServeMux or similar.
var myH http.Handler
// wrappedH wraps myH in order to log every request.
wrappedH := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	m := httpsnoop.CaptureMetrics(myH, w, r)
	log.Printf(
		"%s %s (code=%d dt=%s written=%d)",
		r.Method,
		r.URL,
		m.Code,
		m.Duration,
		m.Written,
	)
})
http.ListenAndServe(":8080", wrappedH)

Why this package exists Copy Chapter

Instrumenting an application's http.Handler is surprisingly difficult.

However if you google for e.g. "capture ResponseWriter status code" you'll find
lots of advise and code examples that suggest it to be a fairly trivial
undertaking. Unfortunately everything I've seen so far has a high chance of
breaking your application.

The main problem is that a http.ResponseWriter often implements additional
interfaces such as http.Flusher, http.CloseNotifier, http.Hijacker, http.Pusher, and
io.ReaderFrom. So the naive approach of just wrapping http.ResponseWriter
in your own struct that also implements the http.ResponseWriter interface
will hide the additional interfaces mentioned above. This has a high change of
introducing subtle bugs into any non-trivial application.

Another approach I've seen people take is to return a struct that implements
all of the interfaces above. However, that's also problematic, because it's
difficult to fake some of these interfaces behaviors when the underlying
http.ResponseWriter doesn't have an implementation. It's also dangerous,
because an application may choose to operate differently, merely because it
detects the presence of these additional interfaces.

This package solves this problem by checking which additional interfaces a
http.ResponseWriter implements, returning a wrapped version implementing the
exact same set of interfaces.

Additionally this package properly handles edge cases such as WriteHeader not
being called, or called more than once, as well as concurrent calls to
http.ResponseWriter methods, and even calls happening after the wrapped
ServeHTTP has already returned.

Unfortunately this package is not perfect either. It's possible that it is
still missing some interfaces provided by the go core (let me know if you find
one), and it won't work for applications adding their own interfaces into the
mix. You can however use httpsnoop.Unwrap(w) to access the underlying
http.ResponseWriter and type-assert the result to its other interfaces.

However, hopefully the explanation above has sufficiently scared you of rolling
your own solution to this problem. httpsnoop may still break your application,
but at least it tries to avoid it as much as possible.

Anyway, the real problem here is that smuggling additional interfaces inside
http.ResponseWriter is a problematic design choice, but it probably goes as
deep as the Go language specification itself. But that's okay, I still prefer
Go over the alternatives ;).

Performance Copy Chapter

BenchmarkBaseline-8      	   20000	     94912 ns/op
BenchmarkCaptureMetrics-8	   20000	     95461 ns/op

As you can see, using CaptureMetrics on a vanilla http.Handler introduces an
overhead of ~500 ns per http request on my machine. However, the margin of
error appears to be larger than that, therefor it should be reasonable to
assume that the overhead introduced by CaptureMetrics is absolutely
negligible.

License Copy Chapter

MIT

Overview (`vendor/github.com/go-logr/logr/README.md`) Copy Section

A minimal logging API for Go

logr offers an(other) opinion on how Go programs and libraries can do logging
without becoming coupled to a particular logging implementation. This is not
an implementation of logging - it is an API. In fact it is two APIs with two
different sets of users.

The Logger type is intended for application and library authors. It provides
a relatively small API which can be used everywhere you want to emit logs. It
defers the actual act of writing logs (to files, to stdout, or whatever) to the
LogSink interface.

The LogSink interface is intended for logging library implementers. It is a
pure interface which can be implemented by logging frameworks to provide the actual logging
functionality.

This decoupling allows application and library developers to write code in
terms of logr.Logger (which has very low dependency fan-out) while the
implementation of logging is managed "up stack" (e.g. in or near main().)
Application developers can then switch out implementations as necessary.

Many people assert that libraries should not be logging, and as such efforts
like this are pointless. Those people are welcome to convince the authors of
the tens-of-thousands of libraries that DO write logs that they are all
wrong. In the meantime, logr takes a more practical approach.

Typical usage Copy Chapter

Somewhere, early in an application's life, it will make a decision about which
logging library (implementation) it actually wants to use. Something like:

func main() {
        // ... other setup code ...

        // Create the "root" logger.  We have chosen the "logimpl" implementation,
        // which takes some initial parameters and returns a logr.Logger.
        logger := logimpl.New(param1, param2)

        // ... other setup code ...

Most apps will call into other libraries, create structures to govern the flow,
etc. The logr.Logger object can be passed to these other libraries, stored
in structs, or even used as a package-global variable, if needed. For example:

app := createTheAppObject(logger)
    app.Run()

Outside of this early setup, no other packages need to know about the choice of
implementation. They write logs in terms of the logr.Logger that they
received:

type appObject struct {
        // ... other fields ...
        logger logr.Logger
        // ... other fields ...
    }

    func (app *appObject) Run() {
        app.logger.Info("starting up", "timestamp", time.Now())

        // ... app code ...

Background Copy Chapter

If the Go standard library had defined an interface for logging, this project
probably would not be needed. Alas, here we are.

When the Go developers started developing such an interface with
slog, they adopted some of the
logr design but also left out some parts and changed others:

The high-level slog API is explicitly meant to be one of many different APIs
that can be layered on top of a shared slog.Handler. logr is one such
alternative API, with interoperability provided by
some conversion functions.

Inspiration Copy Section

Before you consider this package, please read this blog post by the
inimitable Dave Cheney. We really appreciate what
he has to say, and it largely aligns with our own experiences.

Differences from Dave's ideas Copy Section

The main differences are:

1. Dave basically proposes doing away with the notion of a logging API in favor
   of fmt.Printf(). We disagree, especially when you consider things like output
   locations, timestamps, file and line decorations, and structured logging. This
   package restricts the logging API to just 2 types of logs: info and error.

Info logs are things you want to tell the user which are not errors. Error
logs are, well, errors. If your code receives an error from a subordinate
function call and is logging that error and not returning it, use error
logs.

2. Verbosity-levels on info logs. This gives developers a chance to indicate
   arbitrary grades of importance for info logs, without assigning names with
   semantic meaning such as "warning", "trace", and "debug." Superficially this
   may feel very similar, but the primary difference is the lack of semantics.
   Because verbosity is a numerical value, it's safe to assume that an app running
   with higher verbosity means more (and less important) logs will be generated.

Implementations (non-exhaustive) Copy Chapter

There are implementations for the following logging libraries:

• a function (can bridge to non-structured libraries): funcr
• a testing.T (for use in Go tests, with JSON-like output): testr
• github.com/google/glog: glogr
• k8s.io/klog (for Kubernetes): klogr
• a testing.T (with klog-like text output): ktesting
• go.uber.org/zap: zapr
• log (the Go standard library logger): stdr
• github.com/sirupsen/logrus: logrusr
• github.com/wojas/genericr: genericr (makes it easy to implement your own backend)
• logfmt (Heroku style logging): logfmtr
• github.com/rs/zerolog: zerologr
• github.com/go-kit/log: gokitlogr (also compatible with github.com/go-kit/kit/log since v0.12.0)
• bytes.Buffer (writing to a buffer): bufrlogr (useful for ensuring values were logged, like during testing)

slog interoperability Copy Chapter

Interoperability goes both ways, using the logr.Logger API with a slog.Handler
and using the slog.Logger API with a logr.LogSink. FromSlogHandler and
ToSlogHandler convert between a logr.Logger and a slog.Handler.
As usual, slog.New can be used to wrap such a slog.Handler in the high-level
slog API.

Using a `logr.LogSink` as backend for slog Copy Section

Ideally, a logr sink implementation should support both logr and slog by
implementing both the normal logr interface(s) and SlogSink. Because
of a conflict in the parameters of the common Enabled method, it is not
possible to implement both slog.Handler and logr.Sink in the same
type.

If both are supported, log calls can go from the high-level APIs to the backend
without the need to convert parameters. FromSlogHandler and ToSlogHandler can
convert back and forth without adding additional wrappers, with one exception:
when Logger.V was used to adjust the verbosity for a slog.Handler, then
ToSlogHandler has to use a wrapper which adjusts the verbosity for future
log calls.

Such an implementation should also support values that implement specific
interfaces from both packages for logging (logr.Marshaler, slog.LogValuer,
slog.GroupValue). logr does not convert those.

Not supporting slog has several drawbacks:

• Recording source code locations works correctly if the handler gets called
  through slog.Logger, but may be wrong in other cases. That's because a
  logr.Sink does its own stack unwinding instead of using the program counter
  provided by the high-level API.
• slog levels <= 0 can be mapped to logr levels by negating the level without a
  loss of information. But all slog levels > 0 (e.g. slog.LevelWarning as
  used by slog.Logger.Warn) must be mapped to 0 before calling the sink
  because logr does not support "more important than info" levels.
• The slog group concept is supported by prefixing each key in a key/value
  pair with the group names, separated by a dot. For structured output like
  JSON it would be better to group the key/value pairs inside an object.
• Special slog values and interfaces don't work as expected.
• The overhead is likely to be higher.

These drawbacks are severe enough that applications using a mixture of slog and
logr should switch to a different backend.

Using a `slog.Handler` as backend for logr Copy Section

Using a plain slog.Handler without support for logr works better than the
other direction:

• All logr verbosity levels can be mapped 1:1 to their corresponding slog level
  by negating them.
• Stack unwinding is done by the SlogSink and the resulting program
  counter is passed to the slog.Handler.
• Names added via Logger.WithName are gathered and recorded in an additional
  attribute with logger as key and the names separated by slash as value.
• Logger.Error is turned into a log record with slog.LevelError as level
  and an additional attribute with err as key, if an error was provided.

The main drawback is that logr.Marshaler will not be supported. Types should
ideally support both logr.Marshaler and slog.Valuer. If compatibility
with logr implementations without slog support is not important, then
slog.Valuer is sufficient.

Context support for slog Copy Section

Storing a logger in a context.Context is not supported by
slog. NewContextWithSlogLogger and FromContextAsSlogLogger can be
used to fill this gap. They store and retrieve a slog.Logger pointer
under the same context key that is also used by NewContext and
FromContext for logr.Logger value.

When NewContextWithSlogLogger is followed by FromContext, the latter will
automatically convert the slog.Logger to a
logr.Logger. FromContextAsSlogLogger does the same for the other direction.

With this approach, binaries which use either slog or logr are as efficient as
possible with no unnecessary allocations. This is also why the API stores a
slog.Logger pointer: when storing a slog.Handler, creating a slog.Logger
on retrieval would need to allocate one.

The downside is that switching back and forth needs more allocations. Because
logr is the API that is already in use by different packages, in particular
Kubernetes, the recommendation is to use the logr.Logger API in code which
uses contextual logging.

An alternative to adding values to a logger and storing that logger in the
context is to store the values in the context and to configure a logging
backend to extract those values when emitting log entries. This only works when
log calls are passed the context, which is not supported by the logr API.

With the slog API, it is possible, but not
required. https://github.com/veqryn/slog-context is a package for slog which
provides additional support code for this approach. It also contains wrappers
for the context functions in logr, so developers who prefer to not use the logr
APIs directly can use those instead and the resulting code will still be
interoperable with logr.

FAQ Copy Chapter

Conceptual Copy Section

Why structured logging?

• Structured logs are more easily queryable: Since you've got
  key-value pairs, it's much easier to query your structured logs for
  particular values by filtering on the contents of a particular key --
  think searching request logs for error codes, Kubernetes reconcilers for
  the name and namespace of the reconciled object, etc.

• Structured logging makes it easier to have cross-referenceable logs:
  Similarly to searchability, if you maintain conventions around your
  keys, it becomes easy to gather all log lines related to a particular
  concept.

• Structured logs allow better dimensions of filtering: if you have
  structure to your logs, you've got more precise control over how much
  information is logged -- you might choose in a particular configuration
  to log certain keys but not others, only log lines where a certain key
  matches a certain value, etc., instead of just having v-levels and names
  to key off of.

• Structured logs better represent structured data: sometimes, the
  data that you want to log is inherently structured (think tuple-link
  objects.) Structured logs allow you to preserve that structure when
  outputting.

Why V-levels?

V-levels give operators an easy way to control the chattiness of log
operations. V-levels provide a way for a given package to distinguish
the relative importance or verbosity of a given log message. Then, if
a particular logger or package is logging too many messages, the user
of the package can simply change the v-levels for that library.

Why not named levels, like Info/Warning/Error?

Read Dave Cheney's post. Then read Differences
from Dave's ideas.

Why not allow format strings, too?

Format strings negate many of the benefits of structured logs:

• They're not easily searchable without resorting to fuzzy searching,
  regular expressions, etc.

• They don't store structured data well, since contents are flattened into
  a string.

• They're not cross-referenceable.

• They don't compress easily, since the message is not constant.

(Unless you turn positional parameters into key-value pairs with numerical
keys, at which point you've gotten key-value logging with meaningless
keys.)

Practical Copy Section

Why key-value pairs, and not a map?

Key-value pairs are much easier to optimize, especially around
allocations. Zap (a structured logger that inspired logr's interface) has
performance measurements
that show this quite nicely.

While the interface ends up being a little less obvious, you get
potentially better performance, plus avoid making users type
map[string]string{} every time they want to log.

What if my V-levels differ between libraries?

That's fine. Control your V-levels on a per-logger basis, and use the
WithName method to pass different loggers to different libraries.

Generally, you should take care to ensure that you have relatively
consistent V-levels within a given logger, however, as this makes deciding
on what verbosity of logs to request easier.

But I really want to use a format string!

That's not actually a question. Assuming your question is "how do
I convert my mental model of logging with format strings to logging with
constant messages":

1. Figure out what the error actually is, as you'd write in a TL;DR style,
   and use that as a message.

2. For every place you'd write a format specifier, look to the word before
   it, and add that as a key value pair.

For instance, consider the following examples (all taken from spots in the
Kubernetes codebase):

• klog.V(4).Infof("Client is returning errors: code %v, error %v", responseCode, err) becomes logger.Error(err, "client returned an error", "code", responseCode)

• klog.V(4).Infof("Got a Retry-After %ds response for attempt %d to %v", seconds, retries, url) becomes logger.V(4).Info("got a retry-after response when requesting url", "attempt", retries, "after seconds", seconds, "url", url)

If you really must use a format string, use it in a key's value, and
call fmt.Sprintf yourself. For instance: log.Printf("unable to reflect over type %T") becomes logger.Info("unable to reflect over type", "type", fmt.Sprintf("%T")). In general though, the cases where
this is necessary should be few and far between.

How do I choose my V-levels?

This is basically the only hard constraint: increase V-levels to denote
more verbose or more debug-y logs.

Otherwise, you can start out with 0 as "you always want to see this",
1 as "common logging that you might possibly want to turn off", and
10 as "I would like to performance-test your log collection stack."

Then gradually choose levels in between as you need them, working your way
down from 10 (for debug and trace style logs) and up from 1 (for chattier
info-type logs). For reference, slog pre-defines -4 for debug logs
(corresponds to 4 in logr), which matches what is
recommended for Kubernetes.

How do I choose my keys?

Keys are fairly flexible, and can hold more or less any string
value. For best compatibility with implementations and consistency
with existing code in other projects, there are a few conventions you
should consider.

• Make your keys human-readable.
• Constant keys are generally a good idea.
• Be consistent across your codebase.
• Keys should naturally match parts of the message string.
• Use lower case for simple keys and
  lowerCamelCase for
  more complex ones. Kubernetes is one example of a project that has
  adopted that
  convention.

While key names are mostly unrestricted (and spaces are acceptable),
it's generally a good idea to stick to printable ascii characters, or at
least match the general character set of your log lines.

Why should keys be constant values?

The point of structured logging is to make later log processing easier. Your
keys are, effectively, the schema of each log message. If you use different
keys across instances of the same log line, you will make your structured logs
much harder to use. Sprintf() is for values, not for keys!

Why is this not a pure interface?

The Logger type is implemented as a struct in order to allow the Go compiler to
optimize things like high-V Info logs that are not triggered. Not all of
these implementations are implemented yet, but this structure was suggested as
a way to ensure they can be implemented. All of the real work is behind the
LogSink interface.

Overview (`vendor/github.com/go-logr/stdr/README.md`) Copy Section

Minimal Go logging using logr and Go's standard library

This package implements the logr interface
in terms of Go's standard log package(https://pkg.go.dev/log).

Overview (`vendor/github.com/google/s2a-go/README.md`) Copy Section

Secure Session Agent Client Libraries

The Secure Session Agent is a service that enables a workload to offload select
operations from the mTLS handshake and protects a workload's private key
material from exfiltration. Specifically, the workload asks the Secure Session
Agent for the TLS configuration to use during the handshake, to perform private
key operations, and to validate the peer certificate chain. The Secure Session
Agent's client libraries enable applications to communicate with the Secure
Session Agent during the TLS handshake, and to encrypt traffic to the peer
after the TLS handshake is complete.

This repository contains the source code for the Secure Session Agent's Go
client libraries, which allow gRPC and HTTP Go applications to use the Secure Session
Agent.

Overview (`vendor/github.com/google/uuid/README.md`) Copy Section

uuid

The uuid package generates and inspects UUIDs based on
RFC 4122
and DCE 1.1: Authentication and Security Services.

This package is based on the github.com/pborman/uuid package (previously named
code.google.com/p/go-uuid). It differs from these earlier packages in that
a UUID is a 16 byte array rather than a byte slice. One loss due to this
change is the ability to represent an invalid UUID (vs a NIL UUID).

Install

go get github.com/google/uuid

Documentation

Full go doc style documentation for the package can be viewed online without
installing this package by using the GoDoc site here:
http://pkg.go.dev/github.com/google/uuid

Overview (`vendor/github.com/gorilla/handlers/README.md`) Copy Section

gorilla/handlers

Package handlers is a collection of handlers (aka "HTTP middleware") for use
with Go's net/http package (or any framework supporting http.Handler), including:

• LoggingHandler for logging HTTP requests in the Apache Common Log
  Format.
• CombinedLoggingHandler for logging HTTP requests in the Apache Combined Log
  Format commonly used by
  both Apache and nginx.
• CompressHandler for gzipping responses.
• ContentTypeHandler for validating requests against a list of accepted
  content types.
• MethodHandler for matching HTTP methods against handlers in a
  map[string]http.Handler
• ProxyHeaders for populating r.RemoteAddr and r.URL.Scheme based on the
  X-Forwarded-For, X-Real-IP, X-Forwarded-Proto and RFC7239 Forwarded
  headers when running a Go server behind a HTTP reverse proxy.
• CanonicalHost for re-directing to the preferred host when handling multiple
  domains (i.e. multiple CNAME aliases).
• RecoveryHandler for recovering from unexpected panics.

Other handlers are documented on the Gorilla
website.

Example Copy Chapter

A simple example using handlers.LoggingHandler and handlers.CompressHandler:

import (
    "net/http"
    "github.com/gorilla/handlers"
)

func main() {
    r := http.NewServeMux()

    // Only log requests to our admin dashboard to stdout
    r.Handle("/admin", handlers.LoggingHandler(os.Stdout, http.HandlerFunc(ShowAdminDashboard)))
    r.HandleFunc("/", ShowIndex)

    // Wrap our server with our gzip handler to gzip compress all responses.
    http.ListenAndServe(":8000", handlers.CompressHandler(r))
}

License Copy Chapter

BSD licensed. See the included LICENSE file for details.

Overview (`vendor/github.com/gorilla/mux/README.md`) Copy Section

gorilla/mux

Package gorilla/mux implements a request router and dispatcher for matching incoming requests to
their respective handler.

The name mux stands for "HTTP request multiplexer". Like the standard http.ServeMux, mux.Router matches incoming requests against a list of registered routes and calls a handler for the route that matches the URL or other conditions. The main features are:

• It implements the http.Handler interface so it is compatible with the standard http.ServeMux.
• Requests can be matched based on URL host, path, path prefix, schemes, header and query values, HTTP methods or using custom matchers.
• URL hosts, paths and query values can have variables with an optional regular expression.
• Registered URLs can be built, or "reversed", which helps maintaining references to resources.
• Routes can be used as subrouters: nested routes are only tested if the parent route matches. This is useful to define groups of routes that share common conditions like a host, a path prefix or other repeated attributes. As a bonus, this optimizes request matching.

• Install
• Examples
• Matching Routes
• Static Files
• Serving Single Page Applications (e.g. React, Vue, Ember.js, etc.)
• Registered URLs
• Walking Routes
• Graceful Shutdown
• Middleware
• Handling CORS Requests
• Testing Handlers
• Full Example

Install Copy Chapter

With a correctly configured Go toolchain:

go get -u github.com/gorilla/mux

Examples Copy Chapter

Let's start registering a couple of URL paths and handlers:

func main() {
    r := mux.NewRouter()
    r.HandleFunc("/", HomeHandler)
    r.HandleFunc("/products", ProductsHandler)
    r.HandleFunc("/articles", ArticlesHandler)
    http.Handle("/", r)
}

Here we register three routes mapping URL paths to handlers. This is equivalent to how http.HandleFunc() works: if an incoming request URL matches one of the paths, the corresponding handler is called passing (http.ResponseWriter, *http.Request) as parameters.

Paths can have variables. They are defined using the format {name} or {name:pattern}. If a regular expression pattern is not defined, the matched variable will be anything until the next slash. For example:

r := mux.NewRouter()
r.HandleFunc("/products/{key}", ProductHandler)
r.HandleFunc("/articles/{category}/", ArticlesCategoryHandler)
r.HandleFunc("/articles/{category}/{id:[0-9]+}", ArticleHandler)

The names are used to create a map of route variables which can be retrieved calling mux.Vars():

func ArticlesCategoryHandler(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    w.WriteHeader(http.StatusOK)
    fmt.Fprintf(w, "Category: %v\n", vars["category"])
}

And this is all you need to know about the basic usage. More advanced options are explained below.

Matching Routes Copy Section

Routes can also be restricted to a domain or subdomain. Just define a host pattern to be matched. They can also have variables:

r := mux.NewRouter()
// Only matches if domain is "www.example.com".
r.Host("www.example.com")
// Matches a dynamic subdomain.
r.Host("{subdomain:[a-z]+}.example.com")

There are several other matchers that can be added. To match path prefixes:

r.PathPrefix("/products/")

...or HTTP methods:

r.Methods("GET", "POST")

...or URL schemes:

r.Schemes("https")

...or header values:

r.Headers("X-Requested-With", "XMLHttpRequest")

...or query values:

r.Queries("key", "value")

...or to use a custom matcher function:

r.MatcherFunc(func(r *http.Request, rm *RouteMatch) bool {
    return r.ProtoMajor == 0
})

...and finally, it is possible to combine several matchers in a single route:

r.HandleFunc("/products", ProductsHandler).
  Host("www.example.com").
  Methods("GET").
  Schemes("http")

Routes are tested in the order they were added to the router. If two routes match, the first one wins:

r := mux.NewRouter()
r.HandleFunc("/specific", specificHandler)
r.PathPrefix("/").Handler(catchAllHandler)

Setting the same matching conditions again and again can be boring, so we have a way to group several routes that share the same requirements. We call it "subrouting".

For example, let's say we have several URLs that should only match when the host is www.example.com. Create a route for that host and get a "subrouter" from it:

r := mux.NewRouter()
s := r.Host("www.example.com").Subrouter()

Then register routes in the subrouter:

s.HandleFunc("/products/", ProductsHandler)
s.HandleFunc("/products/{key}", ProductHandler)
s.HandleFunc("/articles/{category}/{id:[0-9]+}", ArticleHandler)

The three URL paths we registered above will only be tested if the domain is www.example.com, because the subrouter is tested first. This is not only convenient, but also optimizes request matching. You can create subrouters combining any attribute matchers accepted by a route.

Subrouters can be used to create domain or path "namespaces": you define subrouters in a central place and then parts of the app can register its paths relatively to a given subrouter.

There's one more thing about subroutes. When a subrouter has a path prefix, the inner routes use it as base for their paths:

r := mux.NewRouter()
s := r.PathPrefix("/products").Subrouter()
// "/products/"
s.HandleFunc("/", ProductsHandler)
// "/products/{key}/"
s.HandleFunc("/{key}/", ProductHandler)
// "/products/{key}/details"
s.HandleFunc("/{key}/details", ProductDetailsHandler)

Static Files Copy Section

Note that the path provided to PathPrefix() represents a "wildcard": calling
PathPrefix("/static/").Handler(...) means that the handler will be passed any
request that matches "/static/*". This makes it easy to serve static files with mux:

func main() {
    var dir string

    flag.StringVar(&dir, "dir", ".", "the directory to serve files from. Defaults to the current dir")
    flag.Parse()
    r := mux.NewRouter()

    // This will serve files under http://localhost:8000/static/<filename>
    r.PathPrefix("/static/").Handler(http.StripPrefix("/static/", http.FileServer(http.Dir(dir))))

    srv := &http.Server{
        Handler:      r,
        Addr:         "127.0.0.1:8000",
        // Good practice: enforce timeouts for servers you create!
        WriteTimeout: 15 * time.Second,
        ReadTimeout:  15 * time.Second,
    }

    log.Fatal(srv.ListenAndServe())
}

Serving Single Page Applications Copy Section

Most of the time it makes sense to serve your SPA on a separate web server from your API,
but sometimes it's desirable to serve them both from one place. It's possible to write a simple
handler for serving your SPA (for use with React Router's BrowserRouter for example), and leverage
mux's powerful routing for your API endpoints.

/* Detailed source-code truncated for AI context efficiency. */

Registered URLs Copy Section

Now let's see how to build registered URLs.

Routes can be named. All routes that define a name can have their URLs built, or "reversed". We define a name calling Name() on a route. For example:

r := mux.NewRouter()
r.HandleFunc("/articles/{category}/{id:[0-9]+}", ArticleHandler).
  Name("article")

To build a URL, get the route and call the URL() method, passing a sequence of key/value pairs for the route variables. For the previous route, we would do:

url, err := r.Get("article").URL("category", "technology", "id", "42")

...and the result will be a url.URL with the following path:

"/articles/technology/42"

This also works for host and query value variables:

r := mux.NewRouter()
r.Host("{subdomain}.example.com").
  Path("/articles/{category}/{id:[0-9]+}").
  Queries("filter", "{filter}").
  HandlerFunc(ArticleHandler).
  Name("article")

// url.String() will be "http://news.example.com/articles/technology/42?filter=gorilla"
url, err := r.Get("article").URL("subdomain", "news",
                                 "category", "technology",
                                 "id", "42",
                                 "filter", "gorilla")

All variables defined in the route are required, and their values must conform to the corresponding patterns. These requirements guarantee that a generated URL will always match a registered route -- the only exception is for explicitly defined "build-only" routes which never match.

Regex support also exists for matching Headers within a route. For example, we could do:

r.HeadersRegexp("Content-Type", "application/(text|json)")

...and the route will match both requests with a Content-Type of application/json as well as application/text

There's also a way to build only the URL host or path for a route: use the methods URLHost() or URLPath() instead. For the previous route, we would do:

// "http://news.example.com/"
host, err := r.Get("article").URLHost("subdomain", "news")

// "/articles/technology/42"
path, err := r.Get("article").URLPath("category", "technology", "id", "42")

And if you use subrouters, host and path defined separately can be built as well:

r := mux.NewRouter()
s := r.Host("{subdomain}.example.com").Subrouter()
s.Path("/articles/{category}/{id:[0-9]+}").
  HandlerFunc(ArticleHandler).
  Name("article")

// "http://news.example.com/articles/technology/42"
url, err := r.Get("article").URL("subdomain", "news",
                                 "category", "technology",
                                 "id", "42")

To find all the required variables for a given route when calling URL(), the method GetVarNames() is available:

r := mux.NewRouter()
r.Host("{domain}").
    Path("/{group}/{item_id}").
    Queries("some_data1", "{some_data1}").
    Queries("some_data2", "{some_data2}").
    Name("article")

// Will print [domain group item_id some_data1 some_data2] <nil>
fmt.Println(r.Get("article").GetVarNames())

Walking Routes Copy Section

The Walk function on mux.Router can be used to visit all of the routes that are registered on a router. For example,
the following prints all of the registered routes:

package main

import (
	"fmt"
	"net/http"
	"strings"

	"github.com/gorilla/mux"
)

func handler(w http.ResponseWriter, r *http.Request) {
	return
}

func main() {
	r := mux.NewRouter()
	r.HandleFunc("/", handler)
	r.HandleFunc("/products", handler).Methods("POST")
	r.HandleFunc("/articles", handler).Methods("GET")
	r.HandleFunc("/articles/{id}", handler).Methods("GET", "PUT")
	r.HandleFunc("/authors", handler).Queries("surname", "{surname}")
	err := r.Walk(func(route *mux.Route, router *mux.Router, ancestors []*mux.Route) error {
		pathTemplate, err := route.GetPathTemplate()
		if err == nil {
			fmt.Println("ROUTE:", pathTemplate)
		}
		pathRegexp, err := route.GetPathRegexp()
		if err == nil {
			fmt.Println("Path regexp:", pathRegexp)
		}
		queriesTemplates, err := route.GetQueriesTemplates()
		if err == nil {
			fmt.Println("Queries templates:", strings.Join(queriesTemplates, ","))
		}
		queriesRegexps, err := route.GetQueriesRegexp()
		if err == nil {
			fmt.Println("Queries regexps:", strings.Join(queriesRegexps, ","))
		}
		methods, err := route.GetMethods()
		if err == nil {
			fmt.Println("Methods:", strings.Join(methods, ","))
		}
		fmt.Println()
		return nil
	})

	if err != nil {
		fmt.Println(err)
	}

	http.Handle("/", r)
}

Graceful Shutdown Copy Section

Go 1.8 introduced the ability to gracefully shutdown a *http.Server. Here's how to do that alongside mux:

/* Detailed source-code truncated for AI context efficiency. */

Middleware Copy Section

Mux supports the addition of middlewares to a Router, which are executed in the order they are added if a match is found, including its subrouters.
Middlewares are (typically) small pieces of code which take one request, do something with it, and pass it down to another middleware or the final handler. Some common use cases for middleware are request logging, header manipulation, or ResponseWriter hijacking.

Mux middlewares are defined using the de facto standard type:

type MiddlewareFunc func(http.Handler) http.Handler

Typically, the returned handler is a closure which does something with the http.ResponseWriter and http.Request passed to it, and then calls the handler passed as parameter to the MiddlewareFunc. This takes advantage of closures being able access variables from the context where they are created, while retaining the signature enforced by the receivers.

A very basic middleware which logs the URI of the request being handled could be written as:

func loggingMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Do stuff here
        log.Println(r.RequestURI)
        // Call the next handler, which can be another middleware in the chain, or the final handler.
        next.ServeHTTP(w, r)
    })
}

Middlewares can be added to a router using Router.Use():

r := mux.NewRouter()
r.HandleFunc("/", handler)
r.Use(loggingMiddleware)

A more complex authentication middleware, which maps session token to users, could be written as:

// Define our struct
type authenticationMiddleware struct {
	tokenUsers map[string]string
}

// Initialize it somewhere
func (amw *authenticationMiddleware) Populate() {
	amw.tokenUsers["00000000"] = "user0"
	amw.tokenUsers["aaaaaaaa"] = "userA"
	amw.tokenUsers["05f717e5"] = "randomUser"
	amw.tokenUsers["deadbeef"] = "user0"
}

// Middleware function, which will be called for each request
func (amw *authenticationMiddleware) Middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        token := r.Header.Get("X-Session-Token")

        if user, found := amw.tokenUsers[token]; found {
        	// We found the token in our map
        	log.Printf("Authenticated user %s\n", user)
        	// Pass down the request to the next middleware (or final handler)
        	next.ServeHTTP(w, r)
        } else {
        	// Write an error and stop the handler chain
        	http.Error(w, "Forbidden", http.StatusForbidden)
        }
    })
}

r := mux.NewRouter()
r.HandleFunc("/", handler)

amw := authenticationMiddleware{tokenUsers: make(map[string]string)}
amw.Populate()

r.Use(amw.Middleware)

Note: The handler chain will be stopped if your middleware doesn't call next.ServeHTTP() with the corresponding parameters. This can be used to abort a request if the middleware writer wants to. Middlewares should write to ResponseWriter if they are going to terminate the request, and they should not write to ResponseWriter if they are not going to terminate it.

Handling CORS Requests Copy Section

CORSMethodMiddleware intends to make it easier to strictly set the Access-Control-Allow-Methods response header.

• You will still need to use your own CORS handler to set the other CORS headers such as Access-Control-Allow-Origin
• The middleware will set the Access-Control-Allow-Methods header to all the method matchers (e.g. r.Methods(http.MethodGet, http.MethodPut, http.MethodOptions) -> Access-Control-Allow-Methods: GET,PUT,OPTIONS) on a route
• If you do not specify any methods, then:

Important: there must be an OPTIONS method matcher for the middleware to set the headers.

Here is an example of using CORSMethodMiddleware along with a custom OPTIONS handler to set all the required CORS headers:

package main

import (
	"net/http"
	"github.com/gorilla/mux"
)

func main() {
    r := mux.NewRouter()

    // IMPORTANT: you must specify an OPTIONS method matcher for the middleware to set CORS headers
    r.HandleFunc("/foo", fooHandler).Methods(http.MethodGet, http.MethodPut, http.MethodPatch, http.MethodOptions)
    r.Use(mux.CORSMethodMiddleware(r))
    
    http.ListenAndServe(":8080", r)
}

func fooHandler(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Access-Control-Allow-Origin", "*")
    if r.Method == http.MethodOptions {
        return
    }

    w.Write([]byte("foo"))
}

And an request to /foo using something like:

curl localhost:8080/foo -v

Would look like:

*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /foo HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.59.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Access-Control-Allow-Methods: GET,PUT,PATCH,OPTIONS
< Access-Control-Allow-Origin: *
< Date: Fri, 28 Jun 2019 20:13:30 GMT
< Content-Length: 3
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host localhost left intact
foo

Testing Handlers Copy Section

Testing handlers in a Go web application is straightforward, and mux doesn't complicate this any further. Given two files: endpoints.go and endpoints_test.go, here's how we'd test an application using mux.

First, our simple HTTP handler:

// endpoints.go
package main

func HealthCheckHandler(w http.ResponseWriter, r *http.Request) {
    // A very simple health check.
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusOK)

    // In the future we could report back on the status of our DB, or our cache
    // (e.g. Redis) by performing a simple PING, and include them in the response.
    io.WriteString(w, `{"alive": true}`)
}

func main() {
    r := mux.NewRouter()
    r.HandleFunc("/health", HealthCheckHandler)

    log.Fatal(http.ListenAndServe("localhost:8080", r))
}

Our test code:

// endpoints_test.go
package main

import (
    "net/http"
    "net/http/httptest"
    "testing"
)

func TestHealthCheckHandler(t *testing.T) {
    // Create a request to pass to our handler. We don't have any query parameters for now, so we'll
    // pass 'nil' as the third parameter.
    req, err := http.NewRequest("GET", "/health", nil)
    if err != nil {
        t.Fatal(err)
    }

    // We create a ResponseRecorder (which satisfies http.ResponseWriter) to record the response.
    rr := httptest.NewRecorder()
    handler := http.HandlerFunc(HealthCheckHandler)

    // Our handlers satisfy http.Handler, so we can call their ServeHTTP method
    // directly and pass in our Request and ResponseRecorder.
    handler.ServeHTTP(rr, req)

    // Check the status code is what we expect.
    if status := rr.Code; status != http.StatusOK {
        t.Errorf("handler returned wrong status code: got %v want %v",
            status, http.StatusOK)
    }

    // Check the response body is what we expect.
    expected := `{"alive": true}`
    if rr.Body.String() != expected {
        t.Errorf("handler returned unexpected body: got %v want %v",
            rr.Body.String(), expected)
    }
}

In the case that our routes have variables, we can pass those in the request. We could write
table-driven tests to test multiple
possible route variables as needed.

// endpoints.go
func main() {
    r := mux.NewRouter()
    // A route with a route variable:
    r.HandleFunc("/metrics/{type}", MetricsHandler)

    log.Fatal(http.ListenAndServe("localhost:8080", r))
}

Our test file, with a table-driven test of routeVariables:

// endpoints_test.go
func TestMetricsHandler(t *testing.T) {
    tt := []struct{
        routeVariable string
        shouldPass bool
    }{
        {"goroutines", true},
        {"heap", true},
        {"counters", true},
        {"queries", true},
        {"adhadaeqm3k", false},
    }

    for _, tc := range tt {
        path := fmt.Sprintf("/metrics/%s", tc.routeVariable)
        req, err := http.NewRequest("GET", path, nil)
        if err != nil {
            t.Fatal(err)
        }

        rr := httptest.NewRecorder()
	
	// To add the vars to the context, 
	// we need to create a router through which we can pass the request.
	router := mux.NewRouter()
        router.HandleFunc("/metrics/{type}", MetricsHandler)
        router.ServeHTTP(rr, req)

        // In this case, our MetricsHandler returns a non-200 response
        // for a route variable it doesn't know about.
        if rr.Code == http.StatusOK && !tc.shouldPass {
            t.Errorf("handler should have failed on routeVariable %s: got %v want %v",
                tc.routeVariable, rr.Code, http.StatusOK)
        }
    }
}

Full Example Copy Chapter

Here's a complete, runnable example of a small mux based server:

package main

import (
    "net/http"
    "log"
    "github.com/gorilla/mux"
)

func YourHandler(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("Gorilla!\n"))
}

func main() {
    r := mux.NewRouter()
    // Routes consist of a path and a handler function.
    r.HandleFunc("/", YourHandler)

    // Bind to a port and pass our router in
    log.Fatal(http.ListenAndServe(":8000", r))
}

License Copy Chapter

BSD licensed. See the LICENSE file for details.

Overview (`vendor/github.com/inconshreveable/mousetrap/README.md`) Copy Section

mousetrap

mousetrap is a tiny library that answers a single question.

On a Windows machine, was the process invoked by someone double clicking on
the executable file while browsing in explorer?

Motivation Copy Section

Windows developers unfamiliar with command line tools will often "double-click"
the executable for a tool. Because most CLI tools print the help and then exit
when invoked without arguments, this is often very frustrating for those users.

mousetrap provides a way to detect these invocations so that you can provide
more helpful behavior and instructions on how to run the CLI tool. To see what
this looks like, both from an organizational and a technical perspective, see
https://inconshreveable.com/09-09-2014/sweat-the-small-stuff/

The interface Copy Section

The library exposes a single interface:

func StartedByExplorer() (bool)

Overview (`vendor/github.com/jmespath/go-jmespath/README.md`) Copy Section

go-jmespath - A JMESPath implementation in Go

go-jmespath is a GO implementation of JMESPath,
which is a query language for JSON. It will take a JSON
document and transform it into another JSON document
through a JMESPath expression.

Using go-jmespath is really easy. There's a single function
you use, jmespath.search:

> import "github.com/jmespath/go-jmespath"
>
> var jsondata = []byte(`{"foo": {"bar": {"baz": [0, 1, 2, 3, 4]}}}`) // your data
> var data interface{}
> err := json.Unmarshal(jsondata, &data)
> result, err := jmespath.Search("foo.bar.baz[2]", data)
result = 2

In the example we gave the search function input data of
{"foo": {"bar": {"baz": [0, 1, 2, 3, 4]}}} as well as the JMESPath
expression foo.bar.baz[2], and the search function evaluated
the expression against the input data to produce the result 2.

The JMESPath language can do a lot more than select an element
from a list. Here are a few more examples:

> var jsondata = []byte(`{"foo": {"bar": {"baz": [0, 1, 2, 3, 4]}}}`) // your data
> var data interface{}
> err := json.Unmarshal(jsondata, &data)
> result, err := jmespath.search("foo.bar", data)
result = { "baz": [ 0, 1, 2, 3, 4 ] }


> var jsondata  = []byte(`{"foo": [{"first": "a", "last": "b"},
                           {"first": "c", "last": "d"}]}`) // your data
> var data interface{}
> err := json.Unmarshal(jsondata, &data)
> result, err := jmespath.search({"foo[*].first", data)
result [ 'a', 'c' ]


> var jsondata = []byte(`{"foo": [{"age": 20}, {"age": 25},
                           {"age": 30}, {"age": 35},
                           {"age": 40}]}`) // your data
> var data interface{}
> err := json.Unmarshal(jsondata, &data)
> result, err := jmespath.search("foo[?age > `30`]")
result = [ { age: 35 }, { age: 40 } ]

You can also pre-compile your query. This is usefull if
you are going to run multiple searches with it:

> var jsondata = []byte(`{"foo": "bar"}`)
	> var data interface{}
    > err := json.Unmarshal(jsondata, &data)
	> precompiled, err := Compile("foo")
	> if err != nil{
    >   // ... handle the error
    > }
    > result, err := precompiled.Search(data)
	result = "bar"

More Resources Copy Chapter

The example above only show a small amount of what
a JMESPath expression can do. If you want to take a
tour of the language, the best place to go is the
JMESPath Tutorial.

One of the best things about JMESPath is that it is
implemented in many different programming languages including
python, ruby, php, lua, etc. To see a complete list of libraries,
check out the JMESPath libraries page.

And finally, the full JMESPath specification can be found
on the JMESPath site.

Overview (`vendor/github.com/klauspost/compress/README.md`) Copy Section

compress

This package provides various compression algorithms.

• zstandard compression and decompression in pure Go.
• S2 is a high performance replacement for Snappy.
• Optimized deflate packages which can be used as a dropin replacement for gzip, zip and zlib.
• snappy is a drop-in replacement for github.com/golang/snappy offering better compression and concurrent streams.
• huff0 and FSE implementations for raw entropy encoding.
• gzhttp Provides client and server wrappers for handling gzipped/zstd HTTP requests efficiently.
• pgzip is a separate package that provides a very fast parallel gzip implementation.

package usage

Use go get github.com/klauspost/compress@latest to add it to your project.

This package will support the current Go version and 2 versions back.

• Use the nounsafe tag to disable all use of the "unsafe" package.
• Use the noasm tag to disable all assembly across packages.

Use the links above for more information on each.

changelog

• Jan 16th, 2026 1.18.3

  • Downstream CVE-2025-61728. See golang/go#77102.

• Dec 1st, 2025 - 1.18.2

  • flate: Fix invalid encoding on level 9 with single value input in https://github.com/klauspost/compress/pull/1115
  • flate: reduce stateless allocations by @RXamzin in https://github.com/klauspost/compress/pull/1106

• Oct 20, 2025 - 1.18.1 - RETRACTED

  • zstd: Add simple zstd EncodeTo/DecodeTo functions https://github.com/klauspost/compress/pull/1079
  • zstd: Fix incorrect buffer size in dictionary encodes https://github.com/klauspost/compress/pull/1059
  • s2: check for cap, not len of buffer in EncodeBetter/Best by @vdarulis in https://github.com/klauspost/compress/pull/1080
  • zlib: Avoiding extra allocation in zlib.reader.Reset by @travelpolicy in https://github.com/klauspost/compress/pull/1086
  • gzhttp: remove redundant err check in zstdReader by @ryanfowler in https://github.com/klauspost/compress/pull/1090
  • flate: Faster load+store https://github.com/klauspost/compress/pull/1104
  • flate: Simplify matchlen https://github.com/klauspost/compress/pull/1101
  • flate: Use exact sizes for huffman tables https://github.com/klauspost/compress/pull/1103

• Feb 19th, 2025 - 1.18.0

  • Add unsafe little endian loaders https://github.com/klauspost/compress/pull/1036

  • fix: check r.err != nil but return a nil value error err by @alingse in https://github.com/klauspost/compress/pull/1028

  • flate: Simplify L4-6 loading https://github.com/klauspost/compress/pull/1043

  • flate: Simplify matchlen (remove asm) https://github.com/klauspost/compress/pull/1045

  • s2: Improve small block compression speed w/o asm https://github.com/klauspost/compress/pull/1048

  • flate: Fix matchlen L5+L6 https://github.com/klauspost/compress/pull/1049

  • flate: Cleanup & reduce casts https://github.com/klauspost/compress/pull/1050

    See changes to v1.17.x

• Oct 11th, 2024 - 1.17.11

  • zstd: Fix extra CRC written with multiple Close calls https://github.com/klauspost/compress/pull/1017
  • s2: Don't use stack for index tables https://github.com/klauspost/compress/pull/1014
  • gzhttp: No content-type on no body response code by @juliens in https://github.com/klauspost/compress/pull/1011
  • gzhttp: Do not set the content-type when response has no body by @kevinpollet in https://github.com/klauspost/compress/pull/1013

• Sep 23rd, 2024 - 1.17.10

  • gzhttp: Add TransportAlwaysDecompress option. https://github.com/klauspost/compress/pull/978
  • gzhttp: Add supported decompress request body by @mirecl in https://github.com/klauspost/compress/pull/1002
  • s2: Add EncodeBuffer buffer recycling callback https://github.com/klauspost/compress/pull/982
  • zstd: Improve memory usage on small streaming encodes https://github.com/klauspost/compress/pull/1007
  • flate: read data written with partial flush by @vajexal in https://github.com/klauspost/compress/pull/996

• Jun 12th, 2024 - 1.17.9

  • s2: Reduce ReadFrom temporary allocations https://github.com/klauspost/compress/pull/949
  • flate, zstd: Shave some bytes off amd64 matchLen by @greatroar in https://github.com/klauspost/compress/pull/963
  • Upgrade zip/zlib to 1.22.4 upstream https://github.com/klauspost/compress/pull/970 https://github.com/klauspost/compress/pull/971
  • zstd: BuildDict fails with RLE table https://github.com/klauspost/compress/pull/951

• Apr 9th, 2024 - 1.17.8

  • zstd: Reject blocks where reserved values are not 0 https://github.com/klauspost/compress/pull/885
  • zstd: Add RLE detection+encoding https://github.com/klauspost/compress/pull/938

• Feb 21st, 2024 - 1.17.7

  • s2: Add AsyncFlush method: Complete the block without flushing by @Jille in https://github.com/klauspost/compress/pull/927
  • s2: Fix literal+repeat exceeds dst crash https://github.com/klauspost/compress/pull/930

• Feb 5th, 2024 - 1.17.6

  • zstd: Fix incorrect repeat coding in best mode https://github.com/klauspost/compress/pull/923
  • s2: Fix DecodeConcurrent deadlock on errors https://github.com/klauspost/compress/pull/925

• Jan 26th, 2024 - v1.17.5

  • flate: Fix reset with dictionary on custom window encodes https://github.com/klauspost/compress/pull/912
  • zstd: Add Frame header encoding and stripping https://github.com/klauspost/compress/pull/908
  • zstd: Limit better/best default window to 8MB https://github.com/klauspost/compress/pull/913
  • zstd: Speed improvements by @greatroar in https://github.com/klauspost/compress/pull/896 https://github.com/klauspost/compress/pull/910
  • s2: Fix callbacks for skippable blocks and disallow 0xfe (Padding) by @Jille in https://github.com/klauspost/compress/pull/916 https://github.com/klauspost/compress/pull/917
    https://github.com/klauspost/compress/pull/919 https://github.com/klauspost/compress/pull/918

• Dec 1st, 2023 - v1.17.4

  • huff0: Speed up symbol counting by @greatroar in https://github.com/klauspost/compress/pull/887
  • huff0: Remove byteReader by @greatroar in https://github.com/klauspost/compress/pull/886
  • gzhttp: Allow overriding decompression on transport https://github.com/klauspost/compress/pull/892
  • gzhttp: Clamp compression level https://github.com/klauspost/compress/pull/890
  • gzip: Error out if reserved bits are set https://github.com/klauspost/compress/pull/891

• Nov 15th, 2023 - v1.17.3

  • fse: Fix max header size https://github.com/klauspost/compress/pull/881
  • zstd: Improve better/best compression https://github.com/klauspost/compress/pull/877
  • gzhttp: Fix missing content type on Close https://github.com/klauspost/compress/pull/883

• Oct 22nd, 2023 - v1.17.2

  • zstd: Fix rare CORRUPTION output in "best" mode. See https://github.com/klauspost/compress/pull/876

• Oct 14th, 2023 - v1.17.1

  • s2: Fix S2 "best" dictionary wrong encoding https://github.com/klauspost/compress/pull/871
  • flate: Reduce allocations in decompressor and minor code improvements by @fakefloordiv in https://github.com/klauspost/compress/pull/869
  • s2: Fix EstimateBlockSize on 6&7 length input https://github.com/klauspost/compress/pull/867

• Sept 19th, 2023 - v1.17.0

  • Add experimental dictionary builder https://github.com/klauspost/compress/pull/853
  • Add xerial snappy read/writer https://github.com/klauspost/compress/pull/838
  • flate: Add limited window compression https://github.com/klauspost/compress/pull/843
  • s2: Do 2 overlapping match checks https://github.com/klauspost/compress/pull/839
  • flate: Add amd64 assembly matchlen https://github.com/klauspost/compress/pull/837
  • gzip: Copy bufio.Reader on Reset by @thatguystone in https://github.com/klauspost/compress/pull/860

  See changes to v1.16.x

• July 1st, 2023 - v1.16.7

  • zstd: Fix default level first dictionary encode https://github.com/klauspost/compress/pull/829
  • s2: add GetBufferCapacity() method by @GiedriusS in https://github.com/klauspost/compress/pull/832

• June 13, 2023 - v1.16.6

  • zstd: correctly ignore WithEncoderPadding(1) by @ianlancetaylor in https://github.com/klauspost/compress/pull/806
  • zstd: Add amd64 match length assembly https://github.com/klauspost/compress/pull/824
  • gzhttp: Handle informational headers by @rtribotte in https://github.com/klauspost/compress/pull/815
  • s2: Improve Better compression slightly https://github.com/klauspost/compress/pull/663

• Apr 16, 2023 - v1.16.5

  • zstd: readByte needs to use io.ReadFull by @jnoxon in https://github.com/klauspost/compress/pull/802
  • gzip: Fix WriterTo after initial read https://github.com/klauspost/compress/pull/804

• Apr 5, 2023 - v1.16.4

  • zstd: Improve zstd best efficiency by @greatroar and @klauspost in https://github.com/klauspost/compress/pull/784
  • zstd: Respect WithAllLitEntropyCompression https://github.com/klauspost/compress/pull/792
  • zstd: Fix amd64 not always detecting corrupt data https://github.com/klauspost/compress/pull/785
  • zstd: Various minor improvements by @greatroar in https://github.com/klauspost/compress/pull/788 https://github.com/klauspost/compress/pull/794 https://github.com/klauspost/compress/pull/795
  • s2: Fix huge block overflow https://github.com/klauspost/compress/pull/779
  • s2: Allow CustomEncoder fallback https://github.com/klauspost/compress/pull/780
  • gzhttp: Support ResponseWriter Unwrap() in gzhttp handler by @jgimenez in https://github.com/klauspost/compress/pull/799

• Mar 13, 2023 - v1.16.1

  • zstd: Speed up + improve best encoder by @greatroar in https://github.com/klauspost/compress/pull/776
  • gzhttp: Add optional BREACH mitigation. https://github.com/klauspost/compress/pull/762 https://github.com/klauspost/compress/pull/768 https://github.com/klauspost/compress/pull/769 https://github.com/klauspost/compress/pull/770 https://github.com/klauspost/compress/pull/767
  • s2: Add Intel LZ4s converter https://github.com/klauspost/compress/pull/766
  • zstd: Minor bug fixes https://github.com/klauspost/compress/pull/771 https://github.com/klauspost/compress/pull/772 https://github.com/klauspost/compress/pull/773
  • huff0: Speed up compress1xDo by @greatroar in https://github.com/klauspost/compress/pull/774

• Feb 26, 2023 - v1.16.0

  • s2: Add Dictionary support. https://github.com/klauspost/compress/pull/685
  • s2: Add Compression Size Estimate. https://github.com/klauspost/compress/pull/752
  • s2: Add support for custom stream encoder. https://github.com/klauspost/compress/pull/755
  • s2: Add LZ4 block converter. https://github.com/klauspost/compress/pull/748
  • s2: Support io.ReaderAt in ReadSeeker. https://github.com/klauspost/compress/pull/747
  • s2c/s2sx: Use concurrent decoding. https://github.com/klauspost/compress/pull/746

  See changes to v1.15.x

• Jan 21st, 2023 (v1.15.15)

  • deflate: Improve level 7-9 https://github.com/klauspost/compress/pull/739
  • zstd: Add delta encoding support by @greatroar in https://github.com/klauspost/compress/pull/728
  • zstd: Various speed improvements by @greatroar https://github.com/klauspost/compress/pull/741 https://github.com/klauspost/compress/pull/734 https://github.com/klauspost/compress/pull/736 https://github.com/klauspost/compress/pull/744 https://github.com/klauspost/compress/pull/743 https://github.com/klauspost/compress/pull/745
  • gzhttp: Add SuffixETag() and DropETag() options to prevent ETag collisions on compressed responses by @willbicks in https://github.com/klauspost/compress/pull/740

• Jan 3rd, 2023 (v1.15.14)

  • flate: Improve speed in big stateless blocks https://github.com/klauspost/compress/pull/718
  • zstd: Minor speed tweaks by @greatroar in https://github.com/klauspost/compress/pull/716 https://github.com/klauspost/compress/pull/720
  • export NoGzipResponseWriter for custom ResponseWriter wrappers by @harshavardhana in https://github.com/klauspost/compress/pull/722
  • s2: Add example for indexing and existing stream https://github.com/klauspost/compress/pull/723

• Dec 11, 2022 (v1.15.13)

  • zstd: Add MaxEncodedSize to encoder https://github.com/klauspost/compress/pull/691
  • zstd: Various tweaks and improvements https://github.com/klauspost/compress/pull/693 https://github.com/klauspost/compress/pull/695 https://github.com/klauspost/compress/pull/696 https://github.com/klauspost/compress/pull/701 https://github.com/klauspost/compress/pull/702 https://github.com/klauspost/compress/pull/703 https://github.com/klauspost/compress/pull/704 https://github.com/klauspost/compress/pull/705 https://github.com/klauspost/compress/pull/706 https://github.com/klauspost/compress/pull/707 https://github.com/klauspost/compress/pull/708

• Oct 26, 2022 (v1.15.12)

  • zstd: Tweak decoder allocs. https://github.com/klauspost/compress/pull/680
  • gzhttp: Always delete HeaderNoCompression https://github.com/klauspost/compress/pull/683

• Sept 26, 2022 (v1.15.11)

  • flate: Improve level 1-3 compression https://github.com/klauspost/compress/pull/678
  • zstd: Improve "best" compression by @nightwolfz in https://github.com/klauspost/compress/pull/677
  • zstd: Fix+reduce decompression allocations https://github.com/klauspost/compress/pull/668
  • zstd: Fix non-effective noescape tag https://github.com/klauspost/compress/pull/667

• Sept 16, 2022 (v1.15.10)

  • zstd: Add WithDecodeAllCapLimit https://github.com/klauspost/compress/pull/649
  • Add Go 1.19 - deprecate Go 1.16 https://github.com/klauspost/compress/pull/651
  • flate: Improve level 5+6 compression https://github.com/klauspost/compress/pull/656
  • zstd: Improve "better" compression https://github.com/klauspost/compress/pull/657
  • s2: Improve "best" compression https://github.com/klauspost/compress/pull/658
  • s2: Improve "better" compression. https://github.com/klauspost/compress/pull/635
  • s2: Slightly faster non-assembly decompression https://github.com/klauspost/compress/pull/646
  • Use arrays for constant size copies https://github.com/klauspost/compress/pull/659

• July 21, 2022 (v1.15.9)

  • zstd: Fix decoder crash on amd64 (no BMI) on invalid input https://github.com/klauspost/compress/pull/645
  • zstd: Disable decoder extended memory copies (amd64) due to possible crashes https://github.com/klauspost/compress/pull/644
  • zstd: Allow single segments up to "max decoded size" https://github.com/klauspost/compress/pull/643

• July 13, 2022 (v1.15.8)

  • gzip: fix stack exhaustion bug in Reader.Read https://github.com/klauspost/compress/pull/641
  • s2: Add Index header trim/restore https://github.com/klauspost/compress/pull/638
  • zstd: Optimize seqdeq amd64 asm by @greatroar in https://github.com/klauspost/compress/pull/636
  • zstd: Improve decoder memcopy https://github.com/klauspost/compress/pull/637
  • huff0: Pass a single bitReader pointer to asm by @greatroar in https://github.com/klauspost/compress/pull/634
  • zstd: Branchless getBits for amd64 w/o BMI2 by @greatroar in https://github.com/klauspost/compress/pull/640
  • gzhttp: Remove header before writing https://github.com/klauspost/compress/pull/639

• June 29, 2022 (v1.15.7)

  • s2: Fix absolute forward seeks https://github.com/klauspost/compress/pull/633
  • zip: Merge upstream https://github.com/klauspost/compress/pull/631
  • zip: Re-add zip64 fix https://github.com/klauspost/compress/pull/624
  • zstd: translate fseDecoder.buildDtable into asm by @WojciechMula in https://github.com/klauspost/compress/pull/598
  • flate: Faster histograms https://github.com/klauspost/compress/pull/620
  • deflate: Use compound hcode https://github.com/klauspost/compress/pull/622

• June 3, 2022 (v1.15.6)

  • s2: Improve coding for long, close matches https://github.com/klauspost/compress/pull/613
  • s2c: Add Snappy/S2 stream recompression https://github.com/klauspost/compress/pull/611
  • zstd: Always use configured block size https://github.com/klauspost/compress/pull/605
  • zstd: Fix incorrect hash table placement for dict encoding in default https://github.com/klauspost/compress/pull/606
  • zstd: Apply default config to ZipDecompressor without options https://github.com/klauspost/compress/pull/608
  • gzhttp: Exclude more common archive formats https://github.com/klauspost/compress/pull/612
  • s2: Add ReaderIgnoreCRC https://github.com/klauspost/compress/pull/609
  • s2: Remove sanity load on index creation https://github.com/klauspost/compress/pull/607
  • snappy: Use dedicated function for scoring https://github.com/klauspost/compress/pull/614
  • s2c+s2d: Use official snappy framed extension https://github.com/klauspost/compress/pull/610

• May 25, 2022 (v1.15.5)

  • s2: Add concurrent stream decompression https://github.com/klauspost/compress/pull/602
  • s2: Fix final emit oob read crash on amd64 https://github.com/klauspost/compress/pull/601
  • huff0: asm implementation of Decompress1X by @WojciechMula https://github.com/klauspost/compress/pull/596
  • zstd: Use 1 less goroutine for stream decoding https://github.com/klauspost/compress/pull/588
  • zstd: Copy literal in 16 byte blocks when possible https://github.com/klauspost/compress/pull/592
  • zstd: Speed up when WithDecoderLowmem(false) https://github.com/klauspost/compress/pull/599
  • zstd: faster next state update in BMI2 version of decode by @WojciechMula in https://github.com/klauspost/compress/pull/593
  • huff0: Do not check max size when reading table. https://github.com/klauspost/compress/pull/586
  • flate: Inplace hashing for level 7-9 https://github.com/klauspost/compress/pull/590

• May 11, 2022 (v1.15.4)

  • huff0: decompress directly into output by @WojciechMula in #577
  • inflate: Keep dict on stack #581
  • zstd: Faster decoding memcopy in asm #583
  • zstd: Fix ignored crc #580

• May 5, 2022 (v1.15.3)

  • zstd: Allow to ignore checksum checking by @WojciechMula #572
  • s2: Fix incorrect seek for io.SeekEnd in #575

• Apr 26, 2022 (v1.15.2)

  • zstd: Add x86-64 assembly for decompression on streams and blocks. Contributed by @WojciechMula. Typically 2x faster. #528 #531 #545 #537
  • zstd: Add options to ZipDecompressor and fixes #539
  • s2: Use sorted search for index #555
  • Minimum version is Go 1.16, added CI test on 1.18.

• Mar 11, 2022 (v1.15.1)

  • huff0: Add x86 assembly of Decode4X by @WojciechMula in #512
  • zstd: Reuse zip decoders in #514
  • zstd: Detect extra block data and report as corrupted in #520
  • zstd: Handle zero sized frame content size stricter in #521
  • zstd: Add stricter block size checks in #523

• Mar 3, 2022 (v1.15.0)

  • zstd: Refactor decoder #498
  • zstd: Add stream encoding without goroutines #505
  • huff0: Prevent single blocks exceeding 16 bits by @klauspost in#507
  • flate: Inline literal emission #509
  • gzhttp: Add zstd to transport #400
  • gzhttp: Make content-type optional #510

Both compression and decompression now supports "synchronous" stream operations. This means that whenever "concurrency" is set to 1, they will operate without spawning goroutines.

Stream decompression is now faster on asynchronous, since the goroutine allocation much more effectively splits the workload. On typical streams this will typically use 2 cores fully for decompression. When a stream has finished decoding no goroutines will be left over, so decoders can now safely be pooled and still be garbage collected.

While the release has been extensively tested, it is recommended to testing when upgrading.

See changes to v1.14.x

• Feb 22, 2022 (v1.14.4)

  • flate: Fix rare huffman only (-2) corruption. #503
  • zip: Update deprecated CreateHeaderRaw to correctly call CreateRaw by @saracen in #502
  • zip: don't read data descriptor early by @saracen in #501 #501
  • huff0: Use static decompression buffer up to 30% faster #499 #500

• Feb 17, 2022 (v1.14.3)

  • flate: Improve fastest levels compression speed ~10% more throughput. #482 #489 #490 #491 #494 #478
  • flate: Faster decompression speed, ~5-10%. #483
  • s2: Faster compression with Go v1.18 and amd64 microarch level 3+. #484 #486

• Jan 25, 2022 (v1.14.2)

  • zstd: improve header decoder by @dsnet #476
  • zstd: Add bigger default blocks #469
  • zstd: Remove unused decompression buffer #470
  • zstd: Fix logically dead code by @ningmingxiao #472
  • flate: Improve level 7-9 #471 #473
  • zstd: Add noasm tag for xxhash #475

• Jan 11, 2022 (v1.14.1)

  • s2: Add stream index in #462
  • flate: Speed and efficiency improvements in #439 #461 #455 #452 #458
  • zstd: Performance improvement in #420 #456 #437 #467 #468
  • zstd: add arm64 xxhash assembly in #464
  • Add garbled for binaries for s2 in #445

  See changes to v1.13.x

• Aug 30, 2021 (v1.13.5)

  • gz/zlib/flate: Alias stdlib errors #425
  • s2: Add block support to commandline tools #413
  • zstd: pooledZipWriter should return Writers to the same pool #426
  • Removed golang/snappy as external dependency for tests #421

• Aug 12, 2021 (v1.13.4)

  • Add snappy replacement package.
  • zstd: Fix incorrect encoding in "best" mode #415

• Aug 3, 2021 (v1.13.3)

  • zstd: Improve Best compression #404
  • zstd: Fix WriteTo error forwarding #411
  • gzhttp: Return http.HandlerFunc instead of http.Handler. Unlikely breaking change. #406
  • s2sx: Fix max size error #399
  • zstd: Add optional stream content size on reset #401
  • zstd: use SpeedBestCompression for level >= 10 #410

• Jun 14, 2021 (v1.13.1)

  • s2: Add full Snappy output support #396
  • zstd: Add configurable Decoder window size #394
  • gzhttp: Add header to skip compression #389
  • s2: Improve speed with bigger output margin #395

• Jun 3, 2021 (v1.13.0)

  • Added gzhttp which allows wrapping HTTP servers and clients with GZIP compressors.
  • zstd: Detect short invalid signatures #382
  • zstd: Spawn decoder goroutine only if needed. #380

  See changes to v1.12.x

• May 25, 2021 (v1.12.3)

  • deflate: Better/faster Huffman encoding #374
  • deflate: Allocate less for history. #375
  • zstd: Forward read errors #373

• Apr 27, 2021 (v1.12.2)

  • zstd: Improve better/best compression #360 #364 #365
  • zstd: Add helpers to compress/decompress zstd inside zip files #363
  • deflate: Improve level 5+6 compression #367
  • s2: Improve better/best compression #358 #359
  • s2: Load after checking src limit on amd64. #362
  • s2sx: Limit max executable size #368

• Apr 14, 2021 (v1.12.1)

  • snappy package removed. Upstream added as dependency.
  • s2: Better compression in "best" mode #353
  • s2sx: Add stdin input and detect pre-compressed from signature #352
  • s2c/s2d: Add http as possible input #348
  • s2c/s2d/s2sx: Always truncate when writing files #352
  • zstd: Reduce memory usage further when using WithLowerEncoderMem #346
  • s2: Fix potential problem with amd64 assembly and profilers #349

  See changes to v1.11.x

• Mar 26, 2021 (v1.11.13)

  • zstd: Big speedup on small dictionary encodes #344 #345
  • zstd: Add WithLowerEncoderMem encoder option #336
  • deflate: Improve entropy compression #338
  • s2: Clean up and minor performance improvement in best #341

• Mar 5, 2021 (v1.11.12)

  • s2: Add s2sx binary that creates self extracting archives.
  • s2: Speed up decompression on non-assembly platforms #328

• Mar 1, 2021 (v1.11.9)

  • s2: Add ARM64 decompression assembly. Around 2x output speed. #324
  • s2: Improve "better" speed and efficiency. #325
  • s2: Fix binaries.

• Feb 25, 2021 (v1.11.8)

  • s2: Fixed occasional out-of-bounds write on amd64. Upgrade recommended.
  • s2: Add AMD64 assembly for better mode. 25-50% faster. #315
  • s2: Less upfront decoder allocation. #322
  • zstd: Faster "compression" of incompressible data. #314
  • zip: Fix zip64 headers. #313

• Jan 14, 2021 (v1.11.7)

  • Use Bytes() interface to get bytes across packages. #309
  • s2: Add 'best' compression option. #310
  • s2: Add ReaderMaxBlockSize, changes s2.NewReader signature to include varargs. #311
  • s2: Fix crash on small better buffers. #308
  • s2: Clean up decoder. #312

• Jan 7, 2021 (v1.11.6)

  • zstd: Make decoder allocations smaller #306
  • zstd: Free Decoder resources when Reset is called with a nil io.Reader #305

• Dec 20, 2020 (v1.11.4)

  • zstd: Add Best compression mode #304
  • Add header decoder #299
  • s2: Add uncompressed stream option #297
  • Simplify/speed up small blocks with known max size. #300
  • zstd: Always reset literal dict encoder #303

• Nov 15, 2020 (v1.11.3)

  • inflate: 10-15% faster decompression #293
  • zstd: Tweak DecodeAll default allocation #295

• Oct 11, 2020 (v1.11.2)

  • s2: Fix out of bounds read in "better" block compression #291

• Oct 1, 2020 (v1.11.1)

  • zstd: Set allLitEntropy true in default configuration #286

• Sept 8, 2020 (v1.11.0)

  • zstd: Add experimental compression dictionaries #281
  • zstd: Fix mixed Write and ReadFrom calls #282
  • inflate/gz: Limit variable shifts, ~5% faster decompression #274

  See changes to v1.10.x

• July 8, 2020 (v1.10.11)

  • zstd: Fix extra block when compressing with ReadFrom. #278
  • huff0: Also populate compression table when reading decoding table. #275

• June 23, 2020 (v1.10.10)

  • zstd: Skip entropy compression in fastest mode when no matches. #270

• June 16, 2020 (v1.10.9):

  • zstd: API change for specifying dictionaries. See #268
  • zip: update CreateHeaderRaw to handle zip64 fields. #266
  • Fuzzit tests removed. The service has been purchased and is no longer available.

• June 5, 2020 (v1.10.8):

  • 1.15x faster zstd block decompression. #265

• June 1, 2020 (v1.10.7):

  • Added zstd decompression dictionary support
  • Increase zstd decompression speed up to 1.19x. #259
  • Remove internal reset call in zstd compression and reduce allocations. #263

• May 21, 2020: (v1.10.6)

  • zstd: Reduce allocations while decoding. #258, #252
  • zstd: Stricter decompression checks.

• April 12, 2020: (v1.10.5)

  • s2-commands: Flush output when receiving SIGINT. #239

• Apr 8, 2020: (v1.10.4)

  • zstd: Minor/special case optimizations. #251, #250, #249, #247

• Mar 11, 2020: (v1.10.3)

  • s2: Use S2 encoder in pure Go mode for Snappy output as well. #245
  • s2: Fix pure Go block encoder. #244
  • zstd: Added "better compression" mode. #240
  • zstd: Improve speed of fastest compression mode by 5-10% #241
  • zstd: Skip creating encoders when not needed. #238

• Feb 27, 2020: (v1.10.2)

  • Close to 50% speedup in inflate (gzip/zip decompression). #236 #234 #232
  • Reduce deflate level 1-6 memory usage up to 59%. #227

• Feb 18, 2020: (v1.10.1)

  • Fix zstd crash when resetting multiple times without sending data. #226
  • deflate: Fix dictionary use on level 1-6. #224
  • Remove deflate writer reference when closing. #224

• Feb 4, 2020: (v1.10.0)

  • Add optional dictionary to stateless deflate. Breaking change, send nil for previous behaviour. #216
  • Fix buffer overflow on repeated small block deflate. #218
  • Allow copying content from an existing ZIP file without decompressing+compressing. #214
  • Added S2 AMD64 assembler and various optimizations. Stream speed >10GB/s. #186

  See changes prior to v1.10.0

• Jan 20,2020 (v1.9.8) Optimize gzip/deflate with better size estimates and faster table generation. #207 by luyu6056, #206.

• Jan 11, 2020: S2 Encode/Decode will use provided buffer if capacity is big enough. #204

• Jan 5, 2020: (v1.9.7) Fix another zstd regression in v1.9.5 - v1.9.6 removed.

• Jan 4, 2020: (v1.9.6) Regression in v1.9.5 fixed causing corrupt zstd encodes in rare cases.

• Jan 4, 2020: Faster IO in s2c + s2d commandline tools compression/decompression. #192

• Dec 29, 2019: Removed v1.9.5 since fuzz tests showed a compatibility problem with the reference zstandard decoder.

• Dec 29, 2019: (v1.9.5) zstd: 10-20% faster block compression. #199

• Dec 29, 2019: zip package updated with latest Go features

• Dec 29, 2019: zstd: Single segment flag condintions tweaked. #197

• Dec 18, 2019: s2: Faster compression when ReadFrom is used. #198

• Dec 10, 2019: s2: Fix repeat length output when just above at 16MB limit.

• Dec 10, 2019: zstd: Add function to get decoder as io.ReadCloser. #191

• Dec 3, 2019: (v1.9.4) S2: limit max repeat length. #188

• Dec 3, 2019: Add WithNoEntropyCompression to zstd #187

• Dec 3, 2019: Reduce memory use for tests. Check for leaked goroutines.

• Nov 28, 2019 (v1.9.3) Less allocations in stateless deflate.

• Nov 28, 2019: 5-20% Faster huff0 decode. Impacts zstd as well. #184

• Nov 12, 2019 (v1.9.2) Added Stateless Compression for gzip/deflate.

• Nov 12, 2019: Fixed zstd decompression of large single blocks. #180

• Nov 11, 2019: Set default s2c block size to 4MB.

• Nov 11, 2019: Reduce inflate memory use by 1KB.

• Nov 10, 2019: Less allocations in deflate bit writer.

• Nov 10, 2019: Fix inconsistent error returned by zstd decoder.

• Oct 28, 2019 (v1.9.1) ztsd: Fix crash when compressing blocks. #174

• Oct 24, 2019 (v1.9.0) zstd: Fix rare data corruption #173

• Oct 24, 2019 zstd: Fix huff0 out of buffer write #171 and always return errors #172

• Oct 10, 2019: Big deflate rewrite, 30-40% faster with better compression #105

  See changes prior to v1.9.0

• Oct 10, 2019: (v1.8.6) zstd: Allow partial reads to get flushed data. #169

• Oct 3, 2019: Fix inconsistent results on broken zstd streams.

• Sep 25, 2019: Added -rm (remove source files) and -q (no output except errors) to s2c and s2d commands

• Sep 16, 2019: (v1.8.4) Add s2c and s2d commandline tools.

• Sep 10, 2019: (v1.8.3) Fix s2 decoder Skip.

• Sep 7, 2019: zstd: Added WithWindowSize, contributed by ianwilkes.

• Sep 5, 2019: (v1.8.2) Add WithZeroFrames which adds full zero payload block encoding option.

• Sep 5, 2019: Lazy initialization of zstandard predefined en/decoder tables.

• Aug 26, 2019: (v1.8.1) S2: 1-2% compression increase in "better" compression mode.

• Aug 26, 2019: zstd: Check maximum size of Huffman 1X compressed literals while decoding.

• Aug 24, 2019: (v1.8.0) Added S2 compression, a high performance replacement for Snappy.

• Aug 21, 2019: (v1.7.6) Fixed minor issues found by fuzzer. One could lead to zstd not decompressing.

• Aug 18, 2019: Add fuzzit continuous fuzzing.

• Aug 14, 2019: zstd: Skip incompressible data 2x faster. #147

• Aug 4, 2019 (v1.7.5): Better literal compression. #146

• Aug 4, 2019: Faster zstd compression. #143 #144

• Aug 4, 2019: Faster zstd decompression. #145 #143 #142

• July 15, 2019 (v1.7.4): Fix double EOF block in rare cases on zstd encoder.

• July 15, 2019 (v1.7.3): Minor speedup/compression increase in default zstd encoder.

• July 14, 2019: zstd decoder: Fix decompression error on multiple uses with mixed content.

• July 7, 2019 (v1.7.2): Snappy update, zstd decoder potential race fix.

• June 17, 2019: zstd decompression bugfix.

• June 17, 2019: fix 32 bit builds.

• June 17, 2019: Easier use in modules (less dependencies).

• June 9, 2019: New stronger "default" zstd compression mode. Matches zstd default compression ratio.

• June 5, 2019: 20-40% throughput in zstandard compression and better compression.

• June 5, 2019: deflate/gzip compression: Reduce memory usage of lower compression levels.

• June 2, 2019: Added zstandard compression!

• May 25, 2019: deflate/gzip: 10% faster bit writer, mostly visible in lower levels.

• Apr 22, 2019: zstd decompression added.

• Aug 1, 2018: Added huff0 README.

• Jul 8, 2018: Added Performance Update 2018 below.

• Jun 23, 2018: Merged Go 1.11 inflate optimizations. Go 1.9 is now required. Backwards compatible version tagged with v1.3.0.

• Apr 2, 2018: Added huff0 en/decoder. Experimental for now, API may change.

• Mar 4, 2018: Added FSE Entropy en/decoder. Experimental for now, API may change.

• Nov 3, 2017: Add compression Estimate function.

• May 28, 2017: Reduce allocations when resetting decoder.

• Apr 02, 2017: Change back to official crc32, since changes were merged in Go 1.7.

• Jan 14, 2017: Reduce stack pressure due to array copies. See Issue #18625.

• Oct 25, 2016: Level 2-4 have been rewritten and now offers significantly better performance than before.

• Oct 20, 2016: Port zlib changes from Go 1.7 to fix zlib writer issue. Please update.

• Oct 16, 2016: Go 1.7 changes merged. Apples to apples this package is a few percent faster, but has a significantly better balance between speed and compression per level.

• Mar 24, 2016: Always attempt Huffman encoding on level 4-7. This improves base 64 encoded data compression.

• Mar 24, 2016: Small speedup for level 1-3.

• Feb 19, 2016: Faster bit writer, level -2 is 15% faster, level 1 is 4% faster.

• Feb 19, 2016: Handle small payloads faster in level 1-3.

• Feb 19, 2016: Added faster level 2 + 3 compression modes.

• Feb 19, 2016: Rebalanced compression levels, so there is a more even progression in terms of compression. New default level is 5.

• Feb 14, 2016: Snappy: Merge upstream changes.

• Feb 14, 2016: Snappy: Fix aggressive skipping.

• Feb 14, 2016: Snappy: Update benchmark.

• Feb 13, 2016: Deflate: Fixed assembler problem that could lead to sub-optimal compression.

• Feb 12, 2016: Snappy: Added AMD64 SSE 4.2 optimizations to matching, which makes easy to compress material run faster. Typical speedup is around 25%.

• Feb 9, 2016: Added Snappy package fork. This version is 5-7% faster, much more on hard to compress content.

• Jan 30, 2016: Optimize level 1 to 3 by not considering static dictionary or storing uncompressed. ~4-5% speedup.

• Jan 16, 2016: Optimization on deflate level 1,2,3 compression.

• Jan 8 2016: Merge CL 18317: fix reading, writing of zip64 archives.

• Dec 8 2015: Make level 1 and -2 deterministic even if write size differs.

• Dec 8 2015: Split encoding functions, so hashing and matching can potentially be inlined. 1-3% faster on AMD64. 5% faster on other platforms.

• Dec 8 2015: Fixed rare one byte out-of bounds read. Please update!

• Nov 23 2015: Optimization on token writer. ~2-4% faster. Contributed by @dsnet.

• Nov 20 2015: Small optimization to bit writer on 64 bit systems.

• Nov 17 2015: Fixed out-of-bound errors if the underlying Writer returned an error. See #15.

• Nov 12 2015: Added io.WriterTo support to gzip/inflate.

• Nov 11 2015: Merged CL 16669: archive/zip: enable overriding (de)compressors per file

• Oct 15 2015: Added skipping on uncompressible data. Random data speed up >5x.

deflate usage

The packages are drop-in replacements for standard library deflate, gzip, zip, and zlib. Simply replace the import path to use them:

Typical speed is about 2x of the standard library packages.

You may also be interested in pgzip, which is a drop-in replacement for gzip, which support multithreaded compression on big files and the optimized crc32 package used by these packages.

The packages implement the same API as the standard library, so you can use the original godoc documentation: gzip, zip, zlib, flate.

Currently there is only minor speedup on decompression (mostly CRC32 calculation).

Memory usage is typically 1MB for a Writer. stdlib is in the same range.
If you expect to have a lot of concurrently allocated Writers consider using
the stateless compression described below.

For compression performance, see: this spreadsheet.

To disable all assembly add -tags=noasm. This works across all packages.

Stateless compression

This package offers stateless compression as a special option for gzip/deflate.
It will do compression but without maintaining any state between Write calls.

This means there will be no memory kept between Write calls, but compression and speed will be suboptimal.

This is only relevant in cases where you expect to run many thousands of compressors concurrently,
but with very little activity. This is not intended for regular web servers serving individual requests.

Because of this, the size of actual Write calls will affect output size.

In gzip, specify level -3 / gzip.StatelessCompression to enable.

For direct deflate use, NewStatelessWriter and StatelessDeflate are available. See documentation

A bufio.Writer can of course be used to control write sizes. For example, to use a 4KB buffer:

// replace 'ioutil.Discard' with your output.
	gzw, err := gzip.NewWriterLevel(ioutil.Discard, gzip.StatelessCompression)
	if err != nil {
		return err
	}
	defer gzw.Close()

	w := bufio.NewWriterSize(gzw, 4096)
	defer w.Flush()
	
	// Write to 'w'

This will only use up to 4KB in memory when the writer is idle.

Compression is almost always worse than the fastest compression level
and each write will allocate (a little) memory.

Other packages

Here are other packages of good quality and pure Go (no cgo wrappers or autoconverted code):

• github.com/pierrec/lz4 - strong multithreaded LZ4 compression.
• github.com/cosnicolaou/pbzip2 - multithreaded bzip2 decompression.
• github.com/dsnet/compress - brotli decompression, bzip2 writer.
• github.com/ronanh/intcomp - Integer compression.
• github.com/spenczar/fpc - Float compression.
• github.com/minio/zipindex - External ZIP directory index.
• github.com/ybirader/pzip - Fast concurrent zip archiver and extractor.

license

This code is licensed under the same conditions as the original Go code. See LICENSE file.

Overview (`vendor/github.com/mitchellh/mapstructure/README.md`) Copy Section

mapstructure [](https://godoc.org/github.com/mitchellh/mapstructure)

mapstructure is a Go library for decoding generic map values to structures
and vice versa, while providing helpful error handling.

This library is most useful when decoding values from some data stream (JSON,
Gob, etc.) where you don't quite know the structure of the underlying data
until you read a part of it. You can therefore read a map[string]interface{}
and use this library to decode it into the proper underlying native Go
structure.

Installation Copy Chapter

Standard go get:

$ go get github.com/mitchellh/mapstructure

Usage & Example Copy Chapter

For usage and examples see the Godoc.

The Decode function has examples associated with it there.

But Why?! Copy Chapter

Go offers fantastic standard libraries for decoding formats such as JSON.
The standard method is to have a struct pre-created, and populate that struct
from the bytes of the encoded format. This is great, but the problem is if
you have configuration or an encoding that changes slightly depending on
specific fields. For example, consider this JSON:

{
  "type": "person",
  "name": "Mitchell"
}

Perhaps we can't populate a specific structure without first reading
the "type" field from the JSON. We could always do two passes over the
decoding of the JSON (reading the "type" first, and the rest later).
However, it is much simpler to just decode this into a map[string]interface{}
structure, read the "type" key, then use something like this library
to decode it into the proper structure.

Overview (`vendor/github.com/opencontainers/go-digest/README.md`) Copy Section

go-digest

Common digest package used across the container ecosystem.

Please see the godoc for more information.

What is a digest?

A digest is just a hash.

The most common use case for a digest is to create a content identifier for use in Content Addressable Storage systems:

id := digest.FromBytes([]byte("my content"))

In the example above, the id can be used to uniquely identify the byte slice "my content".
This allows two disparate applications to agree on a verifiable identifier without having to trust one another.

An identifying digest can be verified, as follows:

if id != digest.FromBytes([]byte("my content")) {
  return errors.New("the content has changed!")
}

A Verifier type can be used to handle cases where an io.Reader makes more sense:

rd := getContent()
verifier := id.Verifier()
io.Copy(verifier, rd)

if !verifier.Verified() {
  return errors.New("the content has changed!")
}

Using Merkle DAGs, this can power a rich, safe, content distribution system.

Usage

While the godoc is considered the best resource, a few important items need to be called out when using this package.

1. Make sure to import the hash implementations into your application or the package will panic.
   You should have something like the following in the main (or other entrypoint) of your application:

   go Copy

   import (
       _ "crypto/sha256"
       _ "crypto/sha512"
   )

   This may seem inconvenient but it allows you replace the hash
   implementations with others, such as https://github.com/stevvooe/resumable.

2. Even though digest.Digest may be assemblable as a string, always verify your input with digest.Parse or use Digest.Validate when accepting untrusted input.
   While there are measures to avoid common problems, this will ensure you have valid digests in the rest of your application.

3. While alternative encodings of hash values (digests) are possible (for example, base64), this package deals exclusively with hex-encoded digests.

Stability

The Go API, at this stage, is considered stable, unless otherwise noted.

As always, before using a package export, read the godoc.

Contributing

This package is considered fairly complete.
It has been in production in thousands (millions?) of deployments and is fairly battle-hardened.
New additions will be met with skepticism.
If you think there is a missing feature, please file a bug clearly describing the problem and the alternatives you tried before submitting a PR.

Code of Conduct Copy Chapter

Participation in the OpenContainers community is governed by OpenContainer's Code of Conduct.

Security Copy Chapter

If you find an issue, please follow the security protocol to report it.

Copyright and license

Copyright © 2019, 2020 OCI Contributors
Copyright © 2016 Docker, Inc.
All rights reserved, except as follows.
Code is released under the Apache 2.0 license.
This README.md file and the CONTRIBUTING.md file are licensed under the Creative Commons Attribution 4.0 International License under the terms and conditions set forth in the file LICENSE.docs.
You may obtain a duplicate copy of the same license, titled CC BY-SA 4.0, at http://creativecommons.org/licenses/by-sa/4.0/.

Overview (`vendor/github.com/pkg/browser/README.md`) Copy Section

browser

import "github.com/pkg/browser"

Package browser provides helpers to open files, readers, and urls in a browser window.

The choice of which browser is started is entirely client dependant.

Variables Copy Chapter

var Stderr io.Writer = os.Stderr

Stderr is the io.Writer to which executed commands write standard error.

var Stdout io.Writer = os.Stdout

Stdout is the io.Writer to which executed commands write standard output.

func OpenFile Copy Chapter

func OpenFile(path string) error

OpenFile opens new browser window for the file path.

func OpenReader Copy Chapter

func OpenReader(r io.Reader) error

OpenReader consumes the contents of r and presents the
results in a new browser window.

func OpenURL Copy Chapter

func OpenURL(url string) error

OpenURL opens a new browser window pointing to url.

Generated by godoc2md

Overview (`vendor/github.com/prometheus/otlptranslator/README.md`) Copy Section

OTLP Prometheus Translator

A Go library for converting OpenTelemetry Protocol (OTLP) metric and attribute names to Prometheus-compliant formats. This is an internal library for both Prometheus and Open Telemetry, without any stability guarantees for external usage.

Part of the Prometheus ecosystem, following the OpenTelemetry to Prometheus compatibility specification.

Features Copy Chapter

• Metric Name and Label Translation: Convert OTLP metric names and attributes to Prometheus-compliant format
• Unit Handling: Translate OTLP units to Prometheus unit conventions
• Type-Aware Suffixes: Optionally append _total, _ratio based on metric type
• Namespace Support: Add configurable namespace prefixes
• UTF-8 Support: Choose between Prometheus legacy scheme compliant metric/label names ([a-zA-Z0-9:_]) or untranslated metric/label names
• Translation Strategy Configuration: Select a translation strategy with a standard set of strings.

Installation Copy Chapter

go get github.com/prometheus/otlptranslator

Quick Start Copy Chapter

package main

import (
    "fmt"
    "github.com/prometheus/otlptranslator"
)

func main() {
    // Create a metric namer using traditional Prometheus name translation, with suffixes added and UTF-8 disallowed.
    strategy := otlptranslator.UnderscoreEscapingWithSuffixes
    namer := otlptranslator.NewMetricNamer("myapp", strategy)

    // Translate OTLP metric to Prometheus format
    metric := otlptranslator.Metric{
        Name: "http.server.request.duration",
        Unit: "s",
        Type: otlptranslator.MetricTypeHistogram,
    }
    fmt.Println(namer.Build(metric)) // Output: myapp_http_server_request_duration_seconds

    // Translate label names
    labelNamer := otlptranslator.LabelNamer{UTF8Allowed: false}
    fmt.Println(labelNamer.Build("http.method")) // Output: http_method
}

Usage Examples Copy Chapter

Metric Name Translation Copy Section

namer := otlptranslator.MetricNamer{WithMetricSuffixes: true, UTF8Allowed: false}

// Counter gets _total suffix
counter := otlptranslator.Metric{
    Name: "requests.count", Unit: "1", Type: otlptranslator.MetricTypeMonotonicCounter,
}
fmt.Println(namer.Build(counter)) // requests_count_total

// Gauge with unit conversion
gauge := otlptranslator.Metric{
    Name: "memory.usage", Unit: "By", Type: otlptranslator.MetricTypeGauge,
}
fmt.Println(namer.Build(gauge)) // memory_usage_bytes

// Dimensionless gauge gets _ratio suffix
ratio := otlptranslator.Metric{
    Name: "cpu.utilization", Unit: "1", Type: otlptranslator.MetricTypeGauge,
}
fmt.Println(namer.Build(ratio)) // cpu_utilization_ratio

Label Translation Copy Section

labelNamer := otlptranslator.LabelNamer{UTF8Allowed: false}

labelNamer.Build("http.method")           // http_method
labelNamer.Build("123invalid")            // key_123invalid
labelNamer.Build("_private")              // key_private
labelNamer.Build("__reserved__")          // __reserved__ (preserved)
labelNamer.Build("label@with$symbols")    // label_with_symbols

Unit Translation Copy Section

unitNamer := otlptranslator.UnitNamer{UTF8Allowed: false}

unitNamer.Build("s")           // seconds
unitNamer.Build("By")          // bytes
unitNamer.Build("requests/s")  // requests_per_second
unitNamer.Build("1")           // "" (dimensionless)

Configuration Options Copy Section

// Prometheus-compliant mode - supports [a-zA-Z0-9:_]
compliantNamer := otlptranslator.MetricNamer{UTF8Allowed: false, WithMetricSuffixes: true}

// Transparent pass-through mode, aka "NoTranslation"
utf8Namer := otlptranslator.MetricNamer{UTF8Allowed: true, WithMetricSuffixes: false}
utf8Namer = otlptranslator.NewMetricNamer("", otlpTranslator.NoTranslation)

// With namespace and suffixes
productionNamer := otlptranslator.MetricNamer{
    Namespace:          "myservice",
    WithMetricSuffixes: true,
    UTF8Allowed:        false,
}

License Copy Chapter

Licensed under the Apache License 2.0 - see the LICENSE file for details.

Overview (`vendor/github.com/prometheus/procfs/README.md`) Copy Section

procfs

This package provides functions to retrieve system, kernel, and process
metrics from the pseudo-filesystems /proc and /sys.

WARNING: This package is a work in progress. Its API may still break in
backwards-incompatible ways without warnings. Use it at your own risk.

Usage Copy Chapter

The procfs library is organized by packages based on whether the gathered data is coming from
/proc, /sys, or both. Each package contains an FS type which represents the path to either /proc,
/sys, or both. For example, cpu statistics are gathered from
/proc/stat and are available via the root procfs package. First, the proc filesystem mount
point is initialized, and then the stat information is read.

fs, err := procfs.NewFS("/proc")
stats, err := fs.Stat()

Some sub-packages such as blockdevice, require access to both the proc and sys filesystems.

fs, err := blockdevice.NewFS("/proc", "/sys")
    stats, err := fs.ProcDiskstats()

Package Organization Copy Chapter

The packages in this project are organized according to (1) whether the data comes from the /proc or
/sys filesystem and (2) the type of information being retrieved. For example, most process information
can be gathered from the functions in the root procfs package. Information about block devices such as disk drives
is available in the blockdevices sub-package.

Building and Testing Copy Chapter

The procfs library is intended to be built as part of another application, so there are no distributable binaries.
However, most of the API includes unit tests which can be run with make test.

Updating Test Fixtures Copy Section

The procfs library includes a set of test fixtures which include many example files from
the /proc and /sys filesystems. These fixtures are included as a ttar file
which is extracted automatically during testing. To add/update the test fixtures, first
ensure the testdata/fixtures directory is up to date by removing the existing directory and then
extracting the ttar file using make testdata/fixtures/.unpacked or just make test.

rm -rf testdata/fixtures
make test

Next, make the required changes to the extracted files in the testdata/fixtures directory. When
the changes are complete, run make update_fixtures to create a new fixtures.ttar file
based on the updated fixtures directory. And finally, verify the changes using
git diff testdata/fixtures.ttar.

Overview (`vendor/github.com/sirupsen/logrus/README.md`) Copy Section

Logrus [](https://github.com/sirupsen/logrus/actions?query=workflow%3ACI) [](https://pkg.go.dev/github.com/sirupsen/logrus)

Logrus is a structured logger for Go (golang), completely API compatible with
the standard library logger.

Logrus is in maintenance mode. The project focuses on security, bug fixes,
and performance improvements. New features are not planned, aside from changes
required to provide interoperability with other logging ecosystems (e.g., Go's
log/slog).

I believe Logrus' biggest contribution is to have played a part in today's
widespread use of structured logging in Golang. There doesn't seem to be a
reason to do a major, breaking iteration into Logrus V2, since the fantastic Go
community has built those independently. Many fantastic alternatives have sprung
up. Logrus would look like those, had it been re-designed with what we know
about structured logging in Go today. Check out, for example,
Zerolog, Zap, and Apex.

Nicely color-coded in development (when a TTY is attached, otherwise just
plain text):

With logrus.SetFormatter(&logrus.JSONFormatter{}), for easy parsing by logstash
or Splunk:

{"animal":"walrus","level":"info","msg":"A group of walrus emerges from the ocean","size":10,"time":"2014-03-10 19:57:38.562264131 -0400 EDT"}
{"level":"warning","msg":"The group's number increased tremendously!","number":122,"omg":true,"time":"2014-03-10 19:57:38.562471297 -0400 EDT"}
{"animal":"walrus","level":"info","msg":"A giant walrus appears!","size":10,"time":"2014-03-10 19:57:38.562500591 -0400 EDT"}
{"animal":"walrus","level":"info","msg":"Tremendously sized cow enters the ocean.","size":9,"time":"2014-03-10 19:57:38.562527896 -0400 EDT"}
{"level":"fatal","msg":"The ice breaks!","number":100,"omg":true,"time":"2014-03-10 19:57:38.562543128 -0400 EDT"}

With the default logrus.SetFormatter(&logrus.TextFormatter{}) when a TTY is not
attached, the output is compatible with the
logfmt format:

time="2015-03-26T01:27:38-04:00" level=debug msg="Started observing beach" animal=walrus number=8
time="2015-03-26T01:27:38-04:00" level=info msg="A group of walrus emerges from the ocean" animal=walrus size=10
time="2015-03-26T01:27:38-04:00" level=warning msg="The group's number increased tremendously!" number=122 omg=true
time="2015-03-26T01:27:38-04:00" level=debug msg="Temperature changes" temperature=-4
time="2015-03-26T01:27:38-04:00" level=panic msg="It's over 9000!" animal=orca size=9009
time="2015-03-26T01:27:38-04:00" level=fatal msg="The ice breaks!" animal=orca err="It's over 9000!" number=100 omg=true size=9009

To ensure this behaviour even if a TTY is attached, set your formatter as follows:

logrus.SetFormatter(&logrus.TextFormatter{
    DisableColors: true,
    FullTimestamp: true,
})

Logging Method Name

If you wish to add the calling method as a field, instruct the logger via:

logrus.SetReportCaller(true)

This adds the caller as 'method' like so:

{"animal":"penguin","level":"fatal","method":"github.com/sirupsen/arcticcreatures.migrate","msg":"a penguin swims by","time":"2014-03-10 19:57:38.562543129 -0400 EDT"}

time="2015-03-26T01:27:38-04:00" level=fatal method=github.com/sirupsen/arcticcreatures.migrate msg="a penguin swims by" animal=penguin

Note that this does add measurable overhead - the cost will depend on the version of Go, but is
between 20 and 40% in recent tests with 1.6 and 1.7. You can validate this in your
environment via benchmarks:

go test -bench=ReportCaller

Case-sensitivity

The organization's name was changed to lower-case. If you are getting import
conflicts due to case sensitivity, please use the lower-case import:
github.com/sirupsen/logrus.

Example

The simplest way to use Logrus is simply the package-level exported logger:

package main

import "github.com/sirupsen/logrus"

func main() {
  logrus.WithFields(logrus.Fields{
    "animal": "walrus",
  }).Info("A walrus appears")
}

Note that it's completely api-compatible with the stdlib logger, so you can
replace your log imports everywhere with log "github.com/sirupsen/logrus"
and you'll now have the flexibility of Logrus. You can customize it all you
want:

package main

import (
  "os"

  log "github.com/sirupsen/logrus"
)

func init() {
  // Log as JSON instead of the default ASCII formatter.
  log.SetFormatter(&log.JSONFormatter{})

  // Output to stdout instead of the default stderr
  // Can be any io.Writer, see below for File example
  log.SetOutput(os.Stdout)

  // Only log the warning severity or above.
  log.SetLevel(log.WarnLevel)
}

func main() {
  log.WithFields(log.Fields{
    "animal": "walrus",
    "size":   10,
  }).Info("A group of walrus emerges from the ocean")

  log.WithFields(log.Fields{
    "omg":    true,
    "number": 122,
  }).Warn("The group's number increased tremendously!")

  log.WithFields(log.Fields{
    "omg":    true,
    "number": 100,
  }).Fatal("The ice breaks!")

  // A common pattern is to re-use fields between logging statements by re-using
  // the logrus.Entry returned from WithFields()
  contextLogger := log.WithFields(log.Fields{
    "common": "this is a common field",
    "other": "I also should be logged always",
  })

  contextLogger.Info("I'll be logged with common and other field")
  contextLogger.Info("Me too")
}

For more advanced usage such as logging to multiple locations from the same
application, you can also create an instance of the logrus Logger:

package main

import (
  "os"

  "github.com/sirupsen/logrus"
)

// Create a new instance of the logger. You can have any number of instances.
var logger = logrus.New()

func main() {
  // The API for setting attributes is a little different than the package level
  // exported logger. See Godoc. 
  logger.Out = os.Stdout

  // You could set this to any `io.Writer` such as a file
  // file, err := os.OpenFile("logrus.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
  // if err == nil {
  //  logger.Out = file
  // } else {
  //  logger.Info("Failed to log to file, using default stderr")
  // }

  logger.WithFields(logrus.Fields{
    "animal": "walrus",
    "size":   10,
  }).Info("A group of walrus emerges from the ocean")
}

Fields

Logrus encourages careful, structured logging through logging fields instead of
long, unparseable error messages. For example, instead of: logrus.Fatalf("Failed to send event %s to topic %s with key %d"), you should log the much more
discoverable:

logrus.WithFields(logrus.Fields{
  "event": event,
  "topic": topic,
  "key": key,
}).Fatal("Failed to send event")

We've found this API forces you to think about logging in a way that produces
much more useful logging messages. We've been in countless situations where just
a single added field to a log statement that was already there would've saved us
hours. The WithFields call is optional.

In general, with Logrus using any of the printf-family functions should be
seen as a hint you should add a field, however, you can still use the
printf-family functions with Logrus.

Default Fields

Often it's helpful to have fields always attached to log statements in an
application or parts of one. For example, you may want to always log the
request_id and user_ip in the context of a request. Instead of writing
logger.WithFields(logrus.Fields{"request_id": request_id, "user_ip": user_ip}) on
every line, you can create a logrus.Entry to pass around instead:

requestLogger := logger.WithFields(logrus.Fields{"request_id": request_id, "user_ip": user_ip})
requestLogger.Info("something happened on that request") // will log request_id and user_ip
requestLogger.Warn("something not great happened")

Hooks

You can add hooks for logging levels. For example to send errors to an exception
tracking service on Error, Fatal and Panic, info to StatsD or log to
multiple places simultaneously, e.g. syslog.

Logrus comes with built-in hooks. Add those, or your custom hook, in
init:

package main

import (
  "log/syslog"

  "github.com/sirupsen/logrus"
  airbrake "gopkg.in/gemnasium/logrus-airbrake-hook.v2"
  logrus_syslog "github.com/sirupsen/logrus/hooks/syslog"
)

func init() {

  // Use the Airbrake hook to report errors that have Error severity or above to
  // an exception tracker. You can create custom hooks, see the Hooks section.
  logrus.AddHook(airbrake.NewHook(123, "xyz", "production"))

  hook, err := logrus_syslog.NewSyslogHook("udp", "localhost:514", syslog.LOG_INFO, "")
  if err != nil {
    logrus.Error("Unable to connect to local syslog daemon")
  } else {
    logrus.AddHook(hook)
  }
}

Note: Syslog hooks also support connecting to local syslog (Ex. "/dev/log" or "/var/run/syslog" or "/var/run/log"). For the detail, please check the syslog hook README.

A list of currently known service hooks can be found in this wiki page

Level logging

Logrus has seven logging levels: Trace, Debug, Info, Warning, Error, Fatal and Panic.

logrus.Trace("Something very low level.")
logrus.Debug("Useful debugging information.")
logrus.Info("Something noteworthy happened!")
logrus.Warn("You should probably take a look at this.")
logrus.Error("Something failed but I'm not quitting.")
// Calls os.Exit(1) after logging
logrus.Fatal("Bye.")
// Calls panic() after logging
logrus.Panic("I'm bailing.")

You can set the logging level on a Logger, then it will only log entries with
that severity or anything above it:

// Will log anything that is info or above (warn, error, fatal, panic). Default.
logrus.SetLevel(logrus.InfoLevel)

It may be useful to set logrus.Level = logrus.DebugLevel in a debug or verbose
environment if your application has that.

Note: If you want different log levels for global (logrus.SetLevel(...)) and syslog logging, please check the syslog hook README.

Entries

Besides the fields added with WithField or WithFields some fields are
automatically added to all logging events:

1. time. The timestamp when the entry was created.
2. msg. The logging message passed to {Info,Warn,Error,Fatal,Panic} after
   the AddFields call. E.g. Failed to send event.
3. level. The logging level. E.g. info.

Environments

Logrus has no notion of environment.

If you wish for hooks and formatters to only be used in specific environments,
you should handle that yourself. For example, if your application has a global
variable Environment, which is a string representation of the environment you
could do:

import (
  "github.com/sirupsen/logrus"
)

func init() {
  // do something here to set environment depending on an environment variable
  // or command-line flag
  if Environment == "production" {
    logrus.SetFormatter(&logrus.JSONFormatter{})
  } else {
    // The TextFormatter is default, you don't actually have to do this.
    logrus.SetFormatter(&logrus.TextFormatter{})
  }
}

This configuration is how logrus was intended to be used, but JSON in
production is mostly only useful if you do log aggregation with tools like
Splunk or Logstash.

Formatters

The built-in logging formatters are:

• logrus.TextFormatter
  logs the event in colors if the logger output is a TTY, otherwise without colors.
  • To force colored output when there is no TTY, set the ForceColors
    field to true. To force no colored output even if there is a TTY set the
    DisableColors field to true.
  • On modern Windows terminals with ANSI (Virtual Terminal) support, TextFormatter
    automatically enables colored output.
  • If your environment does not support ANSI escape sequences, wrap the logger output
    using github.com/mattn/go-colorable
    and set ForceColors (or CLICOLOR_FORCE=1) to enable colors through the wrapper.
  • When colors are enabled, levels are truncated to 4 characters by default. To disable
    truncation set the DisableLevelTruncation field to true.
  • When outputting to a TTY, it's often helpful to visually scan down a column where all the levels are the same width. Setting the PadLevelText field to true enables this behavior, by adding padding to the level text.
• logrus.JSONFormatter
  logs fields as JSON.

Third-party logging formatters:

• FluentdFormatter. Formats entries that can be parsed by Kubernetes and Google Container Engine.
• GELF. Formats entries so they comply to Graylog's GELF 1.1 specification.
• logstash. Logs fields as Logstash Events.
• prefixed. Displays log entry source along with alternative layout.
• zalgo. Invoking the Power of Zalgo.
• nested-logrus-formatter. Converts logrus fields to a nested structure.
• powerful-logrus-formatter. get fileName, log's line number and the latest function's name when print log; Save log to files.
• caption-json-formatter. logrus's message json formatter with human-readable caption added.
• easy-logrus-formatter. Provide a user-friendly formatter for logrus.
• redactrus. Redacts sensitive information like password, apikeys, email, etc. from logs.

You can define your formatter by implementing the Formatter interface,
requiring a Format method. Format takes an *Entry. entry.Data is a
Fields type (map[string]any) with all your fields as well as the
default ones (see Entries section above):

type MyJSONFormatter struct{}

logrus.SetFormatter(new(MyJSONFormatter))

func (f *MyJSONFormatter) Format(entry *Entry) ([]byte, error) {
  // Note this doesn't include Time, Level and Message which are available on
  // the Entry. Consult `godoc` on information about those fields or read the
  // source of the official loggers.
  serialized, err := json.Marshal(entry.Data)
    if err != nil {
      return nil, fmt.Errorf("Failed to marshal fields to JSON, %w", err)
    }
  return append(serialized, '\n'), nil
}

Logger as an `io.Writer`

Logrus can be transformed into an io.Writer. That writer is the end of an io.Pipe and it is your responsibility to close it.

w := logger.Writer()
defer w.Close()

srv := http.Server{
    // create a stdlib log.Logger that writes to
    // logrus.Logger.
    ErrorLog: log.New(w, "", 0),
}

Each line written to that writer will be printed the usual way, using formatters
and hooks. The level for those entries is info.

This means that we can override the standard library logger easily:

logger := logrus.New()
logger.Formatter = &logrus.JSONFormatter{}

// Use logrus for standard log output
// Note that `log` here references stdlib's log
// Not logrus imported under the name `log`.
log.SetOutput(logger.Writer())

Rotation

Log rotation is not provided with Logrus. Log rotation should be done by an
external program (like logrotate(8)) that can compress and delete old log
entries. It should not be a feature of the application-level logger.

Tools

Testing

Logrus has a built-in facility for asserting the presence of log messages. This is implemented through the test hook and provides:

• decorators for existing logger (test.NewLocal and test.NewGlobal) which basically just adds the test hook
• a test logger (test.NewNullLogger) that just records log messages (and does not output any):

import(
  "testing"

  "github.com/sirupsen/logrus"
  "github.com/sirupsen/logrus/hooks/test"
  "github.com/stretchr/testify/assert"
)

func TestSomething(t*testing.T){
  logger, hook := test.NewNullLogger()
  logger.Error("Helloerror")

  assert.Equal(t, 1, len(hook.Entries))
  assert.Equal(t, logrus.ErrorLevel, hook.LastEntry().Level)
  assert.Equal(t, "Helloerror", hook.LastEntry().Message)

  hook.Reset()
  assert.Nil(t, hook.LastEntry())
}

Fatal handlers

Logrus can register one or more functions that will be called when any fatal
level message is logged. The registered handlers will be executed before
logrus performs an os.Exit(1). This behavior may be helpful if callers need
to gracefully shut down. Unlike a panic("Something went wrong...") call which can be intercepted with a deferred recover a call to os.Exit(1) can not be intercepted.

// ...
handler := func() {
  // gracefully shut down something...
}
logrus.RegisterExitHandler(handler)
// ...

Thread safety

By default, Logger is protected by a mutex for concurrent writes. The mutex is held when calling hooks and writing logs.
If you are sure such locking is not needed, you can call logger.SetNoLock() to disable the locking.

Situations when locking is not needed include:

• You have no hooks registered, or hooks calling is already thread-safe.

• Writing to logger.Out is already thread-safe, for example:

  1. logger.Out is protected by locks.

  2. logger.Out is an os.File handler opened with O_APPEND flag, and every write is smaller than 4k. (This allows multi-thread/multi-process writing)

     (Refer to http://www.notthewizard.com/2014/06/17/are-files-appends-really-atomic/)

Overview (`vendor/github.com/spf13/cobra/README.md`) Copy Section

Cobra is a library for creating powerful modern CLI applications.

Visit Cobra.dev for extensive documentation

Cobra is used in many Go projects such as Kubernetes,
Hugo, and GitHub CLI to
name a few. This list contains a more extensive list of projects using Cobra.

Overview

Cobra is a library providing a simple interface to create powerful modern CLI
interfaces similar to git & go tools.

Cobra provides:

• Easy subcommand-based CLIs: app server, app fetch, etc.
• Fully POSIX-compliant flags (including short & long versions)
• Nested subcommands
• Global, local and cascading flags
• Intelligent suggestions (app srver... did you mean app server?)
• Automatic help generation for commands and flags
• Grouping help for subcommands
• Automatic help flag recognition of -h, --help, etc.
• Automatically generated shell autocomplete for your application (bash, zsh, fish, powershell)
• Automatically generated man pages for your application
• Command aliases so you can change things without breaking them
• The flexibility to define your own help, usage, etc.
• Optional seamless integration with viper for 12-factor apps

Concepts

Cobra is built on a structure of commands, arguments & flags.

Commands represent actions, Args are things and Flags are modifiers for those actions.

The best applications read like sentences when used, and as a result, users
intuitively know how to interact with them.

The pattern to follow is
APPNAME VERB NOUN --ADJECTIVE
or
APPNAME COMMAND ARG --FLAG.

A few good real world examples may better illustrate this point.

In the following example, 'server' is a command, and 'port' is a flag:

hugo server --port=1313

In this command we are telling Git to clone the url bare.

git clone URL --bare

Commands Copy Chapter

Command is the central point of the application. Each interaction that
the application supports will be contained in a Command. A command can
have children commands and optionally run an action.

In the example above, 'server' is the command.

More about cobra.Command

Flags Copy Chapter

A flag is a way to modify the behavior of a command. Cobra supports
fully POSIX-compliant flags as well as the Go flag package.
A Cobra command can define flags that persist through to children commands
and flags that are only available to that command.

In the example above, 'port' is the flag.

Flag functionality is provided by the pflag
library, a fork of the flag standard library
which maintains the same interface while adding POSIX compliance.

Installing

Using Cobra is easy. First, use go get to install the latest version
of the library.

go get -u github.com/spf13/cobra@latest

Next, include Cobra in your application:

import "github.com/spf13/cobra"

Usage

cobra-cli is a command line program to generate cobra applications and command files.
It will bootstrap your application scaffolding to rapidly
develop a Cobra-based application. It is the easiest way to incorporate Cobra into your application.

It can be installed by running:

go install github.com/spf13/cobra-cli@latest

For complete details on using the Cobra-CLI generator, please read The Cobra Generator README

For complete details on using the Cobra library, please read The Cobra User Guide.

License

Cobra is released under the Apache 2.0 license. See LICENSE.txt

Overview (`vendor/github.com/spf13/pflag/README.md`) Copy Section

Description Copy Chapter

pflag is a drop-in replacement for Go's flag package, implementing
POSIX/GNU-style --flags.

pflag is compatible with the GNU extensions to the POSIX recommendations
for command-line options. For a more precise description, see the
"Command-line flag syntax" section below.

pflag is available under the same style of BSD license as the Go language,
which can be found in the LICENSE file.

Installation Copy Chapter

pflag is available using the standard go get command.

Install by running:

go get github.com/spf13/pflag

Run tests by running:

go test github.com/spf13/pflag

Usage Copy Chapter

pflag is a drop-in replacement of Go's native flag package. If you import
pflag under the name "flag" then all code should continue to function
with no changes.

import flag "github.com/spf13/pflag"

There is one exception to this: if you directly instantiate the Flag struct
there is one more field "Shorthand" that you will need to set.
Most code never instantiates this struct directly, and instead uses
functions such as String(), BoolVar(), and Var(), and is therefore
unaffected.

Define flags using flag.String(), Bool(), Int(), etc.

This declares an integer flag, -flagname, stored in the pointer ip, with type *int.

var ip *int = flag.Int("flagname", 1234, "help message for flagname")

If you like, you can bind the flag to a variable using the Var() functions.

var flagvar int
func init() {
    flag.IntVar(&flagvar, "flagname", 1234, "help message for flagname")
}

Or you can create custom flags that satisfy the Value interface (with
pointer receivers) and couple them to flag parsing by

flag.Var(&flagVal, "name", "help message for flagname")

For such flags, the default value is just the initial value of the variable.

After all flags are defined, call

flag.Parse()

to parse the command line into the defined flags.

Flags may then be used directly. If you're using the flags themselves,
they are all pointers; if you bind to variables, they're values.

fmt.Println("ip has value ", *ip)
fmt.Println("flagvar has value ", flagvar)

There are helper functions available to get the value stored in a Flag if you have a FlagSet but find
it difficult to keep up with all of the pointers in your code.
If you have a pflag.FlagSet with a flag called 'flagname' of type int you
can use GetInt() to get the int value. But notice that 'flagname' must exist
and it must be an int. GetString("flagname") will fail.

i, err := flagset.GetInt("flagname")

After parsing, the arguments after the flag are available as the
slice flag.Args() or individually as flag.Arg(i).
The arguments are indexed from 0 through flag.NArg()-1.

The pflag package also defines some new functions that are not in flag,
that give one-letter shorthands for flags. You can use these by appending
'P' to the name of any function that defines a flag.

var ip = flag.IntP("flagname", "f", 1234, "help message")
var flagvar bool
func init() {
	flag.BoolVarP(&flagvar, "boolname", "b", true, "help message")
}
flag.VarP(&flagVal, "varname", "v", "help message")

Shorthand letters can be used with single dashes on the command line.
Boolean shorthand flags can be combined with other shorthand flags.

The default set of command-line flags is controlled by
top-level functions. The FlagSet type allows one to define
independent sets of flags, such as to implement subcommands
in a command-line interface. The methods of FlagSet are
analogous to the top-level functions for the command-line
flag set.

Setting no option default values for flags Copy Chapter

After you create a flag it is possible to set the pflag.NoOptDefVal for
the given flag. Doing this changes the meaning of the flag slightly. If
a flag has a NoOptDefVal and the flag is set on the command line without
an option the flag will be set to the NoOptDefVal. For example given:

var ip = flag.IntP("flagname", "f", 1234, "help message")
flag.Lookup("flagname").NoOptDefVal = "4321"

Would result in something like

Command line flag syntax Copy Chapter

--flag    // boolean flags, or flags with no option default values
--flag x  // only on flags without a default value
--flag=x

Unlike the flag package, a single dash before an option means something
different than a double dash. Single dashes signify a series of shorthand
letters for flags. All but the last shorthand letter must be boolean flags
or a flag with a default value

// boolean or flags where the 'no option default value' is set
-f
-f=true
-abc
but
-b true is INVALID

// non-boolean and flags without a 'no option default value'
-n 1234
-n=1234
-n1234

// mixed
-abcs "hello"
-absd="hello"
-abcs1234

Flag parsing stops after the terminator "--". Unlike the flag package,
flags can be interspersed with arguments anywhere on the command line
before this terminator.

Integer flags accept 1234, 0664, 0x1234 and may be negative.
Boolean flags (in their long form) accept 1, 0, t, f, true, false,
TRUE, FALSE, True, False.
Duration flags accept any input valid for time.ParseDuration.

Mutating or "Normalizing" Flag names Copy Chapter

It is possible to set a custom flag name 'normalization function.' It allows flag names to be mutated both when created in the code and when used on the command line to some 'normalized' form. The 'normalized' form is used for comparison. Two examples of using the custom normalization func follow.

Example #1: You want -, _, and . in flags to compare the same. aka --my-flag == --my_flag == --my.flag

func wordSepNormalizeFunc(f *pflag.FlagSet, name string) pflag.NormalizedName {
	from := []string{"-", "_"}
	to := "."
	for _, sep := range from {
		name = strings.Replace(name, sep, to, -1)
	}
	return pflag.NormalizedName(name)
}

myFlagSet.SetNormalizeFunc(wordSepNormalizeFunc)

Example #2: You want to alias two flags. aka --old-flag-name == --new-flag-name

func aliasNormalizeFunc(f *pflag.FlagSet, name string) pflag.NormalizedName {
	switch name {
	case "old-flag-name":
		name = "new-flag-name"
		break
	}
	return pflag.NormalizedName(name)
}

myFlagSet.SetNormalizeFunc(aliasNormalizeFunc)

Deprecating a flag or its shorthand Copy Chapter

It is possible to deprecate a flag, or just its shorthand. Deprecating a flag/shorthand hides it from help text and prints a usage message when the deprecated flag/shorthand is used.

Example #1: You want to deprecate a flag named "badflag" as well as inform the users what flag they should use instead.

// deprecate a flag by specifying its name and a usage message
flags.MarkDeprecated("badflag", "please use --good-flag instead")

This hides "badflag" from help text, and prints Flag --badflag has been deprecated, please use --good-flag instead when "badflag" is used.

Example #2: You want to keep a flag name "noshorthandflag" but deprecate its shortname "n".

// deprecate a flag shorthand by specifying its flag name and a usage message
flags.MarkShorthandDeprecated("noshorthandflag", "please use --noshorthandflag only")

This hides the shortname "n" from help text, and prints Flag shorthand -n has been deprecated, please use --noshorthandflag only when the shorthand "n" is used.

Note that usage message is essential here, and it should not be empty.

Hidden flags Copy Chapter

It is possible to mark a flag as hidden, meaning it will still function as normal, however will not show up in usage/help text.

Example: You have a flag named "secretFlag" that you need for internal use only and don't want it showing up in help text, or for its usage text to be available.

// hide a flag by specifying its name
flags.MarkHidden("secretFlag")

Disable sorting of flags Copy Chapter

pflag allows you to disable sorting of flags for help and usage message.

Example:

flags.BoolP("verbose", "v", false, "verbose output")
flags.String("coolflag", "yeaah", "it's really cool flag")
flags.Int("usefulflag", 777, "sometimes it's very useful")
flags.SortFlags = false
flags.PrintDefaults()

Output:

-v, --verbose           verbose output
      --coolflag string   it's really cool flag (default "yeaah")
      --usefulflag int    sometimes it's very useful (default 777)

Supporting Go flags when using pflag Copy Chapter

In order to support flags defined using Go's flag package, they must be added to the pflag flagset. This is usually necessary
to support flags defined by third-party dependencies (e.g. golang/glog).

Example: You want to add the Go flags to the CommandLine flagset

import (
	goflag "flag"
	flag "github.com/spf13/pflag"
)

var ip *int = flag.Int("flagname", 1234, "help message for flagname")

func main() {
	flag.CommandLine.AddGoFlagSet(goflag.CommandLine)
	flag.Parse()
}

Using pflag with go test Copy Section

pflag does not parse the shorthand versions of go test's built-in flags (i.e., those starting with -test.).
For more context, see issues #63 and #238 for more details.

For example, if you use pflag in your TestMain function and call pflag.Parse() after defining your custom flags, running a test like this:

go test /your/tests -run ^YourTest -v --your-test-pflags

will result in the -v flag being ignored. This happens because of the way pflag handles flag parsing, skipping over go test's built-in shorthand flags.
To work around this, you can use the ParseSkippedFlags function, which ensures that go test's flags are parsed separately using the standard flag package.

Example: You want to parse go test flags that are otherwise ignore by pflag.Parse()

import (
	goflag "flag"
	flag "github.com/spf13/pflag"
)

var ip *int = flag.Int("flagname", 1234, "help message for flagname")

func main() {
	flag.CommandLine.AddGoFlagSet(goflag.CommandLine)
    flag.ParseSkippedFlags(os.Args[1:], goflag.CommandLine)
	flag.Parse()
}

More info Copy Chapter

You can see the full reference documentation of the pflag package
at godoc.org, or through go's standard documentation system by
running godoc -http=:6060 and browsing to
http://localhost:6060/pkg/github.com/spf13/pflag after
installation.

Overview (`vendor/go.opentelemetry.io/otel/attribute/README.md`) Copy Section

Attribute

Overview (`vendor/go.opentelemetry.io/otel/baggage/README.md`) Copy Section

Baggage

Overview (`vendor/go.opentelemetry.io/otel/codes/README.md`) Copy Section

Codes

Overview (`vendor/go.opentelemetry.io/otel/log/README.md`) Copy Section

Log API

Overview (`vendor/go.opentelemetry.io/otel/metric/README.md`) Copy Section

Metric API

Overview (`vendor/go.opentelemetry.io/otel/propagation/README.md`) Copy Section

Propagation

Overview (`vendor/go.opentelemetry.io/otel/sdk/README.md`) Copy Section

SDK

Overview (`vendor/go.opentelemetry.io/otel/trace/README.md`) Copy Section

Trace API

Overview (`vendor/go.yaml.in/yaml/v2/README.md`) Copy Section

YAML support for the Go language

Introduction Copy Chapter

The yaml package enables Go programs to comfortably encode and decode YAML
values. It was developed within Canonical as
part of the juju project, and is based on a
pure Go port of the well-known libyaml
C library to parse and generate YAML data quickly and reliably.

Compatibility Copy Chapter

The yaml package supports most of YAML 1.1 and 1.2, including support for
anchors, tags, map merging, etc. Multi-document unmarshalling is not yet
implemented, and base-60 floats from YAML 1.1 are purposefully not
supported since they're a poor design and are gone in YAML 1.2.

Installation and usage Copy Chapter

The import path for the package is go.yaml.in/yaml/v2.

To install it, run:

go get go.yaml.in/yaml/v2

API documentation Copy Chapter

See: https://pkg.go.dev/go.yaml.in/yaml/v2

API stability Copy Chapter

The package API for yaml v2 will remain stable as described in gopkg.in.

License Copy Chapter

The yaml package is licensed under the Apache License 2.0. Please see the LICENSE file for details.

Example Copy Chapter

package main

import (
        "fmt"
        "log"

        "go.yaml.in/yaml/v2"
)

var data = `
a: Easy!
b:
  c: 2
  d: [3, 4]
`

// Note: struct fields must be public in order for unmarshal to
// correctly populate the data.
type T struct {
        A string
        B struct {
                RenamedC int   `yaml:"c"`
                D        []int `yaml:",flow"`
        }
}

func main() {
        t := T{}
    
        err := yaml.Unmarshal([]byte(data), &t)
        if err != nil {
                log.Fatalf("error: %v", err)
        }
        fmt.Printf("--- t:\n%v\n\n", t)
    
        d, err := yaml.Marshal(&t)
        if err != nil {
                log.Fatalf("error: %v", err)
        }
        fmt.Printf("--- t dump:\n%s\n\n", string(d))
    
        m := make(map[interface{}]interface{})
    
        err = yaml.Unmarshal([]byte(data), &m)
        if err != nil {
                log.Fatalf("error: %v", err)
        }
        fmt.Printf("--- m:\n%v\n\n", m)
    
        d, err = yaml.Marshal(&m)
        if err != nil {
                log.Fatalf("error: %v", err)
        }
        fmt.Printf("--- m dump:\n%s\n\n", string(d))
}

This example will generate the following output:

--- t:
{Easy! {2 [3 4]}}

--- t dump:
a: Easy!
b:
  c: 2
  d: [3, 4]


--- m:
map[a:Easy! b:map[c:2 d:[3 4]]]

--- m dump:
a: Easy!
b:
  c: 2
  d:
  - 3
  - 4

Overview (`vendor/go.yaml.in/yaml/v3/README.md`) Copy Section

go.yaml.in/yaml

YAML Support for the Go Language

Introduction Copy Chapter

The yaml package enables Go programs to comfortably encode
and decode YAML values.

It was originally developed within Canonical as
part of the juju project, and is based on a pure Go
port of the well-known libyaml C library to
parse and generate YAML data quickly and reliably.

Project Status Copy Chapter

This project started as a fork of the extremely popular go-yaml
project, and is being maintained by the official YAML organization.

The YAML team took over ongoing maintenance and development of the project after
discussion with go-yaml's author, @niemeyer, following his decision to
label the project repository as "unmaintained" in April 2025.

We have put together a team of dedicated maintainers including representatives
of go-yaml's most important downstream projects.

We will strive to earn the trust of the various go-yaml forks to switch back to
this repository as their upstream.

Please contact us if you
would like to contribute or be involved.

Compatibility Copy Chapter

The yaml package supports most of YAML 1.2, but preserves some behavior from
1.1 for backwards compatibility.

Specifically, v3 of the yaml package:

• Supports YAML 1.1 bools (yes/no, on/off) as long as they are being
  decoded into a typed bool value.
  Otherwise they behave as a string.
  Booleans in YAML 1.2 are true/false only.
• Supports octals encoded and decoded as 0777 per YAML 1.1, rather than
  0o777 as specified in YAML 1.2, because most parsers still use the old
  format.
  Octals in the 0o777 format are supported though, so new files work.
• Does not support base-60 floats.
  These are gone from YAML 1.2, and were actually never supported by this
  package as it's clearly a poor choice.

Installation and Usage Copy Chapter

The import path for the package is go.yaml.in/yaml/v3.

To install it, run:

go get go.yaml.in/yaml/v3

API Documentation Copy Chapter

See: https://pkg.go.dev/go.yaml.in/yaml/v3

API Stability Copy Chapter

The package API for yaml v3 will remain stable as described in gopkg.in.

Example Copy Chapter

package main

import (
	"fmt"
	"log"

	"go.yaml.in/yaml/v3"
)

var data = `
a: Easy!
b:
  c: 2
  d: [3, 4]
`

// Note: struct fields must be public in order for unmarshal to
// correctly populate the data.
type T struct {
	A string
	B struct {
		RenamedC int   `yaml:"c"`
		D	[]int `yaml:",flow"`
	}
}

func main() {
	t := T{}

	err := yaml.Unmarshal([]byte(data), &t)
	if err != nil {
		log.Fatalf("error: %v", err)
	}
	fmt.Printf("--- t:\n%v\n\n", t)

	d, err := yaml.Marshal(&t)
	if err != nil {
		log.Fatalf("error: %v", err)
	}
	fmt.Printf("--- t dump:\n%s\n\n", string(d))

	m := make(map[interface{}]interface{})

	err = yaml.Unmarshal([]byte(data), &m)
	if err != nil {
		log.Fatalf("error: %v", err)
	}
	fmt.Printf("--- m:\n%v\n\n", m)

	d, err = yaml.Marshal(&m)
	if err != nil {
		log.Fatalf("error: %v", err)
	}
	fmt.Printf("--- m dump:\n%s\n\n", string(d))
}

This example will generate the following output:

--- t:
{Easy! {2 [3 4]}}

--- t dump:
a: Easy!
b:
  c: 2
  d: [3, 4]


--- m:
map[a:Easy! b:map[c:2 d:[3 4]]]

--- m dump:
a: Easy!
b:
  c: 2
  d:
  - 3
  - 4

License Copy Chapter

The yaml package is licensed under the MIT and Apache License 2.0 licenses.
Please see the LICENSE file for details.

Overview (`vendor/golang.org/x/oauth2/README.md`) Copy Section

OAuth2 for Go

oauth2 package contains a client implementation for OAuth 2.0 spec.

See pkg.go.dev for further documentation and examples.

• pkg.go.dev/golang.org/x/oauth2
• pkg.go.dev/golang.org/x/oauth2/google

Policy for new endpoints Copy Chapter

We no longer accept new provider-specific packages in this repo if all
they do is add a single endpoint variable. If you just want to add a
single endpoint, add it to the
pkg.go.dev/golang.org/x/oauth2/endpoints
package.

Report Issues / Send Patches Copy Chapter

The main issue tracker for the oauth2 repository is located at
https://github.com/golang/oauth2/issues.

This repository uses Gerrit for code changes. To learn how to submit changes to
this repository, see https://go.dev/doc/contribute.

The git repository is https://go.googlesource.com/oauth2.

Note:

• Excluding trivial changes, all contributions should be connected to an existing issue.
• API changes must go through the change proposal process before they can be accepted.
• The code owners are listed at dev.golang.org/owners.

Overview (`vendor/cloud.google.com/go/compute/metadata/README.md`) Copy Section

Compute API

This is a utility library for communicating with Google Cloud metadata service
on Google Cloud.

Install Copy Chapter

go get cloud.google.com/go/compute/metadata

Go Version Support Copy Chapter

See the Go Versions Supported
section in the root directory's README.

Contributing Copy Chapter

Contributions are welcome. Please, see the CONTRIBUTING
document for details.

Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms. See
Contributor Code of Conduct
for more information.

Overview (`vendor/github.com/alicebob/miniredis/v2/README.md`) Copy Section

Miniredis

Pure Go Redis test server, used in Go unittests.

Copy Chapter

Sometimes you want to test code which uses Redis, without making it a full-blown
integration test.
Miniredis implements (parts of) the Redis server, to be used in unittests. It
enables a simple, cheap, in-memory, Redis replacement, with a real TCP interface. Think of it as the Redis version of net/http/httptest.

It saves you from using mock code, and since the redis server lives in the
test process you can query for values directly, without going through the server
stack.

There are no dependencies on external binaries, so you can easily integrate it in automated build processes.

Be sure to import v2:

import "github.com/alicebob/miniredis/v2"

Commands Copy Chapter

Implemented commands:

• Connection (complete)
  • AUTH -- see RequireAuth()
  • ECHO
  • HELLO -- see RequireUserAuth()
  • PING
  • SELECT
  • SWAPDB
  • QUIT
• Key
  • COPY
  • DEL
  • EXISTS
  • EXPIRE
  • EXPIREAT
  • EXPIRETIME
  • KEYS
  • MOVE
  • PERSIST
  • PEXPIRE
  • PEXPIREAT
  • PEXPIRETIME
  • PTTL
  • RANDOMKEY -- see m.Seed(...)
  • RENAME
  • RENAMENX
  • SCAN
  • TOUCH
  • TTL
  • TYPE
  • UNLINK
• Transactions (complete)
  • DISCARD
  • EXEC
  • MULTI
  • UNWATCH
  • WATCH
• Server
  • DBSIZE
  • FLUSHALL
  • FLUSHDB
  • TIME -- returns time.Now() or value set by SetTime()
  • COMMAND -- partly
  • INFO -- partly, returns only "clients" section with one field "connected_clients"
• String keys (complete)
  • APPEND
  • BITCOUNT
  • BITOP
  • BITPOS
  • DECR
  • DECRBY
  • GET
  • GETBIT
  • GETRANGE
  • GETSET
  • GETDEL
  • GETEX
  • INCR
  • INCRBY
  • INCRBYFLOAT
  • MGET
  • MSET
  • MSETNX
  • PSETEX
  • SET
  • SETBIT
  • SETEX
  • SETNX
  • SETRANGE
  • STRLEN
• Hash keys (complete)
  • HDEL
  • HEXISTS
  • HGET
  • HGETALL
  • HINCRBY
  • HINCRBYFLOAT
  • HKEYS
  • HLEN
  • HMGET
  • HMSET
  • HRANDFIELD
  • HSET
  • HSETNX
  • HSTRLEN
  • HVALS
  • HSCAN
• List keys (complete)
  • BLPOP
  • BRPOP
  • BRPOPLPUSH
  • LINDEX
  • LINSERT
  • LLEN
  • LPOP
  • LPUSH
  • LPUSHX
  • LRANGE
  • LREM
  • LSET
  • LTRIM
  • RPOP
  • RPOPLPUSH
  • RPUSH
  • RPUSHX
  • LMOVE
  • BLMOVE
• Pub/Sub (complete)
  • PSUBSCRIBE
  • PUBLISH
  • PUBSUB
  • PUNSUBSCRIBE
  • SUBSCRIBE
  • UNSUBSCRIBE
• Set keys (complete)
  • SADD
  • SCARD
  • SDIFF
  • SDIFFSTORE
  • SINTER
  • SINTERSTORE
  • SINTERCARD
  • SISMEMBER
  • SMEMBERS
  • SMISMEMBER
  • SMOVE
  • SPOP -- see m.Seed(...)
  • SRANDMEMBER -- see m.Seed(...)
  • SREM
  • SSCAN
  • SUNION
  • SUNIONSTORE
• Sorted Set keys (complete)
  • ZADD
  • ZCARD
  • ZCOUNT
  • ZINCRBY
  • ZINTER
  • ZINTERSTORE
  • ZLEXCOUNT
  • ZPOPMIN
  • ZPOPMAX
  • ZRANDMEMBER
  • ZRANGE
  • ZRANGEBYLEX
  • ZRANGEBYSCORE
  • ZRANK
  • ZREM
  • ZREMRANGEBYLEX
  • ZREMRANGEBYRANK
  • ZREMRANGEBYSCORE
  • ZREVRANGE
  • ZREVRANGEBYLEX
  • ZREVRANGEBYSCORE
  • ZREVRANK
  • ZSCORE
  • ZUNION
  • ZUNIONSTORE
  • ZSCAN
• Stream keys
  • XACK
  • XADD
  • XAUTOCLAIM
  • XCLAIM
  • XDEL
  • XGROUP CREATE
  • XGROUP CREATECONSUMER
  • XGROUP DESTROY
  • XGROUP DELCONSUMER
  • XINFO STREAM -- partly
  • XINFO GROUPS
  • XINFO CONSUMERS -- partly
  • XLEN
  • XRANGE
  • XREAD
  • XREADGROUP
  • XREVRANGE
  • XPENDING
  • XTRIM
• Scripting
  • EVAL
  • EVALSHA
  • SCRIPT LOAD
  • SCRIPT EXISTS
  • SCRIPT FLUSH
• GEO
  • GEOADD
  • GEODIST
  • GEOHASH
  • GEOPOS
  • GEORADIUS
  • GEORADIUS_RO
  • GEORADIUSBYMEMBER
  • GEORADIUSBYMEMBER_RO
• Cluster
  • CLUSTER SLOTS
  • CLUSTER KEYSLOT
  • CLUSTER NODES
• HyperLogLog (complete)
  • PFADD
  • PFCOUNT
  • PFMERGE

TTLs, key expiration, and time Copy Chapter

Since miniredis is intended to be used in unittests TTLs don't decrease
automatically. You can use TTL() to get the TTL (as a time.Duration) of a
key. It will return 0 when no TTL is set.

m.FastForward(d) can be used to decrement all TTLs. All TTLs which become <=
0 will be removed.

EXPIREAT and PEXPIREAT values will be
converted to a duration. For that you can either set m.SetTime(t) to use that
time as the base for the (P)EXPIREAT conversion, or don't call SetTime(), in
which case time.Now() will be used.

SetTime() also sets the value returned by TIME, which defaults to time.Now().
It is not updated by FastForward, only by SetTime.

Randomness and Seed() Copy Chapter

Miniredis will use math/rand's global RNG for randomness unless a seed is
provided by calling m.Seed(...). If a seed is provided, then miniredis will
use its own RNG based on that seed.

Commands which use randomness are: RANDOMKEY, SPOP, and SRANDMEMBER.

Example Copy Chapter

import (
    ...
    "github.com/alicebob/miniredis/v2"
    ...
)

func TestSomething(t *testing.T) {
	s := miniredis.RunT(t)

	// Optionally set some keys your code expects:
	s.Set("foo", "bar")
	s.HSet("some", "other", "key")

	// Run your code and see if it behaves.
	// An example using the redigo library from "github.com/gomodule/redigo/redis":
	c, err := redis.Dial("tcp", s.Addr())
	_, err = c.Do("SET", "foo", "bar")

	// Optionally check values in redis...
	if got, err := s.Get("foo"); err != nil || got != "bar" {
		t.Error("'foo' has the wrong value")
	}
	// ... or use a helper for that:
	s.CheckGet(t, "foo", "bar")

	// TTL and expiration:
	s.Set("foo", "bar")
	s.SetTTL("foo", 10*time.Second)
	s.FastForward(11 * time.Second)
	if s.Exists("foo") {
		t.Fatal("'foo' should not have existed anymore")
	}
}

Not supported Copy Chapter

Commands which will probably not be implemented:

• CLUSTER (all)
  • CLUSTER *
  • READONLY
  • READWRITE
• Key
  • DUMP
  • MIGRATE
  • OBJECT
  • RESTORE
  • WAIT
• Scripting
  • FCALL / FCALL_RO *
  • FUNCTION *
  • SCRIPT DEBUG
  • SCRIPT KILL
• Server
  • BGSAVE
  • BGWRITEAOF
  • CLIENT *
  • CONFIG *
  • DEBUG *
  • LASTSAVE
  • MONITOR
  • ROLE
  • SAVE
  • SHUTDOWN
  • SLAVEOF
  • SLOWLOG
  • SYNC

&c. Copy Chapter

Integration tests are run against Redis 7.2.4. The ./integration subdir
compares miniredis against a real redis instance.

The Redis 6 RESP3 protocol is supported. If there are problems, please open
an issue.

If you want to test Redis Sentinel have a look at minisentinel.

A changelog is kept at CHANGELOG.md.

Overview (`vendor/github.com/cenkalti/backoff/v5/README.md`) Copy Section

Exponential Backoff [![GoDoc][godoc image]][godoc]

This is a Go port of the exponential backoff algorithm from Google's HTTP Client Library for Java.

Exponential backoff
is an algorithm that uses feedback to multiplicatively decrease the rate of some process,
in order to gradually find an acceptable rate.
The retries exponentially increase and stop increasing when a certain threshold is met.

Usage Copy Chapter

Import path is github.com/cenkalti/backoff/v5. Please note the version part at the end.

For most cases, use Retry function. See example_test.go for an example.

If you have specific needs, copy Retry function (from retry.go) into your code and modify it as needed.

Contributing Copy Chapter

• I would like to keep this library as small as possible.
• Please don't send a PR without opening an issue and discussing it first.
• If proposed change is not a common use case, I will probably not accept it.

Overview (`vendor/github.com/cespare/xxhash/v2/README.md`) Copy Section

xxhash

xxhash is a Go implementation of the 64-bit xxHash algorithm, XXH64. This is a
high-quality hashing algorithm that is much faster than anything in the Go
standard library.

This package provides a straightforward API:

func Sum64(b []byte) uint64
func Sum64String(s string) uint64
type Digest struct{ ... }
    func New() *Digest

The Digest type implements hash.Hash64. Its key methods are:

func (*Digest) Write([]byte) (int, error)
func (*Digest) WriteString(string) (int, error)
func (*Digest) Sum64() uint64

The package is written with optimized pure Go and also contains even faster
assembly implementations for amd64 and arm64. If desired, the purego build tag
opts into using the Go code even on those architectures.

Compatibility Copy Chapter

This package is in a module and the latest code is in version 2 of the module.
You need a version of Go with at least "minimal module compatibility" to use
github.com/cespare/xxhash/v2:

• 1.9.7+ for Go 1.9
• 1.10.3+ for Go 1.10
• Go 1.11 or later

I recommend using the latest release of Go.

Benchmarks Copy Chapter

Here are some quick benchmarks comparing the pure-Go and assembly
implementations of Sum64.

These numbers were generated on Ubuntu 20.04 with an Intel Xeon Platinum 8252C
CPU using the following commands under Go 1.19.2:

benchstat <(go test -tags purego -benchtime 500ms -count 15 -bench 'Sum64$')
benchstat <(go test -benchtime 500ms -count 15 -bench 'Sum64$')

Projects using this package Copy Chapter

• InfluxDB
• Prometheus
• VictoriaMetrics
• FreeCache
• FastCache
• Ristretto
• Badger

Overview (`vendor/github.com/go-jose/go-jose/v4/README.md`) Copy Section

Go JOSE

Package jose aims to provide an implementation of the Javascript Object Signing
and Encryption set of standards. This includes support for JSON Web Encryption,
JSON Web Signature, and JSON Web Token standards.

Overview Copy Chapter

The implementation follows the
JSON Web Encryption (RFC 7516),
JSON Web Signature (RFC 7515), and
JSON Web Token (RFC 7519) specifications.
Tables of supported algorithms are shown below. The library supports both
the compact and JWS/JWE JSON Serialization formats, and has optional support for
multiple recipients. It also comes with a small command-line utility
(jose-util)
for dealing with JOSE messages in a shell.

Note: We use a forked version of the encoding/json package from the Go
standard library which uses case-sensitive matching for member names (instead
of case-insensitive matching).
This is to avoid differences in interpretation of messages between go-jose and
libraries in other languages.

Versions Copy Section

The forthcoming Version 5 will be released with several breaking API changes,
and will require Golang's encoding/json/v2, which is currently requires
Go 1.25 built with GOEXPERIMENT=jsonv2.

Version 4 is the current stable version:

import "github.com/go-jose/go-jose/v4"

It supports at least the current and previous Golang release. Currently it
requires Golang 1.24.

Version 3 is only receiving critical security updates. Migration to Version 4 is recommended.

Versions 1 and 2 are obsolete, but can be found in the old repository, square/go-jose.

Supported algorithms Copy Section

See below for a table of supported algorithms. Algorithm identifiers match
the names in the JSON Web Algorithms
standard where possible. The Godoc reference has a list of constants.

^{1. Not supported in multi-recipient mode}

^{2. Only available in version 2 of the package}

Supported key types Copy Section

See below for a table of supported key types. These are understood by the
library, and can be passed to corresponding functions such as NewEncrypter or
NewSigner. Each of these keys can also be wrapped in a JWK if desired, which
allows attaching a key id.

^{1. Only available in version 2 or later of the package}

Examples Copy Chapter

Examples can be found in the Godoc
reference for this package. The
jose-util
subdirectory also contains a small command-line utility which might be useful
as an example as well.

Overview (`vendor/github.com/golang-jwt/jwt/v5/README.md`) Copy Section

jwt-go

A go (or 'golang' for search engine friendliness)
implementation of JSON Web
Tokens.

Starting with v4.0.0
this project adds Go module support, but maintains backward compatibility with
older v3.x.y tags and upstream github.com/dgrijalva/jwt-go. See the
MIGRATION_GUIDE.md for more information. Version
v5.0.0 introduces major improvements to the validation of tokens, but is not
entirely backward compatible.

After the original author of the library suggested migrating the maintenance
of jwt-go, a dedicated team of open source maintainers decided to clone the
existing library into this repository. See
dgrijalva/jwt-go#462 for a
detailed discussion on this topic.

SECURITY NOTICE: Some older versions of Go have a security issue in the
crypto/elliptic. The recommendation is to upgrade to at least 1.15 See issue
dgrijalva/jwt-go#216 for more
detail.

SECURITY NOTICE: It's important that you validate the alg presented is
what you
expect.
This library attempts to make it easy to do the right thing by requiring key
types to match the expected alg, but you should take the extra step to verify it in
your usage. See the examples provided.

Supported Go versions Copy Section

Our support of Go versions is aligned with Go's version release
policy. So we will support a major
version of Go until there are two newer major releases. We no longer support
building jwt-go with unsupported Go versions, as these contain security
vulnerabilities that will not be fixed.

What the heck is a JWT? Copy Chapter

JWT.io has a great introduction to JSON Web
Tokens.

In short, it's a signed JSON object that does something useful (for example,
authentication). It's commonly used for Bearer tokens in Oauth 2. A token is
made of three parts, separated by .'s. The first two parts are JSON objects,
that have been base64url
encoded. The last part is the signature, encoded the same way.

The first part is called the header. It contains the necessary information for
verifying the last part, the signature. For example, which encryption method
was used for signing and what key was used.

The part in the middle is the interesting bit. It's called the Claims and
contains the actual stuff you care about. Refer to RFC
7519 for information about
reserved keys and the proper way to add your own.

What's in the box? Copy Chapter

This library supports the parsing and verification as well as the generation and
signing of JWTs. Current supported signing algorithms are HMAC SHA, RSA,
RSA-PSS, and ECDSA, though hooks are present for adding your own.

Installation Guidelines Copy Chapter

1. To install the jwt package, you first need to have
   Go installed, then you can use the command
   below to add jwt-go as a dependency in your Go program.

go get -u github.com/golang-jwt/jwt/v5

2. Import it in your code:

import "github.com/golang-jwt/jwt/v5"

Usage Copy Chapter

A detailed usage guide, including how to sign and verify tokens can be found on
our documentation website.

Examples Copy Chapter

See the project documentation
for examples of usage:

• Simple example of parsing and validating a
  token
• Simple example of building and signing a
  token
• Directory of
  Examples

Compliance Copy Chapter

This library was last reviewed to comply with RFC
7519 dated May 2015 with a few
notable differences:

• In order to protect against accidental use of Unsecured
  JWTs, tokens using
  alg=none will only be accepted if the constant
  jwt.UnsafeAllowNoneSignatureType is provided as the key.

Project Status & Versioning Copy Chapter

This library is considered production ready. Feedback and feature requests are
appreciated. The API should be considered stable. There should be very few
backward-incompatible changes outside of major version updates (and only with
good reason).

This project uses Semantic Versioning 2.0.0. Accepted pull
requests will land on main. Periodically, versions will be tagged from
main. You can find all the releases on the project releases
page.

BREAKING CHANGES: A full list of breaking changes is available in
VERSION_HISTORY.md. See MIGRATION_GUIDE.md for more information on updating
your code.

Extensions Copy Chapter

This library publishes all the necessary components for adding your own signing
methods or key functions. Simply implement the SigningMethod interface and
register a factory method using RegisterSigningMethod or provide a
jwt.Keyfunc.

A common use case would be integrating with different 3rd party signature
providers, like key management services from various cloud providers or Hardware
Security Modules (HSMs) or to implement additional standards.

Disclaimer: Unless otherwise specified, these integrations are maintained by
third parties and should not be considered as a primary offer by any of the
mentioned cloud providers

More Copy Chapter

Go package documentation can be found on
pkg.go.dev. Additional
documentation can be found on our project
page.

The command line utility included in this project (cmd/jwt) provides a
straightforward example of token creation and parsing as well as a useful tool
for debugging your own integration. You'll also find several implementation
examples in the documentation.

golang-jwt incorporates a modified version
of the JWT logo, which is distributed under the terms of the MIT
License.

Overview (`vendor/github.com/klauspost/compress/fse/README.md`) Copy Section

Finite State Entropy

This package provides Finite State Entropy encoding and decoding.

Finite State Entropy (also referenced as tANS)
encoding provides a fast near-optimal symbol encoding/decoding
for byte blocks as implemented in zstandard.

This can be used for compressing input with a lot of similar input values to the smallest number of bytes.
This does not perform any multi-byte dictionary coding as LZ coders,
but it can be used as a secondary step to compressors (like Snappy) that does not do entropy encoding.

• Godoc documentation

News Copy Chapter

• Feb 2018: First implementation released. Consider this beta software for now.

Usage

This package provides a low level interface that allows to compress single independent blocks.

Each block is separate, and there is no built in integrity checks.
This means that the caller should keep track of block sizes and also do checksums if needed.

Compressing a block is done via the Compress function.
You must provide input and will receive the output and maybe an error.

These error values can be returned:

As can be seen above there are errors that will be returned even under normal operation so it is important to handle these.

To reduce allocations you can provide a Scratch object
that can be re-used for successive calls. Both compression and decompression accepts a Scratch object, and the same
object can be used for both.

Be aware, that when re-using a Scratch object that the output buffer is also re-used, so if you are still using this
you must set the Out field in the scratch to nil. The same buffer is used for compression and decompression output.

Decompressing is done by calling the Decompress function.
You must provide the output from the compression stage, at exactly the size you got back. If you receive an error back
your input was likely corrupted.

It is important to note that a successful decoding does not mean your output matches your original input.
There are no integrity checks, so relying on errors from the decompressor does not assure your data is valid.

For more detailed usage, see examples in the godoc documentation.

Performance

A lot of factors are affecting speed. Block sizes and compressibility of the material are primary factors.
All compression functions are currently only running on the calling goroutine so only one core will be used per block.

The compressor is significantly faster if symbols are kept as small as possible. The highest byte value of the input
is used to reduce some of the processing, so if all your input is above byte value 64 for instance, it may be
beneficial to transpose all your input values down by 64.

With moderate block sizes around 64k speed are typically 200MB/s per core for compression and
around 300MB/s decompression speed.

The same hardware typically does Huffman (deflate) encoding at 125MB/s and decompression at 100MB/s.

Plans

At one point, more internals will be exposed to facilitate more "expert" usage of the components.

A streaming interface is also likely to be implemented. Likely compatible with FSE stream format.

Contributing

Contributions are always welcome. Be aware that adding public functions will require good justification and breaking
changes will likely not be accepted. If in doubt open an issue before writing the PR.

Overview (`vendor/github.com/klauspost/compress/huff0/README.md`) Copy Section

Huff0 entropy compression

This package provides Huff0 encoding and decoding as used in zstd.

Huff0,
a Huffman codec designed for modern CPU, featuring OoO (Out of Order) operations on multiple ALU
(Arithmetic Logic Unit), achieving extremely fast compression and decompression speeds.

This can be used for compressing input with a lot of similar input values to the smallest number of bytes.
This does not perform any multi-byte dictionary coding as LZ coders,
but it can be used as a secondary step to compressors (like Snappy) that does not do entropy encoding.

• Godoc documentation

News Copy Chapter

This is used as part of the zstandard compression and decompression package.

This ensures that most functionality is well tested.

Usage

This package provides a low level interface that allows to compress single independent blocks.

Each block is separate, and there is no built in integrity checks.
This means that the caller should keep track of block sizes and also do checksums if needed.

Compressing a block is done via the Compress1X and
Compress4X functions.
You must provide input and will receive the output and maybe an error.

These error values can be returned:

As can be seen above some of there are errors that will be returned even under normal operation so it is important to handle these.

To reduce allocations you can provide a Scratch object
that can be re-used for successive calls. Both compression and decompression accepts a Scratch object, and the same
object can be used for both.

Be aware, that when re-using a Scratch object that the output buffer is also re-used, so if you are still using this
you must set the Out field in the scratch to nil. The same buffer is used for compression and decompression output.

The Scratch object will retain state that allows to re-use previous tables for encoding and decoding.

Tables and re-use Copy Chapter

Huff0 allows for reusing tables from the previous block to save space if that is expected to give better/faster results.

The Scratch object allows you to set a ReusePolicy
that controls this behaviour. See the documentation for details. This can be altered between each block.

Do however note that this information is not stored in the output block and it is up to the users of the package to
record whether ReadTable should be called,
based on the boolean reported back from the CompressXX call.

If you want to store the table separate from the data, you can access them as OutData and OutTable on the
Scratch object.

Decompressing Copy Chapter

The first part of decoding is to initialize the decoding table through ReadTable.
This will initialize the decoding tables.
You can supply the complete block to ReadTable and it will return the data part of the block
which can be given to the decompressor.

Decompressing is done by calling the Decompress1X
or Decompress4X function.

For concurrently decompressing content with a fixed table a stateless Decoder can be requested which will remain correct as long as the scratch is unchanged. The capacity of the provided slice indicates the expected output size.

You must provide the output from the compression stage, at exactly the size you got back. If you receive an error back
your input was likely corrupted.

It is important to note that a successful decoding does not mean your output matches your original input.
There are no integrity checks, so relying on errors from the decompressor does not assure your data is valid.

Contributing

Contributions are always welcome. Be aware that adding public functions will require good justification and breaking
changes will likely not be accepted. If in doubt open an issue before writing the PR.

Overview (`vendor/github.com/klauspost/compress/zstd/README.md`) Copy Section

zstd

Zstandard is a real-time compression algorithm, providing high compression ratios.
It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder.
A high performance compression algorithm is implemented. For now focused on speed.

This package provides compression to and decompression of Zstandard content.

This package is pure Go. Use noasm and nounsafe to disable relevant features.

The zstd package is provided as open source software using a Go standard license.

Currently the package is heavily optimized for 64 bit processors and will be significantly slower on 32 bit processors.

For seekable zstd streams, see this excellent package.

Installation Copy Chapter

Install using go get -u github.com/klauspost/compress. The package is located in github.com/klauspost/compress/zstd.

Compressor Copy Chapter

Status: Copy Section

STABLE - there may always be subtle bugs, a wide variety of content has been tested and the library is actively
used by several projects. This library is being fuzz-tested for all updates.

There may still be specific combinations of data types/size/settings that could lead to edge cases,
so as always, testing is recommended.

For now, a high speed (fastest) and medium-fast (default) compressor has been implemented.

• The "Fastest" compression ratio is roughly equivalent to zstd level 1.
• The "Default" compression ratio is roughly equivalent to zstd level 3 (default).
• The "Better" compression ratio is roughly equivalent to zstd level 7.
• The "Best" compression ratio is roughly equivalent to zstd level 11.

In terms of speed, it is typically 2x as fast as the stdlib deflate/gzip in its fastest mode.
The compression ratio compared to stdlib is around level 3, but usually 3x as fast.

Usage Copy Section

An Encoder can be used for either compressing a stream via the
io.WriteCloser interface supported by the Encoder or as multiple independent
tasks via the EncodeAll function.
Smaller encodes are encouraged to use the EncodeAll function.
Use NewWriter to create a new instance that can be used for both.

To create a writer with default options, do like this:

// Compress input to output.
func Compress(in io.Reader, out io.Writer) error {
    enc, err := zstd.NewWriter(out)
    if err != nil {
        return err
    }
    _, err = io.Copy(enc, in)
    if err != nil {
        enc.Close()
        return err
    }
    return enc.Close()
}

Now you can encode by writing data to enc. The output will be finished writing when Close() is called.
Even if your encode fails, you should still call Close() to release any resources that may be held up.

The above is fine for big encodes. However, whenever possible try to reuse the writer.

To reuse the encoder, you can use the Reset(io.Writer) function to change to another output.
This will allow the encoder to reuse all resources and avoid wasteful allocations.

Currently stream encoding has 'light' concurrency, meaning up to 2 goroutines can be working on part
of a stream. This is independent of the WithEncoderConcurrency(n), but that is likely to change
in the future. So if you want to limit concurrency for future updates, specify the concurrency
you would like.

If you would like stream encoding to be done without spawning async goroutines, use WithEncoderConcurrency(1)
which will compress input as each block is completed, blocking on writes until each has completed.

You can specify your desired compression level using WithEncoderLevel() option. Currently only pre-defined
compression settings can be specified.

Future Compatibility Guarantees

This will be an evolving project. When using this package it is important to note that both the compression efficiency and speed may change.

The goal will be to keep the default efficiency at the default zstd (level 3).
However the encoding should never be assumed to remain the same,
and you should not use hashes of compressed output for similarity checks.

The Encoder can be assumed to produce the same output from the exact same code version.
However, the may be modes in the future that break this,
although they will not be enabled without an explicit option.

This encoder is not designed to (and will probably never) output the exact same bitstream as the reference encoder.

Also note, that the cgo decompressor currently does not report all errors on invalid input,
omits error checks, ignores checksums
and seems to ignore concatenated streams, even though it is part of the spec.

Blocks

For compressing small blocks, the returned encoder has a function called EncodeAll(src, dst []byte) []byte.

EncodeAll will encode all input in src and append it to dst.
This function can be called concurrently.
Each call will only run on a same goroutine as the caller.

Encoded blocks can be concatenated and the result will be the combined input stream.
Data compressed with EncodeAll can be decoded with the Decoder, using either a stream or DecodeAll.

Especially when encoding blocks you should take special care to reuse the encoder.
This will effectively make it run without allocations after a warmup period.
To make it run completely without allocations, supply a destination buffer with space for all content.

import "github.com/klauspost/compress/zstd"

// Create a writer that caches compressors.
// For this operation type we supply a nil Reader.
var encoder, _ = zstd.NewWriter(nil)

// Compress a buffer. 
// If you have a destination buffer, the allocation in the call can also be eliminated.
func Compress(src []byte) []byte {
    return encoder.EncodeAll(src, make([]byte, 0, len(src)))
}

You can control the maximum number of concurrent encodes using the WithEncoderConcurrency(n)
option when creating the writer.

Using the Encoder for both a stream and individual blocks concurrently is safe.

Performance Copy Section

I have collected some speed examples to compare speed and compression against other compressors.

• file is the input file.
• out is the compressor used. zskp is this package. zstd is the Datadog cgo library. gzstd/gzkp is gzip standard and this library.
• level is the compression level used. For zskp level 1 is "fastest", level 2 is "default"; 3 is "better", 4 is "best".
• insize/outsize is the input/output size.
• millis is the number of milliseconds used for compression.
• mb/s is megabytes (2^20 bytes) per second.

/* Detailed source-code truncated for AI context efficiency. */

Decompressor Copy Chapter

Status: STABLE - there may still be subtle bugs, but a wide variety of content has been tested.

This library is being continuously fuzz-tested,
kindly supplied by fuzzit.dev.
The main purpose of the fuzz testing is to ensure that it is not possible to crash the decoder,
or run it past its limits with ANY input provided.

Usage Copy Section

The package has been designed for two main usages, big streams of data and smaller in-memory buffers.
There are two main usages of the package for these. Both of them are accessed by creating a Decoder.

For streaming use a simple setup could look like this:

import "github.com/klauspost/compress/zstd"

func Decompress(in io.Reader, out io.Writer) error {
    d, err := zstd.NewReader(in)
    if err != nil {
        return err
    }
    defer d.Close()
    
    // Copy content...
    _, err = io.Copy(out, d)
    return err
}

It is important to use the "Close" function when you no longer need the Reader to stop running goroutines,
when running with default settings.
Goroutines will exit once an error has been returned, including io.EOF at the end of a stream.

Streams are decoded concurrently in 4 asynchronous stages to give the best possible throughput.
However, if you prefer synchronous decompression, use WithDecoderConcurrency(1) which will decompress data
as it is being requested only.

For decoding buffers, it could look something like this:

import "github.com/klauspost/compress/zstd"

// Create a reader that caches decompressors.
// For this operation type we supply a nil Reader.
var decoder, _ = zstd.NewReader(nil, zstd.WithDecoderConcurrency(0))

// Decompress a buffer. We don't supply a destination buffer,
// so it will be allocated by the decoder.
func Decompress(src []byte) ([]byte, error) {
    return decoder.DecodeAll(src, nil)
}

Both of these cases should provide the functionality needed.
The decoder can be used for concurrent decompression of multiple buffers.
By default 4 decompressors will be created.

It will only allow a certain number of concurrent operations to run.
To tweak that yourself use the WithDecoderConcurrency(n) option when creating the decoder.
It is possible to use WithDecoderConcurrency(0) to create GOMAXPROCS decoders.

Dictionaries Copy Section

Data compressed with dictionaries can be decompressed.

Dictionaries are added individually to Decoders.
Dictionaries are generated by the zstd --train command and contains an initial state for the decoder.
To add a dictionary use the WithDecoderDicts(dicts ...[]byte) option with the dictionary data.
Several dictionaries can be added at once.

The dictionary will be used automatically for the data that specifies them.
A re-used Decoder will still contain the dictionaries registered.

When registering multiple dictionaries with the same ID, the last one will be used.

It is possible to use dictionaries when compressing data.

To enable a dictionary use WithEncoderDict(dict []byte). Here only one dictionary will be used
and it will likely be used even if it doesn't improve compression.

The used dictionary must be used to decompress the content.

For any real gains, the dictionary should be built with similar data.
If an unsuitable dictionary is used the output may be slightly larger than using no dictionary.
Use the zstd commandline tool to build a dictionary from sample data.
For information see zstd dictionary information.

For now there is a fixed startup performance penalty for compressing content with dictionaries.
This will likely be improved over time. Just be aware to test performance when implementing.

Allocation-less operation Copy Section

The decoder has been designed to operate without allocations after a warmup.

This means that you should store the decoder for best performance.
To re-use a stream decoder, use the Reset(r io.Reader) error to switch to another stream.
A decoder can safely be re-used even if the previous stream failed.

To release the resources, you must call the Close() function on a decoder.
After this it can no longer be reused, but all running goroutines will be stopped.
So you must use this if you will no longer need the Reader.

For decompressing smaller buffers a single decoder can be used.
When decoding buffers, you can supply a destination slice with length 0 and your expected capacity.
In this case no unneeded allocations should be made.

Concurrency Copy Section

The buffer decoder does everything on the same goroutine and does nothing concurrently.
It can however decode several buffers concurrently. Use WithDecoderConcurrency(n) to limit that.

The stream decoder will create goroutines that:

1. Reads input and splits the input into blocks.
2. Decompression of literals.
3. Decompression of sequences.
4. Reconstruction of output stream.

So effectively this also means the decoder will "read ahead" and prepare data to always be available for output.

The concurrency level will, for streams, determine how many blocks ahead the compression will start.

Since "blocks" are quite dependent on the output of the previous block stream decoding will only have limited concurrency.

In practice this means that concurrency is often limited to utilizing about 3 cores effectively.

Benchmarks Copy Section

The first two are streaming decodes and the last are smaller inputs.

Running on AMD Ryzen 9 3950X 16-Core Processor. AMD64 assembly used.

/* Detailed source-code truncated for AI context efficiency. */

This reflects the performance around May 2022, but this may be out of date.

Zstd inside ZIP files Copy Chapter

It is possible to use zstandard to compress individual files inside zip archives.
While this isn't widely supported it can be useful for internal files.

To support the compression and decompression of these files you must register a compressor and decompressor.

It is highly recommended registering the (de)compressors on individual zip Reader/Writer and NOT
use the global registration functions. The main reason for this is that 2 registrations from
different packages will result in a panic.

It is a good idea to only have a single compressor and decompressor, since they can be used for multiple zip
files concurrently, and using a single instance will allow reusing some resources.

See this example for
how to compress and decompress files inside zip archives.

Contributions

Contributions are always welcome.
For new features/fixes, remember to add tests and for performance enhancements include benchmarks.

For general feedback and experience reports, feel free to open an issue or write me on Twitter.

This package includes the excellent github.com/cespare/xxhash package Copyright (c) 2016 Caleb Spare.

Overview (`vendor/github.com/prometheus/client_golang/prometheus/README.md`) Copy Section

See .

Overview (`vendor/github.com/redis/go-redis/v9/README.md`) Copy Section

Redis client for Go

go-redis is brought to you by :star: uptrace/uptrace.
Uptrace is an open-source APM tool that supports distributed tracing, metrics, and logs. You can
use it to monitor applications and set up automatic alerts to receive notifications via email,
Slack, Telegram, and others.

See OpenTelemetry example which
demonstrates how you can use Uptrace to monitor go-redis.

How do I Redis? Copy Chapter

Learn for free at Redis University

Build faster with the Redis Launchpad

Try the Redis Cloud

Dive in developer tutorials

Join the Redis community

Work at Redis

Documentation Copy Chapter

• English
• 简体中文

Resources Copy Chapter

• Discussions
• Chat
• Reference
• Examples

Ecosystem Copy Chapter

• Redis Mock
• Distributed Locks
• Redis Cache
• Rate limiting

This client also works with Kvrocks, a distributed
key value NoSQL database that uses RocksDB as storage engine and is compatible with Redis protocol.

Features Copy Chapter

• Redis commands except QUIT and SYNC.
• Automatic connection pooling.
• Pub/Sub.
• Pipelines and transactions.
• Scripting.
• Redis Sentinel.
• Redis Cluster.
• Redis Ring.
• Redis Performance Monitoring.
• Redis Probabilistic [RedisStack]

Installation Copy Chapter

go-redis supports 2 last Go versions and requires a Go version with
modules support. So make sure to initialize a Go
module:

go mod init github.com/my/repo

Then install go-redis/v9:

go get github.com/redis/go-redis/v9

Quickstart Copy Chapter

import (
    "context"
    "fmt"

    "github.com/redis/go-redis/v9"
)

var ctx = context.Background()

func ExampleClient() {
    rdb := redis.NewClient(&redis.Options{
        Addr:     "localhost:6379",
        Password: "", // no password set
        DB:       0,  // use default DB
    })

    err := rdb.Set(ctx, "key", "value", 0).Err()
    if err != nil {
        panic(err)
    }

    val, err := rdb.Get(ctx, "key").Result()
    if err != nil {
        panic(err)
    }
    fmt.Println("key", val)

    val2, err := rdb.Get(ctx, "key2").Result()
    if err == redis.Nil {
        fmt.Println("key2 does not exist")
    } else if err != nil {
        panic(err)
    } else {
        fmt.Println("key2", val2)
    }
    // Output: key value
    // key2 does not exist
}

The above can be modified to specify the version of the RESP protocol by adding the protocol
option to the Options struct:

rdb := redis.NewClient(&redis.Options{
        Addr:     "localhost:6379",
        Password: "", // no password set
        DB:       0,  // use default DB
        Protocol: 3, // specify 2 for RESP 2 or 3 for RESP 3
    })

Connecting via a redis url Copy Section

go-redis also supports connecting via the
redis uri specification.
The example below demonstrates how the connection can easily be configured using a string, adhering
to this specification.

import (
    "github.com/redis/go-redis/v9"
)

func ExampleClient() *redis.Client {
    url := "redis://user:password@localhost:6379/0?protocol=3"
    opts, err := redis.ParseURL(url)
    if err != nil {
        panic(err)
    }

    return redis.NewClient(opts)
}

Advanced Configuration Copy Section

go-redis supports extending the client identification phase to allow projects to send their own custom client identification.

Default Client Identification

By default, go-redis automatically sends the client library name and version during the connection process. This feature is available in redis-server as of version 7.2. As a result, the command is "fire and forget", meaning it should fail silently, in the case that the redis server does not support this feature.

Disabling Identity Verification

When connection identity verification is not required or needs to be explicitly disabled, a DisableIdentity configuration option exists.
Initially there was a typo and the option was named DisableIndentity instead of DisableIdentity. The misspelled option is marked as Deprecated and will be removed in V10 of this library.
Although both options will work at the moment, the correct option is DisableIdentity. The deprecated option will be removed in V10 of this library, so please use the correct option name to avoid any issues.

To disable verification, set the DisableIdentity option to true in the Redis client options:

rdb := redis.NewClient(&redis.Options{
    Addr:            "localhost:6379",
    Password:        "",
    DB:              0,
    DisableIdentity: true, // Disable set-info on connect
})

Unstable RESP3 Structures for RediSearch Commands

When integrating Redis with application functionalities using RESP3, it's important to note that some response structures aren't final yet. This is especially true for more complex structures like search and query results. We recommend using RESP2 when using the search and query capabilities, but we plan to stabilize the RESP3-based API-s in the coming versions. You can find more guidance in the upcoming release notes.

To enable unstable RESP3, set the option in your client configuration:

redis.NewClient(&redis.Options{
			UnstableResp3: true,
		})

Note: When UnstableResp3 mode is enabled, it's necessary to use RawResult() and RawVal() to retrieve a raw data.
Since, raw response is the only option for unstable search commands Val() and Result() calls wouldn't have any affect on them:

res1, err := client.FTSearchWithArgs(ctx, "txt", "foo bar", &redis.FTSearchOptions{}).RawResult()
val1 := client.FTSearchWithArgs(ctx, "txt", "foo bar", &redis.FTSearchOptions{}).RawVal()

Contributing Copy Chapter

Please see out contributing guidelines to help us improve this library!

Look and feel Copy Chapter

Some corner cases:

// SET key value EX 10 NX
set, err := rdb.SetNX(ctx, "key", "value", 10*time.Second).Result()

// SET key value keepttl NX
set, err := rdb.SetNX(ctx, "key", "value", redis.KeepTTL).Result()

// SORT list LIMIT 0 2 ASC
vals, err := rdb.Sort(ctx, "list", &redis.Sort{Offset: 0, Count: 2, Order: "ASC"}).Result()

// ZRANGEBYSCORE zset -inf +inf WITHSCORES LIMIT 0 2
vals, err := rdb.ZRangeByScoreWithScores(ctx, "zset", &redis.ZRangeBy{
    Min: "-inf",
    Max: "+inf",
    Offset: 0,
    Count: 2,
}).Result()

// ZINTERSTORE out 2 zset1 zset2 WEIGHTS 2 3 AGGREGATE SUM
vals, err := rdb.ZInterStore(ctx, "out", &redis.ZStore{
    Keys: []string{"zset1", "zset2"},
    Weights: []int64{2, 3}
}).Result()

// EVAL "return {KEYS[1],ARGV[1]}" 1 "key" "hello"
vals, err := rdb.Eval(ctx, "return {KEYS[1],ARGV[1]}", []string{"key"}, "hello").Result()

// custom command
res, err := rdb.Do(ctx, "set", "key", "value").Result()

Run the test Copy Chapter

go-redis will start a redis-server and run the test cases.

The paths of redis-server bin file and redis config file are defined in main_test.go:

var (
	redisServerBin, _  = filepath.Abs(filepath.Join("testdata", "redis", "src", "redis-server"))
	redisServerConf, _ = filepath.Abs(filepath.Join("testdata", "redis", "redis.conf"))
)

For local testing, you can change the variables to refer to your local files, or create a soft link
to the corresponding folder for redis-server and copy the config file to testdata/redis/:

ln -s /usr/bin/redis-server ./go-redis/testdata/redis/src
cp ./go-redis/testdata/redis.conf ./go-redis/testdata/redis/

Lastly, run:

go test

Another option is to run your specific tests with an already running redis. The example below, tests
against a redis running on port 9999.:

REDIS_PORT=9999 go test <your options>

See also Copy Chapter

• Golang ORM for PostgreSQL, MySQL, MSSQL, and SQLite
• Golang PostgreSQL
• Golang HTTP router
• Golang ClickHouse ORM

Contributors Copy Chapter

Thanks to all the people who already contributed!

Overview (`vendor/go.opentelemetry.io/contrib/detectors/gcp/README.md`) Copy Section

GCP Resource detector

The GCP resource detector supports detecting resources on:

• Google Compute Engine (GCE)
• Google Kubernetes Engine (GKE)
• Google App Engine (GAE)
• Cloud Run
• Cloud Run jobs
• Cloud Functions

Usage Copy Chapter

ctx := context.Background()
// Detect your resources
res, err := resource.New(ctx,
    // Use the GCP resource detector!
    resource.WithDetectors(gcp.NewDetector()),
    // Keep the default detectors
    resource.WithTelemetrySDK(),
    // Add your own custom attributes to identify your application
    resource.WithAttributes(
        semconv.ServiceNameKey.String("my-application"),
        semconv.ServiceNamespaceKey.String("my-company-frontend-team"),
    ),
)
if err != nil {
    // Handle err
}
// Use the resource in your tracerprovider (or meterprovider)
tp := trace.NewTracerProvider(
    // ... other options
    trace.WithResource(res),
)

Setting Kubernetes attributes Copy Chapter

Previous iterations of GCP resource detection attempted to detect
container.name, k8s.pod.name and k8s.namespace.name. When using this detector,
you should use this in your Pod Spec to set these using
OTEL_RESOURCE_ATTRIBUTES:

env:
- name: POD_NAME
  valueFrom:
    fieldRef:
      fieldPath: metadata.name
- name: NAMESPACE_NAME
  valueFrom:
    fieldRef:
      fieldPath: metadata.namespace
- name: CONTAINER_NAME
  value: my-container-name
- name: OTEL_RESOURCE_ATTRIBUTES
  value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME),k8s.container.name=$(CONTAINER_NAME)

To have a detector unpack the OTEL_RESOURCE_ATTRIBUTES envvar, use the WithFromEnv option:

...
// Detect your resources
res, err := resource.New(ctx,
    resource.WithDetectors(gcp.NewDetector()),
    resource.WithTelemetrySDK(),
    resource.WithFromEnv(), // unpacks OTEL_RESOURCE_ATTRIBUTES
    // Add your own custom attributes to identify your application
    resource.WithAttributes(
        semconv.ServiceNameKey.String("my-application"),
        semconv.ServiceNamespaceKey.String("my-company-frontend-team"),
    ),
)
...

Overview (`vendor/go.opentelemetry.io/otel/exporters/prometheus/README.md`) Copy Section

Prometheus Exporter

Overview (`vendor/go.opentelemetry.io/otel/log/embedded/README.md`) Copy Section

Log Embedded

Overview (`vendor/go.opentelemetry.io/otel/log/noop/README.md`) Copy Section

Log Noop

Overview (`vendor/go.opentelemetry.io/otel/metric/embedded/README.md`) Copy Section

Metric Embedded

Overview (`vendor/go.opentelemetry.io/otel/metric/noop/README.md`) Copy Section

Metric Noop

Overview (`vendor/go.opentelemetry.io/otel/sdk/instrumentation/README.md`) Copy Section

SDK Instrumentation

Overview (`vendor/go.opentelemetry.io/otel/sdk/log/README.md`) Copy Section

Log SDK

Overview (`vendor/go.opentelemetry.io/otel/sdk/metric/README.md`) Copy Section

Metric SDK

Overview (`vendor/go.opentelemetry.io/otel/sdk/resource/README.md`) Copy Section

SDK Resource

Overview (`vendor/go.opentelemetry.io/otel/sdk/trace/README.md`) Copy Section

SDK Trace

Overview (`vendor/go.opentelemetry.io/otel/semconv/v1.10.0/README.md`) Copy Section

Semconv v1.10.0

Overview (`vendor/go.opentelemetry.io/otel/semconv/v1.12.0/README.md`) Copy Section

Semconv v1.12.0

Overview (`vendor/go.opentelemetry.io/otel/semconv/v1.24.0/README.md`) Copy Section

Semconv v1.24.0

Overview (`vendor/go.opentelemetry.io/otel/semconv/v1.30.0/README.md`) Copy Section

Semconv v1.30.0

Overview (`vendor/go.opentelemetry.io/otel/semconv/v1.37.0/README.md`) Copy Section

Semconv v1.37.0

Overview (`vendor/go.opentelemetry.io/otel/semconv/v1.4.0/README.md`) Copy Section

Semconv v1.4.0

Overview (`vendor/go.opentelemetry.io/otel/semconv/v1.40.0/README.md`) Copy Section

Semconv v1.40.0

Overview (`vendor/go.opentelemetry.io/otel/semconv/v1.41.0/README.md`) Copy Section

Semconv v1.41.0

Overview (`vendor/go.opentelemetry.io/otel/semconv/v1.43.0/README.md`) Copy Section

Semconv v1.43.0

Overview (`vendor/go.opentelemetry.io/otel/trace/embedded/README.md`) Copy Section

Trace Embedded

Overview (`vendor/go.opentelemetry.io/otel/trace/noop/README.md`) Copy Section

Trace Noop

Overview (`vendor/golang.org/x/net/http2/README.md`) Copy Section

This package (golang.org/x/net/http2) is the original source of truth
of the Go HTTP/2 implementation.

As of Go 1.27, the source of truth has moved to the standard library
package net/http/internal/http2.
All new feature development should happen in that package.
Only critical bug fixes and security fixes will be backported to x/net.

The x/net package contains two implementations of the HTTP/2 transport and server:

The original implementation (no longer the source of truth).

A reimplementation of the x/net/http2 APIs in terms of net/http.
This is called "the wrapping implementation", since it wraps net/http.

The original implementation is used when the Go version is less than 1.27.

The wrapping implementation is used when the Go version is at least 1.27.
The build tag "http2legacy" may be set to use the original implementation.

Overview (`vendor/golang.org/x/sys/unix/README.md`) Copy Section

Building `sys/unix`

The sys/unix package provides access to the raw system call interface of the
underlying operating system. See: https://godoc.org/golang.org/x/sys/unix

Porting Go to a new architecture/OS combination or adding syscalls, types, or
constants to an existing architecture/OS pair requires some manual effort;
however, there are tools that automate much of the process.

Build Systems Copy Chapter

There are currently two ways we generate the necessary files. We are currently
migrating the build system to use containers so the builds are reproducible.
This is being done on an OS-by-OS basis. Please update this documentation as
components of the build system change.

Old Build System (currently for `GOOS != "linux"`) Copy Section

The old build system generates the Go files based on the C header files
present on your system. This means that files
for a given GOOS/GOARCH pair must be generated on a system with that OS and
architecture. This also means that the generated code can differ from system
to system, based on differences in the header files.

To avoid this, if you are using the old build system, only generate the Go
files on an installation with unmodified header files. It is also important to
keep track of which version of the OS the files were generated from (ex.
Darwin 14 vs Darwin 15). This makes it easier to track the progress of changes
and have each OS upgrade correspond to a single change.

To build the files for your current OS and architecture, make sure GOOS and
GOARCH are set correctly and run mkall.sh. This will generate the files for
your specific system. Running mkall.sh -n shows the commands that will be run.

Requirements: bash, go

New Build System (currently for `GOOS == "linux"`) Copy Section

The new build system uses a Docker container to generate the go files directly
from source checkouts of the kernel and various system libraries. This means
that on any platform that supports Docker, all the files using the new build
system can be generated at once, and generated files will not change based on
what the person running the scripts has installed on their computer.

The OS specific files for the new build system are located in the ${GOOS}
directory, and the build is coordinated by the ${GOOS}/mkall.go program. When
the kernel or system library updates, modify the Dockerfile at
${GOOS}/Dockerfile to checkout the new release of the source.

To build all the files under the new build system, you must be on an amd64/Linux
system and have your GOOS and GOARCH set accordingly. Running mkall.sh will
then generate all of the files for all of the GOOS/GOARCH pairs in the new build
system. Running mkall.sh -n shows the commands that will be run.

Requirements: bash, go, docker

Component files Copy Chapter

This section describes the various files used in the code generation process.
It also contains instructions on how to modify these files to add a new
architecture/OS or to add additional syscalls, types, or constants. Note that
if you are using the new build system, the scripts/programs cannot be called normally.
They must be called from within the docker container.

asm files Copy Section

The hand-written assembly file at asm_${GOOS}_${GOARCH}.s implements system
call dispatch. There are three entry points:

func Syscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr)
  func Syscall6(trap, a1, a2, a3, a4, a5, a6 uintptr) (r1, r2, err uintptr)
  func RawSyscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr)

The first and second are the standard ones; they differ only in how many
arguments can be passed to the kernel. The third is for low-level use by the
ForkExec wrapper. Unlike the first two, it does not call into the scheduler to
let it know that a system call is running.

When porting Go to a new architecture/OS, this file must be implemented for
each GOOS/GOARCH pair.

mksysnum Copy Section

Mksysnum is a Go program located at ${GOOS}/mksysnum.go (or mksysnum_${GOOS}.go
for the old system). This program takes in a list of header files containing the
syscall number declarations and parses them to produce the corresponding list of
Go numeric constants. See zsysnum_${GOOS}_${GOARCH}.go for the generated
constants.

Adding new syscall numbers is mostly done by running the build on a sufficiently
new installation of the target OS (or updating the source checkouts for the
new build system). However, depending on the OS, you may need to update the
parsing in mksysnum.

mksyscall.go Copy Section