Technical Documentation: heroiclabs/nakama

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

Features

Users - Register/login new users via social networks, email, or device ID.
Storage - Store user records, settings, and other objects in collections.
Social - Users can connect with friends, and join groups. Builtin social graph to see how users can be connected.
Chat - 1-on-1, group, and global chat between users. Persist messages for chat history.
Multiplayer - Realtime, or turn-based active and passive multiplayer.
Leaderboards - Dynamic, seasonal, get top members, or members around a user. Have as many as you need.
Tournaments - Invite players to compete together over prizes. Link many together to create leagues.
Parties - Add team play to a game. Users can form a party and communicate with party members.
Purchase Validation - Validate in-app purchases and subscriptions.
In-App Notifications - Send messages and notifications to connected client sockets.
Runtime code - Extend the server with custom logic written in Lua, TypeScript/JavaScript, or native Go code.
Matchmaker, dashboard, metrics, and more.

Build scalable games and apps with a production ready server used by ambitious game studios and app developers all around the world. Have a look at the documentation and join the developer community for more info.

Getting Started

The server is simple to setup and run for local development and can be deployed to any cloud provider. See the deployment notes for recommendations on how to deploy the project for production. Nakama server requires CockroachDB or another Postgres wire-compatible server as it's database.

Docker

The fastest way to run the server and the database is with Docker. Setup Docker and start the daemon.

Set up a docker-compose file and place it in a folder for your project.
Run docker-compose -f ./docker-compose.yml up to download container images and run the servers.

For more detailed instructions have a look at our Docker quickstart guide.

Nakama Docker images are maintained on Docker Hub and prerelease images are occasionally published for cutting edge features of the server.

Binaries

You can run the servers with native binaries for your platform.

Download the server from our releases page and the database.
Follow the database instructions to start it.
Run a migration which will setup or upgrade the database schema:
shell
```
nakama migrate up --database.address "[email protected]:26257"
```

Start Nakama and connect to the database:

shell

nakama --database.address "[email protected]:26257"

When connected you'll see server output which describes all settings the server uses for configuration.

{"level":"info","ts":"2018-04-29T10:14:41.249+0100","msg":"Node","name":"nakama","version":"2.0.0+7e18b09","runtime":"go1.10.1","cpu":4}
{"level":"info","ts":"2018-04-29T10:14:41.249+0100","msg":"Database connections","dsns":["[email protected]:26257"]}
...

Usage

Nakama supports a variety of protocols optimized for various gameplay or app use cases. For request/response it can use GRPC or the HTTP1.1+JSON fallback (REST). For realtime communication you can use WebSockets or rUDP.

For example with the REST API to authenticate a user account with a device identifier.

shell

curl "127.0.0.1:7350/v2/account/authenticate/device?create=true" \
  --user "defaultkey:" \
  --data '{"id": "someuniqueidentifier"}'

Response:

{
"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1MjQ5OTU2NDksInVpZCI6Ijk5Y2Q1YzUyLWE5ODgtNGI2NC04YThhLTVmMTM5YTg4MTgxMiIsInVzbiI6InhBb1RxTUVSdFgifQ.-3_rXNYx3Q4jKuS7RkxeMWBzMNAm0vl93QxzRI8p_IY"
}

There's a number of official client libraries available on GitHub with documentation. The current platform/language support includes: .NET (in C#), Unity engine, JavaScript, Java (with Android), Unreal engine, Godot, Defold, and Swift (with iOS). If you'd like to contribute a client or request one let us know.

Nakama Console

The server provides a web UI which teams can use to inspect various data stored through the server APIs, view lightweight service metrics, manage player data, update storage objects, restrict access to production with permission profiles, and gain visibility into realtime features like active multiplayer matches. There is no separate installation required as it is embedded as part of the single server binary.

You can navigate to it on your browser on http://127.0.0.1:7351.

View Screenshots

Deployment

Nakama can be deployed to any cloud provider such as Google Cloud, Azure, AWS, Digital Ocean, Heroku, or your own private cloud. You should setup and provision separate nodes for Nakama and CockroachDB.

The recommended minimum production infrastructure for CockroachDB is outlined in these docs and Nakama can be run on instance types as small as "g1-small" on Google Cloud although we recommend a minimum of "n1-standard-1" in production. The specific hardware requirements will depend on what features of the server are used. Reach out to us for help and advice on what servers to run.

Heroic Cloud

You can support development, new features, and maintainance of the server by using the Heroic Labs' Heroic Cloud for deployment. This service handles the uptime, replication, backups, logs, data upgrades, and all other tasks involved with production server environments.

Have a look at our Heroic Cloud service for more details.

Contribute

The development roadmap is managed as GitHub issues and pull requests are welcome. If you're interested to add a feature which is not mentioned on the issue tracker please open one to create a discussion or drop in and discuss it in the community forum.

Simple Builds

All dependencies required for a build are vendored as part of the Go project. We recommend a modern release of the Go toolchain and do not store the codebase in the old GOPATH.

Download the source tree.

shell

git clone "https://github.com/heroiclabs/nakama" nakama
cd nakama

Build the project from source.

shell

go build -trimpath -mod=vendor
./nakama --version

Full Source Builds

The codebase uses Protocol Buffers, GRPC, GRPC-Gateway, buf, and the OpenAPI spec as part of the project. These dependencies are generated as sources and committed to the repository to simplify builds for contributors.

To build the codebase and generate all sources follow these steps.

Install the toolchain.

shell

go install \
    "google.golang.org/protobuf/cmd/protoc-gen-go" \
    "google.golang.org/grpc/cmd/protoc-gen-go-grpc" \
    "github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway" \
    "github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2"

Install buf.
Re-generate the protocol buffers and gateway code.
shell
```
# Run the shell script:
./buf.sh
```
Build the codebase.
shell
```
go build -trimpath -mod=vendor
```

Testing

In order to run all the unit and integration tests run:

shell

docker-compose -f ./docker-compose-tests.yml up --build --abort-on-container-exit; docker-compose -f ./docker-compose-tests.yml down -v

This will create an isolated environment with Nakama and database instances, run
all the tests, and drop the environment afterwards.

License

This project is licensed under the Apache-2 License.

Overview (`build/README.md`)

Release Instructions

These instructions guide the release process for new official Nakama server builds.

Steps

To build releases for a variety of platforms we use the excellent xgo project. You will need Docker engine installed. These steps should be followed from the project root folder.

These steps are one off to install the required build utilities.

Install the xgo Docker image.
text
```
docker pull techknowlogick/xgo:latest
```
Install the command line helper tool. Ensure "$GOPATH/bin" is on your system path to access the executable.
text
```
go install src.techknowlogick.com/xgo@latest
```

These steps are run for each new release.

Update the CHANGELOG.
Add the CHANGELOG file and tag a commit.

Note: In source control good semver suggests a "v" prefix on a version. It helps group release tags.
text
```
git add CHANGELOG.md
git commit -m "Nakama 2.1.0 release."
git tag -a v2.1.0 -m "v2.1.0"
git push origin v2.1.0 master
```

Execute the cross-compiled build helper.

text

xgo --targets=darwin/arm64,darwin/amd64,linux/amd64,linux/arm64,windows/amd64 --trimpath --ldflags "-s -w -X main.version=2.1.0 -X main.commitID=$(git rev-parse --short HEAD 2>/dev/null)" github.com/heroiclabs/nakama

This will build binaries for all target platforms supported officially by Heroic Labs.

Package up each release as a compressed bundle.

text

tar -czf "nakama-<os>-<arch>.tar.gz" nakama README.md LICENSE CHANGELOG.md

Create a new draft release on GitHub and publish it with the compressed bundles.

Build Nakama Image

With the release generated we can create the official container image.

These steps are one off to install the required build utilities.

Install Docker Desktop.
Create a new Docker context to create multi-platform builds.
text
```
docker context create builder
```

These steps are run for each new release.

Use an existing builder that supports multi-platform builds.
text
```
docker buildx create --use builder
```

Build the container image and push to the container registry.

text

cd build
docker buildx build . --platform linux/amd64,linux/arm64 --file ./Dockerfile --build-arg commit="$(git rev-parse --short HEAD 2>/dev/null)" --build-arg version=2.1.0 -t heroiclabs/nakama:2.1.0 -t heroiclabs/nakama:latest --push

Build Nakama Image (dSYM)

With the release generated we can also create an official container image which includes debug symbols.

Use an existing builder that supports multi-platform builds.
text
```
docker buildx create --use builder
```

Build the container image and push to the container registry.

text

cd build
docker buildx build "$PWD" --platform linux/amd64,linux/arm64 --file ./Dockerfile.dsym --build-arg commit="$(git rev-parse --short HEAD 2>/dev/null)" --build-arg version=2.1.0 -t heroiclabs/nakama-dsym:2.1.0 -t heroiclabs/nakama-dsym:latest --push

Build Plugin Builder Image

With the official release image generated we can create a container image to help with Go runtime development.

Use an existing builder that supports multi-platform builds.
text
```
docker buildx create --use builder
```

Build the container image.

text

cd build/pluginbuilder
docker buildx build "$PWD" --platform linux/amd64,linux/arm64 --file ./Dockerfile --build-arg commit="$(git rev-parse --short HEAD 2>/dev/null)" --build-arg version=2.1.0 -t heroiclabs/nakama-pluginbuilder:2.1.0 -t heroiclabs/nakama-pluginbuilder:latest --push

Build all images with provided script.

Use an existing builder that supports multi-platform builds.
text
```
docker buildx create --use builder
```
Build the container image.
text
```
cd build
./build.sh
```

Overview (`sample_go_module/README.md`)

Nakama Go Runtime

The game server includes support to develop native code in Go with the plugin package from the Go stdlib. It's used to enable compiled shared objects to be loaded by the game server at startup.

The Go runtime support can be used to develop authoritative multiplayer match handlers, RPC functions, hook into messages processed by the server, and extend the server with any other custom logic. It offers the same capabilities as the Lua runtime support but has the advantage that any package from the Go ecosystem can be used.

For more information and a discussion of the pros/cons with the Go runtime have a look at the docs.

Minimal example

Here's the smallest example of a Go module written with the server runtime.

package main

import (
  "context"
  "database/sql"

  "github.com/heroiclabs/nakama-common/runtime"
)

func InitModule(ctx context.Context, logger runtime.Logger, db *sql.DB, nk runtime.NakamaModule, initializer runtime.Initializer) error {
  logger.Info("module loaded")
  return nil
}

Setup a project

To setup your own project to build modules for the game server you can follow these steps.

Download and install the Go toolchain. It's recommended you follow the official instructions.

Setup a folder for your own plugin code.

bash

mkdir -p "$HOME/plugin_code"
cd "$HOME/plugin_code"

Initialize the Go module for your plugin and add the nakama-common dependency.
bash
```
go mod init "plugin_code"
go get -u "github.com/heroiclabs/[email protected]"
```
⚠️ NOTE: If you're working on Nakama's master branch drop the @v1.23.0 from the above snippet.

⚠️ NOTE: The official Nakama v3.12.+ expects nakama-common v1.23.0 in order to run. If you use v1.22.0, older, or drop the version reference, you might get a plugin was built with a different version of package error while starting up the Nakama server.
Develop your plugin code (you can use the minimal example as a starting point) and save it within your plugin project directory with the .go extension.

Build & load process

In a regular development cycle you will often recompile your plugin code and rerun the server.

Develop and compile your code.

bash

go build -buildmode=plugin -trimpath -o ./plugin_code.so

Use --runtime.path flag when you start the Nakama server binary to load your built plugin. (Note: Also make sure you run the database).
bash
```
./nakama --runtime.path "$HOME/plugin_code"
```
TIP: You can either build and run Nakama from source or you can download a prebuilt binary for your platform here.

For more information on how the server loads modules have a look at these docs. For general instructions on how to run the server give these docs a read.

Docker builds

It's often easiest to run the game server with Docker Compose. It will start the game server and database server together in the right sequence and wraps the process up into a single command. You'll need the Docker engine installed to use it.

For Windows development and environments where you want to use our official Docker images to run your containers we provide a container image to help you build your code.

Use the Docker plugin helper container to compile your project (works for bash/PowerShell):
bash
```
cd "$HOME/plugin_code" # Your project folder. See instructions above.
docker run --rm -w "/builder" -v "${PWD}:/builder" heroiclabs/nakama-pluginbuilder:3.12.0 build -buildmode=plugin -trimpath -o ./modules/plugin_code.so
```
In the command above we bind-mount your current folder into the container and use the Go toolchain inside it to run the build. The output artifacts are written back into your host filesystem.
Use our official Docker Compose file to run all containers together and load in your custom module.

NOTE: You should copy the .so files generated in step 1. to the /modules folder of your Nakama source files and then run the command below from the Nakama root directory.
bash
```
docker-compose up
```
By default the server will be started and look in a folder relative to the current dir called "./modules" to load the plugins.

TIP: Use the same version of your plugin builder image as used in the Docker Compose file for the server version. i.e. "heroiclabs/nakama:2.3.1" <> "heroiclabs/nakama-pluginbuilder:2.3.1", etc.

Bigger Example

Have a look in this repo for more example code on how to create and use various parts of the game server Go runtime support. The project implements a fully authoritative example of tic-tac-toe in Go, Lua and TS.

https://github.com/heroiclabs/nakama-project-template

Overview (`build/do-marketplace/README.md`)

Build Automation with Packer

Packer is a tool for creating images from a single source configuration. Using this Packer template reduces the entire process of creating, configuring, validating, and snapshotting a build Droplet to a single command:

text

packer build marketplace-image.json

This Packer template is configured to create a snapshot with Nakama and CockroachDB.

Usage

To get started, you'll need to install Packer and create a DigitalOcean personal access token and set it to the DIGITALOCEAN_API_TOKEN environment variable. Running packer build marketplace-image.json without any other modifications will create a build Droplet configured with Nakama & CockroachDB, clean and verify it, then power it down and snapshot it.

⚠️ The image validation script in scripts/99-img_check.sh is copied from the top-level scripts directory in the DigitalOcean Marketplace Partnners repository. The top-level location is the script's canonical source, so make sure you're using the latest version from there.

This Packer template uses binaries for both Nakama and CockroachDB. The URL for each binary is set via variables at the top of the template.

Configuration Details

By using Packer's DigitalOcean Builder to integrate with the DigitalOcean API, this template fully automates Marketplace image creation.

This template uses Packer's file provisioner to upload complete directories to the Droplet. The contents of files/var/ will be uploaded to /var/. Likewise, the contents of files/etc/ will be uploaded to /etc/. One important thing to note about the file provisioner, from Packer's docs:

The destination directory must already exist. If you need to create it, use a shell provisioner just prior to the file provisioner in order to create the directory. If the destination directory does not exist, the file provisioner may succeed, but it will have undefined results.

This template also uses Packer's shell provisioner to run scripts from the /scripts directory and install APT packages using an inline task.

Learn more about using Packer in the official Packer documentation.

Overview (`internal/cronexpr/README.md`)

Golang Cron expression parser

Given a cron expression and a time stamp, you can get the next time stamp which satisfies the cron expression.

In another project, I decided to use cron expression syntax to encode scheduling information. Thus this standalone library to parse and apply time stamps to cron expressions.

The time-matching algorithm in this implementation is efficient, it avoids as much as possible to guess the next matching time stamp, a common technique seen in a number of implementations out there.

There is also a companion command-line utility to evaluate cron time expressions: https://github.com/gorhill/cronexpr/tree/master/cronexpr (which of course uses this library).

Implementation

The reference documentation for this implementation is found at
https://en.wikipedia.org/wiki/Cron#CRON_expression, which I copy/pasted here (laziness!) with modifications where this implementation differs:

text

Field name     Mandatory?   Allowed values    Allowed special characters
----------     ----------   --------------    --------------------------
Seconds        No           0-59              * / , -
Minutes        Yes          0-59              * / , -
Hours          Yes          0-23              * / , -
Day of month   Yes          1-31              * / , - L W
Month          Yes          1-12 or JAN-DEC   * / , -
Day of week    Yes          0-6 or SUN-SAT    * / , - L #
Year           No           1970–2099         * / , -

Asterisk ( * )

The asterisk indicates that the cron expression matches for all values of the field. E.g., using an asterisk in the 4th field (month) indicates every month.

Slash ( / )

Slashes describe increments of ranges. For example 3-59/15 in the minute field indicate the third minute of the hour and every 15 minutes thereafter. The form */... is equivalent to the form "first-last/...", that is, an increment over the largest possible range of the field.

Comma ( , )

Commas are used to separate items of a list. For example, using MON,WED,FRI in the 5th field (day of week) means Mondays, Wednesdays and Fridays.

Hyphen ( - )

Hyphens define ranges. For example, 2000-2010 indicates every year between 2000 and 2010 AD, inclusive.

L

L stands for "last". When used in the day-of-week field, it allows you to specify constructs such as "the last Friday" (5L) of a given month. In the day-of-month field, it specifies the last day of the month.

W

The W character is allowed for the day-of-month field. This character is used to specify the business day (Monday-Friday) nearest the given day. As an example, if you were to specify 15W as the value for the day-of-month field, the meaning is: "the nearest business day to the 15th of the month."

So, if the 15th is a Saturday, the trigger fires on Friday the 14th. If the 15th is a Sunday, the trigger fires on Monday the 16th. If the 15th is a Tuesday, then it fires on Tuesday the 15th. However if you specify 1W as the value for day-of-month, and the 1st is a Saturday, the trigger fires on Monday the 3rd, as it does not 'jump' over the boundary of a month's days.

The W character can be specified only when the day-of-month is a single day, not a range or list of days.

The W character can also be combined with L, i.e. LW to mean "the last business day of the month."

Hash ( # )

# is allowed for the day-of-week field, and must be followed by a number between one and five. It allows you to specify constructs such as "the second Friday" of a given month.

Predefined cron expressions

(Copied from https://en.wikipedia.org/wiki/Cron#Predefined_scheduling_definitions, with text modified according to this implementation)

text

Entry       Description                                                             Equivalent to
@annually   Run once a year at midnight in the morning of January 1                 0 0 0 1 1 * *
@yearly     Run once a year at midnight in the morning of January 1                 0 0 0 1 1 * *
@monthly    Run once a month at midnight in the morning of the first of the month   0 0 0 1 * * *
@weekly     Run once a week at midnight in the morning of Sunday                    0 0 0 * * 0 *
@daily      Run once a day at midnight                                              0 0 0 * * * *
@hourly     Run once an hour at the beginning of the hour                           0 0 * * * * *
@reboot     Not supported

Other details

If only six fields are present, a 0 second field is prepended, that is, * * * * * 2013 internally become 0 * * * * * 2013.
If only five fields are present, a 0 second field is prepended and a wildcard year field is appended, that is, * * * * Mon internally become 0 * * * * Mon *.
Domain for day-of-week field is [0-7] instead of [0-6], 7 being Sunday (like 0). This to comply with http://linux.die.net/man/5/crontab#.
As of now, the behavior of the code is undetermined if a malformed cron expression is supplied

Install

text

go get github.com/gorhill/cronexpr

Usage

Import the library:

text

import "github.com/gorhill/cronexpr"
import "time"

Simplest way:

text

nextTime := cronexpr.MustParse("0 0 29 2 *").Next(time.Now())

Assuming time.Now() is "2013-08-29 09:28:00", then nextTime will be "2016-02-29 00:00:00".

You can keep the returned Expression pointer around if you want to reuse it:

text

expr := cronexpr.MustParse("0 0 29 2 *")
nextTime := expr.Next(time.Now())
...
nextTime = expr.Next(nextTime)

Use time.IsZero() to find out whether a valid time was returned. For example,

text

cronexpr.MustParse("* * * * * 1980").Next(time.Now()).IsZero()

will return true, whereas

text

cronexpr.MustParse("* * * * * 2050").Next(time.Now()).IsZero()

will return false (as of 2013-08-29...)

You may also query for n next time stamps:

text

cronexpr.MustParse("0 0 29 2 *").NextN(time.Now(), 5)

which returns a slice of time.Time objects, containing the following time stamps (as of 2013-08-30):

text

2016-02-29 00:00:00
2020-02-29 00:00:00
2024-02-29 00:00:00
2028-02-29 00:00:00
2032-02-29 00:00:00

The time zone of time values returned by Next and NextN is always the
time zone of the time value passed as argument, unless a zero time value is
returned.

API

http://godoc.org/github.com/gorhill/cronexpr

License

License: pick the one which suits you best:

GPL v3 see https://www.gnu.org/licenses/gpl.html
APL v2 see http://www.apache.org/licenses/LICENSE-2.0

Overview (`internal/skiplist/README.md`)

skiplist

reference from redis zskiplist

Usage

package main

import (
	"fmt"
	"github.com/gansidui/skiplist"
	"log"
)

type User struct {
	score float64
	id    string
}

func (u *User) Less(other interface{}) bool {
	if u.score > other.(*User).score {
		return true
	}
	if u.score == other.(*User).score && len(u.id) > len(other.(*User).id) {
		return true
	}
	return false
}

func main() {
	us := make([]*User, 7)
	us[0] = &User{6.6, "hi"}
	us[1] = &User{4.4, "hello"}
	us[2] = &User{2.2, "world"}
	us[3] = &User{3.3, "go"}
	us[4] = &User{1.1, "skip"}
	us[5] = &User{2.2, "list"}
	us[6] = &User{3.3, "lang"}

	// insert
	sl := skiplist.New()
	for i := 0; i < len(us); i++ {
		sl.Insert(us[i])
	}

	// traverse
	for e := sl.Front(); e != nil; e = e.Next() {
		fmt.Println(e.Value.(*User).id, "-->", e.Value.(*User).score)
	}
	fmt.Println()

	// rank
	rank1 := sl.GetRank(&User{2.2, "list"})
	rank2 := sl.GetRank(&User{6.6, "hi"})
	if rank1 != 6 || rank2 != 1 {
		log.Fatal()
	}
	if e := sl.GetElementByRank(2); e.Value.(*User).score != 4.4 || e.Value.(*User).id != "hello" {
		log.Fatal()
	}
}

/* output:

hi --> 6.6
hello --> 4.4
lang --> 3.3
go --> 3.3
world --> 2.2
list --> 2.2
skip --> 1.1

*/

License

MIT

Overview (`internal/cronexpr/cronexpr/README.md`)

cronexpr: command-line utility

A command-line utility written in Go to evaluate cron time expressions.

It is based on the standalone Go library https://github.com/gorhill/cronexpr.

Install

text

go get github.com/gorhill/cronexpr
go install github.com/gorhill/cronexpr

Usage

text

cronexpr [options] "{cron expression}"

Options

-l:

Go-compliant time layout to use for outputting time value(s), see http://golang.org/pkg/time/#pkg-constants.

Default is "Mon, 02 Jan 2006 15:04:05 MST"

-n:

Number of resulting time values to output.

Default is 1.

-t:

Whole or partial RFC3339 time value (i.e. 2006-01-02T15:04:05Z07:00) against which the cron expression is evaluated. Examples of valid values include (assuming EST time zone):

13 = 2013-01-01T00:00:00-05:00
2013 = 2013-01-01T00:00:00-05:00
2013-08 = 2013-08-01T00:00:00-05:00
2013-08-31 = 2013-08-31T00:00:00-05:00
2013-08-31T12 = 2013-08-31T12:00:00-05:00
2013-08-31T12:40 = 2013-08-31T12:40:00-05:00
2013-08-31T12:40:35 = 2013-08-31T12:40:35-05:00
2013-08-31T12:40:35-10:00 = 2013-08-31T12:40:35-10:00

Default time is current time, and default time zone is local time zone.

Examples

Example 1

Midnight on December 31st of any year.

Command:

text

cronexpr -t="2013-08-31" -n=5 "0 0 31 12 *"

Output (assuming computer is in EST time zone):

text

# "0 0 31 12 *" + "2013-08-31T00:00:00-04:00" =
Tue, 31 Dec 2013 00:00:00 EST
Wed, 31 Dec 2014 00:00:00 EST
Thu, 31 Dec 2015 00:00:00 EST
Sat, 31 Dec 2016 00:00:00 EST
Sun, 31 Dec 2017 00:00:00 EST

Example 2

2pm on February 29th of any year.

Command:

text

cronexpr -t=2000 -n=10 "0 14 29 2 *"

Output (assuming computer is in EST time zone):

text

# "0 14 29 2 *" + "2000-01-01T00:00:00-05:00" =
Tue, 29 Feb 2000 14:00:00 EST
Sun, 29 Feb 2004 14:00:00 EST
Fri, 29 Feb 2008 14:00:00 EST
Wed, 29 Feb 2012 14:00:00 EST
Mon, 29 Feb 2016 14:00:00 EST
Sat, 29 Feb 2020 14:00:00 EST
Thu, 29 Feb 2024 14:00:00 EST
Tue, 29 Feb 2028 14:00:00 EST
Sun, 29 Feb 2032 14:00:00 EST
Fri, 29 Feb 2036 14:00:00 EST

Example 3

12pm on the work day closest to the 15th of March and every three month
thereafter.

Command:

text

cronexpr -t=2013-09-01 -n=5 "0 12 15W 3/3 *"

Output (assuming computer is in EST time zone):

text

# "0 12 15W 3/3 *" + "2013-09-01T00:00:00-04:00" =
Mon, 16 Sep 2013 12:00:00 EDT
Mon, 16 Dec 2013 12:00:00 EST
Fri, 14 Mar 2014 12:00:00 EDT
Mon, 16 Jun 2014 12:00:00 EDT
Mon, 15 Sep 2014 12:00:00 EDT

Example 4

Midnight on the fifth Saturday of any month (twist: not all months have a 5th
specific day of week).

Command:

text

cronexpr -t=2013-09-02 -n 5 "0 0 * * 6#5"

Output (assuming computer is in EST time zone):

text

# "0 0 * * 6#5" + "2013-09-02T00:00:00-04:00" =
Sat, 30 Nov 2013 00:00:00 EST
Sat, 29 Mar 2014 00:00:00 EDT
Sat, 31 May 2014 00:00:00 EDT
Sat, 30 Aug 2014 00:00:00 EDT
Sat, 29 Nov 2014 00:00:00 EST

Overview (`vendor/go.uber.org/atomic/README.md`)

atomic [![GoDoc][doc-img]][doc] [![Build Status][ci-img]][ci] [![Coverage Status][cov-img]][cov] [![Go Report Card][reportcard-img]][reportcard]

Simple wrappers for primitive types to enforce atomic access.

Installation

shell

$ go get -u go.uber.org/atomic@v1

Legacy Import Path

As of v1.5.0, the import path go.uber.org/atomic is the only supported way
of using this package. If you are using Go modules, this package will fail to
compile with the legacy import path path github.com/uber-go/atomic.

We recommend migrating your code to the new import path but if you're unable
to do so, or if your dependencies are still using the old import path, you
will have to add a replace directive to your go.mod file downgrading the
legacy import path to an older version.

text

replace github.com/uber-go/atomic => github.com/uber-go/atomic v1.4.0

You can do so automatically by running the following command.

shell

$ go mod edit -replace github.com/uber-go/atomic=github.com/uber-go/[email protected]

Usage

The standard library's sync/atomic is powerful, but it's easy to forget which
variables must be accessed atomically. go.uber.org/atomic preserves all the
functionality of the standard library, but wraps the primitive types to
provide a safer, more convenient API.

var atom atomic.Uint32
atom.Store(42)
atom.Sub(2)
atom.CAS(40, 11)

See the documentation for a complete API specification.

Development Status

Stable.

Released under the MIT License.

Overview (`vendor/go.uber.org/multierr/README.md`)

multierr [![GoDoc][doc-img]][doc] [![Build Status][ci-img]][ci] [![Coverage Status][cov-img]][cov]

multierr allows combining one or more Go errors together.

Features

Idiomatic:
multierr follows best practices in Go, and keeps your code idiomatic.
- It keeps the underlying error type hidden,
  allowing you to deal in error values exclusively.
- It provides APIs to safely append into an error from a defer statement.
Performant:
multierr is optimized for performance:
- It avoids allocations where possible.
- It utilizes slice resizing semantics to optimize common cases
  like appending into the same error object from a loop.
Interoperable:
multierr interoperates with the Go standard library's error APIs seamlessly:
- The errors.Is and errors.As functions just work.
Lightweight:
multierr comes with virtually no dependencies.

Installation

bash

go get -u go.uber.org/multierr@latest

Status

Stable: No breaking changes will be made before 2.0.

Released under the MIT License.

Overview (`vendor/go.uber.org/zap/README.md`)

:zap: zap

Installation

go get -u go.uber.org/zap

Note that zap only supports the two most recent minor versions of Go.

Quick Start

In contexts where performance is nice, but not critical, use the
SugaredLogger. It's 4-10x faster than other structured logging
packages and includes both structured and printf-style APIs.

logger, _ := zap.NewProduction()
defer logger.Sync() // flushes buffer, if any
sugar := logger.Sugar()
sugar.Infow("failed to fetch URL",
  // Structured context as loosely typed key-value pairs.
  "url", url,
  "attempt", 3,
  "backoff", time.Second,
)
sugar.Infof("Failed to fetch URL: %s", url)

When performance and type safety are critical, use the Logger. It's even
faster than the SugaredLogger and allocates far less, but it only supports
structured logging.

logger, _ := zap.NewProduction()
defer logger.Sync()
logger.Info("failed to fetch URL",
  // Structured context as strongly typed Field values.
  zap.String("url", url),
  zap.Int("attempt", 3),
  zap.Duration("backoff", time.Second),
)

See the documentation and FAQ for more details.

Performance

For applications that log in the hot path, reflection-based serialization and
string formatting are prohibitively expensive — they're CPU-intensive
and make many small allocations. Put differently, using encoding/json and
fmt.Fprintf to log tons of interface{}s makes your application slow.

Zap takes a different approach. It includes a reflection-free, zero-allocation
JSON encoder, and the base Logger strives to avoid serialization overhead
and allocations wherever possible. By building the high-level SugaredLogger
on that foundation, zap lets users choose when they need to count every
allocation and when they'd prefer a more familiar, loosely typed API.

As measured by its own benchmarking suite, not only is zap more performant
than comparable structured logging packages — it's also faster than the
standard library. Like all benchmarks, take these with a grain of salt.¹

Log a message and 10 fields:

Package	Time	Time % to zap	Objects Allocated
:zap: zap	656 ns/op	+0%	5 allocs/op
:zap: zap (sugared)	935 ns/op	+43%	10 allocs/op
zerolog	380 ns/op	-42%	1 allocs/op
go-kit	2249 ns/op	+243%	57 allocs/op
slog (LogAttrs)	2479 ns/op	+278%	40 allocs/op
slog	2481 ns/op	+278%	42 allocs/op
apex/log	9591 ns/op	+1362%	63 allocs/op
log15	11393 ns/op	+1637%	75 allocs/op
logrus	11654 ns/op	+1677%	79 allocs/op

Log a message with a logger that already has 10 fields of context:

Package	Time	Time % to zap	Objects Allocated
:zap: zap	67 ns/op	+0%	0 allocs/op
:zap: zap (sugared)	84 ns/op	+25%	1 allocs/op
zerolog	35 ns/op	-48%	0 allocs/op
slog	193 ns/op	+188%	0 allocs/op
slog (LogAttrs)	200 ns/op	+199%	0 allocs/op
go-kit	2460 ns/op	+3572%	56 allocs/op
log15	9038 ns/op	+13390%	70 allocs/op
apex/log	9068 ns/op	+13434%	53 allocs/op
logrus	10521 ns/op	+15603%	68 allocs/op

Log a static string, without any context or printf-style templating:

Package	Time	Time % to zap	Objects Allocated
:zap: zap	63 ns/op	+0%	0 allocs/op
:zap: zap (sugared)	81 ns/op	+29%	1 allocs/op
zerolog	32 ns/op	-49%	0 allocs/op
standard library	124 ns/op	+97%	1 allocs/op
slog	196 ns/op	+211%	0 allocs/op
slog (LogAttrs)	200 ns/op	+217%	0 allocs/op
go-kit	213 ns/op	+238%	9 allocs/op
apex/log	771 ns/op	+1124%	5 allocs/op
logrus	1439 ns/op	+2184%	23 allocs/op
log15	2069 ns/op	+3184%	20 allocs/op

Development Status: Stable

All APIs are finalized, and no breaking changes will be made in the 1.x series
of releases. Users of semver-aware dependency management systems should pin
zap to ^1.

Contributing

We encourage and support an active, healthy community of contributors —
including you! Details are in the contribution guide and
the code of conduct. The zap maintainers keep an eye on
issues and pull requests, but you can also report any negative conduct to
[email protected]. That email list is a private, safe space; even the zap
maintainers don't have access, so don't hesitate to hold us to a high
standard.

Released under the MIT License.

¹ In particular, keep in mind that we may be
benchmarking against slightly older versions of other packages. Versions are
pinned in the benchmarks/go.mod file. ↩

Overview (`vendor/google.golang.org/grpc/README.md`)

gRPC-Go

The Go implementation of gRPC: A high performance, open source, general
RPC framework that puts mobile and HTTP/2 first. For more information see the
Go gRPC docs, or jump directly into the quick start.

Prerequisites

Go: any one of the two latest major releases.

Installation

Simply add the following import to your code, and then go [build|run|test]
will automatically fetch the necessary dependencies:

import "google.golang.org/grpc"

Note: If you are trying to access grpc-go from China, see the
FAQ below.

Learn more

Go gRPC docs, which include a quick start and API
reference among other resources
Low-level technical docs from this repository
Performance benchmark
Examples
Contribution guidelines

FAQ

I/O Timeout Errors

The golang.org domain may be blocked from some countries. go get usually
produces an error like the following when this happens:

console

$ go get -u google.golang.org/grpc
package google.golang.org/grpc: unrecognized import path "google.golang.org/grpc" (https fetch: Get https://google.golang.org/grpc?go-get=1: dial tcp 216.239.37.1:443: i/o timeout)

To build Go code, there are several options:

Set up a VPN and access google.golang.org through that.
With Go module support: it is possible to use the replace feature of go mod to create aliases for golang.org packages. In your project's directory:
sh
```
go mod edit -replace=google.golang.org/grpc=github.com/grpc/grpc-go@latest
go mod tidy
go mod vendor
go build -mod=vendor
```
Again, this will need to be done for all transitive dependencies hosted on
golang.org as well. For details, refer to golang/go issue
#28652.

Compiling error, undefined: grpc.SupportPackageIsVersion

Please update to the latest version of gRPC-Go using
go get google.golang.org/grpc.

How to turn on logging

The default logger is controlled by environment variables. Turn everything on
like this:

console

$ export GRPC_GO_LOG_VERBOSITY_LEVEL=99
$ export GRPC_GO_LOG_SEVERITY_LEVEL=info

The RPC failed with error `"code = Unavailable desc = transport is closing"`

This error means the connection the RPC is using was closed, and there are many
possible reasons, including:

mis-configured transport credentials, connection failed on handshaking
bytes disrupted, possibly by a proxy in between
server shutdown
Keepalive parameters caused connection shutdown, for example if you have
configured your server to terminate connections regularly to trigger DNS
lookups.
If this is the case, you may want to increase your
MaxConnectionAgeGrace,
to allow longer RPC calls to finish.

It can be tricky to debug this because the error happens on the client side but
the root cause of the connection being closed is on the server side. Turn on
logging on both client and server, and see if there are any transport
errors.

Overview (`vendor/gopkg.in/yaml.v3/README.md`)

YAML support for the Go language

Introduction

The yaml package enables Go programs to comfortably encode and decode YAML
values. It was developed within Canonical as
part of the juju project, and is based on a
pure Go port of the well-known libyaml
C library to parse and generate YAML data quickly and reliably.

Compatibility

The yaml package supports most of YAML 1.2, but preserves some behavior
from 1.1 for backwards compatibility.

Specifically, as of v3 of the yaml package:

YAML 1.1 bools (yes/no, on/off) are supported as long as they are being
decoded into a typed bool value. Otherwise they behave as a string. Booleans
in YAML 1.2 are true/false only.
Octals encode and decode as 0777 per YAML 1.1, rather than 0o777
as specified in YAML 1.2, because most parsers still use the old format.
Octals in the 0o777 format are supported though, so new files work.
Does not support base-60 floats. These are gone from YAML 1.2, and were
actually never supported by this package as it's clearly a poor choice.

and offers backwards
compatibility with YAML 1.1 in some cases.
1.2, including support for
anchors, tags, map merging, etc. Multi-document unmarshalling is not yet
implemented, and base-60 floats from YAML 1.1 are purposefully not
supported since they're a poor design and are gone in YAML 1.2.

Installation and usage

The import path for the package is gopkg.in/yaml.v3.

To install it, run:

text

go get gopkg.in/yaml.v3

API documentation

If opened in a browser, the import path itself leads to the API documentation:

https://gopkg.in/yaml.v3

API stability

The package API for yaml v3 will remain stable as described in gopkg.in.

License

The yaml package is licensed under the MIT and Apache License 2.0 licenses.
Please see the LICENSE file for details.

Example

package main

import (
        "fmt"
        "log"

        "gopkg.in/yaml.v3"
)

var data = `
a: Easy!
b:
  c: 2
  d: [3, 4]
`

// Note: struct fields must be public in order for unmarshal to
// correctly populate the data.
type T struct {
        A string
        B struct {
                RenamedC int   `yaml:"c"`
                D        []int `yaml:",flow"`
        }
}

func main() {
        t := T{}
    
        err := yaml.Unmarshal([]byte(data), &t)
        if err != nil {
                log.Fatalf("error: %v", err)
        }
        fmt.Printf("--- t:\n%v\n\n", t)
    
        d, err := yaml.Marshal(&t)
        if err != nil {
                log.Fatalf("error: %v", err)
        }
        fmt.Printf("--- t dump:\n%s\n\n", string(d))
    
        m := make(map[interface{}]interface{})
    
        err = yaml.Unmarshal([]byte(data), &m)
        if err != nil {
                log.Fatalf("error: %v", err)
        }
        fmt.Printf("--- m:\n%v\n\n", m)
    
        d, err = yaml.Marshal(&m)
        if err != nil {
                log.Fatalf("error: %v", err)
        }
        fmt.Printf("--- m dump:\n%s\n\n", string(d))
}

This example will generate the following output:

text

--- t:
{Easy! {2 [3 4]}}

--- t dump:
a: Easy!
b:
  c: 2
  d: [3, 4]


--- m:
map[a:Easy! b:map[c:2 d:[3 4]]]

--- m dump:
a: Easy!
b:
  c: 2
  d:
  - 3
  - 4

Overview (`vendor/github.com/RoaringBitmap/roaring/README.md`)

roaring

=============

This is a go version of the Roaring bitmap data structure.

Roaring bitmaps are used by several major systems such as Apache Lucene and derivative systems such as Solr and
Elasticsearch, Apache Druid (Incubating), LinkedIn Pinot, Netflix Atlas, Apache Spark, OpenSearchServer, anacrolix/torrent, Whoosh, Redpanda, Pilosa, Microsoft Visual Studio Team Services (VSTS), and eBay's Apache Kylin. The YouTube SQL Engine, Google Procella, uses Roaring bitmaps for indexing.

Roaring bitmaps are found to work well in many important applications:

Use Roaring for bitmap compression whenever possible. Do not use other bitmap compression methods (Wang et al., SIGMOD 2017)

The roaring Go library is used by

This library is used in production in several systems, it is part of the Awesome Go collection.

There are also Java and C/C++ versions. The Java, C, C++ and Go version are binary compatible: e.g, you can save bitmaps
from a Java program and load them back in Go, and vice versa. We have a format specification.

This code is licensed under Apache License, Version 2.0 (ASL2.0).

When should you use a bitmap?

Sets are a fundamental abstraction in
software. They can be implemented in various
ways, as hash sets, as trees, and so forth.
In databases and search engines, sets are often an integral
part of indexes. For example, we may need to maintain a set
of all documents or rows (represented by numerical identifier)
that satisfy some property. Besides adding or removing
elements from the set, we need fast functions
to compute the intersection, the union, the difference between sets, and so on.

To implement a set
of integers, a particularly appealing strategy is the
bitmap (also called bitset or bit vector). Using n bits,
we can represent any set made of the integers from the range
[0,n): the ith bit is set to one if integer i is present in the set.
Commodity processors use words of W=32 or W=64 bits. By combining many such words, we can
support large values of n. Intersections, unions and differences can then be implemented
as bitwise AND, OR and ANDNOT operations.
More complicated set functions can also be implemented as bitwise operations.

When the bitset approach is applicable, it can be orders of
magnitude faster than other possible implementation of a set (e.g., as a hash set)
while using several times less memory.

However, a bitset, even a compressed one is not always applicable. For example, if
you have 1000 random-looking integers, then a simple array might be the best representation.
We refer to this case as the "sparse" scenario.

When should you use compressed bitmaps?

An uncompressed BitSet can use a lot of memory. For example, if you take a BitSet
and set the bit at position 1,000,000 to true and you have just over 100kB. That is over 100kB
to store the position of one bit. This is wasteful even if you do not care about memory:
suppose that you need to compute the intersection between this BitSet and another one
that has a bit at position 1,000,001 to true, then you need to go through all these zeroes,
whether you like it or not. That can become very wasteful.

This being said, there are definitively cases where attempting to use compressed bitmaps is wasteful.
For example, if you have a small universe size. E.g., your bitmaps represent sets of integers
from [0,n) where n is small (e.g., n=64 or n=128). If you can use uncompressed BitSet and
it does not blow up your memory usage, then compressed bitmaps are probably not useful
to you. In fact, if you do not need compression, then a BitSet offers remarkable speed.

The sparse scenario is another use case where compressed bitmaps should not be used.
Keep in mind that random-looking data is usually not compressible. E.g., if you have a small set of
32-bit random integers, it is not mathematically possible to use far less than 32 bits per integer,
and attempts at compression can be counterproductive.

How does Roaring compares with the alternatives?

Most alternatives to Roaring are part of a larger family of compressed bitmaps that are run-length-encoded
bitmaps. They identify long runs of 1s or 0s and they represent them with a marker word.
If you have a local mix of 1s and 0, you use an uncompressed word.

There are many formats in this family:

Oracle's BBC is an obsolete format at this point: though it may provide good compression,
it is likely much slower than more recent alternatives due to excessive branching.
WAH is a patented variation on BBC that provides better performance.
Concise is a variation on the patented WAH. It some specific instances, it can compress
much better than WAH (up to 2x better), but it is generally slower.
EWAH is both free of patent, and it is faster than all the above. On the downside, it
does not compress quite as well. It is faster because it allows some form of "skipping"
over uncompressed words. So though none of these formats are great at random access, EWAH
is better than the alternatives.

There is a big problem with these formats however that can hurt you badly in some cases: there is no random access. If you want to check whether a given value is present in the set, you have to start from the beginning and "uncompress" the whole thing. This means that if you want to intersect a big set with a large set, you still have to uncompress the whole big set in the worst case...

Roaring solves this problem. It works in the following manner. It divides the data into chunks of 2¹⁶ integers
(e.g., [0, 2¹⁶), [2¹⁶, 2 x 2¹⁶), ...). Within a chunk, it can use an uncompressed bitmap, a simple list of integers,
or a list of runs. Whatever format it uses, they all allow you to check for the presence of any one value quickly
(e.g., with a binary search). The net result is that Roaring can compute many operations much faster than run-length-encoded
formats like WAH, EWAH, Concise... Maybe surprisingly, Roaring also generally offers better compression ratios.

References

Daniel Lemire, Owen Kaser, Nathan Kurz, Luca Deri, Chris O'Hara, François Saint-Jacques, Gregory Ssi-Yan-Kai, Roaring Bitmaps: Implementation of an Optimized Software Library, Software: Practice and Experience 48 (4), 2018 arXiv:1709.07821
Samy Chambi, Daniel Lemire, Owen Kaser, Robert Godin,
Better bitmap performance with Roaring bitmaps,
Software: Practice and Experience 46 (5), 2016.arXiv:1402.6407 This paper used data from http://lemire.me/data/realroaring2014.html
Daniel Lemire, Gregory Ssi-Yan-Kai, Owen Kaser, Consistently faster and smaller compressed bitmaps with Roaring, Software: Practice and Experience 46 (11), 2016. arXiv:1603.06549

Dependencies

Dependencies are fetched automatically by giving the -t flag to go get.

they include

github.com/bits-and-blooms/bitset
github.com/mschoch/smat
github.com/glycerine/go-unsnap-stream
github.com/philhofer/fwd
github.com/jtolds/gls

Note that the smat library requires Go 1.6 or better.

Installation

go get -t github.com/RoaringBitmap/roaring

Instructions for contributors

Using bash or other common shells:

text

$ git clone [email protected]:RoaringBitmap/roaring.git
$ export GO111MODULE=on 
$ go mod tidy
$ go test -v

Example

Here is a simplified but complete example:

package main

import (
    "fmt"
    "github.com/RoaringBitmap/roaring"
    "bytes"
)


func main() {
    // example inspired by https://github.com/fzandona/goroar
    fmt.Println("==roaring==")
    rb1 := roaring.BitmapOf(1, 2, 3, 4, 5, 100, 1000)
    fmt.Println(rb1.String())

    rb2 := roaring.BitmapOf(3, 4, 1000)
    fmt.Println(rb2.String())

    rb3 := roaring.New()
    fmt.Println(rb3.String())

    fmt.Println("Cardinality: ", rb1.GetCardinality())

    fmt.Println("Contains 3? ", rb1.Contains(3))

    rb1.And(rb2)

    rb3.Add(1)
    rb3.Add(5)

    rb3.Or(rb1)

    // computes union of the three bitmaps in parallel using 4 workers  
    roaring.ParOr(4, rb1, rb2, rb3)
    // computes intersection of the three bitmaps in parallel using 4 workers  
    roaring.ParAnd(4, rb1, rb2, rb3)


    // prints 1, 3, 4, 5, 1000
    i := rb3.Iterator()
    for i.HasNext() {
        fmt.Println(i.Next())
    }
    fmt.Println()

    // next we include an example of serialization
    buf := new(bytes.Buffer)
    rb1.WriteTo(buf) // we omit error handling
    newrb:= roaring.New()
    newrb.ReadFrom(buf)
    if rb1.Equals(newrb) {
    	fmt.Println("I wrote the content to a byte stream and read it back.")
    }
    // you can iterate over bitmaps using ReverseIterator(), Iterator, ManyIterator()
}

If you wish to use serialization and handle errors, you might want to
consider the following sample of code:

rb := BitmapOf(1, 2, 3, 4, 5, 100, 1000)
	buf := new(bytes.Buffer)
	size,err:=rb.WriteTo(buf)
	if err != nil {
		t.Errorf("Failed writing")
	}
	newrb:= New()
	size,err=newrb.ReadFrom(buf)
	if err != nil {
		t.Errorf("Failed reading")
	}
	if ! rb.Equals(newrb) {
		t.Errorf("Cannot retrieve serialized version")
	}

Given N integers in [0,x), then the serialized size in bytes of
a Roaring bitmap should never exceed this bound:

8 + 9 * ((long)x+65535)/65536 + 2 * N

That is, given a fixed overhead for the universe size (x), Roaring
bitmaps never use more than 2 bytes per integer. You can call
BoundSerializedSizeInBytes for a more precise estimate.

64-bit Roaring

By default, roaring is used to stored unsigned 32-bit integers. However, we also offer
an extension dedicated to 64-bit integers. It supports roughly the same functions:

package main

import (
    "fmt"
    "github.com/RoaringBitmap/roaring/roaring64"
    "bytes"
)


func main() {
    // example inspired by https://github.com/fzandona/goroar
    fmt.Println("==roaring64==")
    rb1 := roaring64.BitmapOf(1, 2, 3, 4, 5, 100, 1000)
    fmt.Println(rb1.String())

    rb2 := roaring64.BitmapOf(3, 4, 1000)
    fmt.Println(rb2.String())

    rb3 := roaring64.New()
    fmt.Println(rb3.String())

    fmt.Println("Cardinality: ", rb1.GetCardinality())

    fmt.Println("Contains 3? ", rb1.Contains(3))

    rb1.And(rb2)

    rb3.Add(1)
    rb3.Add(5)

    rb3.Or(rb1)



    // prints 1, 3, 4, 5, 1000
    i := rb3.Iterator()
    for i.HasNext() {
        fmt.Println(i.Next())
    }
    fmt.Println()

    // next we include an example of serialization
    buf := new(bytes.Buffer)
    rb1.WriteTo(buf) // we omit error handling
    newrb:= roaring64.New()
    newrb.ReadFrom(buf)
    if rb1.Equals(newrb) {
    	fmt.Println("I wrote the content to a byte stream and read it back.")
    }
    // you can iterate over bitmaps using ReverseIterator(), Iterator, ManyIterator()
}

Only the 32-bit roaring format is standard and cross-operable between Java, C++, C and Go. There is no guarantee that the 64-bit versions are compatible.

Documentation

Current documentation is available at https://pkg.go.dev/github.com/RoaringBitmap/roaring and https://pkg.go.dev/github.com/RoaringBitmap/roaring/roaring64

Goroutine safety

In general, it should not generally be considered safe to access
the same bitmaps using different goroutines--they are left
unsynchronized for performance. Should you want to access
a Bitmap from more than one goroutine, you should
provide synchronization. Typically this is done by using channels to pass
the *Bitmap around (in Go style; so there is only ever one owner),
or by using sync.Mutex to serialize operations on Bitmaps.

Coverage

We test our software. For a report on our test coverage, see

https://coveralls.io/github/RoaringBitmap/roaring?branch=master

Benchmark

Type

text

go test -bench Benchmark -run -

To run benchmarks on Real Roaring Datasets
run the following:

go get github.com/RoaringBitmap/real-roaring-datasets
BENCH_REAL_DATA=1 go test -bench BenchmarkRealData -run -

Iterative use

You can use roaring with gore:

go get -u github.com/motemen/gore
Make sure that $GOPATH/bin is in your $PATH.
go get github.com/RoaringBitmap/roaring

$ gore
gore version 0.2.6  :help for help
gore> :import github.com/RoaringBitmap/roaring
gore> x:=roaring.New()
gore> x.Add(1)
gore> x.String()
"{1}"

Fuzzy testing

You can help us test further the library with fuzzy testing:

text

go get github.com/dvyukov/go-fuzz/go-fuzz
     go get github.com/dvyukov/go-fuzz/go-fuzz-build
     go test -tags=gofuzz -run=TestGenerateSmatCorpus
     go-fuzz-build github.com/RoaringBitmap/roaring
     go-fuzz -bin=./roaring-fuzz.zip -workdir=workdir/ -timeout=200 -func FuzzSmat

Let it run, and if the # of crashers is > 0, check out the reports in
the workdir where you should be able to find the panic goroutine stack
traces.

You may also replace -func FuzzSmat by -func FuzzSerializationBuffer or -func FuzzSerializationStream.

Alternative in Go

There is a Go version wrapping the C/C++ implementation https://github.com/RoaringBitmap/gocroaring

For an alternative implementation in Go, see https://github.com/fzandona/goroar
The two versions were written independently.

Mailing list/discussion group

https://groups.google.com/forum/#!forum/roaring-bitmaps

Overview (`vendor/github.com/axiomhq/hyperloglog/README.md`)

HyperLogLog - an algorithm for approximating the number of distinct elements

An improved version of HyperLogLog for the count-distinct problem, approximating the number of distinct elements in a multiset. This implementation offers enhanced performance, flexibility, and simplicity while maintaining accuracy.

Note on Implementation History

The initial version of this work (tagged as v0.1.0) was based on "Better with fewer bits: Improving the performance of cardinality estimation of large data streams - Qingjun Xiao, You Zhou, Shigang Chen". However, the current implementation has evolved significantly from this original basis, notably moving away from the tailcut method.

Current Implementation

The current implementation is based on the LogLog-Beta algorithm, as described in:

"LogLog-Beta and More: A New Algorithm for Cardinality Estimation Based on LogLog Counting" by Jason Qin, Denys Kim, and Yumei Tung (2016).

Key features of the current implementation:

Metro hash used instead of xxhash
Sparse representation for lower cardinalities (like HyperLogLog++)
LogLog-Beta for dynamic bias correction across all cardinalities
8-bit registers for convenience and simplified implementation
Order-independent insertions and merging for consistent results regardless of data input order
Removal of tailcut method for a more straightforward approach
Flexible precision allowing for 2^4 to 2^18 registers

This implementation is now more straightforward, efficient, and flexible, while remaining backwards compatible with previous versions. It provides a balance between precision, memory usage, speed, and ease of use.

Precision and Memory Usage

This implementation allows for creating HyperLogLog sketches with arbitrary precision between 2^4 and 2^18 registers. The memory usage scales with the number of registers:

Minimum (2^4 registers): 16 bytes
Default (2^14 registers): 16 KB
Maximum (2^18 registers): 256 KB

Users can choose the precision that best fits their use case, balancing memory usage against estimation accuracy.

Note

A big thank you to Prof. Shigang Chen and his team at the University of Florida who are actively conducting research around "Big Network Data".

Contributing

Kindly check our contributing guide on how to propose bugfixes and improvements, and submitting pull requests to the project

License

Distributed under MIT License (The MIT License).

See LICENSE for more information.

Overview (`vendor/github.com/bits-and-blooms/bitset/README.md`)

bitset

Go language library to map between non-negative integers and boolean values

This library is part of the awesome go collection. It is used in production by several important systems.

Description

Package bitset implements bitsets, a mapping between non-negative integers and boolean values.
It should be more efficient than map[uint] bool.

It provides methods for setting, clearing, flipping, and testing individual integers.

But it also provides set intersection, union, difference, complement, and symmetric operations, as well as tests to check whether any, all, or no bits are set, and querying a bitset's current length and number of positive bits.

BitSets are expanded to the size of the largest set bit; the memory allocation is approximately Max bits, where Max is the largest set bit. BitSets are never shrunk automatically, but Shrink and Compact methods are available. On creation, a hint can be given for the number of bits that will be used.

Many of the methods, including Set, Clear, and Flip, return a BitSet pointer, which allows for chaining.

Example use:

package main

import (
	"fmt"
	"math/rand"

	"github.com/bits-and-blooms/bitset"
)

func main() {
	fmt.Printf("Hello from BitSet!\n")
	var b bitset.BitSet
	// play some Go Fish
	for i := 0; i < 100; i++ {
		card1 := uint(rand.Intn(52))
		card2 := uint(rand.Intn(52))
		b.Set(card1)
		if b.Test(card2) {
			fmt.Println("Go Fish!")
		}
		b.Clear(card1)
	}

	// Chaining
	b.Set(10).Set(11)

	for i, e := b.NextSet(0); e; i, e = b.NextSet(i + 1) {
		fmt.Println("The following bit is set:", i)
	}
	if b.Intersection(bitset.New(100).Set(10)).Count() == 1 {
		fmt.Println("Intersection works.")
	} else {
		fmt.Println("Intersection doesn't work???")
	}
}

If you have Go 1.23 or better, you can iterate over the set bits like so:

for i := range b.EachSet() {}

Package documentation is at: https://pkg.go.dev/github.com/bits-and-blooms/bitset?tab=doc

Serialization

You may serialize a bitset safely and portably to a stream
of bytes as follows:

const length = 9585
	const oneEvery = 97
	bs := bitset.New(length)
	// Add some bits
	for i := uint(0); i < length; i += oneEvery {
		bs = bs.Set(i)
	}

	var buf bytes.Buffer
	n, err := bs.WriteTo(&buf)
	if err != nil {
		// failure
	}
	// Here n == buf.Len()

You can later deserialize the result as follows:

// Read back from buf
	bs = bitset.New()
	n, err = bs.ReadFrom(&buf)
	if err != nil {
		// error
	}
	// n is the number of bytes read

The ReadFrom function attempts to read the data into the existing
BitSet instance, to minimize memory allocations.

Performance tip:
When reading and writing to a file or a network connection, you may get better performance by
wrapping your streams with bufio instances.

E.g.,

f, err := os.Create("myfile")
	w := bufio.NewWriter(f)

f, err := os.Open("myfile")
	r := bufio.NewReader(f)

Memory Usage

The memory usage of a bitset using N bits is at least N/8 bytes. The number of bits in a bitset is at least as large as one plus the greatest bit index you have accessed. Thus it is possible to run out of memory while using a bitset. If you have lots of bits, you might prefer compressed bitsets, like the Roaring bitmaps and its Go implementation.

The roaring library allows you to go back and forth between compressed Roaring bitmaps and the conventional bitset instances:

mybitset := roaringbitmap.ToBitSet()
			newroaringbitmap := roaring.FromBitSet(mybitset)

Goroutine safety

In general, it's not safe to access the same BitSet using different goroutines--they are unsynchronized for performance.

Should you want to access a BitSet from more than one goroutine, you should provide synchronization. Typically this is done by using channels to pass the *BitSet around (in Go style; so there is only ever one owner), or by using sync.Mutex to serialize operations on BitSets.

Installation

bash

go get github.com/bits-and-blooms/bitset

Users

Databases & Storage

milvus-io/milvus — popular open-source vector database
cubefs/cubefs — CNCF distributed filesystem (used at JD.com)
sourcenetwork/defradb — decentralized document database
siglens/siglens — open-source observability/log analytics platform
ByteStorage/FlyDB — high-performance KV store

Search & Indexing

blevesearch/bleve — full-text search library for Go
blevesearch/vellum — FST library used internally by Bleve
blugelabs/bluge — modern Go search library (successor to Bleve)
geange/lucene-go — Lucene port to Go

Blockchain & Cryptography

consensys/gnark — zk-SNARK proving system
consensys/gnark-crypto — elliptic curve cryptography for ZK proofs
NethermindEth/juno — StarkNet full node
onflow/cadence — Flow blockchain smart contract language runtime
cosmos/cosmos-sdk — Cosmos blockchain SDK (slashing module)
babylonlabs-io/babylon — Bitcoin staking protocol
bnb-chain/greenfield — BNB Chain decentralized storage

Infrastructure & Networking

moby/swarmkit — Docker Swarm orchestration (ID management)
coredhcp/coredhcp — pluggable DHCP server (bitmap IP allocation)
Scalingo/sand — private networking for containers (IP allocation)
AliyunContainerService/terway — Alibaba Cloud K8s CNI
pomerium/pomerium — identity-aware access proxy

Distributed Systems & Messaging

apache/pulsar-client-go — official Apache Pulsar Go client
d7y.io/dragonfly — CNCF P2P file distribution system
RoaringBitmap/roaring — compressed bitmap library (interops with bitset)
zhenghaoz/gorse — Go recommendation engine

Machine Learning / Vector Search

fogfish/hnsw — HNSW approximate nearest-neighbor graph
semafind/semadb — vector database

Developer Tools & Security

bearer/bearer — security code scanner (AST traversal)
zrepl/zrepl — ZFS replication tool
NVIDIA/dcgm-exporter — NVIDIA GPU Prometheus exporter
NVIDIA/go-dcgm — NVIDIA datacenter GPU management bindings
gohugoio/hugo — the Hugo static site generator
godoctor/godoctor — Go refactoring tool (dataflow analysis)

Contributing

If you wish to contribute to this project, please branch and issue a pull request against master ("GitHub Flow")

Running all tests

Before committing the code, please check if it passes tests, has adequate coverage, etc.

bash

go test
go test -cover

This fork...

I'm maintaining this fork because the original author was not replying to issues or pull requests. For now I plan on maintaining this fork as necessary.

Status

Go Porter Stemmer

A native Go clean room implementation of the Porter Stemming Algorithm.

This algorithm is of interest to people doing Machine Learning or
Natural Language Processing (NLP).

This is NOT a port. This is a native Go implementation from the human-readable
description of the algorithm.

I've tried to make it (more) efficient by NOT internally using string's, but
instead internally using []rune's and using the same (array) buffer used by
the []rune slice (and sub-slices) at all steps of the algorithm.

For Porter Stemmer algorithm, see:

http://tartarus.org/martin/PorterStemmer/def.txt (URL #1)

http://tartarus.org/martin/PorterStemmer/ (URL #2)

Departures

Also, since when I initially implemented it, it failed the tests at...

http://tartarus.org/martin/PorterStemmer/voc.txt (URL #3)

http://tartarus.org/martin/PorterStemmer/output.txt (URL #4)

... after reading the human-readble text over and over again to try to figure out
what the error I made was (and doing all sorts of things to debug it) I came to the
conclusion that the some of these tests were wrong according to the human-readable
description of the algorithm.

This led me to wonder if maybe other people's code that was passing these tests had
rules that were not in the human-readable description. Which led me to look at the source
code here...

http://tartarus.org/martin/PorterStemmer/c.txt (URL #5)

... When I looked there I noticed that there are some items marked as a "DEPARTURE",
which differ from the original algorithm. (There are 2 of these.)

I implemented these departures, and the tests at URL #3 and URL #4 all passed.

Usage

To use this Golang library, use with something like:

text

package main

import (
  "fmt"
  "github.com/reiver/go-porterstemmer"
)

func main() {
  
  word := "Waxes"
  
  stem := porterstemmer.StemString(word)
  
  fmt.Printf("The word [%s] has the stem [%s].\n", word, stem)
}

Alternatively, if you want to be a bit more efficient, use []rune slices instead, with code like:

text

package main

import (
  "fmt"
  "github.com/reiver/go-porterstemmer"
)

func main() {
  
  word := []rune("Waxes")
  
  stem := porterstemmer.Stem(word)
  
  fmt.Printf("The word [%s] has the stem [%s].\n", string(word), string(stem))
}

Although NOTE that the above code may modify original slice (named "word" in the example) as a side
effect, for efficiency reasons. And that the slice named "stem" in the example above may be a
sub-slice of the slice named "word".

Also alternatively, if you already know that your word is already lowercase (and you don't need
this library to lowercase your word for you) you can instead use code like:

text

package main

import (
  "fmt"
  "github.com/reiver/go-porterstemmer"
)

func main() {
  
  word := []rune("waxes")
  
  stem := porterstemmer.StemWithoutLowerCasing(word)
  
  fmt.Printf("The word [%s] has the stem [%s].\n", string(word), string(stem))
}

Again NOTE (like with the previous example) that the above code may modify original slice (named
"word" in the example) as a side effect, for efficiency reasons. And that the slice named "stem"
in the example above may be a sub-slice of the slice named "word".

Overview (`vendor/github.com/blevesearch/mmap-go/README.md`)

mmap-go

mmap-go is a portable mmap package for the Go programming language.

Operating System Support

This package is tested using GitHub Actions on Linux, macOS, and Windows. It should also work on other Unix-like platforms, but hasn't been tested with them. I'm interested to hear about the results.

This package compiles for Plan 9 and WebAssembly, but its functions always return errors.

Related functions such as mprotect and mincore aren't included. I haven't found a way to implement them on Windows without introducing significant complexity. If you're running on a Unix-like platform and really need these features, it should still be possible to implement them on top of this package via syscall.

Overview (`vendor/github.com/blevesearch/segment/README.md`)

segment

A Go library for performing Unicode Text Segmentation
as described in Unicode Standard Annex #29

Features

Currently only segmentation at Word Boundaries is supported.

License

Apache License Version 2.0

Usage

The functionality is exposed in two ways:

You can use a bufio.Scanner with the SplitWords implementation of SplitFunc.
The SplitWords function will identify the appropriate word boundaries in the input
text and the Scanner will return tokens at the appropriate place.
text
```
scanner := bufio.NewScanner(...)
scanner.Split(segment.SplitWords)
for scanner.Scan() {
    tokenBytes := scanner.Bytes()
}
if err := scanner.Err(); err != nil {
    t.Fatal(err)
}
```
Sometimes you would also like information returned about the type of token.
To do this we have introduce a new type named Segmenter. It works just like Scanner
but additionally a token type is returned.
text
```
segmenter := segment.NewWordSegmenter(...)
for segmenter.Segment() {
    tokenBytes := segmenter.Bytes())
    tokenType := segmenter.Type()
}
if err := segmenter.Err(); err != nil {
    t.Fatal(err)
}
```

Choosing Implementation

By default segment does NOT use the fastest runtime implementation. The reason is that it adds approximately 5s to compilation time and may require more than 1GB of ram on the machine performing compilation.

However, you can choose to build with the fastest runtime implementation by passing the build tag as follows:

text

-tags 'prod'

Generating Code

Several components in this package are generated.

Several Ragel rules files are generated from Unicode properties files.
Ragel machine is generated from the Ragel rules.
Test tables are generated from the Unicode test files.

All of these can be generated by running:

text

go generate

Fuzzing

There is support for fuzzing the segment library with go-fuzz.

Install go-fuzz if you haven't already:

text

go get github.com/dvyukov/go-fuzz/go-fuzz
go get github.com/dvyukov/go-fuzz/go-fuzz-build

Build the package with go-fuzz:

text

go-fuzz-build github.com/blevesearch/segment

Convert the Unicode provided test cases into the initial corpus for go-fuzz:
text
```
go test -v -run=TestGenerateWordSegmentFuzz -tags gofuzz_generate
```

Run go-fuzz:

text

go-fuzz -bin=segment-fuzz.zip -workdir=workdir

Status

Overview (`vendor/github.com/blevesearch/snowballstem/README.md`)

snowballstem

This repository contains the Go stemmers generated by the Snowball project. They are maintained outside of the core bleve package so that they may be more easily be reused in other contexts.

Usage

All these stemmers export a single Stem() method which operates on a snowball Env structure. The Env structure maintains all state for the stemmer. A new Env is created to point at an initial string. After stemming, the results of the Stem() operation can be retrieved using the Current() method. The Env structure can be reused for subsequent calls by using the SetCurrent() method.

Example

text

package main

import (
	"fmt"

	"github.com/blevesearch/snowballstem"
	"github.com/blevesearch/snowballstem/english"
)

func main() {

	// words to stem
	words := []string{
		"running",
		"jumping",
	}

	// build new environment
	env := snowballstem.NewEnv("")

	for _, word := range words {
		// set up environment for word
		env.SetCurrent(word)
		// invoke stemmer
		english.Stem(env)
		// print results
		fmt.Printf("%s stemmed to %s\n", word, env.Current())
	}
}

Produces Output:

text

$ ./snowtest
running stemmed to run
jumping stemmed to jump

Testing

The test harness for these stemmers is hosted in the main Snowball repository. There are functional tests built around the separate snowballstem-data repository, and there is support for fuzz-testing the stemmers there as well.

Generating the Stemmers

text

$ export SNOWBALL=/path/to/github.com/snowballstem/snowball/after/snowball/built
$ go generate

Updated the Go Generate Commands

A simple tool is provided to automate these from the snowball algorithms directory:

text

$ go run gengen.go /path/to/github.com/snowballstem/snowball/algorithms

Overview (`vendor/github.com/blevesearch/vellum/README.md`)

vellum

A Go library implementing an FST (finite state transducer) capable of:

mapping between keys ([]byte) and a value (uint64)
enumerating keys in lexicographic order

Some additional goals of this implementation:

bounded memory use while building the FST
streaming out FST data while building
mmap FST runtime to support very large FTSs (optional)

Usage

Building an FST

To build an FST, create a new builder using the New() method. This method takes an io.Writer as an argument. As the FST is being built, data will be streamed to the writer as soon as possible. With this builder you MUST insert keys in lexicographic order. Inserting keys out of order will result in an error. After inserting the last key into the builder, you MUST call Close() on the builder. This will flush all remaining data to the underlying writer.

In memory:

var buf bytes.Buffer
builder, err := vellum.New(&buf, nil)
if err != nil {
  log.Fatal(err)
}

To disk:

f, err := os.Create("/tmp/vellum.fst")
if err != nil {
  log.Fatal(err)
}
builder, err := vellum.New(f, nil)
if err != nil {
  log.Fatal(err)
}

MUST insert keys in lexicographic order:

err = builder.Insert([]byte("cat"), 1)
if err != nil {
  log.Fatal(err)
}

err = builder.Insert([]byte("dog"), 2)
if err != nil {
  log.Fatal(err)
}

err = builder.Insert([]byte("fish"), 3)
if err != nil {
  log.Fatal(err)
}

err = builder.Close()
if err != nil {
  log.Fatal(err)
}

Using an FST

After closing the builder, the data can be used to instantiate an FST. If the data was written to disk, you can use the Open() method to mmap the file. If the data is already in memory, or you wish to load/mmap the data yourself, you can instantiate the FST with the Load() method.

Load in memory:

fst, err := vellum.Load(buf.Bytes())
if err != nil {
  log.Fatal(err)
}

Open from disk:

fst, err := vellum.Open("/tmp/vellum.fst")
if err != nil {
  log.Fatal(err)
}

Get key/value:

val, exists, err = fst.Get([]byte("dog"))
if err != nil {
  log.Fatal(err)
}
if exists {
  fmt.Printf("contains dog with val: %d\n", val)
} else {
  fmt.Printf("does not contain dog")
}

Iterate key/values:

itr, err := fst.Iterator(startKeyInclusive, endKeyExclusive)
for err == nil {
  key, val := itr.Current()
  fmt.Printf("contains key: %s val: %d", key, val)
  err = itr.Next()
}
if err != nil {
  log.Fatal(err)
}

How does the FST get built?

A full example of the implementation is beyond the scope of this README, but let's consider a small example where we want to insert 3 key/value pairs.

First we insert "are" with the value 4.

Next, we insert "ate" with the value 2.

Notice how the values associated with the transitions were adjusted so that by summing them while traversing we still get the expected value.

At this point, we see that state 5 looks like state 3, and state 4 looks like state 2. But, we cannot yet combine them because future inserts could change this.

Now, we insert "see" with value 3. Once it has been added, we now know that states 5 and 4 can longer change. Since they are identical to 3 and 2, we replace them.

Again, we see that states 7 and 8 appear to be identical to 2 and 3.

Having inserted our last key, we call Close() on the builder.

Now, states 7 and 8 can safely be replaced with 2 and 3.

For additional information, see the references at the bottom of this document.

What does the serialized format look like?

We've broken out a separate document on the vellum disk format v1.

What if I want to use this on a system that doesn't have mmap?

The mmap library itself is guarded with system/architecture build tags, but we've also added an additional build tag in vellum. If you'd like to Open() a file based representation of an FST, but not use mmap, you can build the library with the nommap build tag. NOTE: if you do this, the entire FST will be read into memory.

Can I use this with Unicode strings?

Yes, however this implementation is only aware of the byte representation you choose. In order to find matches, you must work with some canonical byte representation of the string. In the future, some encoding-aware traversals may be possible on top of the lower-level byte transitions.

How did this library come to be?

In my work on the Bleve project I became aware of the power of the FST for many search-related tasks. The obvious starting point for such a thing in Go was the mafsa project. While working with mafsa I encountered some issues. First, it did not stream data to disk while building. Second, it chose to use a rune as the fundamental unit of transition in the FST, but I felt using a byte would be more powerful in the end. My hope is that higher-level encoding-aware traversals will be possible when necessary. Finally, as I reported bugs and submitted PRs I learned that the mafsa project was mainly a research project and no longer being maintained. I wanted to build something that could be used in production. As the project advanced more and more techniques from the BurntSushi/fst were adapted to our implementation.

Are there tools to work with vellum files?

Under the cmd/vellum subdirectory, there's a command-line tool which
features subcommands that can allow you to create, inspect and query
vellum files.

How can I generate a state transition diagram from a vellum file?

The vellum command-line tool has a "dot" subcommand that can emit
graphviz dot output data from an input vellum file. The dot file can
in turn be converted into an image using graphviz tools. Example...

shell

vellum dot myFile.vellum > output.dot
dot -Tpng output.dot -o output.png

Much credit goes to two existing projects:

Most of the original implementation here started with my digging into the internals of mafsa. As the implementation progressed, I continued to borrow ideas/approaches from the BurntSushi/fst library as well.

For a great introduction to this topic, please read the blog post Index 1,600,000,000 Keys with Automata and Rust

Overview (`vendor/github.com/blugelabs/bluge/README.md`)

Bluge

modern text indexing in go - blugelabs.com

Features

Supported field types:
- Text, Numeric, Date, Geo Point
Supported query types:
- Term, Phrase, Match, Match Phrase, Prefix
- Conjunction, Disjunction, Boolean
- Numeric Range, Date Range
BM25 Similarity/Scoring with pluggable interfaces
Search result match highlighting
Extendable Aggregations:
- Bucketing
  - Terms
  - Numeric Range
  - Date Range
- Metrics
  - Min/Max/Count/Sum
  - Avg/Weighted Avg
  - Cardinality Estimation (HyperLogLog++)
  - Quantile Approximation (T-Digest)

Indexing

config := bluge.DefaultConfig(path)
    writer, err := bluge.OpenWriter(config)
    if err != nil {
        log.Fatalf("error opening writer: %v", err)
    }
    defer writer.Close()

    doc := bluge.NewDocument("example").
        AddField(bluge.NewTextField("name", "bluge"))

    err = writer.Update(doc.ID(), doc)
    if err != nil {
        log.Fatalf("error updating document: %v", err)
    }

Querying

reader, err := writer.Reader()
    if err != nil {
        log.Fatalf("error getting index reader: %v", err)
    }
    defer reader.Close()

    query := bluge.NewMatchQuery("bluge").SetField("name")
    request := bluge.NewTopNSearch(10, query).
        WithStandardAggregations()
    documentMatchIterator, err := reader.Search(context.Background(), request)
    if err != nil {
        log.Fatalf("error executing search: %v", err)
    }
    match, err := documentMatchIterator.Next()
    for err == nil && match != nil {
        err = match.VisitStoredFields(func(field string, value []byte) bool {
            if field == "_id" {
                fmt.Printf("match: %s\n", string(value))
            }
            return true
        })
        if err != nil {
            log.Fatalf("error loading stored fields: %v", err)
        }
        match, err = documentMatchIterator.Next()
    }
    if err != nil {
        log.Fatalf("error iterator document matches: %v", err)
    }

Repobeats

License

Apache License Version 2.0

Overview (`vendor/github.com/blugelabs/bluge_segment_api/README.md`)

Bluge Segment API

These interfaces define the relationship between Bluge and it's segment file formats.

The expectation is that these interfaces can be well-defined, and evolve slowly. By doing so, Bluge and the file formats themselves can evolve more quickly, and yet remain compatible.

Overview (`vendor/github.com/blugelabs/ice/README.md`)

ice

The file is written in the reverse order that we typically access data. This helps us write in one pass since later sections of the file require file offsets of things we've already written.

Current usage:

crc-32 bytes and version are in fixed position at the end of the file
reading remainder of footer could be version specific
remainder of footer gives us:
- 3 important offsets (docValue, fields index and stored data index)
- 2 important values (number of docs and chunk factor)
field data is processed once and memoized onto the heap so that we never have to go back to disk for it
access to stored data by doc number means first navigating to the stored data index, then accessing a fixed position offset into that slice, which gives us the actual address of the data. the first bytes of that section tell us the size of data so that we know where it ends.
access to all other indexed data follows the following pattern:
- first know the field name -> convert to id
- next navigate to term dictionary for that field
  - some operations stop here and do dictionary ops
- next use dictionary to navigate to posting list for a specific term
- walk posting list
- if necessary, walk posting details as we go
- if location info is desired, consult location bitmap to see if it is there

stored fields section

for each document
- preparation phase:
  - produce a slice of metadata bytes and data bytes
  - produce these slices in field id order
  - field value is appended to the data slice
  - metadata slice is varint encoded with the following values for each field value
    - field id (uint16)
    - field value start offset in uncompressed data slice (uint64)
    - field value length (uint64)
    - compress the data slice using snappy
- file writing phase:
  - remember the start offset for this document
  - write out meta data length (varint uint64)
  - write out compressed data length (varint uint64)
  - write out the metadata bytes
  - write out the compressed data bytes

stored fields idx

for each document
- write start offset (remembered from previous section) of stored data (big endian uint64)

With this index and a known document number, we have direct access to all the stored field data.

posting details (freq/norm) section

for each posting list
- produce a slice containing multiple consecutive chunks (each chunk is varint stream)
- produce a slice remembering offsets of where each chunk starts
- preparation phase:
  - for each hit in the posting list
  - if this hit is in next chunk close out encoding of last chunk and record offset start of next
  - encode term frequency (uint64)
  - encode norm factor (float32) - similarity specific implementation
- file writing phase:
  - remember start position for this posting list details
  - write out number of chunks that follow (varint uint64)
  - write out length of each chunk (each a varint uint64)
  - write out the byte slice containing all the chunk data

If you know the doc number you're interested in, this format lets you jump to the correct chunk (docNum/chunkFactor) directly and then seek within that chunk until you find it.

posting details (location) section

for each posting list
- produce a slice containing multiple consecutive chunks (each chunk is varint stream)
- produce a slice remembering offsets of where each chunk starts
- preparation phase:
  - for each hit in the posting list
  - if this hit is in next chunk close out encoding of last chunk and record offset start of next
  - encode field (uint16)
  - encode field pos (uint64)
  - encode field start (uint64)
  - encode field end (uint64)
- file writing phase:
  - remember start position for this posting list details
  - write out number of chunks that follow (varint uint64)
  - write out length of each chunk (each a varint uint64)
  - write out the byte slice containing all the chunk data

If you know the doc number you're interested in, this format lets you jump to the correct chunk (docNum/chunkFactor) directly and then seek within that chunk until you find it.

postings list section

for each posting list
- preparation phase:
  - encode roaring bitmap posting list to bytes (so we know the length)
- file writing phase:
  - remember the start position for this posting list
  - write freq/norm details offset (remembered from previous, as varint uint64)
  - write location details offset (remembered from previous, as varint uint64)
  - write length of encoded roaring bitmap
  - write the serialized roaring bitmap data

dictionary

for each field
- preparation phase:
  - encode vellum FST with dictionary data pointing to file offset of posting list (remembered from previous)
- file writing phase:
  - remember the start position of this persistDictionary
  - write length of vellum data (varint uint64)
  - write out vellum data

fields section

for each field
- file writing phase:
  - remember start offset for each field
  - write dictionary address (remembered from previous) (varint uint64)
  - write length of field name (varint uint64)
  - write field name bytes

fields idx

for each field
- file writing phase:
  - write big endian uint64 of start offset for each field

NOTE: currently we don't know or record the length of this fields index. Instead we rely on the fact that we know it immediately precedes a footer of known size.

fields DocValue

for each field
- preparation phase:
  - produce a slice containing multiple consecutive chunks, where each chunk is composed of a meta section followed by compressed columnar field data
  - produce a slice remembering the length of each chunk
- file writing phase:
  - remember the start position of this first field DocValue offset in the footer
  - write out number of chunks that follow (varint uint64)
  - write out length of each chunk (each a varint uint64)
  - write out the byte slice containing all the chunk data

NOTE: currently the meta header inside each chunk gives clue to the location offsets and size of the data pertaining to a given docID and any
read operation leverage that meta information to extract the document specific data from the file.

file writing phase
- write number of docs (big endian uint64)
- write stored field index location (big endian uint64)
- write field index location (big endian uint64)
- write field docValue location (big endian uint64)
- write out chunk factor (big endian uint32)
- write out version (big endian uint32)
- write out file CRC of everything preceding this (big endian uint32)

ice file format diagrams

Legend

Sections

text

|========|
|        | section
|========|

Fixed-size fields

text

|--------|        |----|        |--|        |-|
|        | uint64 |    | uint32 |  | uint16 | | uint8
|--------|        |----|        |--|        |-|

Varints

text

|~~~~~~~~|
|        | varint(up to uint64)
|~~~~~~~~|

Arbitrary-length fields

text

|--------...---|
|              | arbitrary-length field (string, vellum, roaring bitmap)
|--------...---|

Chunked data

text

[--------]
[        ]
[--------]

Overview

Footer section describes the configuration of particular ice file. The format of footer is version-dependent, so it is necessary to check V field before the parsing.

text

|==================================================|
        | Stored Fields                                    |
        |==================================================|
|-----> | Stored Fields Index                              |
|       |==================================================|   
|       | Dictionaries + Postings + DocValues              | 
|       |==================================================|
| |---> | DocValues Index                                  |
| |     |==================================================|   
| |     | Fields                                           |
| |     |==================================================|
| | |-> | Fields Index                                     |
| | |   |========|========|========|========|====|====|====|
| | |   |     D# |     SF |      F |    FDV | CF |  V | CC | (Footer)
| | |   |========|====|===|====|===|====|===|====|====|====|
| | |                 |        |        |
|-+-+-----------------|        |        |
  | |--------------------------|        |
  |-------------------------------------|

 D#. Number of Docs.
 SF. Stored Fields Index Offset.
  F. Field Index Offset.
FDV. Field DocValue Offset.
 CF. Chunk Factor.
  V. Version.
 CC. CRC32.

Stored Fields

Stored Fields Index is D# consecutive 64-bit unsigned integers - offsets, where relevant Stored Fields Data records are located.

text

0                                [SF]                   [SF + D# * 8]
| Stored Fields                  | Stored Fields Index              |
|================================|==================================|
|                                |                                  |
|       |--------------------|   ||--------|--------|. . .|--------||
|   |-> | Stored Fields Data |   ||      0 |      1 |     | D# - 1 ||
|   |   |--------------------|   ||--------|----|---|. . .|--------||
|   |                            |              |                   |
|===|============================|==============|===================|
    |                                           |
    |-------------------------------------------|

Stored Fields Data is an arbitrary size record, which consists of metadata and Snappy-compressed data.

text

Stored Fields Data
|~~~~~~~~|~~~~~~~~|~~~~~~~~...~~~~~~~~|~~~~~~~~...~~~~~~~~|
|    MDS |    CDS |                MD |                CD |
|~~~~~~~~|~~~~~~~~|~~~~~~~~...~~~~~~~~|~~~~~~~~...~~~~~~~~|

MDS. Metadata size.
CDS. Compressed data size.
MD. Metadata.
CD. Snappy-compressed data.

Fields

Fields Index section located between addresses F and len(file) - len(footer) and consist of uint64 values (F1, F2, ...) which are offsets to records in Fields section. We have F# = (len(file) - len(footer) - F) / sizeof(uint64) fields.

text

(...)                                              [F]                       [F + F#]
| Fields                                           | Fields Index.                  |
|==================================================|================================|
|                                                  |                                |
|   |~~~~~~~~|~~~~~~~~|---...---|~~~~~~~~|~~~~~~~~|||--------|--------|...|--------||
||->|   Dict | Length |    Name | # Docs |Tot Freq|||      0 |      1 |   | F# - 1 ||
||  |~~~~~~~~|~~~~~~~~|---...---|~~~~~~~~|~~~~~~~~|||--------|----|---|...|--------||
||                                                 |              |                 |
||=================================================|==============|=================|
 |                                                                |
 |----------------------------------------------------------------|

Dictionaries + Postings

Each of fields has its own dictionary, encoded in Vellum format. Dictionary consists of pairs (term, offset), where offset indicates the position of postings (list of documents) for this particular term.

text

|================================================================|- Dictionaries + 
|                                                                |   Postings +
|                                                                |    DocValues
|    Freq/Norm (chunked)                                         |
|    [~~~~~~|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~]                      |
| |->[ Freq | Norm (float32 under varint) ]                      |
| |  [~~~~~~|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~]                      |
| |                                                              |
| |------------------------------------------------------------| |
|    Location Details (chunked)                                | |
|    [~~~~~~|~~~~~|~~~~~~~|~~~~~]                              | |
| |->[ Size | Pos | Start | End ]                              | |
| |  [~~~~~~|~~~~~|~~~~~~~|~~~~~]                              | |
| |                                                            | |
| |----------------------|                                     | |
|          Postings List |                                     | |
|         |~~~~~~~~|~~~~~|~~|~~~~~~~~|-----------...--|        | |
|      |->|    F/N |     LD | Length | ROARING BITMAP |        | |
|      |  |~~~~~|~~|~~~~~~~~|~~~~~~~~|-----------...--|        | |
|      |        |----------------------------------------------| |
|      |--------------------------------------|                  |
|          Dictionary                         |                  |
|         |~~~~~~~~|--------------------------|-...-|            |
|      |->| Length | VELLUM DATA : (TERM -> OFFSET) |            |
|      |  |~~~~~~~~|----------------------------...-|            |
|      |                                                         |
|======|=========================================================|- DocValues Index
|      |                                                         |
|======|=========================================================|- Fields
|      |                                                         |
| |~~~~|~~~|~~~~~~~~|---...---|                                  |
| |   Dict | Length |    Name |                                  |
| |~~~~~~~~|~~~~~~~~|---...---|                                  |
|                                                                |
|================================================================|

DocValues

DocValues Index is F# pairs of varints, one pair per field. Each pair of varints indicates start and end point of DocValues slice.

text

|================================================================|
|     |------...--|                                              |
|  |->| DocValues |<-|                                           |
|  |  |------...--|  |                                           |
|==|=================|===========================================|- DocValues Index
||~|~~~~~~~~~|~~~~~~~|~~|           |~~~~~~~~~~~~~~|~~~~~~~~~~~~||
|| DV1 START | DV1 STOP | . . . . . | DV(F#) START | DV(F#) END ||
||~~~~~~~~~~~|~~~~~~~~~~|           |~~~~~~~~~~~~~~|~~~~~~~~~~~~||
|================================================================|

DocValues is chunked Snappy-compressed values for each document and field.

text

[~~~~~~~~~~~~~~~|~~~~~~|~~~~~~~~~|-...-|~~~~~~|~~~~~~~~~|--------------------...-]
[ Doc# in Chunk | Doc1 | Offset1 | ... | DocN | OffsetN | SNAPPY COMPRESSED DATA ]
[~~~~~~~~~~~~~~~|~~~~~~|~~~~~~~~~|-...-|~~~~~~|~~~~~~~~~|--------------------...-]

Last 16 bytes are description of chunks.

text

|~~~~~~~~~~~~...~|----------------|----------------|
|   Chunk Sizes  | Chunk Size Arr |         Chunk# |
|~~~~~~~~~~~~...~|----------------|----------------|

Overview (`vendor/github.com/blugelabs/query_string/README.md`)

query_string

An adaptation of the class Bleve query string functionality for Bluge.

Overview (`vendor/github.com/caio/go-tdigest/README.md`)

T-Digest

A fast map-reduce and parallel streaming friendly data-structure for accurate
quantile approximation.

This package provides an implementation of Ted Dunning's t-digest data
structure in Go.

Project Status

This project is actively maintained. We are happy to collaborate on features
and issues if/when they arrive.

Installation

Our releases are tagged and signed following the Semantic Versioning
scheme. If you are using a dependency manager such as dep, the recommended
way to is go about your business normally:

text

go get github.com/caio/go-tdigest

Otherwise we recommend to use the following so that you don't risk breaking
your build because of an API change:

text

go get gopkg.in/caio/go-tdigest.v2

Example Usage

package main

import (
	"fmt"
	"math/rand"

	"github.com/caio/go-tdigest"
)

func main() {
	// Analogue to tdigest.New(tdigest.Compression(100))
	t, _ := tdigest.New()

	for i := 0; i < 10000; i++ {
		// Analogue to t.AddWeighted(rand.Float64(), 1)
		t.Add(rand.Float64())
	}

	fmt.Printf("p(.5) = %.6f\n", t.Quantile(0.5))
	fmt.Printf("CDF(Quantile(.5)) = %.6f\n", t.CDF(t.Quantile(0.5)))
}

Configuration

You can configure your digest upon creation with options documented
at options.go. Example:

// Construct a digest with compression=200 and its own
// (thread-unsafe) RNG seeded with 0xCA10:
digest, _ := tdigest.New(
        tdigest.Compression(200),
        tdigest.LocalRandomNumberGenerator(0xCA10),
)

Porting Existing Code to the v2 API

It's very easy to migrate to the new API:

Replace tdigest.New(100) with tdigest.New()
Replace tdigest.New(number) with tdigest.New(tdigest.Compression(number))
Replace Add(x,1) with Add(x)
Replace Add(x, weight) with AddWeighted(x, weight)
Remove any use of tdigest.Len() (or open an issue)

References

This is a port of the reference implementation with some ideas borrowed
from the python version. If you wanna get a quick grasp of how it works
and why it's useful, this video and companion article is pretty helpful.

Overview (`vendor/github.com/dgryski/dgoogauth/README.md`)

This is a Go implementation of the Google Authenticator library.

It implements the one-time-password algorithms specified in:

RFC 4226 (HOTP: An HMAC-Based One-Time Password Algorithm)
RFC 6238 (TOTP: Time-Based One-Time Password Algorithm)

You can learn more about the Google Authenticator library at its project page:

https://github.com/google/google-authenticator

Overview (`vendor/github.com/dop251/goja/README.md`)

goja

ECMAScript 5.1(+) implementation in Go.

Goja is an implementation of ECMAScript 5.1 in pure Go with emphasis on standard compliance and
performance.

This project was largely inspired by otto.

The minimum required Go version is 1.25.

Features

Full ECMAScript 5.1 support (including regex and strict mode).
Passes nearly all tc39 tests for the features implemented so far. The goal is to
pass all of them. See .tc39_test262_checkout.sh for the latest working commit id.
Capable of running Babel, Typescript compiler and pretty much anything written in ES5.
Sourcemaps.
Most of ES6 functionality, still work in progress, see https://github.com/dop251/goja/milestone/1?closed=1

Known incompatibilities and caveats

JSON

JSON.parse() uses the standard Go library which operates in UTF-8. Therefore, it cannot correctly parse broken UTF-16
surrogate pairs, for example:

javascript

JSON.parse(`"\\uD800"`).charCodeAt(0).toString(16) // returns "fffd" instead of "d800"

Date

Conversion from calendar date to epoch timestamp uses the standard Go library which uses int, rather than float as per
ECMAScript specification. This means if you pass arguments that overflow int to the Date() constructor or if there is
an integer overflow, the result will be incorrect, for example:

javascript

Date.UTC(1970, 0, 1, 80063993375, 29, 1, -288230376151711740) // returns 29256 instead of 29312

FAQ

How fast is it?

Although it's faster than many scripting language implementations in Go I have seen
(for example it's 6-7 times faster than otto on average) it is not a
replacement for V8 or SpiderMonkey or any other general-purpose JavaScript engine.
You can find some benchmarks here.

Why would I want to use it over a V8 wrapper?

It greatly depends on your usage scenario. If most of the work is done in javascript
(for example crypto or any other heavy calculations) you are definitely better off with V8.

If you need a scripting language that drives an engine written in Go so that
you need to make frequent calls between Go and javascript passing complex data structures
then the cgo overhead may outweigh the benefits of having a faster javascript engine.

Because it's written in pure Go there are no cgo dependencies, it's very easy to build and it
should run on any platform supported by Go.

It gives you a much better control over execution environment so can be useful for research.

Is it goroutine-safe?

No. An instance of goja.Runtime can only be used by a single goroutine
at a time. You can create as many instances of Runtime as you like but
it's not possible to pass object values between runtimes.

Where is setTimeout()/setInterval()?

setTimeout() and setInterval() are common functions to provide concurrent execution in ECMAScript environments, but the two functions are not part of the ECMAScript standard.
Browsers and NodeJS just happen to provide similar, but not identical, functions. The hosting application need to control the environment for concurrent execution, e.g. an event loop, and supply the functionality to script code.

There is a separate project aimed at providing some NodeJS functionality,
and it includes an event loop.

Can you implement (feature X from ES6 or higher)?

I will be adding features in their dependency order and as quickly as time permits. Please do not ask
for ETAs. Features that are open in the milestone are either in progress
or will be worked on next.

The ongoing work is done in separate feature branches which are merged into master when appropriate.
Every commit in these branches represents a relatively stable state (i.e. it compiles and passes all enabled tc39 tests),
however because the version of tc39 tests I use is quite old, it may be not as well tested as the ES5.1 functionality. Because there are (usually) no major breaking changes between ECMAScript revisions
it should not break your existing code. You are encouraged to give it a try and report any bugs found. Please do not submit fixes though without discussing it first, as the code could be changed in the meantime.

How do I contribute?

Before submitting a pull request please make sure that:

You followed ECMA standard as close as possible. If adding a new feature make sure you've read the specification,
do not just base it on a couple of examples that work fine.
Your change does not have a significant negative impact on performance (unless it's a bugfix and it's unavoidable)
It passes all relevant tc39 tests.

Current Status

There should be no breaking changes in the API, however it may be extended.
Some of the AnnexB functionality is missing.

Basic Example

Run JavaScript and get the result value.

vm := goja.New()
v, err := vm.RunString("2 + 2")
if err != nil {
    panic(err)
}
if num := v.Export().(int64); num != 4 {
    panic(num)
}

Passing Values to JS

Any Go value can be passed to JS using Runtime.ToValue() method. See the method's documentation for more details.

Exporting Values from JS

A JS value can be exported into its default Go representation using Value.Export() method.

Alternatively it can be exported into a specific Go variable using Runtime.ExportTo() method.

Within a single export operation the same Object will be represented by the same Go value (either the same map, slice or
a pointer to the same struct). This includes circular objects and makes it possible to export them.

Calling JS functions from Go

There are 2 approaches:

Using AssertFunction():

const SCRIPT = `
function sum(a, b) {
    return +a + b;
}
`

vm := goja.New()
_, err := vm.RunString(SCRIPT)
if err != nil {
    panic(err)
}
sum, ok := goja.AssertFunction(vm.Get("sum"))
if !ok {
    panic("Not a function")
}

res, err := sum(goja.Undefined(), vm.ToValue(40), vm.ToValue(2))
if err != nil {
    panic(err)
}
fmt.Println(res)
// Output: 42

Using Runtime.ExportTo():

const SCRIPT = `
function sum(a, b) {
    return +a + b;
}
`

vm := goja.New()
_, err := vm.RunString(SCRIPT)
if err != nil {
    panic(err)
}

var sum func(int, int) int
err = vm.ExportTo(vm.Get("sum"), &sum)
if err != nil {
    panic(err)
}

fmt.Println(sum(40, 2)) // note, _this_ value in the function will be undefined.
// Output: 42

The first one is more low level and allows specifying this value, whereas the second one makes the function look like
a normal Go function.

Mapping struct field and method names

By default, the names are passed through as is which means they are capitalised. This does not match
the standard JavaScript naming convention, so if you need to make your JS code look more natural or if you are
dealing with a 3rd party library, you can use a FieldNameMapper:

vm := goja.New()
vm.SetFieldNameMapper(TagFieldNameMapper("json", true))
type S struct {
    Field int `json:"field"`
}
vm.Set("s", S{Field: 42})
res, _ := vm.RunString(`s.field`) // without the mapper it would have been s.Field
fmt.Println(res.Export())
// Output: 42

There are two standard mappers: TagFieldNameMapper and
UncapFieldNameMapper, or you can use your own implementation.

Native Constructors

In order to implement a constructor function in Go use func (goja.ConstructorCall) *goja.Object.
See Runtime.ToValue() documentation for more details.

Regular Expressions

Goja uses the embedded Go regexp library where possible, otherwise it falls back to regexp2.

Exceptions

Any exception thrown in JavaScript is returned as an error of type *Exception. It is possible to extract the value thrown
by using the Value() method:

vm := goja.New()
_, err := vm.RunString(`

throw("Test");

`)

if jserr, ok := err.(*Exception); ok {
    if jserr.Value().Export() != "Test" {
        panic("wrong value")
    }
} else {
    panic("wrong type")
}

If a native Go function panics with a Value, it is thrown as a Javascript exception (and therefore can be caught):

var vm *Runtime

func Test() {
    panic(vm.ToValue("Error"))
}

vm = goja.New()
vm.Set("Test", Test)
_, err := vm.RunString(`

try {
    Test();
} catch(e) {
    if (e !== "Error") {
        throw e;
    }
}

`)

if err != nil {
    panic(err)
}

Interrupting

func TestInterrupt(t *testing.T) {
    const SCRIPT = `
    var i = 0;
    for (;;) {
        i++;
    }
    `

    vm := goja.New()
    time.AfterFunc(200 * time.Millisecond, func() {
        vm.Interrupt("halt")
    })

    _, err := vm.RunString(SCRIPT)
    if err == nil {
        t.Fatal("Err is nil")
    }
    // err is of type *InterruptError and its Value() method returns whatever has been passed to vm.Interrupt()
}

NodeJS Compatibility

There is a separate project aimed at providing some of the NodeJS functionality.

Overview (`vendor/github.com/felixge/httpsnoop/README.md`)

httpsnoop

Package httpsnoop provides an easy way to capture http related metrics (i.e.
response time, bytes written, and http status code) from your application's
http.Handlers.

Doing this requires non-trivial wrapping of the http.ResponseWriter interface,
which is also exposed for users interested in a more low-level API.

Usage Example

// myH is your app's http handler, perhaps a http.ServeMux or similar.
var myH http.Handler
// wrappedH wraps myH in order to log every request.
wrappedH := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	m := httpsnoop.CaptureMetrics(myH, w, r)
	log.Printf(
		"%s %s (code=%d dt=%s written=%d)",
		r.Method,
		r.URL,
		m.Code,
		m.Duration,
		m.Written,
	)
})
http.ListenAndServe(":8080", wrappedH)

Why this package exists

Instrumenting an application's http.Handler is surprisingly difficult.

However if you google for e.g. "capture ResponseWriter status code" you'll find
lots of advise and code examples that suggest it to be a fairly trivial
undertaking. Unfortunately everything I've seen so far has a high chance of
breaking your application.

The main problem is that a http.ResponseWriter often implements additional
interfaces such as http.Flusher, http.CloseNotifier, http.Hijacker, http.Pusher, and
io.ReaderFrom. So the naive approach of just wrapping http.ResponseWriter
in your own struct that also implements the http.ResponseWriter interface
will hide the additional interfaces mentioned above. This has a high change of
introducing subtle bugs into any non-trivial application.

Another approach I've seen people take is to return a struct that implements
all of the interfaces above. However, that's also problematic, because it's
difficult to fake some of these interfaces behaviors when the underlying
http.ResponseWriter doesn't have an implementation. It's also dangerous,
because an application may choose to operate differently, merely because it
detects the presence of these additional interfaces.

This package solves this problem by checking which additional interfaces a
http.ResponseWriter implements, returning a wrapped version implementing the
exact same set of interfaces.

Additionally this package properly handles edge cases such as WriteHeader not
being called, or called more than once, as well as concurrent calls to
http.ResponseWriter methods, and even calls happening after the wrapped
ServeHTTP has already returned.

Unfortunately this package is not perfect either. It's possible that it is
still missing some interfaces provided by the go core (let me know if you find
one), and it won't work for applications adding their own interfaces into the
mix. You can however use httpsnoop.Unwrap(w) to access the underlying
http.ResponseWriter and type-assert the result to its other interfaces.

However, hopefully the explanation above has sufficiently scared you of rolling
your own solution to this problem. httpsnoop may still break your application,
but at least it tries to avoid it as much as possible.

Anyway, the real problem here is that smuggling additional interfaces inside
http.ResponseWriter is a problematic design choice, but it probably goes as
deep as the Go language specification itself. But that's okay, I still prefer
Go over the alternatives ;).

Performance

text

BenchmarkBaseline-8      	   20000	     94912 ns/op
BenchmarkCaptureMetrics-8	   20000	     95461 ns/op

As you can see, using CaptureMetrics on a vanilla http.Handler introduces an
overhead of ~500 ns per http request on my machine. However, the margin of
error appears to be larger than that, therefor it should be reasonable to
assume that the overhead introduced by CaptureMetrics is absolutely
negligible.

License

MIT

Overview (`vendor/github.com/go-sourcemap/sourcemap/README.md`)

Source maps consumer for Golang

This package is brought to you by :star: uptrace/uptrace.
Uptrace is an open-source APM tool that supports distributed tracing, metrics, and logs. You can
use it to monitor applications and set up automatic alerts to receive notifications via email,
Slack, Telegram, and others.

Installation

Install:

shell

go get -u github.com/go-sourcemap/sourcemap

Quickstart

func ExampleParse() {
	mapURL := "http://code.jquery.com/jquery-2.0.3.min.map"
	resp, err := http.Get(mapURL)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	b, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		panic(err)
	}

	smap, err := sourcemap.Parse(mapURL, b)
	if err != nil {
		panic(err)
	}

	line, column := 5, 6789
	file, fn, line, col, ok := smap.Source(line, column)
	fmt.Println(file, fn, line, col, ok)
	// Output: http://code.jquery.com/jquery-2.0.3.js apply 4360 27 true
}

Overview (`vendor/github.com/gorilla/handlers/README.md`)

gorilla/handlers

Package handlers is a collection of handlers (aka "HTTP middleware") for use
with Go's net/http package (or any framework supporting http.Handler), including:

LoggingHandler for logging HTTP requests in the Apache Common Log
Format.
CombinedLoggingHandler for logging HTTP requests in the Apache Combined Log
Format commonly used by
both Apache and nginx.
CompressHandler for gzipping responses.
ContentTypeHandler for validating requests against a list of accepted
content types.
MethodHandler for matching HTTP methods against handlers in a
map[string]http.Handler
ProxyHeaders for populating r.RemoteAddr and r.URL.Scheme based on the
X-Forwarded-For, X-Real-IP, X-Forwarded-Proto and RFC7239 Forwarded
headers when running a Go server behind a HTTP reverse proxy.
CanonicalHost for re-directing to the preferred host when handling multiple
domains (i.e. multiple CNAME aliases).
RecoveryHandler for recovering from unexpected panics.

Other handlers are documented on the Gorilla
website.

Example

A simple example using handlers.LoggingHandler and handlers.CompressHandler:

import (
    "net/http"
    "github.com/gorilla/handlers"
)

func main() {
    r := http.NewServeMux()

    // Only log requests to our admin dashboard to stdout
    r.Handle("/admin", handlers.LoggingHandler(os.Stdout, http.HandlerFunc(ShowAdminDashboard)))
    r.HandleFunc("/", ShowIndex)

    // Wrap our server with our gzip handler to gzip compress all responses.
    http.ListenAndServe(":8000", handlers.CompressHandler(r))
}

License

BSD licensed. See the included LICENSE file for details.

Overview (`vendor/github.com/gorilla/mux/README.md`)

gorilla/mux

Package gorilla/mux implements a request router and dispatcher for matching incoming requests to
their respective handler.

The name mux stands for "HTTP request multiplexer". Like the standard http.ServeMux, mux.Router matches incoming requests against a list of registered routes and calls a handler for the route that matches the URL or other conditions. The main features are:

It implements the http.Handler interface so it is compatible with the standard http.ServeMux.
Requests can be matched based on URL host, path, path prefix, schemes, header and query values, HTTP methods or using custom matchers.
URL hosts, paths and query values can have variables with an optional regular expression.
Registered URLs can be built, or "reversed", which helps maintaining references to resources.
Routes can be used as subrouters: nested routes are only tested if the parent route matches. This is useful to define groups of routes that share common conditions like a host, a path prefix or other repeated attributes. As a bonus, this optimizes request matching.

Install
Examples
Matching Routes
Static Files
Serving Single Page Applications (e.g. React, Vue, Ember.js, etc.)
Registered URLs
Walking Routes
Graceful Shutdown
Middleware
Handling CORS Requests
Testing Handlers
Full Example

Install

With a correctly configured Go toolchain:

go get -u github.com/gorilla/mux

Examples

Let's start registering a couple of URL paths and handlers:

func main() {
    r := mux.NewRouter()
    r.HandleFunc("/", HomeHandler)
    r.HandleFunc("/products", ProductsHandler)
    r.HandleFunc("/articles", ArticlesHandler)
    http.Handle("/", r)
}

Here we register three routes mapping URL paths to handlers. This is equivalent to how http.HandleFunc() works: if an incoming request URL matches one of the paths, the corresponding handler is called passing (http.ResponseWriter, *http.Request) as parameters.

Paths can have variables. They are defined using the format {name} or {name:pattern}. If a regular expression pattern is not defined, the matched variable will be anything until the next slash. For example:

r := mux.NewRouter()
r.HandleFunc("/products/{key}", ProductHandler)
r.HandleFunc("/articles/{category}/", ArticlesCategoryHandler)
r.HandleFunc("/articles/{category}/{id:[0-9]+}", ArticleHandler)

The names are used to create a map of route variables which can be retrieved calling mux.Vars():

func ArticlesCategoryHandler(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    w.WriteHeader(http.StatusOK)
    fmt.Fprintf(w, "Category: %v\n", vars["category"])
}

And this is all you need to know about the basic usage. More advanced options are explained below.

Matching Routes

Routes can also be restricted to a domain or subdomain. Just define a host pattern to be matched. They can also have variables:

r := mux.NewRouter()
// Only matches if domain is "www.example.com".
r.Host("www.example.com")
// Matches a dynamic subdomain.
r.Host("{subdomain:[a-z]+}.example.com")

There are several other matchers that can be added. To match path prefixes:

r.PathPrefix("/products/")

...or HTTP methods:

r.Methods("GET", "POST")

...or URL schemes:

r.Schemes("https")

...or header values:

r.Headers("X-Requested-With", "XMLHttpRequest")

...or query values:

r.Queries("key", "value")

...or to use a custom matcher function:

r.MatcherFunc(func(r *http.Request, rm *RouteMatch) bool {
    return r.ProtoMajor == 0
})

...and finally, it is possible to combine several matchers in a single route:

r.HandleFunc("/products", ProductsHandler).
  Host("www.example.com").
  Methods("GET").
  Schemes("http")

Routes are tested in the order they were added to the router. If two routes match, the first one wins:

r := mux.NewRouter()
r.HandleFunc("/specific", specificHandler)
r.PathPrefix("/").Handler(catchAllHandler)

Setting the same matching conditions again and again can be boring, so we have a way to group several routes that share the same requirements. We call it "subrouting".

For example, let's say we have several URLs that should only match when the host is www.example.com. Create a route for that host and get a "subrouter" from it:

r := mux.NewRouter()
s := r.Host("www.example.com").Subrouter()

Then register routes in the subrouter:

s.HandleFunc("/products/", ProductsHandler)
s.HandleFunc("/products/{key}", ProductHandler)
s.HandleFunc("/articles/{category}/{id:[0-9]+}", ArticleHandler)

The three URL paths we registered above will only be tested if the domain is www.example.com, because the subrouter is tested first. This is not only convenient, but also optimizes request matching. You can create subrouters combining any attribute matchers accepted by a route.

Subrouters can be used to create domain or path "namespaces": you define subrouters in a central place and then parts of the app can register its paths relatively to a given subrouter.

There's one more thing about subroutes. When a subrouter has a path prefix, the inner routes use it as base for their paths:

r := mux.NewRouter()
s := r.PathPrefix("/products").Subrouter()
// "/products/"
s.HandleFunc("/", ProductsHandler)
// "/products/{key}/"
s.HandleFunc("/{key}/", ProductHandler)
// "/products/{key}/details"
s.HandleFunc("/{key}/details", ProductDetailsHandler)

Static Files

Note that the path provided to PathPrefix() represents a "wildcard": calling
PathPrefix("/static/").Handler(...) means that the handler will be passed any
request that matches "/static/*". This makes it easy to serve static files with mux:

func main() {
    var dir string

    flag.StringVar(&dir, "dir", ".", "the directory to serve files from. Defaults to the current dir")
    flag.Parse()
    r := mux.NewRouter()

    // This will serve files under http://localhost:8000/static/<filename>
    r.PathPrefix("/static/").Handler(http.StripPrefix("/static/", http.FileServer(http.Dir(dir))))

    srv := &http.Server{
        Handler:      r,
        Addr:         "127.0.0.1:8000",
        // Good practice: enforce timeouts for servers you create!
        WriteTimeout: 15 * time.Second,
        ReadTimeout:  15 * time.Second,
    }

    log.Fatal(srv.ListenAndServe())
}

Serving Single Page Applications

Most of the time it makes sense to serve your SPA on a separate web server from your API,
but sometimes it's desirable to serve them both from one place. It's possible to write a simple
handler for serving your SPA (for use with React Router's BrowserRouter for example), and leverage
mux's powerful routing for your API endpoints.

text

/* Detailed source-code truncated for AI context efficiency. */

Registered URLs

Now let's see how to build registered URLs.

Routes can be named. All routes that define a name can have their URLs built, or "reversed". We define a name calling Name() on a route. For example:

r := mux.NewRouter()
r.HandleFunc("/articles/{category}/{id:[0-9]+}", ArticleHandler).
  Name("article")

To build a URL, get the route and call the URL() method, passing a sequence of key/value pairs for the route variables. For the previous route, we would do:

url, err := r.Get("article").URL("category", "technology", "id", "42")

...and the result will be a url.URL with the following path:

text

"/articles/technology/42"

This also works for host and query value variables:

r := mux.NewRouter()
r.Host("{subdomain}.example.com").
  Path("/articles/{category}/{id:[0-9]+}").
  Queries("filter", "{filter}").
  HandlerFunc(ArticleHandler).
  Name("article")

// url.String() will be "http://news.example.com/articles/technology/42?filter=gorilla"
url, err := r.Get("article").URL("subdomain", "news",
                                 "category", "technology",
                                 "id", "42",
                                 "filter", "gorilla")

All variables defined in the route are required, and their values must conform to the corresponding patterns. These requirements guarantee that a generated URL will always match a registered route -- the only exception is for explicitly defined "build-only" routes which never match.

Regex support also exists for matching Headers within a route. For example, we could do:

r.HeadersRegexp("Content-Type", "application/(text|json)")

...and the route will match both requests with a Content-Type of application/json as well as application/text

There's also a way to build only the URL host or path for a route: use the methods URLHost() or URLPath() instead. For the previous route, we would do:

// "http://news.example.com/"
host, err := r.Get("article").URLHost("subdomain", "news")

// "/articles/technology/42"
path, err := r.Get("article").URLPath("category", "technology", "id", "42")

And if you use subrouters, host and path defined separately can be built as well:

r := mux.NewRouter()
s := r.Host("{subdomain}.example.com").Subrouter()
s.Path("/articles/{category}/{id:[0-9]+}").
  HandlerFunc(ArticleHandler).
  Name("article")

// "http://news.example.com/articles/technology/42"
url, err := r.Get("article").URL("subdomain", "news",
                                 "category", "technology",
                                 "id", "42")

To find all the required variables for a given route when calling URL(), the method GetVarNames() is available:

r := mux.NewRouter()
r.Host("{domain}").
    Path("/{group}/{item_id}").
    Queries("some_data1", "{some_data1}").
    Queries("some_data2", "{some_data2}").
    Name("article")

// Will print [domain group item_id some_data1 some_data2] <nil>
fmt.Println(r.Get("article").GetVarNames())

Walking Routes

The Walk function on mux.Router can be used to visit all of the routes that are registered on a router. For example,
the following prints all of the registered routes:

package main

import (
	"fmt"
	"net/http"
	"strings"

	"github.com/gorilla/mux"
)

func handler(w http.ResponseWriter, r *http.Request) {
	return
}

func main() {
	r := mux.NewRouter()
	r.HandleFunc("/", handler)
	r.HandleFunc("/products", handler).Methods("POST")
	r.HandleFunc("/articles", handler).Methods("GET")
	r.HandleFunc("/articles/{id}", handler).Methods("GET", "PUT")
	r.HandleFunc("/authors", handler).Queries("surname", "{surname}")
	err := r.Walk(func(route *mux.Route, router *mux.Router, ancestors []*mux.Route) error {
		pathTemplate, err := route.GetPathTemplate()
		if err == nil {
			fmt.Println("ROUTE:", pathTemplate)
		}
		pathRegexp, err := route.GetPathRegexp()
		if err == nil {
			fmt.Println("Path regexp:", pathRegexp)
		}
		queriesTemplates, err := route.GetQueriesTemplates()
		if err == nil {
			fmt.Println("Queries templates:", strings.Join(queriesTemplates, ","))
		}
		queriesRegexps, err := route.GetQueriesRegexp()
		if err == nil {
			fmt.Println("Queries regexps:", strings.Join(queriesRegexps, ","))
		}
		methods, err := route.GetMethods()
		if err == nil {
			fmt.Println("Methods:", strings.Join(methods, ","))
		}
		fmt.Println()
		return nil
	})

	if err != nil {
		fmt.Println(err)
	}

	http.Handle("/", r)
}

Graceful Shutdown

Go 1.8 introduced the ability to gracefully shutdown a *http.Server. Here's how to do that alongside mux:

text

/* Detailed source-code truncated for AI context efficiency. */

Middleware

Mux supports the addition of middlewares to a Router, which are executed in the order they are added if a match is found, including its subrouters.
Middlewares are (typically) small pieces of code which take one request, do something with it, and pass it down to another middleware or the final handler. Some common use cases for middleware are request logging, header manipulation, or ResponseWriter hijacking.

Mux middlewares are defined using the de facto standard type:

type MiddlewareFunc func(http.Handler) http.Handler

Typically, the returned handler is a closure which does something with the http.ResponseWriter and http.Request passed to it, and then calls the handler passed as parameter to the MiddlewareFunc. This takes advantage of closures being able access variables from the context where they are created, while retaining the signature enforced by the receivers.

A very basic middleware which logs the URI of the request being handled could be written as:

func loggingMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Do stuff here
        log.Println(r.RequestURI)
        // Call the next handler, which can be another middleware in the chain, or the final handler.
        next.ServeHTTP(w, r)
    })
}

Middlewares can be added to a router using Router.Use():

r := mux.NewRouter()
r.HandleFunc("/", handler)
r.Use(loggingMiddleware)

A more complex authentication middleware, which maps session token to users, could be written as:

// Define our struct
type authenticationMiddleware struct {
	tokenUsers map[string]string
}

// Initialize it somewhere
func (amw *authenticationMiddleware) Populate() {
	amw.tokenUsers["00000000"] = "user0"
	amw.tokenUsers["aaaaaaaa"] = "userA"
	amw.tokenUsers["05f717e5"] = "randomUser"
	amw.tokenUsers["deadbeef"] = "user0"
}

// Middleware function, which will be called for each request
func (amw *authenticationMiddleware) Middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        token := r.Header.Get("X-Session-Token")

        if user, found := amw.tokenUsers[token]; found {
        	// We found the token in our map
        	log.Printf("Authenticated user %s\n", user)
        	// Pass down the request to the next middleware (or final handler)
        	next.ServeHTTP(w, r)
        } else {
        	// Write an error and stop the handler chain
        	http.Error(w, "Forbidden", http.StatusForbidden)
        }
    })
}

r := mux.NewRouter()
r.HandleFunc("/", handler)

amw := authenticationMiddleware{tokenUsers: make(map[string]string)}
amw.Populate()

r.Use(amw.Middleware)

Note: The handler chain will be stopped if your middleware doesn't call next.ServeHTTP() with the corresponding parameters. This can be used to abort a request if the middleware writer wants to. Middlewares should write to ResponseWriter if they are going to terminate the request, and they should not write to ResponseWriter if they are not going to terminate it.

Handling CORS Requests

CORSMethodMiddleware intends to make it easier to strictly set the Access-Control-Allow-Methods response header.

You will still need to use your own CORS handler to set the other CORS headers such as Access-Control-Allow-Origin
The middleware will set the Access-Control-Allow-Methods header to all the method matchers (e.g. r.Methods(http.MethodGet, http.MethodPut, http.MethodOptions) -> Access-Control-Allow-Methods: GET,PUT,OPTIONS) on a route
If you do not specify any methods, then:

Important: there must be an OPTIONS method matcher for the middleware to set the headers.

Here is an example of using CORSMethodMiddleware along with a custom OPTIONS handler to set all the required CORS headers:

package main

import (
	"net/http"
	"github.com/gorilla/mux"
)

func main() {
    r := mux.NewRouter()

    // IMPORTANT: you must specify an OPTIONS method matcher for the middleware to set CORS headers
    r.HandleFunc("/foo", fooHandler).Methods(http.MethodGet, http.MethodPut, http.MethodPatch, http.MethodOptions)
    r.Use(mux.CORSMethodMiddleware(r))
    
    http.ListenAndServe(":8080", r)
}

func fooHandler(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Access-Control-Allow-Origin", "*")
    if r.Method == http.MethodOptions {
        return
    }

    w.Write([]byte("foo"))
}

And an request to /foo using something like:

bash

curl localhost:8080/foo -v

Would look like:

bash

*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /foo HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.59.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Access-Control-Allow-Methods: GET,PUT,PATCH,OPTIONS
< Access-Control-Allow-Origin: *
< Date: Fri, 28 Jun 2019 20:13:30 GMT
< Content-Length: 3
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host localhost left intact
foo

Testing Handlers

Testing handlers in a Go web application is straightforward, and mux doesn't complicate this any further. Given two files: endpoints.go and endpoints_test.go, here's how we'd test an application using mux.

First, our simple HTTP handler:

// endpoints.go
package main

func HealthCheckHandler(w http.ResponseWriter, r *http.Request) {
    // A very simple health check.
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusOK)

    // In the future we could report back on the status of our DB, or our cache
    // (e.g. Redis) by performing a simple PING, and include them in the response.
    io.WriteString(w, `{"alive": true}`)
}

func main() {
    r := mux.NewRouter()
    r.HandleFunc("/health", HealthCheckHandler)

    log.Fatal(http.ListenAndServe("localhost:8080", r))
}

Our test code:

// endpoints_test.go
package main

import (
    "net/http"
    "net/http/httptest"
    "testing"
)

func TestHealthCheckHandler(t *testing.T) {
    // Create a request to pass to our handler. We don't have any query parameters for now, so we'll
    // pass 'nil' as the third parameter.
    req, err := http.NewRequest("GET", "/health", nil)
    if err != nil {
        t.Fatal(err)
    }

    // We create a ResponseRecorder (which satisfies http.ResponseWriter) to record the response.
    rr := httptest.NewRecorder()
    handler := http.HandlerFunc(HealthCheckHandler)

    // Our handlers satisfy http.Handler, so we can call their ServeHTTP method
    // directly and pass in our Request and ResponseRecorder.
    handler.ServeHTTP(rr, req)

    // Check the status code is what we expect.
    if status := rr.Code; status != http.StatusOK {
        t.Errorf("handler returned wrong status code: got %v want %v",
            status, http.StatusOK)
    }

    // Check the response body is what we expect.
    expected := `{"alive": true}`
    if rr.Body.String() != expected {
        t.Errorf("handler returned unexpected body: got %v want %v",
            rr.Body.String(), expected)
    }
}

In the case that our routes have variables, we can pass those in the request. We could write
table-driven tests to test multiple
possible route variables as needed.

// endpoints.go
func main() {
    r := mux.NewRouter()
    // A route with a route variable:
    r.HandleFunc("/metrics/{type}", MetricsHandler)

    log.Fatal(http.ListenAndServe("localhost:8080", r))
}

Our test file, with a table-driven test of routeVariables:

// endpoints_test.go
func TestMetricsHandler(t *testing.T) {
    tt := []struct{
        routeVariable string
        shouldPass bool
    }{
        {"goroutines", true},
        {"heap", true},
        {"counters", true},
        {"queries", true},
        {"adhadaeqm3k", false},
    }

    for _, tc := range tt {
        path := fmt.Sprintf("/metrics/%s", tc.routeVariable)
        req, err := http.NewRequest("GET", path, nil)
        if err != nil {
            t.Fatal(err)
        }

        rr := httptest.NewRecorder()
	
	// To add the vars to the context, 
	// we need to create a router through which we can pass the request.
	router := mux.NewRouter()
        router.HandleFunc("/metrics/{type}", MetricsHandler)
        router.ServeHTTP(rr, req)

        // In this case, our MetricsHandler returns a non-200 response
        // for a route variable it doesn't know about.
        if rr.Code == http.StatusOK && !tc.shouldPass {
            t.Errorf("handler should have failed on routeVariable %s: got %v want %v",
                tc.routeVariable, rr.Code, http.StatusOK)
        }
    }
}

Full Example

Here's a complete, runnable example of a small mux based server:

package main

import (
    "net/http"
    "log"
    "github.com/gorilla/mux"
)

func YourHandler(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("Gorilla!\n"))
}

func main() {
    r := mux.NewRouter()
    // Routes consist of a path and a handler function.
    r.HandleFunc("/", YourHandler)

    // Bind to a port and pass our router in
    log.Fatal(http.ListenAndServe(":8000", r))
}

License

BSD licensed. See the LICENSE file for details.

Overview (`vendor/github.com/gorilla/websocket/README.md`)

Gorilla WebSocket

Gorilla WebSocket is a Go implementation of the
WebSocket protocol.

Documentation

Status

The Gorilla WebSocket package provides a complete and tested implementation of
the WebSocket protocol. The
package API is stable.

Installation

text

go get github.com/gorilla/websocket

Protocol Compliance

The Gorilla WebSocket package passes the server tests in the Autobahn Test
Suite using the application in the examples/autobahn
subdirectory.

Overview (`vendor/github.com/heroiclabs/sql-migrate/README.md`)

sql-migrate

This is a fork of the sql-migrate SQL Schema migration tool for Go.

The motivation for this fork is to remove the go-gorp dependency, simplifying the codebase but making it work with the pgx db driver exclusively.

Features

Can embed migrations into your application
Migrations are defined with SQL for full flexibility
Atomic migrations
Up/down migrations to allow rollback

Installation

bash

go get -v github.com/heroiclabs/nakama/sql-migrate/

Usage

Import sql-migrate into your application:

import "github.com/heroiclabs/nakama/sql-migrate"

Set up a source of migrations, this can be from memory, from a set of files, from bindata (more on that later), or from any library that implements http.FileSystem:

// Hardcoded strings in memory:
migrations := &migrate.MemoryMigrationSource{
    Migrations: []*migrate.Migration{
        &migrate.Migration{
            Id:   "123",
            Up:   []string{"CREATE TABLE people (id int)"},
            Down: []string{"DROP TABLE people"},
        },
    },
}

// OR: Read migrations from a folder:
migrations := &migrate.FileMigrationSource{
    Dir: "db/migrations",
}

// OR: Use migrations from a packr box
migrations := &migrate.PackrMigrationSource{
    Box: packr.New("migrations", "./migrations"),
}

// OR: Use pkger which implements `http.FileSystem`
migrationSource := &migrate.HttpFileSystemMigrationSource{
    FileSystem: pkger.Dir("/db/migrations"),
}

// OR: Use migrations from bindata:
migrations := &migrate.AssetMigrationSource{
    Asset:    Asset,
    AssetDir: AssetDir,
    Dir:      "migrations",
}

// OR: Read migrations from a `http.FileSystem`
migrationSource := &migrate.HttpFileSystemMigrationSource{
    FileSystem: httpFS,
}

Then use the Exec function to upgrade your database:

n, err := migrate.Exec(db, migrations, migrate.Up)
if err != nil {
    // Handle errors!
}
fmt.Printf("Applied %d migrations!\n", n)

Note that n can be greater than 0 even if there is an error: any migration that succeeded will remain applied even if a later one fails.

Writing migrations

Migrations are defined in SQL files, which contain a set of SQL statements. Special comments are used to distinguish up and down migrations.

sql

-- +migrate Up
-- SQL in section 'Up' is executed when this migration is applied
CREATE TABLE people (id int);


-- +migrate Down
-- SQL section 'Down' is executed when this migration is rolled back
DROP TABLE people;

You can put multiple statements in each block, as long as you end them with a semicolon (;).

You can alternatively set up a separator string that matches an entire line by setting sqlparse.LineSeparator. This
can be used to imitate, for example, MS SQL Query Analyzer functionality where commands can be separated by a line with
contents of GO. If sqlparse.LineSeparator is matched, it will not be included in the resulting migration scripts.

If you have complex statements which contain semicolons, use StatementBegin and StatementEnd to indicate boundaries:

sql

-- +migrate Up
CREATE TABLE people (id int);

-- +migrate StatementBegin
CREATE OR REPLACE FUNCTION do_something()
returns void AS $$
DECLARE
  create_query text;
BEGIN
  -- Do something here
END;
$$
language plpgsql;
-- +migrate StatementEnd

-- +migrate Down
DROP FUNCTION do_something();
DROP TABLE people;

The order in which migrations are applied is defined through the filename: sql-migrate will sort migrations based on their name. It's recommended to use an increasing version number or a timestamp as the first part of the filename.

Normally each migration is run within a transaction in order to guarantee that it is fully atomic. However some SQL commands (for example creating an index concurrently in PostgreSQL) cannot be executed inside a transaction. In order to execute such a command in a migration, the migration can be run using the notransaction option:

sql

-- +migrate Up notransaction
CREATE UNIQUE INDEX CONCURRENTLY people_unique_id_idx ON people (id);

-- +migrate Down
DROP INDEX people_unique_id_idx;

Because a notransaction migration runs outside of a transaction, it is not atomic: if it fails partway through, the statements that already ran remain applied and the migration is not recorded as complete.

Each statement in a notransaction migration must be a single SQL command. Statements are sent using PostgreSQL's simple query protocol, which wraps a message containing multiple commands in an implicit transaction. Grouping several commands inside one StatementBegin/StatementEnd block therefore reintroduces a transaction.

The notransaction option applies to the whole Up (or Down) section. A file may contain several -- +migrate Up sections, but if any of them sets notransaction then all of that migration's up statements run outside a transaction. You cannot mix transactional and non-transactional statements in one migration, so keep statements that must run outside a transaction in their own migration file.

Embedding migrations with libraries that implement `http.FileSystem`

You can also embed migrations with any library that implements http.FileSystem, like vfsgen, parcello, or go-resources.

migrationSource := &migrate.HttpFileSystemMigrationSource{
    FileSystem: httpFS,
}

Tests

Run the tests by first starting the Postgres docker container.

bash

docker compose up
go test

Extending

Adding a new migration source means implementing MigrationSource.

type MigrationSource interface {
    FindMigrations() ([]*Migration, error)
}

The resulting slice of migrations will be executed in the given order, so it should usually be sorted by the Id field.

License

This library is distributed under the MIT license.

Overview (`vendor/github.com/jackc/pgerrcode/README.md`)

pgerrcode

Package pgerrcode contains constants for PostgreSQL error codes.

License

MIT for this package's code and PostgreSQL License for the underlying data.

Overview (`vendor/github.com/jackc/pgpassfile/README.md`)

pgpassfile

Package pgpassfile is a parser PostgreSQL .pgpass files.

Extracted and rewritten from original implementation in https://github.com/jackc/pgx.

Overview (`vendor/github.com/jackc/pgservicefile/README.md`)

pgservicefile

Package pgservicefile is a parser for PostgreSQL service files (e.g. .pg_service.conf).

Overview (`vendor/github.com/kamstrup/intmap/README.md`)

Fast hashmap with integer keys for Golang

intmap

text

import "github.com/kamstrup/intmap"

Package intmap is a fast hashmap implementation for Golang, specialized for maps with integer type keys.
The values can be of any type.

It is a full port of https://github.com/brentp/intintmap to use type parameters (aka generics).

It interleaves keys and values in the same underlying array to improve locality.
This is also known as open addressing with linear probing.

It is up to 3X faster than the builtin map:

text

name                             time/op
Map64Fill-8                       201ms ± 5%
IntIntMapFill-8                   207ms ±31%
StdMapFill-8                      371ms ±11%
Map64Get10PercentHitRate-8        148µs ±40%
IntIntMapGet10PercentHitRate-8    171µs ±50%
StdMapGet10PercentHitRate-8       171µs ±33%
Map64Get100PercentHitRate-8      4.50ms ± 5%
IntIntMapGet100PercentHitRate-8  4.82ms ± 6%
StdMapGet100PercentHitRate-8     15.5ms ±32%

Usage

m := intmap.New[int64,int64](32768)
m.Put(int64(1234), int64(-222))
m.Put(int64(123), int64(33))

v, ok := m.Get(int64(222))
v, ok := m.Get(int64(333))

m.Del(int64(222))
m.Del(int64(333))

fmt.Println(m.Len())

m.ForEach(func(k int64, v int64) {
    fmt.Printf("key: %d, value: %d\n", k, v)
})

m.Clear() // all gone, but buffers kept

Overview (`vendor/github.com/klauspost/compress/README.md`)

compress

This package provides various compression algorithms.

zstandard compression and decompression in pure Go.
S2 is a high performance replacement for Snappy.
Optimized deflate packages which can be used as a dropin replacement for gzip, zip and zlib.
snappy is a drop-in replacement for github.com/golang/snappy offering better compression and concurrent streams.
lzw is a drop-in replacement for compress/lzw with 1.4-4x faster decompression and 1.1-2.7x faster compression, depending on the data.
huff0 and FSE implementations for raw entropy encoding.
Xpress decompression of the Microsoft XPRESS (MS-XCA) plain LZ77 and LZ77+Huffman formats (the LZ77+Huffman variant is the one used in WIM images and Windows Compact OS / WOF data).
gzhttp Provides client and server wrappers for handling gzipped/zstd HTTP requests efficiently.
pgzip is a separate package that provides a very fast parallel gzip implementation.

package usage

Use go get github.com/klauspost/compress@latest to add it to your project.

This package will support the current Go version and 2 versions back.

Use the nounsafe tag to disable all use of the "unsafe" package.
Use the noasm tag to disable all assembly across packages.

Use the links above for more information on each.

changelog

Jul 1st, 2026 1.19.0
- zstd: Add true concurrent stream encodingin https://github.com/klauspost/compress/pull/1136
- zstd: arm64 decoder asm by @lizthegrey in https://github.com/klauspost/compress/pull/1160
- flate: Add inflate checkpoints in https://github.com/klauspost/compress/pull/1154
- zstd: avoid unused BuildDict encoder allocation by @snissn in https://github.com/klauspost/compress/pull/1147
- snappy/s2: Limit length of varint in decodedLen by @eustas in https://github.com/klauspost/compress/pull/1148
- gzhttp: match qvalue parameter case-insensitively (RFC 7231) by @z9z in https://github.com/klauspost/compress/pull/1149
- zip: add NameDecoder callback for legacy encoding rewrite by @SAY-5 in https://github.com/klauspost/compress/pull/1150
- huff0: Allow building tables from histogram in https://github.com/klauspost/compress/pull/1155
- huff0: Allow building table from oversized histogram in https://github.com/klauspost/compress/pull/1156
- s2sx: Clean symlink targets in https://github.com/klauspost/compress/pull/1163
Feb 9th, 2026 1.18.4
- gzhttp: Add zstandard to server handler wrapper https://github.com/klauspost/compress/pull/1121
- zstd: Add ResetWithOptions to encoder/decoder https://github.com/klauspost/compress/pull/1122
- gzhttp: preserve qvalue when extra parameters follow in Accept-Encoding by @analytically in https://github.com/klauspost/compress/pull/1116
Jan 16th, 2026 1.18.3
- Downstream CVE-2025-61728. See golang/go#77102.
Dec 1st, 2025 - 1.18.2
- flate: Fix invalid encoding on level 9 with single value input in https://github.com/klauspost/compress/pull/1115
- flate: reduce stateless allocations by @RXamzin in https://github.com/klauspost/compress/pull/1106
Oct 20, 2025 - 1.18.1 - RETRACTED
- zstd: Add simple zstd EncodeTo/DecodeTo functions https://github.com/klauspost/compress/pull/1079
- zstd: Fix incorrect buffer size in dictionary encodes https://github.com/klauspost/compress/pull/1059
- s2: check for cap, not len of buffer in EncodeBetter/Best by @vdarulis in https://github.com/klauspost/compress/pull/1080
- zlib: Avoiding extra allocation in zlib.reader.Reset by @travelpolicy in https://github.com/klauspost/compress/pull/1086
- gzhttp: remove redundant err check in zstdReader by @ryanfowler in https://github.com/klauspost/compress/pull/1090
- flate: Faster load+store https://github.com/klauspost/compress/pull/1104
- flate: Simplify matchlen https://github.com/klauspost/compress/pull/1101
- flate: Use exact sizes for huffman tables https://github.com/klauspost/compress/pull/1103
Feb 19th, 2025 - 1.18.0
- Add unsafe little endian loaders https://github.com/klauspost/compress/pull/1036
- fix: check r.err != nil but return a nil value error err by @alingse in https://github.com/klauspost/compress/pull/1028
- flate: Simplify L4-6 loading https://github.com/klauspost/compress/pull/1043
- flate: Simplify matchlen (remove asm) https://github.com/klauspost/compress/pull/1045
- s2: Improve small block compression speed w/o asm https://github.com/klauspost/compress/pull/1048
- flate: Fix matchlen L5+L6 https://github.com/klauspost/compress/pull/1049
- flate: Cleanup & reduce casts https://github.com/klauspost/compress/pull/1050

See changes to v1.17.x

Oct 11th, 2024 - 1.17.11
- zstd: Fix extra CRC written with multiple Close calls https://github.com/klauspost/compress/pull/1017
- s2: Don't use stack for index tables https://github.com/klauspost/compress/pull/1014
- gzhttp: No content-type on no body response code by @juliens in https://github.com/klauspost/compress/pull/1011
- gzhttp: Do not set the content-type when response has no body by @kevinpollet in https://github.com/klauspost/compress/pull/1013
Sep 23rd, 2024 - 1.17.10
- gzhttp: Add TransportAlwaysDecompress option. https://github.com/klauspost/compress/pull/978
- gzhttp: Add supported decompress request body by @mirecl in https://github.com/klauspost/compress/pull/1002
- s2: Add EncodeBuffer buffer recycling callback https://github.com/klauspost/compress/pull/982
- zstd: Improve memory usage on small streaming encodes https://github.com/klauspost/compress/pull/1007
- flate: read data written with partial flush by @vajexal in https://github.com/klauspost/compress/pull/996
Jun 12th, 2024 - 1.17.9
- s2: Reduce ReadFrom temporary allocations https://github.com/klauspost/compress/pull/949
- flate, zstd: Shave some bytes off amd64 matchLen by @greatroar in https://github.com/klauspost/compress/pull/963
- Upgrade zip/zlib to 1.22.4 upstream https://github.com/klauspost/compress/pull/970 https://github.com/klauspost/compress/pull/971
- zstd: BuildDict fails with RLE table https://github.com/klauspost/compress/pull/951
Apr 9th, 2024 - 1.17.8
- zstd: Reject blocks where reserved values are not 0 https://github.com/klauspost/compress/pull/885
- zstd: Add RLE detection+encoding https://github.com/klauspost/compress/pull/938
Feb 21st, 2024 - 1.17.7
- s2: Add AsyncFlush method: Complete the block without flushing by @Jille in https://github.com/klauspost/compress/pull/927
- s2: Fix literal+repeat exceeds dst crash https://github.com/klauspost/compress/pull/930
Feb 5th, 2024 - 1.17.6
- zstd: Fix incorrect repeat coding in best mode https://github.com/klauspost/compress/pull/923
- s2: Fix DecodeConcurrent deadlock on errors https://github.com/klauspost/compress/pull/925
Jan 26th, 2024 - v1.17.5
- flate: Fix reset with dictionary on custom window encodes https://github.com/klauspost/compress/pull/912
- zstd: Add Frame header encoding and stripping https://github.com/klauspost/compress/pull/908
- zstd: Limit better/best default window to 8MB https://github.com/klauspost/compress/pull/913
- zstd: Speed improvements by @greatroar in https://github.com/klauspost/compress/pull/896 https://github.com/klauspost/compress/pull/910
- s2: Fix callbacks for skippable blocks and disallow 0xfe (Padding) by @Jille in https://github.com/klauspost/compress/pull/916 https://github.com/klauspost/compress/pull/917
  https://github.com/klauspost/compress/pull/919 https://github.com/klauspost/compress/pull/918
Dec 1st, 2023 - v1.17.4
- huff0: Speed up symbol counting by @greatroar in https://github.com/klauspost/compress/pull/887
- huff0: Remove byteReader by @greatroar in https://github.com/klauspost/compress/pull/886
- gzhttp: Allow overriding decompression on transport https://github.com/klauspost/compress/pull/892
- gzhttp: Clamp compression level https://github.com/klauspost/compress/pull/890
- gzip: Error out if reserved bits are set https://github.com/klauspost/compress/pull/891
Nov 15th, 2023 - v1.17.3
- fse: Fix max header size https://github.com/klauspost/compress/pull/881
- zstd: Improve better/best compression https://github.com/klauspost/compress/pull/877
- gzhttp: Fix missing content type on Close https://github.com/klauspost/compress/pull/883
Oct 22nd, 2023 - v1.17.2
- zstd: Fix rare CORRUPTION output in "best" mode. See https://github.com/klauspost/compress/pull/876
Oct 14th, 2023 - v1.17.1
- s2: Fix S2 "best" dictionary wrong encoding https://github.com/klauspost/compress/pull/871
- flate: Reduce allocations in decompressor and minor code improvements by @fakefloordiv in https://github.com/klauspost/compress/pull/869
- s2: Fix EstimateBlockSize on 6&7 length input https://github.com/klauspost/compress/pull/867
Sept 19th, 2023 - v1.17.0
- Add experimental dictionary builder https://github.com/klauspost/compress/pull/853
- Add xerial snappy read/writer https://github.com/klauspost/compress/pull/838
- flate: Add limited window compression https://github.com/klauspost/compress/pull/843
- s2: Do 2 overlapping match checks https://github.com/klauspost/compress/pull/839
- flate: Add amd64 assembly matchlen https://github.com/klauspost/compress/pull/837
- gzip: Copy bufio.Reader on Reset by @thatguystone in https://github.com/klauspost/compress/pull/860

See changes to v1.16.x

July 1st, 2023 - v1.16.7
- zstd: Fix default level first dictionary encode https://github.com/klauspost/compress/pull/829
- s2: add GetBufferCapacity() method by @GiedriusS in https://github.com/klauspost/compress/pull/832
June 13, 2023 - v1.16.6
- zstd: correctly ignore WithEncoderPadding(1) by @ianlancetaylor in https://github.com/klauspost/compress/pull/806
- zstd: Add amd64 match length assembly https://github.com/klauspost/compress/pull/824
- gzhttp: Handle informational headers by @rtribotte in https://github.com/klauspost/compress/pull/815
- s2: Improve Better compression slightly https://github.com/klauspost/compress/pull/663
Apr 16, 2023 - v1.16.5
- zstd: readByte needs to use io.ReadFull by @jnoxon in https://github.com/klauspost/compress/pull/802
- gzip: Fix WriterTo after initial read https://github.com/klauspost/compress/pull/804
Apr 5, 2023 - v1.16.4
- zstd: Improve zstd best efficiency by @greatroar and @klauspost in https://github.com/klauspost/compress/pull/784
- zstd: Respect WithAllLitEntropyCompression https://github.com/klauspost/compress/pull/792
- zstd: Fix amd64 not always detecting corrupt data https://github.com/klauspost/compress/pull/785
- zstd: Various minor improvements by @greatroar in https://github.com/klauspost/compress/pull/788 https://github.com/klauspost/compress/pull/794 https://github.com/klauspost/compress/pull/795
- s2: Fix huge block overflow https://github.com/klauspost/compress/pull/779
- s2: Allow CustomEncoder fallback https://github.com/klauspost/compress/pull/780
- gzhttp: Support ResponseWriter Unwrap() in gzhttp handler by @jgimenez in https://github.com/klauspost/compress/pull/799
Mar 13, 2023 - v1.16.1
- zstd: Speed up + improve best encoder by @greatroar in https://github.com/klauspost/compress/pull/776
- gzhttp: Add optional BREACH mitigation. https://github.com/klauspost/compress/pull/762 https://github.com/klauspost/compress/pull/768 https://github.com/klauspost/compress/pull/769 https://github.com/klauspost/compress/pull/770 https://github.com/klauspost/compress/pull/767
- s2: Add Intel LZ4s converter https://github.com/klauspost/compress/pull/766
- zstd: Minor bug fixes https://github.com/klauspost/compress/pull/771 https://github.com/klauspost/compress/pull/772 https://github.com/klauspost/compress/pull/773
- huff0: Speed up compress1xDo by @greatroar in https://github.com/klauspost/compress/pull/774
Feb 26, 2023 - v1.16.0
- s2: Add Dictionary support. https://github.com/klauspost/compress/pull/685
- s2: Add Compression Size Estimate. https://github.com/klauspost/compress/pull/752
- s2: Add support for custom stream encoder. https://github.com/klauspost/compress/pull/755
- s2: Add LZ4 block converter. https://github.com/klauspost/compress/pull/748
- s2: Support io.ReaderAt in ReadSeeker. https://github.com/klauspost/compress/pull/747
- s2c/s2sx: Use concurrent decoding. https://github.com/klauspost/compress/pull/746

See changes to v1.15.x

Jan 21st, 2023 (v1.15.15)
- deflate: Improve level 7-9 https://github.com/klauspost/compress/pull/739
- zstd: Add delta encoding support by @greatroar in https://github.com/klauspost/compress/pull/728
- zstd: Various speed improvements by @greatroar https://github.com/klauspost/compress/pull/741 https://github.com/klauspost/compress/pull/734 https://github.com/klauspost/compress/pull/736 https://github.com/klauspost/compress/pull/744 https://github.com/klauspost/compress/pull/743 https://github.com/klauspost/compress/pull/745
- gzhttp: Add SuffixETag() and DropETag() options to prevent ETag collisions on compressed responses by @willbicks in https://github.com/klauspost/compress/pull/740
Jan 3rd, 2023 (v1.15.14)
- flate: Improve speed in big stateless blocks https://github.com/klauspost/compress/pull/718
- zstd: Minor speed tweaks by @greatroar in https://github.com/klauspost/compress/pull/716 https://github.com/klauspost/compress/pull/720
- export NoGzipResponseWriter for custom ResponseWriter wrappers by @harshavardhana in https://github.com/klauspost/compress/pull/722
- s2: Add example for indexing and existing stream https://github.com/klauspost/compress/pull/723
Dec 11, 2022 (v1.15.13)
- zstd: Add MaxEncodedSize to encoder https://github.com/klauspost/compress/pull/691
- zstd: Various tweaks and improvements https://github.com/klauspost/compress/pull/693 https://github.com/klauspost/compress/pull/695 https://github.com/klauspost/compress/pull/696 https://github.com/klauspost/compress/pull/701 https://github.com/klauspost/compress/pull/702 https://github.com/klauspost/compress/pull/703 https://github.com/klauspost/compress/pull/704 https://github.com/klauspost/compress/pull/705 https://github.com/klauspost/compress/pull/706 https://github.com/klauspost/compress/pull/707 https://github.com/klauspost/compress/pull/708
Oct 26, 2022 (v1.15.12)
- zstd: Tweak decoder allocs. https://github.com/klauspost/compress/pull/680
- gzhttp: Always delete HeaderNoCompression https://github.com/klauspost/compress/pull/683
Sept 26, 2022 (v1.15.11)
- flate: Improve level 1-3 compression https://github.com/klauspost/compress/pull/678
- zstd: Improve "best" compression by @nightwolfz in https://github.com/klauspost/compress/pull/677
- zstd: Fix+reduce decompression allocations https://github.com/klauspost/compress/pull/668
- zstd: Fix non-effective noescape tag https://github.com/klauspost/compress/pull/667
Sept 16, 2022 (v1.15.10)
- zstd: Add WithDecodeAllCapLimit https://github.com/klauspost/compress/pull/649
- Add Go 1.19 - deprecate Go 1.16 https://github.com/klauspost/compress/pull/651
- flate: Improve level 5+6 compression https://github.com/klauspost/compress/pull/656
- zstd: Improve "better" compression https://github.com/klauspost/compress/pull/657
- s2: Improve "best" compression https://github.com/klauspost/compress/pull/658
- s2: Improve "better" compression. https://github.com/klauspost/compress/pull/635
- s2: Slightly faster non-assembly decompression https://github.com/klauspost/compress/pull/646
- Use arrays for constant size copies https://github.com/klauspost/compress/pull/659
July 21, 2022 (v1.15.9)
- zstd: Fix decoder crash on amd64 (no BMI) on invalid input https://github.com/klauspost/compress/pull/645
- zstd: Disable decoder extended memory copies (amd64) due to possible crashes https://github.com/klauspost/compress/pull/644
- zstd: Allow single segments up to "max decoded size" https://github.com/klauspost/compress/pull/643
July 13, 2022 (v1.15.8)
- gzip: fix stack exhaustion bug in Reader.Read https://github.com/klauspost/compress/pull/641
- s2: Add Index header trim/restore https://github.com/klauspost/compress/pull/638
- zstd: Optimize seqdeq amd64 asm by @greatroar in https://github.com/klauspost/compress/pull/636
- zstd: Improve decoder memcopy https://github.com/klauspost/compress/pull/637
- huff0: Pass a single bitReader pointer to asm by @greatroar in https://github.com/klauspost/compress/pull/634
- zstd: Branchless getBits for amd64 w/o BMI2 by @greatroar in https://github.com/klauspost/compress/pull/640
- gzhttp: Remove header before writing https://github.com/klauspost/compress/pull/639
June 29, 2022 (v1.15.7)
- s2: Fix absolute forward seeks https://github.com/klauspost/compress/pull/633
- zip: Merge upstream https://github.com/klauspost/compress/pull/631
- zip: Re-add zip64 fix https://github.com/klauspost/compress/pull/624
- zstd: translate fseDecoder.buildDtable into asm by @WojciechMula in https://github.com/klauspost/compress/pull/598
- flate: Faster histograms https://github.com/klauspost/compress/pull/620
- deflate: Use compound hcode https://github.com/klauspost/compress/pull/622
June 3, 2022 (v1.15.6)
- s2: Improve coding for long, close matches https://github.com/klauspost/compress/pull/613
- s2c: Add Snappy/S2 stream recompression https://github.com/klauspost/compress/pull/611
- zstd: Always use configured block size https://github.com/klauspost/compress/pull/605
- zstd: Fix incorrect hash table placement for dict encoding in default https://github.com/klauspost/compress/pull/606
- zstd: Apply default config to ZipDecompressor without options https://github.com/klauspost/compress/pull/608
- gzhttp: Exclude more common archive formats https://github.com/klauspost/compress/pull/612
- s2: Add ReaderIgnoreCRC https://github.com/klauspost/compress/pull/609
- s2: Remove sanity load on index creation https://github.com/klauspost/compress/pull/607
- snappy: Use dedicated function for scoring https://github.com/klauspost/compress/pull/614
- s2c+s2d: Use official snappy framed extension https://github.com/klauspost/compress/pull/610
May 25, 2022 (v1.15.5)
- s2: Add concurrent stream decompression https://github.com/klauspost/compress/pull/602
- s2: Fix final emit oob read crash on amd64 https://github.com/klauspost/compress/pull/601
- huff0: asm implementation of Decompress1X by @WojciechMula https://github.com/klauspost/compress/pull/596
- zstd: Use 1 less goroutine for stream decoding https://github.com/klauspost/compress/pull/588
- zstd: Copy literal in 16 byte blocks when possible https://github.com/klauspost/compress/pull/592
- zstd: Speed up when WithDecoderLowmem(false) https://github.com/klauspost/compress/pull/599
- zstd: faster next state update in BMI2 version of decode by @WojciechMula in https://github.com/klauspost/compress/pull/593
- huff0: Do not check max size when reading table. https://github.com/klauspost/compress/pull/586
- flate: Inplace hashing for level 7-9 https://github.com/klauspost/compress/pull/590
May 11, 2022 (v1.15.4)
- huff0: decompress directly into output by @WojciechMula in #577
- inflate: Keep dict on stack #581
- zstd: Faster decoding memcopy in asm #583
- zstd: Fix ignored crc #580
May 5, 2022 (v1.15.3)
- zstd: Allow to ignore checksum checking by @WojciechMula #572
- s2: Fix incorrect seek for io.SeekEnd in #575
Apr 26, 2022 (v1.15.2)
- zstd: Add x86-64 assembly for decompression on streams and blocks. Contributed by @WojciechMula. Typically 2x faster. #528 #531 #545 #537
- zstd: Add options to ZipDecompressor and fixes #539
- s2: Use sorted search for index #555
- Minimum version is Go 1.16, added CI test on 1.18.
Mar 11, 2022 (v1.15.1)
- huff0: Add x86 assembly of Decode4X by @WojciechMula in #512
- zstd: Reuse zip decoders in #514
- zstd: Detect extra block data and report as corrupted in #520
- zstd: Handle zero sized frame content size stricter in #521
- zstd: Add stricter block size checks in #523
Mar 3, 2022 (v1.15.0)
- zstd: Refactor decoder #498
- zstd: Add stream encoding without goroutines #505
- huff0: Prevent single blocks exceeding 16 bits by @klauspost in#507
- flate: Inline literal emission #509
- gzhttp: Add zstd to transport #400
- gzhttp: Make content-type optional #510

Both compression and decompression now supports "synchronous" stream operations. This means that whenever "concurrency" is set to 1, they will operate without spawning goroutines.

Stream decompression is now faster on asynchronous, since the goroutine allocation much more effectively splits the workload. On typical streams this will typically use 2 cores fully for decompression. When a stream has finished decoding no goroutines will be left over, so decoders can now safely be pooled and still be garbage collected.

While the release has been extensively tested, it is recommended to testing when upgrading.

See changes to v1.14.x

Feb 22, 2022 (v1.14.4)
- flate: Fix rare huffman only (-2) corruption. #503
- zip: Update deprecated CreateHeaderRaw to correctly call CreateRaw by @saracen in #502
- zip: don't read data descriptor early by @saracen in #501 #501
- huff0: Use static decompression buffer up to 30% faster #499 #500
Feb 17, 2022 (v1.14.3)
- flate: Improve fastest levels compression speed ~10% more throughput. #482 #489 #490 #491 #494 #478
- flate: Faster decompression speed, ~5-10%. #483
- s2: Faster compression with Go v1.18 and amd64 microarch level 3+. #484 #486
Jan 25, 2022 (v1.14.2)
- zstd: improve header decoder by @dsnet #476
- zstd: Add bigger default blocks #469
- zstd: Remove unused decompression buffer #470
- zstd: Fix logically dead code by @ningmingxiao #472
- flate: Improve level 7-9 #471 #473
- zstd: Add noasm tag for xxhash #475
Jan 11, 2022 (v1.14.1)
- s2: Add stream index in #462
- flate: Speed and efficiency improvements in #439 #461 #455 #452 #458
- zstd: Performance improvement in #420 #456 #437 #467 #468
- zstd: add arm64 xxhash assembly in #464
- Add garbled for binaries for s2 in #445

See changes to v1.13.x

Aug 30, 2021 (v1.13.5)
- gz/zlib/flate: Alias stdlib errors #425
- s2: Add block support to commandline tools #413
- zstd: pooledZipWriter should return Writers to the same pool #426
- Removed golang/snappy as external dependency for tests #421
Aug 12, 2021 (v1.13.4)
- Add snappy replacement package.
- zstd: Fix incorrect encoding in "best" mode #415
Aug 3, 2021 (v1.13.3)
- zstd: Improve Best compression #404
- zstd: Fix WriteTo error forwarding #411
- gzhttp: Return http.HandlerFunc instead of http.Handler. Unlikely breaking change. #406
- s2sx: Fix max size error #399
- zstd: Add optional stream content size on reset #401
- zstd: use SpeedBestCompression for level >= 10 #410
Jun 14, 2021 (v1.13.1)
- s2: Add full Snappy output support #396
- zstd: Add configurable Decoder window size #394
- gzhttp: Add header to skip compression #389
- s2: Improve speed with bigger output margin #395
Jun 3, 2021 (v1.13.0)
- Added gzhttp which allows wrapping HTTP servers and clients with GZIP compressors.
- zstd: Detect short invalid signatures #382
- zstd: Spawn decoder goroutine only if needed. #380

See changes to v1.12.x

May 25, 2021 (v1.12.3)
- deflate: Better/faster Huffman encoding #374
- deflate: Allocate less for history. #375
- zstd: Forward read errors #373
Apr 27, 2021 (v1.12.2)
- zstd: Improve better/best compression #360 #364 #365
- zstd: Add helpers to compress/decompress zstd inside zip files #363
- deflate: Improve level 5+6 compression #367
- s2: Improve better/best compression #358 #359
- s2: Load after checking src limit on amd64. #362
- s2sx: Limit max executable size #368
Apr 14, 2021 (v1.12.1)
- snappy package removed. Upstream added as dependency.
- s2: Better compression in "best" mode #353
- s2sx: Add stdin input and detect pre-compressed from signature #352
- s2c/s2d: Add http as possible input #348
- s2c/s2d/s2sx: Always truncate when writing files #352
- zstd: Reduce memory usage further when using WithLowerEncoderMem #346
- s2: Fix potential problem with amd64 assembly and profilers #349

See changes to v1.11.x

Mar 26, 2021 (v1.11.13)
- zstd: Big speedup on small dictionary encodes #344 #345
- zstd: Add WithLowerEncoderMem encoder option #336
- deflate: Improve entropy compression #338
- s2: Clean up and minor performance improvement in best #341
Mar 5, 2021 (v1.11.12)
- s2: Add s2sx binary that creates self extracting archives.
- s2: Speed up decompression on non-assembly platforms #328
Mar 1, 2021 (v1.11.9)
- s2: Add ARM64 decompression assembly. Around 2x output speed. #324
- s2: Improve "better" speed and efficiency. #325
- s2: Fix binaries.
Feb 25, 2021 (v1.11.8)
- s2: Fixed occasional out-of-bounds write on amd64. Upgrade recommended.
- s2: Add AMD64 assembly for better mode. 25-50% faster. #315
- s2: Less upfront decoder allocation. #322
- zstd: Faster "compression" of incompressible data. #314
- zip: Fix zip64 headers. #313
Jan 14, 2021 (v1.11.7)
- Use Bytes() interface to get bytes across packages. #309
- s2: Add 'best' compression option. #310
- s2: Add ReaderMaxBlockSize, changes s2.NewReader signature to include varargs. #311
- s2: Fix crash on small better buffers. #308
- s2: Clean up decoder. #312
Jan 7, 2021 (v1.11.6)
- zstd: Make decoder allocations smaller #306
- zstd: Free Decoder resources when Reset is called with a nil io.Reader #305
Dec 20, 2020 (v1.11.4)
- zstd: Add Best compression mode #304
- Add header decoder #299
- s2: Add uncompressed stream option #297
- Simplify/speed up small blocks with known max size. #300
- zstd: Always reset literal dict encoder #303
Nov 15, 2020 (v1.11.3)
- inflate: 10-15% faster decompression #293
- zstd: Tweak DecodeAll default allocation #295
Oct 11, 2020 (v1.11.2)
- s2: Fix out of bounds read in "better" block compression #291
Oct 1, 2020 (v1.11.1)
- zstd: Set allLitEntropy true in default configuration #286
Sept 8, 2020 (v1.11.0)
- zstd: Add experimental compression dictionaries #281
- zstd: Fix mixed Write and ReadFrom calls #282
- inflate/gz: Limit variable shifts, ~5% faster decompression #274

See changes to v1.10.x

July 8, 2020 (v1.10.11)
- zstd: Fix extra block when compressing with ReadFrom. #278
- huff0: Also populate compression table when reading decoding table. #275
June 23, 2020 (v1.10.10)
- zstd: Skip entropy compression in fastest mode when no matches. #270
June 16, 2020 (v1.10.9):
- zstd: API change for specifying dictionaries. See #268
- zip: update CreateHeaderRaw to handle zip64 fields. #266
- Fuzzit tests removed. The service has been purchased and is no longer available.
June 5, 2020 (v1.10.8):
- 1.15x faster zstd block decompression. #265
June 1, 2020 (v1.10.7):
- Added zstd decompression dictionary support
- Increase zstd decompression speed up to 1.19x. #259
- Remove internal reset call in zstd compression and reduce allocations. #263
May 21, 2020: (v1.10.6)
- zstd: Reduce allocations while decoding. #258, #252
- zstd: Stricter decompression checks.
April 12, 2020: (v1.10.5)
- s2-commands: Flush output when receiving SIGINT. #239
Apr 8, 2020: (v1.10.4)
- zstd: Minor/special case optimizations. #251, #250, #249, #247
Mar 11, 2020: (v1.10.3)
- s2: Use S2 encoder in pure Go mode for Snappy output as well. #245
- s2: Fix pure Go block encoder. #244
- zstd: Added "better compression" mode. #240
- zstd: Improve speed of fastest compression mode by 5-10% #241
- zstd: Skip creating encoders when not needed. #238
Feb 27, 2020: (v1.10.2)
- Close to 50% speedup in inflate (gzip/zip decompression). #236 #234 #232
- Reduce deflate level 1-6 memory usage up to 59%. #227
Feb 18, 2020: (v1.10.1)
- Fix zstd crash when resetting multiple times without sending data. #226
- deflate: Fix dictionary use on level 1-6. #224
- Remove deflate writer reference when closing. #224
Feb 4, 2020: (v1.10.0)
- Add optional dictionary to stateless deflate. Breaking change, send nil for previous behaviour. #216
- Fix buffer overflow on repeated small block deflate. #218
- Allow copying content from an existing ZIP file without decompressing+compressing. #214
- Added S2 AMD64 assembler and various optimizations. Stream speed >10GB/s. #186

See changes prior to v1.10.0

Jan 20,2020 (v1.9.8) Optimize gzip/deflate with better size estimates and faster table generation. #207 by luyu6056, #206.
Jan 11, 2020: S2 Encode/Decode will use provided buffer if capacity is big enough. #204
Jan 5, 2020: (v1.9.7) Fix another zstd regression in v1.9.5 - v1.9.6 removed.
Jan 4, 2020: (v1.9.6) Regression in v1.9.5 fixed causing corrupt zstd encodes in rare cases.
Jan 4, 2020: Faster IO in s2c + s2d commandline tools compression/decompression. #192
Dec 29, 2019: Removed v1.9.5 since fuzz tests showed a compatibility problem with the reference zstandard decoder.
Dec 29, 2019: (v1.9.5) zstd: 10-20% faster block compression. #199
Dec 29, 2019: zip package updated with latest Go features
Dec 29, 2019: zstd: Single segment flag condintions tweaked. #197
Dec 18, 2019: s2: Faster compression when ReadFrom is used. #198
Dec 10, 2019: s2: Fix repeat length output when just above at 16MB limit.
Dec 10, 2019: zstd: Add function to get decoder as io.ReadCloser. #191
Dec 3, 2019: (v1.9.4) S2: limit max repeat length. #188
Dec 3, 2019: Add WithNoEntropyCompression to zstd #187
Dec 3, 2019: Reduce memory use for tests. Check for leaked goroutines.
Nov 28, 2019 (v1.9.3) Less allocations in stateless deflate.
Nov 28, 2019: 5-20% Faster huff0 decode. Impacts zstd as well. #184
Nov 12, 2019 (v1.9.2) Added Stateless Compression for gzip/deflate.
Nov 12, 2019: Fixed zstd decompression of large single blocks. #180
Nov 11, 2019: Set default s2c block size to 4MB.
Nov 11, 2019: Reduce inflate memory use by 1KB.
Nov 10, 2019: Less allocations in deflate bit writer.
Nov 10, 2019: Fix inconsistent error returned by zstd decoder.
Oct 28, 2019 (v1.9.1) ztsd: Fix crash when compressing blocks. #174
Oct 24, 2019 (v1.9.0) zstd: Fix rare data corruption #173
Oct 24, 2019 zstd: Fix huff0 out of buffer write #171 and always return errors #172
Oct 10, 2019: Big deflate rewrite, 30-40% faster with better compression #105

See changes prior to v1.9.0

Oct 10, 2019: (v1.8.6) zstd: Allow partial reads to get flushed data. #169
Oct 3, 2019: Fix inconsistent results on broken zstd streams.
Sep 25, 2019: Added -rm (remove source files) and -q (no output except errors) to s2c and s2d commands
Sep 16, 2019: (v1.8.4) Add s2c and s2d commandline tools.
Sep 10, 2019: (v1.8.3) Fix s2 decoder Skip.
Sep 7, 2019: zstd: Added WithWindowSize, contributed by ianwilkes.
Sep 5, 2019: (v1.8.2) Add WithZeroFrames which adds full zero payload block encoding option.
Sep 5, 2019: Lazy initialization of zstandard predefined en/decoder tables.
Aug 26, 2019: (v1.8.1) S2: 1-2% compression increase in "better" compression mode.
Aug 26, 2019: zstd: Check maximum size of Huffman 1X compressed literals while decoding.
Aug 24, 2019: (v1.8.0) Added S2 compression, a high performance replacement for Snappy.
Aug 21, 2019: (v1.7.6) Fixed minor issues found by fuzzer. One could lead to zstd not decompressing.
Aug 18, 2019: Add fuzzit continuous fuzzing.
Aug 14, 2019: zstd: Skip incompressible data 2x faster. #147
Aug 4, 2019 (v1.7.5): Better literal compression. #146
Aug 4, 2019: Faster zstd compression. #143 #144
Aug 4, 2019: Faster zstd decompression. #145 #143 #142
July 15, 2019 (v1.7.4): Fix double EOF block in rare cases on zstd encoder.
July 15, 2019 (v1.7.3): Minor speedup/compression increase in default zstd encoder.
July 14, 2019: zstd decoder: Fix decompression error on multiple uses with mixed content.
July 7, 2019 (v1.7.2): Snappy update, zstd decoder potential race fix.
June 17, 2019: zstd decompression bugfix.
June 17, 2019: fix 32 bit builds.
June 17, 2019: Easier use in modules (less dependencies).
June 9, 2019: New stronger "default" zstd compression mode. Matches zstd default compression ratio.
June 5, 2019: 20-40% throughput in zstandard compression and better compression.
June 5, 2019: deflate/gzip compression: Reduce memory usage of lower compression levels.
June 2, 2019: Added zstandard compression!
May 25, 2019: deflate/gzip: 10% faster bit writer, mostly visible in lower levels.
Apr 22, 2019: zstd decompression added.
Aug 1, 2018: Added huff0 README.
Jul 8, 2018: Added Performance Update 2018 below.
Jun 23, 2018: Merged Go 1.11 inflate optimizations. Go 1.9 is now required. Backwards compatible version tagged with v1.3.0.
Apr 2, 2018: Added huff0 en/decoder. Experimental for now, API may change.
Mar 4, 2018: Added FSE Entropy en/decoder. Experimental for now, API may change.
Nov 3, 2017: Add compression Estimate function.
May 28, 2017: Reduce allocations when resetting decoder.
Apr 02, 2017: Change back to official crc32, since changes were merged in Go 1.7.
Jan 14, 2017: Reduce stack pressure due to array copies. See Issue #18625.
Oct 25, 2016: Level 2-4 have been rewritten and now offers significantly better performance than before.
Oct 20, 2016: Port zlib changes from Go 1.7 to fix zlib writer issue. Please update.
Oct 16, 2016: Go 1.7 changes merged. Apples to apples this package is a few percent faster, but has a significantly better balance between speed and compression per level.
Mar 24, 2016: Always attempt Huffman encoding on level 4-7. This improves base 64 encoded data compression.
Mar 24, 2016: Small speedup for level 1-3.
Feb 19, 2016: Faster bit writer, level -2 is 15% faster, level 1 is 4% faster.
Feb 19, 2016: Handle small payloads faster in level 1-3.
Feb 19, 2016: Added faster level 2 + 3 compression modes.
Feb 19, 2016: Rebalanced compression levels, so there is a more even progression in terms of compression. New default level is 5.
Feb 14, 2016: Snappy: Merge upstream changes.
Feb 14, 2016: Snappy: Fix aggressive skipping.
Feb 14, 2016: Snappy: Update benchmark.
Feb 13, 2016: Deflate: Fixed assembler problem that could lead to sub-optimal compression.
Feb 12, 2016: Snappy: Added AMD64 SSE 4.2 optimizations to matching, which makes easy to compress material run faster. Typical speedup is around 25%.
Feb 9, 2016: Added Snappy package fork. This version is 5-7% faster, much more on hard to compress content.
Jan 30, 2016: Optimize level 1 to 3 by not considering static dictionary or storing uncompressed. ~4-5% speedup.
Jan 16, 2016: Optimization on deflate level 1,2,3 compression.
Jan 8 2016: Merge CL 18317: fix reading, writing of zip64 archives.
Dec 8 2015: Make level 1 and -2 deterministic even if write size differs.
Dec 8 2015: Split encoding functions, so hashing and matching can potentially be inlined. 1-3% faster on AMD64. 5% faster on other platforms.
Dec 8 2015: Fixed rare one byte out-of bounds read. Please update!
Nov 23 2015: Optimization on token writer. ~2-4% faster. Contributed by @dsnet.
Nov 20 2015: Small optimization to bit writer on 64 bit systems.
Nov 17 2015: Fixed out-of-bound errors if the underlying Writer returned an error. See #15.
Nov 12 2015: Added io.WriterTo support to gzip/inflate.
Nov 11 2015: Merged CL 16669: archive/zip: enable overriding (de)compressors per file
Oct 15 2015: Added skipping on uncompressible data. Random data speed up >5x.

deflate usage

The packages are drop-in replacements for standard library deflate, gzip, zip, and zlib. Simply replace the import path to use them:

Typical speed is about 2x of the standard library packages.

old import	new import	Documentation
`compress/gzip`	`github.com/klauspost/compress/gzip`	gzip
`compress/zlib`	`github.com/klauspost/compress/zlib`	zlib
`archive/zip`	`github.com/klauspost/compress/zip`	zip
`compress/flate`	`github.com/klauspost/compress/flate`	flate

You may also be interested in pgzip, which is a drop-in replacement for gzip, which support multithreaded compression on big files and the optimized crc32 package used by these packages.

The packages implement the same API as the standard library, so you can use the original godoc documentation: gzip, zip, zlib, flate.

Currently there is only minor speedup on decompression (mostly CRC32 calculation).

Memory usage is typically 1MB for a Writer. stdlib is in the same range.
If you expect to have a lot of concurrently allocated Writers consider using
the stateless compression described below.

For compression performance, see: this spreadsheet.

To disable all assembly add -tags=noasm. This works across all packages.

Stateless compression

This package offers stateless compression as a special option for gzip/deflate.
It will do compression but without maintaining any state between Write calls.

This means there will be no memory kept between Write calls, but compression and speed will be suboptimal.

This is only relevant in cases where you expect to run many thousands of compressors concurrently,
but with very little activity. This is not intended for regular web servers serving individual requests.

Because of this, the size of actual Write calls will affect output size.

In gzip, specify level -3 / gzip.StatelessCompression to enable.

For direct deflate use, NewStatelessWriter and StatelessDeflate are available. See documentation

A bufio.Writer can of course be used to control write sizes. For example, to use a 4KB buffer:

// replace 'ioutil.Discard' with your output.
	gzw, err := gzip.NewWriterLevel(ioutil.Discard, gzip.StatelessCompression)
	if err != nil {
		return err
	}
	defer gzw.Close()

	w := bufio.NewWriterSize(gzw, 4096)
	defer w.Flush()
	
	// Write to 'w'

This will only use up to 4KB in memory when the writer is idle.

Compression is almost always worse than the fastest compression level
and each write will allocate (a little) memory.

Other packages

Here are other packages of good quality and pure Go (no cgo wrappers or autoconverted code):

github.com/pierrec/lz4 - strong multithreaded LZ4 compression.
github.com/cosnicolaou/pbzip2 - multithreaded bzip2 decompression.
github.com/dsnet/compress - brotli decompression, bzip2 writer.
github.com/ronanh/intcomp - Integer compression.
github.com/spenczar/fpc - Float compression.
github.com/minio/zipindex - External ZIP directory index.
github.com/ybirader/pzip - Fast concurrent zip archiver and extractor.

license

This code is licensed under the same conditions as the original Go code. See LICENSE file.

Overview (`vendor/github.com/mschoch/smat/README.md`)

smat – State Machine Assisted Testing

The concept is simple, describe valid uses of your library as states and actions. States describe which actions are possible, and with what probability they should occur. Actions mutate the context and transition to another state.

By doing this, two things are possible:

Use go-fuzz to find/test interesting sequences of operations on your library.
Automate longevity testing of your application by performing long sequences of valid operations.

NOTE: both of these can also incorporate validation logic (not just failure detection by building validation into the state machine)

Status

The API is still not stable. This is brand new and we'll probably change things we don't like...

License

Apache 2.0

How do I use it?

smat.Context

Choose a structure to keep track of any state. You pass in an instance of this when you start, and it will be passed to every action when it executes. The actions may mutate this context.

For example, consider a database library, once you open a database handle, you need to use it inside of the other actions. So you might use a structure like:

text

type context struct {
  db *DB
}

smat.State

A state represents a state that your application/library can be in, and the probabilities thats certain actions should be taken.

For example, consider a database library, in a state where the database is open, there many things you can do. Let's consider just two right now, you can set a value, or you can delete a value.

text

func dbOpen(next byte) smat.ActionID {
	return smat.PercentExecute(next,
		smat.PercentAction{50, setValue},
		smat.PercentAction{50, deleteValue},
	)
}

This says that in the open state, there are two valid actions, 50% of the time you should set a value and 50% of the time you should delete a value. NOTE: these percentages are just for characterizing the test workload.

smat.Action

Actions are functions that do some work, optionally mutate the context, and indicate the next state to transition to. Below we see an example action to set value in a database.

text

func setValueFunc(ctx smat.Context) (next smat.State, err error) {
  // type assert to our custom context type
	context := ctx.(*context)
  // perform the operation
  err = context.db.Set("k", "v")
  if err != nil {
    return nil, err
  }
  // return the new state
  return dbOpen, nil
}

smat.ActionID and smat.ActionMap

Actions are just functions, and since we can't compare functions in Go, we need to introduce an external identifier for them. This allows us to build a bi-directional mapping which we'll take advantage of later.

text

const (
  setup smat.ActionID = iota
  teardown
  setValue
  deleteValue
)

var actionMap = smat.ActionMap{
  setup:       setupFunc,
  teardown:    teardownFunc,
	setValue:    setValueFunc,
	deleteValue: deleteValueFunc,
}

smat.ActionSeq

A common way that many users think about a library is as a sequence of actions to be performed. Using the ActionID's that we've already seen we can build up sequences of operations.

text

actionSeq := smat.ActionSeq{
		open,
		setValue,
		setValue,
		setValue,
	}

Notice that we build these actions using the constants we defined above, and because of this we can have a bi-directional mapping between a stream of bytes (driving the state machine) and a sequence of actions to be performed.

Fuzzing

We've built a lot of pieces, lets wire it up to go-fuzz.

text

func Fuzz(data []byte) int {
	return smat.Fuzz(&context{}, setup, teardown, actionMap, data)
}

The first argument is an instance of context structure.
The second argument is the ActionID of our setup function. The setup function does not consume any of the input stream and is used to initialize the context and determine the start state.
The third argument is the teardown function. This will be called unconditionally to clean up any resources associated with the test.
The fourth argument is the actionMap which maps all ActionIDs to Actions.
The fifth argument is the data passed in from the go-fuzz application.

Generating Initial go-fuzz Corpus

Earlier we mentioned the bi-directional mapping between Actions and the byte stream driving the state machine. We can now leverage this to build the inital go-fuzz corpus.

Using the ActinSeqs we learned about earlier we can build up a list of them as:

text

var actionSeqs = []smat.ActionSeq{...}

Then, we can write them out to disk using:

text

for i, actionSeq := range actionSeqs {
  byteSequence, err := actionSeq.ByteEncoding(&context{}, setup, teardown, actionMap)
  if err != nil {
    // handle error
  }
  os.MkdirAll("workdir/corpus", 0700)
  ioutil.WriteFile(fmt.Sprintf("workdir/corpus/%d", i), byteSequence, 0600)
}

You can then either put this into a test case or a main application depending on your needs.

Longevity Testing

Fuzzing is great, but most of your corpus is likely to be shorter meaningful sequences. And go-fuzz works to find shortest sequences that cause problems, but sometimes you actually want to explore longer sequences that appear to go-fuzz as not triggering additional code coverage.

For these cases we have another helper you can use:

text

Longevity(ctx, setup, teardown, actionMap, 0, closeChan)

The first four arguments are the same, the last two are:

random seed used to ensure repeatable tests
closeChan (chan struct{}) - close this channel if you want the function to stop and return ErrClosed, otherwise it will run forever

Examples

See the examples directory for a working example that tests some BoltDB functionality.

Overview (`vendor/github.com/pkg/errors/README.md`)

errors [](https://travis-ci.org/pkg/errors) [](https://ci.appveyor.com/project/davecheney/errors/branch/master) [](http://godoc.org/github.com/pkg/errors) [](https://goreportcard.com/report/github.com/pkg/errors) [](https://sourcegraph.com/github.com/pkg/errors?badge)

Package errors provides simple error handling primitives.

go get github.com/pkg/errors

The traditional error handling idiom in Go is roughly akin to

if err != nil {
        return err
}

which applied recursively up the call stack results in error reports without context or debugging information. The errors package allows programmers to add context to the failure path in their code in a way that does not destroy the original value of the error.

Adding context to an error

The errors.Wrap function returns a new error that adds context to the original error. For example

_, err := ioutil.ReadAll(r)
if err != nil {
        return errors.Wrap(err, "read failed")
}

Retrieving the cause of an error

Using errors.Wrap constructs a stack of errors, adding context to the preceding error. Depending on the nature of the error it may be necessary to reverse the operation of errors.Wrap to retrieve the original error for inspection. Any error value which implements this interface can be inspected by errors.Cause.

type causer interface {
        Cause() error
}

errors.Cause will recursively retrieve the topmost error which does not implement causer, which is assumed to be the original cause. For example:

switch err := errors.Cause(err).(type) {
case *MyError:
        // handle specifically
default:
        // unknown error
}

Read the package documentation for more information.

Roadmap

With the upcoming Go2 error proposals this package is moving into maintenance mode. The roadmap for a 1.0 release is as follows:

0.9. Remove pre Go 1.9 and Go 1.10 support, address outstanding pull requests (if possible)
1.0. Final release.

Contributing

Because of the Go2 errors changes, this package is not accepting proposals for new functionality. With that said, we welcome pull requests, bug fixes and issue reports.

Before sending a PR, please discuss your change by raising an issue.

License

BSD-2-Clause

Overview (`vendor/github.com/prometheus/procfs/README.md`)

procfs

This package provides functions to retrieve system, kernel, and process
metrics from the pseudo-filesystems /proc and /sys.

WARNING: This package is a work in progress. Its API may still break in
backwards-incompatible ways without warnings. Use it at your own risk.

Usage

The procfs library is organized by packages based on whether the gathered data is coming from
/proc, /sys, or both. Each package contains an FS type which represents the path to either /proc,
/sys, or both. For example, cpu statistics are gathered from
/proc/stat and are available via the root procfs package. First, the proc filesystem mount
point is initialized, and then the stat information is read.

fs, err := procfs.NewFS("/proc")
stats, err := fs.Stat()

Some sub-packages such as blockdevice, require access to both the proc and sys filesystems.

fs, err := blockdevice.NewFS("/proc", "/sys")
    stats, err := fs.ProcDiskstats()

Package Organization

The packages in this project are organized according to (1) whether the data comes from the /proc or
/sys filesystem and (2) the type of information being retrieved. For example, most process information
can be gathered from the functions in the root procfs package. Information about block devices such as disk drives
is available in the blockdevices sub-package.

Building and Testing

The procfs library is intended to be built as part of another application, so there are no distributable binaries.
However, most of the API includes unit tests which can be run with make test.

Updating Test Fixtures

The procfs library includes a set of test fixtures which include many example files from
the /proc and /sys filesystems. These fixtures are included as a ttar file
which is extracted automatically during testing. To add/update the test fixtures, first
ensure the testdata/fixtures directory is up to date by removing the existing directory and then
extracting the ttar file using make testdata/fixtures/.unpacked or just make test.

bash

rm -rf testdata/fixtures
make test

Next, make the required changes to the extracted files in the testdata/fixtures directory. When
the changes are complete, run make update_fixtures to create a new fixtures.ttar file
based on the updated fixtures directory. And finally, verify the changes using
git diff testdata/fixtures.ttar.

Overview (`vendor/github.com/twmb/murmur3/README.md`)

murmur3

Native Go implementation of Austin Appleby's third MurmurHash revision (aka
MurmurHash3).

Includes 32, 64, and 128 bit sums, seeding functions, string functions that
hash without converting to a slice, and streaming hashes implementing Go's
standard Hash and
Cloner interfaces.

This library started as a fork of spaolacci/murmur3.
The reference algorithm has been slightly hacked as to support the streaming
mode required by Go's standard Hash interface.

Endianness

Unlike the canonical source, this library always reads bytes as little
endian numbers. This makes the hashes portable across architectures, although
does mean that hashing is a bit slower on big endian architectures.

Safety

This library uses no unsafe. Bytes are read with plain indexing, which the
compiler turns into single word sized loads on architectures that allow
unaligned access, and the loops are shaped so that the compiler proves every
index in bounds. Strings are hashed through one implementation generic over
string | []byte, so hashing a string does not copy it and does not
allocate.

Assembly

Earlier versions shipped hand rolled amd64 assembly for the 64 and 128 bit
sums. That assembly was removed once the compiler's output caught up: as of
Go 1.27, the pure Go code is faster than the old assembly for every input
under 64 bytes, for keys of varying length, and for every 32 bit sum, and 1
to 7 percent slower on fixed inputs of 64 bytes and more. The 32 bit assembly
was removed for the same reason back in Go 1.11. See the benchmarks below.

Testing

Testing includes comparing random inputs against the canonical
implementation,
and testing length 0 through 100 inputs to force the block loop, the trailing
block, and all tail lengths.

Because this code always reads input as little endian, testing against the
canonical source is skipped for big endian architectures. The canonical source
just converts bytes to numbers, meaning on big endian architectures, it will
use different numbers for its hashing.

Benchmarks

Cycles per call from perf (cycles:u at a fixed iteration count, minimum of
three runs) on a Comet Lake i7-10710U, so CPU frequency and thermal
throttling drop out. asm is the amd64 assembly this library shipped through
v1.1.8; the other columns are this code built by that Go release. Go 1.26
is the minimum this module declares. Sizes
rows exercise the block loop, Branches rows the 0 to 16 byte tail, and
RandomLengths rows draw an xorshift length in [0, 64) on every call.

text

benchmark              asm  go1.26  go1.27   1.26/asm 1.27/asm
128Branches/3           25      21      21     -16.7%   -16.7%
128Branches/7           26      21      21     -18.6%   -18.5%
128Branches/13          25      22      22     -12.5%   -14.5%
128Branches/16          27      26      26      -1.4%    -1.3%
128Sizes/32             35      34      34      -2.3%    -2.2%
128Sizes/64             48      48      49      +1.2%    +1.2%
128Sizes/256           140     149     149      +6.9%    +6.8%
128Sizes/1024          534     567     557      +6.2%    +4.3%
128Sizes/8192         4215    4361    4363      +3.5%    +3.5%
64Sizes/32              35      34      34      -3.7%    -3.5%
64Sizes/1024           535     575     557      +7.4%    +4.1%
64Sizes/8192          4221    4360    4366      +3.3%    +3.4%
32Branches/3            15      16      16      +5.2%    +5.6%
32Sizes/32              50      41      41     -18.2%   -18.0%
32Sizes/64              89      77      77     -13.6%   -13.6%
32Sizes/256            322     296     302      -7.9%    -6.2%
32Sizes/1024          1284    1243    1211      -3.2%    -5.7%
32Sizes/8192         10080    9896    9582      -1.8%    -4.9%
RandomLengths128       110      91      90     -18.0%   -18.2%
RandomLengths32         96      83      85     -13.5%   -11.4%

Overview (`vendor/go.yaml.in/yaml/v3/README.md`)

go.yaml.in/yaml

YAML Support for the Go Language

Introduction

The yaml package enables Go programs to comfortably encode
and decode YAML values.

It was originally developed within Canonical as
part of the juju project, and is based on a pure Go
port of the well-known libyaml C library to
parse and generate YAML data quickly and reliably.

Project Status

This project started as a fork of the extremely popular go-yaml
project, and is being maintained by the official YAML organization.

The YAML team took over ongoing maintenance and development of the project after
discussion with go-yaml's author, @niemeyer, following his decision to
label the project repository as "unmaintained" in April 2025.

We have put together a team of dedicated maintainers including representatives
of go-yaml's most important downstream projects.

We will strive to earn the trust of the various go-yaml forks to switch back to
this repository as their upstream.

Please contact us if you
would like to contribute or be involved.

Compatibility

The yaml package supports most of YAML 1.2, but preserves some behavior from
1.1 for backwards compatibility.

Specifically, v3 of the yaml package:

Supports YAML 1.1 bools (yes/no, on/off) as long as they are being
decoded into a typed bool value.
Otherwise they behave as a string.
Booleans in YAML 1.2 are true/false only.
Supports octals encoded and decoded as 0777 per YAML 1.1, rather than
0o777 as specified in YAML 1.2, because most parsers still use the old
format.
Octals in the 0o777 format are supported though, so new files work.
Does not support base-60 floats.
These are gone from YAML 1.2, and were actually never supported by this
package as it's clearly a poor choice.

Installation and Usage

The import path for the package is go.yaml.in/yaml/v3.

To install it, run:

bash

go get go.yaml.in/yaml/v3

API Documentation

See: https://pkg.go.dev/go.yaml.in/yaml/v3

API Stability

The package API for yaml v3 will remain stable as described in gopkg.in.

Example

package main

import (
	"fmt"
	"log"

	"go.yaml.in/yaml/v3"
)

var data = `
a: Easy!
b:
  c: 2
  d: [3, 4]
`

// Note: struct fields must be public in order for unmarshal to
// correctly populate the data.
type T struct {
	A string
	B struct {
		RenamedC int   `yaml:"c"`
		D	[]int `yaml:",flow"`
	}
}

func main() {
	t := T{}

	err := yaml.Unmarshal([]byte(data), &t)
	if err != nil {
		log.Fatalf("error: %v", err)
	}
	fmt.Printf("--- t:\n%v\n\n", t)

	d, err := yaml.Marshal(&t)
	if err != nil {
		log.Fatalf("error: %v", err)
	}
	fmt.Printf("--- t dump:\n%s\n\n", string(d))

	m := make(map[interface{}]interface{})

	err = yaml.Unmarshal([]byte(data), &m)
	if err != nil {
		log.Fatalf("error: %v", err)
	}
	fmt.Printf("--- m:\n%v\n\n", m)

	d, err = yaml.Marshal(&m)
	if err != nil {
		log.Fatalf("error: %v", err)
	}
	fmt.Printf("--- m dump:\n%s\n\n", string(d))
}

This example will generate the following output:

text

--- t:
{Easy! {2 [3 4]}}

--- t dump:
a: Easy!
b:
  c: 2
  d: [3, 4]


--- m:
map[a:Easy! b:map[c:2 d:[3 4]]]

--- m dump:
a: Easy!
b:
  c: 2
  d:
  - 3
  - 4

License

The yaml package is licensed under the MIT and Apache License 2.0 licenses.
Please see the LICENSE file for details.

Overview (`vendor/golang.org/x/oauth2/README.md`)

OAuth2 for Go

oauth2 package contains a client implementation for OAuth 2.0 spec.

See pkg.go.dev for further documentation and examples.

Policy for new endpoints

We no longer accept new provider-specific packages in this repo if all
they do is add a single endpoint variable. If you just want to add a
single endpoint, add it to the
pkg.go.dev/golang.org/x/oauth2/endpoints
package.

Report Issues / Send Patches

The main issue tracker for the oauth2 repository is located at
https://github.com/golang/oauth2/issues.

This repository uses Gerrit for code changes. To learn how to submit changes to
this repository, see https://go.dev/doc/contribute.

The git repository is https://go.googlesource.com/oauth2.

Note:

Excluding trivial changes, all contributions should be connected to an existing issue.
API changes must go through the change proposal process before they can be accepted.
The code owners are listed at dev.golang.org/owners.

Overview (`vendor/gopkg.in/natefinch/lumberjack.v2/README.md`)

lumberjack [](https://godoc.org/gopkg.in/natefinch/lumberjack.v2) [](https://travis-ci.org/natefinch/lumberjack) [](https://ci.appveyor.com/project/natefinch/lumberjack) [](https://coveralls.io/r/natefinch/lumberjack?branch=v2.0)

Lumberjack is a Go package for writing logs to rolling files.

Package lumberjack provides a rolling logger.

Note that this is v2.0 of lumberjack, and should be imported using gopkg.in
thusly:

text

import "gopkg.in/natefinch/lumberjack.v2"

The package name remains simply lumberjack, and the code resides at
https://github.com/natefinch/lumberjack under the v2.0 branch.

Lumberjack is intended to be one part of a logging infrastructure.
It is not an all-in-one solution, but instead is a pluggable
component at the bottom of the logging stack that simply controls the files
to which logs are written.

Lumberjack plays well with any logging package that can write to an
io.Writer, including the standard library's log package.

Lumberjack assumes that only one process is writing to the output files.
Using the same lumberjack configuration from multiple processes on the same
machine will result in improper behavior.

Example

To use lumberjack with the standard library's log package, just pass it into the SetOutput function when your application starts.

Code:

log.SetOutput(&lumberjack.Logger{
    Filename:   "/var/log/myapp/foo.log",
    MaxSize:    500, // megabytes
    MaxBackups: 3,
    MaxAge:     28, //days
    Compress:   true, // disabled by default
})

type Logger

text

/* Detailed source-code truncated for AI context efficiency. */

Logger is an io.WriteCloser that writes to the specified filename.

Logger opens or creates the logfile on first Write. If the file exists and
is less than MaxSize megabytes, lumberjack will open and append to that file.
If the file exists and its size is >= MaxSize megabytes, the file is renamed
by putting the current time in a timestamp in the name immediately before the
file's extension (or the end of the filename if there's no extension). A new
log file is then created using original filename.

Whenever a write would cause the current log file exceed MaxSize megabytes,
the current file is closed, renamed, and a new log file created with the
original name. Thus, the filename you give Logger is always the "current" log
file.

Backups use the log file name given to Logger, in the form name-timestamp.ext
where name is the filename without the extension, timestamp is the time at which
the log was rotated formatted with the time.Time format of
2006-01-02T15-04-05.000 and the extension is the original extension. For
example, if your Logger.Filename is /var/log/foo/server.log, a backup created
at 6:30pm on Nov 11 2016 would use the filename
/var/log/foo/server-2016-11-04T18-30-00.000.log

Cleaning Up Old Log Files

Whenever a new logfile gets created, old log files may be deleted. The most
recent files according to the encoded timestamp will be retained, up to a
number equal to MaxBackups (or all of them if MaxBackups is 0). Any files
with an encoded timestamp older than MaxAge days are deleted, regardless of
MaxBackups. Note that the time encoded in the timestamp is the rotation
time, which may differ from the last time that file was written to.

If MaxBackups and MaxAge are both 0, no old log files will be deleted.

func (\*Logger) Close

func (l *Logger) Close() error

Close implements io.Closer, and closes the current logfile.

func (\*Logger) Rotate

func (l *Logger) Rotate() error

Rotate causes Logger to close the existing log file and immediately create a
new one. This is a helper function for applications that want to initiate
rotations outside of the normal rotation rules, such as in response to
SIGHUP. After rotating, this initiates a cleanup of old log files according
to the normal rules.

Example

Example of how to rotate in response to SIGHUP.

Code:

l := &lumberjack.Logger{}
log.SetOutput(l)
c := make(chan os.Signal, 1)
signal.Notify(c, syscall.SIGHUP)

go func() {
    for {
        <-c
        l.Rotate()
    }
}()

func (\*Logger) Write

func (l *Logger) Write(p []byte) (n int, err error)

Write implements io.Writer. If a write would cause the log file to be larger
than MaxSize, the file is closed, renamed to include a timestamp of the
current time, and a new log file is created using the original log file name.
If the length of the write is greater than MaxSize, an error is returned.

Generated by godoc2md

Overview (`vendor/cloud.google.com/go/compute/metadata/README.md`)

Compute API

This is a utility library for communicating with Google Cloud metadata service
on Google Cloud.

Install

bash

go get cloud.google.com/go/compute/metadata

Go Version Support

See the Go Versions Supported
section in the root directory's README.

Contributing

Contributions are welcome. Please, see the CONTRIBUTING
document for details.

Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms. See
Contributor Code of Conduct
for more information.

Overview (`vendor/github.com/blevesearch/vellum/levenshtein/README.md`)

levenshtein

levenshtein automaton

This package makes it fast and simple to build a finite determinic automaton that computes the levenshtein distance from a given string.

Sample usage:

text

// build a re-usable builder
lb := NewLevenshteinAutomatonBuilder(2, false)

origTerm := "couchbasefts"
dfa := lb.BuildDfa("couchbases", 2)
ed := dfa.eval([]byte(origTerm))
if ed.distance() != 2 {
	log.Errorf("expected distance 2, actual: %d", ed.distance())
}

This implementation is inspired by blog post and is intended to be
a port of original rust implementation: https://github.com/tantivy-search/levenshtein-automata

Micro Benchmark Results against the current vellum/levenshtein is as below.

text

BenchmarkNewEditDistance1-8       	   30000	     52684 ns/op	   89985 B/op	     295 allocs/op
BenchmarkOlderEditDistance1-8     	   10000	    132931 ns/op	  588892 B/op	     363 allocs/op

BenchmarkNewEditDistance2-8       	   10000	    199127 ns/op	  377532 B/op	    1019 allocs/op
BenchmarkOlderEditDistance2-8     	    2000	    988109 ns/op	 4236609 B/op	    1898 allocs/op

Overview (`vendor/github.com/blugelabs/ice/v2/README.md`)

ice

The file is written in the reverse order that we typically access data. This helps us write in one pass since later sections of the file require file offsets of things we've already written.

Current usage:

crc-32 bytes and version are in fixed position at the end of the file
reading remainder of footer could be version specific
remainder of footer gives us:
- 3 important offsets (docValue, fields index and stored data index)
- 2 important values (number of docs and chunk factor)
field data is processed once and memoized onto the heap so that we never have to go back to disk for it
access to stored data by doc number means first navigating to the stored data index, then accessing a fixed position offset into that slice, which gives us the actual address of the data. the first bytes of that section tell us the size of data so that we know where it ends.
access to all other indexed data follows the following pattern:
- first know the field name -> convert to id
- next navigate to term dictionary for that field
  - some operations stop here and do dictionary ops
- next use dictionary to navigate to posting list for a specific term
- walk posting list
- if necessary, walk posting details as we go
- if location info is desired, consult location bitmap to see if it is there

stored fields section

for each document
- preparation phase:
  - produce a slice of metadata bytes and data bytes
  - produce these slices in field id order
  - field value is appended to the data slice
  - metadata slice is varint encoded with the following values for each field value
    - field id (uint16)
    - field value start offset in uncompressed data slice (uint64)
    - field value length (uint64)
    - compress the data slice using snappy
- file writing phase:
  - remember the start offset for this document
  - write out meta data length (varint uint64)
  - write out compressed data length (varint uint64)
  - write out the metadata bytes
  - write out the compressed data bytes

stored fields idx

for each document
- write start offset (remembered from previous section) of stored data (big endian uint64)

With this index and a known document number, we have direct access to all the stored field data.

posting details (freq/norm) section

for each posting list
- produce a slice containing multiple consecutive chunks (each chunk is varint stream)
- produce a slice remembering offsets of where each chunk starts
- preparation phase:
  - for each hit in the posting list
  - if this hit is in next chunk close out encoding of last chunk and record offset start of next
  - encode term frequency (uint64)
  - encode norm factor (float32) - similarity specific implementation
- file writing phase:
  - remember start position for this posting list details
  - write out number of chunks that follow (varint uint64)
  - write out length of each chunk (each a varint uint64)
  - write out the byte slice containing all the chunk data

If you know the doc number you're interested in, this format lets you jump to the correct chunk (docNum/chunkFactor) directly and then seek within that chunk until you find it.

posting details (location) section

for each posting list
- produce a slice containing multiple consecutive chunks (each chunk is varint stream)
- produce a slice remembering offsets of where each chunk starts
- preparation phase:
  - for each hit in the posting list
  - if this hit is in next chunk close out encoding of last chunk and record offset start of next
  - encode field (uint16)
  - encode field pos (uint64)
  - encode field start (uint64)
  - encode field end (uint64)
- file writing phase:
  - remember start position for this posting list details
  - write out number of chunks that follow (varint uint64)
  - write out length of each chunk (each a varint uint64)
  - write out the byte slice containing all the chunk data

If you know the doc number you're interested in, this format lets you jump to the correct chunk (docNum/chunkFactor) directly and then seek within that chunk until you find it.

postings list section

for each posting list
- preparation phase:
  - encode roaring bitmap posting list to bytes (so we know the length)
- file writing phase:
  - remember the start position for this posting list
  - write freq/norm details offset (remembered from previous, as varint uint64)
  - write location details offset (remembered from previous, as varint uint64)
  - write length of encoded roaring bitmap
  - write the serialized roaring bitmap data

dictionary

for each field
- preparation phase:
  - encode vellum FST with dictionary data pointing to file offset of posting list (remembered from previous)
- file writing phase:
  - remember the start position of this persistDictionary
  - write length of vellum data (varint uint64)
  - write out vellum data

fields section

for each field
- file writing phase:
  - remember start offset for each field
  - write dictionary address (remembered from previous) (varint uint64)
  - write length of field name (varint uint64)
  - write field name bytes

fields idx

for each field
- file writing phase:
  - write big endian uint64 of start offset for each field

NOTE: currently we don't know or record the length of this fields index. Instead we rely on the fact that we know it immediately precedes a footer of known size.

fields DocValue

for each field
- preparation phase:
  - produce a slice containing multiple consecutive chunks, where each chunk is composed of a meta section followed by compressed columnar field data
  - produce a slice remembering the length of each chunk
- file writing phase:
  - remember the start position of this first field DocValue offset in the footer
  - write out number of chunks that follow (varint uint64)
  - write out length of each chunk (each a varint uint64)
  - write out the byte slice containing all the chunk data

file writing phase
- write number of docs (big endian uint64)
- write stored field index location (big endian uint64)
- write field index location (big endian uint64)
- write field docValue location (big endian uint64)
- write out chunk factor (big endian uint32)
- write out version (big endian uint32)
- write out file CRC of everything preceding this (big endian uint32)

ice file format diagrams

Legend

Sections

text

|========|
|        | section
|========|

Fixed-size fields

text

|--------|        |----|        |--|        |-|
|        | uint64 |    | uint32 |  | uint16 | | uint8
|--------|        |----|        |--|        |-|

Varints

text

|~~~~~~~~|
|        | varint(up to uint64)
|~~~~~~~~|

Arbitrary-length fields

text

|--------...---|
|              | arbitrary-length field (string, vellum, roaring bitmap)
|--------...---|

Chunked data

text

[--------]
[        ]
[--------]

Overview

Footer section describes the configuration of particular ice file. The format of footer is version-dependent, so it is necessary to check V field before the parsing.

text

|==================================================|
        | Stored Fields                                    |
        |==================================================|
|-----> | Stored Fields Index                              |
|       |==================================================|   
|       | Dictionaries + Postings + DocValues              | 
|       |==================================================|
| |---> | DocValues Index                                  |
| |     |==================================================|   
| |     | Fields                                           |
| |     |==================================================|
| | |-> | Fields Index                                     |
| | |   |========|========|========|========|====|====|====|
| | |   |     D# |     SF |      F |    FDV | CF |  V | CC | (Footer)
| | |   |========|====|===|====|===|====|===|====|====|====|
| | |                 |        |        |
|-+-+-----------------|        |        |
  | |--------------------------|        |
  |-------------------------------------|

 D#. Number of Docs.
 SF. Stored Fields Index Offset.
  F. Field Index Offset.
FDV. Field DocValue Offset.
 CF. Chunk Factor.
  V. Version.
 CC. CRC32.

Stored Fields

Stored Fields Index is D# consecutive 64-bit unsigned integers - offsets, where relevant Stored Fields Data records are located.

text

0                                [SF]                   [SF + D# * 8]
| Stored Fields                  | Stored Fields Index              |
|================================|==================================|
|                                |                                  |
|       |--------------------|   ||--------|--------|. . .|--------||
|   |-> | Stored Fields Data |   ||      0 |      1 |     | D# - 1 ||
|   |   |--------------------|   ||--------|----|---|. . .|--------||
|   |                            |              |                   |
|===|============================|==============|===================|
    |                                           |
    |-------------------------------------------|

Stored Fields Data is an arbitrary size record, which consists of metadata and Snappy-compressed data.

text

Stored Fields Data
|~~~~~~~~|~~~~~~~~|~~~~~~~~...~~~~~~~~|~~~~~~~~...~~~~~~~~|
|    MDS |    CDS |                MD |                CD |
|~~~~~~~~|~~~~~~~~|~~~~~~~~...~~~~~~~~|~~~~~~~~...~~~~~~~~|

MDS. Metadata size.
CDS. Compressed data size.
MD. Metadata.
CD. Snappy-compressed data.

Fields

text

(...)                                              [F]                       [F + F#]
| Fields                                           | Fields Index.                  |
|==================================================|================================|
|                                                  |                                |
|   |~~~~~~~~|~~~~~~~~|---...---|~~~~~~~~|~~~~~~~~|||--------|--------|...|--------||
||->|   Dict | Length |    Name | # Docs |Tot Freq|||      0 |      1 |   | F# - 1 ||
||  |~~~~~~~~|~~~~~~~~|---...---|~~~~~~~~|~~~~~~~~|||--------|----|---|...|--------||
||                                                 |              |                 |
||=================================================|==============|=================|
 |                                                                |
 |----------------------------------------------------------------|

Dictionaries + Postings

text

|================================================================|- Dictionaries + 
|                                                                |   Postings +
|                                                                |    DocValues
|    Freq/Norm (chunked)                                         |
|    [~~~~~~|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~]                      |
| |->[ Freq | Norm (float32 under varint) ]                      |
| |  [~~~~~~|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~]                      |
| |                                                              |
| |------------------------------------------------------------| |
|    Location Details (chunked)                                | |
|    [~~~~~~|~~~~~|~~~~~~~|~~~~~]                              | |
| |->[ Size | Pos | Start | End ]                              | |
| |  [~~~~~~|~~~~~|~~~~~~~|~~~~~]                              | |
| |                                                            | |
| |----------------------|                                     | |
|          Postings List |                                     | |
|         |~~~~~~~~|~~~~~|~~|~~~~~~~~|-----------...--|        | |
|      |->|    F/N |     LD | Length | ROARING BITMAP |        | |
|      |  |~~~~~|~~|~~~~~~~~|~~~~~~~~|-----------...--|        | |
|      |        |----------------------------------------------| |
|      |--------------------------------------|                  |
|          Dictionary                         |                  |
|         |~~~~~~~~|--------------------------|-...-|            |
|      |->| Length | VELLUM DATA : (TERM -> OFFSET) |            |
|      |  |~~~~~~~~|----------------------------...-|            |
|      |                                                         |
|======|=========================================================|- DocValues Index
|      |                                                         |
|======|=========================================================|- Fields
|      |                                                         |
| |~~~~|~~~|~~~~~~~~|---...---|                                  |
| |   Dict | Length |    Name |                                  |
| |~~~~~~~~|~~~~~~~~|---...---|                                  |
|                                                                |
|================================================================|

DocValues

DocValues Index is F# pairs of varints, one pair per field. Each pair of varints indicates start and end point of DocValues slice.

text

|================================================================|
|     |------...--|                                              |
|  |->| DocValues |<-|                                           |
|  |  |------...--|  |                                           |
|==|=================|===========================================|- DocValues Index
||~|~~~~~~~~~|~~~~~~~|~~|           |~~~~~~~~~~~~~~|~~~~~~~~~~~~||
|| DV1 START | DV1 STOP | . . . . . | DV(F#) START | DV(F#) END ||
||~~~~~~~~~~~|~~~~~~~~~~|           |~~~~~~~~~~~~~~|~~~~~~~~~~~~||
|================================================================|

DocValues is chunked Snappy-compressed values for each document and field.

text

[~~~~~~~~~~~~~~~|~~~~~~|~~~~~~~~~|-...-|~~~~~~|~~~~~~~~~|--------------------...-]
[ Doc# in Chunk | Doc1 | Offset1 | ... | DocN | OffsetN | SNAPPY COMPRESSED DATA ]
[~~~~~~~~~~~~~~~|~~~~~~|~~~~~~~~~|-...-|~~~~~~|~~~~~~~~~|--------------------...-]

Last 16 bytes are description of chunks.

text

|~~~~~~~~~~~~...~|----------------|----------------|
|   Chunk Sizes  | Chunk Size Arr |         Chunk# |
|~~~~~~~~~~~~...~|----------------|----------------|

Overview (`vendor/github.com/cespare/xxhash/v2/README.md`)

xxhash

xxhash is a Go implementation of the 64-bit xxHash algorithm, XXH64. This is a
high-quality hashing algorithm that is much faster than anything in the Go
standard library.

This package provides a straightforward API:

text

func Sum64(b []byte) uint64
func Sum64String(s string) uint64
type Digest struct{ ... }
    func New() *Digest

The Digest type implements hash.Hash64. Its key methods are:

text

func (*Digest) Write([]byte) (int, error)
func (*Digest) WriteString(string) (int, error)
func (*Digest) Sum64() uint64

The package is written with optimized pure Go and also contains even faster
assembly implementations for amd64 and arm64. If desired, the purego build tag
opts into using the Go code even on those architectures.

Compatibility

This package is in a module and the latest code is in version 2 of the module.
You need a version of Go with at least "minimal module compatibility" to use
github.com/cespare/xxhash/v2:

1.9.7+ for Go 1.9
1.10.3+ for Go 1.10
Go 1.11 or later

I recommend using the latest release of Go.

Benchmarks

Here are some quick benchmarks comparing the pure-Go and assembly
implementations of Sum64.

input size	purego	asm
4 B	1.3 GB/s	1.2 GB/s
16 B	2.9 GB/s	3.5 GB/s
100 B	6.9 GB/s	8.1 GB/s
4 KB	11.7 GB/s	16.7 GB/s
10 MB	12.0 GB/s	17.3 GB/s

These numbers were generated on Ubuntu 20.04 with an Intel Xeon Platinum 8252C
CPU using the following commands under Go 1.19.2:

text

benchstat <(go test -tags purego -benchtime 500ms -count 15 -bench 'Sum64$')
benchstat <(go test -benchtime 500ms -count 15 -bench 'Sum64$')

Projects using this package

Overview (`vendor/github.com/dlclark/regexp2/v2/README.md`)

regexp2 - full featured regular expressions for Go

Regexp2 is a feature-rich RegExp engine for Go. It doesn't have constant time guarantees like the built-in regexp package, but it allows backtracking and is compatible with Perl5 and .NET. You'll likely be better off with the RE2 engine from the regexp package and should only use this if you need to write very complex patterns or require compatibility with .NET.

Basis of the engine

The engine is ported from the .NET framework's System.Text.RegularExpressions.Regex engine. That engine was open sourced in 2015 under the MIT license. There are some fundamental differences between .NET strings and Go strings that required a bit of borrowing from the Go framework regex engine as well. I cleaned up a couple of the dirtier bits during the port (regexcharclass.cs was terrible), but the parse tree, code emmitted, and therefore patterns matched should be identical.

New Code Generation

For extra performance use regexp2 with regexp2cg. It is a code generation utility for regexp2 and you can likely improve your regexp runtime performance by 3-10x in hot code paths. As always you should benchmark your specifics to confirm the results. Give it a try!

Installing

This is a go-gettable library, so install is easy:

text

go get github.com/dlclark/regexp2/v2@latest

Changes in v2

Version 2 includes changes that may affect compatibility with existing v1 users:

The module path is now github.com/dlclark/regexp2/v2, so imports need to use the /v2 suffix.
The minimum supported Go version is now Go 1.25.
Changes to support https://github.com/dlclark/regexp2cg are merged in to support generated regex engines.
Regexp.Split is now available for splitting strings with regexp matches.
The new compat sub-package provides a regexp compatibility adapter with the same Find* and Match* method signatures as regexp.Regexp, plus a compat.Matcher interface that is implemented by both *regexp.Regexp and the adapter.
The parser, optimizer, and runner internals have changed significantly to support generated regexes and additional matching optimizations.
Compile and MustCompile now use variadic compile options for regex behavior and memory/performance tuning. See Compile options for more details.
Moved regexp2.Debug and regexp2.Compile to new regexp2.OptionDebug() and regexp2.OptionIsCodeGen() compile options.
Some types and constants in the syntax package have been exported or changed to support code generation.
Conceptually changed the goal of the regexp2.ECMAScript option to be closer to the ECMAScript standard rather than C#'s ECMAScript behavior.
Renamed the fields Capture.Index and Capture.Length to Capture.RuneIndex and Capture.RuneLength to be more clear that we're dealing with rune offsets.
Added Capture.ByteRange() to return the byte offset index and length of the captured text. This requires some additional processing to be done behind the scenes the first time it's called for a given capture to convert the native rune offsets to byte offsets.

Usage

Usage is similar to the Go regexp package. Just like in regexp, you start by converting a regex into a state machine via the Compile or MustCompile methods. They ultimately do the same thing, but MustCompile will panic if the regex is invalid. You can then use the provided Regexp struct to find matches repeatedly. A Regexp struct is safe to use across goroutines.

re := regexp2.MustCompile(`Your pattern`)
if isMatch, _ := re.MatchString(`Something to match`); isMatch {
    //do something
}

The *Match* methods can return a timeout error if you set the re.MatchTimeout field, or ErrBacktrackingStackLimit if a match exceeds its configured backtracking stack size. Any other error is a bug in the regexp2 package. If you need more details about capture groups in a match then use the FindStringMatch method, like so:

if m, _ := re.FindStringMatch(`Something to match`); m != nil {
    // the whole match is always group 0
    fmt.Printf("Group 0: %v\n", m.String())

    // you can get all the groups too
    gps := m.Groups()

    // a group can be captured multiple times, so each cap is separately addressable
    fmt.Printf("Group 1, first capture", gps[1].Captures[0].String())
    fmt.Printf("Group 1, second capture", gps[1].Captures[1].String())
}

Group 0 is embedded in the Match. Group 0 is an automatically-assigned group that encompasses the whole pattern. This means that m.String() is the same as m.Group.String() and m.Groups()[0].String()

The last capture is embedded in each group, so g.String() will return the same thing as g.Capture.String() and g.Captures[len(g.Captures)-1].String().

If you want to find multiple matches from a single input string you should use the FindNextMatch method. For example, to implement a function similar to regexp.FindAllString:

func regexp2FindAllString(re *regexp2.Regexp, s string) []string {
	var matches []string
	m, _ := re.FindStringMatch(s)
	for m != nil {
		matches = append(matches, m.String())
		m, _ = re.FindNextMatch(m)
	}
	return matches
}

FindNextMatch is optmized so that it re-uses the underlying string/rune slice.

The internals of regexp2 always operate on []rune so RuneIndex and RuneLength data in a Match always reference a position in runes rather than bytes (even if the input was given as a string). ByteRange() provides UTF-8 byte offsets, matching the original string input for string APIs. It's advisable to use the provided String() methods when you do not need explicit offsets. ByteRange() lazily caches byte offsets on the shared match text, so the first call on captures from the same match is not safe to run concurrently with other ByteRange() calls on that match.

Unicode character classes

regexp2 supports Unicode character classes with \p{...} and negated classes with \P{...}. Outside ECMAScript Unicode mode, it also accepts the RE2/PCRE-style one-letter form, such as \pL.

The class name may be a Go Unicode category, category alias, script, or property exposed by the Go standard library Unicode tables. For example:

letter := regexp2.MustCompile(`\p{L}+`)
katakana := regexp2.MustCompile(`\p{Katakana}+`)
notEmoji := regexp2.MustCompile(`\P{Emoji}+`)

regexp2 also supports Unicode property selection syntax in the form \p{property=value}. Property and value aliases are matched loosely: case, hyphens, and underscores are ignored. For example, \p{GCB=RI}, \p{grapheme_cluster_break=regional_indicator}, and \p{grapheme-cluster-break=regional-indicator} all refer to the same class.

Valid property names and aliases come from Unicode 17.0.0 PropertyAliases.txt. Valid property values and aliases come from Unicode 17.0.0 PropertyValueAliases.txt. The generated tables use Unicode 17.0.0 data from DerivedCoreProperties.txt, emoji/emoji-data.txt, auxiliary/GraphemeBreakProperty.txt, auxiliary/WordBreakProperty.txt, and auxiliary/SentenceBreakProperty.txt for the package-local properties whose data changes more frequently than the Go standard library tables.

Additional Perl and PCRE syntax

The default mode supports the following syntax:

\Q...\E quotes every character between \Q and \E. If \E is omitted, quoting continues to the end of the pattern. This also works inside character classes.
\R matches one Unicode newline sequence: CRLF as a single sequence, or LF, VT, FF, CR, NEL, line separator, or paragraph separator.
\X atomically matches one Unicode 17.0.0 extended grapheme cluster. This includes combining sequences, Hangul syllables, regional-indicator pairs, emoji ZWJ sequences, and Indic conjuncts.
A + after a quantifier makes it possessive: *+, ++, ?+, and {m,n}+. For example, a*+ is equivalent to (?>a*) and will not give characters back when the remainder of the pattern fails.

RE2 mode also supports \Q...\E and possessive quantifiers, but keeps \R and \X as literal R and X identity escapes. ECMAScript mode does not enable any of this syntax: \Q, \E, \R, and \X retain ECMAScript identity-escape behavior, and possessive quantifiers remain invalid.

`regexp` compatibility adapter

The github.com/dlclark/regexp2/v2/compat package provides an adapter for callers that want the same Find* and Match* method signatures as the standard library's regexp.Regexp, while still using the regexp2 engine.

import (
	"github.com/dlclark/regexp2/v2"
	"github.com/dlclark/regexp2/v2/compat"
)

re := compat.MustCompile(`Your pattern`, regexp2.RE2)
if re.MatchString(`Something to match`) {
	// do something
}

matches := re.FindAllString(`abc axbc`, -1)
_ = matches

You can also wrap an existing compiled regexp:

base := regexp2.MustCompile(`Your pattern`)
re := compat.Wrap(base)

The adapter includes the standard-library matching surface: Match, MatchString, MatchReader, and all Find(All)?(String)?(Submatch)?(Index)? methods. Index-returning methods use UTF-8 byte offsets like regexp, not regexp2's rune offsets.

The package also defines compat.Matcher, a common interface implemented by both *regexp.Regexp and *compat.Regexp. Use it when code should accept either the standard library engine or a regexp2-backed adapter:

func findWords(re compat.Matcher, input string) []string {
	return re.FindAllString(input, -1)
}

Because those standard-library method signatures do not return errors, the adapter panics if the wrapped regexp2 matcher returns an error such as a match timeout or ErrBacktrackingStackLimit. Use the main regexp2 APIs directly when you need to handle match errors directly.

Compile options

Compile and MustCompile take variadic compile options. Most users can omit them and get default regex behavior plus bounded shared pools for rune buffers and replacement output buffers, plus per-regexp caches for parsed replacement patterns and ASCII character class bitmaps.

Regex option constants can be passed directly, individually or as a bitmask:

re := regexp2.MustCompile(`Your pattern`, regexp2.IgnoreCase, regexp2.Singleline)
re = regexp2.MustCompile(`Your pattern`, regexp2.IgnoreCase|regexp2.Singleline)

Performance tuning options override the default cache settings:

re := regexp2.MustCompile(`Your pattern`,
	regexp2.IgnoreCase,
	regexp2.OptionMaxBacktrackingStackSize(200000),
	regexp2.OptionMaxCachedRuneBufferLength(64*1024),
	regexp2.OptionMaxCachedReplacerDataEntries(8),
)

Compile-only options configure behavior that is not settable from the pattern:

re := regexp2.MustCompile(`(?<first>This) (is)`, regexp2.OptionMaintainCaptureOrder())

The defaults are intentionally bounded:

Option	Default	Used by	Working-set growth	Tradeoffs
`OptionMaintainCaptureOrder()`	false	Parser capture-slot assignment for mixed named and unnamed captures.	None at match time. This changes compile-time capture numbering only.	Keeps named and unnamed captures in pattern order instead of appending named captures after unnamed captures. This can change numeric backreference meaning, so it is caller-controlled rather than an inline regex option.
`OptionDebug()`	false	Compile dumps and runner tracing.	Debug output volume only.	Useful for diagnostics, but it can produce noisy output and slower traced matching.
`OptionIsCodeGen()`	false	Compile-time find-optimization analysis for `regexp2cg`.	Per compiled regexp, during `Compile` or `MustCompile`.	Enables more expensive analysis intended for generated engines. Do not use it for normal interpreter execution; the interpreter defaults intentionally avoid this extra compile-time cost.
`OptionMaxBacktrackingStackSize(n)`	100,000	The interpreter's per-match backtracking stack.	Per pooled runner. The initial allocation and subsequent growth are capped at the configured number of integer slots; the runner pool may retain stacks at their high-water size for reuse.	Lowering this bounds backtracking memory more tightly but may reject complex matches sooner with `ErrBacktrackingStackLimit`. Raising it permits deeper backtracking and increases possible memory use. A negative value disables the limit.
`OptionMaxCachedRuneBufferLength(n)`	256K runes	String APIs that run through pooled runners, such as `MatchString` and replacement-pattern `Replace`, when converting input strings to the engine's internal `[]rune` representation.	Process-wide shared `sync.Pool` retention by size class. This does not grow per compiled regexp or per input string; the practical working set follows recent and concurrent use across all regexps and can be dropped by GC.	Raising this lets calls use larger pooled rune buffers and can reduce allocations for repeated matches against large strings. Lowering it prevents larger buffers from being borrowed or returned, so large inputs allocate directly.
`OptionMaxCachedReplaceBufferLength(n)`	256 KB	Replacement-pattern `Replace` calls that build output through a shared byte buffer.	Process-wide shared `sync.Pool` retention by size class after replacement-pattern `Replace` runs. It does not grow from evaluator-based `ReplaceFunc` output and is shared across compiled regexps.	Raising this lets larger replacement outputs use pooled buffers and can reduce allocations. Lowering it prevents larger output buffers from being retained, so large replacements allocate directly.
`OptionMaxCachedReplacerDataEntries(n)`	`16`	`Replace` with replacement pattern strings, after the replacement pattern is parsed into reusable replacement data.	Per compiled regexp. The cache grows as distinct cacheable replacement strings are used with `Replace`, up to this entry count.	Raising this helps when a single compiled regexp is used with many recurring replacement patterns. It increases per-regexp cache memory and lock-protected cache bookkeeping. Setting it to `0` disables this cache.
`OptionMaxCachedReplacerDataBytes(n)`	4 KB	The parsed replacement-pattern cache. Replacement strings longer than this are parsed for the call but not retained.	Per compiled regexp, combined with `OptionMaxCachedReplacerDataEntries`. Only replacement strings whose source text is at or below this size can add parsed data to the cache.	Raising this helps if large replacement patterns are reused. It can retain more memory per cached replacement. Lowering it avoids keeping unusual large replacement patterns around.
`OptionDisableCharClassASCIIBitmap()`	false	Compile-time preparation of character classes and first-character prefix sets. By default, character classes with ASCII membership get a small bitmap used by `CharIn`.	Per compiled regexp, during `Compile` or `MustCompile`. Each eligible character class can hold one small bitmap; this does not scale with match concurrency or input size.	Leaving this false speeds up ASCII-heavy character class checks at the cost of a small amount of per-char-class memory and compile-time work. Setting to true can reduce memory for large numbers of compiled char classes in regexps, but ASCII character class matching may be slower.

For OptionMaxBacktrackingStackSize, set n to a negative value to allow unbounded stack growth. Setting it to 0 permits no backtracking stack entries, so most interpreted matches will return ErrBacktrackingStackLimit.

For pooled buffer cache options, set n to 0 to disable pooling, or -1 to allow all built-in size classes. The rune buffer classes are 1K, 4K, 16K, 64K, and 256K runes. The replacement byte buffer classes are 4 KB, 16 KB, 64 KB, 256 KB, and 1 MB. By default the 1 MB pool is unused. For replacement data byte-size cache options, -1 means unbounded. For entry-count cache options, set n to 0 to disable the cache.

Compare `regexp` and `regexp2`

Category	regexp	regexp2
Catastrophic backtracking possible	no, constant execution time guarantees	yes; backtracking stack growth is bounded by default, and `re.MatchTimeout` can also bound match duration
Python-style capture groups `(?P<name>re)`	yes	no (yes in RE2 compat mode)
.NET-style capture groups `(?<name>re)` or `(?'name're)`	yes	yes
comments `(?#comment)`	no	yes
branch numbering reset `(?\|a\|b)`	no	no
atomic group `(?>re)`	no	yes
possessive quantifiers `*+`, `++`, `?+`, `{m,n}+`	no	yes
literal quoting `\Q...\E`	yes	yes
Unicode newline sequence `\R`	no	yes (default mode only)
extended grapheme cluster `\X`	no	yes (default mode only)
positive lookahead `(?=re)`	no	yes
negative lookahead `(?!re)`	no	yes
positive lookbehind `(?<=re)`	no	yes
negative lookbehind `(?<!re)`	no	yes
back reference `\1`	no	yes
named back reference `\k'name'`	no	yes
Python-style named back reference `(?P=name)`	no	no (yes in RE2 compat mode)
named ascii character class `[[:foo:]]`	yes	no (yes in RE2 compat mode)
conditionals `(?(expr)yes\|no)`	no	yes

RE2 compatibility mode

The default behavior of regexp2 is to match the .NET regexp engine, however the RE2 option is provided to change the parsing to increase compatibility with RE2. Using the RE2 option when compiling a regexp will not take away any features, but will change the following behaviors:

add support for named ascii character classes (e.g. [[:foo:]])
add support for python-style capture groups (e.g. (?P<name>re))
add support for python-style named backreferences (e.g. (?P=name))
change singleline behavior for $ to only match end of string (like RE2) (see #24)
change the character classes \d \s and \w to match the same characters as RE2. NOTE: if you also use the ECMAScript option then this will change the \s character class to match ECMAScript instead of RE2. ECMAScript allows more whitespace characters in \s than RE2 (but still fewer than the the default behavior).
allow character escape sequences to have defaults. For example, by default \_ isn't a known character escape and will fail to compile, but in RE2 mode it will match the literal character _
support RE2-style literal quoting with \Q...\E
support possessive quantifiers (*+, ++, ?+, and {m,n}+) as a regexp2 extension

re := regexp2.MustCompile(`Your RE2-compatible pattern`, regexp2.RE2)
if isMatch, _ := re.MatchString(`Something to match`); isMatch {
    //do something
}

This feature is a work in progress and I'm open to ideas for more things to put here (maybe more relaxed character escaping rules?).

Catastrophic Backtracking and Timeouts

regexp2 supports features that can lead to catastrophic backtracking.
Each compiled regexp limits its per-match backtracking stack to 100,000
slots by default. If a match would exceed that limit, it stops and returns
ErrBacktrackingStackLimit. Callers can identify it with
errors.Is(err, regexp2.ErrBacktrackingStackLimit). The limit can be changed at
compile time; a negative value restores the previous unbounded behavior:

re := regexp2.MustCompile(pattern, regexp2.OptionMaxBacktrackingStackSize(200000))
// regexp2.OptionMaxBacktrackingStackSize(-1) disables the limit.

This limit bounds the interpreter's backtracking stack, not total match time or
all memory used by a match. Literal empty expressions repeated any number of
times are optimized away and do not consume backtracking stack space.

Regexp.MatchTimeout can be set to limit the impact of such behavior; the
match will fail with an error after approximately MatchTimeout. No timeout
checks are done by default.

Timeout checking is not free. The current timeout checking implementation starts
a background worker that updates a clock value approximately once every 100
milliseconds. The matching code compares this value against the precomputed
deadline for the match. The performance impact is as follows.

A match with a timeout runs almost as fast as a match without a timeout.
If any live matches have a timeout, there will be a background CPU load
(~0.15% currently on a modern machine). This load will remain constant
regardless of the number of matches done including matches done in parallel.
If no live matches are using a timeout, the background load will remain
until the longest deadline (match timeout + the time when the match started)
is reached. E.g., if you set a timeout of one minute the load will persist
for approximately a minute even if the match finishes quickly.

See PR #58 for more details and
alternatives considered.

Goroutine leak error

If you're using a library during unit tests (e.g. https://github.com/uber-go/goleak) that validates all goroutines are exited then you'll likely get an error if you or any of your dependencies use regex's with a MatchTimeout.
To remedy the problem you'll need to tell the unit test to wait until the backgroup timeout goroutine is exited.

func TestSomething(t *testing.T) {
    defer goleak.VerifyNone(t)
    defer regexp2.StopTimeoutClock()

    // ... test
}

//or

func TestMain(m *testing.M) {
    // setup
    // ...

    // run 
    m.Run()

    //tear down
    regexp2.StopTimeoutClock()
    goleak.VerifyNone(t)
}

This will add ~100ms runtime to each test (or TestMain). If that's too much time you can set the clock cycle rate of the timeout goroutine in an init function in a test file. regexp2.SetTimeoutCheckPeriod isn't threadsafe so it must be setup before starting any regex's with Timeouts.

func init() {
	//speed up testing by making the timeout clock 1ms
	regexp2.SetTimeoutCheckPeriod(time.Millisecond)
}

ECMAScript compatibility mode

In this mode the engine attempts to match the regex engine described in the ECMAScript specification as closely as reasonably possible within regexp2's API and implementation.

This flag should not be treated as compatibility with C#'s RegexOptions.ECMAScript. regexp2's ECMAScript behavior prioritizes ECMAScript specification behavior over matching the C# regex engine's interpretation of that option.

Additionally a Unicode mode is provided which allows parsing of \u{CodePoint} syntax only when both ECMAScript and Unicode are provided.

Perl/PCRE extensions \Q...\E, \R, \X, and possessive quantifiers are intentionally not enabled in this mode. The letter escapes retain the engine's existing ECMAScript identity-escape behavior.

Potential bugs

I've run a battery of tests against regexp2 from various sources and found the debug output matches the .NET engine, but .NET and Go handle strings very differently. I've attempted to handle these differences, but most of my testing deals with basic ASCII with a little bit of multi-byte Unicode. There's a chance that there are bugs in the string handling related to character sets with supplementary Unicode chars. Right-to-Left support is coded, but not well tested either.

Find a bug?

I'm open to new issues and pull requests with tests if you find something odd!

Overview (`vendor/github.com/gofrs/uuid/v5/README.md`)

UUID

Package uuid provides a pure Go implementation of Universally Unique Identifiers
(UUID) variant as defined in RFC-9562. This package supports both the creation
and parsing of UUIDs in different formats.

This package supports the following UUID versions:

Version 1, based on timestamp and MAC address
Version 3, based on MD5 hashing of a named value
Version 4, based on random numbers
Version 5, based on SHA-1 hashing of a named value
Version 6, a k-sortable id based on timestamp, and field-compatible with v1
Version 7, a k-sortable id based on timestamp
Version 8, for custom UUID implementations

Project History

This project was originally forked from the
github.com/satori/go.uuid repository after
it appeared to be no longer maintained, while exhibiting critical
flaws. We have decided to take
over this project to ensure it receives regular maintenance for the benefit of
the larger Go community.

We'd like to thank Maxim Bublis for his hard work on the original iteration of
the package.

License

This source code of this package is released under the MIT License. Please see
the LICENSE for the full
content of the license.

Recommended Package Version

We recommend using v2.0.0+ of this package, as versions prior to 2.0.0 were
created before our fork of the original package and have some known
deficiencies.

Requirements

This package requires Go 1.25 or later

Usage

Here is a quick overview of how to use this package. For more detailed
documentation, please see the GoDoc Page.

package main

import (
	"log"

	"github.com/gofrs/uuid/v5"
)

// Create a Version 4 UUID, panicking on error.
// Use this form to initialize package-level variables.
var u1 = uuid.Must(uuid.NewV4())

func main() {
	// Create a Version 4 UUID.
	u2, err := uuid.NewV4()
	if err != nil {
		log.Fatalf("failed to generate UUID: %v", err)
	}
	log.Printf("generated Version 4 UUID %v", u2)

	// Parse a UUID from a string.
	s := "6ba7b810-9dad-11d1-80b4-00c04fd430c8"
	u3, err := uuid.FromString(s)
	if err != nil {
		log.Fatalf("failed to parse UUID %q: %v", s, err)
	}
	log.Printf("successfully parsed UUID %v", u3)
}

References

RFC-9562 (replaces RFC-4122)
DCE 1.1: Authentication and Security Services

Overview (`vendor/github.com/golang-jwt/jwt/v5/README.md`)

jwt-go

A go (or 'golang' for search engine friendliness)
implementation of JSON Web
Tokens.

Starting with v4.0.0
this project adds Go module support, but maintains backward compatibility with
older v3.x.y tags and upstream github.com/dgrijalva/jwt-go. See the
MIGRATION_GUIDE.md for more information. Version
v5.0.0 introduces major improvements to the validation of tokens, but is not
entirely backward compatible.

After the original author of the library suggested migrating the maintenance
of jwt-go, a dedicated team of open source maintainers decided to clone the
existing library into this repository. See
dgrijalva/jwt-go#462 for a
detailed discussion on this topic.

SECURITY NOTICE: Some older versions of Go have a security issue in the
crypto/elliptic. The recommendation is to upgrade to at least 1.15 See issue
dgrijalva/jwt-go#216 for more
detail.

SECURITY NOTICE: It's important that you validate the alg presented is
what you
expect.
This library attempts to make it easy to do the right thing by requiring key
types to match the expected alg, but you should take the extra step to verify it in
your usage. See the examples provided.

Supported Go versions

Our support of Go versions is aligned with Go's version release
policy. So we will support a major
version of Go until there are two newer major releases. We no longer support
building jwt-go with unsupported Go versions, as these contain security
vulnerabilities that will not be fixed.

What the heck is a JWT?

JWT.io has a great introduction to JSON Web
Tokens.

In short, it's a signed JSON object that does something useful (for example,
authentication). It's commonly used for Bearer tokens in Oauth 2. A token is
made of three parts, separated by .'s. The first two parts are JSON objects,
that have been base64url
encoded. The last part is the signature, encoded the same way.

The first part is called the header. It contains the necessary information for
verifying the last part, the signature. For example, which encryption method
was used for signing and what key was used.

The part in the middle is the interesting bit. It's called the Claims and
contains the actual stuff you care about. Refer to RFC
7519 for information about
reserved keys and the proper way to add your own.

What's in the box?

This library supports the parsing and verification as well as the generation and
signing of JWTs. Current supported signing algorithms are HMAC SHA, RSA,
RSA-PSS, and ECDSA, though hooks are present for adding your own.

Installation Guidelines

To install the jwt package, you first need to have
Go installed, then you can use the command
below to add jwt-go as a dependency in your Go program.

go get -u github.com/golang-jwt/jwt/v5

Import it in your code:

import "github.com/golang-jwt/jwt/v5"

Usage

A detailed usage guide, including how to sign and verify tokens can be found on
our documentation website.

Examples

See the project documentation
for examples of usage:

Compliance

This library was last reviewed to comply with RFC
7519 dated May 2015 with a few
notable differences:

In order to protect against accidental use of Unsecured
JWTs, tokens using
alg=none will only be accepted if the constant
jwt.UnsafeAllowNoneSignatureType is provided as the key.

Project Status & Versioning

This library is considered production ready. Feedback and feature requests are
appreciated. The API should be considered stable. There should be very few
backward-incompatible changes outside of major version updates (and only with
good reason).

This project uses Semantic Versioning 2.0.0. Accepted pull
requests will land on main. Periodically, versions will be tagged from
main. You can find all the releases on the project releases
page.

BREAKING CHANGES: A full list of breaking changes is available in
VERSION_HISTORY.md. See MIGRATION_GUIDE.md for more information on updating
your code.

Extensions

This library publishes all the necessary components for adding your own signing
methods or key functions. Simply implement the SigningMethod interface and
register a factory method using RegisterSigningMethod or provide a
jwt.Keyfunc.

A common use case would be integrating with different 3rd party signature
providers, like key management services from various cloud providers or Hardware
Security Modules (HSMs) or to implement additional standards.

Extension	Purpose	Repo
GCP	Integrates with multiple Google Cloud Platform signing tools (AppEngine, IAM API, Cloud KMS)	https://github.com/someone1/gcp-jwt-go
AWS	Integrates with AWS Key Management Service, KMS	https://github.com/matelang/jwt-go-aws-kms
JWKS	Provides support for JWKS (RFC 7517) as a `jwt.Keyfunc`	https://github.com/MicahParks/keyfunc
TPM	Integrates with Trusted Platform Module (TPM)	https://github.com/salrashid123/golang-jwt-tpm

Disclaimer: Unless otherwise specified, these integrations are maintained by
third parties and should not be considered as a primary offer by any of the
mentioned cloud providers

Go package documentation can be found on
pkg.go.dev. Additional
documentation can be found on our project
page.

The command line utility included in this project (cmd/jwt) provides a
straightforward example of token creation and parsing as well as a useful tool
for debugging your own integration. You'll also find several implementation
examples in the documentation.

golang-jwt incorporates a modified version
of the JWT logo, which is distributed under the terms of the MIT
License.

Overview (`vendor/github.com/heroiclabs/sql-migrate/sqlparse/README.md`)

SQL migration parser

Based on the goose migration parser.

License

This library is distributed under the MIT license.

Overview (`vendor/github.com/jackc/pgx/v5/README.md`)

pgx - PostgreSQL Driver and Toolkit

pgx is a pure Go driver and toolkit for PostgreSQL.

The pgx driver is a low-level, high performance interface that exposes PostgreSQL-specific features such as LISTEN /
NOTIFY and COPY. It also includes an adapter for the standard database/sql interface.

The toolkit component is a related set of packages that implement PostgreSQL functionality such as parsing the wire protocol
and type mapping between PostgreSQL and Go. These underlying packages can be used to implement alternative drivers,
proxies, load balancers, logical replication clients, etc.

Quick Start

Installation

bash

go get github.com/jackc/pgx/v5

Example Usage

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/jackc/pgx/v5"
)

func main() {
	// urlExample := "postgres://username:password@localhost:5432/database_name"
	conn, err := pgx.Connect(context.Background(), os.Getenv("DATABASE_URL"))
	if err != nil {
		fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
		os.Exit(1)
	}
	defer conn.Close(context.Background())

	var name string
	var weight int64
	err = conn.QueryRow(context.Background(), "select name, weight from widgets where id=$1", 42).Scan(&name, &weight)
	if err != nil {
		fmt.Fprintf(os.Stderr, "QueryRow failed: %v\n", err)
		os.Exit(1)
	}

	fmt.Println(name, weight)
}

Connection Configuration

pgx.Connect and pgxpool.New accept PostgreSQL connection URLs (such as postgres://user:pass@host:5432/db?sslmode=verify-full) as well as key=value strings. See pgconn.ParseConfig and the PostgreSQL connection string documentation for supported options and environment variables.

For a step-by-step walkthrough, see the getting started guide.

Documentation

Package documentation and API reference are available on pkg.go.dev:

pgx — base PostgreSQL driver
pgxpool — concurrency-safe connection pool
stdlib — database/sql compatibility adapter

Features

Support for approximately 70 different PostgreSQL types
Automatic statement preparation and caching
Batch queries
Single-round trip query mode
Full TLS connection control
Binary format support for custom types (allows for much quicker encoding/decoding)
COPY protocol support for faster bulk data loads
Tracing and logging support
Connection pool with after-connect hook for arbitrary connection setup
LISTEN / NOTIFY
Conversion of PostgreSQL arrays to Go slice mappings for integers, floats, and strings
hstore support
json and jsonb support
Maps inet and cidr PostgreSQL types to netip.Addr and netip.Prefix
Large object support
NULL mapping to pointer to pointer
Supports database/sql.Scanner and database/sql/driver.Valuer interfaces for custom types
Notice response handling
Simulated nested transactions with savepoints

Choosing Between the pgx and database/sql Interfaces

The pgx interface is faster. Many PostgreSQL specific features such as LISTEN / NOTIFY and COPY are not available
through the database/sql interface.

The pgx interface is recommended when:

The application only targets PostgreSQL.
No other libraries that require database/sql are in use.

It is also possible to use the database/sql interface and convert a connection to the lower-level pgx interface as needed.

Development and Testing

Each checkout has its own PostgreSQL 14-18 clusters and CockroachDB node, so the whole test matrix
is available locally on macOS, on Linux, or in the included devcontainer. Only PostgreSQL 18 stays
running by default; other targets start and stop around their tests:

scripts/setup-host # native macOS (Homebrew required) or Ubuntu: host packages and mise
export PATH="$HOME/.local/bin:$PATH" # if mise was just installed
mise trust
mise install        # project tools
mise run dev:init   # checkout ports and certificates
mise run dev        # start PostgreSQL 18 and the on-demand database supervisor
./test.sh           # the suite against PostgreSQL 18
./test.sh all       # every target, starting and stopping each server as needed

Host setup installs PostgreSQL 14-18 and Ruby build dependencies. It does not install project
tools or initialize the checkout; those remain separate steps above. The devcontainer already
provides the host prerequisites.

See DEVELOPMENT.md for the full setup, and
CONTRIBUTING.md for how to contribute — including how to run the tests against
a PostgreSQL server you already have, without any of the above.

Architecture

See the presentation at Golang Estonia, PGX Top to Bottom for a description of pgx architecture.

Supported Go and PostgreSQL Versions

pgx supports the same versions of Go and PostgreSQL that are supported by their respective teams. For Go that is the two most recent major releases and for PostgreSQL the major releases in the last 5 years. This means pgx supports Go 1.25 and higher and PostgreSQL 14 and higher. pgx also is tested against the latest version of CockroachDB.

Version Policy

pgx follows semantic versioning for the documented public API on stable releases. v5 is the latest stable major version.

PGX Family Libraries

[github.com/jackc/pglogrepl](https://github.com/jackc/pglogrepl)

pglogrepl provides functionality to act as a client for PostgreSQL logical replication.

[github.com/jackc/pgmock](https://github.com/jackc/pgmock)

pgmock offers the ability to create a server that mocks the PostgreSQL wire protocol. This is used internally to test pgx by purposely inducing unusual errors. pgproto3 and pgmock together provide most of the foundational tooling required to implement a PostgreSQL proxy or MitM (such as for a custom connection pooler).

[github.com/jackc/tern](https://github.com/jackc/tern)

tern is a stand-alone SQL migration system.

[github.com/jackc/pgerrcode](https://github.com/jackc/pgerrcode)

pgerrcode contains constants for the PostgreSQL error codes.

Adapters for 3rd Party Types

Adapters for 3rd Party Tracers

Adapters for 3rd Party Loggers

These adapters can be used with the tracelog package.

3rd Party Libraries with PGX Support

[github.com/pashagolub/pgxmock](https://github.com/pashagolub/pgxmock)

pgxmock is a mock library implementing pgx interfaces.
pgxmock has one and only purpose - to simulate pgx behavior in tests, without needing a real database connection.

[github.com/georgysavva/scany](https://github.com/georgysavva/scany)

Library for scanning data from a database into Go structs and more.

[github.com/vingarcia/ksql](https://github.com/vingarcia/ksql)

A carefully designed SQL client for making using SQL easier,
more productive, and less error-prone on Golang.

[github.com/otan/gopgkrb5](https://github.com/otan/gopgkrb5)

Adds GSSAPI / Kerberos authentication support.

[github.com/wcamarao/pmx](https://github.com/wcamarao/pmx)

Explicit data mapping and scanning library for Go structs and slices.

[github.com/stephenafamo/scan](https://github.com/stephenafamo/scan)

Type safe and flexible package for scanning database data into Go types.
Supports, structs, maps, slices and custom mapping functions.

[github.com/z0ne-dev/mgx](https://github.com/z0ne-dev/mgx)

Code first migration library for native pgx (no database/sql abstraction).

[github.com/amirsalarsafaei/sqlc-pgx-monitoring](https://github.com/amirsalarsafaei/sqlc-pgx-monitoring)

A database monitoring/metrics library for pgx and sqlc. Trace, log and monitor your sqlc query performance using OpenTelemetry.

[https://github.com/nikolayk812/pgx-outbox](https://github.com/nikolayk812/pgx-outbox)

Simple Golang implementation for transactional outbox pattern for PostgreSQL using jackc/pgx driver.

[https://github.com/Arlandaren/pgxWrappy](https://github.com/Arlandaren/pgxWrappy)

Simplifies working with the pgx library, providing convenient scanning of nested structures.

[https://github.com/KoNekoD/pgx-colon-query-rewriter](https://github.com/KoNekoD/pgx-colon-query-rewriter)

Implementation of the pgx query rewriter to use ':' instead of '@' in named query parameters.

Overview (`vendor/github.com/jackc/puddle/v2/README.md`)

Puddle

Puddle is a tiny generic resource pool library for Go that uses the standard
context library to signal cancellation of acquires. It is designed to contain
the minimum functionality required for a resource pool. It can be used directly
or it can be used as the base for a domain specific resource pool. For example,
a database connection pool may use puddle internally and implement health checks
and keep-alive behavior without needing to implement any concurrent code of its
own.

Features

Acquire cancellation via context standard library
Statistics API for monitoring pool pressure
No dependencies outside of standard library and golang.org/x/sync
High performance
100% test coverage of reachable code

Example Usage

package main

import (
	"context"
	"log"
	"net"

	"github.com/jackc/puddle/v2"
)

func main() {
	constructor := func(context.Context) (net.Conn, error) {
		return net.Dial("tcp", "127.0.0.1:8080")
	}
	destructor := func(value net.Conn) {
		value.Close()
	}
	maxPoolSize := int32(10)

	pool, err := puddle.NewPool(&puddle.Config[net.Conn]{Constructor: constructor, Destructor: destructor, MaxSize: maxPoolSize})
	if err != nil {
		log.Fatal(err)
	}

	// Acquire resource from the pool.
	res, err := pool.Acquire(context.Background())
	if err != nil {
		log.Fatal(err)
	}

	// Use resource.
	_, err = res.Value().Write([]byte{1})
	if err != nil {
		log.Fatal(err)
	}

	// Release when done.
	res.Release()
}

Status

Puddle is stable and feature complete.

Bug reports and fixes are welcome.
New features will usually not be accepted if they can be feasibly implemented in a wrapper.
Performance optimizations will usually not be accepted unless the performance issue rises to the level of a bug.

Supported Go Versions

puddle supports the same versions of Go that are supported by the Go project. For Go that is the two most recent major releases. This means puddle supports Go 1.19 and higher.

License

MIT

Overview (`vendor/github.com/klauspost/compress/fse/README.md`)

Finite State Entropy

This package provides Finite State Entropy encoding and decoding.

Finite State Entropy (also referenced as tANS)
encoding provides a fast near-optimal symbol encoding/decoding
for byte blocks as implemented in zstandard.

This can be used for compressing input with a lot of similar input values to the smallest number of bytes.
This does not perform any multi-byte dictionary coding as LZ coders,
but it can be used as a secondary step to compressors (like Snappy) that does not do entropy encoding.

Godoc documentation

News

Feb 2018: First implementation released. Consider this beta software for now.

Usage

This package provides a low level interface that allows to compress single independent blocks.

Each block is separate, and there is no built in integrity checks.
This means that the caller should keep track of block sizes and also do checksums if needed.

Compressing a block is done via the Compress function.
You must provide input and will receive the output and maybe an error.

These error values can be returned:

Error	Description
`<nil>`	Everything ok, output is returned
`ErrIncompressible`	Returned when input is judged to be too hard to compress
`ErrUseRLE`	Returned from the compressor when the input is a single byte value repeated
`(error)`	An internal error occurred.

As can be seen above there are errors that will be returned even under normal operation so it is important to handle these.

To reduce allocations you can provide a Scratch object
that can be re-used for successive calls. Both compression and decompression accepts a Scratch object, and the same
object can be used for both.

Be aware, that when re-using a Scratch object that the output buffer is also re-used, so if you are still using this
you must set the Out field in the scratch to nil. The same buffer is used for compression and decompression output.

Decompressing is done by calling the Decompress function.
You must provide the output from the compression stage, at exactly the size you got back. If you receive an error back
your input was likely corrupted.

It is important to note that a successful decoding does not mean your output matches your original input.
There are no integrity checks, so relying on errors from the decompressor does not assure your data is valid.

For more detailed usage, see examples in the godoc documentation.

Performance

A lot of factors are affecting speed. Block sizes and compressibility of the material are primary factors.
All compression functions are currently only running on the calling goroutine so only one core will be used per block.

The compressor is significantly faster if symbols are kept as small as possible. The highest byte value of the input
is used to reduce some of the processing, so if all your input is above byte value 64 for instance, it may be
beneficial to transpose all your input values down by 64.

With moderate block sizes around 64k speed are typically 200MB/s per core for compression and
around 300MB/s decompression speed.

The same hardware typically does Huffman (deflate) encoding at 125MB/s and decompression at 100MB/s.

Plans

At one point, more internals will be exposed to facilitate more "expert" usage of the components.

A streaming interface is also likely to be implemented. Likely compatible with FSE stream format.

Contributing

Contributions are always welcome. Be aware that adding public functions will require good justification and breaking
changes will likely not be accepted. If in doubt open an issue before writing the PR.

Overview (`vendor/github.com/klauspost/compress/huff0/README.md`)

Huff0 entropy compression

This package provides Huff0 encoding and decoding as used in zstd.

Huff0,
a Huffman codec designed for modern CPU, featuring OoO (Out of Order) operations on multiple ALU
(Arithmetic Logic Unit), achieving extremely fast compression and decompression speeds.

Godoc documentation

News

This is used as part of the zstandard compression and decompression package.

This ensures that most functionality is well tested.

Usage

This package provides a low level interface that allows to compress single independent blocks.

Each block is separate, and there is no built in integrity checks.
This means that the caller should keep track of block sizes and also do checksums if needed.

Compressing a block is done via the Compress1X and
Compress4X functions.
You must provide input and will receive the output and maybe an error.

These error values can be returned:

Error	Description
`<nil>`	Everything ok, output is returned
`ErrIncompressible`	Returned when input is judged to be too hard to compress
`ErrUseRLE`	Returned from the compressor when the input is a single byte value repeated
`ErrTooBig`	Returned if the input block exceeds the maximum allowed size (128 Kib)
`(error)`	An internal error occurred.

As can be seen above some of there are errors that will be returned even under normal operation so it is important to handle these.

The Scratch object will retain state that allows to re-use previous tables for encoding and decoding.

Tables and re-use

Huff0 allows for reusing tables from the previous block to save space if that is expected to give better/faster results.

The Scratch object allows you to set a ReusePolicy
that controls this behaviour. See the documentation for details. This can be altered between each block.

Do however note that this information is not stored in the output block and it is up to the users of the package to
record whether ReadTable should be called,
based on the boolean reported back from the CompressXX call.

If you want to store the table separate from the data, you can access them as OutData and OutTable on the
Scratch object.

Decompressing

The first part of decoding is to initialize the decoding table through ReadTable.
This will initialize the decoding tables.
You can supply the complete block to ReadTable and it will return the data part of the block
which can be given to the decompressor.

Decompressing is done by calling the Decompress1X
or Decompress4X function.

For concurrently decompressing content with a fixed table a stateless Decoder can be requested which will remain correct as long as the scratch is unchanged. The capacity of the provided slice indicates the expected output size.

You must provide the output from the compression stage, at exactly the size you got back. If you receive an error back
your input was likely corrupted.

Contributing

Overview (`vendor/github.com/klauspost/compress/zstd/README.md`)

zstd

Zstandard is a real-time compression algorithm, providing high compression ratios.
It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder.
A high performance compression algorithm is implemented. For now focused on speed.

This package provides compression to and decompression of Zstandard content.

This package is pure Go. Use noasm and nounsafe to disable relevant features.

The zstd package is provided as open source software using a Go standard license.

Currently the package is heavily optimized for 64 bit processors and will be significantly slower on 32 bit processors.

For seekable zstd streams, see this excellent package.

Installation

Install using go get -u github.com/klauspost/compress. The package is located in github.com/klauspost/compress/zstd.

Compressor

Status:

STABLE - there may always be subtle bugs, a wide variety of content has been tested and the library is actively
used by several projects. This library is being fuzz-tested for all updates.

There may still be specific combinations of data types/size/settings that could lead to edge cases,
so as always, testing is recommended.

For now, a high speed (fastest) and medium-fast (default) compressor has been implemented.

The "Fastest" compression ratio is roughly equivalent to zstd level 1.
The "Default" compression ratio is roughly equivalent to zstd level 3 (default).
The "Better" compression ratio is roughly equivalent to zstd level 7.
The "Best" compression ratio is roughly equivalent to zstd level 11.

In terms of speed, it is typically 2x as fast as the stdlib deflate/gzip in its fastest mode.
The compression ratio compared to stdlib is around level 3, but usually 3x as fast.

Usage

An Encoder can be used for either compressing a stream via the
io.WriteCloser interface supported by the Encoder or as multiple independent
tasks via the EncodeAll function.
Smaller encodes are encouraged to use the EncodeAll function.
Use NewWriter to create a new instance that can be used for both.

To create a writer with default options, do like this:

// Compress input to output.
func Compress(in io.Reader, out io.Writer) error {
    enc, err := zstd.NewWriter(out)
    if err != nil {
        return err
    }
    _, err = io.Copy(enc, in)
    if err != nil {
        enc.Close()
        return err
    }
    return enc.Close()
}

Now you can encode by writing data to enc. The output will be finished writing when Close() is called.
Even if your encode fails, you should still call Close() to release any resources that may be held up.

The above is fine for big encodes. However, whenever possible try to reuse the writer.

To reuse the encoder, you can use the Reset(io.Writer) function to change to another output.
This will allow the encoder to reuse all resources and avoid wasteful allocations.

By default, stream encoding has 'light' concurrency, meaning up to 2 goroutines can be working on part
of a stream. This is independent of the WithEncoderConcurrency(n), but that is likely to change
in the future. So if you want to limit concurrency for future updates, specify the concurrency
you would like.

If you would like stream encoding to be done without spawning async goroutines, use WithEncoderConcurrency(1)
which will compress input as each block is completed, blocking on writes until each has completed.

Parallel Stream Compression

For maximum throughput on large streams, use WithConcurrentBlocks(true) together with
WithEncoderConcurrency(n) where n is the number of CPU cores you want to use.
This splits the input into large sections (jobs) that are compressed simultaneously by multiple goroutines,
similar to how the C zstd library does multithreaded compression.

enc, err := zstd.NewWriter(out,
    zstd.WithEncoderLevel(zstd.SpeedDefault),
    zstd.WithEncoderConcurrency(runtime.GOMAXPROCS(0)),
    zstd.WithConcurrentBlocks(true),
)

Each non-first job receives an overlap prefix from the previous job for match context,
so compression ratio is only marginally affected. Output is flushed in order,
producing a valid single-frame zstd stream.

Benchmark on 1.8GB GOB stream (AMD Ryzen 9 9950X):

Level	1 thread	4 threads	16 threads	1T ratio	16T ratio
fastest	783 MB/s	2950 MB/s (3.8×)	6939 MB/s (8.9×)	12.24%	12.26%
default	728 MB/s	2533 MB/s (3.5×)	5340 MB/s (7.3×)	10.67%	10.68%
better	434 MB/s	1105 MB/s (2.5×)	2206 MB/s (5.1×)	9.14%	9.21%
best	129 MB/s	367 MB/s (2.8×)	884 MB/s (6.8×)	8.48%	8.63%

Notes:

Not compatible with dictionary encoding.
Flush() dispatches the current partial job, so latency-sensitive callers can force output.
EncodeAll is unaffected — it uses its own concurrency via the encoder pool.

You can specify your desired compression level using WithEncoderLevel() option. Currently only pre-defined
compression settings can be specified.

Future Compatibility Guarantees

This will be an evolving project. When using this package it is important to note that both the compression efficiency and speed may change.

The goal will be to keep the default efficiency at the default zstd (level 3).
However the encoding should never be assumed to remain the same,
and you should not use hashes of compressed output for similarity checks.

The Encoder can be assumed to produce the same output from the exact same code version.
However, the may be modes in the future that break this,
although they will not be enabled without an explicit option.

This encoder is not designed to (and will probably never) output the exact same bitstream as the reference encoder.

Also note, that the cgo decompressor currently does not report all errors on invalid input,
omits error checks, ignores checksums
and seems to ignore concatenated streams, even though it is part of the spec.

Blocks

For compressing small blocks, the returned encoder has a function called EncodeAll(src, dst []byte) []byte.

EncodeAll will encode all input in src and append it to dst.
This function can be called concurrently.
Each call will only run on a same goroutine as the caller.

Encoded blocks can be concatenated and the result will be the combined input stream.
Data compressed with EncodeAll can be decoded with the Decoder, using either a stream or DecodeAll.

Especially when encoding blocks you should take special care to reuse the encoder.
This will effectively make it run without allocations after a warmup period.
To make it run completely without allocations, supply a destination buffer with space for all content.

import "github.com/klauspost/compress/zstd"

// Create a writer that caches compressors.
// For this operation type we supply a nil Reader.
var encoder, _ = zstd.NewWriter(nil)

// Compress a buffer. 
// If you have a destination buffer, the allocation in the call can also be eliminated.
func Compress(src []byte) []byte {
    return encoder.EncodeAll(src, make([]byte, 0, len(src)))
}

You can control the maximum number of concurrent encodes using the WithEncoderConcurrency(n)
option when creating the writer.

Using the Encoder for both a stream and individual blocks concurrently is safe.

Performance

I have collected some speed examples to compare speed and compression against other compressors.

file is the input file.
out is the compressor used. zskp is this package. zstd is the Datadog cgo library. gzstd/gzkp is gzip standard and this library.
level is the compression level used. For zskp level 1 is "fastest", level 2 is "default"; 3 is "better", 4 is "best".
insize/outsize is the input/output size.
millis is the number of milliseconds used for compression.
mb/s is megabytes (2^20 bytes) per second.

text

/* Detailed source-code truncated for AI context efficiency. */

Decompressor

Status: STABLE - there may still be subtle bugs, but a wide variety of content has been tested.

This library is being continuously fuzz-tested,
kindly supplied by fuzzit.dev.
The main purpose of the fuzz testing is to ensure that it is not possible to crash the decoder,
or run it past its limits with ANY input provided.

Usage

The package has been designed for two main usages, big streams of data and smaller in-memory buffers.
There are two main usages of the package for these. Both of them are accessed by creating a Decoder.

For streaming use a simple setup could look like this:

import "github.com/klauspost/compress/zstd"

func Decompress(in io.Reader, out io.Writer) error {
    d, err := zstd.NewReader(in)
    if err != nil {
        return err
    }
    defer d.Close()
    
    // Copy content...
    _, err = io.Copy(out, d)
    return err
}

It is important to use the "Close" function when you no longer need the Reader to stop running goroutines,
when running with default settings.
Goroutines will exit once an error has been returned, including io.EOF at the end of a stream.

Streams are decoded concurrently in 4 asynchronous stages to give the best possible throughput.
However, if you prefer synchronous decompression, use WithDecoderConcurrency(1) which will decompress data
as it is being requested only.

For decoding buffers, it could look something like this:

import "github.com/klauspost/compress/zstd"

// Create a reader that caches decompressors.
// For this operation type we supply a nil Reader.
var decoder, _ = zstd.NewReader(nil, zstd.WithDecoderConcurrency(0))

// Decompress a buffer. We don't supply a destination buffer,
// so it will be allocated by the decoder.
func Decompress(src []byte) ([]byte, error) {
    return decoder.DecodeAll(src, nil)
}

Both of these cases should provide the functionality needed.
The decoder can be used for concurrent decompression of multiple buffers.
By default 4 decompressors will be created.

It will only allow a certain number of concurrent operations to run.
To tweak that yourself use the WithDecoderConcurrency(n) option when creating the decoder.
It is possible to use WithDecoderConcurrency(0) to create GOMAXPROCS decoders.

Dictionaries

Data compressed with dictionaries can be decompressed.

Dictionaries are added individually to Decoders.
Dictionaries are generated by the zstd --train command and contains an initial state for the decoder.
To add a dictionary use the WithDecoderDicts(dicts ...[]byte) option with the dictionary data.
Several dictionaries can be added at once.

The dictionary will be used automatically for the data that specifies them.
A re-used Decoder will still contain the dictionaries registered.

When registering multiple dictionaries with the same ID, the last one will be used.

It is possible to use dictionaries when compressing data.

To enable a dictionary use WithEncoderDict(dict []byte). Here only one dictionary will be used
and it will likely be used even if it doesn't improve compression.

The used dictionary must be used to decompress the content.

For any real gains, the dictionary should be built with similar data.
If an unsuitable dictionary is used the output may be slightly larger than using no dictionary.
Use the zstd commandline tool to build a dictionary from sample data.
For information see zstd dictionary information.

For now there is a fixed startup performance penalty for compressing content with dictionaries.
This will likely be improved over time. Just be aware to test performance when implementing.

Allocation-less operation

The decoder has been designed to operate without allocations after a warmup.

This means that you should store the decoder for best performance.
To re-use a stream decoder, use the Reset(r io.Reader) error to switch to another stream.
A decoder can safely be re-used even if the previous stream failed.

To release the resources, you must call the Close() function on a decoder.
After this it can no longer be reused, but all running goroutines will be stopped.
So you must use this if you will no longer need the Reader.

For decompressing smaller buffers a single decoder can be used.
When decoding buffers, you can supply a destination slice with length 0 and your expected capacity.
In this case no unneeded allocations should be made.

Concurrency

The buffer decoder does everything on the same goroutine and does nothing concurrently.
It can however decode several buffers concurrently. Use WithDecoderConcurrency(n) to limit that.

The stream decoder will create goroutines that:

Reads input and splits the input into blocks.
Decompression of literals.
Decompression of sequences.
Reconstruction of output stream.

So effectively this also means the decoder will "read ahead" and prepare data to always be available for output.

The concurrency level will, for streams, determine how many blocks ahead the compression will start.

Since "blocks" are quite dependent on the output of the previous block stream decoding will only have limited concurrency.

In practice this means that concurrency is often limited to utilizing about 3 cores effectively.

Benchmarks

The first two are streaming decodes and the last are smaller inputs.

Running on AMD Ryzen 9 3950X 16-Core Processor. AMD64 assembly used.

text

/* Detailed source-code truncated for AI context efficiency. */

This reflects the performance around May 2022, but this may be out of date.

Zstd inside ZIP files

It is possible to use zstandard to compress individual files inside zip archives.
While this isn't widely supported it can be useful for internal files.

To support the compression and decompression of these files you must register a compressor and decompressor.

It is highly recommended registering the (de)compressors on individual zip Reader/Writer and NOT
use the global registration functions. The main reason for this is that 2 registrations from
different packages will result in a panic.

It is a good idea to only have a single compressor and decompressor, since they can be used for multiple zip
files concurrently, and using a single instance will allow reusing some resources.

See this example for
how to compress and decompress files inside zip archives.

Contributions

Contributions are always welcome.
For new features/fixes, remember to add tests and for performance enhancements include benchmarks.

For general feedback and experience reports, feel free to open an issue or write me on Twitter.

Overview (`vendor/github.com/prometheus/client_golang/prometheus/README.md`)

See .

Overview (`vendor/github.com/uber-go/tally/v4/README.md`)

:heavy_check_mark: tally [![GoDoc][doc-img]][doc] [![Build Status][ci-img]][ci] [![Coverage Status][cov-img]][cov]

Fast, buffered, hierarchical stats collection in Go.

Installation

go get -u github.com/uber-go/tally

Abstract

Tally provides a common interface for emitting metrics, while letting you not worry about the velocity of metrics emission.

By default it buffers counters, gauges and histograms at a specified interval but does not buffer timer values. This is primarily so timer values can have all their values sampled if desired and if not they can be sampled as summaries or histograms independently by a reporter.

Structure

Scope: Keeps track of metrics, and their common metadata.
Metrics: Counters, Gauges, Timers and Histograms.
Reporter: Implemented by you. Accepts aggregated values from the scope. Forwards the aggregated values to your metrics ingestion pipeline.
- The reporters already available listed alphabetically are:
  - github.com/uber-go/tally/m3: Report m3 metrics, timers are not sampled and forwarded directly.
  - github.com/uber-go/tally/multi: Report to multiple reporters, you can multi-write metrics to other reporters simply.
  - github.com/uber-go/tally/prometheus: Report prometheus metrics, timers by default are made summaries with an option to make them histograms instead.
  - github.com/uber-go/tally/statsd: Report statsd metrics, no support for tags.

Basics

Scopes created with tally provide race-safe registration and use of all metric types Counter, Gauge, Timer, Histogram.
NewRootScope(...) returns a Scope and io.Closer, the second return value is used to stop the scope's goroutine reporting values from the scope to it's reporter. This is to reduce the footprint of Scope from the public API for those implementing it themselves to use in Go packages that take a tally Scope.

Acquire a Scope

reporter = NewMyStatsReporter()  // Implement as you will
tags := map[string]string{
	"dc": "east-1",
	"type": "master",
}
reportEvery := time.Second

scope := tally.NewRootScope(tally.ScopeOptions{
	Tags: tags,
	Reporter: reporter,
}, reportEvery)

Get/Create a metric, use it

// Get a counter, increment a counter
reqCounter := scope.Counter("requests")  // cache me
reqCounter.Inc(1)

queueGauge := scope.Gauge("queue_length")  // cache me
queueGauge.Update(42)

Report your metrics

Use the inbuilt statsd reporter:

import (
	"io"
	"github.com/cactus/go-statsd-client/v5/statsd"
	"github.com/uber-go/tally"
	tallystatsd "github.com/uber-go/tally/statsd"
	// ...
)

func newScope() (tally.Scope, io.Closer) {
	statter, _ := statsd.NewBufferedClient("127.0.0.1:8125",
		"stats", 100*time.Millisecond, 1440)

	reporter := tallystatsd.NewReporter(statter, tallystatsd.Options{
		SampleRate: 1.0,
	})

	scope, closer := tally.NewRootScope(tally.ScopeOptions{
		Prefix:   "my-service",
		Tags:     map[string]string{},
		Reporter: reporter,
	}, time.Second)

	return scope, closer
}

Implement your own reporter using the StatsReporter interface:

// BaseStatsReporter implements the shared reporter methods.
type BaseStatsReporter interface {
	Capabilities() Capabilities
	Flush()
}

// StatsReporter is a backend for Scopes to report metrics to.
type StatsReporter interface {
	BaseStatsReporter

	// ReportCounter reports a counter value
	ReportCounter(
		name string,
		tags map[string]string,
		value int64,
	)

	// ReportGauge reports a gauge value
	ReportGauge(
		name string,
		tags map[string]string,
		value float64,
	)

	// ReportTimer reports a timer value
	ReportTimer(
		name string,
		tags map[string]string,
		interval time.Duration,
	)

	// ReportHistogramValueSamples reports histogram samples for a bucket
	ReportHistogramValueSamples(
		name string,
		tags map[string]string,
		buckets Buckets,
		bucketLowerBound,
		bucketUpperBound float64,
		samples int64,
	)

	// ReportHistogramDurationSamples reports histogram samples for a bucket
	ReportHistogramDurationSamples(
		name string,
		tags map[string]string,
		buckets Buckets,
		bucketLowerBound,
		bucketUpperBound time.Duration,
		samples int64,
	)
}

Or implement your own metrics implementation that matches the tally Scope interface to use different buffering semantics:

text

/* Detailed source-code truncated for AI context efficiency. */

Performance

This stuff needs to be fast. With that in mind, we avoid locks and unnecessary memory allocations.

text

BenchmarkCounterInc-8               	200000000	         7.68 ns/op
BenchmarkReportCounterNoData-8      	300000000	         4.88 ns/op
BenchmarkReportCounterWithData-8    	100000000	        21.6 ns/op
BenchmarkGaugeSet-8                 	100000000	        16.0 ns/op
BenchmarkReportGaugeNoData-8        	100000000	        10.4 ns/op
BenchmarkReportGaugeWithData-8      	50000000	        27.6 ns/op
BenchmarkTimerInterval-8            	50000000	        37.7 ns/op
BenchmarkTimerReport-8              	300000000	         5.69 ns/op

Released under the MIT License.

Overview (`vendor/golang.org/x/net/http2/README.md`)

This package (golang.org/x/net/http2) is the original source of truth
of the Go HTTP/2 implementation.

As of Go 1.27, the source of truth has moved to the standard library
package net/http/internal/http2.
All new feature development should happen in that package.
Only critical bug fixes and security fixes will be backported to x/net.

The x/net package contains two implementations of the HTTP/2 transport and server:

The original implementation (no longer the source of truth).

A reimplementation of the x/net/http2 APIs in terms of net/http.
This is called "the wrapping implementation", since it wraps net/http.

The original implementation is used when the Go version is less than 1.27.

The wrapping implementation is used when the Go version is at least 1.27.
The build tag "http2legacy" may be set to use the original implementation.

Overview (`vendor/golang.org/x/sys/unix/README.md`)

Building `sys/unix`

The sys/unix package provides access to the raw system call interface of the
underlying operating system. See: https://godoc.org/golang.org/x/sys/unix

Porting Go to a new architecture/OS combination or adding syscalls, types, or
constants to an existing architecture/OS pair requires some manual effort;
however, there are tools that automate much of the process.

Build Systems

There are currently two ways we generate the necessary files. We are currently
migrating the build system to use containers so the builds are reproducible.
This is being done on an OS-by-OS basis. Please update this documentation as
components of the build system change.

Old Build System (currently for `GOOS != "linux"`)

The old build system generates the Go files based on the C header files
present on your system. This means that files
for a given GOOS/GOARCH pair must be generated on a system with that OS and
architecture. This also means that the generated code can differ from system
to system, based on differences in the header files.

To avoid this, if you are using the old build system, only generate the Go
files on an installation with unmodified header files. It is also important to
keep track of which version of the OS the files were generated from (ex.
Darwin 14 vs Darwin 15). This makes it easier to track the progress of changes
and have each OS upgrade correspond to a single change.

To build the files for your current OS and architecture, make sure GOOS and
GOARCH are set correctly and run mkall.sh. This will generate the files for
your specific system. Running mkall.sh -n shows the commands that will be run.

Requirements: bash, go

New Build System (currently for `GOOS == "linux"`)

The new build system uses a Docker container to generate the go files directly
from source checkouts of the kernel and various system libraries. This means
that on any platform that supports Docker, all the files using the new build
system can be generated at once, and generated files will not change based on
what the person running the scripts has installed on their computer.

The OS specific files for the new build system are located in the ${GOOS}
directory, and the build is coordinated by the ${GOOS}/mkall.go program. When
the kernel or system library updates, modify the Dockerfile at
${GOOS}/Dockerfile to checkout the new release of the source.

To build all the files under the new build system, you must be on an amd64/Linux
system and have your GOOS and GOARCH set accordingly. Running mkall.sh will
then generate all of the files for all of the GOOS/GOARCH pairs in the new build
system. Running mkall.sh -n shows the commands that will be run.

Requirements: bash, go, docker

Component files

This section describes the various files used in the code generation process.
It also contains instructions on how to modify these files to add a new
architecture/OS or to add additional syscalls, types, or constants. Note that
if you are using the new build system, the scripts/programs cannot be called normally.
They must be called from within the docker container.

asm files

The hand-written assembly file at asm_${GOOS}_${GOARCH}.s implements system
call dispatch. There are three entry points:

text

func Syscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr)
  func Syscall6(trap, a1, a2, a3, a4, a5, a6 uintptr) (r1, r2, err uintptr)
  func RawSyscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr)

The first and second are the standard ones; they differ only in how many
arguments can be passed to the kernel. The third is for low-level use by the
ForkExec wrapper. Unlike the first two, it does not call into the scheduler to
let it know that a system call is running.

When porting Go to a new architecture/OS, this file must be implemented for
each GOOS/GOARCH pair.

mksysnum

Mksysnum is a Go program located at ${GOOS}/mksysnum.go (or mksysnum_${GOOS}.go
for the old system). This program takes in a list of header files containing the
syscall number declarations and parses them to produce the corresponding list of
Go numeric constants. See zsysnum_${GOOS}_${GOARCH}.go for the generated
constants.

Adding new syscall numbers is mostly done by running the build on a sufficiently
new installation of the target OS (or updating the source checkouts for the
new build system). However, depending on the OS, you may need to update the
parsing in mksysnum.

mksyscall.go

The syscall.go, syscall_${GOOS}.go, syscall_${GOOS}_${GOARCH}.go are
hand-written Go files which implement system calls (for unix, the specific OS,
or the specific OS/Architecture pair respectively) that need special handling
and list //sys comments giving prototypes for ones that can be generated.

The mksyscall.go program takes the //sys and //sysnb comments and converts
them into syscalls. This requires the name of the prototype in the comment to
match a syscall number in the zsysnum_${GOOS}_${GOARCH}.go file. The function
prototype can be exported (capitalized) or not.

Adding a new syscall often just requires adding a new //sys function prototype
with the desired arguments and a capitalized name so it is exported. However, if
you want the interface to the syscall to be different, often one will make an
unexported //sys prototype, and then write a custom wrapper in
syscall_${GOOS}.go.

types files

For each OS, there is a hand-written Go file at ${GOOS}/types.go (or
types_${GOOS}.go on the old system). This file includes standard C headers and
creates Go type aliases to the corresponding C types. The file is then fed
through godef to get the Go compatible definitions. Finally, the generated code
is fed though mkpost.go to format the code correctly and remove any hidden or
private identifiers. This cleaned-up code is written to
ztypes_${GOOS}_${GOARCH}.go.

The hardest part about preparing this file is figuring out which headers to
include and which symbols need to be #defined to get the actual data
structures that pass through to the kernel system calls. Some C libraries
preset alternate versions for binary compatibility and translate them on the
way in and out of system calls, but there is almost always a #define that can
get the real ones.
See types_darwin.go and linux/types.go for examples.

To add a new type, add in the necessary include statement at the top of the
file (if it is not already there) and add in a type alias line. Note that if
your type is significantly different on different architectures, you may need
some #if/#elif macros in your include statements.

mkerrors.sh

This script is used to generate the system's various constants. This doesn't
just include the error numbers and error strings, but also the signal numbers
and a wide variety of miscellaneous constants. The constants come from the list
of include files in the includes_${uname} variable. A regex then picks out
the desired #define statements, and generates the corresponding Go constants.
The error numbers and strings are generated from #include <errno.h>, and the
signal numbers and strings are generated from #include <signal.h>. All of
these constants are written to zerrors_${GOOS}_${GOARCH}.go via a C program,
_errors.c, which prints out all the constants.

To add a constant, add the header that includes it to the appropriate variable.
Then, edit the regex (if necessary) to match the desired constant. Avoid making
the regex too broad to avoid matching unintended constants.

internal/mkmerge

This program is used to extract duplicate const, func, and type declarations
from the generated architecture-specific files listed below, and merge these
into a common file for each OS.

The merge is performed in the following steps:

Construct the set of common code that is identical in all architecture-specific files.
Write this common code to the merged file.
Remove the common code from all architecture-specific files.

Generated files

`zerrors_${GOOS}_${GOARCH}.go`

A file containing all of the system's generated error numbers, error strings,
signal numbers, and constants. Generated by mkerrors.sh (see above).

`zsyscall_${GOOS}_${GOARCH}.go`

A file containing all the generated syscalls for a specific GOOS and GOARCH.
Generated by mksyscall.go (see above).

`zsysnum_${GOOS}_${GOARCH}.go`

A list of numeric constants for all the syscall number of the specific GOOS
and GOARCH. Generated by mksysnum (see above).

`ztypes_${GOOS}_${GOARCH}.go`

A file containing Go types for passing into (or returning from) syscalls.
Generated by godefs and the types file (see above).

Overview (`vendor/google.golang.org/grpc/cmd/protoc-gen-go-grpc/README.md`)

protoc-gen-go-grpc

This tool generates Go language bindings of services in protobuf definition
files for gRPC. For usage information, please see our quick start
guide.

Future-proofing services

By default, to register services using the methods generated by this tool, the
service implementations must embed the corresponding
Unimplemented<ServiceName>Server for future compatibility. This is a behavior
change from the grpc code generator previously included with protoc-gen-go.
To restore this behavior, set the option require_unimplemented_servers=false.
E.g.:

protoc --go-grpc_out=. --go-grpc_opt=require_unimplemented_servers=false[,other options...] \

Note that this is not recommended, and the option is only provided to restore
backward compatibility with previously-generated code.

When embedding the Unimplemented<ServiceName>Server in a struct that
implements the service, it should be embedded by value instead of as a
pointer. If it is embedded as a pointer, it must be assigned to a valid,
non-nil pointer or else unimplemented methods would panic when called. This is
tested at service registration time, and will lead to a panic in
Register<ServiceName>Server if it is not embedded properly.

Overview (`vendor/github.com/blugelabs/bluge/numeric/geo/README.md`)

geo support in blube

First, all of this geo code is a Go adaptation of the Lucene 5.3.2 sandbox geo support.

Notes

All of the APIs will use float64 for lon/lat values.
When describing a point in function arguments or return values, we always use the order lon, lat.
High level APIs will use TopLeft and BottomRight to describe bounding boxes. This may not map cleanly to min/max lon/lat when crossing the dateline. The lower level APIs will use min/max lon/lat and require the higher-level code to split boxes accordingly.

Overview (`vendor/github.com/jackc/pgx/v5/pgconn/README.md`)

pgconn

Package pgconn is a low-level PostgreSQL database driver. It operates at nearly the same level as the C library libpq.
It is primarily intended to serve as the foundation for higher level libraries such as https://github.com/jackc/pgx.
Applications should handle normal queries with a higher level library and only use pgconn directly when required for
low-level access to PostgreSQL functionality.

Example Usage

pgConn, err := pgconn.Connect(context.Background(), os.Getenv("DATABASE_URL"))
if err != nil {
	log.Fatalln("pgconn failed to connect:", err)
}
defer pgConn.Close(context.Background())

result := pgConn.ExecParams(context.Background(), "SELECT email FROM users WHERE id=$1", [][]byte{[]byte("123")}, nil, nil, nil)
for result.NextRow() {
	fmt.Println("User 123 has email:", string(result.Values()[0]))
}
_, err = result.Close()
if err != nil {
	log.Fatalln("failed reading result:", err)
}

Testing

See CONTRIBUTING.md for setup instructions.

Overview (`vendor/github.com/jackc/pgx/v5/pgproto3/README.md`)

pgproto3

Package pgproto3 is an encoder and decoder of the PostgreSQL wire protocol version 3.

pgproto3 can be used as a foundation for PostgreSQL drivers, proxies, mock servers, load balancers and more.

See example/pgfortune for a playful example of a fake PostgreSQL server.

Overview (`vendor/github.com/stretchr/testify/internal/spew/README.md`)

go-spew

Go-spew implements a deep pretty printer for Go data structures to aid in
debugging. A comprehensive suite of tests with 100% test coverage is provided
to ensure proper functionality.

License

Go-spew is licensed under the copyfree ISC License.

Overview (`vendor/github.com/uber-go/tally/v4/prometheus/README.md`)

A buffered Prometheus reporter

See examples/prometheus_main.go for an end to end example.

Options

You can use a specific Prometheus registry, and you can use
either summaries or histograms for timers.

The reporter options are:

// Options is a set of options for the tally reporter.
type Options struct {
	// Registerer is the prometheus registerer to register
	// metrics with. Use nil to specify the default registerer.
	Registerer prom.Registerer

	// DefaultTimerType is the default type timer type to create
	// when using timers. It's default value is a histogram timer type.
	DefaultTimerType TimerType

	// DefaultHistogramBuckets is the default histogram buckets
	// to use. Use nil to specify the default histogram buckets.
	DefaultHistogramBuckets []float64

	// DefaultSummaryObjectives is the default summary objectives
	// to use. Use nil to specify the default summary objectives.
	DefaultSummaryObjectives map[float64]float64

	// OnRegisterError defines a method to call to when registering
	// a metric with the registerer fails. Use nil to specify
	// to panic by default when registering a metric fails.
	OnRegisterError func(err error)
}

The timer types are:

// TimerType describes a type of timer
type TimerType int

const (
	// SummaryTimerType is a timer type that reports into a summary
	SummaryTimerType TimerType = iota

	// HistogramTimerType is a timer type that reports into a histogram
	HistogramTimerType
)

You can also pre-register help description text ahead of using a metric
that will be named and tagged identically with tally. You can also
access the Prometheus HTTP handler directly.

The returned reporter interface:

// Reporter is a Prometheus backed tally reporter.
type Reporter interface {
	tally.CachedStatsReporter

	// HTTPHandler provides the Prometheus HTTP scrape handler.
	HTTPHandler() http.Handler

	// RegisterCounter is a helper method to initialize a counter
	// in the Prometheus backend with a given help text.
	// If not called explicitly, the Reporter will create one for
	// you on first use, with a not super helpful HELP string.
	RegisterCounter(
		name string,
		tagKeys []string,
		desc string,
	) (*prom.CounterVec, error)

	// RegisterGauge is a helper method to initialize a gauge
	// in the prometheus backend with a given help text.
	// If not called explicitly, the Reporter will create one for
	// you on first use, with a not super helpful HELP string.
	RegisterGauge(
		name string,
		tagKeys []string,
		desc string,
	) (*prom.GaugeVec, error)

	// RegisterTimer is a helper method to initialize a timer
	// summary or histogram vector in the prometheus backend
	// with a given help text.
	// If not called explicitly, the Reporter will create one for
	// you on first use, with a not super helpful HELP string.
	// You may pass opts as nil to get the default timer type
	// and objectives/buckets.
	// You may also pass objectives/buckets as nil in opts to
	// get the default objectives/buckets for the specified
	// timer type.
	RegisterTimer(
		name string,
		tagKeys []string,
		desc string,
		opts *RegisterTimerOptions,
	) (TimerUnion, error)
}

The register timer options:

// RegisterTimerOptions provides options when registering a timer on demand.
// By default you can pass nil for the options to get the reporter defaults.
type RegisterTimerOptions struct {
	TimerType         TimerType
	HistogramBuckets  []float64
	SummaryObjectives map[float64]float64
}

Overview (`vendor/golang.org/x/net/internal/httpcommon/README.md`)

These two packages are identical:

net/http/internal/httpcommon
golang.org/x/net/internal/httpcommon

The two are kept in sync manually.
Any change to one should be made in the other.

Overview (`vendor/github.com/grpc-ecosystem/grpc-gateway/v2/internal/casing/README.md`)

Case conversion

This package contains two functions:

Camel copied from the github.com/golang/protobuf/protoc-gen-go/generator package.
JSONCamelCase copied from the github.com/protocolbuffers/protobuf-go/internal/strs package.

Both these modules are licensed by The Go Authors, as reflected in this package's [LICENSE.md].

Overview (`vendor/github.com/jackc/pgx/v5/internal/pgio/README.md`)

pgio

Package pgio is a low-level toolkit building messages in the PostgreSQL wire protocol.

pgio provides functions for appending integers to a []byte while doing byte
order conversion.

Overview (`vendor/github.com/klauspost/compress/zstd/internal/xxhash/README.md`)

xxhash

VENDORED: Go to github.com/cespare/xxhash for original package.

xxhash is a Go implementation of the 64-bit xxHash algorithm, XXH64. This is a
high-quality hashing algorithm that is much faster than anything in the Go
standard library.

This package provides a straightforward API:

text

func Sum64(b []byte) uint64
func Sum64String(s string) uint64
type Digest struct{ ... }
    func New() *Digest

The Digest type implements hash.Hash64. Its key methods are:

text

func (*Digest) Write([]byte) (int, error)
func (*Digest) WriteString(string) (int, error)
func (*Digest) Sum64() uint64

Compatibility

This package is in a module and the latest code is in version 2 of the module.
You need a version of Go with at least "minimal module compatibility" to use
github.com/cespare/xxhash/v2:

1.9.7+ for Go 1.9
1.10.3+ for Go 1.10
Go 1.11 or later

I recommend using the latest release of Go.

Benchmarks

Here are some quick benchmarks comparing the pure-Go and assembly
implementations of Sum64.

input size	purego	asm
4 B	1.3 GB/s	1.2 GB/s
16 B	2.9 GB/s	3.5 GB/s
100 B	6.9 GB/s	8.1 GB/s
4 KB	11.7 GB/s	16.7 GB/s
10 MB	12.0 GB/s	17.3 GB/s

These numbers were generated on Ubuntu 20.04 with an Intel Xeon Platinum 8252C
CPU using the following commands under Go 1.19.2:

text

benchstat <(go test -tags purego -benchtime 500ms -count 15 -bench 'Sum64$')
benchstat <(go test -benchtime 500ms -count 15 -bench 'Sum64$')

Projects using this package

--- METRICS ---

Files Extracted: 72
Estimated Token Budget: ~92215 tokens
Canonical Reference: https://codewiki.google/github.com/heroiclabs/nakama

Docker & Self-Hosted Deployment Verified OCI

Technical Documentation: heroiclabs/nakama

Overview (`README.md`) Copy Section

Features Copy Chapter

Getting Started Copy Chapter

Docker Copy Section

Binaries Copy Section

Usage Copy Chapter

Nakama Console Copy Chapter

Deployment Copy Chapter

Heroic Cloud Copy Section

Contribute Copy Chapter

Simple Builds Copy Section

Full Source Builds Copy Section

Testing Copy Section

License Copy Section

Overview (`build/README.md`) Copy Section

Release Instructions

Steps Copy Chapter

Build Nakama Image Copy Chapter

Build Nakama Image (dSYM) Copy Chapter

Build Plugin Builder Image Copy Chapter

Build all images with provided script. Copy Chapter

Overview (`sample_go_module/README.md`) Copy Section

Nakama Go Runtime

Minimal example Copy Chapter

Setup a project Copy Chapter

Build & load process Copy Chapter

Docker builds Copy Section

Bigger Example Copy Chapter

Overview (`build/do-marketplace/README.md`) Copy Section

Build Automation with Packer

Usage Copy Chapter

Configuration Details Copy Chapter

Overview (`internal/cronexpr/README.md`) Copy Section

Golang Cron expression parser

Implementation Copy Chapter

Asterisk ( * )

Slash ( / )

Comma ( , )

Hyphen ( - )

L

W

Hash ( # )

Predefined cron expressions Copy Chapter

Other details Copy Chapter

Install Copy Chapter

Usage Copy Chapter

API Copy Chapter

License Copy Chapter

Overview (`internal/skiplist/README.md`) Copy Section

skiplist

Usage

License

Overview (`internal/cronexpr/cronexpr/README.md`) Copy Section

cronexpr: command-line utility

Install Copy Chapter

Usage Copy Chapter

Options Copy Chapter

Examples Copy Chapter

Example 1

Example 2

Example 3

Example 4

Overview (`vendor/go.uber.org/atomic/README.md`) Copy Section

atomic [![GoDoc][doc-img]][doc] [![Build Status][ci-img]][ci] [![Coverage Status][cov-img]][cov] [![Go Report Card][reportcard-img]][reportcard]

Installation Copy Chapter

Legacy Import Path Copy Section

Usage Copy Chapter

Development Status Copy Chapter

Overview (`vendor/go.uber.org/multierr/README.md`) Copy Section

multierr [![GoDoc][doc-img]][doc] [![Build Status][ci-img]][ci] [![Coverage Status][cov-img]][cov]

Features Copy Chapter

Installation Copy Chapter

Status Copy Chapter

Overview (`vendor/go.uber.org/zap/README.md`) Copy Section

:zap: zap

Installation Copy Chapter

Quick Start Copy Chapter

Performance Copy Chapter

Overview (`README.md`)

Features

Getting Started

Docker

Binaries

Usage

Nakama Console

Deployment

Heroic Cloud

Contribute

Simple Builds

Full Source Builds

Testing

License

Overview (`build/README.md`)

Steps

Build Nakama Image

Build Nakama Image (dSYM)

Build Plugin Builder Image

Build all images with provided script.

Overview (`sample_go_module/README.md`)

Minimal example

Setup a project

Build & load process

Docker builds

Bigger Example

Overview (`build/do-marketplace/README.md`)

Usage

Configuration Details

Overview (`internal/cronexpr/README.md`)

Implementation

Predefined cron expressions

Other details

Install

Usage

API

License

Overview (`internal/skiplist/README.md`)

Overview (`internal/cronexpr/cronexpr/README.md`)

Install

Usage

Options

Examples

Overview (`vendor/go.uber.org/atomic/README.md`)

Installation

Legacy Import Path

Usage

Development Status

Overview (`vendor/go.uber.org/multierr/README.md`)

Features

Installation

Status

Overview (`vendor/go.uber.org/zap/README.md`)

Installation

Quick Start

Performance

Development Status: Stable

Contributing

Overview (`vendor/google.golang.org/grpc/README.md`)

Prerequisites

Installation

Learn more

FAQ

I/O Timeout Errors

Compiling error, undefined: grpc.SupportPackageIsVersion

How to turn on logging

The RPC failed with error `"code = Unavailable desc = transport is closing"`

Overview (`vendor/gopkg.in/yaml.v3/README.md`)

Introduction

Compatibility

Installation and usage

API documentation

API stability

License

Example

Overview (`vendor/github.com/RoaringBitmap/roaring/README.md`)

References

Dependencies

Instructions for contributors

Example