## 1. Project Overview & Quickstart (heroiclabs/nakama) # Technical Documentation: heroiclabs/nakama > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/heroiclabs/nakama) ### Overview (`README.md`)

Nakama - Distributed server for social and realtime games and apps

Version Downloads License Nakama Forum Nakama Documentation

## Features * **Users** - Register/login new users via social networks, email, or device ID. * **Storage** - Store user records, settings, and other objects in collections. * **Social** - Users can connect with friends, and join groups. Builtin social graph to see how users can be connected. * **Chat** - 1-on-1, group, and global chat between users. Persist messages for chat history. * **Multiplayer** - Realtime, or turn-based active and passive multiplayer. * **Leaderboards** - Dynamic, seasonal, get top members, or members around a user. Have as many as you need. * **Tournaments** - Invite players to compete together over prizes. Link many together to create leagues. * **Parties** - Add team play to a game. Users can form a party and communicate with party members. * **Purchase Validation** - Validate in-app purchases and subscriptions. * **In-App Notifications** - Send messages and notifications to connected client sockets. * **Runtime code** - Extend the server with custom logic written in Lua, TypeScript/JavaScript, or native Go code. * **Matchmaker**, **dashboard**, **metrics**, and [more](https://heroiclabs.com/docs). Build scalable games and apps with a production ready server used by ambitious game studios and app developers [all around the world](https://heroiclabs.com/customers/). Have a look at the [documentation](https://heroiclabs.com/docs) and join the [developer community](https://forum.heroiclabs.com) for more info. ## Getting Started The server is simple to setup and run for local development and can be deployed to any cloud provider. See the [deployment notes](#deployment) for recommendations on how to deploy the project for production. Nakama server requires CockroachDB or another Postgres wire-compatible server as it's database. ### Docker The fastest way to run the server and the database is with Docker. Setup Docker and start the daemon. 1. Set up a [docker-compose file](https://heroiclabs.com/docs/nakama/getting-started/install/docker/#running-nakama) and place it in a folder for your project. 2. Run `docker-compose -f ./docker-compose.yml up` to download container images and run the servers. For more detailed instructions have a look at our [Docker quickstart](https://heroiclabs.com/docs/nakama/getting-started/install/docker) guide. Nakama Docker images are maintained on [Docker Hub](https://hub.docker.com/r/heroiclabs/nakama/tags) and [prerelease](https://hub.docker.com/r/heroiclabs/nakama-prerelease/tags) images are occasionally published for cutting edge features of the server. ### Binaries You can run the servers with native binaries for your platform. 1. Download the server from our [releases](https://github.com/heroiclabs/nakama/releases) page and the [database](https://www.cockroachlabs.com/docs/stable/install-cockroachdb.html). 2. Follow the database [instructions](https://www.cockroachlabs.com/docs/stable/start-a-local-cluster.html#before-you-begin) to start it. 3. Run a migration which will setup or upgrade the database schema: ```shell nakama migrate up --database.address "root@127.0.0.1:26257" ``` 4. Start Nakama and connect to the database: ```shell nakama --database.address "root@127.0.0.1:26257" ``` When connected you'll see server output which describes all settings the server uses for [configuration](https://heroiclabs.com/docs/nakama/getting-started/configuration). > {"level":"info","ts":"2018-04-29T10:14:41.249+0100","msg":"Node","name":"nakama","version":"2.0.0+7e18b09","runtime":"go1.10.1","cpu":4}
> {"level":"info","ts":"2018-04-29T10:14:41.249+0100","msg":"Database connections","dsns":["root@127.0.0.1:26257"]}
> ... ## Usage Nakama supports a variety of protocols optimized for various gameplay or app use cases. For request/response it can use GRPC or the HTTP1.1+JSON fallback (REST). For realtime communication you can use WebSockets or rUDP. For example with the REST API to authenticate a user account with a device identifier. ```shell curl "127.0.0.1:7350/v2/account/authenticate/device?create=true" \ --user "defaultkey:" \ --data '{"id": "someuniqueidentifier"}' ``` Response: > {
> "token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1MjQ5OTU2NDksInVpZCI6Ijk5Y2Q1YzUyLWE5ODgtNGI2NC04YThhLTVmMTM5YTg4MTgxMiIsInVzbiI6InhBb1RxTUVSdFgifQ.-3_rXNYx3Q4jKuS7RkxeMWBzMNAm0vl93QxzRI8p_IY"
> } There's a number of official [client libraries](https://github.com/heroiclabs) available on GitHub with [documentation](https://heroiclabs.com/docs). The current platform/language support includes: .NET (in C#), Unity engine, JavaScript, Java (with Android), Unreal engine, Godot, Defold, and Swift (with iOS). If you'd like to contribute a client or request one let us know. ## Nakama Console The server provides a web UI which teams can use to inspect various data stored through the server APIs, view lightweight service metrics, manage player data, update storage objects, restrict access to production with permission profiles, and gain visibility into realtime features like active multiplayer matches. There is no separate installation required as it is embedded as part of the single server binary. You can navigate to it on your browser on [http://127.0.0.1:7351](http://127.0.0.1:7351).
View Screenshots Nakama Console dashboard view Nakama Console players view Nakama Console API explorer view Nakama Console storage view Nakama Console modules view
## Deployment Nakama can be deployed to any cloud provider such as Google Cloud, Azure, AWS, Digital Ocean, Heroku, or your own private cloud. You should setup and provision separate nodes for Nakama and CockroachDB. The recommended minimum production infrastructure for CockroachDB is outlined in [these docs](https://www.cockroachlabs.com/docs/stable/recommended-production-settings.html#basic-hardware-recommendations) and Nakama can be run on instance types as small as "g1-small" on Google Cloud although we recommend a minimum of "n1-standard-1" in production. The specific hardware requirements will depend on what features of the server are used. Reach out to us for help and advice on what servers to run. ### Heroic Cloud You can support development, new features, and maintainance of the server by using the Heroic Labs' [Heroic Cloud](https://heroiclabs.com/heroic-cloud/) for deployment. This service handles the uptime, replication, backups, logs, data upgrades, and all other tasks involved with production server environments. Have a look at our [Heroic Cloud](https://heroiclabs.com/heroic-cloud/) service for more details. ## Contribute The development roadmap is managed as GitHub issues and pull requests are welcome. If you're interested to add a feature which is not mentioned on the issue tracker please open one to create a discussion or drop in and discuss it in the [community forum](https://forum.heroiclabs.com). ### Simple Builds All dependencies required for a build are vendored as part of the Go project. We recommend a modern release of the Go toolchain and do not store the codebase in the old GOPATH. 1. Download the source tree. ```shell git clone "https://github.com/heroiclabs/nakama" nakama cd nakama ``` 2. Build the project from source. ```shell go build -trimpath -mod=vendor ./nakama --version ``` ### Full Source Builds The codebase uses Protocol Buffers, GRPC, GRPC-Gateway, buf, and the OpenAPI spec as part of the project. These dependencies are generated as sources and committed to the repository to simplify builds for contributors. To build the codebase and generate all sources follow these steps. 1. Install the toolchain. ```shell go install \ "google.golang.org/protobuf/cmd/protoc-gen-go" \ "google.golang.org/grpc/cmd/protoc-gen-go-grpc" \ "github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway" \ "github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2" ``` 2. Install [buf](https://buf.build/docs/cli/installation/). 3. Re-generate the protocol buffers and gateway code. ```shell # Run the shell script: ./buf.sh ``` 4. Build the codebase. ```shell go build -trimpath -mod=vendor ``` ### Testing In order to run all the unit and integration tests run: ```shell docker-compose -f ./docker-compose-tests.yml up --build --abort-on-container-exit; docker-compose -f ./docker-compose-tests.yml down -v ``` This will create an isolated environment with Nakama and database instances, run all the tests, and drop the environment afterwards. ### License This project is licensed under the [Apache-2 License](https://github.com/heroiclabs/nakama/blob/master/LICENSE). --- ### Overview (`build/README.md`) Release Instructions === These instructions guide the release process for new official Nakama server builds. ## Steps To build releases for a variety of platforms we use the excellent [xgo](https://github.com/techknowlogick/xgo) project. You will need Docker engine installed. These steps should be followed from the project root folder. These steps are one off to install the required build utilities. 1. Install the xgo Docker image. ``` docker pull techknowlogick/xgo:latest ``` 2. Install the command line helper tool. Ensure "$GOPATH/bin" is on your system path to access the executable. ``` go install src.techknowlogick.com/xgo@latest ``` These steps are run for each new release. 1. Update the CHANGELOG. 2. Add the CHANGELOG file and tag a commit. __Note__: In source control good semver suggests a "v" prefix on a version. It helps group release tags. ``` git add CHANGELOG.md git commit -m "Nakama 2.1.0 release." git tag -a v2.1.0 -m "v2.1.0" git push origin v2.1.0 master ``` 3. Execute the cross-compiled build helper. ``` xgo --targets=darwin/arm64,darwin/amd64,linux/amd64,linux/arm64,windows/amd64 --trimpath --ldflags "-s -w -X main.version=2.1.0 -X main.commitID=$(git rev-parse --short HEAD 2>/dev/null)" github.com/heroiclabs/nakama ``` This will build binaries for all target platforms supported officially by Heroic Labs. 4. Package up each release as a compressed bundle. ``` tar -czf "nakama--.tar.gz" nakama README.md LICENSE CHANGELOG.md ``` 5. Create a new draft release on GitHub and publish it with the compressed bundles. ## Build Nakama Image With the release generated we can create the official container image. These steps are one off to install the required build utilities. 1. Install Docker Desktop. 2. Create a new Docker context to create multi-platform builds. ``` docker context create builder ``` These steps are run for each new release. 1. Use an existing builder that supports multi-platform builds. ``` docker buildx create --use builder ``` 2. Build the container image and push to the container registry. ``` cd build docker buildx build . --platform linux/amd64,linux/arm64 --file ./Dockerfile --build-arg commit="$(git rev-parse --short HEAD 2>/dev/null)" --build-arg version=2.1.0 -t heroiclabs/nakama:2.1.0 -t heroiclabs/nakama:latest --push ``` ## Build Nakama Image (dSYM) With the release generated we can also create an official container image which includes debug symbols. 1. Use an existing builder that supports multi-platform builds. ``` docker buildx create --use builder ``` 2. Build the container image and push to the container registry. ``` cd build docker buildx build "$PWD" --platform linux/amd64,linux/arm64 --file ./Dockerfile.dsym --build-arg commit="$(git rev-parse --short HEAD 2>/dev/null)" --build-arg version=2.1.0 -t heroiclabs/nakama-dsym:2.1.0 -t heroiclabs/nakama-dsym:latest --push ``` ## Build Plugin Builder Image With the official release image generated we can create a container image to help with Go runtime development. 1. Use an existing builder that supports multi-platform builds. ``` docker buildx create --use builder ``` 2. Build the container image. ``` cd build/pluginbuilder docker buildx build "$PWD" --platform linux/amd64,linux/arm64 --file ./Dockerfile --build-arg commit="$(git rev-parse --short HEAD 2>/dev/null)" --build-arg version=2.1.0 -t heroiclabs/nakama-pluginbuilder:2.1.0 -t heroiclabs/nakama-pluginbuilder:latest --push ``` ## Build all images with provided script. 1. Use an existing builder that supports multi-platform builds. ``` docker buildx create --use builder ``` 2. Build the container image. ``` cd build ./build.sh ``` --- ### Overview (`sample_go_module/README.md`) # Nakama Go Runtime The game server includes support to develop native code in Go with the [plugin](https://golang.org/pkg/plugin/) package from the Go stdlib. It's used to enable compiled shared objects to be loaded by the game server at startup. The Go runtime support can be used to develop authoritative multiplayer match handlers, RPC functions, hook into messages processed by the server, and extend the server with any other custom logic. It offers the same capabilities as the [Lua runtime](https://heroiclabs.com/docs/runtime-code-basics/) support but has the advantage that any package from the Go ecosystem can be used. For more information and a discussion of the pros/cons with the Go runtime have a look at the [docs](https://heroiclabs.com/docs). ## Minimal example Here's the smallest example of a Go module written with the server runtime. ```go package main import ( "context" "database/sql" "github.com/heroiclabs/nakama-common/runtime" ) func InitModule(ctx context.Context, logger runtime.Logger, db *sql.DB, nk runtime.NakamaModule, initializer runtime.Initializer) error { logger.Info("module loaded") return nil } ``` ## Setup a project To setup your own project to build modules for the game server you can follow these steps. 1. Download and install the Go toolchain. It's recommended you follow the [official instructions](https://golang.org/doc/install). 2. Setup a folder for your own plugin code. ```bash mkdir -p "$HOME/plugin_code" cd "$HOME/plugin_code" ``` 3. Initialize the Go module for your plugin and add the nakama-common dependency. ```bash go mod init "plugin_code" go get -u "github.com/heroiclabs/nakama-common@v1.23.0" ``` ⚠️ __NOTE__: If you're working on Nakama's master branch drop the `@v1.23.0` from the above snippet. ⚠️ __NOTE__: The official Nakama v3.12.+ expects nakama-common v1.23.0 in order to run. If you use v1.22.0, older, or drop the version reference, you might get a `plugin was built with a different version of package` error while starting up the Nakama server. 4. Develop your plugin code (you can use the [minimal example](#minimal-example) as a starting point) and save it within your plugin project directory with the `.go` extension. ## Build & load process In a regular development cycle you will often recompile your plugin code and rerun the server. 1. Develop and compile your code. ```bash go build -buildmode=plugin -trimpath -o ./plugin_code.so ``` 2. Use `--runtime.path` flag when you start the Nakama server binary to load your built plugin. (Note: Also make sure you run the database). ```bash ./nakama --runtime.path "$HOME/plugin_code" ``` __TIP__: You can either build and run Nakama from source or you can download a prebuilt binary for your platform [here](https://github.com/heroiclabs/nakama/releases). For more information on how the server loads modules have a look at [these](https://heroiclabs.com/docs/runtime-code-basics/#load-modules) docs. For general instructions on how to run the server give [these](https://heroiclabs.com/docs/install-start-server/#start-nakama) docs a read. ### Docker builds It's often easiest to run the game server with Docker Compose. It will start the game server and database server together in the right sequence and wraps the process up into a single command. You'll need the Docker engine installed to use it. For Windows development and environments where you want to use our official Docker images to run your containers we provide a container image to help you build your code. 1. Use the Docker plugin helper container to compile your project (works for bash/PowerShell): ```bash cd "$HOME/plugin_code" # Your project folder. See instructions above. docker run --rm -w "/builder" -v "${PWD}:/builder" heroiclabs/nakama-pluginbuilder:3.12.0 build -buildmode=plugin -trimpath -o ./modules/plugin_code.so ``` In the command above we bind-mount your current folder into the container and use the Go toolchain inside it to run the build. The output artifacts are written back into your host filesystem. 2. Use our official Docker Compose [file](https://heroiclabs.com/docs/nakama/getting-started/install/docker/#running-nakama) to run all containers together and load in your custom module. __NOTE:__ You should copy the `.so` files generated in step 1. to the `/modules` folder of your Nakama source files and then run the command below from the Nakama root directory. ```bash docker-compose up ``` By default the server will be started and look in a folder relative to the current dir called "./modules" to load the plugins. __TIP__: Use the same version of your plugin builder image as used in the Docker Compose file for the server version. i.e. "heroiclabs/nakama:2.3.1" <> "heroiclabs/nakama-pluginbuilder:2.3.1", etc. ## Bigger Example Have a look in this repo for more example code on how to create and use various parts of the game server Go runtime support. The project implements a fully authoritative example of tic-tac-toe in Go, Lua and TS. https://github.com/heroiclabs/nakama-project-template --- ### Overview (`build/do-marketplace/README.md`) # Build Automation with Packer [Packer](https://www.packer.io/intro/index.html) is a tool for creating images from a single source configuration. Using this Packer template reduces the entire process of creating, configuring, validating, and snapshotting a build Droplet to a single command: ``` packer build marketplace-image.json ``` This Packer template is configured to create a snapshot with Nakama and CockroachDB. ## Usage To get started, you'll need to [install Packer](https://www.packer.io/intro/getting-started/install.html) and [create a DigitalOcean personal access token](https://www.digitalocean.com/docs/api/create-personal-access-token/) and set it to the `DIGITALOCEAN_API_TOKEN` environment variable. Running `packer build marketplace-image.json` without any other modifications will create a build Droplet configured with Nakama & CockroachDB, clean and verify it, then power it down and snapshot it. > ⚠️ The image validation script in `scripts/99-img_check.sh` is copied from the [top-level `scripts` directory](../scripts) in the [DigitalOcean Marketplace Partnners repository](https://github.com/digitalocean/marketplace-partners). The top-level location is the script's canonical source, so make sure you're using the latest version from there. This Packer template uses binaries for both Nakama and CockroachDB. The URL for each binary is set via variables at the top of the template. ## Configuration Details By using [Packer's DigitalOcean Builder](https://www.packer.io/docs/builders/digitalocean.html) to integrate with the [DigitalOcean API](https://developers.digitalocean.com/), this template fully automates Marketplace image creation. This template uses Packer's [file provisioner](https://www.packer.io/docs/provisioners/file.html) to upload complete directories to the Droplet. The contents of `files/var/` will be uploaded to `/var/`. Likewise, the contents of `files/etc/` will be uploaded to `/etc/`. One important thing to note about the file provisioner, from Packer's docs: > The destination directory must already exist. If you need to create it, use a shell provisioner just prior to the file provisioner in order to create the directory. If the destination directory does not exist, the file provisioner may succeed, but it will have undefined results. This template also uses Packer's [shell provisioner](https://www.packer.io/docs/provisioners/shell.html) to run scripts from the `/scripts` directory and install APT packages using an inline task. Learn more about using Packer in [the official Packer documentation](https://www.packer.io/docs/index.html). --- ### Overview (`internal/cronexpr/README.md`) Golang Cron expression parser ============================= Given a cron expression and a time stamp, you can get the next time stamp which satisfies the cron expression. In another project, I decided to use cron expression syntax to encode scheduling information. Thus this standalone library to parse and apply time stamps to cron expressions. The time-matching algorithm in this implementation is efficient, it avoids as much as possible to guess the next matching time stamp, a common technique seen in a number of implementations out there. There is also a companion command-line utility to evaluate cron time expressions: (which of course uses this library). Implementation -------------- The reference documentation for this implementation is found at , which I copy/pasted here (laziness!) with modifications where this implementation differs: Field name Mandatory? Allowed values Allowed special characters ---------- ---------- -------------- -------------------------- Seconds No 0-59 * / , - Minutes Yes 0-59 * / , - Hours Yes 0-23 * / , - Day of month Yes 1-31 * / , - L W Month Yes 1-12 or JAN-DEC * / , - Day of week Yes 0-6 or SUN-SAT * / , - L # Year No 1970–2099 * / , - #### Asterisk ( * ) The asterisk indicates that the cron expression matches for all values of the field. E.g., using an asterisk in the 4th field (month) indicates every month. #### Slash ( / ) Slashes describe increments of ranges. For example `3-59/15` in the minute field indicate the third minute of the hour and every 15 minutes thereafter. The form `*/...` is equivalent to the form "first-last/...", that is, an increment over the largest possible range of the field. #### Comma ( , ) Commas are used to separate items of a list. For example, using `MON,WED,FRI` in the 5th field (day of week) means Mondays, Wednesdays and Fridays. #### Hyphen ( - ) Hyphens define ranges. For example, 2000-2010 indicates every year between 2000 and 2010 AD, inclusive. #### L `L` stands for "last". When used in the day-of-week field, it allows you to specify constructs such as "the last Friday" (`5L`) of a given month. In the day-of-month field, it specifies the last day of the month. #### W The `W` character is allowed for the day-of-month field. This character is used to specify the business day (Monday-Friday) nearest the given day. As an example, if you were to specify `15W` as the value for the day-of-month field, the meaning is: "the nearest business day to the 15th of the month." So, if the 15th is a Saturday, the trigger fires on Friday the 14th. If the 15th is a Sunday, the trigger fires on Monday the 16th. If the 15th is a Tuesday, then it fires on Tuesday the 15th. However if you specify `1W` as the value for day-of-month, and the 1st is a Saturday, the trigger fires on Monday the 3rd, as it does not 'jump' over the boundary of a month's days. The `W` character can be specified only when the day-of-month is a single day, not a range or list of days. The `W` character can also be combined with `L`, i.e. `LW` to mean "the last business day of the month." #### Hash ( # ) `#` is allowed for the day-of-week field, and must be followed by a number between one and five. It allows you to specify constructs such as "the second Friday" of a given month. Predefined cron expressions --------------------------- (Copied from , with text modified according to this implementation) Entry Description Equivalent to @annually Run once a year at midnight in the morning of January 1 0 0 0 1 1 * * @yearly Run once a year at midnight in the morning of January 1 0 0 0 1 1 * * @monthly Run once a month at midnight in the morning of the first of the month 0 0 0 1 * * * @weekly Run once a week at midnight in the morning of Sunday 0 0 0 * * 0 * @daily Run once a day at midnight 0 0 0 * * * * @hourly Run once an hour at the beginning of the hour 0 0 * * * * * @reboot Not supported Other details ------------- * If only six fields are present, a `0` second field is prepended, that is, `* * * * * 2013` internally become `0 * * * * * 2013`. * If only five fields are present, a `0` second field is prepended and a wildcard year field is appended, that is, `* * * * Mon` internally become `0 * * * * Mon *`. * Domain for day-of-week field is [0-7] instead of [0-6], 7 being Sunday (like 0). This to comply with http://linux.die.net/man/5/crontab#. * As of now, the behavior of the code is undetermined if a malformed cron expression is supplied Install ------- go get github.com/gorhill/cronexpr Usage ----- Import the library: import "github.com/gorhill/cronexpr" import "time" Simplest way: nextTime := cronexpr.MustParse("0 0 29 2 *").Next(time.Now()) Assuming `time.Now()` is "2013-08-29 09:28:00", then `nextTime` will be "2016-02-29 00:00:00". You can keep the returned Expression pointer around if you want to reuse it: expr := cronexpr.MustParse("0 0 29 2 *") nextTime := expr.Next(time.Now()) ... nextTime = expr.Next(nextTime) Use `time.IsZero()` to find out whether a valid time was returned. For example, cronexpr.MustParse("* * * * * 1980").Next(time.Now()).IsZero() will return `true`, whereas cronexpr.MustParse("* * * * * 2050").Next(time.Now()).IsZero() will return `false` (as of 2013-08-29...) You may also query for `n` next time stamps: cronexpr.MustParse("0 0 29 2 *").NextN(time.Now(), 5) which returns a slice of time.Time objects, containing the following time stamps (as of 2013-08-30): 2016-02-29 00:00:00 2020-02-29 00:00:00 2024-02-29 00:00:00 2028-02-29 00:00:00 2032-02-29 00:00:00 The time zone of time values returned by `Next` and `NextN` is always the time zone of the time value passed as argument, unless a zero time value is returned. API --- License ------- License: pick the one which suits you best: - GPL v3 see - APL v2 see --- ### Overview (`internal/skiplist/README.md`) skiplist =============== reference from redis [zskiplist](https://github.com/antirez/redis) Usage =============== ~~~Go package main import ( "fmt" "github.com/gansidui/skiplist" "log" ) type User struct { score float64 id string } func (u *User) Less(other interface{}) bool { if u.score > other.(*User).score { return true } if u.score == other.(*User).score && len(u.id) > len(other.(*User).id) { return true } return false } func main() { us := make([]*User, 7) us[0] = &User{6.6, "hi"} us[1] = &User{4.4, "hello"} us[2] = &User{2.2, "world"} us[3] = &User{3.3, "go"} us[4] = &User{1.1, "skip"} us[5] = &User{2.2, "list"} us[6] = &User{3.3, "lang"} // insert sl := skiplist.New() for i := 0; i < len(us); i++ { sl.Insert(us[i]) } // traverse for e := sl.Front(); e != nil; e = e.Next() { fmt.Println(e.Value.(*User).id, "-->", e.Value.(*User).score) } fmt.Println() // rank rank1 := sl.GetRank(&User{2.2, "list"}) rank2 := sl.GetRank(&User{6.6, "hi"}) if rank1 != 6 || rank2 != 1 { log.Fatal() } if e := sl.GetElementByRank(2); e.Value.(*User).score != 4.4 || e.Value.(*User).id != "hello" { log.Fatal() } } /* output: hi --> 6.6 hello --> 4.4 lang --> 3.3 go --> 3.3 world --> 2.2 list --> 2.2 skip --> 1.1 */ ~~~ License =============== MIT --- ### Overview (`internal/cronexpr/cronexpr/README.md`) cronexpr: command-line utility ============================== A command-line utility written in Go to evaluate cron time expressions. It is based on the standalone Go library . ## Install go get github.com/gorhill/cronexpr go install github.com/gorhill/cronexpr ## Usage cronexpr [options] "{cron expression}" ## Options `-l`: Go-compliant time layout to use for outputting time value(s), see . Default is `"Mon, 02 Jan 2006 15:04:05 MST"` `-n`: Number of resulting time values to output. Default is 1. `-t`: Whole or partial RFC3339 time value (i.e. `2006-01-02T15:04:05Z07:00`) against which the cron expression is evaluated. Examples of valid values include (assuming EST time zone): `13` = 2013-01-01T00:00:00-05:00 `2013` = 2013-01-01T00:00:00-05:00 `2013-08` = 2013-08-01T00:00:00-05:00 `2013-08-31` = 2013-08-31T00:00:00-05:00 `2013-08-31T12` = 2013-08-31T12:00:00-05:00 `2013-08-31T12:40` = 2013-08-31T12:40:00-05:00 `2013-08-31T12:40:35` = 2013-08-31T12:40:35-05:00 `2013-08-31T12:40:35-10:00` = 2013-08-31T12:40:35-10:00 Default time is current time, and default time zone is local time zone. ## Examples #### Example 1 Midnight on December 31st of any year. Command: cronexpr -t="2013-08-31" -n=5 "0 0 31 12 *" Output (assuming computer is in EST time zone): # "0 0 31 12 *" + "2013-08-31T00:00:00-04:00" = Tue, 31 Dec 2013 00:00:00 EST Wed, 31 Dec 2014 00:00:00 EST Thu, 31 Dec 2015 00:00:00 EST Sat, 31 Dec 2016 00:00:00 EST Sun, 31 Dec 2017 00:00:00 EST #### Example 2 2pm on February 29th of any year. Command: cronexpr -t=2000 -n=10 "0 14 29 2 *" Output (assuming computer is in EST time zone): # "0 14 29 2 *" + "2000-01-01T00:00:00-05:00" = Tue, 29 Feb 2000 14:00:00 EST Sun, 29 Feb 2004 14:00:00 EST Fri, 29 Feb 2008 14:00:00 EST Wed, 29 Feb 2012 14:00:00 EST Mon, 29 Feb 2016 14:00:00 EST Sat, 29 Feb 2020 14:00:00 EST Thu, 29 Feb 2024 14:00:00 EST Tue, 29 Feb 2028 14:00:00 EST Sun, 29 Feb 2032 14:00:00 EST Fri, 29 Feb 2036 14:00:00 EST #### Example 3 12pm on the work day closest to the 15th of March and every three month thereafter. Command: cronexpr -t=2013-09-01 -n=5 "0 12 15W 3/3 *" Output (assuming computer is in EST time zone): # "0 12 15W 3/3 *" + "2013-09-01T00:00:00-04:00" = Mon, 16 Sep 2013 12:00:00 EDT Mon, 16 Dec 2013 12:00:00 EST Fri, 14 Mar 2014 12:00:00 EDT Mon, 16 Jun 2014 12:00:00 EDT Mon, 15 Sep 2014 12:00:00 EDT #### Example 4 Midnight on the fifth Saturday of any month (twist: not all months have a 5th specific day of week). Command: cronexpr -t=2013-09-02 -n 5 "0 0 * * 6#5" Output (assuming computer is in EST time zone): # "0 0 * * 6#5" + "2013-09-02T00:00:00-04:00" = Sat, 30 Nov 2013 00:00:00 EST Sat, 29 Mar 2014 00:00:00 EDT Sat, 31 May 2014 00:00:00 EDT Sat, 30 Aug 2014 00:00:00 EDT Sat, 29 Nov 2014 00:00:00 EST --- ### Overview (`vendor/go.uber.org/atomic/README.md`) # atomic [![GoDoc][doc-img]][doc] [![Build Status][ci-img]][ci] [![Coverage Status][cov-img]][cov] [![Go Report Card][reportcard-img]][reportcard] Simple wrappers for primitive types to enforce atomic access. ## Installation ```shell $ go get -u go.uber.org/atomic@v1 ``` ### Legacy Import Path As of v1.5.0, the import path `go.uber.org/atomic` is the only supported way of using this package. If you are using Go modules, this package will fail to compile with the legacy import path path `github.com/uber-go/atomic`. We recommend migrating your code to the new import path but if you're unable to do so, or if your dependencies are still using the old import path, you will have to add a `replace` directive to your `go.mod` file downgrading the legacy import path to an older version. ``` replace github.com/uber-go/atomic => github.com/uber-go/atomic v1.4.0 ``` You can do so automatically by running the following command. ```shell $ go mod edit -replace github.com/uber-go/atomic=github.com/uber-go/atomic@v1.4.0 ``` ## Usage The standard library's `sync/atomic` is powerful, but it's easy to forget which variables must be accessed atomically. `go.uber.org/atomic` preserves all the functionality of the standard library, but wraps the primitive types to provide a safer, more convenient API. ```go var atom atomic.Uint32 atom.Store(42) atom.Sub(2) atom.CAS(40, 11) ``` See the [documentation][doc] for a complete API specification. ## Development Status Stable. --- Released under the [MIT License](LICENSE.txt). [doc-img]: https://godoc.org/github.com/uber-go/atomic?status.svg [doc]: https://godoc.org/go.uber.org/atomic [ci-img]: https://github.com/uber-go/atomic/actions/workflows/go.yml/badge.svg [ci]: https://github.com/uber-go/atomic/actions/workflows/go.yml [cov-img]: https://codecov.io/gh/uber-go/atomic/branch/master/graph/badge.svg [cov]: https://codecov.io/gh/uber-go/atomic [reportcard-img]: https://goreportcard.com/badge/go.uber.org/atomic [reportcard]: https://goreportcard.com/report/go.uber.org/atomic --- ### Overview (`vendor/go.uber.org/multierr/README.md`) # multierr [![GoDoc][doc-img]][doc] [![Build Status][ci-img]][ci] [![Coverage Status][cov-img]][cov] `multierr` allows combining one or more Go `error`s together. ## Features - **Idiomatic**: multierr follows best practices in Go, and keeps your code idiomatic. - It keeps the underlying error type hidden, allowing you to deal in `error` values exclusively. - It provides APIs to safely append into an error from a `defer` statement. - **Performant**: multierr is optimized for performance: - It avoids allocations where possible. - It utilizes slice resizing semantics to optimize common cases like appending into the same error object from a loop. - **Interoperable**: multierr interoperates with the Go standard library's error APIs seamlessly: - The `errors.Is` and `errors.As` functions *just work*. - **Lightweight**: multierr comes with virtually no dependencies. ## Installation ```bash go get -u go.uber.org/multierr@latest ``` ## Status Stable: No breaking changes will be made before 2.0. ------------------------------------------------------------------------------- Released under the [MIT License]. [MIT License]: LICENSE.txt [doc-img]: https://pkg.go.dev/badge/go.uber.org/multierr [doc]: https://pkg.go.dev/go.uber.org/multierr [ci-img]: https://github.com/uber-go/multierr/actions/workflows/go.yml/badge.svg [cov-img]: https://codecov.io/gh/uber-go/multierr/branch/master/graph/badge.svg [ci]: https://github.com/uber-go/multierr/actions/workflows/go.yml [cov]: https://codecov.io/gh/uber-go/multierr --- ### Overview (`vendor/go.uber.org/zap/README.md`) # :zap: zap
Blazing fast, structured, leveled logging in Go. [![GoDoc][doc-img]][doc] [![Build Status][ci-img]][ci] [![Coverage Status][cov-img]][cov]
## Installation `go get -u go.uber.org/zap` Note that zap only supports the two most recent minor versions of Go. ## Quick Start In contexts where performance is nice, but not critical, use the `SugaredLogger`. It's 4-10x faster than other structured logging packages and includes both structured and `printf`-style APIs. ```go logger, _ := zap.NewProduction() defer logger.Sync() // flushes buffer, if any sugar := logger.Sugar() sugar.Infow("failed to fetch URL", // Structured context as loosely typed key-value pairs. "url", url, "attempt", 3, "backoff", time.Second, ) sugar.Infof("Failed to fetch URL: %s", url) ``` When performance and type safety are critical, use the `Logger`. It's even faster than the `SugaredLogger` and allocates far less, but it only supports structured logging. ```go logger, _ := zap.NewProduction() defer logger.Sync() logger.Info("failed to fetch URL", // Structured context as strongly typed Field values. zap.String("url", url), zap.Int("attempt", 3), zap.Duration("backoff", time.Second), ) ``` See the [documentation][doc] and [FAQ](FAQ.md) for more details. ## Performance For applications that log in the hot path, reflection-based serialization and string formatting are prohibitively expensive — they're CPU-intensive and make many small allocations. Put differently, using `encoding/json` and `fmt.Fprintf` to log tons of `interface{}`s makes your application slow. Zap takes a different approach. It includes a reflection-free, zero-allocation JSON encoder, and the base `Logger` strives to avoid serialization overhead and allocations wherever possible. By building the high-level `SugaredLogger` on that foundation, zap lets users *choose* when they need to count every allocation and when they'd prefer a more familiar, loosely typed API. As measured by its own [benchmarking suite][], not only is zap more performant than comparable structured logging packages — it's also faster than the standard library. Like all benchmarks, take these with a grain of salt.[1](#footnote-versions) Log a message and 10 fields: | Package | Time | Time % to zap | Objects Allocated | | :------ | :--: | :-----------: | :---------------: | | :zap: zap | 656 ns/op | +0% | 5 allocs/op | :zap: zap (sugared) | 935 ns/op | +43% | 10 allocs/op | zerolog | 380 ns/op | -42% | 1 allocs/op | go-kit | 2249 ns/op | +243% | 57 allocs/op | slog (LogAttrs) | 2479 ns/op | +278% | 40 allocs/op | slog | 2481 ns/op | +278% | 42 allocs/op | apex/log | 9591 ns/op | +1362% | 63 allocs/op | log15 | 11393 ns/op | +1637% | 75 allocs/op | logrus | 11654 ns/op | +1677% | 79 allocs/op Log a message with a logger that already has 10 fields of context: | Package | Time | Time % to zap | Objects Allocated | | :------ | :--: | :-----------: | :---------------: | | :zap: zap | 67 ns/op | +0% | 0 allocs/op | :zap: zap (sugared) | 84 ns/op | +25% | 1 allocs/op | zerolog | 35 ns/op | -48% | 0 allocs/op | slog | 193 ns/op | +188% | 0 allocs/op | slog (LogAttrs) | 200 ns/op | +199% | 0 allocs/op | go-kit | 2460 ns/op | +3572% | 56 allocs/op | log15 | 9038 ns/op | +13390% | 70 allocs/op | apex/log | 9068 ns/op | +13434% | 53 allocs/op | logrus | 10521 ns/op | +15603% | 68 allocs/op Log a static string, without any context or `printf`-style templating: | Package | Time | Time % to zap | Objects Allocated | | :------ | :--: | :-----------: | :---------------: | | :zap: zap | 63 ns/op | +0% | 0 allocs/op | :zap: zap (sugared) | 81 ns/op | +29% | 1 allocs/op | zerolog | 32 ns/op | -49% | 0 allocs/op | standard library | 124 ns/op | +97% | 1 allocs/op | slog | 196 ns/op | +211% | 0 allocs/op | slog (LogAttrs) | 200 ns/op | +217% | 0 allocs/op | go-kit | 213 ns/op | +238% | 9 allocs/op | apex/log | 771 ns/op | +1124% | 5 allocs/op | logrus | 1439 ns/op | +2184% | 23 allocs/op | log15 | 2069 ns/op | +3184% | 20 allocs/op ## Development Status: Stable All APIs are finalized, and no breaking changes will be made in the 1.x series of releases. Users of semver-aware dependency management systems should pin zap to `^1`. ## Contributing We encourage and support an active, healthy community of contributors — including you! Details are in the [contribution guide](CONTRIBUTING.md) and the [code of conduct](CODE_OF_CONDUCT.md). The zap maintainers keep an eye on issues and pull requests, but you can also report any negative conduct to oss-conduct@uber.com. That email list is a private, safe space; even the zap maintainers don't have access, so don't hesitate to hold us to a high standard.
Released under the [MIT License](LICENSE). 1 In particular, keep in mind that we may be benchmarking against slightly older versions of other packages. Versions are pinned in the [benchmarks/go.mod][] file. [↩](#anchor-versions) [doc-img]: https://pkg.go.dev/badge/go.uber.org/zap [doc]: https://pkg.go.dev/go.uber.org/zap [ci-img]: https://github.com/uber-go/zap/actions/workflows/go.yml/badge.svg [ci]: https://github.com/uber-go/zap/actions/workflows/go.yml [cov-img]: https://codecov.io/gh/uber-go/zap/branch/master/graph/badge.svg [cov]: https://codecov.io/gh/uber-go/zap [benchmarking suite]: https://github.com/uber-go/zap/tree/master/benchmarks [benchmarks/go.mod]: https://github.com/uber-go/zap/blob/master/benchmarks/go.mod --- ### Overview (`vendor/google.golang.org/grpc/README.md`) # gRPC-Go [][API] [](https://goreportcard.com/report/github.com/grpc/grpc-go) [](https://codecov.io/gh/grpc/grpc-go) The [Go][] implementation of [gRPC][]: A high performance, open source, general RPC framework that puts mobile and HTTP/2 first. For more information see the [Go gRPC docs][], or jump directly into the [quick start][]. ## Prerequisites - **[Go][]**: any one of the **two latest major** [releases][go-releases]. ## Installation Simply add the following import to your code, and then `go [build|run|test]` will automatically fetch the necessary dependencies: ```go import "google.golang.org/grpc" ``` > **Note:** If you are trying to access `grpc-go` from **China**, see the > [FAQ](#FAQ) below. ## Learn more - [Go gRPC docs][], which include a [quick start][] and [API reference][API] among other resources - [Low-level technical docs](Documentation) from this repository - [Performance benchmark][] - [Examples](examples) - [Contribution guidelines](CONTRIBUTING.md) ## FAQ ### I/O Timeout Errors The `golang.org` domain may be blocked from some countries. `go get` usually produces an error like the following when this happens: ```console $ go get -u google.golang.org/grpc package google.golang.org/grpc: unrecognized import path "google.golang.org/grpc" (https fetch: Get https://google.golang.org/grpc?go-get=1: dial tcp 216.239.37.1:443: i/o timeout) ``` To build Go code, there are several options: - Set up a VPN and access google.golang.org through that. - With Go module support: it is possible to use the `replace` feature of `go mod` to create aliases for golang.org packages. In your project's directory: ```sh go mod edit -replace=google.golang.org/grpc=github.com/grpc/grpc-go@latest go mod tidy go mod vendor go build -mod=vendor ``` Again, this will need to be done for all transitive dependencies hosted on golang.org as well. For details, refer to [golang/go issue #28652](https://github.com/golang/go/issues/28652). ### Compiling error, undefined: grpc.SupportPackageIsVersion Please update to the latest version of gRPC-Go using `go get google.golang.org/grpc`. ### How to turn on logging The default logger is controlled by environment variables. Turn everything on like this: ```console $ export GRPC_GO_LOG_VERBOSITY_LEVEL=99 $ export GRPC_GO_LOG_SEVERITY_LEVEL=info ``` ### The RPC failed with error `"code = Unavailable desc = transport is closing"` This error means the connection the RPC is using was closed, and there are many possible reasons, including: 1. mis-configured transport credentials, connection failed on handshaking 1. bytes disrupted, possibly by a proxy in between 1. server shutdown 1. Keepalive parameters caused connection shutdown, for example if you have configured your server to terminate connections regularly to [trigger DNS lookups](https://github.com/grpc/grpc-go/issues/3170#issuecomment-552517779). If this is the case, you may want to increase your [MaxConnectionAgeGrace](https://pkg.go.dev/google.golang.org/grpc/keepalive?tab=doc#ServerParameters), to allow longer RPC calls to finish. It can be tricky to debug this because the error happens on the client side but the root cause of the connection being closed is on the server side. Turn on logging on __both client and server__, and see if there are any transport errors. [API]: https://pkg.go.dev/google.golang.org/grpc [Go]: https://golang.org [Go module]: https://github.com/golang/go/wiki/Modules [gRPC]: https://grpc.io [Go gRPC docs]: https://grpc.io/docs/languages/go [Performance benchmark]: https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5180705743044608 [quick start]: https://grpc.io/docs/languages/go/quickstart [go-releases]: https://golang.org/doc/devel/release.html --- ### Overview (`vendor/gopkg.in/yaml.v3/README.md`) # YAML support for the Go language Introduction ------------ The yaml package enables Go programs to comfortably encode and decode YAML values. It was developed within [Canonical](https://www.canonical.com) as part of the [juju](https://juju.ubuntu.com) project, and is based on a pure Go port of the well-known [libyaml](http://pyyaml.org/wiki/LibYAML) C library to parse and generate YAML data quickly and reliably. Compatibility ------------- The yaml package supports most of YAML 1.2, but preserves some behavior from 1.1 for backwards compatibility. Specifically, as of v3 of the yaml package: - YAML 1.1 bools (_yes/no, on/off_) are supported as long as they are being decoded into a typed bool value. Otherwise they behave as a string. Booleans in YAML 1.2 are _true/false_ only. - Octals encode and decode as _0777_ per YAML 1.1, rather than _0o777_ as specified in YAML 1.2, because most parsers still use the old format. Octals in the _0o777_ format are supported though, so new files work. - Does not support base-60 floats. These are gone from YAML 1.2, and were actually never supported by this package as it's clearly a poor choice. and offers backwards compatibility with YAML 1.1 in some cases. 1.2, including support for anchors, tags, map merging, etc. Multi-document unmarshalling is not yet implemented, and base-60 floats from YAML 1.1 are purposefully not supported since they're a poor design and are gone in YAML 1.2. Installation and usage ---------------------- The import path for the package is *gopkg.in/yaml.v3*. To install it, run: go get gopkg.in/yaml.v3 API documentation ----------------- If opened in a browser, the import path itself leads to the API documentation: - [https://gopkg.in/yaml.v3](https://gopkg.in/yaml.v3) API stability ------------- The package API for yaml v3 will remain stable as described in [gopkg.in](https://gopkg.in). License ------- The yaml package is licensed under the MIT and Apache License 2.0 licenses. Please see the LICENSE file for details. Example ------- ```Go package main import ( "fmt" "log" "gopkg.in/yaml.v3" ) var data = ` a: Easy! b: c: 2 d: [3, 4] ` // Note: struct fields must be public in order for unmarshal to // correctly populate the data. type T struct { A string B struct { RenamedC int `yaml:"c"` D []int `yaml:",flow"` } } func main() { t := T{} err := yaml.Unmarshal([]byte(data), &t) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- t:\n%v\n\n", t) d, err := yaml.Marshal(&t) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- t dump:\n%s\n\n", string(d)) m := make(map[interface{}]interface{}) err = yaml.Unmarshal([]byte(data), &m) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- m:\n%v\n\n", m) d, err = yaml.Marshal(&m) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- m dump:\n%s\n\n", string(d)) } ``` This example will generate the following output: ``` --- t: {Easy! {2 [3 4]}} --- t dump: a: Easy! b: c: 2 d: [3, 4] --- m: map[a:Easy! b:map[c:2 d:[3 4]]] --- m dump: a: Easy! b: c: 2 d: - 3 - 4 ``` --- ### Overview (`vendor/github.com/RoaringBitmap/roaring/README.md`) # roaring [](https://godoc.org/github.com/RoaringBitmap/roaring) [](https://goreportcard.com/report/github.com/RoaringBitmap/roaring) ============= This is a go version of the Roaring bitmap data structure. Roaring bitmaps are used by several major systems such as [Apache Lucene][lucene] and derivative systems such as [Solr][solr] and [Elasticsearch][elasticsearch], [Apache Druid (Incubating)][druid], [LinkedIn Pinot][pinot], [Netflix Atlas][atlas], [Apache Spark][spark], [OpenSearchServer][opensearchserver], [anacrolix/torrent][anacrolix/torrent], [Whoosh][whoosh], [Redpanda](https://github.com/redpanda-data/redpanda), [Pilosa][pilosa], [Microsoft Visual Studio Team Services (VSTS)][vsts], and eBay's [Apache Kylin][kylin]. The YouTube SQL Engine, [Google Procella](https://research.google/pubs/pub48388/), uses Roaring bitmaps for indexing. [lucene]: https://lucene.apache.org/ [solr]: https://lucene.apache.org/solr/ [elasticsearch]: https://www.elastic.co/products/elasticsearch [druid]: https://druid.apache.org/ [spark]: https://spark.apache.org/ [opensearchserver]: http://www.opensearchserver.com [anacrolix/torrent]: https://github.com/anacrolix/torrent [whoosh]: https://bitbucket.org/mchaput/whoosh/wiki/Home [pilosa]: https://www.pilosa.com/ [kylin]: http://kylin.apache.org/ [pinot]: http://github.com/linkedin/pinot/wiki [vsts]: https://www.visualstudio.com/team-services/ [atlas]: https://github.com/Netflix/atlas Roaring bitmaps are found to work well in many important applications: > Use Roaring for bitmap compression whenever possible. Do not use other bitmap compression methods ([Wang et al., SIGMOD 2017](http://db.ucsd.edu/wp-content/uploads/2017/03/sidm338-wangA.pdf)) The ``roaring`` Go library is used by * [anacrolix/torrent] * [InfluxDB](https://www.influxdata.com) * [Pilosa](https://www.pilosa.com/) * [Bleve](http://www.blevesearch.com) * [Weaviate](https://github.com/weaviate/weaviate) * [lindb](https://github.com/lindb/lindb) * [Elasticell](https://github.com/deepfabric/elasticell) * [SourceGraph](https://github.com/sourcegraph/sourcegraph) * [M3](https://github.com/m3db/m3) * [trident](https://github.com/NetApp/trident) * [Husky](https://www.datadoghq.com/blog/engineering/introducing-husky/) * [FrostDB](https://github.com/polarsignals/frostdb) This library is used in production in several systems, it is part of the [Awesome Go collection](https://awesome-go.com). There are also [Java](https://github.com/RoaringBitmap/RoaringBitmap) and [C/C++](https://github.com/RoaringBitmap/CRoaring) versions. The Java, C, C++ and Go version are binary compatible: e.g, you can save bitmaps from a Java program and load them back in Go, and vice versa. We have a [format specification](https://github.com/RoaringBitmap/RoaringFormatSpec). This code is licensed under Apache License, Version 2.0 (ASL2.0). Copyright 2016-... by the authors. When should you use a bitmap? =================================== Sets are a fundamental abstraction in software. They can be implemented in various ways, as hash sets, as trees, and so forth. In databases and search engines, sets are often an integral part of indexes. For example, we may need to maintain a set of all documents or rows (represented by numerical identifier) that satisfy some property. Besides adding or removing elements from the set, we need fast functions to compute the intersection, the union, the difference between sets, and so on. To implement a set of integers, a particularly appealing strategy is the bitmap (also called bitset or bit vector). Using n bits, we can represent any set made of the integers from the range [0,n): the ith bit is set to one if integer i is present in the set. Commodity processors use words of W=32 or W=64 bits. By combining many such words, we can support large values of n. Intersections, unions and differences can then be implemented as bitwise AND, OR and ANDNOT operations. More complicated set functions can also be implemented as bitwise operations. When the bitset approach is applicable, it can be orders of magnitude faster than other possible implementation of a set (e.g., as a hash set) while using several times less memory. However, a bitset, even a compressed one is not always applicable. For example, if you have 1000 random-looking integers, then a simple array might be the best representation. We refer to this case as the "sparse" scenario. When should you use compressed bitmaps? =================================== An uncompressed BitSet can use a lot of memory. For example, if you take a BitSet and set the bit at position 1,000,000 to true and you have just over 100kB. That is over 100kB to store the position of one bit. This is wasteful even if you do not care about memory: suppose that you need to compute the intersection between this BitSet and another one that has a bit at position 1,000,001 to true, then you need to go through all these zeroes, whether you like it or not. That can become very wasteful. This being said, there are definitively cases where attempting to use compressed bitmaps is wasteful. For example, if you have a small universe size. E.g., your bitmaps represent sets of integers from [0,n) where n is small (e.g., n=64 or n=128). If you can use uncompressed BitSet and it does not blow up your memory usage, then compressed bitmaps are probably not useful to you. In fact, if you do not need compression, then a BitSet offers remarkable speed. The sparse scenario is another use case where compressed bitmaps should not be used. Keep in mind that random-looking data is usually not compressible. E.g., if you have a small set of 32-bit random integers, it is not mathematically possible to use far less than 32 bits per integer, and attempts at compression can be counterproductive. How does Roaring compares with the alternatives? ================================================== Most alternatives to Roaring are part of a larger family of compressed bitmaps that are run-length-encoded bitmaps. They identify long runs of 1s or 0s and they represent them with a marker word. If you have a local mix of 1s and 0, you use an uncompressed word. There are many formats in this family: * Oracle's BBC is an obsolete format at this point: though it may provide good compression, it is likely much slower than more recent alternatives due to excessive branching. * WAH is a patented variation on BBC that provides better performance. * Concise is a variation on the patented WAH. It some specific instances, it can compress much better than WAH (up to 2x better), but it is generally slower. * EWAH is both free of patent, and it is faster than all the above. On the downside, it does not compress quite as well. It is faster because it allows some form of "skipping" over uncompressed words. So though none of these formats are great at random access, EWAH is better than the alternatives. There is a big problem with these formats however that can hurt you badly in some cases: there is no random access. If you want to check whether a given value is present in the set, you have to start from the beginning and "uncompress" the whole thing. This means that if you want to intersect a big set with a large set, you still have to uncompress the whole big set in the worst case... Roaring solves this problem. It works in the following manner. It divides the data into chunks of 216 integers (e.g., [0, 216), [216, 2 x 216), ...). Within a chunk, it can use an uncompressed bitmap, a simple list of integers, or a list of runs. Whatever format it uses, they all allow you to check for the presence of any one value quickly (e.g., with a binary search). The net result is that Roaring can compute many operations much faster than run-length-encoded formats like WAH, EWAH, Concise... Maybe surprisingly, Roaring also generally offers better compression ratios. ### References - Daniel Lemire, Owen Kaser, Nathan Kurz, Luca Deri, Chris O'Hara, François Saint-Jacques, Gregory Ssi-Yan-Kai, Roaring Bitmaps: Implementation of an Optimized Software Library, Software: Practice and Experience 48 (4), 2018 [arXiv:1709.07821](https://arxiv.org/abs/1709.07821) - Samy Chambi, Daniel Lemire, Owen Kaser, Robert Godin, Better bitmap performance with Roaring bitmaps, Software: Practice and Experience 46 (5), 2016.[arXiv:1402.6407](http://arxiv.org/abs/1402.6407) This paper used data from http://lemire.me/data/realroaring2014.html - Daniel Lemire, Gregory Ssi-Yan-Kai, Owen Kaser, Consistently faster and smaller compressed bitmaps with Roaring, Software: Practice and Experience 46 (11), 2016. [arXiv:1603.06549](http://arxiv.org/abs/1603.06549) ### Dependencies Dependencies are fetched automatically by giving the `-t` flag to `go get`. they include - github.com/bits-and-blooms/bitset - github.com/mschoch/smat - github.com/glycerine/go-unsnap-stream - github.com/philhofer/fwd - github.com/jtolds/gls Note that the smat library requires Go 1.6 or better. #### Installation - go get -t github.com/RoaringBitmap/roaring ### Instructions for contributors Using bash or other common shells: ``` $ git clone git@github.com:RoaringBitmap/roaring.git $ export GO111MODULE=on $ go mod tidy $ go test -v ``` ### Example Here is a simplified but complete example: ```go package main import ( "fmt" "github.com/RoaringBitmap/roaring" "bytes" ) func main() { // example inspired by https://github.com/fzandona/goroar fmt.Println("==roaring==") rb1 := roaring.BitmapOf(1, 2, 3, 4, 5, 100, 1000) fmt.Println(rb1.String()) rb2 := roaring.BitmapOf(3, 4, 1000) fmt.Println(rb2.String()) rb3 := roaring.New() fmt.Println(rb3.String()) fmt.Println("Cardinality: ", rb1.GetCardinality()) fmt.Println("Contains 3? ", rb1.Contains(3)) rb1.And(rb2) rb3.Add(1) rb3.Add(5) rb3.Or(rb1) // computes union of the three bitmaps in parallel using 4 workers roaring.ParOr(4, rb1, rb2, rb3) // computes intersection of the three bitmaps in parallel using 4 workers roaring.ParAnd(4, rb1, rb2, rb3) // prints 1, 3, 4, 5, 1000 i := rb3.Iterator() for i.HasNext() { fmt.Println(i.Next()) } fmt.Println() // next we include an example of serialization buf := new(bytes.Buffer) rb1.WriteTo(buf) // we omit error handling newrb:= roaring.New() newrb.ReadFrom(buf) if rb1.Equals(newrb) { fmt.Println("I wrote the content to a byte stream and read it back.") } // you can iterate over bitmaps using ReverseIterator(), Iterator, ManyIterator() } ``` If you wish to use serialization and handle errors, you might want to consider the following sample of code: ```go rb := BitmapOf(1, 2, 3, 4, 5, 100, 1000) buf := new(bytes.Buffer) size,err:=rb.WriteTo(buf) if err != nil { t.Errorf("Failed writing") } newrb:= New() size,err=newrb.ReadFrom(buf) if err != nil { t.Errorf("Failed reading") } if ! rb.Equals(newrb) { t.Errorf("Cannot retrieve serialized version") } ``` Given N integers in [0,x), then the serialized size in bytes of a Roaring bitmap should never exceed this bound: `` 8 + 9 * ((long)x+65535)/65536 + 2 * N `` That is, given a fixed overhead for the universe size (x), Roaring bitmaps never use more than 2 bytes per integer. You can call ``BoundSerializedSizeInBytes`` for a more precise estimate. ### 64-bit Roaring By default, roaring is used to stored unsigned 32-bit integers. However, we also offer an extension dedicated to 64-bit integers. It supports roughly the same functions: ```go package main import ( "fmt" "github.com/RoaringBitmap/roaring/roaring64" "bytes" ) func main() { // example inspired by https://github.com/fzandona/goroar fmt.Println("==roaring64==") rb1 := roaring64.BitmapOf(1, 2, 3, 4, 5, 100, 1000) fmt.Println(rb1.String()) rb2 := roaring64.BitmapOf(3, 4, 1000) fmt.Println(rb2.String()) rb3 := roaring64.New() fmt.Println(rb3.String()) fmt.Println("Cardinality: ", rb1.GetCardinality()) fmt.Println("Contains 3? ", rb1.Contains(3)) rb1.And(rb2) rb3.Add(1) rb3.Add(5) rb3.Or(rb1) // prints 1, 3, 4, 5, 1000 i := rb3.Iterator() for i.HasNext() { fmt.Println(i.Next()) } fmt.Println() // next we include an example of serialization buf := new(bytes.Buffer) rb1.WriteTo(buf) // we omit error handling newrb:= roaring64.New() newrb.ReadFrom(buf) if rb1.Equals(newrb) { fmt.Println("I wrote the content to a byte stream and read it back.") } // you can iterate over bitmaps using ReverseIterator(), Iterator, ManyIterator() } ``` Only the 32-bit roaring format is standard and cross-operable between Java, C++, C and Go. There is no guarantee that the 64-bit versions are compatible. ### Documentation Current documentation is available at https://pkg.go.dev/github.com/RoaringBitmap/roaring and https://pkg.go.dev/github.com/RoaringBitmap/roaring/roaring64 ### Goroutine safety In general, it should not generally be considered safe to access the same bitmaps using different goroutines--they are left unsynchronized for performance. Should you want to access a Bitmap from more than one goroutine, you should provide synchronization. Typically this is done by using channels to pass the *Bitmap around (in Go style; so there is only ever one owner), or by using `sync.Mutex` to serialize operations on Bitmaps. ### Coverage We test our software. For a report on our test coverage, see https://coveralls.io/github/RoaringBitmap/roaring?branch=master ### Benchmark Type go test -bench Benchmark -run - To run benchmarks on [Real Roaring Datasets](https://github.com/RoaringBitmap/real-roaring-datasets) run the following: ```sh go get github.com/RoaringBitmap/real-roaring-datasets BENCH_REAL_DATA=1 go test -bench BenchmarkRealData -run - ``` ### Iterative use You can use roaring with gore: - go get -u github.com/motemen/gore - Make sure that ``$GOPATH/bin`` is in your ``$PATH``. - go get github.com/RoaringBitmap/roaring ```go $ gore gore version 0.2.6 :help for help gore> :import github.com/RoaringBitmap/roaring gore> x:=roaring.New() gore> x.Add(1) gore> x.String() "{1}" ``` ### Fuzzy testing You can help us test further the library with fuzzy testing: go get github.com/dvyukov/go-fuzz/go-fuzz go get github.com/dvyukov/go-fuzz/go-fuzz-build go test -tags=gofuzz -run=TestGenerateSmatCorpus go-fuzz-build github.com/RoaringBitmap/roaring go-fuzz -bin=./roaring-fuzz.zip -workdir=workdir/ -timeout=200 -func FuzzSmat Let it run, and if the # of crashers is > 0, check out the reports in the workdir where you should be able to find the panic goroutine stack traces. You may also replace `-func FuzzSmat` by `-func FuzzSerializationBuffer` or `-func FuzzSerializationStream`. ### Alternative in Go There is a Go version wrapping the C/C++ implementation https://github.com/RoaringBitmap/gocroaring For an alternative implementation in Go, see https://github.com/fzandona/goroar The two versions were written independently. ### Mailing list/discussion group https://groups.google.com/forum/#!forum/roaring-bitmaps --- ### Overview (`vendor/github.com/axiomhq/hyperloglog/README.md`) # HyperLogLog - an algorithm for approximating the number of distinct elements [](https://godoc.org/github.com/axiomhq/hyperloglog) [](https://goreportcard.com/report/github.com/axiomhq/hyperloglog) [](https://circleci.com/gh/axiomhq/hyperloglog/tree/master) An improved version of [HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog) for the count-distinct problem, approximating the number of distinct elements in a multiset. This implementation offers enhanced performance, flexibility, and simplicity while maintaining accuracy. ## Note on Implementation History The initial version of this work (tagged as v0.1.0) was based on ["Better with fewer bits: Improving the performance of cardinality estimation of large data streams - Qingjun Xiao, You Zhou, Shigang Chen"](https://www.cise.ufl.edu/~sgchen/Publications/XZC17.pdf). However, the current implementation has evolved significantly from this original basis, notably moving away from the tailcut method. ## Current Implementation The current implementation is based on the LogLog-Beta algorithm, as described in: ["LogLog-Beta and More: A New Algorithm for Cardinality Estimation Based on LogLog Counting"](https://arxiv.org/pdf/1612.02284) by Jason Qin, Denys Kim, and Yumei Tung (2016). Key features of the current implementation: * **Metro hash** used instead of xxhash * **Sparse representation** for lower cardinalities (like HyperLogLog++) * **LogLog-Beta** for dynamic bias correction across all cardinalities * **8-bit registers** for convenience and simplified implementation * **Order-independent insertions and merging** for consistent results regardless of data input order * **Removal of tailcut method** for a more straightforward approach * **Flexible precision** allowing for 2^4 to 2^18 registers This implementation is now more straightforward, efficient, and flexible, while remaining backwards compatible with previous versions. It provides a balance between precision, memory usage, speed, and ease of use. ## Precision and Memory Usage This implementation allows for creating HyperLogLog sketches with arbitrary precision between 2^4 and 2^18 registers. The memory usage scales with the number of registers: * Minimum (2^4 registers): 16 bytes * Default (2^14 registers): 16 KB * Maximum (2^18 registers): 256 KB Users can choose the precision that best fits their use case, balancing memory usage against estimation accuracy. ## Note A big thank you to Prof. Shigang Chen and his team at the University of Florida who are actively conducting research around "Big Network Data". ## Contributing Kindly check our [contributing guide](https://github.com/axiomhq/hyperloglog/blob/main/Contributing.md) on how to propose bugfixes and improvements, and submitting pull requests to the project ## License © Axiom, Inc., 2024 Distributed under MIT License (`The MIT License`). See [LICENSE](LICENSE) for more information. --- ### Overview (`vendor/github.com/bits-and-blooms/bitset/README.md`) # bitset *Go language library to map between non-negative integers and boolean values* [](https://github.com/bits-and-blooms/bitset/actions?query=workflow%3ATest) [](https://pkg.go.dev/github.com/bits-and-blooms/bitset?tab=doc) This library is part of the [awesome go collection](https://github.com/avelino/awesome-go). It is used in production by several important systems. ## Description Package bitset implements bitsets, a mapping between non-negative integers and boolean values. It should be more efficient than map[uint] bool. It provides methods for setting, clearing, flipping, and testing individual integers. But it also provides set intersection, union, difference, complement, and symmetric operations, as well as tests to check whether any, all, or no bits are set, and querying a bitset's current length and number of positive bits. BitSets are expanded to the size of the largest set bit; the memory allocation is approximately Max bits, where Max is the largest set bit. BitSets are never shrunk automatically, but `Shrink` and `Compact` methods are available. On creation, a hint can be given for the number of bits that will be used. Many of the methods, including Set, Clear, and Flip, return a BitSet pointer, which allows for chaining. ### Example use: ```go package main import ( "fmt" "math/rand" "github.com/bits-and-blooms/bitset" ) func main() { fmt.Printf("Hello from BitSet!\n") var b bitset.BitSet // play some Go Fish for i := 0; i < 100; i++ { card1 := uint(rand.Intn(52)) card2 := uint(rand.Intn(52)) b.Set(card1) if b.Test(card2) { fmt.Println("Go Fish!") } b.Clear(card1) } // Chaining b.Set(10).Set(11) for i, e := b.NextSet(0); e; i, e = b.NextSet(i + 1) { fmt.Println("The following bit is set:", i) } if b.Intersection(bitset.New(100).Set(10)).Count() == 1 { fmt.Println("Intersection works.") } else { fmt.Println("Intersection doesn't work???") } } ``` If you have Go 1.23 or better, you can iterate over the set bits like so: ```go for i := range b.EachSet() {} ``` Package documentation is at: https://pkg.go.dev/github.com/bits-and-blooms/bitset?tab=doc ## Serialization You may serialize a bitset safely and portably to a stream of bytes as follows: ```Go const length = 9585 const oneEvery = 97 bs := bitset.New(length) // Add some bits for i := uint(0); i < length; i += oneEvery { bs = bs.Set(i) } var buf bytes.Buffer n, err := bs.WriteTo(&buf) if err != nil { // failure } // Here n == buf.Len() ``` You can later deserialize the result as follows: ```Go // Read back from buf bs = bitset.New() n, err = bs.ReadFrom(&buf) if err != nil { // error } // n is the number of bytes read ``` The `ReadFrom` function attempts to read the data into the existing BitSet instance, to minimize memory allocations. *Performance tip*: When reading and writing to a file or a network connection, you may get better performance by wrapping your streams with `bufio` instances. E.g., ```Go f, err := os.Create("myfile") w := bufio.NewWriter(f) ``` ```Go f, err := os.Open("myfile") r := bufio.NewReader(f) ``` ## Memory Usage The memory usage of a bitset using `N` bits is at least `N/8` bytes. The number of bits in a bitset is at least as large as one plus the greatest bit index you have accessed. Thus it is possible to run out of memory while using a bitset. If you have lots of bits, you might prefer compressed bitsets, like the [Roaring bitmaps](https://roaringbitmap.org) and its [Go implementation](https://github.com/RoaringBitmap/roaring). The `roaring` library allows you to go back and forth between compressed Roaring bitmaps and the conventional bitset instances: ```Go mybitset := roaringbitmap.ToBitSet() newroaringbitmap := roaring.FromBitSet(mybitset) ``` ### Goroutine safety In general, it's not safe to access the same BitSet using different goroutines--they are unsynchronized for performance. Should you want to access a BitSet from more than one goroutine, you should provide synchronization. Typically this is done by using channels to pass the *BitSet around (in Go style; so there is only ever one owner), or by using `sync.Mutex` to serialize operations on BitSets. ## Installation ```bash go get github.com/bits-and-blooms/bitset ``` ## Users Databases & Storage - [milvus-io/milvus](https://github.com/milvus-io/milvus) — popular open-source vector database - [cubefs/cubefs](https://github.com/cubefs/cubefs) — CNCF distributed filesystem (used at JD.com) - [sourcenetwork/defradb](https://github.com/sourcenetwork/defradb) — decentralized document database - [siglens/siglens](https://github.com/siglens/siglens) — open-source observability/log analytics platform - [ByteStorage/FlyDB](https://github.com/ByteStorage/FlyDB) — high-performance KV store Search & Indexing - [blevesearch/bleve](https://github.com/blevesearch/bleve) — full-text search library for Go - [blevesearch/vellum](https://github.com/blevesearch/vellum) — FST library used internally by Bleve - [blugelabs/bluge](https://github.com/blugelabs/bluge) — modern Go search library (successor to Bleve) - [geange/lucene-go](https://github.com/geange/lucene-go) — Lucene port to Go Blockchain & Cryptography - [consensys/gnark](https://github.com/consensys/gnark) — zk-SNARK proving system - [consensys/gnark-crypto](https://github.com/consensys/gnark-crypto) — elliptic curve cryptography for ZK proofs - [NethermindEth/juno](https://github.com/NethermindEth/juno) — StarkNet full node - [onflow/cadence](https://github.com/onflow/cadence) — Flow blockchain smart contract language runtime - [cosmos/cosmos-sdk](https://github.com/cosmos/cosmos-sdk) — Cosmos blockchain SDK (slashing module) - [babylonlabs-io/babylon](https://github.com/babylonlabs-io/babylon) — Bitcoin staking protocol - [bnb-chain/greenfield](https://github.com/bnb-chain/greenfield) — BNB Chain decentralized storage Infrastructure & Networking - [moby/swarmkit](https://github.com/moby/swarmkit) — Docker Swarm orchestration (ID management) - [coredhcp/coredhcp](https://github.com/coredhcp/coredhcp) — pluggable DHCP server (bitmap IP allocation) - [Scalingo/sand](https://github.com/Scalingo/sand) — private networking for containers (IP allocation) - [AliyunContainerService/terway](https://github.com/AliyunContainerService/terway) — Alibaba Cloud K8s CNI - [pomerium/pomerium](https://github.com/pomerium/pomerium) — identity-aware access proxy Distributed Systems & Messaging - [apache/pulsar-client-go](https://github.com/apache/pulsar-client-go) — official Apache Pulsar Go client - [d7y.io/dragonfly](https://github.com/dragonflyoss/Dragonfly2) — CNCF P2P file distribution system - [RoaringBitmap/roaring](https://github.com/RoaringBitmap/roaring) — compressed bitmap library (interops with bitset) - [zhenghaoz/gorse](https://github.com/zhenghaoz/gorse) — Go recommendation engine Machine Learning / Vector Search - [fogfish/hnsw](https://github.com/fogfish/hnsw) — HNSW approximate nearest-neighbor graph - [semafind/semadb](https://github.com/semafind/semadb) — vector database Developer Tools & Security - [bearer/bearer](https://github.com/bearer/bearer) — security code scanner (AST traversal) - [zrepl/zrepl](https://github.com/zrepl/zrepl) — ZFS replication tool - [NVIDIA/dcgm-exporter](https://github.com/NVIDIA/dcgm-exporter) — NVIDIA GPU Prometheus exporter - [NVIDIA/go-dcgm](https://github.com/NVIDIA/go-dcgm) — NVIDIA datacenter GPU management bindings - [gohugoio/hugo](https://github.com/gohugoio/hugo) — the Hugo static site generator - [godoctor/godoctor](https://github.com/godoctor/godoctor) — Go refactoring tool (dataflow analysis) ## Contributing If you wish to contribute to this project, please branch and issue a pull request against master ("[GitHub Flow](https://guides.github.com/introduction/flow/)") ## Running all tests Before committing the code, please check if it passes tests, has adequate coverage, etc. ```bash go test go test -cover ``` ## Further reading

Mastering Programming: From Testing to Performance in Go

--- ### Overview (`vendor/github.com/blevesearch/go-porterstemmer/README.md`) # This fork... I'm maintaining this fork because the original author was not replying to issues or pull requests. For now I plan on maintaining this fork as necessary. ## Status [](https://travis-ci.org/blevesearch/go-porterstemmer) [](https://coveralls.io/r/blevesearch/go-porterstemmer?branch=HEAD) # Go Porter Stemmer A native Go clean room implementation of the Porter Stemming Algorithm. This algorithm is of interest to people doing Machine Learning or Natural Language Processing (NLP). This is NOT a port. This is a native Go implementation from the human-readable description of the algorithm. I've tried to make it (more) efficient by NOT internally using string's, but instead internally using []rune's and using the same (array) buffer used by the []rune slice (and sub-slices) at all steps of the algorithm. For Porter Stemmer algorithm, see: http://tartarus.org/martin/PorterStemmer/def.txt (URL #1) http://tartarus.org/martin/PorterStemmer/ (URL #2) # Departures Also, since when I initially implemented it, it failed the tests at... http://tartarus.org/martin/PorterStemmer/voc.txt (URL #3) http://tartarus.org/martin/PorterStemmer/output.txt (URL #4) ... after reading the human-readble text over and over again to try to figure out what the error I made was (and doing all sorts of things to debug it) I came to the conclusion that the some of these tests were wrong according to the human-readable description of the algorithm. This led me to wonder if maybe other people's code that was passing these tests had rules that were not in the human-readable description. Which led me to look at the source code here... http://tartarus.org/martin/PorterStemmer/c.txt (URL #5) ... When I looked there I noticed that there are some items marked as a "DEPARTURE", which differ from the original algorithm. (There are 2 of these.) I implemented these departures, and the tests at URL #3 and URL #4 all passed. ## Usage To use this Golang library, use with something like: package main import ( "fmt" "github.com/reiver/go-porterstemmer" ) func main() { word := "Waxes" stem := porterstemmer.StemString(word) fmt.Printf("The word [%s] has the stem [%s].\n", word, stem) } Alternatively, if you want to be a bit more efficient, use []rune slices instead, with code like: package main import ( "fmt" "github.com/reiver/go-porterstemmer" ) func main() { word := []rune("Waxes") stem := porterstemmer.Stem(word) fmt.Printf("The word [%s] has the stem [%s].\n", string(word), string(stem)) } Although NOTE that the above code may modify original slice (named "word" in the example) as a side effect, for efficiency reasons. And that the slice named "stem" in the example above may be a sub-slice of the slice named "word". Also alternatively, if you already know that your word is already lowercase (and you don't need this library to lowercase your word for you) you can instead use code like: package main import ( "fmt" "github.com/reiver/go-porterstemmer" ) func main() { word := []rune("waxes") stem := porterstemmer.StemWithoutLowerCasing(word) fmt.Printf("The word [%s] has the stem [%s].\n", string(word), string(stem)) } Again NOTE (like with the previous example) that the above code may modify original slice (named "word" in the example) as a side effect, for efficiency reasons. And that the slice named "stem" in the example above may be a sub-slice of the slice named "word". --- ### Overview (`vendor/github.com/blevesearch/mmap-go/README.md`) mmap-go ======= [](https://github.com/blevesearch/mmap-go/actions/workflows/tests.yml?query=event%3Apush+branch%3Amaster) [](https://pkg.go.dev/github.com/blevesearch/mmap-go) mmap-go is a portable mmap package for the [Go programming language](http://golang.org). Operating System Support ======================== This package is tested using GitHub Actions on Linux, macOS, and Windows. It should also work on other Unix-like platforms, but hasn't been tested with them. I'm interested to hear about the results. This package compiles for Plan 9 and WebAssembly, but its functions always return errors. Related functions such as `mprotect` and `mincore` aren't included. I haven't found a way to implement them on Windows without introducing significant complexity. If you're running on a Unix-like platform and really need these features, it should still be possible to implement them on top of this package via `syscall`. --- ### Overview (`vendor/github.com/blevesearch/segment/README.md`) # segment [](https://github.com/blevesearch/segment/actions?query=workflow%3ATests+event%3Apush+branch%3Amaster) A Go library for performing Unicode Text Segmentation as described in [Unicode Standard Annex #29](http://www.unicode.org/reports/tr29/) ## Features * Currently only segmentation at Word Boundaries is supported. ## License Apache License Version 2.0 ## Usage The functionality is exposed in two ways: 1. You can use a bufio.Scanner with the SplitWords implementation of SplitFunc. The SplitWords function will identify the appropriate word boundaries in the input text and the Scanner will return tokens at the appropriate place. scanner := bufio.NewScanner(...) scanner.Split(segment.SplitWords) for scanner.Scan() { tokenBytes := scanner.Bytes() } if err := scanner.Err(); err != nil { t.Fatal(err) } 2. Sometimes you would also like information returned about the type of token. To do this we have introduce a new type named Segmenter. It works just like Scanner but additionally a token type is returned. segmenter := segment.NewWordSegmenter(...) for segmenter.Segment() { tokenBytes := segmenter.Bytes()) tokenType := segmenter.Type() } if err := segmenter.Err(); err != nil { t.Fatal(err) } ## Choosing Implementation By default segment does NOT use the fastest runtime implementation. The reason is that it adds approximately 5s to compilation time and may require more than 1GB of ram on the machine performing compilation. However, you can choose to build with the fastest runtime implementation by passing the build tag as follows: -tags 'prod' ## Generating Code Several components in this package are generated. 1. Several Ragel rules files are generated from Unicode properties files. 2. Ragel machine is generated from the Ragel rules. 3. Test tables are generated from the Unicode test files. All of these can be generated by running: go generate ## Fuzzing There is support for fuzzing the segment library with [go-fuzz](https://github.com/dvyukov/go-fuzz). 1. Install go-fuzz if you haven't already: go get github.com/dvyukov/go-fuzz/go-fuzz go get github.com/dvyukov/go-fuzz/go-fuzz-build 2. Build the package with go-fuzz: go-fuzz-build github.com/blevesearch/segment 3. Convert the Unicode provided test cases into the initial corpus for go-fuzz: go test -v -run=TestGenerateWordSegmentFuzz -tags gofuzz_generate 4. Run go-fuzz: go-fuzz -bin=segment-fuzz.zip -workdir=workdir ## Status [](https://travis-ci.org/blevesearch/segment) [](https://coveralls.io/r/blevesearch/segment?branch=master) [](https://godoc.org/github.com/blevesearch/segment) --- ### Overview (`vendor/github.com/blevesearch/snowballstem/README.md`) # snowballstem This repository contains the Go stemmers generated by the [Snowball](https://github.com/snowballstem/snowball) project. They are maintained outside of the core bleve package so that they may be more easily be reused in other contexts. ## Usage All these stemmers export a single `Stem()` method which operates on a snowball `Env` structure. The `Env` structure maintains all state for the stemmer. A new `Env` is created to point at an initial string. After stemming, the results of the `Stem()` operation can be retrieved using the `Current()` method. The `Env` structure can be reused for subsequent calls by using the `SetCurrent()` method. ## Example ``` package main import ( "fmt" "github.com/blevesearch/snowballstem" "github.com/blevesearch/snowballstem/english" ) func main() { // words to stem words := []string{ "running", "jumping", } // build new environment env := snowballstem.NewEnv("") for _, word := range words { // set up environment for word env.SetCurrent(word) // invoke stemmer english.Stem(env) // print results fmt.Printf("%s stemmed to %s\n", word, env.Current()) } } ``` Produces Output: ``` $ ./snowtest running stemmed to run jumping stemmed to jump ``` ## Testing The test harness for these stemmers is hosted in the main [Snowball](https://github.com/snowballstem/snowball) repository. There are functional tests built around the separate [snowballstem-data](https://github.com/snowballstem/snowball-data) repository, and there is support for fuzz-testing the stemmers there as well. ## Generating the Stemmers ``` $ export SNOWBALL=/path/to/github.com/snowballstem/snowball/after/snowball/built $ go generate ``` ## Updated the Go Generate Commands A simple tool is provided to automate these from the snowball algorithms directory: ``` $ go run gengen.go /path/to/github.com/snowballstem/snowball/algorithms ``` --- ### Overview (`vendor/github.com/blevesearch/vellum/README.md`) # vellum [](https://github.com/blevesearch/vellum/actions/workflows/tests.yml?query=event%3Apush+branch%3Amaster) [](https://pkg.go.dev/github.com/blevesearch/vellum) [](https://goreportcard.com/report/github.com/blevesearch/vellum) [](https://opensource.org/licenses/Apache-2.0) A Go library implementing an FST (finite state transducer) capable of: - mapping between keys ([]byte) and a value (uint64) - enumerating keys in lexicographic order Some additional goals of this implementation: - bounded memory use while building the FST - streaming out FST data while building - mmap FST runtime to support very large FTSs (optional) ## Usage ### Building an FST To build an FST, create a new builder using the `New()` method. This method takes an `io.Writer` as an argument. As the FST is being built, data will be streamed to the writer as soon as possible. With this builder you **MUST** insert keys in lexicographic order. Inserting keys out of order will result in an error. After inserting the last key into the builder, you **MUST** call `Close()` on the builder. This will flush all remaining data to the underlying writer. In memory: ```go var buf bytes.Buffer builder, err := vellum.New(&buf, nil) if err != nil { log.Fatal(err) } ``` To disk: ```go f, err := os.Create("/tmp/vellum.fst") if err != nil { log.Fatal(err) } builder, err := vellum.New(f, nil) if err != nil { log.Fatal(err) } ``` **MUST** insert keys in lexicographic order: ```go err = builder.Insert([]byte("cat"), 1) if err != nil { log.Fatal(err) } err = builder.Insert([]byte("dog"), 2) if err != nil { log.Fatal(err) } err = builder.Insert([]byte("fish"), 3) if err != nil { log.Fatal(err) } err = builder.Close() if err != nil { log.Fatal(err) } ``` ### Using an FST After closing the builder, the data can be used to instantiate an FST. If the data was written to disk, you can use the `Open()` method to mmap the file. If the data is already in memory, or you wish to load/mmap the data yourself, you can instantiate the FST with the `Load()` method. Load in memory: ```go fst, err := vellum.Load(buf.Bytes()) if err != nil { log.Fatal(err) } ``` Open from disk: ```go fst, err := vellum.Open("/tmp/vellum.fst") if err != nil { log.Fatal(err) } ``` Get key/value: ```go val, exists, err = fst.Get([]byte("dog")) if err != nil { log.Fatal(err) } if exists { fmt.Printf("contains dog with val: %d\n", val) } else { fmt.Printf("does not contain dog") } ``` Iterate key/values: ```go itr, err := fst.Iterator(startKeyInclusive, endKeyExclusive) for err == nil { key, val := itr.Current() fmt.Printf("contains key: %s val: %d", key, val) err = itr.Next() } if err != nil { log.Fatal(err) } ``` ### How does the FST get built? A full example of the implementation is beyond the scope of this README, but let's consider a small example where we want to insert 3 key/value pairs. First we insert "are" with the value 4. Next, we insert "ate" with the value 2. Notice how the values associated with the transitions were adjusted so that by summing them while traversing we still get the expected value. At this point, we see that state 5 looks like state 3, and state 4 looks like state 2. But, we cannot yet combine them because future inserts could change this. Now, we insert "see" with value 3. Once it has been added, we now know that states 5 and 4 can longer change. Since they are identical to 3 and 2, we replace them. Again, we see that states 7 and 8 appear to be identical to 2 and 3. Having inserted our last key, we call `Close()` on the builder. Now, states 7 and 8 can safely be replaced with 2 and 3. For additional information, see the references at the bottom of this document. ### What does the serialized format look like? We've broken out a separate document on the [vellum disk format v1](docs/format.md). ### What if I want to use this on a system that doesn't have mmap? The mmap library itself is guarded with system/architecture build tags, but we've also added an additional build tag in vellum. If you'd like to Open() a file based representation of an FST, but not use mmap, you can build the library with the `nommap` build tag. NOTE: if you do this, the entire FST will be read into memory. ### Can I use this with Unicode strings? Yes, however this implementation is only aware of the byte representation you choose. In order to find matches, you must work with some canonical byte representation of the string. In the future, some encoding-aware traversals may be possible on top of the lower-level byte transitions. ### How did this library come to be? In my work on the [Bleve](https://github.com/blevesearch/bleve) project I became aware of the power of the FST for many search-related tasks. The obvious starting point for such a thing in Go was the [mafsa](https://github.com/smartystreets/mafsa) project. While working with mafsa I encountered some issues. First, it did not stream data to disk while building. Second, it chose to use a rune as the fundamental unit of transition in the FST, but I felt using a byte would be more powerful in the end. My hope is that higher-level encoding-aware traversals will be possible when necessary. Finally, as I reported bugs and submitted PRs I learned that the mafsa project was mainly a research project and no longer being maintained. I wanted to build something that could be used in production. As the project advanced more and more techniques from the [BurntSushi/fst](https://github.com/BurntSushi/fst) were adapted to our implementation. ### Are there tools to work with vellum files? Under the cmd/vellum subdirectory, there's a command-line tool which features subcommands that can allow you to create, inspect and query vellum files. ### How can I generate a state transition diagram from a vellum file? The vellum command-line tool has a "dot" subcommand that can emit graphviz dot output data from an input vellum file. The dot file can in turn be converted into an image using graphviz tools. Example... ```shell vellum dot myFile.vellum > output.dot dot -Tpng output.dot -o output.png ``` ## Related Work Much credit goes to two existing projects: - [mafsa](https://github.com/smartystreets/mafsa) - [BurntSushi/fst](https://github.com/BurntSushi/fst) Most of the original implementation here started with my digging into the internals of mafsa. As the implementation progressed, I continued to borrow ideas/approaches from the BurntSushi/fst library as well. For a great introduction to this topic, please read the blog post [Index 1,600,000,000 Keys with Automata and Rust](http://blog.burntsushi.net/transducers/) --- ### Overview (`vendor/github.com/blugelabs/bluge/README.md`) # Bluge [](https://pkg.go.dev/github.com/blugelabs/bluge) [](https://github.com/blugelabs/bluge/actions?query=workflow%3ATests+event%3Apush+branch%3Amaster) [](https://github.com/blugelabs/bluge/actions?query=workflow%3ALint+event%3Apush+branch%3Amaster) modern text indexing in go - [blugelabs.com](https://www.blugelabs.com/) ## Features * Supported field types: * Text, Numeric, Date, Geo Point * Supported query types: * Term, Phrase, Match, Match Phrase, Prefix * Conjunction, Disjunction, Boolean * Numeric Range, Date Range * BM25 Similarity/Scoring with pluggable interfaces * Search result match highlighting * Extendable Aggregations: * Bucketing * Terms * Numeric Range * Date Range * Metrics * Min/Max/Count/Sum * Avg/Weighted Avg * Cardinality Estimation ([HyperLogLog++](https://github.com/axiomhq/hyperloglog)) * Quantile Approximation ([T-Digest](https://github.com/caio/go-tdigest)) ## Indexing ```go config := bluge.DefaultConfig(path) writer, err := bluge.OpenWriter(config) if err != nil { log.Fatalf("error opening writer: %v", err) } defer writer.Close() doc := bluge.NewDocument("example"). AddField(bluge.NewTextField("name", "bluge")) err = writer.Update(doc.ID(), doc) if err != nil { log.Fatalf("error updating document: %v", err) } ``` ## Querying ```go reader, err := writer.Reader() if err != nil { log.Fatalf("error getting index reader: %v", err) } defer reader.Close() query := bluge.NewMatchQuery("bluge").SetField("name") request := bluge.NewTopNSearch(10, query). WithStandardAggregations() documentMatchIterator, err := reader.Search(context.Background(), request) if err != nil { log.Fatalf("error executing search: %v", err) } match, err := documentMatchIterator.Next() for err == nil && match != nil { err = match.VisitStoredFields(func(field string, value []byte) bool { if field == "_id" { fmt.Printf("match: %s\n", string(value)) } return true }) if err != nil { log.Fatalf("error loading stored fields: %v", err) } match, err = documentMatchIterator.Next() } if err != nil { log.Fatalf("error iterator document matches: %v", err) } ``` ## Repobeats ## License Apache License Version 2.0 --- ### Overview (`vendor/github.com/blugelabs/bluge_segment_api/README.md`) # Bluge Segment API [](https://pkg.go.dev/github.com/blugelabs/bluge_segment_api) [](https://github.com/blugelabs/bluge_segment_api/actions?query=workflow%3ATests+event%3Apush+branch%3Amaster) [](https://github.com/blugelabs/bluge_segment_api/actions?query=workflow%3ALint+event%3Apush+branch%3Amaster) These interfaces define the relationship between Bluge and it's segment file formats. The expectation is that these interfaces can be well-defined, and evolve slowly. By doing so, Bluge and the file formats themselves can evolve more quickly, and yet remain compatible. --- ### Overview (`vendor/github.com/blugelabs/ice/README.md`) # ice [](https://pkg.go.dev/github.com/blugelabs/ice) [](https://github.com/blugelabs/ice/actions?query=workflow%3ATests+event%3Apush+branch%3Amaster) [](https://github.com/blugelabs/ice/actions?query=workflow%3ALint+event%3Apush+branch%3Amaster) The file is written in the reverse order that we typically access data. This helps us write in one pass since later sections of the file require file offsets of things we've already written. Current usage: - crc-32 bytes and version are in fixed position at the end of the file - reading remainder of footer could be version specific - remainder of footer gives us: - 3 important offsets (docValue, fields index and stored data index) - 2 important values (number of docs and chunk factor) - field data is processed once and memoized onto the heap so that we never have to go back to disk for it - access to stored data by doc number means first navigating to the stored data index, then accessing a fixed position offset into that slice, which gives us the actual address of the data. the first bytes of that section tell us the size of data so that we know where it ends. - access to all other indexed data follows the following pattern: - first know the field name -> convert to id - next navigate to term dictionary for that field - some operations stop here and do dictionary ops - next use dictionary to navigate to posting list for a specific term - walk posting list - if necessary, walk posting details as we go - if location info is desired, consult location bitmap to see if it is there ## stored fields section - for each document - preparation phase: - produce a slice of metadata bytes and data bytes - produce these slices in field id order - field value is appended to the data slice - metadata slice is varint encoded with the following values for each field value - field id (uint16) - field value start offset in uncompressed data slice (uint64) - field value length (uint64) - compress the data slice using snappy - file writing phase: - remember the start offset for this document - write out meta data length (varint uint64) - write out compressed data length (varint uint64) - write out the metadata bytes - write out the compressed data bytes ## stored fields idx - for each document - write start offset (remembered from previous section) of stored data (big endian uint64) With this index and a known document number, we have direct access to all the stored field data. ## posting details (freq/norm) section - for each posting list - produce a slice containing multiple consecutive chunks (each chunk is varint stream) - produce a slice remembering offsets of where each chunk starts - preparation phase: - for each hit in the posting list - if this hit is in next chunk close out encoding of last chunk and record offset start of next - encode term frequency (uint64) - encode norm factor (float32) - similarity specific implementation - file writing phase: - remember start position for this posting list details - write out number of chunks that follow (varint uint64) - write out length of each chunk (each a varint uint64) - write out the byte slice containing all the chunk data If you know the doc number you're interested in, this format lets you jump to the correct chunk (docNum/chunkFactor) directly and then seek within that chunk until you find it. ## posting details (location) section - for each posting list - produce a slice containing multiple consecutive chunks (each chunk is varint stream) - produce a slice remembering offsets of where each chunk starts - preparation phase: - for each hit in the posting list - if this hit is in next chunk close out encoding of last chunk and record offset start of next - encode field (uint16) - encode field pos (uint64) - encode field start (uint64) - encode field end (uint64) - file writing phase: - remember start position for this posting list details - write out number of chunks that follow (varint uint64) - write out length of each chunk (each a varint uint64) - write out the byte slice containing all the chunk data If you know the doc number you're interested in, this format lets you jump to the correct chunk (docNum/chunkFactor) directly and then seek within that chunk until you find it. ## postings list section - for each posting list - preparation phase: - encode roaring bitmap posting list to bytes (so we know the length) - file writing phase: - remember the start position for this posting list - write freq/norm details offset (remembered from previous, as varint uint64) - write location details offset (remembered from previous, as varint uint64) - write length of encoded roaring bitmap - write the serialized roaring bitmap data ## dictionary - for each field - preparation phase: - encode vellum FST with dictionary data pointing to file offset of posting list (remembered from previous) - file writing phase: - remember the start position of this persistDictionary - write length of vellum data (varint uint64) - write out vellum data ## fields section - for each field - file writing phase: - remember start offset for each field - write dictionary address (remembered from previous) (varint uint64) - write length of field name (varint uint64) - write field name bytes ## fields idx - for each field - file writing phase: - write big endian uint64 of start offset for each field NOTE: currently we don't know or record the length of this fields index. Instead we rely on the fact that we know it immediately precedes a footer of known size. ## fields DocValue - for each field - preparation phase: - produce a slice containing multiple consecutive chunks, where each chunk is composed of a meta section followed by compressed columnar field data - produce a slice remembering the length of each chunk - file writing phase: - remember the start position of this first field DocValue offset in the footer - write out number of chunks that follow (varint uint64) - write out length of each chunk (each a varint uint64) - write out the byte slice containing all the chunk data NOTE: currently the meta header inside each chunk gives clue to the location offsets and size of the data pertaining to a given docID and any read operation leverage that meta information to extract the document specific data from the file. ## footer - file writing phase - write number of docs (big endian uint64) - write stored field index location (big endian uint64) - write field index location (big endian uint64) - write field docValue location (big endian uint64) - write out chunk factor (big endian uint32) - write out version (big endian uint32) - write out file CRC of everything preceding this (big endian uint32) --- # ice file format diagrams ## Legend ### Sections |========| | | section |========| ### Fixed-size fields |--------| |----| |--| |-| | | uint64 | | uint32 | | uint16 | | uint8 |--------| |----| |--| |-| ### Varints |~~~~~~~~| | | varint(up to uint64) |~~~~~~~~| ### Arbitrary-length fields |--------...---| | | arbitrary-length field (string, vellum, roaring bitmap) |--------...---| ### Chunked data [--------] [ ] [--------] ## Overview Footer section describes the configuration of particular ice file. The format of footer is version-dependent, so it is necessary to check `V` field before the parsing. |==================================================| | Stored Fields | |==================================================| |-----> | Stored Fields Index | | |==================================================| | | Dictionaries + Postings + DocValues | | |==================================================| | |---> | DocValues Index | | | |==================================================| | | | Fields | | | |==================================================| | | |-> | Fields Index | | | | |========|========|========|========|====|====|====| | | | | D# | SF | F | FDV | CF | V | CC | (Footer) | | | |========|====|===|====|===|====|===|====|====|====| | | | | | | |-+-+-----------------| | | | |--------------------------| | |-------------------------------------| D#. Number of Docs. SF. Stored Fields Index Offset. F. Field Index Offset. FDV. Field DocValue Offset. CF. Chunk Factor. V. Version. CC. CRC32. ## Stored Fields Stored Fields Index is `D#` consecutive 64-bit unsigned integers - offsets, where relevant Stored Fields Data records are located. 0 [SF] [SF + D# * 8] | Stored Fields | Stored Fields Index | |================================|==================================| | | | | |--------------------| ||--------|--------|. . .|--------|| | |-> | Stored Fields Data | || 0 | 1 | | D# - 1 || | | |--------------------| ||--------|----|---|. . .|--------|| | | | | | |===|============================|==============|===================| | | |-------------------------------------------| Stored Fields Data is an arbitrary size record, which consists of metadata and [Snappy](https://github.com/golang/snappy)-compressed data. Stored Fields Data |~~~~~~~~|~~~~~~~~|~~~~~~~~...~~~~~~~~|~~~~~~~~...~~~~~~~~| | MDS | CDS | MD | CD | |~~~~~~~~|~~~~~~~~|~~~~~~~~...~~~~~~~~|~~~~~~~~...~~~~~~~~| MDS. Metadata size. CDS. Compressed data size. MD. Metadata. CD. Snappy-compressed data. ## Fields Fields Index section located between addresses `F` and `len(file) - len(footer)` and consist of `uint64` values (`F1`, `F2`, ...) which are offsets to records in Fields section. We have `F# = (len(file) - len(footer) - F) / sizeof(uint64)` fields. (...) [F] [F + F#] | Fields | Fields Index. | |==================================================|================================| | | | | |~~~~~~~~|~~~~~~~~|---...---|~~~~~~~~|~~~~~~~~|||--------|--------|...|--------|| ||->| Dict | Length | Name | # Docs |Tot Freq||| 0 | 1 | | F# - 1 || || |~~~~~~~~|~~~~~~~~|---...---|~~~~~~~~|~~~~~~~~|||--------|----|---|...|--------|| || | | | ||=================================================|==============|=================| | | |----------------------------------------------------------------| ## Dictionaries + Postings Each of fields has its own dictionary, encoded in [Vellum](https://github.com/couchbase/vellum) format. Dictionary consists of pairs `(term, offset)`, where `offset` indicates the position of postings (list of documents) for this particular term. |================================================================|- Dictionaries + | | Postings + | | DocValues | Freq/Norm (chunked) | | [~~~~~~|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~] | | |->[ Freq | Norm (float32 under varint) ] | | | [~~~~~~|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~] | | | | | |------------------------------------------------------------| | | Location Details (chunked) | | | [~~~~~~|~~~~~|~~~~~~~|~~~~~] | | | |->[ Size | Pos | Start | End ] | | | | [~~~~~~|~~~~~|~~~~~~~|~~~~~] | | | | | | | |----------------------| | | | Postings List | | | | |~~~~~~~~|~~~~~|~~|~~~~~~~~|-----------...--| | | | |->| F/N | LD | Length | ROARING BITMAP | | | | | |~~~~~|~~|~~~~~~~~|~~~~~~~~|-----------...--| | | | | |----------------------------------------------| | | |--------------------------------------| | | Dictionary | | | |~~~~~~~~|--------------------------|-...-| | | |->| Length | VELLUM DATA : (TERM -> OFFSET) | | | | |~~~~~~~~|----------------------------...-| | | | | |======|=========================================================|- DocValues Index | | | |======|=========================================================|- Fields | | | | |~~~~|~~~|~~~~~~~~|---...---| | | | Dict | Length | Name | | | |~~~~~~~~|~~~~~~~~|---...---| | | | |================================================================| ## DocValues DocValues Index is `F#` pairs of varints, one pair per field. Each pair of varints indicates start and end point of DocValues slice. |================================================================| | |------...--| | | |->| DocValues |<-| | | | |------...--| | | |==|=================|===========================================|- DocValues Index ||~|~~~~~~~~~|~~~~~~~|~~| |~~~~~~~~~~~~~~|~~~~~~~~~~~~|| || DV1 START | DV1 STOP | . . . . . | DV(F#) START | DV(F#) END || ||~~~~~~~~~~~|~~~~~~~~~~| |~~~~~~~~~~~~~~|~~~~~~~~~~~~|| |================================================================| DocValues is chunked Snappy-compressed values for each document and field. [~~~~~~~~~~~~~~~|~~~~~~|~~~~~~~~~|-...-|~~~~~~|~~~~~~~~~|--------------------...-] [ Doc# in Chunk | Doc1 | Offset1 | ... | DocN | OffsetN | SNAPPY COMPRESSED DATA ] [~~~~~~~~~~~~~~~|~~~~~~|~~~~~~~~~|-...-|~~~~~~|~~~~~~~~~|--------------------...-] Last 16 bytes are description of chunks. |~~~~~~~~~~~~...~|----------------|----------------| | Chunk Sizes | Chunk Size Arr | Chunk# | |~~~~~~~~~~~~...~|----------------|----------------| --- ### Overview (`vendor/github.com/blugelabs/query_string/README.md`) # query_string An adaptation of the class Bleve query string functionality for Bluge. --- ### Overview (`vendor/github.com/caio/go-tdigest/README.md`) # T-Digest A fast map-reduce and parallel streaming friendly data-structure for accurate quantile approximation. This package provides an implementation of Ted Dunning's t-digest data structure in Go. [](http://godoc.org/github.com/caio/go-tdigest) [](https://goreportcard.com/report/github.com/caio/go-tdigest) ## Project Status This project is actively maintained. We are happy to collaborate on features and issues if/when they arrive. ## Installation Our releases are tagged and signed following the [Semantic Versioning][semver] scheme. If you are using a dependency manager such as [dep][], the recommended way to is go about your business normally: go get github.com/caio/go-tdigest Otherwise we recommend to use the following so that you don't risk breaking your build because of an API change: go get gopkg.in/caio/go-tdigest.v2 [semver]: http://semver.org/ [dep]: https://github.com/golang/dep ## Example Usage ```go package main import ( "fmt" "math/rand" "github.com/caio/go-tdigest" ) func main() { // Analogue to tdigest.New(tdigest.Compression(100)) t, _ := tdigest.New() for i := 0; i < 10000; i++ { // Analogue to t.AddWeighted(rand.Float64(), 1) t.Add(rand.Float64()) } fmt.Printf("p(.5) = %.6f\n", t.Quantile(0.5)) fmt.Printf("CDF(Quantile(.5)) = %.6f\n", t.CDF(t.Quantile(0.5))) } ``` ## Configuration You can configure your digest upon creation with options documented at [options.go](options.go). Example: ```go // Construct a digest with compression=200 and its own // (thread-unsafe) RNG seeded with 0xCA10: digest, _ := tdigest.New( tdigest.Compression(200), tdigest.LocalRandomNumberGenerator(0xCA10), ) ``` ## Porting Existing Code to the v2 API It's very easy to migrate to the new API: - Replace `tdigest.New(100)` with `tdigest.New()` - Replace `tdigest.New(number)` with `tdigest.New(tdigest.Compression(number))` - Replace `Add(x,1)` with `Add(x)` - Replace `Add(x, weight)` with `AddWeighted(x, weight)` - Remove any use of `tdigest.Len()` (or [open an issue][issues]) [issues]: https://github.com/caio/go-tdigest/issues/new ## References This is a port of the [reference][1] implementation with some ideas borrowed from the [python version][2]. If you wanna get a quick grasp of how it works and why it's useful, [this video and companion article is pretty helpful][3]. [1]: https://github.com/tdunning/t-digest [2]: https://github.com/CamDavidsonPilon/tdigest [3]: https://www.mapr.com/blog/better-anomaly-detection-t-digest-whiteboard-walkthrough --- ### Overview (`vendor/github.com/dgryski/dgoogauth/README.md`) This is a Go implementation of the Google Authenticator library. [](https://godoc.org/github.com/dgryski/dgoogauth) [](https://travis-ci.org/dgryski/dgoogauth) Copyright (c) 2012 Damian Gryski This code is licensed under the Apache License, version 2.0 It implements the one-time-password algorithms specified in: * RFC 4226 (HOTP: An HMAC-Based One-Time Password Algorithm) * RFC 6238 (TOTP: Time-Based One-Time Password Algorithm) You can learn more about the Google Authenticator library at its project page: * https://github.com/google/google-authenticator --- ### Overview (`vendor/github.com/dop251/goja/README.md`) goja ==== ECMAScript 5.1(+) implementation in Go. [](https://pkg.go.dev/github.com/dop251/goja) Goja is an implementation of ECMAScript 5.1 in pure Go with emphasis on standard compliance and performance. This project was largely inspired by [otto](https://github.com/robertkrimen/otto). The minimum required Go version is 1.25. Features -------- * Full ECMAScript 5.1 support (including regex and strict mode). * Passes nearly all [tc39 tests](https://github.com/tc39/test262) for the features implemented so far. The goal is to pass all of them. See .tc39_test262_checkout.sh for the latest working commit id. * Capable of running Babel, Typescript compiler and pretty much anything written in ES5. * Sourcemaps. * Most of ES6 functionality, still work in progress, see https://github.com/dop251/goja/milestone/1?closed=1 Known incompatibilities and caveats ----------------------------------- ### JSON `JSON.parse()` uses the standard Go library which operates in UTF-8. Therefore, it cannot correctly parse broken UTF-16 surrogate pairs, for example: ```javascript JSON.parse(`"\\uD800"`).charCodeAt(0).toString(16) // returns "fffd" instead of "d800" ``` ### Date Conversion from calendar date to epoch timestamp uses the standard Go library which uses `int`, rather than `float` as per ECMAScript specification. This means if you pass arguments that overflow int to the `Date()` constructor or if there is an integer overflow, the result will be incorrect, for example: ```javascript Date.UTC(1970, 0, 1, 80063993375, 29, 1, -288230376151711740) // returns 29256 instead of 29312 ``` FAQ --- ### How fast is it? Although it's faster than many scripting language implementations in Go I have seen (for example it's 6-7 times faster than otto on average) it is not a replacement for V8 or SpiderMonkey or any other general-purpose JavaScript engine. You can find some benchmarks [here](https://github.com/dop251/goja/issues/2). ### Why would I want to use it over a V8 wrapper? It greatly depends on your usage scenario. If most of the work is done in javascript (for example crypto or any other heavy calculations) you are definitely better off with V8. If you need a scripting language that drives an engine written in Go so that you need to make frequent calls between Go and javascript passing complex data structures then the cgo overhead may outweigh the benefits of having a faster javascript engine. Because it's written in pure Go there are no cgo dependencies, it's very easy to build and it should run on any platform supported by Go. It gives you a much better control over execution environment so can be useful for research. ### Is it goroutine-safe? No. An instance of goja.Runtime can only be used by a single goroutine at a time. You can create as many instances of Runtime as you like but it's not possible to pass object values between runtimes. ### Where is setTimeout()/setInterval()? setTimeout() and setInterval() are common functions to provide concurrent execution in ECMAScript environments, but the two functions are not part of the ECMAScript standard. Browsers and NodeJS just happen to provide similar, but not identical, functions. The hosting application need to control the environment for concurrent execution, e.g. an event loop, and supply the functionality to script code. There is a [separate project](https://github.com/dop251/goja_nodejs) aimed at providing some NodeJS functionality, and it includes an event loop. ### Can you implement (feature X from ES6 or higher)? I will be adding features in their dependency order and as quickly as time permits. Please do not ask for ETAs. Features that are open in the [milestone](https://github.com/dop251/goja/milestone/1) are either in progress or will be worked on next. The ongoing work is done in separate feature branches which are merged into master when appropriate. Every commit in these branches represents a relatively stable state (i.e. it compiles and passes all enabled tc39 tests), however because the version of tc39 tests I use is quite old, it may be not as well tested as the ES5.1 functionality. Because there are (usually) no major breaking changes between ECMAScript revisions it should not break your existing code. You are encouraged to give it a try and report any bugs found. Please do not submit fixes though without discussing it first, as the code could be changed in the meantime. ### How do I contribute? Before submitting a pull request please make sure that: - You followed ECMA standard as close as possible. If adding a new feature make sure you've read the specification, do not just base it on a couple of examples that work fine. - Your change does not have a significant negative impact on performance (unless it's a bugfix and it's unavoidable) - It passes all relevant tc39 tests. Current Status -------------- * There should be no breaking changes in the API, however it may be extended. * Some of the AnnexB functionality is missing. Basic Example ------------- Run JavaScript and get the result value. ```go vm := goja.New() v, err := vm.RunString("2 + 2") if err != nil { panic(err) } if num := v.Export().(int64); num != 4 { panic(num) } ``` Passing Values to JS -------------------- Any Go value can be passed to JS using Runtime.ToValue() method. See the method's [documentation](https://pkg.go.dev/github.com/dop251/goja#Runtime.ToValue) for more details. Exporting Values from JS ------------------------ A JS value can be exported into its default Go representation using Value.Export() method. Alternatively it can be exported into a specific Go variable using [Runtime.ExportTo()](https://pkg.go.dev/github.com/dop251/goja#Runtime.ExportTo) method. Within a single export operation the same Object will be represented by the same Go value (either the same map, slice or a pointer to the same struct). This includes circular objects and makes it possible to export them. Calling JS functions from Go ---------------------------- There are 2 approaches: - Using [AssertFunction()](https://pkg.go.dev/github.com/dop251/goja#AssertFunction): ```go const SCRIPT = ` function sum(a, b) { return +a + b; } ` vm := goja.New() _, err := vm.RunString(SCRIPT) if err != nil { panic(err) } sum, ok := goja.AssertFunction(vm.Get("sum")) if !ok { panic("Not a function") } res, err := sum(goja.Undefined(), vm.ToValue(40), vm.ToValue(2)) if err != nil { panic(err) } fmt.Println(res) // Output: 42 ``` - Using [Runtime.ExportTo()](https://pkg.go.dev/github.com/dop251/goja#Runtime.ExportTo): ```go const SCRIPT = ` function sum(a, b) { return +a + b; } ` vm := goja.New() _, err := vm.RunString(SCRIPT) if err != nil { panic(err) } var sum func(int, int) int err = vm.ExportTo(vm.Get("sum"), &sum) if err != nil { panic(err) } fmt.Println(sum(40, 2)) // note, _this_ value in the function will be undefined. // Output: 42 ``` The first one is more low level and allows specifying _this_ value, whereas the second one makes the function look like a normal Go function. Mapping struct field and method names ------------------------------------- By default, the names are passed through as is which means they are capitalised. This does not match the standard JavaScript naming convention, so if you need to make your JS code look more natural or if you are dealing with a 3rd party library, you can use a [FieldNameMapper](https://pkg.go.dev/github.com/dop251/goja#FieldNameMapper): ```go vm := goja.New() vm.SetFieldNameMapper(TagFieldNameMapper("json", true)) type S struct { Field int `json:"field"` } vm.Set("s", S{Field: 42}) res, _ := vm.RunString(`s.field`) // without the mapper it would have been s.Field fmt.Println(res.Export()) // Output: 42 ``` There are two standard mappers: [TagFieldNameMapper](https://pkg.go.dev/github.com/dop251/goja#TagFieldNameMapper) and [UncapFieldNameMapper](https://pkg.go.dev/github.com/dop251/goja#UncapFieldNameMapper), or you can use your own implementation. Native Constructors ------------------- In order to implement a constructor function in Go use `func (goja.ConstructorCall) *goja.Object`. See [Runtime.ToValue()](https://pkg.go.dev/github.com/dop251/goja#Runtime.ToValue) documentation for more details. Regular Expressions ------------------- Goja uses the embedded Go regexp library where possible, otherwise it falls back to [regexp2](https://github.com/dlclark/regexp2). Exceptions ---------- Any exception thrown in JavaScript is returned as an error of type *Exception. It is possible to extract the value thrown by using the Value() method: ```go vm := goja.New() _, err := vm.RunString(` throw("Test"); `) if jserr, ok := err.(*Exception); ok { if jserr.Value().Export() != "Test" { panic("wrong value") } } else { panic("wrong type") } ``` If a native Go function panics with a Value, it is thrown as a Javascript exception (and therefore can be caught): ```go var vm *Runtime func Test() { panic(vm.ToValue("Error")) } vm = goja.New() vm.Set("Test", Test) _, err := vm.RunString(` try { Test(); } catch(e) { if (e !== "Error") { throw e; } } `) if err != nil { panic(err) } ``` Interrupting ------------ ```go func TestInterrupt(t *testing.T) { const SCRIPT = ` var i = 0; for (;;) { i++; } ` vm := goja.New() time.AfterFunc(200 * time.Millisecond, func() { vm.Interrupt("halt") }) _, err := vm.RunString(SCRIPT) if err == nil { t.Fatal("Err is nil") } // err is of type *InterruptError and its Value() method returns whatever has been passed to vm.Interrupt() } ``` NodeJS Compatibility -------------------- There is a [separate project](https://github.com/dop251/goja_nodejs) aimed at providing some of the NodeJS functionality. --- ### Overview (`vendor/github.com/felixge/httpsnoop/README.md`) # httpsnoop Package httpsnoop provides an easy way to capture http related metrics (i.e. response time, bytes written, and http status code) from your application's http.Handlers. Doing this requires non-trivial wrapping of the http.ResponseWriter interface, which is also exposed for users interested in a more low-level API. [](https://pkg.go.dev/github.com/felixge/httpsnoop) [](https://github.com/felixge/httpsnoop/actions/workflows/main.yaml) ## Usage Example ```go // myH is your app's http handler, perhaps a http.ServeMux or similar. var myH http.Handler // wrappedH wraps myH in order to log every request. wrappedH := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { m := httpsnoop.CaptureMetrics(myH, w, r) log.Printf( "%s %s (code=%d dt=%s written=%d)", r.Method, r.URL, m.Code, m.Duration, m.Written, ) }) http.ListenAndServe(":8080", wrappedH) ``` ## Why this package exists Instrumenting an application's http.Handler is surprisingly difficult. However if you google for e.g. "capture ResponseWriter status code" you'll find lots of advise and code examples that suggest it to be a fairly trivial undertaking. Unfortunately everything I've seen so far has a high chance of breaking your application. The main problem is that a `http.ResponseWriter` often implements additional interfaces such as `http.Flusher`, `http.CloseNotifier`, `http.Hijacker`, `http.Pusher`, and `io.ReaderFrom`. So the naive approach of just wrapping `http.ResponseWriter` in your own struct that also implements the `http.ResponseWriter` interface will hide the additional interfaces mentioned above. This has a high change of introducing subtle bugs into any non-trivial application. Another approach I've seen people take is to return a struct that implements all of the interfaces above. However, that's also problematic, because it's difficult to fake some of these interfaces behaviors when the underlying `http.ResponseWriter` doesn't have an implementation. It's also dangerous, because an application may choose to operate differently, merely because it detects the presence of these additional interfaces. This package solves this problem by checking which additional interfaces a `http.ResponseWriter` implements, returning a wrapped version implementing the exact same set of interfaces. Additionally this package properly handles edge cases such as `WriteHeader` not being called, or called more than once, as well as concurrent calls to `http.ResponseWriter` methods, and even calls happening after the wrapped `ServeHTTP` has already returned. Unfortunately this package is not perfect either. It's possible that it is still missing some interfaces provided by the go core (let me know if you find one), and it won't work for applications adding their own interfaces into the mix. You can however use `httpsnoop.Unwrap(w)` to access the underlying `http.ResponseWriter` and type-assert the result to its other interfaces. However, hopefully the explanation above has sufficiently scared you of rolling your own solution to this problem. httpsnoop may still break your application, but at least it tries to avoid it as much as possible. Anyway, the real problem here is that smuggling additional interfaces inside `http.ResponseWriter` is a problematic design choice, but it probably goes as deep as the Go language specification itself. But that's okay, I still prefer Go over the alternatives ;). ## Performance ``` BenchmarkBaseline-8 20000 94912 ns/op BenchmarkCaptureMetrics-8 20000 95461 ns/op ``` As you can see, using `CaptureMetrics` on a vanilla http.Handler introduces an overhead of ~500 ns per http request on my machine. However, the margin of error appears to be larger than that, therefor it should be reasonable to assume that the overhead introduced by `CaptureMetrics` is absolutely negligible. ## License MIT --- ### Overview (`vendor/github.com/go-sourcemap/sourcemap/README.md`) # Source maps consumer for Golang [](https://pkg.go.dev/github.com/go-sourcemap/sourcemap) > This package is brought to you by :star: [**uptrace/uptrace**](https://github.com/uptrace/uptrace). > Uptrace is an open-source APM tool that supports distributed tracing, metrics, and logs. You can > use it to monitor applications and set up automatic alerts to receive notifications via email, > Slack, Telegram, and others. ## Installation Install: ```shell go get -u github.com/go-sourcemap/sourcemap ``` ## Quickstart ```go func ExampleParse() { mapURL := "http://code.jquery.com/jquery-2.0.3.min.map" resp, err := http.Get(mapURL) if err != nil { panic(err) } defer resp.Body.Close() b, err := ioutil.ReadAll(resp.Body) if err != nil { panic(err) } smap, err := sourcemap.Parse(mapURL, b) if err != nil { panic(err) } line, column := 5, 6789 file, fn, line, col, ok := smap.Source(line, column) fmt.Println(file, fn, line, col, ok) // Output: http://code.jquery.com/jquery-2.0.3.js apply 4360 27 true } ``` --- ### Overview (`vendor/github.com/gorilla/handlers/README.md`) # gorilla/handlers [](https://codecov.io/github/gorilla/handlers) [](https://godoc.org/github.com/gorilla/handlers) [](https://sourcegraph.com/github.com/gorilla/handlers?badge) Package handlers is a collection of handlers (aka "HTTP middleware") for use with Go's `net/http` package (or any framework supporting `http.Handler`), including: * [**LoggingHandler**](https://godoc.org/github.com/gorilla/handlers#LoggingHandler) for logging HTTP requests in the Apache [Common Log Format](http://httpd.apache.org/docs/2.2/logs.html#common). * [**CombinedLoggingHandler**](https://godoc.org/github.com/gorilla/handlers#CombinedLoggingHandler) for logging HTTP requests in the Apache [Combined Log Format](http://httpd.apache.org/docs/2.2/logs.html#combined) commonly used by both Apache and nginx. * [**CompressHandler**](https://godoc.org/github.com/gorilla/handlers#CompressHandler) for gzipping responses. * [**ContentTypeHandler**](https://godoc.org/github.com/gorilla/handlers#ContentTypeHandler) for validating requests against a list of accepted content types. * [**MethodHandler**](https://godoc.org/github.com/gorilla/handlers#MethodHandler) for matching HTTP methods against handlers in a `map[string]http.Handler` * [**ProxyHeaders**](https://godoc.org/github.com/gorilla/handlers#ProxyHeaders) for populating `r.RemoteAddr` and `r.URL.Scheme` based on the `X-Forwarded-For`, `X-Real-IP`, `X-Forwarded-Proto` and RFC7239 `Forwarded` headers when running a Go server behind a HTTP reverse proxy. * [**CanonicalHost**](https://godoc.org/github.com/gorilla/handlers#CanonicalHost) for re-directing to the preferred host when handling multiple domains (i.e. multiple CNAME aliases). * [**RecoveryHandler**](https://godoc.org/github.com/gorilla/handlers#RecoveryHandler) for recovering from unexpected panics. Other handlers are documented [on the Gorilla website](https://www.gorillatoolkit.org/pkg/handlers). ## Example A simple example using `handlers.LoggingHandler` and `handlers.CompressHandler`: ```go import ( "net/http" "github.com/gorilla/handlers" ) func main() { r := http.NewServeMux() // Only log requests to our admin dashboard to stdout r.Handle("/admin", handlers.LoggingHandler(os.Stdout, http.HandlerFunc(ShowAdminDashboard))) r.HandleFunc("/", ShowIndex) // Wrap our server with our gzip handler to gzip compress all responses. http.ListenAndServe(":8000", handlers.CompressHandler(r)) } ``` ## License BSD licensed. See the included LICENSE file for details. --- ### Overview (`vendor/github.com/gorilla/mux/README.md`) # gorilla/mux [](https://codecov.io/github/gorilla/mux) [](https://godoc.org/github.com/gorilla/mux) [](https://sourcegraph.com/github.com/gorilla/mux?badge) Package `gorilla/mux` implements a request router and dispatcher for matching incoming requests to their respective handler. The name mux stands for "HTTP request multiplexer". Like the standard `http.ServeMux`, `mux.Router` matches incoming requests against a list of registered routes and calls a handler for the route that matches the URL or other conditions. The main features are: * It implements the `http.Handler` interface so it is compatible with the standard `http.ServeMux`. * Requests can be matched based on URL host, path, path prefix, schemes, header and query values, HTTP methods or using custom matchers. * URL hosts, paths and query values can have variables with an optional regular expression. * Registered URLs can be built, or "reversed", which helps maintaining references to resources. * Routes can be used as subrouters: nested routes are only tested if the parent route matches. This is useful to define groups of routes that share common conditions like a host, a path prefix or other repeated attributes. As a bonus, this optimizes request matching. --- * [Install](#install) * [Examples](#examples) * [Matching Routes](#matching-routes) * [Static Files](#static-files) * [Serving Single Page Applications](#serving-single-page-applications) (e.g. React, Vue, Ember.js, etc.) * [Registered URLs](#registered-urls) * [Walking Routes](#walking-routes) * [Graceful Shutdown](#graceful-shutdown) * [Middleware](#middleware) * [Handling CORS Requests](#handling-cors-requests) * [Testing Handlers](#testing-handlers) * [Full Example](#full-example) --- ## Install With a [correctly configured](https://golang.org/doc/install#testing) Go toolchain: ```sh go get -u github.com/gorilla/mux ``` ## Examples Let's start registering a couple of URL paths and handlers: ```go func main() { r := mux.NewRouter() r.HandleFunc("/", HomeHandler) r.HandleFunc("/products", ProductsHandler) r.HandleFunc("/articles", ArticlesHandler) http.Handle("/", r) } ``` Here we register three routes mapping URL paths to handlers. This is equivalent to how `http.HandleFunc()` works: if an incoming request URL matches one of the paths, the corresponding handler is called passing (`http.ResponseWriter`, `*http.Request`) as parameters. Paths can have variables. They are defined using the format `{name}` or `{name:pattern}`. If a regular expression pattern is not defined, the matched variable will be anything until the next slash. For example: ```go r := mux.NewRouter() r.HandleFunc("/products/{key}", ProductHandler) r.HandleFunc("/articles/{category}/", ArticlesCategoryHandler) r.HandleFunc("/articles/{category}/{id:[0-9]+}", ArticleHandler) ``` The names are used to create a map of route variables which can be retrieved calling `mux.Vars()`: ```go func ArticlesCategoryHandler(w http.ResponseWriter, r *http.Request) { vars := mux.Vars(r) w.WriteHeader(http.StatusOK) fmt.Fprintf(w, "Category: %v\n", vars["category"]) } ``` And this is all you need to know about the basic usage. More advanced options are explained below. ### Matching Routes Routes can also be restricted to a domain or subdomain. Just define a host pattern to be matched. They can also have variables: ```go r := mux.NewRouter() // Only matches if domain is "www.example.com". r.Host("www.example.com") // Matches a dynamic subdomain. r.Host("{subdomain:[a-z]+}.example.com") ``` There are several other matchers that can be added. To match path prefixes: ```go r.PathPrefix("/products/") ``` ...or HTTP methods: ```go r.Methods("GET", "POST") ``` ...or URL schemes: ```go r.Schemes("https") ``` ...or header values: ```go r.Headers("X-Requested-With", "XMLHttpRequest") ``` ...or query values: ```go r.Queries("key", "value") ``` ...or to use a custom matcher function: ```go r.MatcherFunc(func(r *http.Request, rm *RouteMatch) bool { return r.ProtoMajor == 0 }) ``` ...and finally, it is possible to combine several matchers in a single route: ```go r.HandleFunc("/products", ProductsHandler). Host("www.example.com"). Methods("GET"). Schemes("http") ``` Routes are tested in the order they were added to the router. If two routes match, the first one wins: ```go r := mux.NewRouter() r.HandleFunc("/specific", specificHandler) r.PathPrefix("/").Handler(catchAllHandler) ``` Setting the same matching conditions again and again can be boring, so we have a way to group several routes that share the same requirements. We call it "subrouting". For example, let's say we have several URLs that should only match when the host is `www.example.com`. Create a route for that host and get a "subrouter" from it: ```go r := mux.NewRouter() s := r.Host("www.example.com").Subrouter() ``` Then register routes in the subrouter: ```go s.HandleFunc("/products/", ProductsHandler) s.HandleFunc("/products/{key}", ProductHandler) s.HandleFunc("/articles/{category}/{id:[0-9]+}", ArticleHandler) ``` The three URL paths we registered above will only be tested if the domain is `www.example.com`, because the subrouter is tested first. This is not only convenient, but also optimizes request matching. You can create subrouters combining any attribute matchers accepted by a route. Subrouters can be used to create domain or path "namespaces": you define subrouters in a central place and then parts of the app can register its paths relatively to a given subrouter. There's one more thing about subroutes. When a subrouter has a path prefix, the inner routes use it as base for their paths: ```go r := mux.NewRouter() s := r.PathPrefix("/products").Subrouter() // "/products/" s.HandleFunc("/", ProductsHandler) // "/products/{key}/" s.HandleFunc("/{key}/", ProductHandler) // "/products/{key}/details" s.HandleFunc("/{key}/details", ProductDetailsHandler) ``` ### Static Files Note that the path provided to `PathPrefix()` represents a "wildcard": calling `PathPrefix("/static/").Handler(...)` means that the handler will be passed any request that matches "/static/\*". This makes it easy to serve static files with mux: ```go func main() { var dir string flag.StringVar(&dir, "dir", ".", "the directory to serve files from. Defaults to the current dir") flag.Parse() r := mux.NewRouter() // This will serve files under http://localhost:8000/static/ r.PathPrefix("/static/").Handler(http.StripPrefix("/static/", http.FileServer(http.Dir(dir)))) srv := &http.Server{ Handler: r, Addr: "127.0.0.1:8000", // Good practice: enforce timeouts for servers you create! WriteTimeout: 15 * time.Second, ReadTimeout: 15 * time.Second, } log.Fatal(srv.ListenAndServe()) } ``` ### Serving Single Page Applications Most of the time it makes sense to serve your SPA on a separate web server from your API, but sometimes it's desirable to serve them both from one place. It's possible to write a simple handler for serving your SPA (for use with React Router's [BrowserRouter](https://reacttraining.com/react-router/web/api/BrowserRouter) for example), and leverage mux's powerful routing for your API endpoints. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Registered URLs Now let's see how to build registered URLs. Routes can be named. All routes that define a name can have their URLs built, or "reversed". We define a name calling `Name()` on a route. For example: ```go r := mux.NewRouter() r.HandleFunc("/articles/{category}/{id:[0-9]+}", ArticleHandler). Name("article") ``` To build a URL, get the route and call the `URL()` method, passing a sequence of key/value pairs for the route variables. For the previous route, we would do: ```go url, err := r.Get("article").URL("category", "technology", "id", "42") ``` ...and the result will be a `url.URL` with the following path: ``` "/articles/technology/42" ``` This also works for host and query value variables: ```go r := mux.NewRouter() r.Host("{subdomain}.example.com"). Path("/articles/{category}/{id:[0-9]+}"). Queries("filter", "{filter}"). HandlerFunc(ArticleHandler). Name("article") // url.String() will be "http://news.example.com/articles/technology/42?filter=gorilla" url, err := r.Get("article").URL("subdomain", "news", "category", "technology", "id", "42", "filter", "gorilla") ``` All variables defined in the route are required, and their values must conform to the corresponding patterns. These requirements guarantee that a generated URL will always match a registered route -- the only exception is for explicitly defined "build-only" routes which never match. Regex support also exists for matching Headers within a route. For example, we could do: ```go r.HeadersRegexp("Content-Type", "application/(text|json)") ``` ...and the route will match both requests with a Content-Type of `application/json` as well as `application/text` There's also a way to build only the URL host or path for a route: use the methods `URLHost()` or `URLPath()` instead. For the previous route, we would do: ```go // "http://news.example.com/" host, err := r.Get("article").URLHost("subdomain", "news") // "/articles/technology/42" path, err := r.Get("article").URLPath("category", "technology", "id", "42") ``` And if you use subrouters, host and path defined separately can be built as well: ```go r := mux.NewRouter() s := r.Host("{subdomain}.example.com").Subrouter() s.Path("/articles/{category}/{id:[0-9]+}"). HandlerFunc(ArticleHandler). Name("article") // "http://news.example.com/articles/technology/42" url, err := r.Get("article").URL("subdomain", "news", "category", "technology", "id", "42") ``` To find all the required variables for a given route when calling `URL()`, the method `GetVarNames()` is available: ```go r := mux.NewRouter() r.Host("{domain}"). Path("/{group}/{item_id}"). Queries("some_data1", "{some_data1}"). Queries("some_data2", "{some_data2}"). Name("article") // Will print [domain group item_id some_data1 some_data2] fmt.Println(r.Get("article").GetVarNames()) ``` ### Walking Routes The `Walk` function on `mux.Router` can be used to visit all of the routes that are registered on a router. For example, the following prints all of the registered routes: ```go package main import ( "fmt" "net/http" "strings" "github.com/gorilla/mux" ) func handler(w http.ResponseWriter, r *http.Request) { return } func main() { r := mux.NewRouter() r.HandleFunc("/", handler) r.HandleFunc("/products", handler).Methods("POST") r.HandleFunc("/articles", handler).Methods("GET") r.HandleFunc("/articles/{id}", handler).Methods("GET", "PUT") r.HandleFunc("/authors", handler).Queries("surname", "{surname}") err := r.Walk(func(route *mux.Route, router *mux.Router, ancestors []*mux.Route) error { pathTemplate, err := route.GetPathTemplate() if err == nil { fmt.Println("ROUTE:", pathTemplate) } pathRegexp, err := route.GetPathRegexp() if err == nil { fmt.Println("Path regexp:", pathRegexp) } queriesTemplates, err := route.GetQueriesTemplates() if err == nil { fmt.Println("Queries templates:", strings.Join(queriesTemplates, ",")) } queriesRegexps, err := route.GetQueriesRegexp() if err == nil { fmt.Println("Queries regexps:", strings.Join(queriesRegexps, ",")) } methods, err := route.GetMethods() if err == nil { fmt.Println("Methods:", strings.Join(methods, ",")) } fmt.Println() return nil }) if err != nil { fmt.Println(err) } http.Handle("/", r) } ``` ### Graceful Shutdown Go 1.8 introduced the ability to [gracefully shutdown](https://golang.org/doc/go1.8#http_shutdown) a `*http.Server`. Here's how to do that alongside `mux`: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Middleware Mux supports the addition of middlewares to a [Router](https://godoc.org/github.com/gorilla/mux#Router), which are executed in the order they are added if a match is found, including its subrouters. Middlewares are (typically) small pieces of code which take one request, do something with it, and pass it down to another middleware or the final handler. Some common use cases for middleware are request logging, header manipulation, or `ResponseWriter` hijacking. Mux middlewares are defined using the de facto standard type: ```go type MiddlewareFunc func(http.Handler) http.Handler ``` Typically, the returned handler is a closure which does something with the http.ResponseWriter and http.Request passed to it, and then calls the handler passed as parameter to the MiddlewareFunc. This takes advantage of closures being able access variables from the context where they are created, while retaining the signature enforced by the receivers. A very basic middleware which logs the URI of the request being handled could be written as: ```go func loggingMiddleware(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Do stuff here log.Println(r.RequestURI) // Call the next handler, which can be another middleware in the chain, or the final handler. next.ServeHTTP(w, r) }) } ``` Middlewares can be added to a router using `Router.Use()`: ```go r := mux.NewRouter() r.HandleFunc("/", handler) r.Use(loggingMiddleware) ``` A more complex authentication middleware, which maps session token to users, could be written as: ```go // Define our struct type authenticationMiddleware struct { tokenUsers map[string]string } // Initialize it somewhere func (amw *authenticationMiddleware) Populate() { amw.tokenUsers["00000000"] = "user0" amw.tokenUsers["aaaaaaaa"] = "userA" amw.tokenUsers["05f717e5"] = "randomUser" amw.tokenUsers["deadbeef"] = "user0" } // Middleware function, which will be called for each request func (amw *authenticationMiddleware) Middleware(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { token := r.Header.Get("X-Session-Token") if user, found := amw.tokenUsers[token]; found { // We found the token in our map log.Printf("Authenticated user %s\n", user) // Pass down the request to the next middleware (or final handler) next.ServeHTTP(w, r) } else { // Write an error and stop the handler chain http.Error(w, "Forbidden", http.StatusForbidden) } }) } ``` ```go r := mux.NewRouter() r.HandleFunc("/", handler) amw := authenticationMiddleware{tokenUsers: make(map[string]string)} amw.Populate() r.Use(amw.Middleware) ``` Note: The handler chain will be stopped if your middleware doesn't call `next.ServeHTTP()` with the corresponding parameters. This can be used to abort a request if the middleware writer wants to. Middlewares _should_ write to `ResponseWriter` if they _are_ going to terminate the request, and they _should not_ write to `ResponseWriter` if they _are not_ going to terminate it. ### Handling CORS Requests [CORSMethodMiddleware](https://godoc.org/github.com/gorilla/mux#CORSMethodMiddleware) intends to make it easier to strictly set the `Access-Control-Allow-Methods` response header. * You will still need to use your own CORS handler to set the other CORS headers such as `Access-Control-Allow-Origin` * The middleware will set the `Access-Control-Allow-Methods` header to all the method matchers (e.g. `r.Methods(http.MethodGet, http.MethodPut, http.MethodOptions)` -> `Access-Control-Allow-Methods: GET,PUT,OPTIONS`) on a route * If you do not specify any methods, then: > _Important_: there must be an `OPTIONS` method matcher for the middleware to set the headers. Here is an example of using `CORSMethodMiddleware` along with a custom `OPTIONS` handler to set all the required CORS headers: ```go package main import ( "net/http" "github.com/gorilla/mux" ) func main() { r := mux.NewRouter() // IMPORTANT: you must specify an OPTIONS method matcher for the middleware to set CORS headers r.HandleFunc("/foo", fooHandler).Methods(http.MethodGet, http.MethodPut, http.MethodPatch, http.MethodOptions) r.Use(mux.CORSMethodMiddleware(r)) http.ListenAndServe(":8080", r) } func fooHandler(w http.ResponseWriter, r *http.Request) { w.Header().Set("Access-Control-Allow-Origin", "*") if r.Method == http.MethodOptions { return } w.Write([]byte("foo")) } ``` And an request to `/foo` using something like: ```bash curl localhost:8080/foo -v ``` Would look like: ```bash * Trying ::1... * TCP_NODELAY set * Connected to localhost (::1) port 8080 (#0) > GET /foo HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.59.0 > Accept: */* > < HTTP/1.1 200 OK < Access-Control-Allow-Methods: GET,PUT,PATCH,OPTIONS < Access-Control-Allow-Origin: * < Date: Fri, 28 Jun 2019 20:13:30 GMT < Content-Length: 3 < Content-Type: text/plain; charset=utf-8 < * Connection #0 to host localhost left intact foo ``` ### Testing Handlers Testing handlers in a Go web application is straightforward, and _mux_ doesn't complicate this any further. Given two files: `endpoints.go` and `endpoints_test.go`, here's how we'd test an application using _mux_. First, our simple HTTP handler: ```go // endpoints.go package main func HealthCheckHandler(w http.ResponseWriter, r *http.Request) { // A very simple health check. w.Header().Set("Content-Type", "application/json") w.WriteHeader(http.StatusOK) // In the future we could report back on the status of our DB, or our cache // (e.g. Redis) by performing a simple PING, and include them in the response. io.WriteString(w, `{"alive": true}`) } func main() { r := mux.NewRouter() r.HandleFunc("/health", HealthCheckHandler) log.Fatal(http.ListenAndServe("localhost:8080", r)) } ``` Our test code: ```go // endpoints_test.go package main import ( "net/http" "net/http/httptest" "testing" ) func TestHealthCheckHandler(t *testing.T) { // Create a request to pass to our handler. We don't have any query parameters for now, so we'll // pass 'nil' as the third parameter. req, err := http.NewRequest("GET", "/health", nil) if err != nil { t.Fatal(err) } // We create a ResponseRecorder (which satisfies http.ResponseWriter) to record the response. rr := httptest.NewRecorder() handler := http.HandlerFunc(HealthCheckHandler) // Our handlers satisfy http.Handler, so we can call their ServeHTTP method // directly and pass in our Request and ResponseRecorder. handler.ServeHTTP(rr, req) // Check the status code is what we expect. if status := rr.Code; status != http.StatusOK { t.Errorf("handler returned wrong status code: got %v want %v", status, http.StatusOK) } // Check the response body is what we expect. expected := `{"alive": true}` if rr.Body.String() != expected { t.Errorf("handler returned unexpected body: got %v want %v", rr.Body.String(), expected) } } ``` In the case that our routes have [variables](#examples), we can pass those in the request. We could write [table-driven tests](https://dave.cheney.net/2013/06/09/writing-table-driven-tests-in-go) to test multiple possible route variables as needed. ```go // endpoints.go func main() { r := mux.NewRouter() // A route with a route variable: r.HandleFunc("/metrics/{type}", MetricsHandler) log.Fatal(http.ListenAndServe("localhost:8080", r)) } ``` Our test file, with a table-driven test of `routeVariables`: ```go // endpoints_test.go func TestMetricsHandler(t *testing.T) { tt := []struct{ routeVariable string shouldPass bool }{ {"goroutines", true}, {"heap", true}, {"counters", true}, {"queries", true}, {"adhadaeqm3k", false}, } for _, tc := range tt { path := fmt.Sprintf("/metrics/%s", tc.routeVariable) req, err := http.NewRequest("GET", path, nil) if err != nil { t.Fatal(err) } rr := httptest.NewRecorder() // To add the vars to the context, // we need to create a router through which we can pass the request. router := mux.NewRouter() router.HandleFunc("/metrics/{type}", MetricsHandler) router.ServeHTTP(rr, req) // In this case, our MetricsHandler returns a non-200 response // for a route variable it doesn't know about. if rr.Code == http.StatusOK && !tc.shouldPass { t.Errorf("handler should have failed on routeVariable %s: got %v want %v", tc.routeVariable, rr.Code, http.StatusOK) } } } ``` ## Full Example Here's a complete, runnable example of a small `mux` based server: ```go package main import ( "net/http" "log" "github.com/gorilla/mux" ) func YourHandler(w http.ResponseWriter, r *http.Request) { w.Write([]byte("Gorilla!\n")) } func main() { r := mux.NewRouter() // Routes consist of a path and a handler function. r.HandleFunc("/", YourHandler) // Bind to a port and pass our router in log.Fatal(http.ListenAndServe(":8000", r)) } ``` ## License BSD licensed. See the LICENSE file for details. --- ### Overview (`vendor/github.com/gorilla/websocket/README.md`) # Gorilla WebSocket [](https://godoc.org/github.com/gorilla/websocket) [](https://circleci.com/gh/gorilla/websocket) Gorilla WebSocket is a [Go](http://golang.org/) implementation of the [WebSocket](http://www.rfc-editor.org/rfc/rfc6455.txt) protocol. ### Documentation * [API Reference](https://pkg.go.dev/github.com/gorilla/websocket?tab=doc) * [Chat example](https://github.com/gorilla/websocket/tree/master/examples/chat) * [Command example](https://github.com/gorilla/websocket/tree/master/examples/command) * [Client and server example](https://github.com/gorilla/websocket/tree/master/examples/echo) * [File watch example](https://github.com/gorilla/websocket/tree/master/examples/filewatch) ### Status The Gorilla WebSocket package provides a complete and tested implementation of the [WebSocket](http://www.rfc-editor.org/rfc/rfc6455.txt) protocol. The package API is stable. ### Installation go get github.com/gorilla/websocket ### Protocol Compliance The Gorilla WebSocket package passes the server tests in the [Autobahn Test Suite](https://github.com/crossbario/autobahn-testsuite) using the application in the [examples/autobahn subdirectory](https://github.com/gorilla/websocket/tree/master/examples/autobahn). --- ### Overview (`vendor/github.com/heroiclabs/sql-migrate/README.md`) # sql-migrate This is a fork of the [sql-migrate](https://github.com/rubenv/sql-migrate/) SQL Schema migration tool for [Go](https://golang.org/). The motivation for this fork is to remove the `go-gorp` dependency, simplifying the codebase but making it work with the [pgx](https://github.com/jackc/pgx) db driver exclusively. [](https://travis-ci.org/rubenv/sql-migrate) [](https://godoc.org/github.com/rubenv/sql-migrate) ## Features * Can embed migrations into your application * Migrations are defined with SQL for full flexibility * Atomic migrations * Up/down migrations to allow rollback ## Installation ```bash go get -v github.com/heroiclabs/nakama/sql-migrate/ ``` ## Usage Import sql-migrate into your application: ```go import "github.com/heroiclabs/nakama/sql-migrate" ``` Set up a source of migrations, this can be from memory, from a set of files, from bindata (more on that later), or from any library that implements [`http.FileSystem`](https://godoc.org/net/http#FileSystem): ```go // Hardcoded strings in memory: migrations := &migrate.MemoryMigrationSource{ Migrations: []*migrate.Migration{ &migrate.Migration{ Id: "123", Up: []string{"CREATE TABLE people (id int)"}, Down: []string{"DROP TABLE people"}, }, }, } // OR: Read migrations from a folder: migrations := &migrate.FileMigrationSource{ Dir: "db/migrations", } // OR: Use migrations from a packr box migrations := &migrate.PackrMigrationSource{ Box: packr.New("migrations", "./migrations"), } // OR: Use pkger which implements `http.FileSystem` migrationSource := &migrate.HttpFileSystemMigrationSource{ FileSystem: pkger.Dir("/db/migrations"), } // OR: Use migrations from bindata: migrations := &migrate.AssetMigrationSource{ Asset: Asset, AssetDir: AssetDir, Dir: "migrations", } // OR: Read migrations from a `http.FileSystem` migrationSource := &migrate.HttpFileSystemMigrationSource{ FileSystem: httpFS, } ``` Then use the `Exec` function to upgrade your database: ```go n, err := migrate.Exec(db, migrations, migrate.Up) if err != nil { // Handle errors! } fmt.Printf("Applied %d migrations!\n", n) ``` Note that `n` can be greater than `0` even if there is an error: any migration that succeeded will remain applied even if a later one fails. ## Writing migrations Migrations are defined in SQL files, which contain a set of SQL statements. Special comments are used to distinguish up and down migrations. ```sql -- +migrate Up -- SQL in section 'Up' is executed when this migration is applied CREATE TABLE people (id int); -- +migrate Down -- SQL section 'Down' is executed when this migration is rolled back DROP TABLE people; ``` You can put multiple statements in each block, as long as you end them with a semicolon (`;`). You can alternatively set up a separator string that matches an entire line by setting `sqlparse.LineSeparator`. This can be used to imitate, for example, MS SQL Query Analyzer functionality where commands can be separated by a line with contents of `GO`. If `sqlparse.LineSeparator` is matched, it will not be included in the resulting migration scripts. If you have complex statements which contain semicolons, use `StatementBegin` and `StatementEnd` to indicate boundaries: ```sql -- +migrate Up CREATE TABLE people (id int); -- +migrate StatementBegin CREATE OR REPLACE FUNCTION do_something() returns void AS $$ DECLARE create_query text; BEGIN -- Do something here END; $$ language plpgsql; -- +migrate StatementEnd -- +migrate Down DROP FUNCTION do_something(); DROP TABLE people; ``` The order in which migrations are applied is defined through the filename: sql-migrate will sort migrations based on their name. It's recommended to use an increasing version number or a timestamp as the first part of the filename. Normally each migration is run within a transaction in order to guarantee that it is fully atomic. However some SQL commands (for example creating an index concurrently in PostgreSQL) cannot be executed inside a transaction. In order to execute such a command in a migration, the migration can be run using the `notransaction` option: ```sql -- +migrate Up notransaction CREATE UNIQUE INDEX CONCURRENTLY people_unique_id_idx ON people (id); -- +migrate Down DROP INDEX people_unique_id_idx; ``` Because a `notransaction` migration runs outside of a transaction, it is **not atomic**: if it fails partway through, the statements that already ran remain applied and the migration is *not* recorded as complete. Each statement in a `notransaction` migration must be a single SQL command. Statements are sent using PostgreSQL's simple query protocol, which wraps a message containing multiple commands in an implicit transaction. Grouping several commands inside one `StatementBegin`/`StatementEnd` block therefore reintroduces a transaction. The `notransaction` option applies to the whole `Up` (or `Down`) section. A file may contain several `-- +migrate Up` sections, but if any of them sets `notransaction` then *all* of that migration's up statements run outside a transaction. You cannot mix transactional and non-transactional statements in one migration, so keep statements that must run outside a transaction in their own migration file. ## Embedding migrations with libraries that implement `http.FileSystem` You can also embed migrations with any library that implements `http.FileSystem`, like [`vfsgen`](https://github.com/shurcooL/vfsgen), [`parcello`](https://github.com/phogolabs/parcello), or [`go-resources`](https://github.com/omeid/go-resources). ```go migrationSource := &migrate.HttpFileSystemMigrationSource{ FileSystem: httpFS, } ``` ## Tests Run the tests by first starting the Postgres docker container. ```bash docker compose up go test ``` ## Extending Adding a new migration source means implementing `MigrationSource`. ```go type MigrationSource interface { FindMigrations() ([]*Migration, error) } ``` The resulting slice of migrations will be executed in the given order, so it should usually be sorted by the `Id` field. ## License This library is distributed under the [MIT](LICENSE) license. --- ### Overview (`vendor/github.com/jackc/pgerrcode/README.md`) [](https://godoc.org/github.com/jackc/pgerrcode) # pgerrcode Package pgerrcode contains constants for PostgreSQL error codes. ## License MIT for this package's code and PostgreSQL License for the underlying data. --- ### Overview (`vendor/github.com/jackc/pgpassfile/README.md`) [](https://godoc.org/github.com/jackc/pgpassfile) [](https://travis-ci.org/jackc/pgpassfile) # pgpassfile Package pgpassfile is a parser PostgreSQL .pgpass files. Extracted and rewritten from original implementation in https://github.com/jackc/pgx. --- ### Overview (`vendor/github.com/jackc/pgservicefile/README.md`) [](https://pkg.go.dev/github.com/jackc/pgservicefile) [](https://github.com/jackc/pgservicefile/actions/workflows/ci.yml) # pgservicefile Package pgservicefile is a parser for PostgreSQL service files (e.g. `.pg_service.conf`). --- ### Overview (`vendor/github.com/kamstrup/intmap/README.md`) Fast hashmap with integer keys for Golang [](https://godoc.org/github.com/kamstrup/intmap) [](https://goreportcard.com/report/github.com/kamstrup/intmap) # intmap import "github.com/kamstrup/intmap" Package intmap is a fast hashmap implementation for Golang, specialized for maps with integer type keys. The values can be of any type. It is a full port of https://github.com/brentp/intintmap to use type parameters (aka generics). It interleaves keys and values in the same underlying array to improve locality. This is also known as open addressing with linear probing. It is up to 3X faster than the builtin map: ``` name time/op Map64Fill-8 201ms ± 5% IntIntMapFill-8 207ms ±31% StdMapFill-8 371ms ±11% Map64Get10PercentHitRate-8 148µs ±40% IntIntMapGet10PercentHitRate-8 171µs ±50% StdMapGet10PercentHitRate-8 171µs ±33% Map64Get100PercentHitRate-8 4.50ms ± 5% IntIntMapGet100PercentHitRate-8 4.82ms ± 6% StdMapGet100PercentHitRate-8 15.5ms ±32% ``` ## Usage ```go m := intmap.New[int64,int64](32768) m.Put(int64(1234), int64(-222)) m.Put(int64(123), int64(33)) v, ok := m.Get(int64(222)) v, ok := m.Get(int64(333)) m.Del(int64(222)) m.Del(int64(333)) fmt.Println(m.Len()) m.ForEach(func(k int64, v int64) { fmt.Printf("key: %d, value: %d\n", k, v) }) m.Clear() // all gone, but buffers kept ``` --- ### Overview (`vendor/github.com/klauspost/compress/README.md`) # compress This package provides various compression algorithms. * [zstandard](https://github.com/klauspost/compress/tree/master/zstd#zstd) compression and decompression in pure Go. * [S2](https://github.com/klauspost/compress/tree/master/s2#s2-compression) is a high performance replacement for Snappy. * Optimized [deflate](https://godoc.org/github.com/klauspost/compress/flate) packages which can be used as a dropin replacement for [gzip](https://godoc.org/github.com/klauspost/compress/gzip), [zip](https://godoc.org/github.com/klauspost/compress/zip) and [zlib](https://godoc.org/github.com/klauspost/compress/zlib). * [snappy](https://github.com/klauspost/compress/tree/master/snappy) is a drop-in replacement for `github.com/golang/snappy` offering better compression and concurrent streams. * [lzw](https://github.com/klauspost/compress/tree/master/lzw) is a drop-in replacement for `compress/lzw` with 1.4-4x faster decompression and 1.1-2.7x faster compression, depending on the data. * [huff0](https://github.com/klauspost/compress/tree/master/huff0) and [FSE](https://github.com/klauspost/compress/tree/master/fse) implementations for raw entropy encoding. * [Xpress](https://github.com/klauspost/compress/tree/master/xpress) decompression of the Microsoft XPRESS (MS-XCA) plain LZ77 and LZ77+Huffman formats (the LZ77+Huffman variant is the one used in WIM images and Windows Compact OS / WOF data). * [gzhttp](https://github.com/klauspost/compress/tree/master/gzhttp) Provides client and server wrappers for handling gzipped/zstd HTTP requests efficiently. * [pgzip](https://github.com/klauspost/pgzip) is a separate package that provides a very fast parallel gzip implementation. [](https://pkg.go.dev/github.com/klauspost/compress?tab=subdirectories) [](https://github.com/klauspost/compress/actions/workflows/go.yml) [](https://sourcegraph.com/github.com/klauspost/compress?badge) # package usage Use `go get github.com/klauspost/compress@latest` to add it to your project. This package will support the current Go version and 2 versions back. * Use the `nounsafe` tag to disable all use of the "unsafe" package. * Use the `noasm` tag to disable all assembly across packages. Use the links above for more information on each. # changelog * Jul 1st, 2026 [1.19.0](https://github.com/klauspost/compress/releases/tag/v1.19.0) * zstd: Add true concurrent stream encodingin https://github.com/klauspost/compress/pull/1136 * zstd: arm64 decoder asm by @lizthegrey in https://github.com/klauspost/compress/pull/1160 * flate: Add inflate checkpoints in https://github.com/klauspost/compress/pull/1154 * zstd: avoid unused BuildDict encoder allocation by @snissn in https://github.com/klauspost/compress/pull/1147 * snappy/s2: Limit length of varint in `decodedLen` by @eustas in https://github.com/klauspost/compress/pull/1148 * gzhttp: match qvalue parameter case-insensitively (RFC 7231) by @z9z in https://github.com/klauspost/compress/pull/1149 * zip: add NameDecoder callback for legacy encoding rewrite by @SAY-5 in https://github.com/klauspost/compress/pull/1150 * huff0: Allow building tables from histogram in https://github.com/klauspost/compress/pull/1155 * huff0: Allow building table from oversized histogram in https://github.com/klauspost/compress/pull/1156 * s2sx: Clean symlink targets in https://github.com/klauspost/compress/pull/1163 * Feb 9th, 2026 [1.18.4](https://github.com/klauspost/compress/releases/tag/v1.18.4) * gzhttp: Add zstandard to server handler wrapper https://github.com/klauspost/compress/pull/1121 * zstd: Add ResetWithOptions to encoder/decoder https://github.com/klauspost/compress/pull/1122 * gzhttp: preserve qvalue when extra parameters follow in Accept-Encoding by @analytically in https://github.com/klauspost/compress/pull/1116 * Jan 16th, 2026 [1.18.3](https://github.com/klauspost/compress/releases/tag/v1.18.3) * Downstream CVE-2025-61728. See [golang/go#77102](https://github.com/golang/go/issues/77102). * Dec 1st, 2025 - [1.18.2](https://github.com/klauspost/compress/releases/tag/v1.18.2) * flate: Fix invalid encoding on level 9 with single value input in https://github.com/klauspost/compress/pull/1115 * flate: reduce stateless allocations by @RXamzin in https://github.com/klauspost/compress/pull/1106 * Oct 20, 2025 - [1.18.1](https://github.com/klauspost/compress/releases/tag/v1.18.1) - RETRACTED * zstd: Add simple zstd EncodeTo/DecodeTo functions https://github.com/klauspost/compress/pull/1079 * zstd: Fix incorrect buffer size in dictionary encodes https://github.com/klauspost/compress/pull/1059 * s2: check for cap, not len of buffer in EncodeBetter/Best by @vdarulis in https://github.com/klauspost/compress/pull/1080 * zlib: Avoiding extra allocation in zlib.reader.Reset by @travelpolicy in https://github.com/klauspost/compress/pull/1086 * gzhttp: remove redundant err check in zstdReader by @ryanfowler in https://github.com/klauspost/compress/pull/1090 * flate: Faster load+store https://github.com/klauspost/compress/pull/1104 * flate: Simplify matchlen https://github.com/klauspost/compress/pull/1101 * flate: Use exact sizes for huffman tables https://github.com/klauspost/compress/pull/1103 * Feb 19th, 2025 - [1.18.0](https://github.com/klauspost/compress/releases/tag/v1.18.0) * Add unsafe little endian loaders https://github.com/klauspost/compress/pull/1036 * fix: check `r.err != nil` but return a nil value error `err` by @alingse in https://github.com/klauspost/compress/pull/1028 * flate: Simplify L4-6 loading https://github.com/klauspost/compress/pull/1043 * flate: Simplify matchlen (remove asm) https://github.com/klauspost/compress/pull/1045 * s2: Improve small block compression speed w/o asm https://github.com/klauspost/compress/pull/1048 * flate: Fix matchlen L5+L6 https://github.com/klauspost/compress/pull/1049 * flate: Cleanup & reduce casts https://github.com/klauspost/compress/pull/1050
See changes to v1.17.x * Oct 11th, 2024 - [1.17.11](https://github.com/klauspost/compress/releases/tag/v1.17.11) * zstd: Fix extra CRC written with multiple Close calls https://github.com/klauspost/compress/pull/1017 * s2: Don't use stack for index tables https://github.com/klauspost/compress/pull/1014 * gzhttp: No content-type on no body response code by @juliens in https://github.com/klauspost/compress/pull/1011 * gzhttp: Do not set the content-type when response has no body by @kevinpollet in https://github.com/klauspost/compress/pull/1013 * Sep 23rd, 2024 - [1.17.10](https://github.com/klauspost/compress/releases/tag/v1.17.10) * gzhttp: Add TransportAlwaysDecompress option. https://github.com/klauspost/compress/pull/978 * gzhttp: Add supported decompress request body by @mirecl in https://github.com/klauspost/compress/pull/1002 * s2: Add EncodeBuffer buffer recycling callback https://github.com/klauspost/compress/pull/982 * zstd: Improve memory usage on small streaming encodes https://github.com/klauspost/compress/pull/1007 * flate: read data written with partial flush by @vajexal in https://github.com/klauspost/compress/pull/996 * Jun 12th, 2024 - [1.17.9](https://github.com/klauspost/compress/releases/tag/v1.17.9) * s2: Reduce ReadFrom temporary allocations https://github.com/klauspost/compress/pull/949 * flate, zstd: Shave some bytes off amd64 matchLen by @greatroar in https://github.com/klauspost/compress/pull/963 * Upgrade zip/zlib to 1.22.4 upstream https://github.com/klauspost/compress/pull/970 https://github.com/klauspost/compress/pull/971 * zstd: BuildDict fails with RLE table https://github.com/klauspost/compress/pull/951 * Apr 9th, 2024 - [1.17.8](https://github.com/klauspost/compress/releases/tag/v1.17.8) * zstd: Reject blocks where reserved values are not 0 https://github.com/klauspost/compress/pull/885 * zstd: Add RLE detection+encoding https://github.com/klauspost/compress/pull/938 * Feb 21st, 2024 - [1.17.7](https://github.com/klauspost/compress/releases/tag/v1.17.7) * s2: Add AsyncFlush method: Complete the block without flushing by @Jille in https://github.com/klauspost/compress/pull/927 * s2: Fix literal+repeat exceeds dst crash https://github.com/klauspost/compress/pull/930 * Feb 5th, 2024 - [1.17.6](https://github.com/klauspost/compress/releases/tag/v1.17.6) * zstd: Fix incorrect repeat coding in best mode https://github.com/klauspost/compress/pull/923 * s2: Fix DecodeConcurrent deadlock on errors https://github.com/klauspost/compress/pull/925 * Jan 26th, 2024 - [v1.17.5](https://github.com/klauspost/compress/releases/tag/v1.17.5) * flate: Fix reset with dictionary on custom window encodes https://github.com/klauspost/compress/pull/912 * zstd: Add Frame header encoding and stripping https://github.com/klauspost/compress/pull/908 * zstd: Limit better/best default window to 8MB https://github.com/klauspost/compress/pull/913 * zstd: Speed improvements by @greatroar in https://github.com/klauspost/compress/pull/896 https://github.com/klauspost/compress/pull/910 * s2: Fix callbacks for skippable blocks and disallow 0xfe (Padding) by @Jille in https://github.com/klauspost/compress/pull/916 https://github.com/klauspost/compress/pull/917 https://github.com/klauspost/compress/pull/919 https://github.com/klauspost/compress/pull/918 * Dec 1st, 2023 - [v1.17.4](https://github.com/klauspost/compress/releases/tag/v1.17.4) * huff0: Speed up symbol counting by @greatroar in https://github.com/klauspost/compress/pull/887 * huff0: Remove byteReader by @greatroar in https://github.com/klauspost/compress/pull/886 * gzhttp: Allow overriding decompression on transport https://github.com/klauspost/compress/pull/892 * gzhttp: Clamp compression level https://github.com/klauspost/compress/pull/890 * gzip: Error out if reserved bits are set https://github.com/klauspost/compress/pull/891 * Nov 15th, 2023 - [v1.17.3](https://github.com/klauspost/compress/releases/tag/v1.17.3) * fse: Fix max header size https://github.com/klauspost/compress/pull/881 * zstd: Improve better/best compression https://github.com/klauspost/compress/pull/877 * gzhttp: Fix missing content type on Close https://github.com/klauspost/compress/pull/883 * Oct 22nd, 2023 - [v1.17.2](https://github.com/klauspost/compress/releases/tag/v1.17.2) * zstd: Fix rare *CORRUPTION* output in "best" mode. See https://github.com/klauspost/compress/pull/876 * Oct 14th, 2023 - [v1.17.1](https://github.com/klauspost/compress/releases/tag/v1.17.1) * s2: Fix S2 "best" dictionary wrong encoding https://github.com/klauspost/compress/pull/871 * flate: Reduce allocations in decompressor and minor code improvements by @fakefloordiv in https://github.com/klauspost/compress/pull/869 * s2: Fix EstimateBlockSize on 6&7 length input https://github.com/klauspost/compress/pull/867 * Sept 19th, 2023 - [v1.17.0](https://github.com/klauspost/compress/releases/tag/v1.17.0) * Add experimental dictionary builder https://github.com/klauspost/compress/pull/853 * Add xerial snappy read/writer https://github.com/klauspost/compress/pull/838 * flate: Add limited window compression https://github.com/klauspost/compress/pull/843 * s2: Do 2 overlapping match checks https://github.com/klauspost/compress/pull/839 * flate: Add amd64 assembly matchlen https://github.com/klauspost/compress/pull/837 * gzip: Copy bufio.Reader on Reset by @thatguystone in https://github.com/klauspost/compress/pull/860
See changes to v1.16.x * July 1st, 2023 - [v1.16.7](https://github.com/klauspost/compress/releases/tag/v1.16.7) * zstd: Fix default level first dictionary encode https://github.com/klauspost/compress/pull/829 * s2: add GetBufferCapacity() method by @GiedriusS in https://github.com/klauspost/compress/pull/832 * June 13, 2023 - [v1.16.6](https://github.com/klauspost/compress/releases/tag/v1.16.6) * zstd: correctly ignore WithEncoderPadding(1) by @ianlancetaylor in https://github.com/klauspost/compress/pull/806 * zstd: Add amd64 match length assembly https://github.com/klauspost/compress/pull/824 * gzhttp: Handle informational headers by @rtribotte in https://github.com/klauspost/compress/pull/815 * s2: Improve Better compression slightly https://github.com/klauspost/compress/pull/663 * Apr 16, 2023 - [v1.16.5](https://github.com/klauspost/compress/releases/tag/v1.16.5) * zstd: readByte needs to use io.ReadFull by @jnoxon in https://github.com/klauspost/compress/pull/802 * gzip: Fix WriterTo after initial read https://github.com/klauspost/compress/pull/804 * Apr 5, 2023 - [v1.16.4](https://github.com/klauspost/compress/releases/tag/v1.16.4) * zstd: Improve zstd best efficiency by @greatroar and @klauspost in https://github.com/klauspost/compress/pull/784 * zstd: Respect WithAllLitEntropyCompression https://github.com/klauspost/compress/pull/792 * zstd: Fix amd64 not always detecting corrupt data https://github.com/klauspost/compress/pull/785 * zstd: Various minor improvements by @greatroar in https://github.com/klauspost/compress/pull/788 https://github.com/klauspost/compress/pull/794 https://github.com/klauspost/compress/pull/795 * s2: Fix huge block overflow https://github.com/klauspost/compress/pull/779 * s2: Allow CustomEncoder fallback https://github.com/klauspost/compress/pull/780 * gzhttp: Support ResponseWriter Unwrap() in gzhttp handler by @jgimenez in https://github.com/klauspost/compress/pull/799 * Mar 13, 2023 - [v1.16.1](https://github.com/klauspost/compress/releases/tag/v1.16.1) * zstd: Speed up + improve best encoder by @greatroar in https://github.com/klauspost/compress/pull/776 * gzhttp: Add optional [BREACH mitigation](https://github.com/klauspost/compress/tree/master/gzhttp#breach-mitigation). https://github.com/klauspost/compress/pull/762 https://github.com/klauspost/compress/pull/768 https://github.com/klauspost/compress/pull/769 https://github.com/klauspost/compress/pull/770 https://github.com/klauspost/compress/pull/767 * s2: Add Intel LZ4s converter https://github.com/klauspost/compress/pull/766 * zstd: Minor bug fixes https://github.com/klauspost/compress/pull/771 https://github.com/klauspost/compress/pull/772 https://github.com/klauspost/compress/pull/773 * huff0: Speed up compress1xDo by @greatroar in https://github.com/klauspost/compress/pull/774 * Feb 26, 2023 - [v1.16.0](https://github.com/klauspost/compress/releases/tag/v1.16.0) * s2: Add [Dictionary](https://github.com/klauspost/compress/tree/master/s2#dictionaries) support. https://github.com/klauspost/compress/pull/685 * s2: Add Compression Size Estimate. https://github.com/klauspost/compress/pull/752 * s2: Add support for custom stream encoder. https://github.com/klauspost/compress/pull/755 * s2: Add LZ4 block converter. https://github.com/klauspost/compress/pull/748 * s2: Support io.ReaderAt in ReadSeeker. https://github.com/klauspost/compress/pull/747 * s2c/s2sx: Use concurrent decoding. https://github.com/klauspost/compress/pull/746
See changes to v1.15.x * Jan 21st, 2023 (v1.15.15) * deflate: Improve level 7-9 https://github.com/klauspost/compress/pull/739 * zstd: Add delta encoding support by @greatroar in https://github.com/klauspost/compress/pull/728 * zstd: Various speed improvements by @greatroar https://github.com/klauspost/compress/pull/741 https://github.com/klauspost/compress/pull/734 https://github.com/klauspost/compress/pull/736 https://github.com/klauspost/compress/pull/744 https://github.com/klauspost/compress/pull/743 https://github.com/klauspost/compress/pull/745 * gzhttp: Add SuffixETag() and DropETag() options to prevent ETag collisions on compressed responses by @willbicks in https://github.com/klauspost/compress/pull/740 * Jan 3rd, 2023 (v1.15.14) * flate: Improve speed in big stateless blocks https://github.com/klauspost/compress/pull/718 * zstd: Minor speed tweaks by @greatroar in https://github.com/klauspost/compress/pull/716 https://github.com/klauspost/compress/pull/720 * export NoGzipResponseWriter for custom ResponseWriter wrappers by @harshavardhana in https://github.com/klauspost/compress/pull/722 * s2: Add example for indexing and existing stream https://github.com/klauspost/compress/pull/723 * Dec 11, 2022 (v1.15.13) * zstd: Add [MaxEncodedSize](https://pkg.go.dev/github.com/klauspost/compress@v1.15.13/zstd#Encoder.MaxEncodedSize) to encoder https://github.com/klauspost/compress/pull/691 * zstd: Various tweaks and improvements https://github.com/klauspost/compress/pull/693 https://github.com/klauspost/compress/pull/695 https://github.com/klauspost/compress/pull/696 https://github.com/klauspost/compress/pull/701 https://github.com/klauspost/compress/pull/702 https://github.com/klauspost/compress/pull/703 https://github.com/klauspost/compress/pull/704 https://github.com/klauspost/compress/pull/705 https://github.com/klauspost/compress/pull/706 https://github.com/klauspost/compress/pull/707 https://github.com/klauspost/compress/pull/708 * Oct 26, 2022 (v1.15.12) * zstd: Tweak decoder allocs. https://github.com/klauspost/compress/pull/680 * gzhttp: Always delete `HeaderNoCompression` https://github.com/klauspost/compress/pull/683 * Sept 26, 2022 (v1.15.11) * flate: Improve level 1-3 compression https://github.com/klauspost/compress/pull/678 * zstd: Improve "best" compression by @nightwolfz in https://github.com/klauspost/compress/pull/677 * zstd: Fix+reduce decompression allocations https://github.com/klauspost/compress/pull/668 * zstd: Fix non-effective noescape tag https://github.com/klauspost/compress/pull/667 * Sept 16, 2022 (v1.15.10) * zstd: Add [WithDecodeAllCapLimit](https://pkg.go.dev/github.com/klauspost/compress@v1.15.10/zstd#WithDecodeAllCapLimit) https://github.com/klauspost/compress/pull/649 * Add Go 1.19 - deprecate Go 1.16 https://github.com/klauspost/compress/pull/651 * flate: Improve level 5+6 compression https://github.com/klauspost/compress/pull/656 * zstd: Improve "better" compression https://github.com/klauspost/compress/pull/657 * s2: Improve "best" compression https://github.com/klauspost/compress/pull/658 * s2: Improve "better" compression. https://github.com/klauspost/compress/pull/635 * s2: Slightly faster non-assembly decompression https://github.com/klauspost/compress/pull/646 * Use arrays for constant size copies https://github.com/klauspost/compress/pull/659 * July 21, 2022 (v1.15.9) * zstd: Fix decoder crash on amd64 (no BMI) on invalid input https://github.com/klauspost/compress/pull/645 * zstd: Disable decoder extended memory copies (amd64) due to possible crashes https://github.com/klauspost/compress/pull/644 * zstd: Allow single segments up to "max decoded size" https://github.com/klauspost/compress/pull/643 * July 13, 2022 (v1.15.8) * gzip: fix stack exhaustion bug in Reader.Read https://github.com/klauspost/compress/pull/641 * s2: Add Index header trim/restore https://github.com/klauspost/compress/pull/638 * zstd: Optimize seqdeq amd64 asm by @greatroar in https://github.com/klauspost/compress/pull/636 * zstd: Improve decoder memcopy https://github.com/klauspost/compress/pull/637 * huff0: Pass a single bitReader pointer to asm by @greatroar in https://github.com/klauspost/compress/pull/634 * zstd: Branchless getBits for amd64 w/o BMI2 by @greatroar in https://github.com/klauspost/compress/pull/640 * gzhttp: Remove header before writing https://github.com/klauspost/compress/pull/639 * June 29, 2022 (v1.15.7) * s2: Fix absolute forward seeks https://github.com/klauspost/compress/pull/633 * zip: Merge upstream https://github.com/klauspost/compress/pull/631 * zip: Re-add zip64 fix https://github.com/klauspost/compress/pull/624 * zstd: translate fseDecoder.buildDtable into asm by @WojciechMula in https://github.com/klauspost/compress/pull/598 * flate: Faster histograms https://github.com/klauspost/compress/pull/620 * deflate: Use compound hcode https://github.com/klauspost/compress/pull/622 * June 3, 2022 (v1.15.6) * s2: Improve coding for long, close matches https://github.com/klauspost/compress/pull/613 * s2c: Add Snappy/S2 stream recompression https://github.com/klauspost/compress/pull/611 * zstd: Always use configured block size https://github.com/klauspost/compress/pull/605 * zstd: Fix incorrect hash table placement for dict encoding in default https://github.com/klauspost/compress/pull/606 * zstd: Apply default config to ZipDecompressor without options https://github.com/klauspost/compress/pull/608 * gzhttp: Exclude more common archive formats https://github.com/klauspost/compress/pull/612 * s2: Add ReaderIgnoreCRC https://github.com/klauspost/compress/pull/609 * s2: Remove sanity load on index creation https://github.com/klauspost/compress/pull/607 * snappy: Use dedicated function for scoring https://github.com/klauspost/compress/pull/614 * s2c+s2d: Use official snappy framed extension https://github.com/klauspost/compress/pull/610 * May 25, 2022 (v1.15.5) * s2: Add concurrent stream decompression https://github.com/klauspost/compress/pull/602 * s2: Fix final emit oob read crash on amd64 https://github.com/klauspost/compress/pull/601 * huff0: asm implementation of Decompress1X by @WojciechMula https://github.com/klauspost/compress/pull/596 * zstd: Use 1 less goroutine for stream decoding https://github.com/klauspost/compress/pull/588 * zstd: Copy literal in 16 byte blocks when possible https://github.com/klauspost/compress/pull/592 * zstd: Speed up when WithDecoderLowmem(false) https://github.com/klauspost/compress/pull/599 * zstd: faster next state update in BMI2 version of decode by @WojciechMula in https://github.com/klauspost/compress/pull/593 * huff0: Do not check max size when reading table. https://github.com/klauspost/compress/pull/586 * flate: Inplace hashing for level 7-9 https://github.com/klauspost/compress/pull/590 * May 11, 2022 (v1.15.4) * huff0: decompress directly into output by @WojciechMula in [#577](https://github.com/klauspost/compress/pull/577) * inflate: Keep dict on stack [#581](https://github.com/klauspost/compress/pull/581) * zstd: Faster decoding memcopy in asm [#583](https://github.com/klauspost/compress/pull/583) * zstd: Fix ignored crc [#580](https://github.com/klauspost/compress/pull/580) * May 5, 2022 (v1.15.3) * zstd: Allow to ignore checksum checking by @WojciechMula [#572](https://github.com/klauspost/compress/pull/572) * s2: Fix incorrect seek for io.SeekEnd in [#575](https://github.com/klauspost/compress/pull/575) * Apr 26, 2022 (v1.15.2) * zstd: Add x86-64 assembly for decompression on streams and blocks. Contributed by [@WojciechMula](https://github.com/WojciechMula). Typically 2x faster. [#528](https://github.com/klauspost/compress/pull/528) [#531](https://github.com/klauspost/compress/pull/531) [#545](https://github.com/klauspost/compress/pull/545) [#537](https://github.com/klauspost/compress/pull/537) * zstd: Add options to ZipDecompressor and fixes [#539](https://github.com/klauspost/compress/pull/539) * s2: Use sorted search for index [#555](https://github.com/klauspost/compress/pull/555) * Minimum version is Go 1.16, added CI test on 1.18. * Mar 11, 2022 (v1.15.1) * huff0: Add x86 assembly of Decode4X by @WojciechMula in [#512](https://github.com/klauspost/compress/pull/512) * zstd: Reuse zip decoders in [#514](https://github.com/klauspost/compress/pull/514) * zstd: Detect extra block data and report as corrupted in [#520](https://github.com/klauspost/compress/pull/520) * zstd: Handle zero sized frame content size stricter in [#521](https://github.com/klauspost/compress/pull/521) * zstd: Add stricter block size checks in [#523](https://github.com/klauspost/compress/pull/523) * Mar 3, 2022 (v1.15.0) * zstd: Refactor decoder [#498](https://github.com/klauspost/compress/pull/498) * zstd: Add stream encoding without goroutines [#505](https://github.com/klauspost/compress/pull/505) * huff0: Prevent single blocks exceeding 16 bits by @klauspost in[#507](https://github.com/klauspost/compress/pull/507) * flate: Inline literal emission [#509](https://github.com/klauspost/compress/pull/509) * gzhttp: Add zstd to transport [#400](https://github.com/klauspost/compress/pull/400) * gzhttp: Make content-type optional [#510](https://github.com/klauspost/compress/pull/510) Both compression and decompression now supports "synchronous" stream operations. This means that whenever "concurrency" is set to 1, they will operate without spawning goroutines. Stream decompression is now faster on asynchronous, since the goroutine allocation much more effectively splits the workload. On typical streams this will typically use 2 cores fully for decompression. When a stream has finished decoding no goroutines will be left over, so decoders can now safely be pooled and still be garbage collected. While the release has been extensively tested, it is recommended to testing when upgrading.
See changes to v1.14.x * Feb 22, 2022 (v1.14.4) * flate: Fix rare huffman only (-2) corruption. [#503](https://github.com/klauspost/compress/pull/503) * zip: Update deprecated CreateHeaderRaw to correctly call CreateRaw by @saracen in [#502](https://github.com/klauspost/compress/pull/502) * zip: don't read data descriptor early by @saracen in [#501](https://github.com/klauspost/compress/pull/501) #501 * huff0: Use static decompression buffer up to 30% faster [#499](https://github.com/klauspost/compress/pull/499) [#500](https://github.com/klauspost/compress/pull/500) * Feb 17, 2022 (v1.14.3) * flate: Improve fastest levels compression speed ~10% more throughput. [#482](https://github.com/klauspost/compress/pull/482) [#489](https://github.com/klauspost/compress/pull/489) [#490](https://github.com/klauspost/compress/pull/490) [#491](https://github.com/klauspost/compress/pull/491) [#494](https://github.com/klauspost/compress/pull/494) [#478](https://github.com/klauspost/compress/pull/478) * flate: Faster decompression speed, ~5-10%. [#483](https://github.com/klauspost/compress/pull/483) * s2: Faster compression with Go v1.18 and amd64 microarch level 3+. [#484](https://github.com/klauspost/compress/pull/484) [#486](https://github.com/klauspost/compress/pull/486) * Jan 25, 2022 (v1.14.2) * zstd: improve header decoder by @dsnet [#476](https://github.com/klauspost/compress/pull/476) * zstd: Add bigger default blocks [#469](https://github.com/klauspost/compress/pull/469) * zstd: Remove unused decompression buffer [#470](https://github.com/klauspost/compress/pull/470) * zstd: Fix logically dead code by @ningmingxiao [#472](https://github.com/klauspost/compress/pull/472) * flate: Improve level 7-9 [#471](https://github.com/klauspost/compress/pull/471) [#473](https://github.com/klauspost/compress/pull/473) * zstd: Add noasm tag for xxhash [#475](https://github.com/klauspost/compress/pull/475) * Jan 11, 2022 (v1.14.1) * s2: Add stream index in [#462](https://github.com/klauspost/compress/pull/462) * flate: Speed and efficiency improvements in [#439](https://github.com/klauspost/compress/pull/439) [#461](https://github.com/klauspost/compress/pull/461) [#455](https://github.com/klauspost/compress/pull/455) [#452](https://github.com/klauspost/compress/pull/452) [#458](https://github.com/klauspost/compress/pull/458) * zstd: Performance improvement in [#420]( https://github.com/klauspost/compress/pull/420) [#456](https://github.com/klauspost/compress/pull/456) [#437](https://github.com/klauspost/compress/pull/437) [#467](https://github.com/klauspost/compress/pull/467) [#468](https://github.com/klauspost/compress/pull/468) * zstd: add arm64 xxhash assembly in [#464](https://github.com/klauspost/compress/pull/464) * Add garbled for binaries for s2 in [#445](https://github.com/klauspost/compress/pull/445)
See changes to v1.13.x * Aug 30, 2021 (v1.13.5) * gz/zlib/flate: Alias stdlib errors [#425](https://github.com/klauspost/compress/pull/425) * s2: Add block support to commandline tools [#413](https://github.com/klauspost/compress/pull/413) * zstd: pooledZipWriter should return Writers to the same pool [#426](https://github.com/klauspost/compress/pull/426) * Removed golang/snappy as external dependency for tests [#421](https://github.com/klauspost/compress/pull/421) * Aug 12, 2021 (v1.13.4) * Add [snappy replacement package](https://github.com/klauspost/compress/tree/master/snappy). * zstd: Fix incorrect encoding in "best" mode [#415](https://github.com/klauspost/compress/pull/415) * Aug 3, 2021 (v1.13.3) * zstd: Improve Best compression [#404](https://github.com/klauspost/compress/pull/404) * zstd: Fix WriteTo error forwarding [#411](https://github.com/klauspost/compress/pull/411) * gzhttp: Return http.HandlerFunc instead of http.Handler. Unlikely breaking change. [#406](https://github.com/klauspost/compress/pull/406) * s2sx: Fix max size error [#399](https://github.com/klauspost/compress/pull/399) * zstd: Add optional stream content size on reset [#401](https://github.com/klauspost/compress/pull/401) * zstd: use SpeedBestCompression for level >= 10 [#410](https://github.com/klauspost/compress/pull/410) * Jun 14, 2021 (v1.13.1) * s2: Add full Snappy output support [#396](https://github.com/klauspost/compress/pull/396) * zstd: Add configurable [Decoder window](https://pkg.go.dev/github.com/klauspost/compress/zstd#WithDecoderMaxWindow) size [#394](https://github.com/klauspost/compress/pull/394) * gzhttp: Add header to skip compression [#389](https://github.com/klauspost/compress/pull/389) * s2: Improve speed with bigger output margin [#395](https://github.com/klauspost/compress/pull/395) * Jun 3, 2021 (v1.13.0) * Added [gzhttp](https://github.com/klauspost/compress/tree/master/gzhttp#gzip-handler) which allows wrapping HTTP servers and clients with GZIP compressors. * zstd: Detect short invalid signatures [#382](https://github.com/klauspost/compress/pull/382) * zstd: Spawn decoder goroutine only if needed. [#380](https://github.com/klauspost/compress/pull/380)
See changes to v1.12.x * May 25, 2021 (v1.12.3) * deflate: Better/faster Huffman encoding [#374](https://github.com/klauspost/compress/pull/374) * deflate: Allocate less for history. [#375](https://github.com/klauspost/compress/pull/375) * zstd: Forward read errors [#373](https://github.com/klauspost/compress/pull/373) * Apr 27, 2021 (v1.12.2) * zstd: Improve better/best compression [#360](https://github.com/klauspost/compress/pull/360) [#364](https://github.com/klauspost/compress/pull/364) [#365](https://github.com/klauspost/compress/pull/365) * zstd: Add helpers to compress/decompress zstd inside zip files [#363](https://github.com/klauspost/compress/pull/363) * deflate: Improve level 5+6 compression [#367](https://github.com/klauspost/compress/pull/367) * s2: Improve better/best compression [#358](https://github.com/klauspost/compress/pull/358) [#359](https://github.com/klauspost/compress/pull/358) * s2: Load after checking src limit on amd64. [#362](https://github.com/klauspost/compress/pull/362) * s2sx: Limit max executable size [#368](https://github.com/klauspost/compress/pull/368) * Apr 14, 2021 (v1.12.1) * snappy package removed. Upstream added as dependency. * s2: Better compression in "best" mode [#353](https://github.com/klauspost/compress/pull/353) * s2sx: Add stdin input and detect pre-compressed from signature [#352](https://github.com/klauspost/compress/pull/352) * s2c/s2d: Add http as possible input [#348](https://github.com/klauspost/compress/pull/348) * s2c/s2d/s2sx: Always truncate when writing files [#352](https://github.com/klauspost/compress/pull/352) * zstd: Reduce memory usage further when using [WithLowerEncoderMem](https://pkg.go.dev/github.com/klauspost/compress/zstd#WithLowerEncoderMem) [#346](https://github.com/klauspost/compress/pull/346) * s2: Fix potential problem with amd64 assembly and profilers [#349](https://github.com/klauspost/compress/pull/349)
See changes to v1.11.x * Mar 26, 2021 (v1.11.13) * zstd: Big speedup on small dictionary encodes [#344](https://github.com/klauspost/compress/pull/344) [#345](https://github.com/klauspost/compress/pull/345) * zstd: Add [WithLowerEncoderMem](https://pkg.go.dev/github.com/klauspost/compress/zstd#WithLowerEncoderMem) encoder option [#336](https://github.com/klauspost/compress/pull/336) * deflate: Improve entropy compression [#338](https://github.com/klauspost/compress/pull/338) * s2: Clean up and minor performance improvement in best [#341](https://github.com/klauspost/compress/pull/341) * Mar 5, 2021 (v1.11.12) * s2: Add `s2sx` binary that creates [self extracting archives](https://github.com/klauspost/compress/tree/master/s2#s2sx-self-extracting-archives). * s2: Speed up decompression on non-assembly platforms [#328](https://github.com/klauspost/compress/pull/328) * Mar 1, 2021 (v1.11.9) * s2: Add ARM64 decompression assembly. Around 2x output speed. [#324](https://github.com/klauspost/compress/pull/324) * s2: Improve "better" speed and efficiency. [#325](https://github.com/klauspost/compress/pull/325) * s2: Fix binaries. * Feb 25, 2021 (v1.11.8) * s2: Fixed occasional out-of-bounds write on amd64. Upgrade recommended. * s2: Add AMD64 assembly for better mode. 25-50% faster. [#315](https://github.com/klauspost/compress/pull/315) * s2: Less upfront decoder allocation. [#322](https://github.com/klauspost/compress/pull/322) * zstd: Faster "compression" of incompressible data. [#314](https://github.com/klauspost/compress/pull/314) * zip: Fix zip64 headers. [#313](https://github.com/klauspost/compress/pull/313) * Jan 14, 2021 (v1.11.7) * Use Bytes() interface to get bytes across packages. [#309](https://github.com/klauspost/compress/pull/309) * s2: Add 'best' compression option. [#310](https://github.com/klauspost/compress/pull/310) * s2: Add ReaderMaxBlockSize, changes `s2.NewReader` signature to include varargs. [#311](https://github.com/klauspost/compress/pull/311) * s2: Fix crash on small better buffers. [#308](https://github.com/klauspost/compress/pull/308) * s2: Clean up decoder. [#312](https://github.com/klauspost/compress/pull/312) * Jan 7, 2021 (v1.11.6) * zstd: Make decoder allocations smaller [#306](https://github.com/klauspost/compress/pull/306) * zstd: Free Decoder resources when Reset is called with a nil io.Reader [#305](https://github.com/klauspost/compress/pull/305) * Dec 20, 2020 (v1.11.4) * zstd: Add Best compression mode [#304](https://github.com/klauspost/compress/pull/304) * Add header decoder [#299](https://github.com/klauspost/compress/pull/299) * s2: Add uncompressed stream option [#297](https://github.com/klauspost/compress/pull/297) * Simplify/speed up small blocks with known max size. [#300](https://github.com/klauspost/compress/pull/300) * zstd: Always reset literal dict encoder [#303](https://github.com/klauspost/compress/pull/303) * Nov 15, 2020 (v1.11.3) * inflate: 10-15% faster decompression [#293](https://github.com/klauspost/compress/pull/293) * zstd: Tweak DecodeAll default allocation [#295](https://github.com/klauspost/compress/pull/295) * Oct 11, 2020 (v1.11.2) * s2: Fix out of bounds read in "better" block compression [#291](https://github.com/klauspost/compress/pull/291) * Oct 1, 2020 (v1.11.1) * zstd: Set allLitEntropy true in default configuration [#286](https://github.com/klauspost/compress/pull/286) * Sept 8, 2020 (v1.11.0) * zstd: Add experimental compression [dictionaries](https://github.com/klauspost/compress/tree/master/zstd#dictionaries) [#281](https://github.com/klauspost/compress/pull/281) * zstd: Fix mixed Write and ReadFrom calls [#282](https://github.com/klauspost/compress/pull/282) * inflate/gz: Limit variable shifts, ~5% faster decompression [#274](https://github.com/klauspost/compress/pull/274)
See changes to v1.10.x * July 8, 2020 (v1.10.11) * zstd: Fix extra block when compressing with ReadFrom. [#278](https://github.com/klauspost/compress/pull/278) * huff0: Also populate compression table when reading decoding table. [#275](https://github.com/klauspost/compress/pull/275) * June 23, 2020 (v1.10.10) * zstd: Skip entropy compression in fastest mode when no matches. [#270](https://github.com/klauspost/compress/pull/270) * June 16, 2020 (v1.10.9): * zstd: API change for specifying dictionaries. See [#268](https://github.com/klauspost/compress/pull/268) * zip: update CreateHeaderRaw to handle zip64 fields. [#266](https://github.com/klauspost/compress/pull/266) * Fuzzit tests removed. The service has been purchased and is no longer available. * June 5, 2020 (v1.10.8): * 1.15x faster zstd block decompression. [#265](https://github.com/klauspost/compress/pull/265) * June 1, 2020 (v1.10.7): * Added zstd decompression [dictionary support](https://github.com/klauspost/compress/tree/master/zstd#dictionaries) * Increase zstd decompression speed up to 1.19x. [#259](https://github.com/klauspost/compress/pull/259) * Remove internal reset call in zstd compression and reduce allocations. [#263](https://github.com/klauspost/compress/pull/263) * May 21, 2020: (v1.10.6) * zstd: Reduce allocations while decoding. [#258](https://github.com/klauspost/compress/pull/258), [#252](https://github.com/klauspost/compress/pull/252) * zstd: Stricter decompression checks. * April 12, 2020: (v1.10.5) * s2-commands: Flush output when receiving SIGINT. [#239](https://github.com/klauspost/compress/pull/239) * Apr 8, 2020: (v1.10.4) * zstd: Minor/special case optimizations. [#251](https://github.com/klauspost/compress/pull/251), [#250](https://github.com/klauspost/compress/pull/250), [#249](https://github.com/klauspost/compress/pull/249), [#247](https://github.com/klauspost/compress/pull/247) * Mar 11, 2020: (v1.10.3) * s2: Use S2 encoder in pure Go mode for Snappy output as well. [#245](https://github.com/klauspost/compress/pull/245) * s2: Fix pure Go block encoder. [#244](https://github.com/klauspost/compress/pull/244) * zstd: Added "better compression" mode. [#240](https://github.com/klauspost/compress/pull/240) * zstd: Improve speed of fastest compression mode by 5-10% [#241](https://github.com/klauspost/compress/pull/241) * zstd: Skip creating encoders when not needed. [#238](https://github.com/klauspost/compress/pull/238) * Feb 27, 2020: (v1.10.2) * Close to 50% speedup in inflate (gzip/zip decompression). [#236](https://github.com/klauspost/compress/pull/236) [#234](https://github.com/klauspost/compress/pull/234) [#232](https://github.com/klauspost/compress/pull/232) * Reduce deflate level 1-6 memory usage up to 59%. [#227](https://github.com/klauspost/compress/pull/227) * Feb 18, 2020: (v1.10.1) * Fix zstd crash when resetting multiple times without sending data. [#226](https://github.com/klauspost/compress/pull/226) * deflate: Fix dictionary use on level 1-6. [#224](https://github.com/klauspost/compress/pull/224) * Remove deflate writer reference when closing. [#224](https://github.com/klauspost/compress/pull/224) * Feb 4, 2020: (v1.10.0) * Add optional dictionary to [stateless deflate](https://pkg.go.dev/github.com/klauspost/compress/flate?tab=doc#StatelessDeflate). Breaking change, send `nil` for previous behaviour. [#216](https://github.com/klauspost/compress/pull/216) * Fix buffer overflow on repeated small block deflate. [#218](https://github.com/klauspost/compress/pull/218) * Allow copying content from an existing ZIP file without decompressing+compressing. [#214](https://github.com/klauspost/compress/pull/214) * Added [S2](https://github.com/klauspost/compress/tree/master/s2#s2-compression) AMD64 assembler and various optimizations. Stream speed >10GB/s. [#186](https://github.com/klauspost/compress/pull/186)
See changes prior to v1.10.0 * Jan 20,2020 (v1.9.8) Optimize gzip/deflate with better size estimates and faster table generation. [#207](https://github.com/klauspost/compress/pull/207) by [luyu6056](https://github.com/luyu6056), [#206](https://github.com/klauspost/compress/pull/206). * Jan 11, 2020: S2 Encode/Decode will use provided buffer if capacity is big enough. [#204](https://github.com/klauspost/compress/pull/204) * Jan 5, 2020: (v1.9.7) Fix another zstd regression in v1.9.5 - v1.9.6 removed. * Jan 4, 2020: (v1.9.6) Regression in v1.9.5 fixed causing corrupt zstd encodes in rare cases. * Jan 4, 2020: Faster IO in [s2c + s2d commandline tools](https://github.com/klauspost/compress/tree/master/s2#commandline-tools) compression/decompression. [#192](https://github.com/klauspost/compress/pull/192) * Dec 29, 2019: Removed v1.9.5 since fuzz tests showed a compatibility problem with the reference zstandard decoder. * Dec 29, 2019: (v1.9.5) zstd: 10-20% faster block compression. [#199](https://github.com/klauspost/compress/pull/199) * Dec 29, 2019: [zip](https://godoc.org/github.com/klauspost/compress/zip) package updated with latest Go features * Dec 29, 2019: zstd: Single segment flag condintions tweaked. [#197](https://github.com/klauspost/compress/pull/197) * Dec 18, 2019: s2: Faster compression when ReadFrom is used. [#198](https://github.com/klauspost/compress/pull/198) * Dec 10, 2019: s2: Fix repeat length output when just above at 16MB limit. * Dec 10, 2019: zstd: Add function to get decoder as io.ReadCloser. [#191](https://github.com/klauspost/compress/pull/191) * Dec 3, 2019: (v1.9.4) S2: limit max repeat length. [#188](https://github.com/klauspost/compress/pull/188) * Dec 3, 2019: Add [WithNoEntropyCompression](https://godoc.org/github.com/klauspost/compress/zstd#WithNoEntropyCompression) to zstd [#187](https://github.com/klauspost/compress/pull/187) * Dec 3, 2019: Reduce memory use for tests. Check for leaked goroutines. * Nov 28, 2019 (v1.9.3) Less allocations in stateless deflate. * Nov 28, 2019: 5-20% Faster huff0 decode. Impacts zstd as well. [#184](https://github.com/klauspost/compress/pull/184) * Nov 12, 2019 (v1.9.2) Added [Stateless Compression](#stateless-compression) for gzip/deflate. * Nov 12, 2019: Fixed zstd decompression of large single blocks. [#180](https://github.com/klauspost/compress/pull/180) * Nov 11, 2019: Set default [s2c](https://github.com/klauspost/compress/tree/master/s2#commandline-tools) block size to 4MB. * Nov 11, 2019: Reduce inflate memory use by 1KB. * Nov 10, 2019: Less allocations in deflate bit writer. * Nov 10, 2019: Fix inconsistent error returned by zstd decoder. * Oct 28, 2019 (v1.9.1) ztsd: Fix crash when compressing blocks. [#174](https://github.com/klauspost/compress/pull/174) * Oct 24, 2019 (v1.9.0) zstd: Fix rare data corruption [#173](https://github.com/klauspost/compress/pull/173) * Oct 24, 2019 zstd: Fix huff0 out of buffer write [#171](https://github.com/klauspost/compress/pull/171) and always return errors [#172](https://github.com/klauspost/compress/pull/172) * Oct 10, 2019: Big deflate rewrite, 30-40% faster with better compression [#105](https://github.com/klauspost/compress/pull/105)
See changes prior to v1.9.0 * Oct 10, 2019: (v1.8.6) zstd: Allow partial reads to get flushed data. [#169](https://github.com/klauspost/compress/pull/169) * Oct 3, 2019: Fix inconsistent results on broken zstd streams. * Sep 25, 2019: Added `-rm` (remove source files) and `-q` (no output except errors) to `s2c` and `s2d` [commands](https://github.com/klauspost/compress/tree/master/s2#commandline-tools) * Sep 16, 2019: (v1.8.4) Add `s2c` and `s2d` [commandline tools](https://github.com/klauspost/compress/tree/master/s2#commandline-tools). * Sep 10, 2019: (v1.8.3) Fix s2 decoder [Skip](https://godoc.org/github.com/klauspost/compress/s2#Reader.Skip). * Sep 7, 2019: zstd: Added [WithWindowSize](https://godoc.org/github.com/klauspost/compress/zstd#WithWindowSize), contributed by [ianwilkes](https://github.com/ianwilkes). * Sep 5, 2019: (v1.8.2) Add [WithZeroFrames](https://godoc.org/github.com/klauspost/compress/zstd#WithZeroFrames) which adds full zero payload block encoding option. * Sep 5, 2019: Lazy initialization of zstandard predefined en/decoder tables. * Aug 26, 2019: (v1.8.1) S2: 1-2% compression increase in "better" compression mode. * Aug 26, 2019: zstd: Check maximum size of Huffman 1X compressed literals while decoding. * Aug 24, 2019: (v1.8.0) Added [S2 compression](https://github.com/klauspost/compress/tree/master/s2#s2-compression), a high performance replacement for Snappy. * Aug 21, 2019: (v1.7.6) Fixed minor issues found by fuzzer. One could lead to zstd not decompressing. * Aug 18, 2019: Add [fuzzit](https://fuzzit.dev/) continuous fuzzing. * Aug 14, 2019: zstd: Skip incompressible data 2x faster. [#147](https://github.com/klauspost/compress/pull/147) * Aug 4, 2019 (v1.7.5): Better literal compression. [#146](https://github.com/klauspost/compress/pull/146) * Aug 4, 2019: Faster zstd compression. [#143](https://github.com/klauspost/compress/pull/143) [#144](https://github.com/klauspost/compress/pull/144) * Aug 4, 2019: Faster zstd decompression. [#145](https://github.com/klauspost/compress/pull/145) [#143](https://github.com/klauspost/compress/pull/143) [#142](https://github.com/klauspost/compress/pull/142) * July 15, 2019 (v1.7.4): Fix double EOF block in rare cases on zstd encoder. * July 15, 2019 (v1.7.3): Minor speedup/compression increase in default zstd encoder. * July 14, 2019: zstd decoder: Fix decompression error on multiple uses with mixed content. * July 7, 2019 (v1.7.2): Snappy update, zstd decoder potential race fix. * June 17, 2019: zstd decompression bugfix. * June 17, 2019: fix 32 bit builds. * June 17, 2019: Easier use in modules (less dependencies). * June 9, 2019: New stronger "default" [zstd](https://github.com/klauspost/compress/tree/master/zstd#zstd) compression mode. Matches zstd default compression ratio. * June 5, 2019: 20-40% throughput in [zstandard](https://github.com/klauspost/compress/tree/master/zstd#zstd) compression and better compression. * June 5, 2019: deflate/gzip compression: Reduce memory usage of lower compression levels. * June 2, 2019: Added [zstandard](https://github.com/klauspost/compress/tree/master/zstd#zstd) compression! * May 25, 2019: deflate/gzip: 10% faster bit writer, mostly visible in lower levels. * Apr 22, 2019: [zstd](https://github.com/klauspost/compress/tree/master/zstd#zstd) decompression added. * Aug 1, 2018: Added [huff0 README](https://github.com/klauspost/compress/tree/master/huff0#huff0-entropy-compression). * Jul 8, 2018: Added [Performance Update 2018](#performance-update-2018) below. * Jun 23, 2018: Merged [Go 1.11 inflate optimizations](https://go-review.googlesource.com/c/go/+/102235). Go 1.9 is now required. Backwards compatible version tagged with [v1.3.0](https://github.com/klauspost/compress/releases/tag/v1.3.0). * Apr 2, 2018: Added [huff0](https://godoc.org/github.com/klauspost/compress/huff0) en/decoder. Experimental for now, API may change. * Mar 4, 2018: Added [FSE Entropy](https://godoc.org/github.com/klauspost/compress/fse) en/decoder. Experimental for now, API may change. * Nov 3, 2017: Add compression [Estimate](https://godoc.org/github.com/klauspost/compress#Estimate) function. * May 28, 2017: Reduce allocations when resetting decoder. * Apr 02, 2017: Change back to official crc32, since changes were merged in Go 1.7. * Jan 14, 2017: Reduce stack pressure due to array copies. See [Issue #18625](https://github.com/golang/go/issues/18625). * Oct 25, 2016: Level 2-4 have been rewritten and now offers significantly better performance than before. * Oct 20, 2016: Port zlib changes from Go 1.7 to fix zlib writer issue. Please update. * Oct 16, 2016: Go 1.7 changes merged. Apples to apples this package is a few percent faster, but has a significantly better balance between speed and compression per level. * Mar 24, 2016: Always attempt Huffman encoding on level 4-7. This improves base 64 encoded data compression. * Mar 24, 2016: Small speedup for level 1-3. * Feb 19, 2016: Faster bit writer, level -2 is 15% faster, level 1 is 4% faster. * Feb 19, 2016: Handle small payloads faster in level 1-3. * Feb 19, 2016: Added faster level 2 + 3 compression modes. * Feb 19, 2016: [Rebalanced compression levels](https://blog.klauspost.com/rebalancing-deflate-compression-levels/), so there is a more even progression in terms of compression. New default level is 5. * Feb 14, 2016: Snappy: Merge upstream changes. * Feb 14, 2016: Snappy: Fix aggressive skipping. * Feb 14, 2016: Snappy: Update benchmark. * Feb 13, 2016: Deflate: Fixed assembler problem that could lead to sub-optimal compression. * Feb 12, 2016: Snappy: Added AMD64 SSE 4.2 optimizations to matching, which makes easy to compress material run faster. Typical speedup is around 25%. * Feb 9, 2016: Added Snappy package fork. This version is 5-7% faster, much more on hard to compress content. * Jan 30, 2016: Optimize level 1 to 3 by not considering static dictionary or storing uncompressed. ~4-5% speedup. * Jan 16, 2016: Optimization on deflate level 1,2,3 compression. * Jan 8 2016: Merge [CL 18317](https://go-review.googlesource.com/#/c/18317): fix reading, writing of zip64 archives. * Dec 8 2015: Make level 1 and -2 deterministic even if write size differs. * Dec 8 2015: Split encoding functions, so hashing and matching can potentially be inlined. 1-3% faster on AMD64. 5% faster on other platforms. * Dec 8 2015: Fixed rare [one byte out-of bounds read](https://github.com/klauspost/compress/issues/20). Please update! * Nov 23 2015: Optimization on token writer. ~2-4% faster. Contributed by [@dsnet](https://github.com/dsnet). * Nov 20 2015: Small optimization to bit writer on 64 bit systems. * Nov 17 2015: Fixed out-of-bound errors if the underlying Writer returned an error. See [#15](https://github.com/klauspost/compress/issues/15). * Nov 12 2015: Added [io.WriterTo](https://golang.org/pkg/io/#WriterTo) support to gzip/inflate. * Nov 11 2015: Merged [CL 16669](https://go-review.googlesource.com/#/c/16669/4): archive/zip: enable overriding (de)compressors per file * Oct 15 2015: Added skipping on uncompressible data. Random data speed up >5x.
# deflate usage The packages are drop-in replacements for standard library [deflate](https://godoc.org/github.com/klauspost/compress/flate), [gzip](https://godoc.org/github.com/klauspost/compress/gzip), [zip](https://godoc.org/github.com/klauspost/compress/zip), and [zlib](https://godoc.org/github.com/klauspost/compress/zlib). Simply replace the import path to use them: Typical speed is about 2x of the standard library packages. | old import | new import | Documentation | |------------------|---------------------------------------|-------------------------------------------------------------------------| | `compress/gzip` | `github.com/klauspost/compress/gzip` | [gzip](https://pkg.go.dev/github.com/klauspost/compress/gzip?tab=doc) | | `compress/zlib` | `github.com/klauspost/compress/zlib` | [zlib](https://pkg.go.dev/github.com/klauspost/compress/zlib?tab=doc) | | `archive/zip` | `github.com/klauspost/compress/zip` | [zip](https://pkg.go.dev/github.com/klauspost/compress/zip?tab=doc) | | `compress/flate` | `github.com/klauspost/compress/flate` | [flate](https://pkg.go.dev/github.com/klauspost/compress/flate?tab=doc) | You may also be interested in [pgzip](https://github.com/klauspost/pgzip), which is a drop-in replacement for gzip, which support multithreaded compression on big files and the optimized [crc32](https://github.com/klauspost/crc32) package used by these packages. The packages implement the same API as the standard library, so you can use the original godoc documentation: [gzip](http://golang.org/pkg/compress/gzip/), [zip](http://golang.org/pkg/archive/zip/), [zlib](http://golang.org/pkg/compress/zlib/), [flate](http://golang.org/pkg/compress/flate/). Currently there is only minor speedup on decompression (mostly CRC32 calculation). Memory usage is typically 1MB for a Writer. stdlib is in the same range. If you expect to have a lot of concurrently allocated Writers consider using the stateless compression described below. For compression performance, see: [this spreadsheet](https://docs.google.com/spreadsheets/d/1nuNE2nPfuINCZJRMt6wFWhKpToF95I47XjSsc-1rbPQ/edit?usp=sharing). To disable all assembly add `-tags=noasm`. This works across all packages. # Stateless compression This package offers stateless compression as a special option for gzip/deflate. It will do compression but without maintaining any state between Write calls. This means there will be no memory kept between Write calls, but compression and speed will be suboptimal. This is only relevant in cases where you expect to run many thousands of compressors concurrently, but with very little activity. This is *not* intended for regular web servers serving individual requests. Because of this, the size of actual Write calls will affect output size. In gzip, specify level `-3` / `gzip.StatelessCompression` to enable. For direct deflate use, NewStatelessWriter and StatelessDeflate are available. See [documentation](https://godoc.org/github.com/klauspost/compress/flate#NewStatelessWriter) A `bufio.Writer` can of course be used to control write sizes. For example, to use a 4KB buffer: ```go // replace 'ioutil.Discard' with your output. gzw, err := gzip.NewWriterLevel(ioutil.Discard, gzip.StatelessCompression) if err != nil { return err } defer gzw.Close() w := bufio.NewWriterSize(gzw, 4096) defer w.Flush() // Write to 'w' ``` This will only use up to 4KB in memory when the writer is idle. Compression is almost always worse than the fastest compression level and each write will allocate (a little) memory. # Other packages Here are other packages of good quality and pure Go (no cgo wrappers or autoconverted code): * [github.com/pierrec/lz4](https://github.com/pierrec/lz4) - strong multithreaded LZ4 compression. * [github.com/cosnicolaou/pbzip2](https://github.com/cosnicolaou/pbzip2) - multithreaded bzip2 decompression. * [github.com/dsnet/compress](https://github.com/dsnet/compress) - brotli decompression, bzip2 writer. * [github.com/ronanh/intcomp](https://github.com/ronanh/intcomp) - Integer compression. * [github.com/spenczar/fpc](https://github.com/spenczar/fpc) - Float compression. * [github.com/minio/zipindex](https://github.com/minio/zipindex) - External ZIP directory index. * [github.com/ybirader/pzip](https://github.com/ybirader/pzip) - Fast concurrent zip archiver and extractor. # license This code is licensed under the same conditions as the original Go code. See LICENSE file. --- ### Overview (`vendor/github.com/mschoch/smat/README.md`) # smat – State Machine Assisted Testing The concept is simple, describe valid uses of your library as states and actions. States describe which actions are possible, and with what probability they should occur. Actions mutate the context and transition to another state. By doing this, two things are possible: 1. Use [go-fuzz](https://github.com/dvyukov/go-fuzz) to find/test interesting sequences of operations on your library. 2. Automate longevity testing of your application by performing long sequences of valid operations. **NOTE**: both of these can also incorporate validation logic (not just failure detection by building validation into the state machine) ## Status The API is still not stable. This is brand new and we'll probably change things we don't like... [](https://travis-ci.org/mschoch/smat) [](https://coveralls.io/github/mschoch/smat?branch=master) [](https://godoc.org/github.com/mschoch/smat) [](https://codebeat.co/projects/github-com-mschoch-smat) [](https://goreportcard.com/report/github.com/mschoch/smat) ## License Apache 2.0 ## How do I use it? ### smat.Context Choose a structure to keep track of any state. You pass in an instance of this when you start, and it will be passed to every action when it executes. The actions may mutate this context. For example, consider a database library, once you open a database handle, you need to use it inside of the other actions. So you might use a structure like: ``` type context struct { db *DB } ``` ### smat.State A state represents a state that your application/library can be in, and the probabilities thats certain actions should be taken. For example, consider a database library, in a state where the database is open, there many things you can do. Let's consider just two right now, you can set a value, or you can delete a value. ``` func dbOpen(next byte) smat.ActionID { return smat.PercentExecute(next, smat.PercentAction{50, setValue}, smat.PercentAction{50, deleteValue}, ) } ``` This says that in the open state, there are two valid actions, 50% of the time you should set a value and 50% of the time you should delete a value. **NOTE**: these percentages are just for characterizing the test workload. ### smat.Action Actions are functions that do some work, optionally mutate the context, and indicate the next state to transition to. Below we see an example action to set value in a database. ``` func setValueFunc(ctx smat.Context) (next smat.State, err error) { // type assert to our custom context type context := ctx.(*context) // perform the operation err = context.db.Set("k", "v") if err != nil { return nil, err } // return the new state return dbOpen, nil } ``` ### smat.ActionID and smat.ActionMap Actions are just functions, and since we can't compare functions in Go, we need to introduce an external identifier for them. This allows us to build a bi-directional mapping which we'll take advantage of later. ``` const ( setup smat.ActionID = iota teardown setValue deleteValue ) var actionMap = smat.ActionMap{ setup: setupFunc, teardown: teardownFunc, setValue: setValueFunc, deleteValue: deleteValueFunc, } ``` ### smat.ActionSeq A common way that many users think about a library is as a sequence of actions to be performed. Using the ActionID's that we've already seen we can build up sequences of operations. ``` actionSeq := smat.ActionSeq{ open, setValue, setValue, setValue, } ``` Notice that we build these actions using the constants we defined above, and because of this we can have a bi-directional mapping between a stream of bytes (driving the state machine) and a sequence of actions to be performed. ## Fuzzing We've built a lot of pieces, lets wire it up to go-fuzz. ``` func Fuzz(data []byte) int { return smat.Fuzz(&context{}, setup, teardown, actionMap, data) } ``` * The first argument is an instance of context structure. * The second argument is the ActionID of our setup function. The setup function does not consume any of the input stream and is used to initialize the context and determine the start state. * The third argument is the teardown function. This will be called unconditionally to clean up any resources associated with the test. * The fourth argument is the actionMap which maps all ActionIDs to Actions. * The fifth argument is the data passed in from the go-fuzz application. ### Generating Initial go-fuzz Corpus Earlier we mentioned the bi-directional mapping between Actions and the byte stream driving the state machine. We can now leverage this to build the inital go-fuzz corpus. Using the `ActinSeq`s we learned about earlier we can build up a list of them as: var actionSeqs = []smat.ActionSeq{...} Then, we can write them out to disk using: ``` for i, actionSeq := range actionSeqs { byteSequence, err := actionSeq.ByteEncoding(&context{}, setup, teardown, actionMap) if err != nil { // handle error } os.MkdirAll("workdir/corpus", 0700) ioutil.WriteFile(fmt.Sprintf("workdir/corpus/%d", i), byteSequence, 0600) } ``` You can then either put this into a test case or a main application depending on your needs. ## Longevity Testing Fuzzing is great, but most of your corpus is likely to be shorter meaningful sequences. And go-fuzz works to find shortest sequences that cause problems, but sometimes you actually want to explore longer sequences that appear to go-fuzz as not triggering additional code coverage. For these cases we have another helper you can use: ``` Longevity(ctx, setup, teardown, actionMap, 0, closeChan) ``` The first four arguments are the same, the last two are: * random seed used to ensure repeatable tests * closeChan (chan struct{}) - close this channel if you want the function to stop and return ErrClosed, otherwise it will run forever ## Examples See the examples directory for a working example that tests some BoltDB functionality. --- ### Overview (`vendor/github.com/pkg/errors/README.md`) # errors [](https://travis-ci.org/pkg/errors) [](https://ci.appveyor.com/project/davecheney/errors/branch/master) [](http://godoc.org/github.com/pkg/errors) [](https://goreportcard.com/report/github.com/pkg/errors) [](https://sourcegraph.com/github.com/pkg/errors?badge) Package errors provides simple error handling primitives. `go get github.com/pkg/errors` The traditional error handling idiom in Go is roughly akin to ```go if err != nil { return err } ``` which applied recursively up the call stack results in error reports without context or debugging information. The errors package allows programmers to add context to the failure path in their code in a way that does not destroy the original value of the error. ## Adding context to an error The errors.Wrap function returns a new error that adds context to the original error. For example ```go _, err := ioutil.ReadAll(r) if err != nil { return errors.Wrap(err, "read failed") } ``` ## Retrieving the cause of an error Using `errors.Wrap` constructs a stack of errors, adding context to the preceding error. Depending on the nature of the error it may be necessary to reverse the operation of errors.Wrap to retrieve the original error for inspection. Any error value which implements this interface can be inspected by `errors.Cause`. ```go type causer interface { Cause() error } ``` `errors.Cause` will recursively retrieve the topmost error which does not implement `causer`, which is assumed to be the original cause. For example: ```go switch err := errors.Cause(err).(type) { case *MyError: // handle specifically default: // unknown error } ``` [Read the package documentation for more information](https://godoc.org/github.com/pkg/errors). ## Roadmap With the upcoming [Go2 error proposals](https://go.googlesource.com/proposal/+/master/design/go2draft.md) this package is moving into maintenance mode. The roadmap for a 1.0 release is as follows: - 0.9. Remove pre Go 1.9 and Go 1.10 support, address outstanding pull requests (if possible) - 1.0. Final release. ## Contributing Because of the Go2 errors changes, this package is not accepting proposals for new functionality. With that said, we welcome pull requests, bug fixes and issue reports. Before sending a PR, please discuss your change by raising an issue. ## License BSD-2-Clause --- ### Overview (`vendor/github.com/prometheus/procfs/README.md`) # procfs This package provides functions to retrieve system, kernel, and process metrics from the pseudo-filesystems /proc and /sys. *WARNING*: This package is a work in progress. Its API may still break in backwards-incompatible ways without warnings. Use it at your own risk. [](https://pkg.go.dev/github.com/prometheus/procfs) [](https://github.com/prometheus/procfs/actions/workflows/ci.yml) [](https://goreportcard.com/report/github.com/prometheus/procfs) ## Usage The procfs library is organized by packages based on whether the gathered data is coming from /proc, /sys, or both. Each package contains an `FS` type which represents the path to either /proc, /sys, or both. For example, cpu statistics are gathered from `/proc/stat` and are available via the root procfs package. First, the proc filesystem mount point is initialized, and then the stat information is read. ```go fs, err := procfs.NewFS("/proc") stats, err := fs.Stat() ``` Some sub-packages such as `blockdevice`, require access to both the proc and sys filesystems. ```go fs, err := blockdevice.NewFS("/proc", "/sys") stats, err := fs.ProcDiskstats() ``` ## Package Organization The packages in this project are organized according to (1) whether the data comes from the `/proc` or `/sys` filesystem and (2) the type of information being retrieved. For example, most process information can be gathered from the functions in the root `procfs` package. Information about block devices such as disk drives is available in the `blockdevices` sub-package. ## Building and Testing The procfs library is intended to be built as part of another application, so there are no distributable binaries. However, most of the API includes unit tests which can be run with `make test`. ### Updating Test Fixtures The procfs library includes a set of test fixtures which include many example files from the `/proc` and `/sys` filesystems. These fixtures are included as a [ttar](https://github.com/ideaship/ttar) file which is extracted automatically during testing. To add/update the test fixtures, first ensure the `testdata/fixtures` directory is up to date by removing the existing directory and then extracting the ttar file using `make testdata/fixtures/.unpacked` or just `make test`. ```bash rm -rf testdata/fixtures make test ``` Next, make the required changes to the extracted files in the `testdata/fixtures` directory. When the changes are complete, run `make update_fixtures` to create a new `fixtures.ttar` file based on the updated `fixtures` directory. And finally, verify the changes using `git diff testdata/fixtures.ttar`. --- ### Overview (`vendor/github.com/twmb/murmur3/README.md`) murmur3 ======= [](https://pkg.go.dev/github.com/twmb/murmur3) [](https://github.com/twmb/murmur3/actions/workflows/ci.yml) Native Go implementation of Austin Appleby's third MurmurHash revision (aka MurmurHash3). Includes 32, 64, and 128 bit sums, seeding functions, string functions that hash without converting to a slice, and streaming hashes implementing Go's standard [Hash](https://pkg.go.dev/hash#Hash) and [Cloner](https://pkg.go.dev/hash#Cloner) interfaces. This library started as a fork of [spaolacci/murmur3](https://github.com/spaolacci/murmur3). The reference algorithm has been slightly hacked as to support the streaming mode required by Go's standard Hash interface. Endianness ========== Unlike the canonical source, this library **always** reads bytes as little endian numbers. This makes the hashes portable across architectures, although does mean that hashing is a bit slower on big endian architectures. Safety ====== This library uses no `unsafe`. Bytes are read with plain indexing, which the compiler turns into single word sized loads on architectures that allow unaligned access, and the loops are shaped so that the compiler proves every index in bounds. Strings are hashed through one implementation generic over `string | []byte`, so hashing a string does not copy it and does not allocate. Assembly ======== Earlier versions shipped hand rolled amd64 assembly for the 64 and 128 bit sums. That assembly was removed once the compiler's output caught up: as of Go 1.27, the pure Go code is faster than the old assembly for every input under 64 bytes, for keys of varying length, and for every 32 bit sum, and 1 to 7 percent slower on fixed inputs of 64 bytes and more. The 32 bit assembly was removed for the same reason back in Go 1.11. See the benchmarks below. Testing ======= Testing includes comparing random inputs against the [canonical implementation](https://github.com/aappleby/smhasher/blob/master/src/MurmurHash3.cpp), and testing length 0 through 100 inputs to force the block loop, the trailing block, and all tail lengths. Because this code always reads input as little endian, testing against the canonical source is skipped for big endian architectures. The canonical source just converts bytes to numbers, meaning on big endian architectures, it will use different numbers for its hashing. Benchmarks ========== Cycles per call from perf (`cycles:u` at a fixed iteration count, minimum of three runs) on a Comet Lake i7-10710U, so CPU frequency and thermal throttling drop out. `asm` is the amd64 assembly this library shipped through v1.1.8; the other columns are this code built by that Go release. Go 1.26 is the minimum this module declares. `Sizes` rows exercise the block loop, `Branches` rows the 0 to 16 byte tail, and `RandomLengths` rows draw an xorshift length in [0, 64) on every call. ``` benchmark asm go1.26 go1.27 1.26/asm 1.27/asm 128Branches/3 25 21 21 -16.7% -16.7% 128Branches/7 26 21 21 -18.6% -18.5% 128Branches/13 25 22 22 -12.5% -14.5% 128Branches/16 27 26 26 -1.4% -1.3% 128Sizes/32 35 34 34 -2.3% -2.2% 128Sizes/64 48 48 49 +1.2% +1.2% 128Sizes/256 140 149 149 +6.9% +6.8% 128Sizes/1024 534 567 557 +6.2% +4.3% 128Sizes/8192 4215 4361 4363 +3.5% +3.5% 64Sizes/32 35 34 34 -3.7% -3.5% 64Sizes/1024 535 575 557 +7.4% +4.1% 64Sizes/8192 4221 4360 4366 +3.3% +3.4% 32Branches/3 15 16 16 +5.2% +5.6% 32Sizes/32 50 41 41 -18.2% -18.0% 32Sizes/64 89 77 77 -13.6% -13.6% 32Sizes/256 322 296 302 -7.9% -6.2% 32Sizes/1024 1284 1243 1211 -3.2% -5.7% 32Sizes/8192 10080 9896 9582 -1.8% -4.9% RandomLengths128 110 91 90 -18.0% -18.2% RandomLengths32 96 83 85 -13.5% -11.4% ``` --- ### Overview (`vendor/go.yaml.in/yaml/v3/README.md`) go.yaml.in/yaml =============== YAML Support for the Go Language ## Introduction The `yaml` package enables [Go](https://go.dev/) programs to comfortably encode and decode [YAML](https://yaml.org/) values. It was originally developed within [Canonical](https://www.canonical.com) as part of the [juju](https://juju.ubuntu.com) project, and is based on a pure Go port of the well-known [libyaml](http://pyyaml.org/wiki/LibYAML) C library to parse and generate YAML data quickly and reliably. ## Project Status This project started as a fork of the extremely popular [go-yaml]( https://github.com/go-yaml/yaml/) project, and is being maintained by the official [YAML organization]( https://github.com/yaml/). The YAML team took over ongoing maintenance and development of the project after discussion with go-yaml's author, @niemeyer, following his decision to [label the project repository as "unmaintained"]( https://github.com/go-yaml/yaml/blob/944c86a7d2/README.md) in April 2025. We have put together a team of dedicated maintainers including representatives of go-yaml's most important downstream projects. We will strive to earn the trust of the various go-yaml forks to switch back to this repository as their upstream. Please [contact us](https://cloud-native.slack.com/archives/C08PPAT8PS7) if you would like to contribute or be involved. ## Compatibility The `yaml` package supports most of YAML 1.2, but preserves some behavior from 1.1 for backwards compatibility. Specifically, v3 of the `yaml` package: * Supports YAML 1.1 bools (`yes`/`no`, `on`/`off`) as long as they are being decoded into a typed bool value. Otherwise they behave as a string. Booleans in YAML 1.2 are `true`/`false` only. * Supports octals encoded and decoded as `0777` per YAML 1.1, rather than `0o777` as specified in YAML 1.2, because most parsers still use the old format. Octals in the `0o777` format are supported though, so new files work. * Does not support base-60 floats. These are gone from YAML 1.2, and were actually never supported by this package as it's clearly a poor choice. ## Installation and Usage The import path for the package is *go.yaml.in/yaml/v3*. To install it, run: ```bash go get go.yaml.in/yaml/v3 ``` ## API Documentation See: ## API Stability The package API for yaml v3 will remain stable as described in [gopkg.in]( https://gopkg.in). ## Example ```go package main import ( "fmt" "log" "go.yaml.in/yaml/v3" ) var data = ` a: Easy! b: c: 2 d: [3, 4] ` // Note: struct fields must be public in order for unmarshal to // correctly populate the data. type T struct { A string B struct { RenamedC int `yaml:"c"` D []int `yaml:",flow"` } } func main() { t := T{} err := yaml.Unmarshal([]byte(data), &t) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- t:\n%v\n\n", t) d, err := yaml.Marshal(&t) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- t dump:\n%s\n\n", string(d)) m := make(map[interface{}]interface{}) err = yaml.Unmarshal([]byte(data), &m) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- m:\n%v\n\n", m) d, err = yaml.Marshal(&m) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- m dump:\n%s\n\n", string(d)) } ``` This example will generate the following output: ``` --- t: {Easy! {2 [3 4]}} --- t dump: a: Easy! b: c: 2 d: [3, 4] --- m: map[a:Easy! b:map[c:2 d:[3 4]]] --- m dump: a: Easy! b: c: 2 d: - 3 - 4 ``` ## License The yaml package is licensed under the MIT and Apache License 2.0 licenses. Please see the LICENSE file for details. --- ### Overview (`vendor/golang.org/x/oauth2/README.md`) # OAuth2 for Go [](https://pkg.go.dev/golang.org/x/oauth2) [](https://travis-ci.org/golang/oauth2) oauth2 package contains a client implementation for OAuth 2.0 spec. See pkg.go.dev for further documentation and examples. * [pkg.go.dev/golang.org/x/oauth2](https://pkg.go.dev/golang.org/x/oauth2) * [pkg.go.dev/golang.org/x/oauth2/google](https://pkg.go.dev/golang.org/x/oauth2/google) ## Policy for new endpoints We no longer accept new provider-specific packages in this repo if all they do is add a single endpoint variable. If you just want to add a single endpoint, add it to the [pkg.go.dev/golang.org/x/oauth2/endpoints](https://pkg.go.dev/golang.org/x/oauth2/endpoints) package. ## Report Issues / Send Patches The main issue tracker for the oauth2 repository is located at https://github.com/golang/oauth2/issues. This repository uses Gerrit for code changes. To learn how to submit changes to this repository, see https://go.dev/doc/contribute. The git repository is https://go.googlesource.com/oauth2. Note: * Excluding trivial changes, all contributions should be connected to an existing issue. * API changes must go through the [change proposal process](https://go.dev/s/proposal-process) before they can be accepted. * The code owners are listed at [dev.golang.org/owners](https://dev.golang.org/owners#:~:text=x/oauth2). --- ### Overview (`vendor/gopkg.in/natefinch/lumberjack.v2/README.md`) # lumberjack [](https://godoc.org/gopkg.in/natefinch/lumberjack.v2) [](https://travis-ci.org/natefinch/lumberjack) [](https://ci.appveyor.com/project/natefinch/lumberjack) [](https://coveralls.io/r/natefinch/lumberjack?branch=v2.0) ### Lumberjack is a Go package for writing logs to rolling files. Package lumberjack provides a rolling logger. Note that this is v2.0 of lumberjack, and should be imported using gopkg.in thusly: import "gopkg.in/natefinch/lumberjack.v2" The package name remains simply lumberjack, and the code resides at https://github.com/natefinch/lumberjack under the v2.0 branch. Lumberjack is intended to be one part of a logging infrastructure. It is not an all-in-one solution, but instead is a pluggable component at the bottom of the logging stack that simply controls the files to which logs are written. Lumberjack plays well with any logging package that can write to an io.Writer, including the standard library's log package. Lumberjack assumes that only one process is writing to the output files. Using the same lumberjack configuration from multiple processes on the same machine will result in improper behavior. **Example** To use lumberjack with the standard library's log package, just pass it into the SetOutput function when your application starts. Code: ```go log.SetOutput(&lumberjack.Logger{ Filename: "/var/log/myapp/foo.log", MaxSize: 500, // megabytes MaxBackups: 3, MaxAge: 28, //days Compress: true, // disabled by default }) ``` ## type Logger ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Logger is an io.WriteCloser that writes to the specified filename. Logger opens or creates the logfile on first Write. If the file exists and is less than MaxSize megabytes, lumberjack will open and append to that file. If the file exists and its size is >= MaxSize megabytes, the file is renamed by putting the current time in a timestamp in the name immediately before the file's extension (or the end of the filename if there's no extension). A new log file is then created using original filename. Whenever a write would cause the current log file exceed MaxSize megabytes, the current file is closed, renamed, and a new log file created with the original name. Thus, the filename you give Logger is always the "current" log file. Backups use the log file name given to Logger, in the form `name-timestamp.ext` where name is the filename without the extension, timestamp is the time at which the log was rotated formatted with the time.Time format of `2006-01-02T15-04-05.000` and the extension is the original extension. For example, if your Logger.Filename is `/var/log/foo/server.log`, a backup created at 6:30pm on Nov 11 2016 would use the filename `/var/log/foo/server-2016-11-04T18-30-00.000.log` ### Cleaning Up Old Log Files Whenever a new logfile gets created, old log files may be deleted. The most recent files according to the encoded timestamp will be retained, up to a number equal to MaxBackups (or all of them if MaxBackups is 0). Any files with an encoded timestamp older than MaxAge days are deleted, regardless of MaxBackups. Note that the time encoded in the timestamp is the rotation time, which may differ from the last time that file was written to. If MaxBackups and MaxAge are both 0, no old log files will be deleted. ### func (\*Logger) Close ``` go func (l *Logger) Close() error ``` Close implements io.Closer, and closes the current logfile. ### func (\*Logger) Rotate ``` go func (l *Logger) Rotate() error ``` Rotate causes Logger to close the existing log file and immediately create a new one. This is a helper function for applications that want to initiate rotations outside of the normal rotation rules, such as in response to SIGHUP. After rotating, this initiates a cleanup of old log files according to the normal rules. **Example** Example of how to rotate in response to SIGHUP. Code: ```go l := &lumberjack.Logger{} log.SetOutput(l) c := make(chan os.Signal, 1) signal.Notify(c, syscall.SIGHUP) go func() { for { <-c l.Rotate() } }() ``` ### func (\*Logger) Write ``` go func (l *Logger) Write(p []byte) (n int, err error) ``` Write implements io.Writer. If a write would cause the log file to be larger than MaxSize, the file is closed, renamed to include a timestamp of the current time, and a new log file is created using the original log file name. If the length of the write is greater than MaxSize, an error is returned. - - - Generated by [godoc2md](http://godoc.org/github.com/davecheney/godoc2md) --- ### Overview (`vendor/cloud.google.com/go/compute/metadata/README.md`) # Compute API [](https://pkg.go.dev/cloud.google.com/go/compute/metadata) This is a utility library for communicating with Google Cloud metadata service on Google Cloud. ## Install ```bash go get cloud.google.com/go/compute/metadata ``` ## Go Version Support See the [Go Versions Supported](https://github.com/googleapis/google-cloud-go#go-versions-supported) section in the root directory's README. ## Contributing Contributions are welcome. Please, see the [CONTRIBUTING](https://github.com/GoogleCloudPlatform/google-cloud-go/blob/main/CONTRIBUTING.md) document for details. Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See [Contributor Code of Conduct](https://github.com/GoogleCloudPlatform/google-cloud-go/blob/main/CONTRIBUTING.md#contributor-code-of-conduct) for more information. --- ### Overview (`vendor/github.com/blevesearch/vellum/levenshtein/README.md`) # levenshtein levenshtein automaton This package makes it fast and simple to build a finite determinic automaton that computes the levenshtein distance from a given string. # Sample usage: ``` // build a re-usable builder lb := NewLevenshteinAutomatonBuilder(2, false) origTerm := "couchbasefts" dfa := lb.BuildDfa("couchbases", 2) ed := dfa.eval([]byte(origTerm)) if ed.distance() != 2 { log.Errorf("expected distance 2, actual: %d", ed.distance()) } ``` This implementation is inspired by [blog post](https://fulmicoton.com/posts/levenshtein/) and is intended to be a port of original rust implementation: https://github.com/tantivy-search/levenshtein-automata Micro Benchmark Results against the current vellum/levenshtein is as below. ``` BenchmarkNewEditDistance1-8 30000 52684 ns/op 89985 B/op 295 allocs/op BenchmarkOlderEditDistance1-8 10000 132931 ns/op 588892 B/op 363 allocs/op BenchmarkNewEditDistance2-8 10000 199127 ns/op 377532 B/op 1019 allocs/op BenchmarkOlderEditDistance2-8 2000 988109 ns/op 4236609 B/op 1898 allocs/op ``` --- ### Overview (`vendor/github.com/blugelabs/ice/v2/README.md`) # ice [](https://pkg.go.dev/github.com/blugelabs/ice) [](https://github.com/blugelabs/ice/actions?query=workflow%3ATests+event%3Apush+branch%3Amaster) [](https://github.com/blugelabs/ice/actions?query=workflow%3ALint+event%3Apush+branch%3Amaster) The file is written in the reverse order that we typically access data. This helps us write in one pass since later sections of the file require file offsets of things we've already written. Current usage: - crc-32 bytes and version are in fixed position at the end of the file - reading remainder of footer could be version specific - remainder of footer gives us: - 3 important offsets (docValue, fields index and stored data index) - 2 important values (number of docs and chunk factor) - field data is processed once and memoized onto the heap so that we never have to go back to disk for it - access to stored data by doc number means first navigating to the stored data index, then accessing a fixed position offset into that slice, which gives us the actual address of the data. the first bytes of that section tell us the size of data so that we know where it ends. - access to all other indexed data follows the following pattern: - first know the field name -> convert to id - next navigate to term dictionary for that field - some operations stop here and do dictionary ops - next use dictionary to navigate to posting list for a specific term - walk posting list - if necessary, walk posting details as we go - if location info is desired, consult location bitmap to see if it is there ## stored fields section - for each document - preparation phase: - produce a slice of metadata bytes and data bytes - produce these slices in field id order - field value is appended to the data slice - metadata slice is varint encoded with the following values for each field value - field id (uint16) - field value start offset in uncompressed data slice (uint64) - field value length (uint64) - compress the data slice using snappy - file writing phase: - remember the start offset for this document - write out meta data length (varint uint64) - write out compressed data length (varint uint64) - write out the metadata bytes - write out the compressed data bytes ## stored fields idx - for each document - write start offset (remembered from previous section) of stored data (big endian uint64) With this index and a known document number, we have direct access to all the stored field data. ## posting details (freq/norm) section - for each posting list - produce a slice containing multiple consecutive chunks (each chunk is varint stream) - produce a slice remembering offsets of where each chunk starts - preparation phase: - for each hit in the posting list - if this hit is in next chunk close out encoding of last chunk and record offset start of next - encode term frequency (uint64) - encode norm factor (float32) - similarity specific implementation - file writing phase: - remember start position for this posting list details - write out number of chunks that follow (varint uint64) - write out length of each chunk (each a varint uint64) - write out the byte slice containing all the chunk data If you know the doc number you're interested in, this format lets you jump to the correct chunk (docNum/chunkFactor) directly and then seek within that chunk until you find it. ## posting details (location) section - for each posting list - produce a slice containing multiple consecutive chunks (each chunk is varint stream) - produce a slice remembering offsets of where each chunk starts - preparation phase: - for each hit in the posting list - if this hit is in next chunk close out encoding of last chunk and record offset start of next - encode field (uint16) - encode field pos (uint64) - encode field start (uint64) - encode field end (uint64) - file writing phase: - remember start position for this posting list details - write out number of chunks that follow (varint uint64) - write out length of each chunk (each a varint uint64) - write out the byte slice containing all the chunk data If you know the doc number you're interested in, this format lets you jump to the correct chunk (docNum/chunkFactor) directly and then seek within that chunk until you find it. ## postings list section - for each posting list - preparation phase: - encode roaring bitmap posting list to bytes (so we know the length) - file writing phase: - remember the start position for this posting list - write freq/norm details offset (remembered from previous, as varint uint64) - write location details offset (remembered from previous, as varint uint64) - write length of encoded roaring bitmap - write the serialized roaring bitmap data ## dictionary - for each field - preparation phase: - encode vellum FST with dictionary data pointing to file offset of posting list (remembered from previous) - file writing phase: - remember the start position of this persistDictionary - write length of vellum data (varint uint64) - write out vellum data ## fields section - for each field - file writing phase: - remember start offset for each field - write dictionary address (remembered from previous) (varint uint64) - write length of field name (varint uint64) - write field name bytes ## fields idx - for each field - file writing phase: - write big endian uint64 of start offset for each field NOTE: currently we don't know or record the length of this fields index. Instead we rely on the fact that we know it immediately precedes a footer of known size. ## fields DocValue - for each field - preparation phase: - produce a slice containing multiple consecutive chunks, where each chunk is composed of a meta section followed by compressed columnar field data - produce a slice remembering the length of each chunk - file writing phase: - remember the start position of this first field DocValue offset in the footer - write out number of chunks that follow (varint uint64) - write out length of each chunk (each a varint uint64) - write out the byte slice containing all the chunk data NOTE: currently the meta header inside each chunk gives clue to the location offsets and size of the data pertaining to a given docID and any read operation leverage that meta information to extract the document specific data from the file. ## footer - file writing phase - write number of docs (big endian uint64) - write stored field index location (big endian uint64) - write field index location (big endian uint64) - write field docValue location (big endian uint64) - write out chunk factor (big endian uint32) - write out version (big endian uint32) - write out file CRC of everything preceding this (big endian uint32) --- # ice file format diagrams ## Legend ### Sections |========| | | section |========| ### Fixed-size fields |--------| |----| |--| |-| | | uint64 | | uint32 | | uint16 | | uint8 |--------| |----| |--| |-| ### Varints |~~~~~~~~| | | varint(up to uint64) |~~~~~~~~| ### Arbitrary-length fields |--------...---| | | arbitrary-length field (string, vellum, roaring bitmap) |--------...---| ### Chunked data [--------] [ ] [--------] ## Overview Footer section describes the configuration of particular ice file. The format of footer is version-dependent, so it is necessary to check `V` field before the parsing. |==================================================| | Stored Fields | |==================================================| |-----> | Stored Fields Index | | |==================================================| | | Dictionaries + Postings + DocValues | | |==================================================| | |---> | DocValues Index | | | |==================================================| | | | Fields | | | |==================================================| | | |-> | Fields Index | | | | |========|========|========|========|====|====|====| | | | | D# | SF | F | FDV | CF | V | CC | (Footer) | | | |========|====|===|====|===|====|===|====|====|====| | | | | | | |-+-+-----------------| | | | |--------------------------| | |-------------------------------------| D#. Number of Docs. SF. Stored Fields Index Offset. F. Field Index Offset. FDV. Field DocValue Offset. CF. Chunk Factor. V. Version. CC. CRC32. ## Stored Fields Stored Fields Index is `D#` consecutive 64-bit unsigned integers - offsets, where relevant Stored Fields Data records are located. 0 [SF] [SF + D# * 8] | Stored Fields | Stored Fields Index | |================================|==================================| | | | | |--------------------| ||--------|--------|. . .|--------|| | |-> | Stored Fields Data | || 0 | 1 | | D# - 1 || | | |--------------------| ||--------|----|---|. . .|--------|| | | | | | |===|============================|==============|===================| | | |-------------------------------------------| Stored Fields Data is an arbitrary size record, which consists of metadata and [Snappy](https://github.com/golang/snappy)-compressed data. Stored Fields Data |~~~~~~~~|~~~~~~~~|~~~~~~~~...~~~~~~~~|~~~~~~~~...~~~~~~~~| | MDS | CDS | MD | CD | |~~~~~~~~|~~~~~~~~|~~~~~~~~...~~~~~~~~|~~~~~~~~...~~~~~~~~| MDS. Metadata size. CDS. Compressed data size. MD. Metadata. CD. Snappy-compressed data. ## Fields Fields Index section located between addresses `F` and `len(file) - len(footer)` and consist of `uint64` values (`F1`, `F2`, ...) which are offsets to records in Fields section. We have `F# = (len(file) - len(footer) - F) / sizeof(uint64)` fields. (...) [F] [F + F#] | Fields | Fields Index. | |==================================================|================================| | | | | |~~~~~~~~|~~~~~~~~|---...---|~~~~~~~~|~~~~~~~~|||--------|--------|...|--------|| ||->| Dict | Length | Name | # Docs |Tot Freq||| 0 | 1 | | F# - 1 || || |~~~~~~~~|~~~~~~~~|---...---|~~~~~~~~|~~~~~~~~|||--------|----|---|...|--------|| || | | | ||=================================================|==============|=================| | | |----------------------------------------------------------------| ## Dictionaries + Postings Each of fields has its own dictionary, encoded in [Vellum](https://github.com/couchbase/vellum) format. Dictionary consists of pairs `(term, offset)`, where `offset` indicates the position of postings (list of documents) for this particular term. |================================================================|- Dictionaries + | | Postings + | | DocValues | Freq/Norm (chunked) | | [~~~~~~|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~] | | |->[ Freq | Norm (float32 under varint) ] | | | [~~~~~~|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~] | | | | | |------------------------------------------------------------| | | Location Details (chunked) | | | [~~~~~~|~~~~~|~~~~~~~|~~~~~] | | | |->[ Size | Pos | Start | End ] | | | | [~~~~~~|~~~~~|~~~~~~~|~~~~~] | | | | | | | |----------------------| | | | Postings List | | | | |~~~~~~~~|~~~~~|~~|~~~~~~~~|-----------...--| | | | |->| F/N | LD | Length | ROARING BITMAP | | | | | |~~~~~|~~|~~~~~~~~|~~~~~~~~|-----------...--| | | | | |----------------------------------------------| | | |--------------------------------------| | | Dictionary | | | |~~~~~~~~|--------------------------|-...-| | | |->| Length | VELLUM DATA : (TERM -> OFFSET) | | | | |~~~~~~~~|----------------------------...-| | | | | |======|=========================================================|- DocValues Index | | | |======|=========================================================|- Fields | | | | |~~~~|~~~|~~~~~~~~|---...---| | | | Dict | Length | Name | | | |~~~~~~~~|~~~~~~~~|---...---| | | | |================================================================| ## DocValues DocValues Index is `F#` pairs of varints, one pair per field. Each pair of varints indicates start and end point of DocValues slice. |================================================================| | |------...--| | | |->| DocValues |<-| | | | |------...--| | | |==|=================|===========================================|- DocValues Index ||~|~~~~~~~~~|~~~~~~~|~~| |~~~~~~~~~~~~~~|~~~~~~~~~~~~|| || DV1 START | DV1 STOP | . . . . . | DV(F#) START | DV(F#) END || ||~~~~~~~~~~~|~~~~~~~~~~| |~~~~~~~~~~~~~~|~~~~~~~~~~~~|| |================================================================| DocValues is chunked Snappy-compressed values for each document and field. [~~~~~~~~~~~~~~~|~~~~~~|~~~~~~~~~|-...-|~~~~~~|~~~~~~~~~|--------------------...-] [ Doc# in Chunk | Doc1 | Offset1 | ... | DocN | OffsetN | SNAPPY COMPRESSED DATA ] [~~~~~~~~~~~~~~~|~~~~~~|~~~~~~~~~|-...-|~~~~~~|~~~~~~~~~|--------------------...-] Last 16 bytes are description of chunks. |~~~~~~~~~~~~...~|----------------|----------------| | Chunk Sizes | Chunk Size Arr | Chunk# | |~~~~~~~~~~~~...~|----------------|----------------| --- ### Overview (`vendor/github.com/cespare/xxhash/v2/README.md`) # xxhash [](https://pkg.go.dev/github.com/cespare/xxhash/v2) [](https://github.com/cespare/xxhash/actions/workflows/test.yml) xxhash is a Go implementation of the 64-bit [xxHash] algorithm, XXH64. This is a high-quality hashing algorithm that is much faster than anything in the Go standard library. This package provides a straightforward API: ``` func Sum64(b []byte) uint64 func Sum64String(s string) uint64 type Digest struct{ ... } func New() *Digest ``` The `Digest` type implements hash.Hash64. Its key methods are: ``` func (*Digest) Write([]byte) (int, error) func (*Digest) WriteString(string) (int, error) func (*Digest) Sum64() uint64 ``` The package is written with optimized pure Go and also contains even faster assembly implementations for amd64 and arm64. If desired, the `purego` build tag opts into using the Go code even on those architectures. [xxHash]: http://cyan4973.github.io/xxHash/ ## Compatibility This package is in a module and the latest code is in version 2 of the module. You need a version of Go with at least "minimal module compatibility" to use github.com/cespare/xxhash/v2: * 1.9.7+ for Go 1.9 * 1.10.3+ for Go 1.10 * Go 1.11 or later I recommend using the latest release of Go. ## Benchmarks Here are some quick benchmarks comparing the pure-Go and assembly implementations of Sum64. | input size | purego | asm | | ---------- | --------- | --------- | | 4 B | 1.3 GB/s | 1.2 GB/s | | 16 B | 2.9 GB/s | 3.5 GB/s | | 100 B | 6.9 GB/s | 8.1 GB/s | | 4 KB | 11.7 GB/s | 16.7 GB/s | | 10 MB | 12.0 GB/s | 17.3 GB/s | These numbers were generated on Ubuntu 20.04 with an Intel Xeon Platinum 8252C CPU using the following commands under Go 1.19.2: ``` benchstat <(go test -tags purego -benchtime 500ms -count 15 -bench 'Sum64$') benchstat <(go test -benchtime 500ms -count 15 -bench 'Sum64$') ``` ## Projects using this package - [InfluxDB](https://github.com/influxdata/influxdb) - [Prometheus](https://github.com/prometheus/prometheus) - [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) - [FreeCache](https://github.com/coocood/freecache) - [FastCache](https://github.com/VictoriaMetrics/fastcache) - [Ristretto](https://github.com/dgraph-io/ristretto) - [Badger](https://github.com/dgraph-io/badger) --- ### Overview (`vendor/github.com/dlclark/regexp2/v2/README.md`) # regexp2 - full featured regular expressions for Go Regexp2 is a feature-rich RegExp engine for Go. It doesn't have constant time guarantees like the built-in `regexp` package, but it allows backtracking and is compatible with Perl5 and .NET. You'll likely be better off with the RE2 engine from the `regexp` package and should only use this if you need to write very complex patterns or require compatibility with .NET. ## Basis of the engine The engine is ported from the .NET framework's System.Text.RegularExpressions.Regex engine. That engine was open sourced in 2015 under the MIT license. There are some fundamental differences between .NET strings and Go strings that required a bit of borrowing from the Go framework regex engine as well. I cleaned up a couple of the dirtier bits during the port (regexcharclass.cs was terrible), but the parse tree, code emmitted, and therefore patterns matched should be identical. ## New Code Generation For extra performance use `regexp2` with [`regexp2cg`](https://github.com/dlclark/regexp2cg). It is a code generation utility for `regexp2` and you can likely improve your regexp runtime performance by 3-10x in hot code paths. As always you should benchmark your specifics to confirm the results. Give it a try! ## Installing This is a go-gettable library, so install is easy: go get github.com/dlclark/regexp2/v2@latest ## Changes in v2 Version 2 includes changes that may affect compatibility with existing v1 users: * The module path is now `github.com/dlclark/regexp2/v2`, so imports need to use the `/v2` suffix. * The minimum supported Go version is now Go 1.25. * Changes to support https://github.com/dlclark/regexp2cg are merged in to support generated regex engines. * `Regexp.Split` is now available for splitting strings with regexp matches. * The new `compat` sub-package provides a [`regexp` compatibility adapter](#regexp-compatibility-adapter) with the same `Find*` and `Match*` method signatures as `regexp.Regexp`, plus a `compat.Matcher` interface that is implemented by both `*regexp.Regexp` and the adapter. * The parser, optimizer, and runner internals have changed significantly to support generated regexes and additional matching optimizations. * `Compile` and `MustCompile` now use variadic compile options for regex behavior and memory/performance tuning. See [Compile options](#compile-options) for more details. * Moved `regexp2.Debug` and `regexp2.Compile` to new `regexp2.OptionDebug()` and `regexp2.OptionIsCodeGen()` compile options. * Some types and constants in the `syntax` package have been exported or changed to support code generation. * Conceptually changed the goal of the `regexp2.ECMAScript` option to be closer to the ECMAScript standard rather than C#'s ECMAScript behavior. * Renamed the fields `Capture.Index` and `Capture.Length` to `Capture.RuneIndex` and `Capture.RuneLength` to be more clear that we're dealing with rune offsets. * Added `Capture.ByteRange()` to return the byte offset index and length of the captured text. This requires some additional processing to be done behind the scenes the first time it's called for a given capture to convert the native rune offsets to byte offsets. ## Usage Usage is similar to the Go `regexp` package. Just like in `regexp`, you start by converting a regex into a state machine via the `Compile` or `MustCompile` methods. They ultimately do the same thing, but `MustCompile` will panic if the regex is invalid. You can then use the provided `Regexp` struct to find matches repeatedly. A `Regexp` struct is safe to use across goroutines. ```go re := regexp2.MustCompile(`Your pattern`) if isMatch, _ := re.MatchString(`Something to match`); isMatch { //do something } ``` The `*Match*` methods can return a timeout error if you set the `re.MatchTimeout` field, or `ErrBacktrackingStackLimit` if a match exceeds its configured backtracking stack size. Any other error is a bug in the `regexp2` package. If you need more details about capture groups in a match then use the `FindStringMatch` method, like so: ```go if m, _ := re.FindStringMatch(`Something to match`); m != nil { // the whole match is always group 0 fmt.Printf("Group 0: %v\n", m.String()) // you can get all the groups too gps := m.Groups() // a group can be captured multiple times, so each cap is separately addressable fmt.Printf("Group 1, first capture", gps[1].Captures[0].String()) fmt.Printf("Group 1, second capture", gps[1].Captures[1].String()) } ``` Group 0 is embedded in the Match. Group 0 is an automatically-assigned group that encompasses the whole pattern. This means that `m.String()` is the same as `m.Group.String()` and `m.Groups()[0].String()` The __last__ capture is embedded in each group, so `g.String()` will return the same thing as `g.Capture.String()` and `g.Captures[len(g.Captures)-1].String()`. If you want to find multiple matches from a single input string you should use the `FindNextMatch` method. For example, to implement a function similar to `regexp.FindAllString`: ```go func regexp2FindAllString(re *regexp2.Regexp, s string) []string { var matches []string m, _ := re.FindStringMatch(s) for m != nil { matches = append(matches, m.String()) m, _ = re.FindNextMatch(m) } return matches } ``` `FindNextMatch` is optmized so that it re-uses the underlying string/rune slice. The internals of `regexp2` always operate on `[]rune` so `RuneIndex` and `RuneLength` data in a `Match` always reference a position in `rune`s rather than `byte`s (even if the input was given as a string). `ByteRange()` provides UTF-8 byte offsets, matching the original string input for string APIs. It's advisable to use the provided `String()` methods when you do not need explicit offsets. `ByteRange()` lazily caches byte offsets on the shared match text, so the first call on captures from the same match is not safe to run concurrently with other `ByteRange()` calls on that match. ## Unicode character classes `regexp2` supports Unicode character classes with `\p{...}` and negated classes with `\P{...}`. Outside ECMAScript Unicode mode, it also accepts the RE2/PCRE-style one-letter form, such as `\pL`. The class name may be a Go Unicode category, category alias, script, or property exposed by the Go standard library Unicode tables. For example: ```go letter := regexp2.MustCompile(`\p{L}+`) katakana := regexp2.MustCompile(`\p{Katakana}+`) notEmoji := regexp2.MustCompile(`\P{Emoji}+`) ``` `regexp2` also supports Unicode property selection syntax in the form `\p{property=value}`. Property and value aliases are matched loosely: case, hyphens, and underscores are ignored. For example, `\p{GCB=RI}`, `\p{grapheme_cluster_break=regional_indicator}`, and `\p{grapheme-cluster-break=regional-indicator}` all refer to the same class. Valid property names and aliases come from Unicode 17.0.0 [`PropertyAliases.txt`](https://www.unicode.org/Public/17.0.0/ucd/PropertyAliases.txt). Valid property values and aliases come from Unicode 17.0.0 [`PropertyValueAliases.txt`](https://www.unicode.org/Public/17.0.0/ucd/PropertyValueAliases.txt). The generated tables use Unicode 17.0.0 data from [`DerivedCoreProperties.txt`](https://www.unicode.org/Public/17.0.0/ucd/DerivedCoreProperties.txt), [`emoji/emoji-data.txt`](https://www.unicode.org/Public/17.0.0/ucd/emoji/emoji-data.txt), [`auxiliary/GraphemeBreakProperty.txt`](https://www.unicode.org/Public/17.0.0/ucd/auxiliary/GraphemeBreakProperty.txt), [`auxiliary/WordBreakProperty.txt`](https://www.unicode.org/Public/17.0.0/ucd/auxiliary/WordBreakProperty.txt), and [`auxiliary/SentenceBreakProperty.txt`](https://www.unicode.org/Public/17.0.0/ucd/auxiliary/SentenceBreakProperty.txt) for the package-local properties whose data changes more frequently than the Go standard library tables. ## Additional Perl and PCRE syntax The default mode supports the following syntax: * `\Q...\E` quotes every character between `\Q` and `\E`. If `\E` is omitted, quoting continues to the end of the pattern. This also works inside character classes. * `\R` matches one Unicode newline sequence: CRLF as a single sequence, or LF, VT, FF, CR, NEL, line separator, or paragraph separator. * `\X` atomically matches one Unicode 17.0.0 extended grapheme cluster. This includes combining sequences, Hangul syllables, regional-indicator pairs, emoji ZWJ sequences, and Indic conjuncts. * A `+` after a quantifier makes it possessive: `*+`, `++`, `?+`, and `{m,n}+`. For example, `a*+` is equivalent to `(?>a*)` and will not give characters back when the remainder of the pattern fails. `RE2` mode also supports `\Q...\E` and possessive quantifiers, but keeps `\R` and `\X` as literal `R` and `X` identity escapes. `ECMAScript` mode does not enable any of this syntax: `\Q`, `\E`, `\R`, and `\X` retain ECMAScript identity-escape behavior, and possessive quantifiers remain invalid. ## `regexp` compatibility adapter The `github.com/dlclark/regexp2/v2/compat` package provides an adapter for callers that want the same `Find*` and `Match*` method signatures as the standard library's `regexp.Regexp`, while still using the `regexp2` engine. ```go import ( "github.com/dlclark/regexp2/v2" "github.com/dlclark/regexp2/v2/compat" ) re := compat.MustCompile(`Your pattern`, regexp2.RE2) if re.MatchString(`Something to match`) { // do something } matches := re.FindAllString(`abc axbc`, -1) _ = matches ``` You can also wrap an existing compiled regexp: ```go base := regexp2.MustCompile(`Your pattern`) re := compat.Wrap(base) ``` The adapter includes the standard-library matching surface: `Match`, `MatchString`, `MatchReader`, and all `Find(All)?(String)?(Submatch)?(Index)?` methods. Index-returning methods use UTF-8 byte offsets like `regexp`, not regexp2's rune offsets. The package also defines `compat.Matcher`, a common interface implemented by both `*regexp.Regexp` and `*compat.Regexp`. Use it when code should accept either the standard library engine or a regexp2-backed adapter: ```go func findWords(re compat.Matcher, input string) []string { return re.FindAllString(input, -1) } ``` Because those standard-library method signatures do not return errors, the adapter panics if the wrapped regexp2 matcher returns an error such as a match timeout or `ErrBacktrackingStackLimit`. Use the main `regexp2` APIs directly when you need to handle match errors directly. ## Compile options `Compile` and `MustCompile` take variadic compile options. Most users can omit them and get default regex behavior plus bounded shared pools for rune buffers and replacement output buffers, plus per-regexp caches for parsed replacement patterns and ASCII character class bitmaps. Regex option constants can be passed directly, individually or as a bitmask: ```go re := regexp2.MustCompile(`Your pattern`, regexp2.IgnoreCase, regexp2.Singleline) re = regexp2.MustCompile(`Your pattern`, regexp2.IgnoreCase|regexp2.Singleline) ``` Performance tuning options override the default cache settings: ```go re := regexp2.MustCompile(`Your pattern`, regexp2.IgnoreCase, regexp2.OptionMaxBacktrackingStackSize(200000), regexp2.OptionMaxCachedRuneBufferLength(64*1024), regexp2.OptionMaxCachedReplacerDataEntries(8), ) ``` Compile-only options configure behavior that is not settable from the pattern: ```go re := regexp2.MustCompile(`(?This) (is)`, regexp2.OptionMaintainCaptureOrder()) ``` The defaults are intentionally bounded: | Option | Default | Used by | Working-set growth | Tradeoffs | | --- | ---: | --- | --- | --- | | `OptionMaintainCaptureOrder()` | false | Parser capture-slot assignment for mixed named and unnamed captures. | None at match time. This changes compile-time capture numbering only. | Keeps named and unnamed captures in pattern order instead of appending named captures after unnamed captures. This can change numeric backreference meaning, so it is caller-controlled rather than an inline regex option. | | `OptionDebug()` | false | Compile dumps and runner tracing. | Debug output volume only. | Useful for diagnostics, but it can produce noisy output and slower traced matching. | | `OptionIsCodeGen()` | false | Compile-time find-optimization analysis for [`regexp2cg`](https://github.com/dlclark/regexp2cg). | Per compiled regexp, during `Compile` or `MustCompile`. | Enables more expensive analysis intended for generated engines. Do not use it for normal interpreter execution; the interpreter defaults intentionally avoid this extra compile-time cost. | | `OptionMaxBacktrackingStackSize(n)` | 100,000 | The interpreter's per-match backtracking stack. | Per pooled runner. The initial allocation and subsequent growth are capped at the configured number of integer slots; the runner pool may retain stacks at their high-water size for reuse. | Lowering this bounds backtracking memory more tightly but may reject complex matches sooner with `ErrBacktrackingStackLimit`. Raising it permits deeper backtracking and increases possible memory use. A negative value disables the limit. | | `OptionMaxCachedRuneBufferLength(n)` | 256K runes | String APIs that run through pooled runners, such as `MatchString` and replacement-pattern `Replace`, when converting input strings to the engine's internal `[]rune` representation. | Process-wide shared `sync.Pool` retention by size class. This does not grow per compiled regexp or per input string; the practical working set follows recent and concurrent use across all regexps and can be dropped by GC. | Raising this lets calls use larger pooled rune buffers and can reduce allocations for repeated matches against large strings. Lowering it prevents larger buffers from being borrowed or returned, so large inputs allocate directly. | | `OptionMaxCachedReplaceBufferLength(n)` | 256 KB | Replacement-pattern `Replace` calls that build output through a shared byte buffer. | Process-wide shared `sync.Pool` retention by size class after replacement-pattern `Replace` runs. It does not grow from evaluator-based `ReplaceFunc` output and is shared across compiled regexps. | Raising this lets larger replacement outputs use pooled buffers and can reduce allocations. Lowering it prevents larger output buffers from being retained, so large replacements allocate directly. | | `OptionMaxCachedReplacerDataEntries(n)` | `16` | `Replace` with replacement pattern strings, after the replacement pattern is parsed into reusable replacement data. | Per compiled regexp. The cache grows as distinct cacheable replacement strings are used with `Replace`, up to this entry count. | Raising this helps when a single compiled regexp is used with many recurring replacement patterns. It increases per-regexp cache memory and lock-protected cache bookkeeping. Setting it to `0` disables this cache. | | `OptionMaxCachedReplacerDataBytes(n)` | 4 KB | The parsed replacement-pattern cache. Replacement strings longer than this are parsed for the call but not retained. | Per compiled regexp, combined with `OptionMaxCachedReplacerDataEntries`. Only replacement strings whose source text is at or below this size can add parsed data to the cache. | Raising this helps if large replacement patterns are reused. It can retain more memory per cached replacement. Lowering it avoids keeping unusual large replacement patterns around. | | `OptionDisableCharClassASCIIBitmap()` | false | Compile-time preparation of character classes and first-character prefix sets. By default, character classes with ASCII membership get a small bitmap used by `CharIn`. | Per compiled regexp, during `Compile` or `MustCompile`. Each eligible character class can hold one small bitmap; this does not scale with match concurrency or input size. | Leaving this false speeds up ASCII-heavy character class checks at the cost of a small amount of per-char-class memory and compile-time work. Setting to true can reduce memory for large numbers of compiled char classes in regexps, but ASCII character class matching may be slower. | For `OptionMaxBacktrackingStackSize`, set `n` to a negative value to allow unbounded stack growth. Setting it to `0` permits no backtracking stack entries, so most interpreted matches will return `ErrBacktrackingStackLimit`. For pooled buffer cache options, set `n` to `0` to disable pooling, or `-1` to allow all built-in size classes. The rune buffer classes are 1K, 4K, 16K, 64K, and 256K runes. The replacement byte buffer classes are 4 KB, 16 KB, 64 KB, 256 KB, and 1 MB. By default the 1 MB pool is unused. For replacement data byte-size cache options, `-1` means unbounded. For entry-count cache options, set `n` to `0` to disable the cache. ## Compare `regexp` and `regexp2` | Category | regexp | regexp2 | | --- | --- | --- | | Catastrophic backtracking possible | no, constant execution time guarantees | yes; backtracking stack growth is bounded by default, and `re.MatchTimeout` can also bound match duration | | Python-style capture groups `(?Pre)` | yes | no (yes in RE2 compat mode) | | .NET-style capture groups `(?re)` or `(?'name're)` | yes | yes | | comments `(?#comment)` | no | yes | | branch numbering reset `(?\|a\|b)` | no | no | | atomic group `(?>re)` | no | yes | | possessive quantifiers `*+`, `++`, `?+`, `{m,n}+` | no | yes | | literal quoting `\Q...\E` | yes | yes | | Unicode newline sequence `\R` | no | yes (default mode only) | | extended grapheme cluster `\X` | no | yes (default mode only) | | positive lookahead `(?=re)` | no | yes | | negative lookahead `(?!re)` | no | yes | | positive lookbehind `(?<=re)` | no | yes | | negative lookbehind `(?re)`) * add support for python-style named backreferences (e.g. `(?P=name)`) * change singleline behavior for `$` to only match end of string (like RE2) (see [#24](https://github.com/dlclark/regexp2/issues/24)) * change the character classes `\d` `\s` and `\w` to match the same characters as RE2. NOTE: if you also use the `ECMAScript` option then this will change the `\s` character class to match ECMAScript instead of RE2. ECMAScript allows more whitespace characters in `\s` than RE2 (but still fewer than the the default behavior). * allow character escape sequences to have defaults. For example, by default `\_` isn't a known character escape and will fail to compile, but in RE2 mode it will match the literal character `_` * support RE2-style literal quoting with `\Q...\E` * support possessive quantifiers (`*+`, `++`, `?+`, and `{m,n}+`) as a regexp2 extension ```go re := regexp2.MustCompile(`Your RE2-compatible pattern`, regexp2.RE2) if isMatch, _ := re.MatchString(`Something to match`); isMatch { //do something } ``` This feature is a work in progress and I'm open to ideas for more things to put here (maybe more relaxed character escaping rules?). ## Catastrophic Backtracking and Timeouts `regexp2` supports features that can lead to catastrophic backtracking. Each compiled regexp limits its per-match backtracking stack to 100,000 slots by default. If a match would exceed that limit, it stops and returns `ErrBacktrackingStackLimit`. Callers can identify it with `errors.Is(err, regexp2.ErrBacktrackingStackLimit)`. The limit can be changed at compile time; a negative value restores the previous unbounded behavior: ```go re := regexp2.MustCompile(pattern, regexp2.OptionMaxBacktrackingStackSize(200000)) // regexp2.OptionMaxBacktrackingStackSize(-1) disables the limit. ``` This limit bounds the interpreter's backtracking stack, not total match time or all memory used by a match. Literal empty expressions repeated any number of times are optimized away and do not consume backtracking stack space. `Regexp.MatchTimeout` can be set to limit the impact of such behavior; the match will fail with an error after approximately MatchTimeout. No timeout checks are done by default. Timeout checking is not free. The current timeout checking implementation starts a background worker that updates a clock value approximately once every 100 milliseconds. The matching code compares this value against the precomputed deadline for the match. The performance impact is as follows. 1. A match with a timeout runs almost as fast as a match without a timeout. 2. If any live matches have a timeout, there will be a background CPU load (`~0.15%` currently on a modern machine). This load will remain constant regardless of the number of matches done including matches done in parallel. 3. If no live matches are using a timeout, the background load will remain until the longest deadline (match timeout + the time when the match started) is reached. E.g., if you set a timeout of one minute the load will persist for approximately a minute even if the match finishes quickly. See [PR #58](https://github.com/dlclark/regexp2/pull/58) for more details and alternatives considered. ## Goroutine leak error If you're using a library during unit tests (e.g. https://github.com/uber-go/goleak) that validates all goroutines are exited then you'll likely get an error if you or any of your dependencies use regex's with a MatchTimeout. To remedy the problem you'll need to tell the unit test to wait until the backgroup timeout goroutine is exited. ```go func TestSomething(t *testing.T) { defer goleak.VerifyNone(t) defer regexp2.StopTimeoutClock() // ... test } //or func TestMain(m *testing.M) { // setup // ... // run m.Run() //tear down regexp2.StopTimeoutClock() goleak.VerifyNone(t) } ``` This will add ~100ms runtime to each test (or TestMain). If that's too much time you can set the clock cycle rate of the timeout goroutine in an init function in a test file. `regexp2.SetTimeoutCheckPeriod` isn't threadsafe so it must be setup before starting any regex's with Timeouts. ```go func init() { //speed up testing by making the timeout clock 1ms regexp2.SetTimeoutCheckPeriod(time.Millisecond) } ``` ## ECMAScript compatibility mode In this mode the engine attempts to match the [regex engine](https://tc39.es/ecma262/multipage/text-processing.html#sec-regexp-regular-expression-objects) described in the ECMAScript specification as closely as reasonably possible within regexp2's API and implementation. This flag should not be treated as compatibility with C#'s `RegexOptions.ECMAScript`. regexp2's ECMAScript behavior prioritizes ECMAScript specification behavior over matching the C# regex engine's interpretation of that option. Additionally a Unicode mode is provided which allows parsing of `\u{CodePoint}` syntax only when both `ECMAScript` and `Unicode` are provided. Perl/PCRE extensions `\Q...\E`, `\R`, `\X`, and possessive quantifiers are intentionally not enabled in this mode. The letter escapes retain the engine's existing ECMAScript identity-escape behavior. ## Potential bugs I've run a battery of tests against regexp2 from various sources and found the debug output matches the .NET engine, but .NET and Go handle strings very differently. I've attempted to handle these differences, but most of my testing deals with basic ASCII with a little bit of multi-byte Unicode. There's a chance that there are bugs in the string handling related to character sets with supplementary Unicode chars. Right-to-Left support is coded, but not well tested either. ## Find a bug? I'm open to new issues and pull requests with tests if you find something odd! --- ### Overview (`vendor/github.com/gofrs/uuid/v5/README.md`) # UUID [](https://github.com/gofrs/uuid/blob/master/LICENSE) [](https://github.com/gofrs/uuid/actions/workflows/go.yml) [](https://pkg.go.dev/github.com/gofrs/uuid/v5) [](https://codecov.io/gh/gofrs/uuid/) [](https://github.com/gofrs/uuid/actions/workflows/codeql.yml) [](https://www.bestpractices.dev/projects/8929) [](https://scorecard.dev/viewer/?uri=github.com/gofrs/uuid) Package uuid provides a pure Go implementation of Universally Unique Identifiers (UUID) variant as defined in RFC-9562. This package supports both the creation and parsing of UUIDs in different formats. This package supports the following UUID versions: * Version 1, based on timestamp and MAC address * Version 3, based on MD5 hashing of a named value * Version 4, based on random numbers * Version 5, based on SHA-1 hashing of a named value * Version 6, a k-sortable id based on timestamp, and field-compatible with v1 * Version 7, a k-sortable id based on timestamp * Version 8, for custom UUID implementations ## Project History This project was originally forked from the [github.com/satori/go.uuid](https://github.com/satori/go.uuid) repository after it appeared to be no longer maintained, while exhibiting [critical flaws](https://github.com/satori/go.uuid/issues/73). We have decided to take over this project to ensure it receives regular maintenance for the benefit of the larger Go community. We'd like to thank Maxim Bublis for his hard work on the original iteration of the package. ## License This source code of this package is released under the MIT License. Please see the [LICENSE](https://github.com/gofrs/uuid/blob/master/LICENSE) for the full content of the license. ## Recommended Package Version We recommend using v2.0.0+ of this package, as versions prior to 2.0.0 were created before our fork of the original package and have some known deficiencies. ## Requirements This package requires Go 1.25 or later ## Usage Here is a quick overview of how to use this package. For more detailed documentation, please see the [GoDoc Page](http://godoc.org/github.com/gofrs/uuid). ```go package main import ( "log" "github.com/gofrs/uuid/v5" ) // Create a Version 4 UUID, panicking on error. // Use this form to initialize package-level variables. var u1 = uuid.Must(uuid.NewV4()) func main() { // Create a Version 4 UUID. u2, err := uuid.NewV4() if err != nil { log.Fatalf("failed to generate UUID: %v", err) } log.Printf("generated Version 4 UUID %v", u2) // Parse a UUID from a string. s := "6ba7b810-9dad-11d1-80b4-00c04fd430c8" u3, err := uuid.FromString(s) if err != nil { log.Fatalf("failed to parse UUID %q: %v", s, err) } log.Printf("successfully parsed UUID %v", u3) } ``` ## References - [RFC-9562](https://tools.ietf.org/html/rfc9562) (replaces RFC-4122) - [DCE 1.1: Authentication and Security Services](http://pubs.opengroup.org/onlinepubs/9696989899/chap5.htm#tagcjh_08_02_01_01) --- ### Overview (`vendor/github.com/golang-jwt/jwt/v5/README.md`) # jwt-go [](https://github.com/golang-jwt/jwt/actions/workflows/build.yml) [![Go Reference](https://pkg.go.dev/badge/github.com/golang-jwt/jwt/v5.svg)](https://pkg.go.dev/github.com/golang-jwt/jwt/v5) [](https://coveralls.io/github/golang-jwt/jwt?branch=main) A [go](http://www.golang.org) (or 'golang' for search engine friendliness) implementation of [JSON Web Tokens](https://datatracker.ietf.org/doc/html/rfc7519). Starting with [v4.0.0](https://github.com/golang-jwt/jwt/releases/tag/v4.0.0) this project adds Go module support, but maintains backward compatibility with older `v3.x.y` tags and upstream `github.com/dgrijalva/jwt-go`. See the [`MIGRATION_GUIDE.md`](./MIGRATION_GUIDE.md) for more information. Version v5.0.0 introduces major improvements to the validation of tokens, but is not entirely backward compatible. > After the original author of the library suggested migrating the maintenance > of `jwt-go`, a dedicated team of open source maintainers decided to clone the > existing library into this repository. See > [dgrijalva/jwt-go#462](https://github.com/dgrijalva/jwt-go/issues/462) for a > detailed discussion on this topic. **SECURITY NOTICE:** Some older versions of Go have a security issue in the crypto/elliptic. The recommendation is to upgrade to at least 1.15 See issue [dgrijalva/jwt-go#216](https://github.com/dgrijalva/jwt-go/issues/216) for more detail. **SECURITY NOTICE:** It's important that you [validate the `alg` presented is what you expect](https://auth0.com/blog/critical-vulnerabilities-in-json-web-token-libraries/). This library attempts to make it easy to do the right thing by requiring key types to match the expected alg, but you should take the extra step to verify it in your usage. See the examples provided. ### Supported Go versions Our support of Go versions is aligned with Go's [version release policy](https://golang.org/doc/devel/release#policy). So we will support a major version of Go until there are two newer major releases. We no longer support building jwt-go with unsupported Go versions, as these contain security vulnerabilities that will not be fixed. ## What the heck is a JWT? JWT.io has [a great introduction](https://jwt.io/introduction) to JSON Web Tokens. In short, it's a signed JSON object that does something useful (for example, authentication). It's commonly used for `Bearer` tokens in Oauth 2. A token is made of three parts, separated by `.`'s. The first two parts are JSON objects, that have been [base64url](https://datatracker.ietf.org/doc/html/rfc4648) encoded. The last part is the signature, encoded the same way. The first part is called the header. It contains the necessary information for verifying the last part, the signature. For example, which encryption method was used for signing and what key was used. The part in the middle is the interesting bit. It's called the Claims and contains the actual stuff you care about. Refer to [RFC 7519](https://datatracker.ietf.org/doc/html/rfc7519) for information about reserved keys and the proper way to add your own. ## What's in the box? This library supports the parsing and verification as well as the generation and signing of JWTs. Current supported signing algorithms are HMAC SHA, RSA, RSA-PSS, and ECDSA, though hooks are present for adding your own. ## Installation Guidelines 1. To install the jwt package, you first need to have [Go](https://go.dev/doc/install) installed, then you can use the command below to add `jwt-go` as a dependency in your Go program. ```sh go get -u github.com/golang-jwt/jwt/v5 ``` 2. Import it in your code: ```go import "github.com/golang-jwt/jwt/v5" ``` ## Usage A detailed usage guide, including how to sign and verify tokens can be found on our [documentation website](https://golang-jwt.github.io/jwt/usage/create/). ## Examples See [the project documentation](https://pkg.go.dev/github.com/golang-jwt/jwt/v5) for examples of usage: * [Simple example of parsing and validating a token](https://pkg.go.dev/github.com/golang-jwt/jwt/v5#example-Parse-Hmac) * [Simple example of building and signing a token](https://pkg.go.dev/github.com/golang-jwt/jwt/v5#example-New-Hmac) * [Directory of Examples](https://pkg.go.dev/github.com/golang-jwt/jwt/v5#pkg-examples) ## Compliance This library was last reviewed to comply with [RFC 7519](https://datatracker.ietf.org/doc/html/rfc7519) dated May 2015 with a few notable differences: * In order to protect against accidental use of [Unsecured JWTs](https://datatracker.ietf.org/doc/html/rfc7519#section-6), tokens using `alg=none` will only be accepted if the constant `jwt.UnsafeAllowNoneSignatureType` is provided as the key. ## Project Status & Versioning This library is considered production ready. Feedback and feature requests are appreciated. The API should be considered stable. There should be very few backward-incompatible changes outside of major version updates (and only with good reason). This project uses [Semantic Versioning 2.0.0](http://semver.org). Accepted pull requests will land on `main`. Periodically, versions will be tagged from `main`. You can find all the releases on [the project releases page](https://github.com/golang-jwt/jwt/releases). **BREAKING CHANGES:** A full list of breaking changes is available in `VERSION_HISTORY.md`. See [`MIGRATION_GUIDE.md`](./MIGRATION_GUIDE.md) for more information on updating your code. ## Extensions This library publishes all the necessary components for adding your own signing methods or key functions. Simply implement the `SigningMethod` interface and register a factory method using `RegisterSigningMethod` or provide a `jwt.Keyfunc`. A common use case would be integrating with different 3rd party signature providers, like key management services from various cloud providers or Hardware Security Modules (HSMs) or to implement additional standards. | Extension | Purpose | Repo | | --------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | GCP | Integrates with multiple Google Cloud Platform signing tools (AppEngine, IAM API, Cloud KMS) | https://github.com/someone1/gcp-jwt-go | | AWS | Integrates with AWS Key Management Service, KMS | https://github.com/matelang/jwt-go-aws-kms | | JWKS | Provides support for JWKS ([RFC 7517](https://datatracker.ietf.org/doc/html/rfc7517)) as a `jwt.Keyfunc` | https://github.com/MicahParks/keyfunc | | TPM | Integrates with Trusted Platform Module (TPM) | https://github.com/salrashid123/golang-jwt-tpm | *Disclaimer*: Unless otherwise specified, these integrations are maintained by third parties and should not be considered as a primary offer by any of the mentioned cloud providers ## More Go package documentation can be found [on pkg.go.dev](https://pkg.go.dev/github.com/golang-jwt/jwt/v5). Additional documentation can be found on [our project page](https://golang-jwt.github.io/jwt/). The command line utility included in this project (cmd/jwt) provides a straightforward example of token creation and parsing as well as a useful tool for debugging your own integration. You'll also find several implementation examples in the documentation. [golang-jwt](https://github.com/orgs/golang-jwt) incorporates a modified version of the JWT logo, which is distributed under the terms of the [MIT License](https://github.com/jsonwebtoken/jsonwebtoken.github.io/blob/master/LICENSE.txt). --- ### Overview (`vendor/github.com/heroiclabs/sql-migrate/sqlparse/README.md`) # SQL migration parser Based on the [goose](https://bitbucket.org/liamstask/goose) migration parser. ## License This library is distributed under the [MIT](LICENSE) license. --- ### Overview (`vendor/github.com/jackc/pgx/v5/README.md`) [](https://pkg.go.dev/github.com/jackc/pgx/v5) [](https://github.com/jackc/pgx/actions/workflows/ci.yml) # pgx - PostgreSQL Driver and Toolkit pgx is a pure Go driver and toolkit for PostgreSQL. The pgx driver is a low-level, high performance interface that exposes PostgreSQL-specific features such as `LISTEN` / `NOTIFY` and `COPY`. It also includes an adapter for the standard `database/sql` interface. The toolkit component is a related set of packages that implement PostgreSQL functionality such as parsing the wire protocol and type mapping between PostgreSQL and Go. These underlying packages can be used to implement alternative drivers, proxies, load balancers, logical replication clients, etc. ## Quick Start ### Installation ```bash go get github.com/jackc/pgx/v5 ``` ### Example Usage ```go package main import ( "context" "fmt" "os" "github.com/jackc/pgx/v5" ) func main() { // urlExample := "postgres://username:password@localhost:5432/database_name" conn, err := pgx.Connect(context.Background(), os.Getenv("DATABASE_URL")) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) os.Exit(1) } defer conn.Close(context.Background()) var name string var weight int64 err = conn.QueryRow(context.Background(), "select name, weight from widgets where id=$1", 42).Scan(&name, &weight) if err != nil { fmt.Fprintf(os.Stderr, "QueryRow failed: %v\n", err) os.Exit(1) } fmt.Println(name, weight) } ``` ### Connection Configuration `pgx.Connect` and `pgxpool.New` accept PostgreSQL connection URLs (such as `postgres://user:pass@host:5432/db?sslmode=verify-full`) as well as `key=value` strings. See [`pgconn.ParseConfig`](https://pkg.go.dev/github.com/jackc/pgx/v5/pgconn#ParseConfig) and the [PostgreSQL connection string documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) for supported options and environment variables. For a step-by-step walkthrough, see the [getting started guide](https://github.com/jackc/pgx/wiki/Getting-started-with-pgx). ## Documentation Package documentation and API reference are available on [pkg.go.dev](https://pkg.go.dev/github.com/jackc/pgx/v5): * [`pgx`](https://pkg.go.dev/github.com/jackc/pgx/v5) — base PostgreSQL driver * [`pgxpool`](https://pkg.go.dev/github.com/jackc/pgx/v5/pgxpool) — concurrency-safe connection pool * [`stdlib`](https://pkg.go.dev/github.com/jackc/pgx/v5/stdlib) — `database/sql` compatibility adapter ## Features * Support for approximately 70 different PostgreSQL types * Automatic statement preparation and caching * Batch queries * Single-round trip query mode * Full TLS connection control * Binary format support for custom types (allows for much quicker encoding/decoding) * `COPY` protocol support for faster bulk data loads * Tracing and logging support * Connection pool with after-connect hook for arbitrary connection setup * `LISTEN` / `NOTIFY` * Conversion of PostgreSQL arrays to Go slice mappings for integers, floats, and strings * `hstore` support * `json` and `jsonb` support * Maps `inet` and `cidr` PostgreSQL types to `netip.Addr` and `netip.Prefix` * Large object support * NULL mapping to pointer to pointer * Supports `database/sql.Scanner` and `database/sql/driver.Valuer` interfaces for custom types * Notice response handling * Simulated nested transactions with savepoints ## Choosing Between the pgx and database/sql Interfaces The pgx interface is faster. Many PostgreSQL specific features such as `LISTEN` / `NOTIFY` and `COPY` are not available through the `database/sql` interface. The pgx interface is recommended when: 1. The application only targets PostgreSQL. 2. No other libraries that require `database/sql` are in use. It is also possible to use the `database/sql` interface and convert a connection to the lower-level pgx interface as needed. ## Development and Testing Each checkout has its own PostgreSQL 14-18 clusters and CockroachDB node, so the whole test matrix is available locally on macOS, on Linux, or in the included devcontainer. Only PostgreSQL 18 stays running by default; other targets start and stop around their tests: ```sh scripts/setup-host # native macOS (Homebrew required) or Ubuntu: host packages and mise export PATH="$HOME/.local/bin:$PATH" # if mise was just installed mise trust mise install # project tools mise run dev:init # checkout ports and certificates mise run dev # start PostgreSQL 18 and the on-demand database supervisor ./test.sh # the suite against PostgreSQL 18 ./test.sh all # every target, starting and stopping each server as needed ``` Host setup installs PostgreSQL 14-18 and Ruby build dependencies. It does not install project tools or initialize the checkout; those remain separate steps above. The devcontainer already provides the host prerequisites. See [DEVELOPMENT.md](./DEVELOPMENT.md) for the full setup, and [CONTRIBUTING.md](./CONTRIBUTING.md) for how to contribute — including how to run the tests against a PostgreSQL server you already have, without any of the above. ## Architecture See the presentation at Golang Estonia, [PGX Top to Bottom](https://www.youtube.com/watch?v=sXMSWhcHCf8) for a description of pgx architecture. ## Supported Go and PostgreSQL Versions pgx supports the same versions of Go and PostgreSQL that are supported by their respective teams. For [Go](https://golang.org/doc/devel/release.html#policy) that is the two most recent major releases and for [PostgreSQL](https://www.postgresql.org/support/versioning/) the major releases in the last 5 years. This means pgx supports Go 1.25 and higher and PostgreSQL 14 and higher. pgx also is tested against the latest version of [CockroachDB](https://www.cockroachlabs.com/product/). ## Version Policy pgx follows semantic versioning for the documented public API on stable releases. `v5` is the latest stable major version. ## PGX Family Libraries ### [github.com/jackc/pglogrepl](https://github.com/jackc/pglogrepl) pglogrepl provides functionality to act as a client for PostgreSQL logical replication. ### [github.com/jackc/pgmock](https://github.com/jackc/pgmock) pgmock offers the ability to create a server that mocks the PostgreSQL wire protocol. This is used internally to test pgx by purposely inducing unusual errors. pgproto3 and pgmock together provide most of the foundational tooling required to implement a PostgreSQL proxy or MitM (such as for a custom connection pooler). ### [github.com/jackc/tern](https://github.com/jackc/tern) tern is a stand-alone SQL migration system. ### [github.com/jackc/pgerrcode](https://github.com/jackc/pgerrcode) pgerrcode contains constants for the PostgreSQL error codes. ## Adapters for 3rd Party Types * [github.com/jackc/pgx-gofrs-uuid](https://github.com/jackc/pgx-gofrs-uuid) * [github.com/jackc/pgx-shopspring-decimal](https://github.com/jackc/pgx-shopspring-decimal) * [github.com/ColeBurch/pgx-govalues-decimal](https://github.com/ColeBurch/pgx-govalues-decimal) * [github.com/twpayne/pgx-geos](https://github.com/twpayne/pgx-geos) ([PostGIS](https://postgis.net/) and [GEOS](https://libgeos.org/) via [go-geos](https://github.com/twpayne/go-geos)) * [github.com/vgarvardt/pgx-google-uuid](https://github.com/vgarvardt/pgx-google-uuid) ## Adapters for 3rd Party Tracers * [github.com/jackhopner/pgx-xray-tracer](https://github.com/jackhopner/pgx-xray-tracer) * [github.com/exaring/otelpgx](https://github.com/exaring/otelpgx) ## Adapters for 3rd Party Loggers These adapters can be used with the tracelog package. * [github.com/jackc/pgx-go-kit-log](https://github.com/jackc/pgx-go-kit-log) * [github.com/jackc/pgx-log15](https://github.com/jackc/pgx-log15) * [github.com/jackc/pgx-logrus](https://github.com/jackc/pgx-logrus) * [github.com/jackc/pgx-zap](https://github.com/jackc/pgx-zap) * [github.com/jackc/pgx-zerolog](https://github.com/jackc/pgx-zerolog) * [github.com/mcosta74/pgx-slog](https://github.com/mcosta74/pgx-slog) * [github.com/kataras/pgx-golog](https://github.com/kataras/pgx-golog) ## 3rd Party Libraries with PGX Support ### [github.com/pashagolub/pgxmock](https://github.com/pashagolub/pgxmock) pgxmock is a mock library implementing pgx interfaces. pgxmock has one and only purpose - to simulate pgx behavior in tests, without needing a real database connection. ### [github.com/georgysavva/scany](https://github.com/georgysavva/scany) Library for scanning data from a database into Go structs and more. ### [github.com/vingarcia/ksql](https://github.com/vingarcia/ksql) A carefully designed SQL client for making using SQL easier, more productive, and less error-prone on Golang. ### [github.com/otan/gopgkrb5](https://github.com/otan/gopgkrb5) Adds GSSAPI / Kerberos authentication support. ### [github.com/wcamarao/pmx](https://github.com/wcamarao/pmx) Explicit data mapping and scanning library for Go structs and slices. ### [github.com/stephenafamo/scan](https://github.com/stephenafamo/scan) Type safe and flexible package for scanning database data into Go types. Supports, structs, maps, slices and custom mapping functions. ### [github.com/z0ne-dev/mgx](https://github.com/z0ne-dev/mgx) Code first migration library for native pgx (no database/sql abstraction). ### [github.com/amirsalarsafaei/sqlc-pgx-monitoring](https://github.com/amirsalarsafaei/sqlc-pgx-monitoring) A database monitoring/metrics library for pgx and sqlc. Trace, log and monitor your sqlc query performance using OpenTelemetry. ### [https://github.com/nikolayk812/pgx-outbox](https://github.com/nikolayk812/pgx-outbox) Simple Golang implementation for transactional outbox pattern for PostgreSQL using jackc/pgx driver. ### [https://github.com/Arlandaren/pgxWrappy](https://github.com/Arlandaren/pgxWrappy) Simplifies working with the pgx library, providing convenient scanning of nested structures. ### [https://github.com/KoNekoD/pgx-colon-query-rewriter](https://github.com/KoNekoD/pgx-colon-query-rewriter) Implementation of the pgx query rewriter to use ':' instead of '@' in named query parameters. --- ### Overview (`vendor/github.com/jackc/puddle/v2/README.md`) [](https://pkg.go.dev/github.com/jackc/puddle/v2) # Puddle Puddle is a tiny generic resource pool library for Go that uses the standard context library to signal cancellation of acquires. It is designed to contain the minimum functionality required for a resource pool. It can be used directly or it can be used as the base for a domain specific resource pool. For example, a database connection pool may use puddle internally and implement health checks and keep-alive behavior without needing to implement any concurrent code of its own. ## Features * Acquire cancellation via context standard library * Statistics API for monitoring pool pressure * No dependencies outside of standard library and golang.org/x/sync * High performance * 100% test coverage of reachable code ## Example Usage ```go package main import ( "context" "log" "net" "github.com/jackc/puddle/v2" ) func main() { constructor := func(context.Context) (net.Conn, error) { return net.Dial("tcp", "127.0.0.1:8080") } destructor := func(value net.Conn) { value.Close() } maxPoolSize := int32(10) pool, err := puddle.NewPool(&puddle.Config[net.Conn]{Constructor: constructor, Destructor: destructor, MaxSize: maxPoolSize}) if err != nil { log.Fatal(err) } // Acquire resource from the pool. res, err := pool.Acquire(context.Background()) if err != nil { log.Fatal(err) } // Use resource. _, err = res.Value().Write([]byte{1}) if err != nil { log.Fatal(err) } // Release when done. res.Release() } ``` ## Status Puddle is stable and feature complete. * Bug reports and fixes are welcome. * New features will usually not be accepted if they can be feasibly implemented in a wrapper. * Performance optimizations will usually not be accepted unless the performance issue rises to the level of a bug. ## Supported Go Versions puddle supports the same versions of Go that are supported by the Go project. For [Go](https://golang.org/doc/devel/release.html#policy) that is the two most recent major releases. This means puddle supports Go 1.19 and higher. ## License MIT --- ### Overview (`vendor/github.com/klauspost/compress/fse/README.md`) # Finite State Entropy This package provides Finite State Entropy encoding and decoding. Finite State Entropy (also referenced as [tANS](https://en.wikipedia.org/wiki/Asymmetric_numeral_systems#tANS)) encoding provides a fast near-optimal symbol encoding/decoding for byte blocks as implemented in [zstandard](https://github.com/facebook/zstd). This can be used for compressing input with a lot of similar input values to the smallest number of bytes. This does not perform any multi-byte [dictionary coding](https://en.wikipedia.org/wiki/Dictionary_coder) as LZ coders, but it can be used as a secondary step to compressors (like Snappy) that does not do entropy encoding. * [Godoc documentation](https://godoc.org/github.com/klauspost/compress/fse) ## News * Feb 2018: First implementation released. Consider this beta software for now. # Usage This package provides a low level interface that allows to compress single independent blocks. Each block is separate, and there is no built in integrity checks. This means that the caller should keep track of block sizes and also do checksums if needed. Compressing a block is done via the [`Compress`](https://godoc.org/github.com/klauspost/compress/fse#Compress) function. You must provide input and will receive the output and maybe an error. These error values can be returned: | Error | Description | |---------------------|-----------------------------------------------------------------------------| | `` | Everything ok, output is returned | | `ErrIncompressible` | Returned when input is judged to be too hard to compress | | `ErrUseRLE` | Returned from the compressor when the input is a single byte value repeated | | `(error)` | An internal error occurred. | As can be seen above there are errors that will be returned even under normal operation so it is important to handle these. To reduce allocations you can provide a [`Scratch`](https://godoc.org/github.com/klauspost/compress/fse#Scratch) object that can be re-used for successive calls. Both compression and decompression accepts a `Scratch` object, and the same object can be used for both. Be aware, that when re-using a `Scratch` object that the *output* buffer is also re-used, so if you are still using this you must set the `Out` field in the scratch to nil. The same buffer is used for compression and decompression output. Decompressing is done by calling the [`Decompress`](https://godoc.org/github.com/klauspost/compress/fse#Decompress) function. You must provide the output from the compression stage, at exactly the size you got back. If you receive an error back your input was likely corrupted. It is important to note that a successful decoding does *not* mean your output matches your original input. There are no integrity checks, so relying on errors from the decompressor does not assure your data is valid. For more detailed usage, see examples in the [godoc documentation](https://godoc.org/github.com/klauspost/compress/fse#pkg-examples). # Performance A lot of factors are affecting speed. Block sizes and compressibility of the material are primary factors. All compression functions are currently only running on the calling goroutine so only one core will be used per block. The compressor is significantly faster if symbols are kept as small as possible. The highest byte value of the input is used to reduce some of the processing, so if all your input is above byte value 64 for instance, it may be beneficial to transpose all your input values down by 64. With moderate block sizes around 64k speed are typically 200MB/s per core for compression and around 300MB/s decompression speed. The same hardware typically does Huffman (deflate) encoding at 125MB/s and decompression at 100MB/s. # Plans At one point, more internals will be exposed to facilitate more "expert" usage of the components. A streaming interface is also likely to be implemented. Likely compatible with [FSE stream format](https://github.com/Cyan4973/FiniteStateEntropy/blob/dev/programs/fileio.c#L261). # Contributing Contributions are always welcome. Be aware that adding public functions will require good justification and breaking changes will likely not be accepted. If in doubt open an issue before writing the PR. --- ### Overview (`vendor/github.com/klauspost/compress/huff0/README.md`) # Huff0 entropy compression This package provides Huff0 encoding and decoding as used in zstd. [Huff0](https://github.com/Cyan4973/FiniteStateEntropy#new-generation-entropy-coders), a Huffman codec designed for modern CPU, featuring OoO (Out of Order) operations on multiple ALU (Arithmetic Logic Unit), achieving extremely fast compression and decompression speeds. This can be used for compressing input with a lot of similar input values to the smallest number of bytes. This does not perform any multi-byte [dictionary coding](https://en.wikipedia.org/wiki/Dictionary_coder) as LZ coders, but it can be used as a secondary step to compressors (like Snappy) that does not do entropy encoding. * [Godoc documentation](https://godoc.org/github.com/klauspost/compress/huff0) ## News This is used as part of the [zstandard](https://github.com/klauspost/compress/tree/master/zstd#zstd) compression and decompression package. This ensures that most functionality is well tested. # Usage This package provides a low level interface that allows to compress single independent blocks. Each block is separate, and there is no built in integrity checks. This means that the caller should keep track of block sizes and also do checksums if needed. Compressing a block is done via the [`Compress1X`](https://godoc.org/github.com/klauspost/compress/huff0#Compress1X) and [`Compress4X`](https://godoc.org/github.com/klauspost/compress/huff0#Compress4X) functions. You must provide input and will receive the output and maybe an error. These error values can be returned: | Error | Description | |---------------------|-----------------------------------------------------------------------------| | `` | Everything ok, output is returned | | `ErrIncompressible` | Returned when input is judged to be too hard to compress | | `ErrUseRLE` | Returned from the compressor when the input is a single byte value repeated | | `ErrTooBig` | Returned if the input block exceeds the maximum allowed size (128 Kib) | | `(error)` | An internal error occurred. | As can be seen above some of there are errors that will be returned even under normal operation so it is important to handle these. To reduce allocations you can provide a [`Scratch`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch) object that can be re-used for successive calls. Both compression and decompression accepts a `Scratch` object, and the same object can be used for both. Be aware, that when re-using a `Scratch` object that the *output* buffer is also re-used, so if you are still using this you must set the `Out` field in the scratch to nil. The same buffer is used for compression and decompression output. The `Scratch` object will retain state that allows to re-use previous tables for encoding and decoding. ## Tables and re-use Huff0 allows for reusing tables from the previous block to save space if that is expected to give better/faster results. The Scratch object allows you to set a [`ReusePolicy`](https://godoc.org/github.com/klauspost/compress/huff0#ReusePolicy) that controls this behaviour. See the documentation for details. This can be altered between each block. Do however note that this information is *not* stored in the output block and it is up to the users of the package to record whether [`ReadTable`](https://godoc.org/github.com/klauspost/compress/huff0#ReadTable) should be called, based on the boolean reported back from the CompressXX call. If you want to store the table separate from the data, you can access them as `OutData` and `OutTable` on the [`Scratch`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch) object. ## Decompressing The first part of decoding is to initialize the decoding table through [`ReadTable`](https://godoc.org/github.com/klauspost/compress/huff0#ReadTable). This will initialize the decoding tables. You can supply the complete block to `ReadTable` and it will return the data part of the block which can be given to the decompressor. Decompressing is done by calling the [`Decompress1X`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch.Decompress1X) or [`Decompress4X`](https://godoc.org/github.com/klauspost/compress/huff0#Scratch.Decompress4X) function. For concurrently decompressing content with a fixed table a stateless [`Decoder`](https://godoc.org/github.com/klauspost/compress/huff0#Decoder) can be requested which will remain correct as long as the scratch is unchanged. The capacity of the provided slice indicates the expected output size. You must provide the output from the compression stage, at exactly the size you got back. If you receive an error back your input was likely corrupted. It is important to note that a successful decoding does *not* mean your output matches your original input. There are no integrity checks, so relying on errors from the decompressor does not assure your data is valid. # Contributing Contributions are always welcome. Be aware that adding public functions will require good justification and breaking changes will likely not be accepted. If in doubt open an issue before writing the PR. --- ### Overview (`vendor/github.com/klauspost/compress/zstd/README.md`) # zstd [Zstandard](https://facebook.github.io/zstd/) is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder. A high performance compression algorithm is implemented. For now focused on speed. This package provides [compression](#Compressor) to and [decompression](#Decompressor) of Zstandard content. This package is pure Go. Use `noasm` and `nounsafe` to disable relevant features. The `zstd` package is provided as open source software using a Go standard license. Currently the package is heavily optimized for 64 bit processors and will be significantly slower on 32 bit processors. For seekable zstd streams, see [this excellent package](https://github.com/SaveTheRbtz/zstd-seekable-format-go). ## Installation Install using `go get -u github.com/klauspost/compress`. The package is located in `github.com/klauspost/compress/zstd`. [](https://pkg.go.dev/github.com/klauspost/compress/zstd) ## Compressor ### Status: STABLE - there may always be subtle bugs, a wide variety of content has been tested and the library is actively used by several projects. This library is being [fuzz-tested](https://github.com/klauspost/compress-fuzz) for all updates. There may still be specific combinations of data types/size/settings that could lead to edge cases, so as always, testing is recommended. For now, a high speed (fastest) and medium-fast (default) compressor has been implemented. * The "Fastest" compression ratio is roughly equivalent to zstd level 1. * The "Default" compression ratio is roughly equivalent to zstd level 3 (default). * The "Better" compression ratio is roughly equivalent to zstd level 7. * The "Best" compression ratio is roughly equivalent to zstd level 11. In terms of speed, it is typically 2x as fast as the stdlib deflate/gzip in its fastest mode. The compression ratio compared to stdlib is around level 3, but usually 3x as fast. ### Usage An Encoder can be used for either compressing a stream via the `io.WriteCloser` interface supported by the Encoder or as multiple independent tasks via the `EncodeAll` function. Smaller encodes are encouraged to use the EncodeAll function. Use `NewWriter` to create a new instance that can be used for both. To create a writer with default options, do like this: ```Go // Compress input to output. func Compress(in io.Reader, out io.Writer) error { enc, err := zstd.NewWriter(out) if err != nil { return err } _, err = io.Copy(enc, in) if err != nil { enc.Close() return err } return enc.Close() } ``` Now you can encode by writing data to `enc`. The output will be finished writing when `Close()` is called. Even if your encode fails, you should still call `Close()` to release any resources that may be held up. The above is fine for big encodes. However, whenever possible try to *reuse* the writer. To reuse the encoder, you can use the `Reset(io.Writer)` function to change to another output. This will allow the encoder to reuse all resources and avoid wasteful allocations. By default, stream encoding has 'light' concurrency, meaning up to 2 goroutines can be working on part of a stream. This is independent of the `WithEncoderConcurrency(n)`, but that is likely to change in the future. So if you want to limit concurrency for future updates, specify the concurrency you would like. If you would like stream encoding to be done without spawning async goroutines, use `WithEncoderConcurrency(1)` which will compress input as each block is completed, blocking on writes until each has completed. #### Parallel Stream Compression For maximum throughput on large streams, use `WithConcurrentBlocks(true)` together with `WithEncoderConcurrency(n)` where n is the number of CPU cores you want to use. This splits the input into large sections (jobs) that are compressed simultaneously by multiple goroutines, similar to how the C zstd library does multithreaded compression. ```Go enc, err := zstd.NewWriter(out, zstd.WithEncoderLevel(zstd.SpeedDefault), zstd.WithEncoderConcurrency(runtime.GOMAXPROCS(0)), zstd.WithConcurrentBlocks(true), ) ``` Each non-first job receives an overlap prefix from the previous job for match context, so compression ratio is only marginally affected. Output is flushed in order, producing a valid single-frame zstd stream. Benchmark on 1.8GB GOB stream (AMD Ryzen 9 9950X): | Level | 1 thread | 4 threads | 16 threads | 1T ratio | 16T ratio | |---------|:----------:|:------------------:|:-------------------:|:--------:|:---------:| | fastest | 783 MB/s | 2950 MB/s (3.8×) | 6939 MB/s (8.9×) | 12.24% | 12.26% | | default | 728 MB/s | 2533 MB/s (3.5×) | 5340 MB/s (7.3×) | 10.67% | 10.68% | | better | 434 MB/s | 1105 MB/s (2.5×) | 2206 MB/s (5.1×) | 9.14% | 9.21% | | best | 129 MB/s | 367 MB/s (2.8×) | 884 MB/s (6.8×) | 8.48% | 8.63% | Notes: * Not compatible with dictionary encoding. * `Flush()` dispatches the current partial job, so latency-sensitive callers can force output. * `EncodeAll` is unaffected — it uses its own concurrency via the encoder pool. You can specify your desired compression level using `WithEncoderLevel()` option. Currently only pre-defined compression settings can be specified. #### Future Compatibility Guarantees This will be an evolving project. When using this package it is important to note that both the compression efficiency and speed may change. The goal will be to keep the default efficiency at the default zstd (level 3). However the encoding should never be assumed to remain the same, and you should not use hashes of compressed output for similarity checks. The Encoder can be assumed to produce the same output from the exact same code version. However, the may be modes in the future that break this, although they will not be enabled without an explicit option. This encoder is not designed to (and will probably never) output the exact same bitstream as the reference encoder. Also note, that the cgo decompressor currently does not [report all errors on invalid input](https://github.com/DataDog/zstd/issues/59), [omits error checks](https://github.com/DataDog/zstd/issues/61), [ignores checksums](https://github.com/DataDog/zstd/issues/43) and seems to ignore concatenated streams, even though [it is part of the spec](https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#frames). #### Blocks For compressing small blocks, the returned encoder has a function called `EncodeAll(src, dst []byte) []byte`. `EncodeAll` will encode all input in src and append it to dst. This function can be called concurrently. Each call will only run on a same goroutine as the caller. Encoded blocks can be concatenated and the result will be the combined input stream. Data compressed with EncodeAll can be decoded with the Decoder, using either a stream or `DecodeAll`. Especially when encoding blocks you should take special care to reuse the encoder. This will effectively make it run without allocations after a warmup period. To make it run completely without allocations, supply a destination buffer with space for all content. ```Go import "github.com/klauspost/compress/zstd" // Create a writer that caches compressors. // For this operation type we supply a nil Reader. var encoder, _ = zstd.NewWriter(nil) // Compress a buffer. // If you have a destination buffer, the allocation in the call can also be eliminated. func Compress(src []byte) []byte { return encoder.EncodeAll(src, make([]byte, 0, len(src))) } ``` You can control the maximum number of concurrent encodes using the `WithEncoderConcurrency(n)` option when creating the writer. Using the Encoder for both a stream and individual blocks concurrently is safe. ### Performance I have collected some speed examples to compare speed and compression against other compressors. * `file` is the input file. * `out` is the compressor used. `zskp` is this package. `zstd` is the Datadog cgo library. `gzstd/gzkp` is gzip standard and this library. * `level` is the compression level used. For `zskp` level 1 is "fastest", level 2 is "default"; 3 is "better", 4 is "best". * `insize`/`outsize` is the input/output size. * `millis` is the number of milliseconds used for compression. * `mb/s` is megabytes (2^20 bytes) per second. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Decompressor Status: STABLE - there may still be subtle bugs, but a wide variety of content has been tested. This library is being continuously [fuzz-tested](https://github.com/klauspost/compress-fuzz), kindly supplied by [fuzzit.dev](https://fuzzit.dev/). The main purpose of the fuzz testing is to ensure that it is not possible to crash the decoder, or run it past its limits with ANY input provided. ### Usage The package has been designed for two main usages, big streams of data and smaller in-memory buffers. There are two main usages of the package for these. Both of them are accessed by creating a `Decoder`. For streaming use a simple setup could look like this: ```Go import "github.com/klauspost/compress/zstd" func Decompress(in io.Reader, out io.Writer) error { d, err := zstd.NewReader(in) if err != nil { return err } defer d.Close() // Copy content... _, err = io.Copy(out, d) return err } ``` It is important to use the "Close" function when you no longer need the Reader to stop running goroutines, when running with default settings. Goroutines will exit once an error has been returned, including `io.EOF` at the end of a stream. Streams are decoded concurrently in 4 asynchronous stages to give the best possible throughput. However, if you prefer synchronous decompression, use `WithDecoderConcurrency(1)` which will decompress data as it is being requested only. For decoding buffers, it could look something like this: ```Go import "github.com/klauspost/compress/zstd" // Create a reader that caches decompressors. // For this operation type we supply a nil Reader. var decoder, _ = zstd.NewReader(nil, zstd.WithDecoderConcurrency(0)) // Decompress a buffer. We don't supply a destination buffer, // so it will be allocated by the decoder. func Decompress(src []byte) ([]byte, error) { return decoder.DecodeAll(src, nil) } ``` Both of these cases should provide the functionality needed. The decoder can be used for *concurrent* decompression of multiple buffers. By default 4 decompressors will be created. It will only allow a certain number of concurrent operations to run. To tweak that yourself use the `WithDecoderConcurrency(n)` option when creating the decoder. It is possible to use `WithDecoderConcurrency(0)` to create GOMAXPROCS decoders. ### Dictionaries Data compressed with [dictionaries](https://github.com/facebook/zstd#the-case-for-small-data-compression) can be decompressed. Dictionaries are added individually to Decoders. Dictionaries are generated by the `zstd --train` command and contains an initial state for the decoder. To add a dictionary use the `WithDecoderDicts(dicts ...[]byte)` option with the dictionary data. Several dictionaries can be added at once. The dictionary will be used automatically for the data that specifies them. A re-used Decoder will still contain the dictionaries registered. When registering multiple dictionaries with the same ID, the last one will be used. It is possible to use dictionaries when compressing data. To enable a dictionary use `WithEncoderDict(dict []byte)`. Here only one dictionary will be used and it will likely be used even if it doesn't improve compression. The used dictionary must be used to decompress the content. For any real gains, the dictionary should be built with similar data. If an unsuitable dictionary is used the output may be slightly larger than using no dictionary. Use the [zstd commandline tool](https://github.com/facebook/zstd/releases) to build a dictionary from sample data. For information see [zstd dictionary information](https://github.com/facebook/zstd#the-case-for-small-data-compression). For now there is a fixed startup performance penalty for compressing content with dictionaries. This will likely be improved over time. Just be aware to test performance when implementing. ### Allocation-less operation The decoder has been designed to operate without allocations after a warmup. This means that you should *store* the decoder for best performance. To re-use a stream decoder, use the `Reset(r io.Reader) error` to switch to another stream. A decoder can safely be re-used even if the previous stream failed. To release the resources, you must call the `Close()` function on a decoder. After this it can *no longer be reused*, but all running goroutines will be stopped. So you *must* use this if you will no longer need the Reader. For decompressing smaller buffers a single decoder can be used. When decoding buffers, you can supply a destination slice with length 0 and your expected capacity. In this case no unneeded allocations should be made. ### Concurrency The buffer decoder does everything on the same goroutine and does nothing concurrently. It can however decode several buffers concurrently. Use `WithDecoderConcurrency(n)` to limit that. The stream decoder will create goroutines that: 1) Reads input and splits the input into blocks. 2) Decompression of literals. 3) Decompression of sequences. 4) Reconstruction of output stream. So effectively this also means the decoder will "read ahead" and prepare data to always be available for output. The concurrency level will, for streams, determine how many blocks ahead the compression will start. Since "blocks" are quite dependent on the output of the previous block stream decoding will only have limited concurrency. In practice this means that concurrency is often limited to utilizing about 3 cores effectively. ### Benchmarks The first two are streaming decodes and the last are smaller inputs. Running on AMD Ryzen 9 3950X 16-Core Processor. AMD64 assembly used. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` This reflects the performance around May 2022, but this may be out of date. ## Zstd inside ZIP files It is possible to use zstandard to compress individual files inside zip archives. While this isn't widely supported it can be useful for internal files. To support the compression and decompression of these files you must register a compressor and decompressor. It is highly recommended registering the (de)compressors on individual zip Reader/Writer and NOT use the global registration functions. The main reason for this is that 2 registrations from different packages will result in a panic. It is a good idea to only have a single compressor and decompressor, since they can be used for multiple zip files concurrently, and using a single instance will allow reusing some resources. See [this example](https://pkg.go.dev/github.com/klauspost/compress/zstd#example-ZipCompressor) for how to compress and decompress files inside zip archives. # Contributions Contributions are always welcome. For new features/fixes, remember to add tests and for performance enhancements include benchmarks. For general feedback and experience reports, feel free to open an issue or write me on [Twitter](https://twitter.com/sh0dan). This package includes the excellent [`github.com/cespare/xxhash`](https://github.com/cespare/xxhash) package Copyright (c) 2016 Caleb Spare. --- ### Overview (`vendor/github.com/prometheus/client_golang/prometheus/README.md`) See [](https://pkg.go.dev/github.com/prometheus/client_golang/prometheus). --- ### Overview (`vendor/github.com/uber-go/tally/v4/README.md`) # :heavy_check_mark: tally [![GoDoc][doc-img]][doc] [![Build Status][ci-img]][ci] [![Coverage Status][cov-img]][cov] Fast, buffered, hierarchical stats collection in Go. ## Installation `go get -u github.com/uber-go/tally` ## Abstract Tally provides a common interface for emitting metrics, while letting you not worry about the velocity of metrics emission. By default it buffers counters, gauges and histograms at a specified interval but does not buffer timer values. This is primarily so timer values can have all their values sampled if desired and if not they can be sampled as summaries or histograms independently by a reporter. ## Structure - Scope: Keeps track of metrics, and their common metadata. - Metrics: Counters, Gauges, Timers and Histograms. - Reporter: Implemented by you. Accepts aggregated values from the scope. Forwards the aggregated values to your metrics ingestion pipeline. - The reporters already available listed alphabetically are: - `github.com/uber-go/tally/m3`: Report m3 metrics, timers are not sampled and forwarded directly. - `github.com/uber-go/tally/multi`: Report to multiple reporters, you can multi-write metrics to other reporters simply. - `github.com/uber-go/tally/prometheus`: Report prometheus metrics, timers by default are made summaries with an option to make them histograms instead. - `github.com/uber-go/tally/statsd`: Report statsd metrics, no support for tags. ### Basics - Scopes created with tally provide race-safe registration and use of all metric types `Counter`, `Gauge`, `Timer`, `Histogram`. - `NewRootScope(...)` returns a `Scope` and `io.Closer`, the second return value is used to stop the scope's goroutine reporting values from the scope to it's reporter. This is to reduce the footprint of `Scope` from the public API for those implementing it themselves to use in Go packages that take a tally `Scope`. ### Acquire a Scope ### ```go reporter = NewMyStatsReporter() // Implement as you will tags := map[string]string{ "dc": "east-1", "type": "master", } reportEvery := time.Second scope := tally.NewRootScope(tally.ScopeOptions{ Tags: tags, Reporter: reporter, }, reportEvery) ``` ### Get/Create a metric, use it ### ```go // Get a counter, increment a counter reqCounter := scope.Counter("requests") // cache me reqCounter.Inc(1) queueGauge := scope.Gauge("queue_length") // cache me queueGauge.Update(42) ``` ### Report your metrics ### Use the inbuilt statsd reporter: ```go import ( "io" "github.com/cactus/go-statsd-client/v5/statsd" "github.com/uber-go/tally" tallystatsd "github.com/uber-go/tally/statsd" // ... ) func newScope() (tally.Scope, io.Closer) { statter, _ := statsd.NewBufferedClient("127.0.0.1:8125", "stats", 100*time.Millisecond, 1440) reporter := tallystatsd.NewReporter(statter, tallystatsd.Options{ SampleRate: 1.0, }) scope, closer := tally.NewRootScope(tally.ScopeOptions{ Prefix: "my-service", Tags: map[string]string{}, Reporter: reporter, }, time.Second) return scope, closer } ``` Implement your own reporter using the `StatsReporter` interface: ```go // BaseStatsReporter implements the shared reporter methods. type BaseStatsReporter interface { Capabilities() Capabilities Flush() } // StatsReporter is a backend for Scopes to report metrics to. type StatsReporter interface { BaseStatsReporter // ReportCounter reports a counter value ReportCounter( name string, tags map[string]string, value int64, ) // ReportGauge reports a gauge value ReportGauge( name string, tags map[string]string, value float64, ) // ReportTimer reports a timer value ReportTimer( name string, tags map[string]string, interval time.Duration, ) // ReportHistogramValueSamples reports histogram samples for a bucket ReportHistogramValueSamples( name string, tags map[string]string, buckets Buckets, bucketLowerBound, bucketUpperBound float64, samples int64, ) // ReportHistogramDurationSamples reports histogram samples for a bucket ReportHistogramDurationSamples( name string, tags map[string]string, buckets Buckets, bucketLowerBound, bucketUpperBound time.Duration, samples int64, ) } ``` Or implement your own metrics implementation that matches the tally `Scope` interface to use different buffering semantics: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Performance This stuff needs to be fast. With that in mind, we avoid locks and unnecessary memory allocations. ``` BenchmarkCounterInc-8 200000000 7.68 ns/op BenchmarkReportCounterNoData-8 300000000 4.88 ns/op BenchmarkReportCounterWithData-8 100000000 21.6 ns/op BenchmarkGaugeSet-8 100000000 16.0 ns/op BenchmarkReportGaugeNoData-8 100000000 10.4 ns/op BenchmarkReportGaugeWithData-8 50000000 27.6 ns/op BenchmarkTimerInterval-8 50000000 37.7 ns/op BenchmarkTimerReport-8 300000000 5.69 ns/op ```
Released under the [MIT License](LICENSE). [doc-img]: https://godoc.org/github.com/uber-go/tally?status.svg [doc]: https://godoc.org/github.com/uber-go/tally [ci-img]: https://travis-ci.org/uber-go/tally.svg?branch=master [ci]: https://travis-ci.org/uber-go/tally [cov-img]: https://coveralls.io/repos/github/uber-go/tally/badge.svg?branch=master [cov]: https://coveralls.io/github/uber-go/tally?branch=master [glide.lock]: https://github.com/uber-go/tally/blob/master/glide.lock [v1]: https://github.com/uber-go/tally/milestones --- ### Overview (`vendor/golang.org/x/net/http2/README.md`) This package (golang.org/x/net/http2) is the original source of truth of the Go HTTP/2 implementation. As of Go 1.27, the source of truth has moved to the standard library package net/http/internal/http2. All new feature development should happen in that package. Only critical bug fixes and security fixes will be backported to x/net. The x/net package contains two implementations of the HTTP/2 transport and server: The original implementation (no longer the source of truth). A reimplementation of the x/net/http2 APIs in terms of net/http. This is called "the wrapping implementation", since it wraps net/http. The original implementation is used when the Go version is less than 1.27. The wrapping implementation is used when the Go version is at least 1.27. The build tag "http2legacy" may be set to use the original implementation. --- ### Overview (`vendor/golang.org/x/sys/unix/README.md`) # Building `sys/unix` The sys/unix package provides access to the raw system call interface of the underlying operating system. See: https://godoc.org/golang.org/x/sys/unix Porting Go to a new architecture/OS combination or adding syscalls, types, or constants to an existing architecture/OS pair requires some manual effort; however, there are tools that automate much of the process. ## Build Systems There are currently two ways we generate the necessary files. We are currently migrating the build system to use containers so the builds are reproducible. This is being done on an OS-by-OS basis. Please update this documentation as components of the build system change. ### Old Build System (currently for `GOOS != "linux"`) The old build system generates the Go files based on the C header files present on your system. This means that files for a given GOOS/GOARCH pair must be generated on a system with that OS and architecture. This also means that the generated code can differ from system to system, based on differences in the header files. To avoid this, if you are using the old build system, only generate the Go files on an installation with unmodified header files. It is also important to keep track of which version of the OS the files were generated from (ex. Darwin 14 vs Darwin 15). This makes it easier to track the progress of changes and have each OS upgrade correspond to a single change. To build the files for your current OS and architecture, make sure GOOS and GOARCH are set correctly and run `mkall.sh`. This will generate the files for your specific system. Running `mkall.sh -n` shows the commands that will be run. Requirements: bash, go ### New Build System (currently for `GOOS == "linux"`) The new build system uses a Docker container to generate the go files directly from source checkouts of the kernel and various system libraries. This means that on any platform that supports Docker, all the files using the new build system can be generated at once, and generated files will not change based on what the person running the scripts has installed on their computer. The OS specific files for the new build system are located in the `${GOOS}` directory, and the build is coordinated by the `${GOOS}/mkall.go` program. When the kernel or system library updates, modify the Dockerfile at `${GOOS}/Dockerfile` to checkout the new release of the source. To build all the files under the new build system, you must be on an amd64/Linux system and have your GOOS and GOARCH set accordingly. Running `mkall.sh` will then generate all of the files for all of the GOOS/GOARCH pairs in the new build system. Running `mkall.sh -n` shows the commands that will be run. Requirements: bash, go, docker ## Component files This section describes the various files used in the code generation process. It also contains instructions on how to modify these files to add a new architecture/OS or to add additional syscalls, types, or constants. Note that if you are using the new build system, the scripts/programs cannot be called normally. They must be called from within the docker container. ### asm files The hand-written assembly file at `asm_${GOOS}_${GOARCH}.s` implements system call dispatch. There are three entry points: ``` func Syscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr) func Syscall6(trap, a1, a2, a3, a4, a5, a6 uintptr) (r1, r2, err uintptr) func RawSyscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr) ``` The first and second are the standard ones; they differ only in how many arguments can be passed to the kernel. The third is for low-level use by the ForkExec wrapper. Unlike the first two, it does not call into the scheduler to let it know that a system call is running. When porting Go to a new architecture/OS, this file must be implemented for each GOOS/GOARCH pair. ### mksysnum Mksysnum is a Go program located at `${GOOS}/mksysnum.go` (or `mksysnum_${GOOS}.go` for the old system). This program takes in a list of header files containing the syscall number declarations and parses them to produce the corresponding list of Go numeric constants. See `zsysnum_${GOOS}_${GOARCH}.go` for the generated constants. Adding new syscall numbers is mostly done by running the build on a sufficiently new installation of the target OS (or updating the source checkouts for the new build system). However, depending on the OS, you may need to update the parsing in mksysnum. ### mksyscall.go The `syscall.go`, `syscall_${GOOS}.go`, `syscall_${GOOS}_${GOARCH}.go` are hand-written Go files which implement system calls (for unix, the specific OS, or the specific OS/Architecture pair respectively) that need special handling and list `//sys` comments giving prototypes for ones that can be generated. The mksyscall.go program takes the `//sys` and `//sysnb` comments and converts them into syscalls. This requires the name of the prototype in the comment to match a syscall number in the `zsysnum_${GOOS}_${GOARCH}.go` file. The function prototype can be exported (capitalized) or not. Adding a new syscall often just requires adding a new `//sys` function prototype with the desired arguments and a capitalized name so it is exported. However, if you want the interface to the syscall to be different, often one will make an unexported `//sys` prototype, and then write a custom wrapper in `syscall_${GOOS}.go`. ### types files For each OS, there is a hand-written Go file at `${GOOS}/types.go` (or `types_${GOOS}.go` on the old system). This file includes standard C headers and creates Go type aliases to the corresponding C types. The file is then fed through godef to get the Go compatible definitions. Finally, the generated code is fed though mkpost.go to format the code correctly and remove any hidden or private identifiers. This cleaned-up code is written to `ztypes_${GOOS}_${GOARCH}.go`. The hardest part about preparing this file is figuring out which headers to include and which symbols need to be `#define`d to get the actual data structures that pass through to the kernel system calls. Some C libraries preset alternate versions for binary compatibility and translate them on the way in and out of system calls, but there is almost always a `#define` that can get the real ones. See `types_darwin.go` and `linux/types.go` for examples. To add a new type, add in the necessary include statement at the top of the file (if it is not already there) and add in a type alias line. Note that if your type is significantly different on different architectures, you may need some `#if/#elif` macros in your include statements. ### mkerrors.sh This script is used to generate the system's various constants. This doesn't just include the error numbers and error strings, but also the signal numbers and a wide variety of miscellaneous constants. The constants come from the list of include files in the `includes_${uname}` variable. A regex then picks out the desired `#define` statements, and generates the corresponding Go constants. The error numbers and strings are generated from `#include `, and the signal numbers and strings are generated from `#include `. All of these constants are written to `zerrors_${GOOS}_${GOARCH}.go` via a C program, `_errors.c`, which prints out all the constants. To add a constant, add the header that includes it to the appropriate variable. Then, edit the regex (if necessary) to match the desired constant. Avoid making the regex too broad to avoid matching unintended constants. ### internal/mkmerge This program is used to extract duplicate const, func, and type declarations from the generated architecture-specific files listed below, and merge these into a common file for each OS. The merge is performed in the following steps: 1. Construct the set of common code that is identical in all architecture-specific files. 2. Write this common code to the merged file. 3. Remove the common code from all architecture-specific files. ## Generated files ### `zerrors_${GOOS}_${GOARCH}.go` A file containing all of the system's generated error numbers, error strings, signal numbers, and constants. Generated by `mkerrors.sh` (see above). ### `zsyscall_${GOOS}_${GOARCH}.go` A file containing all the generated syscalls for a specific GOOS and GOARCH. Generated by `mksyscall.go` (see above). ### `zsysnum_${GOOS}_${GOARCH}.go` A list of numeric constants for all the syscall number of the specific GOOS and GOARCH. Generated by mksysnum (see above). ### `ztypes_${GOOS}_${GOARCH}.go` A file containing Go types for passing into (or returning from) syscalls. Generated by godefs and the types file (see above). --- ### Overview (`vendor/google.golang.org/grpc/cmd/protoc-gen-go-grpc/README.md`) # protoc-gen-go-grpc This tool generates Go language bindings of `service`s in protobuf definition files for gRPC. For usage information, please see our [quick start guide](https://grpc.io/docs/languages/go/quickstart/). ## Future-proofing services By default, to register services using the methods generated by this tool, the service implementations must embed the corresponding `UnimplementedServer` for future compatibility. This is a behavior change from the grpc code generator previously included with `protoc-gen-go`. To restore this behavior, set the option `require_unimplemented_servers=false`. E.g.: ```sh protoc --go-grpc_out=. --go-grpc_opt=require_unimplemented_servers=false[,other options...] \ ``` Note that this is not recommended, and the option is only provided to restore backward compatibility with previously-generated code. When embedding the `UnimplementedServer` in a struct that implements the service, it should be embedded by _value_ instead of as a _pointer_. If it is embedded as a pointer, it must be assigned to a valid, non-nil pointer or else unimplemented methods would panic when called. This is tested at service registration time, and will lead to a panic in `RegisterServer` if it is not embedded properly. --- ### Overview (`vendor/github.com/blugelabs/bluge/numeric/geo/README.md`) # geo support in blube First, all of this geo code is a Go adaptation of the [Lucene 5.3.2 sandbox geo support](https://lucene.apache.org/core/5_3_2/sandbox/org/apache/lucene/util/package-summary.html). ## Notes - All of the APIs will use float64 for lon/lat values. - When describing a point in function arguments or return values, we always use the order lon, lat. - High level APIs will use TopLeft and BottomRight to describe bounding boxes. This may not map cleanly to min/max lon/lat when crossing the dateline. The lower level APIs will use min/max lon/lat and require the higher-level code to split boxes accordingly. --- ### Overview (`vendor/github.com/jackc/pgx/v5/pgconn/README.md`) # pgconn Package pgconn is a low-level PostgreSQL database driver. It operates at nearly the same level as the C library libpq. It is primarily intended to serve as the foundation for higher level libraries such as https://github.com/jackc/pgx. Applications should handle normal queries with a higher level library and only use pgconn directly when required for low-level access to PostgreSQL functionality. ## Example Usage ```go pgConn, err := pgconn.Connect(context.Background(), os.Getenv("DATABASE_URL")) if err != nil { log.Fatalln("pgconn failed to connect:", err) } defer pgConn.Close(context.Background()) result := pgConn.ExecParams(context.Background(), "SELECT email FROM users WHERE id=$1", [][]byte{[]byte("123")}, nil, nil, nil) for result.NextRow() { fmt.Println("User 123 has email:", string(result.Values()[0])) } _, err = result.Close() if err != nil { log.Fatalln("failed reading result:", err) } ``` ## Testing See CONTRIBUTING.md for setup instructions. --- ### Overview (`vendor/github.com/jackc/pgx/v5/pgproto3/README.md`) # pgproto3 Package pgproto3 is an encoder and decoder of the PostgreSQL wire protocol version 3. pgproto3 can be used as a foundation for PostgreSQL drivers, proxies, mock servers, load balancers and more. See example/pgfortune for a playful example of a fake PostgreSQL server. --- ### Overview (`vendor/github.com/stretchr/testify/internal/spew/README.md`) go-spew ======= [](http://copyfree.org) Go-spew implements a deep pretty printer for Go data structures to aid in debugging. A comprehensive suite of tests with 100% test coverage is provided to ensure proper functionality. ## License Go-spew is licensed under the [copyfree](http://copyfree.org) ISC License. --- ### Overview (`vendor/github.com/uber-go/tally/v4/prometheus/README.md`) # A buffered Prometheus reporter See `examples/prometheus_main.go` for an end to end example. ## Options You can use a specific Prometheus registry, and you can use either summaries or histograms for timers. The reporter options are: ```go // Options is a set of options for the tally reporter. type Options struct { // Registerer is the prometheus registerer to register // metrics with. Use nil to specify the default registerer. Registerer prom.Registerer // DefaultTimerType is the default type timer type to create // when using timers. It's default value is a histogram timer type. DefaultTimerType TimerType // DefaultHistogramBuckets is the default histogram buckets // to use. Use nil to specify the default histogram buckets. DefaultHistogramBuckets []float64 // DefaultSummaryObjectives is the default summary objectives // to use. Use nil to specify the default summary objectives. DefaultSummaryObjectives map[float64]float64 // OnRegisterError defines a method to call to when registering // a metric with the registerer fails. Use nil to specify // to panic by default when registering a metric fails. OnRegisterError func(err error) } ``` The timer types are: ```go // TimerType describes a type of timer type TimerType int const ( // SummaryTimerType is a timer type that reports into a summary SummaryTimerType TimerType = iota // HistogramTimerType is a timer type that reports into a histogram HistogramTimerType ) ``` You can also pre-register help description text ahead of using a metric that will be named and tagged identically with `tally`. You can also access the Prometheus HTTP handler directly. The returned reporter interface: ```go // Reporter is a Prometheus backed tally reporter. type Reporter interface { tally.CachedStatsReporter // HTTPHandler provides the Prometheus HTTP scrape handler. HTTPHandler() http.Handler // RegisterCounter is a helper method to initialize a counter // in the Prometheus backend with a given help text. // If not called explicitly, the Reporter will create one for // you on first use, with a not super helpful HELP string. RegisterCounter( name string, tagKeys []string, desc string, ) (*prom.CounterVec, error) // RegisterGauge is a helper method to initialize a gauge // in the prometheus backend with a given help text. // If not called explicitly, the Reporter will create one for // you on first use, with a not super helpful HELP string. RegisterGauge( name string, tagKeys []string, desc string, ) (*prom.GaugeVec, error) // RegisterTimer is a helper method to initialize a timer // summary or histogram vector in the prometheus backend // with a given help text. // If not called explicitly, the Reporter will create one for // you on first use, with a not super helpful HELP string. // You may pass opts as nil to get the default timer type // and objectives/buckets. // You may also pass objectives/buckets as nil in opts to // get the default objectives/buckets for the specified // timer type. RegisterTimer( name string, tagKeys []string, desc string, opts *RegisterTimerOptions, ) (TimerUnion, error) } ``` The register timer options: ```go // RegisterTimerOptions provides options when registering a timer on demand. // By default you can pass nil for the options to get the reporter defaults. type RegisterTimerOptions struct { TimerType TimerType HistogramBuckets []float64 SummaryObjectives map[float64]float64 } ``` --- ### Overview (`vendor/golang.org/x/net/internal/httpcommon/README.md`) These two packages are identical: - net/http/internal/httpcommon - golang.org/x/net/internal/httpcommon The two are kept in sync manually. Any change to one should be made in the other. --- ### Overview (`vendor/github.com/grpc-ecosystem/grpc-gateway/v2/internal/casing/README.md`) # Case conversion This package contains two functions: - `Camel` copied from the `github.com/golang/protobuf/protoc-gen-go/generator` package. - `JSONCamelCase` copied from the `github.com/protocolbuffers/protobuf-go/internal/strs` package. Both these modules are licensed by The Go Authors, as reflected in this package's [LICENSE.md]. --- ### Overview (`vendor/github.com/jackc/pgx/v5/internal/pgio/README.md`) # pgio Package pgio is a low-level toolkit building messages in the PostgreSQL wire protocol. pgio provides functions for appending integers to a []byte while doing byte order conversion. --- ### Overview (`vendor/github.com/klauspost/compress/zstd/internal/xxhash/README.md`) # xxhash VENDORED: Go to [github.com/cespare/xxhash](https://github.com/cespare/xxhash) for original package. xxhash is a Go implementation of the 64-bit [xxHash] algorithm, XXH64. This is a high-quality hashing algorithm that is much faster than anything in the Go standard library. This package provides a straightforward API: ``` func Sum64(b []byte) uint64 func Sum64String(s string) uint64 type Digest struct{ ... } func New() *Digest ``` The `Digest` type implements hash.Hash64. Its key methods are: ``` func (*Digest) Write([]byte) (int, error) func (*Digest) WriteString(string) (int, error) func (*Digest) Sum64() uint64 ``` The package is written with optimized pure Go and also contains even faster assembly implementations for amd64 and arm64. If desired, the `purego` build tag opts into using the Go code even on those architectures. [xxHash]: http://cyan4973.github.io/xxHash/ ## Compatibility This package is in a module and the latest code is in version 2 of the module. You need a version of Go with at least "minimal module compatibility" to use github.com/cespare/xxhash/v2: * 1.9.7+ for Go 1.9 * 1.10.3+ for Go 1.10 * Go 1.11 or later I recommend using the latest release of Go. ## Benchmarks Here are some quick benchmarks comparing the pure-Go and assembly implementations of Sum64. | input size | purego | asm | | ---------- | --------- | --------- | | 4 B | 1.3 GB/s | 1.2 GB/s | | 16 B | 2.9 GB/s | 3.5 GB/s | | 100 B | 6.9 GB/s | 8.1 GB/s | | 4 KB | 11.7 GB/s | 16.7 GB/s | | 10 MB | 12.0 GB/s | 17.3 GB/s | These numbers were generated on Ubuntu 20.04 with an Intel Xeon Platinum 8252C CPU using the following commands under Go 1.19.2: ``` benchstat <(go test -tags purego -benchtime 500ms -count 15 -bench 'Sum64$') benchstat <(go test -benchtime 500ms -count 15 -bench 'Sum64$') ``` ## Projects using this package - [InfluxDB](https://github.com/influxdata/influxdb) - [Prometheus](https://github.com/prometheus/prometheus) - [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) - [FreeCache](https://github.com/coocood/freecache) - [FastCache](https://github.com/VictoriaMetrics/fastcache) --- --- METRICS --- - Files Extracted: 72 - Estimated Token Budget: ~92215 tokens - Canonical Reference: https://codewiki.google/github.com/heroiclabs/nakama ## 2. Official Technical Reference & Guides (heroiclabs/nakama-docs) ## File: docs/nakama/tutorials/unity/pirate-panic/authentication.md # Authentication We will begin with one of the more important features of a multiplayer game: the ability for players to establish themselves as a user, and communicate with one another. Nakama helps us do this by making it easy to create a **centralized server**, where data is stored and processed, and allowing our players (**clients**) to connect to the server. ## Establishing a connection Here we connect the Unity client and the Nakama server. The first step is to register a new Client object upon game start in Unity. This object will be the interface for any interactions between the client and the server. Below is the snippet from the Main Menu script that initializes the client: === "Scene01MainMenuController.cs" ```csharp var client = new Client("http", "localhost", 7350, "defaultkey", UnityWebRequestAdapter.Instance); client.Timeout = 5; ``` Here, we're creating a connection to the server that you started locally (available at `http://localhost:7350`) by passing the following connection values: * `Scheme`: The connection scheme, `http` here. * `Host`: The server host, `localhost` for this example. * `Port`: The server port, set to `7350` by default. * `ServerKey`: By default `defaultkey` is used. Can be changed via the [server configuration](../../../getting-started/configuration.md#socket). ## Device authentication After establishing the server connection, we authenticate the player so we can connect them to an identity. When starting Pirate Panic, you may notice that you are automatically authenticated and assigned a username. This type of seamless authentication is device authentication, which creates accounts linked to the device that is running the game, such as a phone or computer. In Unity, device authentication can be done by fetching the Device ID and passing it into `AuthenticateDeviceAsync`: ```csharp string deviceId = SystemInfo.deviceUniqueIdentifier; session = await client.AuthenticateDeviceAsync(deviceId); ``` !!! note "Note" If building for WebGL you may have to use `System.Guid.NewGuid().ToString()` instead of `deviceUniqueIdentifier`. When authentication is complete, the player joins a ***session***, which represents the period of time a player is logged in. See [Sessions](../../../concepts/session.md) to learn more. The session object allows us to make requests and access user information as an authenticated client, primarily through the `Socket` and `Account` objects: ```csharp var socket = client.NewSocket(useMainThread: true); await socket.ConnectAsync(session); account = await client.GetAccountAsync(session); ``` In Pirate Panic, the `client`, `socket`, `account`, and `session` properties are saved in the `GameConnection` object. So any access to `_connection` later on, remember that it is referring to these properties. !!! note "Note" You can organize the server connection however you wish, just be sure it can be accessed by other classes. ## Facebook authentication Device authentication makes it easy for players to jump right into the game, but linking accounts to devices means that if the device is ever lost or reset the account is lost. Connecting via an external social provider, such as Facebook, to user accounts enables you to mitigate this issue and fetch additional user information. !!! note "Note" The Facebook SDK is included in the Pirate Panic code if you cloned the project repository. If you're starting your own project, you'll need to [get the SDK][https://developers.facebook.com/docs/unity]. Include the Facebook SDK in the scene script and initialize it as follows: ```csharp using Facebook.Unity; .. private void InitializeFacebook() { FB.Init(() => { FB.ActivateApp(); }); } ``` Then, the following snippet handles the log in: === "ProfileUpdatePanel.cs" ```csharp private void LinkFacebook() { List permissions = new List(); permissions.Add("public_profile"); FB.LogInWithReadPermissions(permissions, async result => { string facebookToken = result.AccessToken.TokenString; await _connection.Client.AuthenticateFacebookAsync(facebookToken); }); } ``` !!! note "Note" You may want to bind `LinkFacebook` to a **Connect Facebook** button so players can choose when to login rather than being forced to do so right away. Here, we're giving the `public_profile` permission so that Nakama can access basic user information like name or profile picture. You can also add other permissions and features, which can be found in the [Facebook SDK documentation][https://developers.facebook.com/docs/unity]. Logging into Facebook will automatically add a player's Facebook friends into their in-game [friends](friends.md) list. ## Initializing a new player After players have a way to log in, next is setting up the initial player information so they can start adding friends, collecting gems, completing quests, or anything else that requires storing user data over time. This can be done on the server side using a [register hook](../../../server-framework/basics.md#register-hooks). There is a different hook for each authentication method. For example, since we set up device and Facebook authentication, we should use `registerAfterAuthenticateDevice` and `registerAfterAuthenticateFacebook`. We bind a function to them that runs when the hook is triggered: === "main.ts" ```typescript initializer.registerAfterAuthenticateDevice(afterAuthenticateDeviceFn); const afterAuthenticateDeviceFn: nkruntime.AfterHookFunction = function(ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, data: nkruntime.Session, req: nkruntime.AuthenticateDeviceRequest) { afterAuthenticate(ctx, logger, nk, data); if (!data.created) { return; // Account already exists. } ... // Player initialization goes here. Write initial stats, add items to inventory, etc. } ``` ## Session tokens For security reasons, player sessions will automatically expire after the time period defined in your [Nakama configuration](../../../getting-started/configuration.md#common-properties). To avoid forcing players to constantly log back in, we can set up a method to automatically request a new session when the old one is about to expire. This is done by saving **tokens** for authentication on the client, and periodically passing these back to the server to request new tokens. There are two types of tokens: * **Access tokens**: Tell the server about the identity of the client and allows the server to trust this client. * **Refresh tokens**: Used to request new access tokens. With Nakama, we can use the `client` object to request new tokens with `SessionRefreshAsync`: === "Scene01MainMenuController.cs" ```csharp string authToken = PlayerPrefs.GetString("nakama.authToken", null); string refreshToken = PlayerPrefs.GetString("nakama.refreshToken", null); session = Session.Restore(authToken, refreshToken); // Check whether a session is close to expiry. if (session.HasExpired(DateTime.UtcNow.AddDays(1))) { try { // get a new access token session = await client.SessionRefreshAsync(session); } catch (ApiResponseException) { // get a new refresh token session = await client.AuthenticateDeviceAsync(deviceId); PlayerPrefs.SetString("nakama.refreshToken", session.RefreshToken); } PlayerPrefs.SetString("nakama.authToken", session.AuthToken); } ``` Here, we use Unity's built in `PlayerPrefs` engine to store the access token and refresh token into `nakama.authToken` and `nakama.refreshToken` respectively. If the session is less than a day away from expiring, the script will attempt to request a new access token using `SessionRefreshAsync`. If the refresh token is invalid, this will fail and we'll need to re-authenticate all over again. ## Further reading Learn more about the topics and features, and view the complete source code, discussed above: * [Authentication](../../../concepts/authentication.md) * [Session Management](../../../concepts/session-management.md) * [GameConnection.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/GameConnection.cs) * [Scene01MainMenuController.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Scene01MainMenuController.cs) * [ProfileUpdatePanel.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/Profile/ProfileUpdatePanel.cs) * [Server main.ts](https://github.com/heroiclabs/unity-sampleproject/blob/master/ServerModules/src/main.ts) --- ## File: docs/nakama/tutorials/unity/pirate-panic/clans.md # Setting Up Clans Nakama enables the ability to create groups, also known as clans, which are small communities for players to team up or hang out together. With Pirate Panic, we'll learn how to enable players to: * Create new clans * Join or disband existing clans * Search public clans * Customize shared clan information, such as the name and emblem * Chat with other members in the clan !!! note "Note" This tutorial does not cover private groups, which require new players to submit join requests before being allowed to join. You can learn more about this at the [official Nakama documentation](../../../concepts/groups-clans.md). ## Clans server module Before jumping into the Unity client implementation, let's take a look at the server side. Like other modules, we can register custom server behaviors for particular events, such as when someone creates a new clan. For groups, we're focusing on the following: - `registerAfterJoinGroup` - `registerAfterKickGroupUsers` - `registerAfterLeaveGroup` - `registerAfterPromoteGroupUsers` - `registerAfterAddFriends` - `registerBeforeDeleteGroup` See the [full list of these events](https://github.com/heroiclabs/nakama-common/blob/master/runtime/runtime.go). ### Sending notifications As an example, a function to send a notification to everyone in the clan when a new player joins would look as follows: === "clans.ts" ```typescript enum ClanNotificationCode { Refresh = 2, Delete = 3 } function sendGroupNotification(nk: nkruntime.Nakama, groupId: string, code: ClanNotificationCode, subject: string) { const members = nk.groupUsersList(groupId, 100); const count = (members.groupUsers ?? []).length; if (count < 1) { return; } const notifications: nkruntime.NotificationRequest[] = new Array(count); members.groupUsers?.forEach(user => { const n: nkruntime.NotificationRequest = { code: code, content: {}, persistent: false, subject: subject, userId: user.user.userId, } notifications.push(n); }); nk.notificationsSend(notifications); } ``` Here we create custom notification codes to differentiate different types of notifications. These can be any positive numbers. We then obtain a list of all of the members using `groupUsersList`, and for each user in the list, we'll fetch their ID using `user.user.id` and send a `NotificationRequest` to them. ### Setting up runtime hooks Now, we need to call the function we just created in a way that Nakama will automatically call it every time someone joins a group: === "clans.ts" ```typescript /** * Send an in-app notification to all clan members when a new member joins. */ const afterJoinGroupFn: nkruntime.AfterHookFunction = function( ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, data: void, request: nkruntime.JoinGroupRequest) { sendGroupNotification(nk, request.groupId ?? "", ClanNotificationCode.Refresh, "New Member Joined!"); } ``` Here we are handling a `JoinGroupRequest`, which contains a `groupId` that we can use to figure out where this notification should be sent. We can use similar code to extend the notification functionality to other events, for example, notifying everyone in the group if someone was kicked: === "clans.ts" ```typescript /** * Send an in-app notification to all clan members when one or more members are kicked. */ const afterKickGroupUsersFn: nkruntime.AfterHookFunction = function(ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, data: void, request: nkruntime.KickGroupUsersRequest) { sendGroupNotification(nk, request.groupId ?? "", ClanNotificationCode.Refresh, "Member(s) Have Been Kicked!"); } ``` ### Registering the hooks The final step in getting our functions to work as intended is to register them in our main module. For each of the functions we just created, we can add them as follows: === "main.ts" ```typescript initializer.registerAfterKickGroupUsers(afterKickGroupUsersFn); initializer.registerAfterLeaveGroup(afterLeaveGroupFn); initializer.registerAfterPromoteGroupUsers(afterPromoteGroupUsersFn); initializer.registerAfterAddFriends(afterAddFriendsFn); initializer.registerBeforeDeleteGroup(beforeDeleteGroupFn); ... ``` Here each `register` function is from the [list of runtime events][https://github.com/heroiclabs/nakama-common/blob/master/runtime/runtime.go] and the parameter we pass in is one of the functions we just created in `clans.ts`. ## Creating the client Next we'll create an interface in Unity for players to interact with the server. In Pirate Panic it will look like this: !!! note "Note" This tutorial does not cover the creation of Unity UI components in detail. See the [official Unity UI tutorial](https://learn.unity.com/tutorial/ui-components#5c7f8528edbc2a002053b4d6) to learn more about this.) Full code examples for clans can be found in the [Pirate Panic project](https://github.com/heroiclabs/unity-sampleproject/tree/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/Clans). ### Creating a clan To create a new clan we use the `CreateGroupAsync(session, name, ...)` function, where `session` is a server connection and `name` is the new clan. This function also takes in additional optional parameters which we can use to store information, such as a description or avatar image. For the Pirate Panic clan creation panel, this looks like: === "ClanCreationPanel.cs" ```csharp private async void CreateClan() { string name = _clanName.text; try { IApiGroup group = await _connection.Client.CreateGroupAsync(_connection.Session, name, "A super great clan.", _avatarImage.name); if (OnClanCreated != null) { OnClanCreated(group); } } catch (ApiResponseException e) { Debug.LogError("Error creating clan: " + e.Message); } } ``` In the snippet above: * `_clanName`: A Unity textbox where players can enter the new clan's name. We are fetching its text here. * `_connection`: The `GameConnection` object initialized earlier in the code. * Two optional parameters provided are a description and avatar name. * Once the clan is created the `OnClanCreated` function is called. This property was assigned on `Awake` in [ClansMenuUI.cs]. Finally, we wrap this async function around a function and bind that to a button action: === "ClanCreationPanel.cs" ```csharp ... _doneButton.onClick.AddListener(() => { Hide(); }); ... public override void Hide (bool isMuteSoundManager = false) { CreateClan(); base.Hide(isMuteSoundManager); } ``` ### Clan chat We can create a channel for an entire group to chat with one another. In Pirate Panic it looks like: For clan chat we pass in the clan's group ID from above when joining a chat channel, so that everyone in the clan will end up in the same channel. To join a chat channel, we use `JoinChatAsync`: === "ClansMenuUI.cs" ```csharp private async void StartChat(ClanMenuUIState state) ... channel = await _connection.Socket.JoinChatAsync(state.UserClan.Id, ChannelType.Group, persistence: true, hidden: true); ... } ``` Here we join a group channel of ID `state.UserClan.Id`. This channel is **persistent**, meaning messages get saved to the database and will show up even if you disconnect and reconnect. And the player joins as a **hidden** member, meaning they won't show up in the member listing. These settings can be configured for different types of rooms. Learn more about different settings in the [Realtime Chat documentation](../../../concepts/realtime-chat.md). Once a player has joined a chat channel, sending and receiving messages is done in the same manner as with [direct messages](friends.md#chatting-with-friends). ### Leaving or disbanding a clan When a player no longer wants to be part of their clan, there are two possible actions: **leave** or **disband**. Leaving a clan keeps it active, and can only be done if there is another `superadmin` (owner) still in the clan. Disbanding a clan will delete it from the server and remove all members. For example, to leave a clan: === "ClansMenuUI.cs" ```csharp await _connection.Client.LeaveGroupAsync(_connection.Session, _state.DisplayedClan.Id); ``` Here, `_state_` is a `ClansMenuUIState`, a custom class (defined in `ClansMenuUIState.cs`) that contains information about what is currently on the player's screen. And `DisplayedClan.Id` is the ID of the clan the player wants to leave. To disband a clan: === "ClansMenuUI.cs" ```csharp await _connection.Client.DeleteGroupAsync(_connection.Session, _state.UserClan.Id); ``` ## Further reading Learn more about the topics and features, and view the complete source code, discussed above: * [Groups and Clans](../../../concepts/groups-clans.md) * [Function Reference - Groups](../../../server-framework/function-reference.md#groups) * [Runtime list](https://github.com/heroiclabs/nakama-common/blob/master/runtime/runtime.go) * [ClanCreationPanel.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/Clans/ClanCreationPanel.cs) * [ClansMenuUI.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/ClansMenuUI.cs) * [ClansMenuUIState.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/Clans/ClanMenuUIState.cs) * [ChatChannelUI.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/Clans/Chat/ChatChannelUI.cs) * [Server main.ts](https://github.com/heroiclabs/unity-sampleproject/blob/master/ServerModules/src/main.ts) * [Server clans.ts](https://github.com/heroiclabs/unity-sampleproject/blob/master/ServerModules/src/clans.ts) --- ## File: docs/nakama/tutorials/unity/pirate-panic/friends.md # Friends A core feature of Nakama is the Friends system, which makes building individual connections within a social network more accessible. Supported by an invite system and the ability for players to keep their own friends lists, friends can easily interact by chatting with one another, competing on leaderboards, or joining games together. Here we'll learn how to use Nakama's Friends features to: * Search the player database for a specific person * Add new friends and manage invites * Direct message friends * Get messages from friends even when offline In Pirate Panic, it will look like: You will notice that code throughout this example is often split into **client-side code**, which interfaces directly with Unity, and **server-side code**, which exists in the `ServerModules` folder and is written in Typescript. This separation allows us to manage and store sensitive information in a central location, and control which parts of it can be seen by players. For example, we might need to store a database of all players and their connections on the server, but players themselves should only be able to see their own friends. ## Finding friends In order to send and receive data between the client and server, we can write RPC's (remote procedure calls). A common structure of RPC code is the need to read in a JSON string using `JSON.parse`, operate on the input, then turn it back into a string using `JSON.stringify` to return. The use of JSON allows us to standardize the format of data to make it easier to communicate between multiple engines and languages. An example of when we need to use an RPC is how to allow players to search for friends' usernames so they are able to add them. Since we don't want clients to have direct access to other players' information we handle it as follows: 1. A player types in a phrase they want to search for 2. The client sends this phrase to the server using an RPC 3. The server queries the database directly to see if there are any matches 4. The server returns the matches to the client so they can be displayed ### Registering RPCs on the server We start by implementing the RPC on the server-side so that the client will have something to call: === "main.ts" ```typescript const rpcSearchUsernameFn: nkruntime.RpcFunction = function( ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, payload: string): string { const input: any = JSON.parse(payload) // NOTE: Must be very careful with custom SQL queries to performance check them. const query = ` SELECT id, username FROM users WHERE username ILIKE concat($1, '%') ` const result = nk.sqlQuery(query, [input.username]); return JSON.stringify(result); } ``` This RPC is a function that outputs a JSON object back to the client. The [context object](../../../server-framework/basics.md#context) contains a number of useful properties that let us know where this call originated from. The Nakama instance allows us to interface with the Nakama runtime API, and the payload is a JSON object that contains any custom information sent by the client. Here, for example, a payload might look like `{username: "BobNorris101"}` if we're looking for our friend named "BobNorris101". Once we've parsed the JSON, we can then use the information to interface directly with the database using a SQL query. The call to `nk.sqlQuery` will return an object that might look like `{{id: '3ea5608a-43c3-11e7-90f9-7b9397165f34', username: 'BobNorris101'} ...}` We don't need to explicitly call this function anywhere, we just register the RPC to an event: ```typescript initializer.registerRpc('search_username', rpcSearchUsernameFn); ``` So whenever the client calls `search_username`, our function will run and automatically send the result back to the client. ### Calling RPCs on the client To call our RPC defined above on the Unity client: === "UsernameSearcher.cs" ```csharp private struct SearchResult { public string username; } ... private async void SearchUsers(string text) { ... // Do some error handling (see source for examples) // Construct a payload var searchRequest = new Dictionary() { { "username", text } }; // Convert the payload to JSON for delivery string payload = searchRequest.ToJson(); //RPC method ID from server string rpcid = "search_username"; Task searchTask; // Can also put this in a try/catch to handle errors //creating search task - sending request for running "search_username" method on server with text parameter searchTask = _connection.Client.RpcAsync(_connection.Session, rpcid, payload); //awaiting for server returning value IApiRpc searchResult = await searchTask; //unpacking results to SearchResult struct object SearchResult[] usernames = searchResult.Payload.FromJson(); ... } ``` The key function here is `RpcAsync`, which takes in the Nakama client session, the RPC identifier `search_username`, and our payload. Since this is a asynchronous network request, we have to make sure to handle the request asynchronously using `await searchTask` so that we can process the output when it arrives. Our RPC converts and returns our database query as a JSON string, so we need to convert this information to a C# object to work with it. To do this, we make a custom `SearchResult` `struct` that has the field `username` since that's the format we expect the JSON to be in. ## Adding and inviting friends Now that we have the validated username returned from the server, we can use the `AddFriendsAsync` function to add them: === "FriendsMenuUI.cs" ```csharp public async void AddFriend() { // This is from the UsernameSearcher we made earlier string[] usernames = new[] { _usernameSearcher.InputFieldValue }; await _connection.Client.AddFriendsAsync(_connection.Session, new string[] { }, usernames); var friends = await _connection.Client.ListFriendsAsync(_connection.Session); // Do stuff with friends list (friends[i].State...) ... } ... ``` Here, we can add a list of friends using either ID's or usernames. Since we use the username in this example, the ID list is left blank. After adding a friend, we then update the UI to show the new status using `ListFriendsAsync`. This function asynchronously returns a list of `IApiFriend` objects. Each `IApiFriend` contains a `User` property and a `State` property which can be used to display information about each friend. The `User` property contains all of the common information about the user (ID, username, etc.). The `State` property describes the relationship between a player and their friend: * State `0` means they are friends with each other * State `1` means A has sent an invitation, but B has not yet accepted * State `2` is the opposite of `1`: B sent an invitation to A * State `3` means that B was blocked by A Calling `AddFriendsAsync` will put a player into state `1`. The other player then needs to call `AddFriendsAsync` on their end to turn both states into `0`. Finally we need to create an interface that lets players view their invites, and ideally press an 'Accept' button to call `AddFriendsAsync` for them. The code for doing so is very similar to the code for adding a friend, and can be found in [FriendPanel.cs][https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/Friends/FriendPanel.cs] for reference. ## Blocking and unblocking friends A player can also decide to block any of their friends. This action is handled as follows: === "FriendsPanel.cs" ```csharp private async void BlockOrUnblockThisFriend() { //Checking if player status is blocked. if (!_blocked) { //if is not blocked, then block this friend string[] ids = new[] { _friend.User.Id }; await _connection.Client.BlockFriendsAsync(_connection.Session, ids); OnDataChanged(); } else { //unblock this friend await _connection.Client.DeleteFriendsAsync(_connection.Session, new[] { _friend.User.Id }); } } ``` Here, `_friend` is an `IApiFriend` object that was passed into the `FriendPanel` class when it was created, and `_blocked` is a boolean that flips whenever this friend is blocked or unblocked. The important part is the `BlockFriendsAsync` function, which takes in a list of ID's and sets all of the friend states to `3`. It might be counterintuitive why we call `DeleteFriendsAsync` to unblock. The alternative is to change the Friend State to either an invitation (`1` or `2`) or friendship (`0`). We don't want to do this so we just delete the Friend object entirely. You are then free to reinvite the friend. `DeleteFriendsAsync` can also be used in the same manner to remove a friend who is not blocked. !!! note "Note" There is a difference between blocking and banning a user. A ban is done server-side and prevents a player from joining the game entirely. Learn more about [banning](../../../concepts/friends.md#ban-a-user). ## Chatting with friends Nakama makes it easy to enable chat between friends. Chat within the server is divided into channels, each with a particular ID. If we want a private channel, we pass in the ID of the player we want to chat with (since these ID's are unique). To join a chat channel, we can use `JoinChatAsync`: === "FriendsMenuUI.cs: ```csharp private async void StartChatWithUser(string userId, string username) { IChannel chatChannel; ... chatChannel = await _connection.Socket.JoinChatAsync(userId, ChannelType.DirectMessage, persistence: true, hidden: true); ... _chatChannelUI.SetChatChannel(chatChannel); _chatChannelUI.gameObject.SetActive(true); } ``` Let's examine the parameters of the `JoinChatAsync` function: There are three main types of chat channels: * `DirectMessage`: Used for creating private rooms between two players * `Group`:Used to support private chats with multiple people in the same clan * `Room`: Creates a dynamic public chat that anybody can join. Once everyone leaves the room, the channel no longer exists until someone joins back. - You can learn more about the differences between these three channels [here](https://heroiclabs.com/docs/social-realtime-chat/#join-chat). `Persistent` means that messages get saved to the database and are available for message history listing at a later date. And `hidden` members won't show up in the member listing. Now that a player has joined a channel, they can send chat messages: === "ChatChannelUI.cs" ```csharp var content = new Dictionary(){{"content", _chatInputField.text}}.ToJson(); _connection.Socket.WriteChatMessageAsync(_chatChannel.Id, content); ``` Here, `_chatChannel` is the same channel we created with `JoinChatAsync` in the previous function. The format for the `content` we pass in comes in a dictionary where the key is a particular custom field (in this case, `content`) and the value is the information we want to send. To receive a message sent to the group, we can add a function to the `ReceivedChannelMessage` handler like so: ```csharp _connection.Socket.ReceivedChannelMessage += AddMessage; ``` Here `AddMessage` is what we want to do with the received message. For example: === "ChatChannelUI.cs" ```csharp private void AddMessage(IApiChannelMessage message) { // Manage message prefabs, etc. (not shown in this snippet) ... // If the message is from the same user as newest we should hide his username on this message bool hideUsername = (message.Username == _lastMessageUsername) && !message.Persistent; // Initializing message with ID, username, message content, time of creation, and whether or not we should hide it messageUI.InitMessage(message.MessageId, message.Username, message.Content.FromJson>()["content"], message.CreateTime, hideUsername); ... //If message is historical change order in hierarchy, the latest historical message is the oldest if (message.Persistent) { messageUI.transform.SetSiblingIndex(1); } } ``` Some constructs, like `MessagePrefab` or `messageUI`, are specific to this tutorial and can be customized further depending on your need, the above code gives an example of how we can use the `IApiChannelMessage` object passed in by the `ReceivedChannelMessage` handler. We can see that the `message` object has several attributes: * `SenderId`: The client that sent this message * `Username`: The name of the player who sent it * `Persistent`: Boolean that tells us if the message is saved or not * `CreateTime`: When the message was sent * `Content`: JSON object that can be converted to its actual contents using `message.Content.FromJson>()["content"]` ## Subscribing to channels By default, when a client disconnects and reconnects to the server (e.g. if a player quits the game), they will not be automatically re-subscribed to any chat channels. If this is something you want in your game (e.g. you want players to receive notifications when they get messages), you will need to make sure that clients re-join chat channels on start. To subscribe to all friend channels: === "Scene01MainMenuController.cs" ```csharp private async void Start() { var friends = await _connection.Client.ListFriendsAsync(_connection.Session); foreach(IApiFriend friend in friends) { var userId = friend.User.Id; await _connection.Socket.JoinChatAsync(userId, ChannelType.DirectMessage, persistence: true, hidden: true); } ... } ``` This code loops through each friend in the friends list on app start, and joins each friend's direct message channel so that any chats delivered by any friend will be received automatically. ## Further reading Learn more about the topics and features, and view the complete source code, discussed above: * [Friends](../../../concepts/friends.md) * [Realtime Chat](../../../concepts/realtime-chat.md) * [UsernameSearcher.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/Friends/UsernameSearcher.cs) * [FriendsMenuUI.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/FriendsMenuUI.cs) * [FriendPanel.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/Friends/FriendPanel.cs) * [ChatChannelUI.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/Clans/Chat/ChatChannelUI.cs) * [Server main.ts](https://github.com/heroiclabs/unity-sampleproject/blob/master/ServerModules/src/main.ts) --- ## File: docs/nakama/tutorials/unity/pirate-panic/intro.md # Pirate Panic Tutorial Welcome to the Pirate Panic tutorial! In this tutorial, we will cover all of the major Nakama features implemented in [Pirate Panic](https://github.com/heroiclabs/unity-sampleproject). This tutorial is split into multiple sections, each exploring a different feature of Nakama and how it can be used with Unity in the context of Pirate Panic. This resource provides an example of Nakama in a full-scale project and is meant to help you adapt the big ideas and code snippets easily into your own games! ## Prerequisites So that you can follow along with this tutorial, please ensure you have done the following before proceeding: * [Installed Nakama](../../..//getting-started/docker-quickstart.md) * [Download](https://unity3d.com/get-unity/download) and install Unity * [Install the Nakama Unity SDK](https://github.com/heroiclabs/nakama-unity) * [Download the Pirate Panic Unity project]((https://github.com/heroiclabs/unity-sampleproject)) ## Topics This tutorial is composed of the following sections: * [Authentication](authentication.md) * [Friends and Chat](friends.md) * [Clans](clans.md) * [Storage](storage.md) * [Matchmaking](matchmaking.md) * [Realtime Multiplayer](realtime-multiplayer.md) * [Leaderboards](leaderboards.md) * [Notifications](notifications.md) ## Source files All of the files referenced in this tutorial can be found in our [Unity Sample Project](https://github.com/heroiclabs/unity-sampleproject) repository. --- ## File: docs/nakama/tutorials/unity/pirate-panic/leaderboards.md # Leaderboards We have set up matches, friends, clans, and ways to store data. Let's put them all together to display a [leaderboard](../../../concepts/leaderboards.md) of top players to foster some competition. ## Setting up the leaderboard For each of the leaderboards we want to create, we run `leaderboardCreate`. This function takes in several parameters: * The ID/name * Boolean for if the leaderboard is authoritative (if yes, then only the server is allowed to update the leaderboard) * Sort order (`SortOrder.ASCENDING` or `SortOrder.DESCENDING`) * Operator to aggregate scores * `Operator.BEST`: Always taking the best score submitted * `Operator.SET`: Always taking the latest score submitted * `Operator.INCR`: Adding submitted scores to the total * Reset schedule, which is a [CRON expression](https://en.wikipedia.org/wiki/Cron) or `null` for a leaderboard that never resets * Metadata to store within the leaderboard For Pirate Panic, we set up a global leaderboard that shows the best high score of all time from all players: === "main.ts" ```typescript const id = "global_leaderboard"; const authoritative = false; const metadata = {}; const scoreOperator = nkruntime.Operator.BEST; const sortOrder = nkruntime.SortOrder.DESCENDING; const resetSchedule = null; nk.leaderboardCreate(id, authoritative, sortOrder, scoreOperator, resetSchedule, metadata); ``` Now the leaderboard is ready to accept new records. ## Adding to the Leaderboard After every match ends we will record the player scores to the leaderboard. Whenever a match is over the client sends an RPC request to the server to handle end-game behaviors: === "LeaderboardsMenuUI.cs" ```typescript interface MatchEndRequest { // Create a structure to align the payload to matchId : string; placement: MatchEndPlacement; time : number; towersDestroyed : number; } const rpcHandleMatchEnd: nkruntime.RpcFunction = function(ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, payload: string): string { ... // Add gems to wallet, calculate score, etc. etc. let request : MatchEndRequest = JSON.parse(payload); let score = calculateScore(request.placement == MatchEndPlacement.Winner, request.towersDestroyed, request.time); nk.leaderboardRecordWrite(globalLeaderboard, ctx.userId, ctx.username, score); ... } ``` How you want to calculate the score is up to you and the mechanics of your game. You could take into account who won, how long it took to win, or how many towers they destroyed like we did in this example. All of this information is passed into the payload from the client. Once a score is calculated, we pass it as a number into `leaderboardRecordWrite` along with the user ID and the username to display, and the leaderboard engine will take care of the rest. In addition, it's possible to do this on the Unity side using [WriteLeaderboardRecordAsync](../../../concepts/leaderboards.md#submit-a-score). ## Listing leaderboard records Now that the server is set up to handle new leaderboard records, let's make a way for players to view what's stored on the server This is done on the client-side using `ListLeaderboardRecordsAsync`: === "LeaderboardsMenuUI.cs" ```csharp [SerializeField] private int _recordsPerPage = 100; ... public async void ShowGlobalLeaderboards(string cursor = null) { // Fetch all records from the leaderboard "global" IApiLeaderboardRecordList records = await _connection.Client.ListLeaderboardRecordsAsync(_connection.Session, "global", ownerIds: null, expiry: null, _recordsPerPage, cursor); SetLeaderboardsCursor(records, ShowGlobalLeaderboards); ... } ``` For the example above, `SetLeaderboardsCursor` might look like: === "LeaderboardsMenuUI.cs" ```csharp private void SetLeaderboardsCursor(IApiLeaderboardRecordList records, Action caller) { if (records.PrevCursor != null) { _prevPageButton.interactable = true; _prevPageButton.onClick.RemoveAllListeners(); _prevPageButton.onClick.AddListener(() => caller(records.PrevCursor)); } else { _prevPageButton.interactable = false; } if (records.NextCursor != null) { _nextPageButton.interactable = true; _nextPageButton.onClick.RemoveAllListeners(); _nextPageButton.onClick.AddListener(() => caller(records.NextCursor)); } else { _nextPageButton.interactable = false; } } ``` Here the list of `records` returned by `ListLeaderboardRecordsAsync` comes with two functions, a `PrevCursor` and `NextCursor`, which each grab the previous or next page of records (if such a page exists). By binding these functions to buttons, we can allow the player to browse records easily. Both `PrevCursor` and `NextCursor` abide by the records per page as set in the original `ListLeaderboardRecordsAsync` call, and will update the `records` object to hold the next page of records. To actually display these records, we can use a `foreach` loop over each record in `records`: === "LeaderboardsMenuUI.cs" ```csharp foreach (IApiLeaderboardRecord record in records) { string username = record.Username; if (localId == record.OwnerId) { username += " (You)"; } ... } ``` Each `record` holds a `Username`, `OwnerId`, `Score`, as well as any other custom fields stored in the optional metadata. ## Filtering leaderboards We can also filter out the global leaderboard to see how our friends or clans are doing. In the previous section we left `ownerIds` as `null` since we wanted to get all of the entries. Now, we can use this parameter to get only the entries with certain owners. For example, we can get all of our friends using `ListFriendsAsync`: ```csharp var friends = await _connection.Client.ListFriendsAsync(_connection.Session); ``` We can then extract all of their user ID's and store them into a list: ```csharp List ids = friends.Friends.Select(x => x.User.Id).ToList(); ``` Finally, we can pass in the `ids` list into the filter, and only the users in that list will be displayed in the `records`: ```csharp IApiLeaderboardRecordList records = await _connection.Client.ListLeaderboardRecordsAsync(_connection.Session, "global", ids, null, 1, cursor); ``` Here, we're fetching records from the same leaderboard (`global`), but now passing in `ids` as `ownerIds` and setting the number of records per page to `1`. A very similar process can be done to filter the leaderboard to only show scores from members of your clan: ```csharp var users = await _connection.Client.ListGroupUsersAsync(_connection.Session, _userClan.Group.Id, null, 1, null); IEnumerable ids = users.GroupUsers.Select(x => x.User.Id); IApiLeaderboardRecordList list = await _connection.Client.ListLeaderboardRecordsAsync(_connection.Session, "global", ids, null, 1, cursor); ``` ## Further reading Learn more about the topics and features, and view the complete source code, discussed above: * [Leaderboards](../../../concepts/leaderboards.md) * [Server-side function reference](../../../server-framework/function-reference.md#leaderboards) * [BattleMenuUI.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/BattleMenuUI.cs) * [LeaderboardsMenuUI.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/LeaderboardsMenuUI.cs) * [Server main.ts](https://github.com/heroiclabs/unity-sampleproject/blob/master/ServerModules/src/main.ts) --- ## File: docs/nakama/tutorials/unity/pirate-panic/matchmaking.md # Setting Up Matchmaking A core feature of many multiplayer games is the ability to find and play against random opponents. Since Nakama is aware of other online players, it is the best place to arrange matches between players. The Nakama [Matchmaking](../../../concepts/matches.md) feature simplifies the creation and management of matches so that you don't have to build the infrastructure yourself. ## Server-side settings Nakama handles matchmaking on the server side by polling for tickets submitted by players from the client. These tickets typically contain information about the player and the type of match they are looking for. First, let's set up the matchmaking settings on the server so that we can accept incoming tickets. There are three options that can be changed regarding how the server handles match requests: * `max_tickets`: Limits the total number of tickets a player (or group) can submit at one time * `interval_sec`: Changes the time, in seconds, between each attempt at forming a new match. A lower interval means lower wait times, at the potential cost of needing more server resources to handle requests * `max_intervals`: Sets the number of times to try at the max player limit before using the min player limit These are all server settings that cannot be changed by players. In order to configure these options, we create a [configuration file](../../../getting-started/configuration.md). In Pirate Panic, our configuration file is `local.yml`, but in your game this can be whatever you want. Inside the configuration file, we create a section for the matchmaker: === "local.yml" ```yml matchmaker: max_tickets: 2 interval_sec: 15 max_intervals: 3 ``` A full list of parameters and their defaults for other parts of the server configuration can be explored [in the documentation](../../../getting-started/configuration.md#matchmaker). ## Requesting a match Now that we've set up the server to handle requests, we can let players create tickets by using `AddMatchmakerAsync`: ```csharp _ticket = await _connection.Socket.AddMatchmakerAsync( query: "*", minCount: 2, maxCount: 2, stringProperties: null, numericProperties: null); ``` This ticket describes a player looking to find any match (`*` wildcard query) with exactly two total players. !!! note "Note" You can also restrict queries to find specific players that have matching properties. For example, to only match with other players in Europe you could make a `query = "+region:europe"`. See the [matchmaker documentation](../../../concepts/matches.md#query) for more examples. `stringProperties` and `numericProperties` can be used to store user data (e.g. name, region, or rank) for use with the matchmaker. Since we have no preference we can leave this `null` in this example. To give players the option to cancel their matchmaking request if a match hasn't been found yet, use `RemoveMatchmakerAsync` on the ticket that was returned by `AddMatchmakerAsync`: ```csharp await _connection.Socket.RemoveMatchmakerAsync(_ticket); ``` ## Joining a match Once a ticket is submitted, the server will handle matchmaking and assign all the matched players with a new match, if one can be found. We can then register a callback function on the client side to run once this occurs which can be used to switch scenes or otherwise prepare the game for joining the match: ```csharp _connection.Socket.ReceivedMatchmakerMatched += OnMatchmakerMatched; ... private void OnMatchmakerMatched(IMatchmakerMatched matched) { ... _connection.Socket.ReceivedMatchmakerMatched -= OnMatchmakerMatched; // Unregister callback function SceneManager.LoadScene(GameConfigurationManager.Instance.GameConfiguration.SceneNameBattle); // Switch scene to battle scene } ``` Here, `ReceivedMatchmakerMatched` is a [register hook](../../../server-framework/function-reference.md#register-hooks) that automatically fires when the server finds an opponent. ## Further reading Learn more about the topics and features, and view the complete source code, discussed above: * [Leaderboards](../../../concepts/leaderboards.md) * [Matchmaking](../../../concepts/matches.md) * [Server-side function reference](../../../runtime-code-function-reference/#leaderboards) * [BattleMenuUI.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/BattleMenuUI.cs) * [LeaderboardsMenuUI.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/LeaderboardsMenuUI.cs) * [Server main.ts](https://github.com/heroiclabs/unity-sampleproject/blob/master/ServerModules/src/main.ts) * [Nakama Config local.yml](https://github.com/heroiclabs/unity-sampleproject/blob/master/ServerModules/local.yml) * [dockerconfig](https://github.com/heroiclabs/unity-sampleproject/blob/master/ServerModules/docker-compose.yml) --- ## File: docs/nakama/tutorials/unity/pirate-panic/notifications.md # In-App Notifications Notifications are a great way to let players know about interesting events while in game. In this section we'll learn how to use the Nakama [notification](../../../concepts/in-app-notifications.md) engine to send alerts and messages to players whenever a particular action triggers them. In-app notifications can be sent from the server, then received by clients and displayed in-game. If they are marked as persistent, then the server can accept notifications even if their recipient is offline. Remember that in-app notifications are different from push notifications you might get on your phone. They only appear when players are in-game. ## Sending notifications First, let's set up the server-side code to trigger notification sends. Notifications can be bound to [register hooks](../../../server-framework/function-reference.md#register-hooks) that fire before or after a certain event. Nakama handles the logic of detecting when events occur. For example, if we want to reward players when they add their first friend , there is a `RegisterAfterAddFriends` hook that runs after a friend is added: === "quests.ts" ```typescript let afterAddFriendsFn: nkruntime.AfterHookFunction = function(ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, data: void, request: nkruntime.AddFriendsRequest) { let subject = JSON.stringify("A new friend!"); let content = { reward: 1000 }; let code = 1; let senderId = null; // Server sent let persistent = true; nk.notificationSend(ctx.userId, subject, content, code, senderId, persistent); } ... initializer.registerAfterAddFriends(afterAddFriendsFn); // Don't forget to register the hook! ``` Here, the function `notificationSend` will deliver a notification to the user `ctx.userId` when they add a new friend. In this function: * `Content` is a JSON object with custom key-value pairs for whatever data you'd like to send in the notification * The `code` is used to specify custom notification types and can be anything desired greater than `0`. Codes `0` and below are reserved by the [system](../../../concepts/in-app-notifications.md#notification-codes) * If a notification was sent by another player, then `senderId` will be the user ID of whoever sent it. In this case, the message was generated by the server so it should be `null` * If a notification has `persistent = true`, then it will be sent even if the player is offline. When the player rejoins, they can then read the list of pending notifications using [ListNotificationsAsync](../../../concepts/in-app-notifications.md#list-notifications) ## Receiving notifications Now that we have a system to send notifications, let's set up the client side. For this we also bind a function to a hook, this time the `ReceivedNotification` hook for when a notification arrives. We then use this function to process the data we sent from the server: === "NotificationPopup.cs" ```csharp public void Init(GameConnection connection) { connection.Socket.ReceivedNotification += NotificationReceived; } private class Reward { public int reward = 0; } private void NotificationReceived(IApiNotification notification) { if (notification.Code == 1) { Reward reward = JsonUtility.FromJson(notification.Content); _titleText.text = notification.Subject; _descriptionText.text = "Received reward: " + reward.reward; ... } } ``` Here we receive the notification in the form of an `IApiNotification`, which then contains all of the data we inputted to `notificationSend` on the server. Since `content` is a JSON object, we need to unpack it into a C# object. We know that the expected format is an object with a single key `reward` containing a number, so we can create a class `Reward` with this structure and convert `Content` into it using `JsonUtility.FromJson`. We then set Text objects `_titleText` and `_descriptionText` to assign with the desired subject and description. ## Further reading Learn more about the topics and features, and view the complete source code, discussed above: * [In-app Notifications](../../../concepts/in-app-notifications.md) * [Register hooks](../../../server-framework/function-reference.md#register-hooks) * [Full hooks list](https://github.com/heroiclabs/nakama-common/blob/master/runtime/runtime.go) * [NotificationPopup.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/UI/Notifications/NotificationPopup.cs) * [Server quests.ts](https://github.com/heroiclabs/unity-sampleproject/blob/master/ServerModules/src/main.ts) --- ## File: docs/nakama/tutorials/unity/pirate-panic/realtime-multiplayer.md # Realtime Multiplayer [Realtime multiplayer](../../../concepts/client-relayed-multiplayer.md), also known as client-relayed multiplayer, makes it easy for players in a match to quickly send and receive data between one another. When clients connect to a match, they can send messages to the server. Nakama's realtime multiplayer engine then automatically routes these messages to everyone else in the match. Once messages are received, each client can then process them and prompt actions, updates, or other changes to the game state. Nakama also supports [server-authoritative multiplayer](../../../concepts/server-authoritative-multiplayer.md), which puts the server at the center of each match. All clients send and receive data directly to the server, which then processes the data and routes it to other clients, if necessary. Server-authoritative multiplayer allows for more flexibility in how data is handled, but can be more complicated to develop and maintain. For Pirate Panic we will be using realtime multiplayer. ## Joining a match Once a ticket is submitted, the server will handle matchmaking and assign all the matched players with a new match, if one can be found. We can then register a callback function on the client side to run once this occurs which can be used to switch scenes or otherwise prepare the game for joining the match. For this, we use `ReceivedMatchmakerMatched`, a [register hook](../../../server-framework/function-reference.md#register-hooks) that automatically fires when the server finds an opponent. ```csharp _connection.Socket.ReceivedMatchmakerMatched += OnMatchmakerMatched; ... private void OnMatchmakerMatched(IMatchmakerMatched matched) { ... _connection.Socket.ReceivedMatchmakerMatched -= OnMatchmakerMatched; // Unregister callback function _connection.BattleConnection = new BattleConnection(matched); // Save Matched object to BattleConnection for later use SceneManager.LoadScene(GameConfigurationManager.Instance.GameConfiguration.SceneNameBattle); // Switch scene to battle scene } ``` We'll also need to save the `IMatchmakerMatched` object since it will be necessary to join the match once we've loaded in the battle scene. In PiratePanic, we've created the `BattleConnection` object to store this information, which allows us to access it from multiple classes. === "BattleConnection.cs" ```csharp public class BattleConnection { public string MatchId { get; set; } public string HostId { get; set; } public string OpponentId { get; set; } public IMatchmakerMatched Matched { get; set; } // More properties can be stored here! public BattleConnection(IMatchmakerMatched matched) { Matched = matched; } } ``` Once we loaded into the battle scene, we need to call `JoinMatchAsync` on start by passing in the matchmaker token, which is an identity and reservation token provided by the matchmaker. This will allow the server to add this client to the correct match and ensure that all messages sent to and from other players in this match will also be sent to this client: ```csharp protected async void Start() { IMatch match = await _connection.Socket.JoinMatchAsync(_connection.BattleConnection.Matched); _connection.BattleConnection.MatchId = match.Id; ... } ``` Players can leave a match by calling `_connection.Socket.LeaveMatchAsync(matchId)`, where `matchId` is the ID that was returned by `JoinMatchAsync`. ## Sending realtime data Now that a player has joined the match, let's send some information to other players. This is done with a `socket.SendMatchStateAsync()` call. For example, when a player casts a spell during the game, the other players should receive information (e.g. type of spell, where it was cast, etc.) so that their game client can render it. === "MatchMessageSpellActivated,cs" ```csharp using Nakama.TinyJson; ... public class MatchMessageSpellActivated { public readonly string OwnerId; ... // More message data goes here public MatchMessageSpellActivated(string ownerId, ...) { OwnerId = ownerId; ... } } ``` Then, inside a function somewhere else: ```csharp long opCode = 5; // custom opcode for spells MatchMessageSpellActivated message = new MatchMessageSpellActivated(playerId, ...); string json = JsonWriter.ToJson(message); // Converts C# object to JSON object _connection.Socket.SendMatchStateAsync(_connection.BattleConnection.MatchId, opCode, json); ``` Here we created a new object called `MatchMessageSpellActivated` that holds the necessary information. We need to make sure to standardize the object so that it can be easily converted on the other end, JSON in this case. Creating a custom class and then converting it to JSON is a common pattern for sending any type of data. You can add different properties to the class to send whatever you want. The `opCode` lets the recipient client handle messages differently based on their category without needing to investigate the structure of the payload itself. The opcode can be any positive integer, no inherent meaning is pre-defined by Nakama so you can create your own schemes. ## Receiving realtime data Now that we sent a message to the match, the other players in the match need to be able to receive it. To do this, Nakama has the register hook `ReceivedMatchState`, which fires every time any message is received by the client. For our cast spell example, we need to receive that message to show the spell on our client as well as process any damage that it dealt: ```csharp void Start() { _connection.Socket.ReceivedMatchState += ReceiveMatchStateMessage; // Bind function to hook } ... private void ReceiveMatchStateMessage(IMatchState matchState) { string messageJson = System.Text.Encoding.UTF8.GetString(matchState.State); if (matchState.opCode == 5) { MatchMessageSpellActivated spell = JsonParser.FromJson(messageJson); // Do stuff with the spell (instantiate object, destroy towers, etc.) } ... // Handle more opcodes, or alternatively create a switch statement } ``` Here, we're creating a custom function, `ReceiveMatchStateMessage`, and binding it to `ReceivedMatchState` so it gets called on every message received. The message is stored in `matchState.State` and, since we used the opcode `5` to specify the message is of type `MatchMessageSpellActivated`, we can convert the JSON object back to C# to read its properties. The `ReceiveMatchStateMessage` function can be used to handle different message types specified by their respective `opcodes`. ## Match presence Every player in a match has a `presence` in the match. Nakama internally tracks this presence information as a match roster. Whenever a player joins or leaves the match, a delta of updates is sent to the client so that the clients can update their own player list view or perform game-specific actions. When a client initially joins a match, Nakama will automatically provide them with a list of current joined opponents. This list can be accessed via the `match.Presences` object. As a reminder, `match` is the `IMatch` object that was returned by `JoinMatchAsync`. Additionally, we can use the `ReceivedMatchPresence` hook to run a function every time a player joins or leaves. The list of joins can be accessed as `match.Joins`, and leaves as `match.Leaves`. For example, the code below prints a message if a player left the game and there is still someone remaining: ```csharp void Start() { _connection.Socket.ReceivedMatchPresence += OnMatchPresence; } ... private void OnMatchPresence(IMatchPresenceEvent e) { if (e.Leaves.Count() > 0) { Debug.LogWarning($"OnMatchPresence() User(s) left the game"); } } ``` We can also use presence information to assign a host player responsible for handling events that should only be done one time. For example, in the beginning of the game we might want to distribute initial card hands to each player. Assigning a host can be done by assigning as host the first player that joins the match. We can do this by checking if the number of presences in the match is `0` (presence count doesn't include the current client): ```csharp if (match.Presences.Count() == 0) { _connection.BattleConnection.HostId = _connection.Session.UserId; ... } else { string opponentId = match.Presences.First().UserId; _connection.BattleConnection.OpponentId = opponentId; _connection.BattleConnection.HostId = opponentId; ... } ``` Then, if we want to perform a host-only action, we can check if the host ID is equal to the current user's ID: ```csharp if (_connection.BattleConnection.HostId == _connection.Session.UserId) { // Handle host-only behaviors } ``` ## Using RPC You may want to offload sensitive calculations to the server rather than trusting the host to perform them. For example, after the match ends we want to reward the winner with gems. In Pirate Panic, once a player's main fort is destroyed we want to send a message to the server to end the match: ```csharp private async void OnAfterMainFortDestroyed() { ... response = await _connection.Client.RpcAsync(_connection.Session, "handle_match_end", matchEndRequest.ToJson()); } ``` Here, `matchEndRequest` is any object containing the information to send to the server. We need to convert it to JSON to send it via RPC so that it can then be decoded on the server side to Typescript: ```typescript const rpcHandleMatchEnd: nkruntime.RpcFunction = function(ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, payload: string): string { let request : MatchEndRequest = JSON.parse(payload); ... } ``` See an example of the behavior of such a RPC in the [Leaderboards](leaderboards.md) section. ## Further reading Learn more about the topics and features, and view the complete source code, discussed above: * [Authoritative Multiplayer](../../../concepts/server-authoritative-multiplayer.md) * [Realtime Multiplayer](../../../concepts/client-relayed-multiplayer.md) * [BattleConnection.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/Battle/Multiplayer/BattleConnection.cs) * [Scene02BattleController.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Scene02BattleController.cs) * [MatchMessage.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/Battle/Multiplayer/MatchStates/MatchMessage.cs) * [MatchMessageSpellActivated.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/Battle/Multiplayer/MatchStates/Spells/MatchMessageSpellActivated.cs) --- ## File: docs/nakama/tutorials/unity/pirate-panic/storage.md # Persistent Storage For most games, it is necessary to keep track of important player data (e.g. inventory or game stats). In a single-player game this can be done by storing information locally on the client, but for multiplayer games data often needs to be sent to multiple clients at the same time, or protected against tampering. The Nakama [Storage Engine](../../../concepts/collections.md) makes it easy to securely store and access per-user information on the server In this section, we'll explore how to: * Read and write from the server database * Create a wallet for holding virtual currencies * Manage transactions with a wallet ledger * Include custom metadata in wallets ## Collections In Nakama, custom user data is organized into `collections` which contain groups of `objects` that hold some data for a particular user. This structure allows us to store similar information together for easy access. In Pirate Panic, each player holds a deck of cards to bring into battle: We want to store the state of this deck on the server so that each user has their own deck that gets saved even if they close the game and rejoin later. ### Read and write from server First, let's set up a way to give each player some cards to start: === "deck.ts" ```typescript let deck: any = {}; const DefaultDeckCards = [ { type: 1, level: 1, }, ... // more cards here ] DefaultDeckCards.forEach(c => { deck[nk.uuidv4()] = c; }); ``` Here, our original deck data was represented as an array of object literals. Each object contains two numeric properties: what type of card it is and the level it has been upgraded to. The details of what type and level represent are specific to Pirate Panic, so your game will likely have its own custom information to store. For example, each object could also include the date obtained, special variants, or anything else players might want to know. We then need to convert the data into storage objects by assigning each one a unique ID using the `uuidv4()` function. This is necessary to keep track of cards that might be otherwise identical. Once we have this deck, we can store it using the `storageWrite` function: === "deck.ts" ```typescript nk.storageWrite([ { key: "card_collection", collection: "user_cards", userId: userId, value: deck, permissionRead: 2, permissionWrite: 0, } ]); ``` This will create a collection that has the following structure: ```typescript card_collection user_cards: { userId: some-user-id-3fb1d6, // some arbitrary ID permissionRead: 2, permissionWrite: 2, ... // some other properties value: { some-random-id: { // another arbitrary ID type: 1, level: 1, } ... // more cards } } ``` Now, if we want to access these cards in the future, we can call `storageRead` using a request object: === "deck.ts" ```typescript function loadUserCards(nk: nkruntime.Nakama, logger: nkruntime.Logger, userId: string): CardCollection { let storageReadReq: nkruntime.StorageReadRequest = { key: DeckCollectionKey, collection: DeckCollectionName, userId: userId, } let objects: nkruntime.StorageObject[] = nk.storageRead([storageReadReq]); // Get the original collection for processing let storedCardCollection = objects[0].value; ... } ``` Multiple requests can be made at the same time using a single `storageRead` call by appending multiple `StorageReadRequest`s to the array in `storageRead`. ### Read and write from client For interfacing with collections from the Unity client we use RPCs like we did with [finding friends](friends.md#finding-friends). This helps us control the types of requests users can make, and keeps all sensitive operations strictly on the server. It is also possible for the local player to read their data using the function `ReadStorageObjectsAsync` directly: === "ProfilePanel.cs" ```csharp StorageObjectId personalStorageId = new StorageObjectId(); personalStorageId.Collection = "personal"; personalStorageId.UserId = _connection.Session.UserId; personalStorageId.Key = "player_data"; IApiStorageObjects personalStorageObjects = await _connection.Client.ReadStorageObjectsAsync(_connection.Session, personalStorageId); ``` Here we pass in a `Collection`, `UserId`, and `Key` property into a `StorageObjectId` `struct` which is used to identify the object we're looking for. ### Adding permissions Nakama supports handling of read and write permissions from the database to prevent unauthorized access of sensitive data. Objects can be protected for reads with three different `permissionRead` levels: * `0`: Nobody can read the object (except for the server) * `1`: Only the user that owns this object (the `userId` matches) can read it * `2`: Anybody can read this object Writes have two levels, `0` for no writes allowed or `1` for owner writes allowed. As with the reads, the server bypasses these permissions and can write to any object. Level `0` is often used so that write requests can be checked on the server rather than letting users edit their data themselves. For example, we want user stats to be able to be seen by everyone but they can't be changed except by the server: ```typescript const writeStats: nkruntime.StorageWriteRequest = { collection: "stats", key: "public", permissionRead: 2, permissionWrite: 0, value: initialState, userId: ctx.userId, } ``` ### Conditional reads and writes Multiple users may need to access (and possibly write to) the same object. In order to make sure that modifications don't conflict, you can pass in a `version` property into any write request. For this use either `writeStorageObjects` on the server or `WriteStorageObjectsAsync` on the client and the write will only be accepted if the version inputted matches the version stored in the database. This feature is not showcased in Pirate Panic, but you can learn more about the [conditional storage](../../../concepts/collections.md#conditional-writes) and implement it in your own games. ## Wallet On top of normal collections, Nakama also has a wallet feature that specifically supports storing and making transactions with a currency in-game. In Pirate Panic, this currency comes in the form of Gems that can be used to purchase new cards. ### Updating wallet amounts Unlike collections, wallets can only be directly updated from the server to make it more difficult for unintended or exploitative transactions to occur. Changing the value in the wallet server-side can be done using `walletUpdate`, which takes four parameters: * The user ID of the wallet to change * An object containing key-value pairs matching every currency name with an amount to update by * A metadata object * Boolean that tells the server whether it should update the ledger Since we only have one currency in Pirate Panic it is convenient to write a helper function so that we can just pass in a number rather than reconstructing `changeset` every time we want to add some gems: === "economy.ts" ```typescript const currencyKeyName = "gems" // Can be changed to whatever you want! ... function updateWallet(nk: nkruntime.Nakama, userId: string, amount: number, metadata: {[key: string]: any}): nkruntime.WalletUpdateResult { const changeset = { [currencyKeyName]: amount, } let result = nk.walletUpdate(userId, changeset, metadata, true); return result; } ``` Note is that `amount` is how much we want to change the current wallet balance by and does not completely replace the current balance. For example, if our wallet had 900 gems and `updateWallet` is called with amount 100, then we can expect the new balance to be 1000 (not 100). We can then wrap this in an RPC to let the Unity client add gems in certain situations. For example, let's say we're feeling generous and want to make a button that gives free gems: The RPC would look like: ```typescript const rpcAddUserGems: nkruntime.RpcFunction = function(ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama): string { let walletUpdateResult = updateWallet(nk, ctx.userId, 100, {}); let updateString = JSON.stringify(walletUpdateResult); logger.debug("Added 100 gems to user %s wallet: %s", ctx.userId, updateString); return updateString; } ``` And to call it from Unity: ```csharp private void Awake() { ... _getFreeGemsButton.onClick.AddListener(HandleAddFreeGems); // Bind function to button } ... private async void HandleAddFreeGems() { // Call the RPC we just made! IApiRpc newGems = await _connection.Client.RpcAsync(_connection.Session, "add_user_gems"); // IMPORTANT: Update the account instance to get the latest results! _connection.Account = await _connection.Client.GetAccountAsync(_connection.Session); ... } ``` We can also subtract gems from a player's wallet by simply passing in a negative amount. For example, if we want to buy a card using 100 gems we could call `updateWallet(nk, ctx.userId, -100, {});`. ### Custom metadata The metadata object enables you to put in any custom information into each wallet transaction. Every item in `walletLedgerList` will then have this information carried with it so it can be referenced in the future. For example, we might want to tag each reward with its match ID so we can make sure they are all coming from the same place and print out the total reward for players to see: ```typescript let metadata = { source: "match_reward", match_id: request.matchId, }; updateWallet(nk, ctx.userId, score, metadata); ``` ### Fetching wallet amounts On the client side wallet information is stored in `connection.Account.Wallet`, where `connection.Account` is an `IApiAccount` instance of the `GameConnection` class that holds the connection information we established while setting up authentication. Once we get this `Wallet` variable, which comes in the form of a stringified JSON object, we can extract the number of gems in the wallet using the following code: ```csharp private int GetGems(string wallet) { Dictionary currency = wallet.FromJson>(); if (currency.ContainsKey("gems")) { return currency["gems"]; } return 0; } ``` In Pirate Panic, since we created `GameConnection`, we can call `GetGems` using: ```csharp GetGems(_connection.Account.Wallet); ``` ### Transaction ledgers There is one final parameter, `updateLedger`, that we set to true above. Every time `walletUpdate` is called with `updateLedger` set to `true`, a new ledger entry is created. This allows us to go through all previous transactions and see their details. To do so call `walletLedgerList` which by default will grab a list of all of the transactions sorted from least to most recent. We can then use the `cursor` to skip to any entry desired. For example, if we want to get the latest cursor update: === "match.ts" ```typescript let items = nk.walletLedgerList(ctx.userId, 100); while (items.cursor) { items = nk.walletLedgerList(ctx.userId, 100, items.cursor); } ``` This is useful, for example, to skip to the current match rewards. If we set the initial reward to have a cursor update, and continue listing incremental updates (like tower capture) with no cursor update, we can then get a list of all transactions in the latest match using this strategy. ## Further reading Learn more about the topics and features, and view the complete source code, discussed above: * [Server-side functions reference - Storage](../../../server-framework/function-reference.md#storage) * [Server-side functions reference - Wallet](../../../server-framework/function-reference.md#wallet) * [Storage Collections](../../../concepts/collections.md) * [Permissions](../../../concepts/access-controls.md) * [Wallets](../../../concepts/user-accounts.md#virtual-wallet) * [ProfilePanel.cs](https://github.com/heroiclabs/unity-sampleproject/blob/master/PiratePanic/Assets/PiratePanic/Scripts/Menus/Profile/ProfilePanel.cs) * [GameConnection.cs](https://github.com/heroiclabs/unity-sampleproject/tree/master/PiratePanic/Assets/PiratePanic/Scripts) * [Server economy.ts](https://github.com/heroiclabs/unity-sampleproject/blob/master/ServerModules/src/economy.ts) * [Server deck.ts](https://github.com/heroiclabs/unity-sampleproject/blob/master/ServerModules/src/deck.ts) * [Server match.ts](https://github.com/heroiclabs/unity-sampleproject/blob/master/ServerModules/src/match.ts) --- ## File: docs/nakama/tutorials/unity/fishgame.md # Fish Game Unity Tutorial Fish Game is an open-source game project developed using the Unity game engine. It is a fast-paced 2D competitive shooter, inspired by [Duck Game](https://store.steampowered.com/app/312530/Duck_Game/), for up to four players. This game and tutorial highlight the powerful [authentication](../../concepts/authentication.md), [matchmaking](../../concepts/matches.md), and [realtime multiplayer](../../concepts/client-relayed-multiplayer.md) features of Nakama. !!! note "Note" Check out the accompanying [blog post](https://heroiclabs.com/blog/tutorials/unity-fishgame/) and [video series](https://www.youtube.com/playlist?list=PLOAExZcDNj9tut2gSUlw46OKK4iB--uW1) to this tutorial for additional content. ## Prerequisites To easily follow with this tutorial, perform the following before proceeding: * [Install Nakama](../../getting-started/docker-quickstart.md) * [Download](https://unity3d.com/get-unity/download) and install Unity 2019.4 * [Install the Nakama Unity SDK](https://github.com/heroiclabs/nakama-unity) * [Download the Fish Game Unity project](https://github.com/heroiclabs/fishgame-unity) * Familiarize yourself with the [Project](#project-hierarchy) and [Asset](#asset-hierarchy) hierarchies ### Project hierarchy Open the Fish Game project in Unity and see the project hierarchy: The following game objects are listed: * **GameManager**: Controls all match specific game logic, including handling matchmaking. * **AudioManager**: Handles the game audio. * **UI** * **AudioMenu**: A simple audio mute toggle. * **MainMenu**: Menu enabling the player to find a match, cancel a matchmaking request, or view the credits. * **InGameMenu**: Menu that the player can invoke mid-game by pressing ESC to quit the match. * **Canvas**: A UI canvas that simply contains the winning player notification label. * **DefaultMap**: The game level including spawn points and tilemap. * **Main Camera**: The game 2D camera and a background sprite. * **EventSystem**: The default Unity event system that handles UI interaction. ### Asset hierarchy The relevant folders for Nakama features are: * **Entities** > **Player**: Contains the player prefabs, three in total, and all `MonoBehaviour` components associated with them. * **Managers**: Contains the components responsible for handling global game elements (e.g. `GameManager`) and various menu scripts. * **Nakama**: Contains the `NakamaConnection` class, which is a `ScriptableObject`, and the `LocalNakamaConnection` asset. * **Weapons**: Contains the components responsible for handling weapons and projectiles. ## Connecting to Nakama From the Asset hierarchy, above, you see that our Fish Game project already contains a `NakamaConnection` class and `LocalNakamaConnection` asset. The `NakamaConnection` inherits from Unity’s `ScriptableObject` class, meaning we can create instances of the `NakamaConnection` class in our project and save them as assets that can be passed as dependencies to other scripts in our game. To learn about how to connect to our Nakama server, let’s open up the `NakamaConnection.cs` file. After declaring our `NakamaConnection` class, we have defined some public variables that allow us to specify which Nakama server this connection object should connect to: === "NakamaConnection.cs" ```csharp public string Scheme = "http"; public string Host = "localhost"; public int Port = 7350; public string ServerKey = "defaultkey"; ``` Next let's look at the workings of the `Connect` function. First we create an instance of the `Nakama.Client` object and pass in our connection values: === "NakamaConnection.cs" ```csharp Client = new Nakama.Client(Scheme, Host, Port, ServerKey, UnityWebRequestAdapter.Instance); ``` This `Client` object is then used to interact with Nakama. Next we attempt to restore an existing user session, for example, if a user previously launched the game and connected to the Nakama server we want to restore that same session if it hasn’t already expired: === "NakamaConnection.cs" ```csharp var authToken = PlayerPrefs.GetString(SessionPrefName); if (!string.IsNullOrEmpty(authToken)) { var session = Nakama.Session.Restore(authToken); if (!session.IsExpired) { Session = session; } } ``` Here we attempt to get an authentication token from `PlayerPrefs`. If found, we call the `Nakama.Session.Restore` function and pass in the authentication token to retrieve a `Session` object. We also perform a check to ensure that the session has not already expired. If the session has not expired, we assign this session object to our private `Session` variable. Otherwise, we need to reauthenticate to create a brand new session. This happens inside an `if` statement that checks if the `Session` variable is null: === "NakamaConnection.cs" ```csharp // If we weren't able to restore an existing session, authenticate to create a new user session. if (Session == null) { string deviceId; // ... ``` This means we have two potential paths, either the user has played the game before and we have stored their device’s unique identifier or this is their first time playing and we need to get a unique identifier for their device to authenticate with the Nakama server. For a previous player, we check to see if there is already a value for the `DeviceIdentifierPrefName` key in `PlayerPrefs` and, if yes, grab the value and store it in the `deviceId` variable to be used later in the function: === "NakamaConnection.cs" ```csharp // If we've already stored a device identifier in PlayerPrefs then use that. if (PlayerPrefs.HasKey(DeviceIdentifierPrefName)) { deviceId = PlayerPrefs.GetString(DeviceIdentifierPrefName); } ``` For a new player we attempt to grab their device’s unique identifier from `SystemInfo.deviceUniqueIdentifier`. If this does not return an appropriate identifier value, it will return a value of `SystemInfo.unsupportedIdentifier`. We can check for this and, if we receive this value, generate our own unique identifier using a Guid before storing it in `PlayerPrefs` for future use: === "NakamaConnection.cs" ```csharp if (deviceId == SystemInfo.unsupportedIdentifier) { deviceId = System.Guid.NewGuid().ToString(); } // Store the device identifier to ensure we use the same one each time from now on. PlayerPrefs.SetString(DeviceIdentifierPrefName, deviceId); ``` We can now authenticate with the Nakama server and store the authentication token to `PlayerPrefs`: === "NakamaConnection.cs" ```csharp // Use Nakama Device authentication to create a new session using the device identifier. Session = await Client.AuthenticateDeviceAsync(deviceId); // Store the auth token that comes back so that we can restore the session later if necessary. PlayerPrefs.SetString(SessionPrefName, Session.AuthToken); ``` Now that we have a valid Nakama Session we go ahead and open a `Socket`: === "NakamaConnection.cs" ```csharp // Open a new Socket for realtime communication. Socket = Client.NewSocket(true); await Socket.ConnectAsync(Session, true); ``` This allows us to begin communicating with the Nakama server and the other players connected to our game. Calling this connection in Fish Game is accomplished via the `GameManager` object: ### Creating a connection By using the `CreateAssetMenu` attribute and defining public variables for our connection, this enables us to modify each `NakamaConnection` asset in the Unity Inspector. This makes it simple to connect to multiple Nakama servers when needed, for example when you have a local, development, and production server. To create a new Nakama connection asset: 1. From the **Assets** menu, select **Create** > **Nakama Connection**. 2. In the **Inspector** pane, edit the connection properties: * Scheme: The connection scheme, whether `http` or `https`. * Host: The host name or IP of the server. * Port: The port used, `7350` by default. * Server Key: The secret key for connecting to the server, `defaultKey` by default. !!! note "Note" For production assets, be sure to add `NakamaConnection` assets to your `.gitignore` file so that you don’t commit them to source control and expose your production server secret key. For example: ``` # Heroic Cloud Nakama Connection asset HeroicCloudNakamaConnection.asset HeroicCloudNakamaConnection.asset.meta ``` ## Matchmaking !!! note "Note" Our [documentation](../../concepts/matches.md) covers matchmaking in Nakama in more detail. To review how matchmaking in Fish Game works, we will dive into the `GameManager`, `NakamaConnection`, and `MainMenu` classes and explore the relevant functions. Here you are creating a new `Dictionary` to store the connected players and subscribing to the following events: * **ReceivedMatchmakerMatched**: Enables you to react when Nakama has found a match. * **ReceivedMatchPresence**: Enables you to react when player(s) join or leave the match. * **ReceivedMatchState**: Enables you to react when receiving a message from the Nakama server. === "GameManager.cs" ```csharp /// /// Called by Unity when the GameObject starts. /// private async void Start() { // Create an empty dictionary to hold references to the currently connected players. players = new Dictionary(); // ... // Setup network event handlers. NakamaConnection.Socket.ReceivedMatchmakerMatched += OnReceivedMatchmakerMatched; NakamaConnection.Socket.ReceivedMatchPresence += OnReceivedMatchPresence; NakamaConnection.Socket.ReceivedMatchState += async m => await OnReceivedMatchState(m); // ... } ``` From the `MainMenu` class, you can see how the matchmaking request is triggered via the `FindMatch` function. Here you are calling the `FindMatch` function on the `NakamaConnection` class via a reference on the `GameManager`, and passing in the amount of players (selected via drop-down) to match with: === "MainMenu.cs" ```csharp /// /// Begins the matchmaking process. /// public async void FindMatch() { // ... await gameManager.NakamaConnection.FindMatch(int.Parse(PlayersDropdown.options[PlayersDropdown.value].text)); } ``` In the `FindMatch` function in the `NakamaConnection` class, a `Dictionary` is declared with a single `key,value` pair that specifies we’re using the Unity engine, and a function to add this request to the matchmaking pool based on this filter and any other specified parameters: === "NakamaConnection.cs" ```csharp /// /// Starts looking for a match with a given number of minimum players. /// public async Task FindMatch(int minPlayers = 2) { // Set some matchmaking properties to ensure we only look for games that are using the Unity client. // This is not a required when using the Unity Nakama SDK, // however in this instance we are using it to differentiate different matchmaking requests across multiple platforms using the same Nakama server. var matchmakingProperties = new Dictionary { { "engine", "unity" } }; // Add this client to the matchmaking pool and get a ticket. var matchmakerTicket = await Socket.AddMatchmakerAsync("+properties.engine:unity", minPlayers, minPlayers, matchmakingProperties); currentMatchmakingTicket = matchmakerTicket.Ticket; } ``` Nakama will then respond by giving you a `MatchmakingTicket`. This ticket is what allows you to join a match that Nakama has found, or to cancel your existing matchmaking request: === "NakamaConnection.cs" ```csharp /// /// Cancels the current matchmaking request. /// public async Task CancelMatchmaking() { await Socket.RemoveMatchmakerAsync(currentMatchmakingTicket); } ``` Now that you have been added to the matchmaking pool, Nakama searches for a match and notifies you via the `ReceivedMatchmakerMatched` event listener when one is found: === "GameManager.cs" ```csharp /// /// Called when a MatchmakerMatched event is received from the Nakama server. /// /// The MatchmakerMatched data. private async void OnReceivedMatchmakerMatched(IMatchmakerMatched matched) { // Cache a reference to the local user. localUser = matched.Self.Presence; // Join the match. var match = await NakamaConnection.Socket.JoinMatchAsync(matched); // ... // Spawn a player instance for each connected user. foreach (var user in match.Presences) { SpawnPlayer(match.Id, user); } // Cache a reference to the current match. currentMatch = match; } ``` The stored reference to the `Self.Presence` value enables quick and easy access to the local player’s Nakama presence, useful for grabbing things such as their Session ID, User ID and Username. The stored reference to the `JoinMatchAsync()` return value enables access to things such as the current match ID and connected users. Next we loop through all of the connected users via the `match.Presences` property and [spawn a player](#spawning-players) for each. The final thing to consider for matchmaking is what to do when a user connects and disconnects. This is done via the `OnReceivedMatchPresence` event handler: === "GameManager.cs" ```csharp /// /// Called when a player/s joins or leaves the match. /// /// The MatchPresenceEvent data. private void OnReceivedMatchPresence(IMatchPresenceEvent matchPresenceEvent) { // For each new user that joins, spawn a player for them. foreach (var user in matchPresenceEvent.Joins) { SpawnPlayer(matchPresenceEvent.MatchId, user); } // For each player that leaves, despawn their player. foreach (var user in matchPresenceEvent.Leaves) { if (players.ContainsKey(user.SessionId)) { Destroy(players[user.SessionId]); players.Remove(user.SessionId); } } } ``` Here the `IMatchPresenceEvent` gives us two things, a list of users who have joined the match and a list of users who have left the match. For users who have joined, we spawn a player object for them just like we did in the `OnReceivedMatchmakerMatched` event handler. For users who have left, we check to see if we currently have a reference for that user in the `players` Dictionary based on their Session ID. If we do, we first destroy the `GameObject` for that player, followed by removing them from our players `Dictionary`. ## Spawning players Here we review the `SpawnPlayer` function called by the `GameManager` in response to matchmaking or user presence events. First we perform a check to see if we already have a reference to this user in our players Dictionary based on their Session ID. If yes, we return early as we have already spawned this player: === "GameManager.cs" ```csharp // If the player has already been spawned, return early. if (players.ContainsKey(user.SessionId)) { return; } ``` Next we determine where to spawn this player. If a `spawnIndex` hasn’t been passed (`spawnIndex == -1`) then we choose a random spawn point for this player. If a spawn index has been passed we will use that instead: === "GameManager.cs" ```csharp // If the spawnIndex is -1 then pick a spawn point at random, otherwise spawn the player at the specified spawn point. var spawnPoint = spawnIndex == -1 ? SpawnPoints.transform.GetChild(Random.Range(0, SpawnPoints.transform.childCount - 1)) : SpawnPoints.transform.GetChild(spawnIndex); ``` Next we check to see whether or not this player is our own local player or a remote player. Based on this we decide which Player prefab to spawn, and then instantiate an instance of the prefab within the game at the relevant spawn point: === "GameManager.cs" ```csharp // Set a variable to check if the player is the local player or not based on session ID. var isLocal = user.SessionId == localUser.SessionId; // Choose the appropriate player prefab based on if it's the local player or not. var playerPrefab = isLocal ? NetworkLocalPlayerPrefab : NetworkRemotePlayerPrefab; // Spawn the new player. var player = Instantiate(playerPrefab, spawnPoint.transform.position, Quaternion.identity); ``` Next we see if this is a remotely connected player and, if so, grab a reference to the `PlayerNetworkRemoteSync` component and set appropriate network data. We pass it the current Match ID and a reference to this user’s `IUserPresence` object: === "GameManager.cs" ```csharp // Setup the appropriate network data values if this is a remote player. if (!isLocal) { player.GetComponent().NetworkData = new RemotePlayerNetworkData { MatchId = matchId, User = user }; } ``` This is used when receiving data to check if that data is meant for that specific player. We then add the player to our `players` Dictionary using their Session ID as the key and passing in the player `GameObject` as the value: === "GameManager.cs" ```csharp // Add the player to the players array. players.Add(user.SessionId, player); ``` Next we check to see if this is our local player and, if so, setup an event listener on their `PlayerDied` event: === "GameManager.cs" ```csharp // If this is our local player, add a listener for the PlayerDied event. if (isLocal) { localPlayer = player; player.GetComponent().PlayerDied.AddListener(OnLocalPlayerDied); } ``` ## Player prefabs Found in then **Entities** > **Player** folder, Fish Game has two player prefabs: `NetworkLocalPlayer` and `NetworkRemotePlayer`. ### NetworkLocalPlayer This is a simple wrapper around the network agnostic Player prefab, which has components to handle movement, physics, weapon and colour. The root `GameObject` of this prefab has a few noteworthy components: * **PlayerNetworkLocalSync**: Handles sending out state updates across the network at a specific interval. * **PlayerInputController**: Handles player input and passes it to the underlying movement controller. * **PlayerHealthController** Handles player health and is responsible for triggering the `PlayerDied` event. * **PlayerCameraController** A simple component that ensures the camera follows this player. ### NetworkRemotePlayer Also a wrapper around the Player prefab, here we have just one component on the root: `PlayerNetworkRemoteSync`. This component receives state updates from the network and handles the smooth interpolation of it’s current state to the new correct state received. For example, if this object was at position `{ 100, 100 }` and we received a network message to say that the correct position should be `{ 120, 100 }`, this component would handle smoothly correcting this objects position over `LerpTime` seconds. ## Sending player position To review how we send out position and input updates across the network from `NetworkLocalPlayer`, let's look at `Entities` > `Player` > `PlayerNetworkLocalSync.cs`. First we have a public variable that determines how often network state syncs updates are sent: === "PlayerNetworkLocalSync.cs" ```csharp /// /// How often to send the player's velocity and position across the network, in seconds. /// public float StateFrequency = 0.1f; ``` Changing the value here will increase or decrease the frequency with which state updates are sent across the network. Finding a good balance here is a crucial part of creating a smooth feeling multiplayer game. You can see the default value here of `0.1f`, or 10 ticks per second. For our Fish Game example, let's set this to `0.05f`, doubling our rate to 20 ticks per second: ```csharp public float StateFrequency = 0.1f; ``` A faster rate will send updates more frequently, which should result in a smoother feel for other players, while sending updates less frequently will result in player positions becoming out of sync on other connected clients and the interpolations/corrections being much more jarring. Next let's focus on the `LateUpdate` function, used as we only want to send updates after the player’s position has been updated. First is a check to see if the state sync timer has elapsed and, if so, then we need to send a new update: === "PlayerNetworkLocalSync.cs" ```csharp // Send a network packet containing the player's velocity and position. gameManager.SendMatchState( OpCodes.VelocityAndPosition, MatchDataJson.VelocityAndPosition(playerRigidbody.velocity, playerTransform.position)); ``` Next we check to see if the local player’s input has changed since the last time this function ran and, if not, return early: === "PlayerNetworkLocalSync.cs" ```csharp // If the players input hasn't changed, return early. if (!playerInputController.InputChanged) { return; } ``` Finally, we send the player’s current input across the network: === "PlayerNetworkLocalSync.cs" ```csharp // Send network packet with the player's current input. gameManager.SendMatchState( OpCodes.Input, MatchDataJson.Input(playerInputController.HorizontalInput, playerInputController.Jump, playerInputController.JumpHeld, playerInputController.Attack) ); ``` ### Operation codes Used in the functions above, operation codes (`OpCodes`) are a numeric representation of specific network actions that can occur in the game. You can see all the `OpCodes` we have in Fish Game by looking at `Entities` > `Player`> `OpCodes.cs`. These include: * `VelocityAndPosition`: See above. Send a network message containing the player's velocity and position. * `Input`: See above. Indicates that a player's input has changed since the last update, and sends a message with the current input. * `Died`: Indicates death of a player, after which their GameObject is destroyed and they are removed from the `players` Dictionary. * `Respawned`: Send a message across the network informing other players where you have spawned. * `NewRound`: Send a message across the network informing other players that you won, and initiate a new round. ## Receiving player position To review how we receive position and input updates across the network from a `NetworkRemotePlayer`, let's look at `Entities` > `Player` > `PlayerNetworkRemoteSync.cs`. First we have two public variables: one storing a reference to the current Match ID as well as the player’s `IUserPresence` object, and the other determining the speed at which we interpolate the player’s position: === "PlayerNetworkRemoteSync.cs" ```csharp public RemotePlayerNetworkData NetworkData; /// /// The speed (in seconds) in which to smoothly interpolate to the player's actual position when receiving corrected data. /// public float LerpTime = 0.05f; ``` Within the `Start` function, we set up a a listener for whenever we receive data from the network: === "PlayerNetworkRemoteSync.cs" ```csharp // Add an event listener to handle incoming match state data. gameManager.NakamaConnection.Socket.ReceivedMatchState += OnReceivedMatchState; // ... /// /// Called when receiving match data from the Nakama server. /// /// The incoming match state data. private void OnReceivedMatchState(IMatchState matchState) { // If the incoming data is not related to this remote player, ignore it and return early. if (matchState.UserPresence.SessionId != NetworkData.User.SessionId) { return; } // Decide what to do based on the Operation Code of the incoming state data as defined in OpCodes. switch (matchState.OpCode) { case OpCodes.VelocityAndPosition: UpdateVelocityAndPositionFromState(matchState.State); break; case OpCodes.Input: SetInputFromState(matchState.State); break; case OpCodes.Died: playerMovementController.PlayDeathAnimation(); break; default: break; } } ``` Here, we first determine whether or not the message we are receiving is meant for this object or not by comparing this object’s `NetworkData.User.SessionId` with the `matchState.UserPresence.SessionId` value: * **If they do not match**, we don’t need to handle this message and return early from this function. * **If they do match**, we perform a switch on the `matchState.OpCode`. Depending on the `OpCode` we pass the state to one of three functions to be handled, either for: * [Receiving position and velocity](#receiving-position-and-velocity) * [Receiving input](#receiving-input) * [When a player dies](#when-a-player-dies) ### Receiving position and velocity For this, let's look at the `UpdateVelocityAndPositionFromState` function: === "PlayerNetworkRemoteSync.cs" ```csharp /// /// Updates the player's velocity and position based on incoming state data. /// /// The incoming state byte array. private void UpdateVelocityAndPositionFromState(byte[] state) { var stateDictionary = GetStateAsDictionary(state); playerRigidbody.velocity = new Vector2(float.Parse(stateDictionary["velocity.x"]), float.Parse(stateDictionary["velocity.y"])); var position = new Vector3( float.Parse(stateDictionary["position.x"]), float.Parse(stateDictionary["position.y"]), 0); // Begin lerping to the corrected position. lerpFromPosition = playerTransform.position; lerpToPosition = position; lerpTimer = 0; lerpPosition = true; } ``` Here we grab the state as a `Dictionary` using the helper function `GetStateAsDictionary`, then update the player’s rigid body velocity property with the incoming `velocity.x` and `velocity.y` values. Next we create a new `Vector3` with the incoming `position.x` and `position.y` values. We then begin the interpolation process by setting the player’s current position as the `lerpFromPosition` and the new position as the `lerpToPosition`. We reset the lerp timer and set the `lerpPosition` value to `true` which will ensure the the interpolation code that lives in `LateUpdate` will run. ### Receiving input For this, let's look at the `SetInputFromState` function: === "PlayerNetworkRemoteSync.cs" ```csharp /// /// Sets the appropriate input values on the PlayerMovementController and PlayerWeaponController based on incoming state data. /// /// The incoming state Dictionary. private void SetInputFromState(byte[] state) { var stateDictionary = GetStateAsDictionary(state); playerMovementController.SetHorizontalMovement(float.Parse(stateDictionary["horizontalInput"])); playerMovementController.SetJump(bool.Parse(stateDictionary["jump"])); playerMovementController.SetJumpHeld(bool.Parse(stateDictionary["jumpHeld"])); if (bool.Parse(stateDictionary["attack"])) { playerWeaponController.Attack(); } } ``` Again we grab the state as a `Dictionary`, then set the appropriate values on the `PlayerMovementController` from the incoming state data.If the `stateDictionary["attack"]` value is `true` we also update the `PlayerWeaponController` by calling the `Attack` function. A final consideration here is the `OnDestroy` function. It is always good practice to remove event listeners when no longer needed, such as the `OnReceivedMatchState` handler here when this object is destroyed: === "PlayerNetworkRemoteSync.cs" ```csharp /// /// Called when this GameObject is being destroyed. /// private void OnDestroy() { if (gameManager != null) { gameManager.NakamaConnection.Socket.ReceivedMatchState -= OnReceivedMatchState; } } ``` ### When a player dies To review how we handle player death let's look at the `Entities` > `Player` > `PlayerHealthController`. A `PlayerDied` event is declared, and called in the `TakeDamage` function when a player's health is less than 1: === "PlayerHealthController.cs" ```csharp public PlayerDiedEvent PlayerDied; // ... /// /// Reduces the players health by damage, triggers the PlayerDied event if health is 0 or below. /// /// The amount of damage the player should receive. public void TakeDamage(int damage = 1) { // ... // If health falls to 0 or below, disable player input controls, play the death animation and fire the PlayerDied event. if (health <= 0) { // ... PlayerDied.Invoke(gameObject); } } ``` The local player death is handled via the `OnLocalPlayerDied` function in the `GameManager` class: === "GameManager.cs" ```csharp /// /// Called when the local player dies. /// /// The local player. private async void OnLocalPlayerDied(GameObject player) { // Send a network message telling everyone that we died. await SendMatchStateAsync(OpCodes.Died, MatchDataJson.Died(player.transform.position)); // Remove ourself from the players array and destroy our GameObject after 0.5 seconds. players.Remove(localUser.SessionId); Destroy(player, 0.5f); } ``` Here we send a message across the network to let other players know that we died, and our location. Then we are removed from the `players` Dictionary and destroyed. All remaining players then receive the `OpCodes.Died` network action, the response is then handled via the `OnReceivedMatchState` function: === "GameManager.cs" ```csharp /// /// Called when new match state is received. /// /// The MatchState data. private async Task OnReceivedMatchState(IMatchState matchState) { // Get the local user's session ID. var userSessionId = matchState.UserPresence.SessionId; // If the matchState object has any state length, decode it as a Dictionary. var state = matchState.State.Length > 0 ? System.Text.Encoding.UTF8.GetString(matchState.State).FromJson>() : null; // Decide what to do based on the Operation Code as defined in OpCodes. switch(matchState.OpCode) { case OpCodes.Died: // Get a reference to the player who died and destroy their GameObject after 0.5 seconds and remove them from our players array. var playerToDestroy = players[userSessionId]; Destroy(playerToDestroy, 0.5f); players.Remove(userSessionId); // If there is only one player left and that us, announce the winner and start a new round. if (players.Count == 1 && players.First().Key == localUser.SessionId) { AnnounceWinnerAndStartNewRound(); } break; case OpCodes.Respawned: // Spawn the player at the chosen spawn index. SpawnPlayer(currentMatch.Id, matchState.UserPresence, int.Parse(state["spawnIndex"])); break; case OpCodes.NewRound: // Display the winning player's name and begin a new round. await AnnounceWinnerAndRespawn(state["winningPlayerName"]); break; default: break; } } ``` Here we are getting a reference to the Session ID of the connected user who sent the message and storing it in `userSessionId`, then see if there is any state data inside the network message by checking the `matchState.State.Length` property. Lastly we do a switch over the `matchState.OpCode` and handle each one as necessary. ## Announcing the winner On each player death, we destroy their `GameObject` and remove them from the `players` Dictionary based on their Session ID. Next we check to see if they are the only player remaining. If so, the `AnnounceWinnerAndStartNewRound` function is called: === "GameManager.cs" ```csharp /// /// Sends a network message that indicates a player has won and a new round is being started. /// /// public async void AnnounceWinnerAndStartNewRound() { // ... var winningPlayerName = localDisplayName; // Send a network message telling everyone else that we won. await SendMatchStateAsync(OpCodes.NewRound, MatchDataJson.StartNewRound(winningPlayerName)); // Display the winning player message and respawn our player. await AnnounceWinnerAndRespawn(winningPlayerName); } ``` Here we are grabbing the winning player’s name and sending out a network message with the `OpCodes.NewRound` action telling everyone that the match has been won and the winning player. We then call the `AnnounceWinnerAndRespawn` function: === "GameManager.cs" ```csharp /// /// Displays the winning player message and respawns the player. /// /// The name of the winning player. private async Task AnnounceWinnerAndRespawn(string winningPlayerName) { // Set the winning player text label. WinningPlayerText.text = string.Format("{0} won this round!", winningPlayerName); // Wait for 2 seconds. await Task.Delay(2000); // Reset the winner player text label. WinningPlayerText.text = ""; // Remove ourself from the players array and destroy our player. players.Remove(localUser.SessionId); Destroy(localPlayer); // Choose a new spawn position and spawn our local player. var spawnIndex = Random.Range(0, SpawnPoints.transform.childCount - 1); SpawnPlayer(currentMatch.Id, localUser, spawnIndex); // Tell everyone where we respawned. SendMatchState(OpCodes.Respawned, MatchDataJson.Respawned(spawnIndex)); } ``` This displays the winning player message and, after a short delay (2 seconds here), removes the message. Then we destroy the local player `GameObject` and remove them from the `players` Dictionary before respawning them at a random location. ## Further reading Learn more about the topics and features in this tutorial with the following: * [Unity Client Guide](../../client-libraries/unity-client-guide.md) * [Authentication](../../concepts/authentication.md) * [Sessions](../../concepts/session.md) * [Matchmaker](../../concepts/matches.md) --- ## File: docs/nakama/tutorials/server/daily-rewards.md # Daily Rewards Tutorial A common way of engaging and retaining players is to offer them a daily reward for logging into your game each day. In this example, you'll learn how to implement a daily reward system using Server Runtime Code. ## Prerequisites To easily follow with this tutorial, perform the following before proceeding: * Download the [Nakama Project Template](https://github.com/heroiclabs/nakama-project-template) or * Follow the [TypeScript Setup Guide](/runtime-code-typescript-setup) ## Registering the RPCs The daily reward sample defines two RPCs ([Remote Procedure Calls](../../server-framework/basics.md#rpc-example)) to check eligibility and issue the reward. The RPCs are then registered to Nakama events in either `main.ts`, `main.go` or `main.lua`. === "TypeScript" ```typescript function InitModule(ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, initializer: nkruntime.Initializer) { initializer.registerRpc('canclaimdailyreward_js', rpcCanClaimDailyReward); initializer.registerRpc('claimdailyreward_js', rpcClaimDailyReward); logger.info('JavaScript logic loaded.'); } ``` === "Go" ```go package main import ( "context" "database/sql" "time" "github.com/heroiclabs/nakama-common/runtime" "github.com/heroiclabs/unity-devrel-samples/modules" ) const ( rpcIdCanClaimDailyReward = "canclaimdailyreward_go" rpcIdClaimDailyReward = "claimdailyreward_go" ) //noinspection GoUnusedExportedFunction func InitModule(ctx context.Context, logger runtime.Logger, db *sql.DB, nk runtime.NakamaModule, initializer runtime.Initializer) error { initStart := time.Now() if err := initializer.RegisterRpc(rpcIdCanClaimDailyReward, modules.RpcCanClaimDailyReward); err != nil { return err } if err := initializer.RegisterRpc(rpcIdClaimDailyReward, modules.RpcClaimDailyReward); err != nil { return err } logger.Info("Plugin loaded in '%d' msec.", time.Since(initStart).Milliseconds()) return nil } ``` === "Lua" ```lua local nk = require("nakama") local daily_reward = require("daily_reward") nk.register_rpc(daily_reward.rpc_can_claim_daily_reward, "canclaimdailyreward_lua") nk.register_rpc(daily_reward.rpc_claim_daily_reward, "claimdailyreward_lua") ``` To register an RPC with the server, you need to specify a string identifier as well as a function to run when the RPC is called by a client. ## Implementing the RPCs The RPCs implement the following logic: ### `canClaimDailyReward` * Get the latest daily reward object from the [Nakama Storage Engine](../../concepts/collections.md) * Check to see if the last time the user claimed a reward was before 00:00 * Return a JSON response indicating if the user can claim a daily reward ### `claimDailyReward` * Get the latest daily reward object from the [Nakama Storage Engine](../../concepts/collections.md) * Check to see if the last time the user claimed a reward was before 00:00 * Update the user's [Wallet](../../server-framework/function-reference.md#wallet) * Send a [Notification](../../concepts/in-app-notifications.md) to the user * Update or create the daily reward object in the [Nakama Storage Engine](../../concepts/collections.md) * Return a JSON response with the number of coins received ### Module code (Go/Lua) This section is specifically for people using Go or Lua. There is some additional code you will need to include in your daily reward module scripts for each respective language. === "Go" ```go package modules import ( "context" "database/sql" "encoding/json" "time" "github.com/heroiclabs/nakama-common/api" "github.com/heroiclabs/nakama-common/runtime" ) var ( errInternalError = runtime.NewError("internal server error", 13) // INTERNAL errMarshal = runtime.NewError("cannot marshal type", 13) // INTERNAL errNoInputAllowed = runtime.NewError("no input allowed", 3) // INVALID_ARGUMENT errNoUserIdFound = runtime.NewError("no user ID in context", 3) // INVALID_ARGUMENT errUnmarshal = runtime.NewError("cannot unmarshal type", 13) // INTERNAL ) ``` === "Lua" ```lua local nk = require("nakama") local M = { } -- Module code goes here return M ``` ### Getting the last daily reward object Let's take a look at the code for retrieving the latest daily reward object from the Nakama Storage Engine. === "TypeScript" ```typescript function getLastDailyRewardObject(context: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, payload: string) : any { if (!context.userId) { throw Error('No user ID in context'); } if (payload) { throw Error('No input allowed'); } var objectId: nkruntime.StorageReadRequest = { collection: 'reward', key: 'daily', userId: context.userId, } var objects: nkruntime.StorageObject[]; try { objects = nk.storageRead([ objectId ]); } catch (error) { logger.error('storageRead error: %s', error); throw error; } var dailyReward: any = { lastClaimUnix: 0, } objects.forEach(object => { if (object.key == 'daily') { dailyReward = object.value; } }); return dailyReward; } ``` === "Go" ```go type dailyReward struct { LastClaimUnix int64 `json:"last_claim_unix"` // The last time the user claimed the reward in UNIX time. } func getLastDailyRewardObject(ctx context.Context, logger runtime.Logger, nk runtime.NakamaModule, payload string) (dailyReward, *api.StorageObject, error) { var d dailyReward d.LastClaimUnix = 0 userID, ok := ctx.Value(runtime.RUNTIME_CTX_USER_ID).(string) if !ok { return d, nil, errNoUserIdFound } if len(payload) > 0 { return d, nil, errNoInputAllowed } objects, err := nk.StorageRead(ctx, []*runtime.StorageRead{{ Collection: "reward", Key: "daily", UserID: userID, }}) if err != nil { logger.Error("StorageRead error: %v", err) return d, nil, errInternalError } var o *api.StorageObject for _, object := range objects { switch object.GetKey() { case "daily": if err := json.Unmarshal([]byte(object.GetValue()), &d); err != nil { logger.Error("Unmarshal error: %v", err) return d, nil, errUnmarshal } return d, object, nil } } return d, o, nil } ``` === "Lua" ```lua function get_last_daily_reward_object(context, payload) if (not context.user_id or #context.user_id < 1) then error({ "no user ID in context", 3 }) end if (#payload > 0) then error({ "no input allowed", 3 }) end local objectid = { collection = "reward", key = "daily", user_id = context.user_id } local success, objects = pcall(nk.storage_read, { objectid }) if (not success) then nk.logger_error(string.format("storage_read error: %q", objects)) error({ "internal server error", 13 }) end local daily_reward = { ["last_claim_unix"] = 0 } for _, object in ipairs(objects) do if (object.key == "daily") then daily_reward = object.value break end end return daily_reward end ``` Regardless of the language you use, the core logic remains the same. * Check the context to ensure there is a valid user ID * Check the user did NOT pass anything in the payload * Query the storage engine for a `daily` object in the `reward` collection * Return the daily reward object or a default one ### Checking if the user is eligible to receive a daily reward === "TypeScript" ```typescript function canUserClaimDailyReward(dailyReward: any) { if (!dailyReward.lastClaimUnix) { dailyReward.lastClaimUnix = 0; } var d = new Date(); d.setHours(0, 0, 0, 0); return dailyReward.lastClaimUnix < msecToSec(d.getTime()); } function msecToSec(n: number): number { return Math.floor(n / 1000); } ``` === "Go" ```go func canUserClaimDailyReward(d dailyReward) bool { t := time.Now() midnight := time.Date(t.Year(), t.Month(), t.Day(), 0, 0, 0, 0, time.Local) return time.Unix(d.LastClaimUnix, 0).Before(midnight) } ``` === "Lua" ```lua function can_user_claim_daily_reward(daily_reward) local dt = os.date("*t") local elapsed_sec_from_midnight = (dt.hour * 3600 + dt.min * 60 + dt.sec) % 86400 return daily_reward.last_claim_unix < (os.time() - elapsed_sec_from_midnight) end ``` This function checks the last claim Unix timestamp value of the daily reward object. If it is less than the timestamp for midnight of the previous day, it returns true, otherwise it returns false. ### CanClaimDailyReward RPC With the two helper functions complete it's time to implement the first of the RPCs. This RPC will return the value of the helper function that checks the user's eligibility as a JSON object. === "TypeScript" ```typescript function rpcCanClaimDailyReward(context: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, payload: string): string { var dailyReward = getLastDailyRewardObject(context, logger, nk, payload); var response = { canClaimDailyReward: canUserClaimDailyReward(dailyReward) } var result = JSON.stringify(response); logger.debug('rpcCanClaimDailyReward response: %q', result); return result; } ``` === "Go" ```go func RpcCanClaimDailyReward(ctx context.Context, logger runtime.Logger, db *sql.DB, nk runtime.NakamaModule, payload string) (string, error) { var resp struct { CanClaimDailyReward bool `json:"canClaimDailyReward"` } dailyReward, _, err := getLastDailyRewardObject(ctx, logger, nk, payload) if err != nil { logger.Error("Error getting daily reward: %v", err) return "", errInternalError } resp.CanClaimDailyReward = canUserClaimDailyReward(dailyReward) out, err := json.Marshal(resp) if err != nil { logger.Error("Marshal error: %v", err) return "", errMarshal } logger.Debug("rpcCanClaimDailyReward resp: %v", string(out)) return string(out), nil } ``` === "Lua" ```lua function M.rpc_can_claim_daily_reward(context, payload) local daily_reward = get_last_daily_reward_object(context, payload) local resp = { ["canClaimDailyReward"] = can_user_claim_daily_reward(daily_reward) } local success, result = pcall(nk.json_encode, resp) if (not success) then nk.logger_error(string.format("json_encode error: %q", result)) error({ "internal server error", 13 }) end nk.logger_debug(string.format("rpc_can_claim_daily_reward resp: %q", result)) return result end ``` ### ClaimDailyReward RPC This RPC will ensure the user is eligible to receive the daily reward, update the user's Wallet, send out a notification and then update the user's daily reward in the Storage Engine. === "TypeScript" ```typescript function rpcClaimDailyReward(context: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, payload: string): string { var response = { coinsReceived: 0 }; var dailyReward = getLastDailyRewardObject(context, logger, nk, payload); if (canUserClaimDailyReward(dailyReward)) { response.coinsReceived = 500; var changeset = { coins: response.coinsReceived, } try { nk.walletUpdate(context.userId, changeset, {}, false); } catch (error) { logger.error('walletUpdate error: %q', error); throw error; } var notification: nkruntime.NotificationRequest = { code: 1001, content: changeset, persistent: true, subject: "You've received your daily reward!", userId: context.userId, } try { nk.notificationsSend([notification]); } catch (error) { logger.error('notificationsSend error: %q', error); throw error; } dailyReward.lastClaimUnix = msecToSec(Date.now()); var write: nkruntime.StorageWriteRequest = { collection: 'reward', key: 'daily', permissionRead: 1, permissionWrite: 0, value: dailyReward, userId: context.userId, } if (dailyReward.version) { // Use OCC to prevent concurrent writes. write.version = dailyReward.version } // Update daily reward storage object for user. try { nk.storageWrite([ write ]) } catch (error) { logger.error('storageWrite error: %q', error); throw error; } } var result = JSON.stringify(response); logger.debug('rpcClaimDailyReward response: %q', result) return result; } ``` === "Go" ```go func RpcClaimDailyReward(ctx context.Context, logger runtime.Logger, db *sql.DB, nk runtime.NakamaModule, payload string) (string, error) { userID, ok := ctx.Value(runtime.RUNTIME_CTX_USER_ID).(string) if !ok { return "", errNoUserIdFound } var resp struct { CoinsReceived int64 `json:"coinsReceived"` } resp.CoinsReceived = int64(0) dailyReward, dailyRewardObject, err := getLastDailyRewardObject(ctx, logger, nk, payload) if err != nil { logger.Error("Error getting daily reward: %v", err) return "", errInternalError } if canUserClaimDailyReward(dailyReward) { resp.CoinsReceived = 500 // Update player wallet. changeset := map[string]int64{ "coins": resp.CoinsReceived, } if _, _, err := nk.WalletUpdate(ctx, userID, changeset, map[string]interface{}{}, false); err != nil { logger.Error("WalletUpdate error: %v", err) return "", errInternalError } err := nk.NotificationsSend(ctx, []*runtime.NotificationSend{{ Code: 1001, Content: map[string]interface{}{ "coins": changeset["coins"], }, Persistent: true, Sender: "", // Server sent. Subject: "You've received your daily reward!", UserID: userID, }}) if err != nil { logger.Error("NotificationsSend error: %v", err) return "", errInternalError } dailyReward.LastClaimUnix = time.Now().Unix() object, err := json.Marshal(dailyReward) if err != nil { logger.Error("Marshal error: %v", err) return "", errInternalError } version := "" if dailyRewardObject != nil { // Use OCC to prevent concurrent writes. version = dailyRewardObject.GetVersion() } // Update daily reward storage object for user. _, err = nk.StorageWrite(ctx, []*runtime.StorageWrite{{ Collection: "reward", Key: "daily", PermissionRead: 1, PermissionWrite: 0, // No client write. Value: string(object), Version: version, UserID: userID, }}) if err != nil { logger.Error("StorageWrite error: %v", err) return "", errInternalError } } out, err := json.Marshal(resp) if err != nil { logger.Error("Marshal error: %v", err) return "", errMarshal } logger.Debug("rpcClaimDailyReward resp: %v", string(out)) return string(out), nil } ``` === "Lua" ```lua function M.rpc_claim_daily_reward(context, payload) local resp = { ["coinsReceived"] = 0 } local daily_reward = get_last_daily_reward_object(context, payload) -- If last claimed is before the new day grant a new reward! if (can_user_claim_daily_reward(daily_reward)) then resp.coinsReceived = 500 -- Update player wallet. local changeset = { ["coins"] = resp.coinsReceived } local success, result = pcall(nk.wallet_update, context.user_id, changeset, {}, false) if (not success) then nk.logger_error(string.format("wallet_update error: %q", result)) error({ "internal server error", 13 }) end local notification = { code = 1001, content = changeset, persistent = true, sender = "", subject = "You've received your daily reward!", user_id = context.user_id } local success, result = pcall(nk.notifications_send, { notification }) if (not success) then nk.logger_error(string.format("notifications_send error: %q", result)) error({ "internal server error", 13 }) end daily_reward.last_claim_unix = os.time() local version = nil if (daily_reward.version) then -- Use OCC to prevent concurrent writes. version = daily_reward.version end -- Update daily reward storage object for user. local write = { collection = "reward", key = "daily", permission_read = 1, permission_write = 0, value = daily_reward, version = version, user_id = context.user_id } local success, result = pcall(nk.storage_write, { write }) if (not success) then nk.logger_error(string.format("storage_write error: %q", result)) error({ "internal server error", 13 }) end end local success, result = pcall(nk.json_encode, resp) if (not success) then nk.logger_error(string.format("json_encode error: %q", result)) error({ "internal server error", 13 }) end nk.logger_debug(string.format("rpc_claim_daily_reward resp: %q", result)) return result end ``` ## Exploring in the Nakama Console Spin up your server ([using Docker](../../getting-started/docker-quickstart)) and test the RPCs using the [Nakama Console](../../getting-started/console-overview). You can access the Nakama Console by opening a browser and going to [http://localhost:7351](http://localhost:7351). Once there, you can try interacting with your RPCs via the [API Explorer](http://127.0.0.1:7351/#/apiexplorer) by selecting them from the dropdown and specifying a user ID as a context. ## Wrap Up With those two RPCs implemented you now have a simple daily reward system, congratulations! Feel free to experiment further by adding more complicated eligibility criteria or other such features. --- ## File: docs/nakama/tutorials/server/initialize-new-user.md # Initialize a new user It's often useful when a new user registers to have a bunch of records setup for them. In games this could be needed for a user's virtual wallet, initial inventory items, etc. In this tutorial we'll cover a few different ways to handle this use case. ## After register callback The simplest approach is to write records in the success callback for the register function in a client. This code demonstrates how to do it with a condensed example. In real application code you'll break up the [authentication](../../concepts/authentication.md) and connect logic from the storage writes based on how you manage connect and reconnect. === "Unity" ```csharp var deviceId = SystemInfo.deviceUniqueIdentifier; var session = await client.AuthenticateDeviceAsync(deviceId); var json = "{\"coins\": 100, \"gems\": 10, \"artifacts\": 0}"; var object = new WriteStorageObject = { "collection" = "wallets", "key" = "mywallet", "value" = json }; const storageWriteAck = await client.WriteStorageObjectsAsync(session, objects); Debug.Log("Successfully setup new user's records."); ``` This code has tradeoffs which should be noted. A disconnect can happen before the records are written to storage. This may leave the setup of the user incomplete and the application in a bad state. This option is only worth choosing when you want to avoid writing server-side code or have built retry logic on top of a client. ## Server-side hook Another way to write records for the new user is to run server-side code after registration has completed. This can be done with a [register hook](../../server-framework/basics.md#register-hooks). The ["register_after"](../../server-framework/function-reference.md#register-hooks) hook can be used with one of the `"authenticaterequest_*"` message types to tell the server to run a function after that message has been processed. It's important to note that the server does not distinguish between register and login messages so we use a [conditional write](../../concepts/collections.md#conditional-writes) to store the records. === "Lua" ```lua local function initialize_user(context, payload) if payload.created then -- Only run this logic if the account that has authenticated is new. local changeset = { coins = 10, -- Add 10 coins to the user's wallet. gems = 5 -- Add 5 gems from the user's wallet. artifacts = 0 -- No artifacts to start with. } local metadata = {} nk.wallet_update(context.user_id, changeset, metadata, true) end end -- change to whatever message name matches your authentication type. nk.register_req_after(initialize_user, "AuthenticateDevice") ``` === "Go" ```go func InitializeUser(ctx context.Context, logger Logger, db *sql.DB, nk NakamaModule, out *api.Session, in *api.AuthenticateDeviceRequest) error { if out.Created { // Only run this logic if the account that has authenticated is new. userID, ok := ctx.Value(runtime.RUNTIME_CTX_USER_ID).(string) if !ok { return "", errors.New("Invalid context") } changeset := map[string]interface{}{ "coins": 10, // Add 10 coins to the user's wallet. "gems": 5, // Add 5 gems from the user's wallet. "artifacts": 0, // No artifacts to start with. } var metadata map[string]interface{} if err := nk.WalletUpdate(ctx, userID, changeset, metadata, true); err != nil { // Handle error. } } } // Register as after hook, this call should be in InitModule. if err := initializer.RegisterAfterAuthenticateDevice(InitializeUser); err != nil { logger.Error("Unable to register: %v", err) return err } ``` This approach avoids the tradeoff with client disconnects but requires a database write to happen after every login or register message. This could be acceptable depending on how frequently you write data to the storage engine and can be minimized if you [cache a user's session](../../concepts/authentication.md#sessions) for quick reconnects.