ZeroTierOne Documentation

Technical Documentation: zerotier/ZeroTierOne

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

ZeroTier - Global Area Networking

Quick Links

ZeroTier Documentation - Start here for downloads, installation, and usage
How to build - Build instructions and platform requirements
Corporate Site
Downloads
Service API Reference
Network Controller
Commercial Support
License Information

About

ZeroTier is a smart programmable Ethernet switch for planet Earth. It allows all networked devices, VMs, containers, and applications to communicate as if they all reside in the same physical data center or cloud region.

This is accomplished by combining a cryptographically addressed and secure peer-to-peer network (termed VL1) with an Ethernet emulation layer somewhat similar to VXLAN (termed VL2). Our VL2 Ethernet virtualization layer includes advanced enterprise SDN features like fine grained access control rules for network micro-segmentation and security monitoring.

All ZeroTier traffic is encrypted end-to-end using secret keys that only you control. Most traffic flows peer-to-peer, though we offer free (but slow) relaying for users who cannot establish peer-to-peer connections.

Apps for Android and iOS are available for free in the Google Play and Apple app stores.

For repository layout, build instructions, platform requirements, and information about running ZeroTier, see build.md.

License

See LICENSE-MPL.txt for all code in node/, osdep/. service/, and everywhere else except ext/ and nonfree/.

See nonfree/LICENSE.md for all non-free ("source available") portions of this repository.

Code in ext/ is external code included for build convenience or backward compatibility and retains its original license.

Overview (`doc/README.md`)

Manual Pages and Other Documentation

Use "./build.sh" to build the manual pages.

You'll need either Node.js/npm installed (script will then automatically install the npm marked-man package) or /usr/bin/ronn. The latter is a Ruby program packaged on some distributions as rubygem-ronn or ruby-ronn or installable as gem install ronn. The Node marked-man package and ronn from RubyGems are two roughly equivalent alternatives for compiling Markdown into roff/man format.

Overview (`ext/README.md`)

Miscellaneous Stuff

This subfolder contains:

Bundled third party libraries that are compiled into the binary on platforms and Linux distributions where they are not available on the system.
Pre-compiled binaries for some platforms, such as pre-built and signed drivers for Mac and Windows.
Miscellaneous files used by installers and packages on various platform targets.

Overview (`include/README.md`)

ZeroTier Node API

This is the externally facing plain C API, which wraps the Node class in the node/ folder. It provides a platform-agnostic interface to the core ZeroTier network virtualization engine.

Overview (`java/README.md`)

ZeroTier One SDK - Android JNI Wrapper

Building

Requires:

JDK
ANT
Android NDK

Required Environment Variables:

NDK_BUILD_LOC - Path do the ndk-build script in the Android NDK
ANDROID_PLATFORM - path to the directory android.jar lives (on Windows: C:\Users\AppData\Local\Android\sdk\platforms\android-21)

Overview (`node/README.md`)

ZeroTier Network Hypervisor Core

This directory contains the real ZeroTier: a completely OS-independent global virtual Ethernet switch engine. This is where the magic happens.

Give it wire packets and it gives you Ethernet packets, and vice versa. The core contains absolutely no actual I/O, port configuration, or other OS-specific code (except Utils::getSecureRandom()). It provides a simple C API via /include/ZeroTierOne.h. It's designed to be small and maximally portable for future use on small embedded and special purpose systems.

Code in here follows these guidelines:

Keep it minimal, especially in terms of code footprint and memory use.
There should be no OS-dependent code here unless absolutely necessary (e.g. getSecureRandom).
If it's not part of the core virtual Ethernet switch it does not belong here.
No C++11 or C++14 since older and embedded compilers don't support it yet and this should be maximally portable.
Minimize the use of complex C++ features since at some point we might end up "minus-minus'ing" this code if doing so proves necessary to port to tiny embedded systems.

Overview (`osdep/README.md`)

OS-Dependent and OS-Interface Things

This folder contains stuff that interfaces with the base operating system
like Phy for network access and the various OS-specific Ethernet tap
drivers.

Overview (`pkg/README.md`)

Third-party packaging

For package documentation see the Devices section here: docs.zerotier.com

Overview (`rule-compiler/README.md`)

ZeroTier Rules Compiler

This script converts ZeroTier rules in human-readable format into rules suitable for import into a ZeroTier network controller. It's the script that is used in the rules editor on ZeroTier Central.

A command line interface is included that may be invoked as: node cli.js <rules script>.

See the manual for information about the rules engine and rules script syntax.

Overview (`service/README.md`)

ZeroTier One Network Virtualization Service

This is the actual implementation of ZeroTier One, a service providing connectivity to ZeroTier virtual networks for desktops, laptops, servers, VMs, etc. (Mobile versions for iOS and Android have their own implementations in native Java and Objective C that leverage only the ZeroTier core engine.)

Local Configuration File

A file called local.conf in the ZeroTier home folder contains configuration options that apply to the local node. (It does not exist unless you create it). It can be used to set up trusted paths, blacklist physical paths, set up physical path hints for certain nodes, and define trusted upstream devices (federated roots). In a large deployment it can be deployed using a tool like Puppet, Chef, SaltStack, etc. to set a uniform configuration across systems.

It's a JSON format file that can also be edited and rewritten by ZeroTier One itself, so ensure that proper JSON formatting is used. To validate your config, paste it into a website like jsonlint.com, or use a tool like jq.

Check the output of zerotier-cli info -j to see if your configuration is being loaded.

Settings available in local.conf (this is not valid JSON, and JSON does not allow comments):

text

/* Detailed source-code truncated for AI context efficiency. */

trustedPathId: A trusted path is a physical network over which encryption and authentication are not required. This provides a performance boost but sacrifices all ZeroTier's security features when communicating over this path. Only use this if you know what you are doing and really need the performance! To set up a trusted path, all devices using it MUST have the same trusted path ID for the same network. Trusted path IDs are arbitrary positive non-zero integers. For example a group of devices on a LAN with IPs in 10.0.0.0/24 could use it as a fast trusted path if they all had the same trusted path ID of "25" defined for that network.

An example local.conf:

javascript

{
	"physical": {
		"10.0.0.0/24": {
			"blacklist": true
		},
		"10.10.10.0/24": {
			"trustedPathId": 101010024
		},
	},
	"virtual": {
		"feedbeef12": {
			"role": "UPSTREAM",
			"try": [ "10.10.20.1/9993" ],
			"blacklist": [ "192.168.0.0/24" ]
		}
	},
	"settings": {
		"softwareUpdate": "apply",
		"softwareUpdateChannel": "release"
	}
}

Network Virtualization Service API

The JSON API supports GET, POST/PUT, and DELETE. PUT is treated as a synonym for POST. Other methods including HEAD are not supported.

Values POSTed to the JSON API are extremely type sensitive. Things must be of the indicated type, otherwise they will be ignored or will generate an error. Anything quoted is a string so booleans and integers must lack quotes. Booleans must be true or false and nothing else. Integers cannot contain decimal points or they are floats (and vice versa). If something seems to be getting ignored or set to a strange value, or if you receive errors, check the type of all JSON fields you are submitting against the types listed below. Unrecognized fields in JSON objects are also ignored.

API requests must be authenticated via an authentication token. ZeroTier One saves this token in the authtoken.secret file in its working directory. This token may be supplied via the auth URL parameter (e.g. '?auth=...') or via the X-ZT1-Auth HTTP request header. Static UI pages are the only thing the server will allow without authentication.

A jsonp URL argument may be supplied to request JSONP encapsulation. A JSONP response is sent as a script with its JSON response payload wrapped in a call to the function name supplied as the argument to jsonp.

/status

Purpose: Get running node status and addressing info
Methods: GET
Returns: { object }

Field	Type	Description	Writable
address	string	10-digit hex ZeroTier address of this node	no
publicIdentity	string	This node's ZeroTier identity.public	no
worldId	integer	ZeroTier world ID (never changes except for test)	no
worldTimestamp	integer	Timestamp of most recent world definition	no
online	boolean	If true at least one upstream peer is reachable	no
tcpFallbackActive	boolean	If true we are using slow TCP fallback	no
relayPolicy	string	Relay policy: ALWAYS, TRUSTED, or NEVER	no
versionMajor	integer	Software major version	no
versionMinor	integer	Software minor version	no
versionRev	integer	Software revision	no
version	string	major.minor.revision	no
clock	integer	Current system clock at node (ms since epoch)	no

/network

Purpose: Get all network memberships
Methods: GET
Returns: [ {object}, ... ]

Getting /network returns an array of all networks that this node has joined. See below for network object format.

/network/\

Purpose: Get, join, or leave a network
Methods: GET, POST, DELETE
Returns: { object }

To join a network, POST to it. Since networks have no mandatory writable parameters, POST data is optional and may be omitted. Example: POST to /network/8056c2e21c000001 to join the public "Earth" network. To leave a network, DELETE it e.g. DELETE /network/8056c2e21c000001.

Most network settings are not writable, as they are defined by the network controller.

Field	Type	Description	Writable
id	string	16-digit hex network ID	no
nwid	string	16-digit hex network ID (legacy field)	no
mac	string	MAC address of network device for this network	no
name	string	Short name of this network (from controller)	no
status	string	Network status (OK, ACCESS_DENIED, etc.)	no
type	string	Network type (PUBLIC or PRIVATE)	no
mtu	integer	Ethernet MTU	no
dhcp	boolean	If true, DHCP should be used to get IP info	no
bridge	boolean	If true, this device can bridge others	no
broadcastEnabled	boolean	If true ff:ff:ff:ff:ff:ff broadcasts work	no
portError	integer	Error code returned by underlying tap driver	no
netconfRevision	integer	Network configuration revision ID	no
assignedAddresses	[string]	Array of ZeroTier-assigned IP addresses (/bits)	no
routes	[object]	Array of ZeroTier-assigned routes (see below)	no
portDeviceName	string	Name of virtual network device (if any)	no
allowManaged	boolean	Allow IP and route management	yes
allowGlobal	boolean	Allow IPs and routes that overlap with global IPs	yes
allowDefault	boolean	Allow overriding of system default route	yes
allowDNS	boolean	Allow configuration of DNS on network	yes

Route objects:

Field	Type	Description	Writable
target	string	Target network / netmask bits	no
via	string	Gateway IP address (next hop) or null for LAN	no
flags	integer	Flags, currently always 0	no
metric	integer	Route metric (not currently used)	no

/peer

Purpose: Get all peers
Methods: GET
Returns: [ {object}, ... ]

Getting /peer returns an array of peer objects for all current peers. See below for peer object format.

/peer/\

Purpose: Get or set information about a peer
Methods: GET, POST
Returns: { object }

Field	Type	Description	Writable
address	string	10-digit hex ZeroTier address of peer	no
versionMajor	integer	Major version of remote (if known)	no
versionMinor	integer	Minor version of remote (if known)	no
versionRev	integer	Software revision of remote (if known)	no
version	string	major.minor.revision	no
latency	integer	Latency in milliseconds if known	no
role	string	LEAF, UPSTREAM, ROOT or PLANET	no
paths	[object]	Currently active physical paths (see below)	no

Path objects:

Field	Type	Description	Writable
address	string	Physical socket address e.g. IP/port	no
lastSend	integer	Time of last send through this path	no
lastReceive	integer	Time of last receive through this path	no
active	boolean	Is this path in use?	no
expired	boolean	Is this path expired?	no
preferred	boolean	Is this a current preferred path?	no
trustedPathId	integer	If nonzero this is a trusted path (unencrypted)	no

Overview (`tcp-proxy/README.md`)

TCP Proxy Server

This is the TCP proxy server we run for TCP tunneling from peers behind difficult NATs. Regular users won't have much use for this.

How to run your own

Currently you must build it and distribute it to your server manually.

To reduce latency, the tcp-relay should be as close as possible to the nodes it is serving. A datacenter in the same city or the LAN would be ideal.

Build

cd tcp-relay
make

Point your node at it

The default tcp relay is at 204.80.128.1/443 -an anycast address.

Option 1 - local.conf configuration

See Service docs for more info on local.conf
{ "settings": { "tcpFallbackRelay": "198.51.100.123/443", "forceTcpRelay": true } }

In this example, forceTcpRelay is enabled. This is helpful for testing or if you know you'll need tcp relay. It takes a few minutes for zerotier-one to realize it needs to relay otherwise.

Option 2 - redirect 204.80.128.1 to your own IP

If you are the admin of the network that is blocking ZeroTier UDP, you can transparently redirect 204.80.128.1 to one of your IP addresses. Users won't need to edit their local client configuration.

Configuring this in your Enterprise Firewall is left as an exercise to the reader.

Here is an iptables example for illustrative purposes:

shell

-A PREROUTING -p tcp -d 204.80.128.1 --dport 443 -j DNAT --to-destination 198.51.100.123
-A POSTROUTING -p tcp -d 198.51.100.123 --dport 443 -j SNAT --to-source 204.80.128.1

Overview (`windows/README.md`)

This folder contains the Windows driver code, Windows-specific service code, and the Microsoft Visual Studio projects and "solution" for doing Windows builds.

This code may also build with MinGW but this hasn't been tested.

Building

text

MSBuild ZeroTierOne.sln /property:Configuration=Release /property:Platform=ARM64

(substitute e.g. Debug, x64, etc. as needed above, or use Visual Studio to select a target)

Testing

PowerShell admin session #1 inside \windows\Build\<ARCH>\<TARGET>:

text

PS C:\Users\<USER>\ZeroTierOne\windows\Build\ARM64\Release> .\zerotier-one_arm64.exe -p9994 -C C:\<SOME_TEMP_DIR>\
Starting Control Plane...
Starting V6 Control Plane...

Session #2:

text

PS C:\Users\<USER>\ZeroTierOne\windows\Build\ARM64\Release> zerotier-cli.bat -p9994 -D"\Users\rcode\scratch\zt1_home\" info
info c3e46fa070 1.12.2 ONLINE
PS C:\Users\<USER>\code\ZeroTierOne\windows\Build\ARM64\Release> zerotier-cli.bat -p9994 -D"\Users\<SOME_TEMP_DIR>\scratch\zt1_home\" join ebe7fbd4458bde97
200 join OK                                                                                                             PS C:\Users\<USER>\code\ZeroTierOne\windows\Build\ARM64\Release>

Overview (`ext/arm32-neon-salsa2012-asm/README.md`)

ARM NEON (32-bit) ASM implementation of Salsa20/12

This is from supercop and was originally written by Daniel J. Bernstein. Code is in the public domain like the rest of Salsa20. It's much faster than the naive implementation.

It's included automatically in 32-bit Linux ARM builds. It likely will not work on 64-bit ARM, so it'll need to be ported at least. That will unfortunately keep it out of mobile versions for now since those are all going 64-bit.

Overview (`ext/central-controller-docker/README.md`)

ZeroTier Central Controller Docker Image

Dockerfile & startup script for use with ZeroTier Central. Not intended for public use.

Overview (`ext/hiredis-0.14.1/README.md`)

This Readme reflects the latest changed in the master branch. See v0.14.1 for the Readme and documentation for the latest release.

HIREDIS

Hiredis is a minimalistic C client library for the Redis database.

It is minimalistic because it just adds minimal support for the protocol, but
at the same time it uses a high level printf-alike API in order to make it
much higher level than otherwise suggested by its minimal code base and the
lack of explicit bindings for every Redis command.

Apart from supporting sending commands and receiving replies, it comes with
a reply parser that is decoupled from the I/O layer. It
is a stream parser designed for easy reusability, which can for instance be used
in higher level language bindings for efficient reply parsing.

Hiredis only supports the binary-safe Redis protocol, so you can use it with any
Redis version >= 1.2.0.

The library comes with multiple APIs. There is the
synchronous API, the asynchronous API and the reply parsing API.

IMPORTANT: Breaking changes when upgrading from 0.13.x -> 0.14.x

Bulk and multi-bulk lengths less than -1 or greater than LLONG_MAX are now
protocol errors. This is consistent with the RESP specification. On 32-bit
platforms, the upper bound is lowered to SIZE_MAX.

Change redisReply.len to size_t, as it denotes the the size of a string

User code should compare this to size_t values as well. If it was used to
compare to other values, casting might be necessary or can be removed, if
casting was applied before.

For a detailed list of changes please view our Changelog.

Synchronous API

To consume the synchronous API, there are only a few function calls that need to be introduced:

redisContext *redisConnect(const char *ip, int port);
void *redisCommand(redisContext *c, const char *format, ...);
void freeReplyObject(void *reply);

Connecting

The function redisConnect is used to create a so-called redisContext. The
context is where Hiredis holds state for a connection. The redisContext
struct has an integer err field that is non-zero when the connection is in
an error state. The field errstr will contain a string with a description of
the error. More information on errors can be found in the Errors section.
After trying to connect to Redis using redisConnect you should
check the err field to see if establishing the connection was successful:

redisContext *c = redisConnect("127.0.0.1", 6379);
if (c == NULL || c->err) {
    if (c) {
        printf("Error: %s\n", c->errstr);
        // handle error
    } else {
        printf("Can't allocate redis context\n");
    }
}

Note: A redisContext is not thread-safe.

Sending commands

There are several ways to issue commands to Redis. The first that will be introduced is
redisCommand. This function takes a format similar to printf. In the simplest form,
it is used like this:

reply = redisCommand(context, "SET foo bar");

The specifier %s interpolates a string in the command, and uses strlen to
determine the length of the string:

reply = redisCommand(context, "SET foo %s", value);

When you need to pass binary safe strings in a command, the %b specifier can be
used. Together with a pointer to the string, it requires a size_t length argument
of the string:

reply = redisCommand(context, "SET foo %b", value, (size_t) valuelen);

Internally, Hiredis splits the command in different arguments and will
convert it to the protocol used to communicate with Redis.
One or more spaces separates arguments, so you can use the specifiers
anywhere in an argument:

reply = redisCommand(context, "SET key:%s %s", myid, value);

Using replies

The return value of redisCommand holds a reply when the command was
successfully executed. When an error occurs, the return value is NULL and
the err field in the context will be set (see section on Errors).
Once an error is returned the context cannot be reused and you should set up
a new connection.

The standard replies that redisCommand are of the type redisReply. The
type field in the redisReply should be used to test what kind of reply
was received:

REDIS_REPLY_STATUS:
- The command replied with a status reply. The status string can be accessed using reply->str.
  The length of this string can be accessed using reply->len.
REDIS_REPLY_ERROR:
- The command replied with an error. The error string can be accessed identical to REDIS_REPLY_STATUS.
REDIS_REPLY_INTEGER:
- The command replied with an integer. The integer value can be accessed using the
  reply->integer field of type long long.
REDIS_REPLY_NIL:
- The command replied with a nil object. There is no data to access.
REDIS_REPLY_STRING:
- A bulk (string) reply. The value of the reply can be accessed using reply->str.
  The length of this string can be accessed using reply->len.
REDIS_REPLY_ARRAY:
- A multi bulk reply. The number of elements in the multi bulk reply is stored in
  reply->elements. Every element in the multi bulk reply is a redisReply object as well
  and can be accessed via reply->element[..index..].
  Redis may reply with nested arrays but this is fully supported.

Replies should be freed using the freeReplyObject() function.
Note that this function will take care of freeing sub-reply objects
contained in arrays and nested arrays, so there is no need for the user to
free the sub replies (it is actually harmful and will corrupt the memory).

Important: the current version of hiredis (0.10.0) frees replies when the
asynchronous API is used. This means you should not call freeReplyObject when
you use this API. The reply is cleaned up by hiredis after the callback
returns. This behavior will probably change in future releases, so make sure to
keep an eye on the changelog when upgrading (see issue #39).

Cleaning up

To disconnect and free the context the following function can be used:

void redisFree(redisContext *c);

This function immediately closes the socket and then frees the allocations done in
creating the context.

Sending commands (cont'd)

Together with redisCommand, the function redisCommandArgv can be used to issue commands.
It has the following prototype:

void *redisCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);

It takes the number of arguments argc, an array of strings argv and the lengths of the
arguments argvlen. For convenience, argvlen may be set to NULL and the function will
use strlen(3) on every argument to determine its length. Obviously, when any of the arguments
need to be binary safe, the entire array of lengths argvlen should be provided.

The return value has the same semantic as redisCommand.

Pipelining

To explain how Hiredis supports pipelining in a blocking connection, there needs to be
understanding of the internal execution flow.

When any of the functions in the redisCommand family is called, Hiredis first formats the
command according to the Redis protocol. The formatted command is then put in the output buffer
of the context. This output buffer is dynamic, so it can hold any number of commands.
After the command is put in the output buffer, redisGetReply is called. This function has the
following two execution paths:

The input buffer is non-empty:
- Try to parse a single reply from the input buffer and return it
- If no reply could be parsed, continue at 2
The input buffer is empty:
- Write the entire output buffer to the socket
- Read from the socket until a single reply could be parsed

The function redisGetReply is exported as part of the Hiredis API and can be used when a reply
is expected on the socket. To pipeline commands, the only things that needs to be done is
filling up the output buffer. For this cause, two commands can be used that are identical
to the redisCommand family, apart from not returning a reply:

void redisAppendCommand(redisContext *c, const char *format, ...);
void redisAppendCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);

After calling either function one or more times, redisGetReply can be used to receive the
subsequent replies. The return value for this function is either REDIS_OK or REDIS_ERR, where
the latter means an error occurred while reading a reply. Just as with the other commands,
the err field in the context can be used to find out what the cause of this error is.

The following examples shows a simple pipeline (resulting in only a single call to write(2) and
a single call to read(2)):

redisReply *reply;
redisAppendCommand(context,"SET foo bar");
redisAppendCommand(context,"GET foo");
redisGetReply(context,&reply); // reply for SET
freeReplyObject(reply);
redisGetReply(context,&reply); // reply for GET
freeReplyObject(reply);

This API can also be used to implement a blocking subscriber:

reply = redisCommand(context,"SUBSCRIBE foo");
freeReplyObject(reply);
while(redisGetReply(context,&reply) == REDIS_OK) {
    // consume message
    freeReplyObject(reply);
}

Errors

When a function call is not successful, depending on the function either NULL or REDIS_ERR is
returned. The err field inside the context will be non-zero and set to one of the
following constants:

REDIS_ERR_IO:
There was an I/O error while creating the connection, trying to write
to the socket or read from the socket. If you included errno.h in your
application, you can use the global errno variable to find out what is
wrong.
REDIS_ERR_EOF:
The server closed the connection which resulted in an empty read.
REDIS_ERR_PROTOCOL:
There was an error while parsing the protocol.
REDIS_ERR_OTHER:
Any other error. Currently, it is only used when a specified hostname to connect
to cannot be resolved.

In every case, the errstr field in the context will be set to hold a string representation
of the error.

Asynchronous API

Hiredis comes with an asynchronous API that works easily with any event library.
Examples are bundled that show using Hiredis with libev
and libevent.

Connecting

The function redisAsyncConnect can be used to establish a non-blocking connection to
Redis. It returns a pointer to the newly created redisAsyncContext struct. The err field
should be checked after creation to see if there were errors creating the connection.
Because the connection that will be created is non-blocking, the kernel is not able to
instantly return if the specified host and port is able to accept a connection.

Note: A redisAsyncContext is not thread-safe.

redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
    printf("Error: %s\n", c->errstr);
    // handle error
}

The asynchronous context can hold a disconnect callback function that is called when the
connection is disconnected (either because of an error or per user request). This function should
have the following prototype:

void(const redisAsyncContext *c, int status);

On a disconnect, the status argument is set to REDIS_OK when disconnection was initiated by the
user, or REDIS_ERR when the disconnection was caused by an error. When it is REDIS_ERR, the err
field in the context can be accessed to find out the cause of the error.

The context object is always freed after the disconnect callback fired. When a reconnect is needed,
the disconnect callback is a good point to do so.

Setting the disconnect callback can only be done once per context. For subsequent calls it will
return REDIS_ERR. The function to set the disconnect callback has the following prototype:

int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);

Sending commands and their callbacks

In an asynchronous context, commands are automatically pipelined due to the nature of an event loop.
Therefore, unlike the synchronous API, there is only a single way to send commands.
Because commands are sent to Redis asynchronously, issuing a command requires a callback function
that is called when the reply is received. Reply callbacks should have the following prototype:

void(redisAsyncContext *c, void *reply, void *privdata);

The privdata argument can be used to curry arbitrary data to the callback from the point where
the command is initially queued for execution.

The functions that can be used to issue commands in an asynchronous context are:

int redisAsyncCommand(
  redisAsyncContext *ac, redisCallbackFn *fn, void *privdata,
  const char *format, ...);
int redisAsyncCommandArgv(
  redisAsyncContext *ac, redisCallbackFn *fn, void *privdata,
  int argc, const char **argv, const size_t *argvlen);

Both functions work like their blocking counterparts. The return value is REDIS_OK when the command
was successfully added to the output buffer and REDIS_ERR otherwise. Example: when the connection
is being disconnected per user-request, no new commands may be added to the output buffer and REDIS_ERR is
returned on calls to the redisAsyncCommand family.

If the reply for a command with a NULL callback is read, it is immediately freed. When the callback
for a command is non-NULL, the memory is freed immediately following the callback: the reply is only
valid for the duration of the callback.

All pending callbacks are called with a NULL reply when the context encountered an error.

Disconnecting

An asynchronous connection can be terminated using:

void redisAsyncDisconnect(redisAsyncContext *ac);

When this function is called, the connection is not immediately terminated. Instead, new
commands are no longer accepted and the connection is only terminated when all pending commands
have been written to the socket, their respective replies have been read and their respective
callbacks have been executed. After this, the disconnection callback is executed with the
REDIS_OK status and the context object is freed.

Hooking it up to event library X

There are a few hooks that need to be set on the context object after it is created.
See the adapters/ directory for bindings to libev and libevent.

Reply parsing API

Hiredis comes with a reply parsing API that makes it easy for writing higher
level language bindings.

The reply parsing API consists of the following functions:

redisReader *redisReaderCreate(void);
void redisReaderFree(redisReader *reader);
int redisReaderFeed(redisReader *reader, const char *buf, size_t len);
int redisReaderGetReply(redisReader *reader, void **reply);

The same set of functions are used internally by hiredis when creating a
normal Redis context, the above API just exposes it to the user for a direct
usage.

Usage

The function redisReaderCreate creates a redisReader structure that holds a
buffer with unparsed data and state for the protocol parser.

Incoming data -- most likely from a socket -- can be placed in the internal
buffer of the redisReader using redisReaderFeed. This function will make a
copy of the buffer pointed to by buf for len bytes. This data is parsed
when redisReaderGetReply is called. This function returns an integer status
and a reply object (as described above) via void **reply. The returned status
can be either REDIS_OK or REDIS_ERR, where the latter means something went
wrong (either a protocol error, or an out of memory error).

The parser limits the level of nesting for multi bulk payloads to 7. If the
multi bulk nesting level is higher than this, the parser returns an error.

Customizing replies

The function redisReaderGetReply creates redisReply and makes the function
argument reply point to the created redisReply variable. For instance, if
the response of type REDIS_REPLY_STATUS then the str field of redisReply
will hold the status as a vanilla C string. However, the functions that are
responsible for creating instances of the redisReply can be customized by
setting the fn field on the redisReader struct. This should be done
immediately after creating the redisReader.

For example, hiredis-rb
uses customized reply object functions to create Ruby objects.

Reader max buffer

Both when using the Reader API directly or when using it indirectly via a
normal Redis context, the redisReader structure uses a buffer in order to
accumulate data from the server.
Usually this buffer is destroyed when it is empty and is larger than 16
KiB in order to avoid wasting memory in unused buffers

However when working with very big payloads destroying the buffer may slow
down performances considerably, so it is possible to modify the max size of
an idle buffer changing the value of the maxbuf field of the reader structure
to the desired value. The special value of 0 means that there is no maximum
value for an idle buffer, so the buffer will never get freed.

For instance if you have a normal Redis context you can set the maximum idle
buffer to zero (unlimited) just with:

context->reader->maxbuf = 0;

This should be done only in order to maximize performances when working with
large payloads. The context should be set back to REDIS_READER_MAX_BUF again
as soon as possible in order to prevent allocation of useless memory.

AUTHORS

Hiredis was written by Salvatore Sanfilippo (antirez at gmail) and
Pieter Noordhuis (pcnoordhuis at gmail) and is released under the BSD license.
Hiredis is currently maintained by Matt Stancliff (matt at genges dot com) and
Jan-Erik Rediger (janerik at fnordig dot com)

Overview (`ext/hiredis-1.0.2/README.md`)

This Readme reflects the latest changed in the master branch. See v1.0.0 for the Readme and documentation for the latest release (API/ABI history).

HIREDIS

Hiredis is a minimalistic C client library for the Redis database.

Hiredis only supports the binary-safe Redis protocol, so you can use it with any
Redis version >= 1.2.0.

The library comes with multiple APIs. There is the
synchronous API, the asynchronous API and the reply parsing API.

Upgrading to `1.0.2`

NOTE: v1.0.1 erroneously bumped SONAME, which is why it is skipped here.

Version 1.0.2 is simply 1.0.0 with a fix for CVE-2021-32765. They are otherwise identical.

Upgrading to `1.0.0`

Version 1.0.0 marks the first stable release of Hiredis.
It includes some minor breaking changes, mostly to make the exposed API more uniform and self-explanatory.
It also bundles the updated sds library, to sync up with upstream and Redis.
For code changes see the Changelog.

Note: As described below, a few member names have been changed but most applications should be able to upgrade with minor code changes and recompiling.

IMPORTANT: Breaking changes from `0.14.1` -> `1.0.0`

redisContext has two additional members (free_privdata, and privctx).
redisOptions.timeout has been renamed to redisOptions.connect_timeout, and we've added redisOptions.command_timeout.
redisReplyObjectFunctions.createArray now takes size_t instead of int for its length parameter.

IMPORTANT: Breaking changes when upgrading from 0.13.x -> 0.14.x

Change redisReply.len to size_t, as it denotes the the size of a string

User code should compare this to size_t values as well. If it was used to
compare to other values, casting might be necessary or can be removed, if
casting was applied before.

Upgrading from `<0.9.0`

Version 0.9.0 is a major overhaul of hiredis in every aspect. However, upgrading existing
code using hiredis should not be a big pain. The key thing to keep in mind when
upgrading is that hiredis >= 0.9.0 uses a redisContext* to keep state, in contrast to
the stateless 0.0.1 that only has a file descriptor to work with.

Synchronous API

To consume the synchronous API, there are only a few function calls that need to be introduced:

redisContext *redisConnect(const char *ip, int port);
void *redisCommand(redisContext *c, const char *format, ...);
void freeReplyObject(void *reply);

Connecting

redisContext *c = redisConnect("127.0.0.1", 6379);
if (c == NULL || c->err) {
    if (c) {
        printf("Error: %s\n", c->errstr);
        // handle error
    } else {
        printf("Can't allocate redis context\n");
    }
}

Note: A redisContext is not thread-safe.

Sending commands

There are several ways to issue commands to Redis. The first that will be introduced is
redisCommand. This function takes a format similar to printf. In the simplest form,
it is used like this:

reply = redisCommand(context, "SET foo bar");

The specifier %s interpolates a string in the command, and uses strlen to
determine the length of the string:

reply = redisCommand(context, "SET foo %s", value);

When you need to pass binary safe strings in a command, the %b specifier can be
used. Together with a pointer to the string, it requires a size_t length argument
of the string:

reply = redisCommand(context, "SET foo %b", value, (size_t) valuelen);

reply = redisCommand(context, "SET key:%s %s", myid, value);

Using replies

The standard replies that redisCommand are of the type redisReply. The
type field in the redisReply should be used to test what kind of reply
was received:

RESP2

REDIS_REPLY_STATUS:
- The command replied with a status reply. The status string can be accessed using reply->str.
  The length of this string can be accessed using reply->len.
REDIS_REPLY_ERROR:
- The command replied with an error. The error string can be accessed identical to REDIS_REPLY_STATUS.
REDIS_REPLY_INTEGER:
- The command replied with an integer. The integer value can be accessed using the
  reply->integer field of type long long.
REDIS_REPLY_NIL:
- The command replied with a nil object. There is no data to access.
REDIS_REPLY_STRING:
- A bulk (string) reply. The value of the reply can be accessed using reply->str.
  The length of this string can be accessed using reply->len.
REDIS_REPLY_ARRAY:
- A multi bulk reply. The number of elements in the multi bulk reply is stored in
  reply->elements. Every element in the multi bulk reply is a redisReply object as well
  and can be accessed via reply->element[..index..].
  Redis may reply with nested arrays but this is fully supported.

RESP3

Hiredis also supports every new RESP3 data type which are as follows. For more information about the protocol see the RESP3 specification.

REDIS_REPLY_DOUBLE:
- The command replied with a double-precision floating point number.
  The value is stored as a string in the str member, and can be converted with strtod or similar.
REDIS_REPLY_BOOL:
- A boolean true/false reply.
  The value is stored in the integer member and will be either 0 or 1.
REDIS_REPLY_MAP:
- An array with the added invariant that there will always be an even number of elements.
  The MAP is functionally equivelant to REDIS_REPLY_ARRAY except for the previously mentioned invariant.
REDIS_REPLY_SET:
- An array response where each entry is unique.
  Like the MAP type, the data is identical to an array response except there are no duplicate values.
REDIS_REPLY_PUSH:
- An array that can be generated spontaneously by Redis.
  This array response will always contain at least two subelements. The first contains the type of PUSH message (e.g. message, or invalidate), and the second being a sub-array with the PUSH payload itself.
REDIS_REPLY_ATTR:
- An array structurally identical to a MAP but intended as meta-data about a reply.
  As of Redis 6.0.6 this reply type is not used in Redis
REDIS_REPLY_BIGNUM:
- A string representing an arbitrarily large signed or unsigned integer value.
  The number will be encoded as a string in the str member of redisReply.
REDIS_REPLY_VERB:
- A verbatim string, intended to be presented to the user without modification.
  The string payload is stored in the str memeber, and type data is stored in the vtype member (e.g. txt for raw text or md for markdown).

Important: the current version of hiredis (1.0.0) frees replies when the
asynchronous API is used. This means you should not call freeReplyObject when
you use this API. The reply is cleaned up by hiredis after the callback
returns. We may introduce a flag to make this configurable in future versions of the library.

Cleaning up

To disconnect and free the context the following function can be used:

void redisFree(redisContext *c);

This function immediately closes the socket and then frees the allocations done in
creating the context.

Sending commands (cont'd)

Together with redisCommand, the function redisCommandArgv can be used to issue commands.
It has the following prototype:

void *redisCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);

The return value has the same semantic as redisCommand.

Pipelining

To explain how Hiredis supports pipelining in a blocking connection, there needs to be
understanding of the internal execution flow.

The input buffer is non-empty:
- Try to parse a single reply from the input buffer and return it
- If no reply could be parsed, continue at 2
The input buffer is empty:
- Write the entire output buffer to the socket
- Read from the socket until a single reply could be parsed

void redisAppendCommand(redisContext *c, const char *format, ...);
void redisAppendCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);

The following examples shows a simple pipeline (resulting in only a single call to write(2) and
a single call to read(2)):

redisReply *reply;
redisAppendCommand(context,"SET foo bar");
redisAppendCommand(context,"GET foo");
redisGetReply(context,(void *)&reply); // reply for SET
freeReplyObject(reply);
redisGetReply(context,(void *)&reply); // reply for GET
freeReplyObject(reply);

This API can also be used to implement a blocking subscriber:

reply = redisCommand(context,"SUBSCRIBE foo");
freeReplyObject(reply);
while(redisGetReply(context,(void *)&reply) == REDIS_OK) {
    // consume message
    freeReplyObject(reply);
}

Errors

REDIS_ERR_IO:
There was an I/O error while creating the connection, trying to write
to the socket or read from the socket. If you included errno.h in your
application, you can use the global errno variable to find out what is
wrong.
REDIS_ERR_EOF:
The server closed the connection which resulted in an empty read.
REDIS_ERR_PROTOCOL:
There was an error while parsing the protocol.
REDIS_ERR_OTHER:
Any other error. Currently, it is only used when a specified hostname to connect
to cannot be resolved.

In every case, the errstr field in the context will be set to hold a string representation
of the error.

Asynchronous API

Hiredis comes with an asynchronous API that works easily with any event library.
Examples are bundled that show using Hiredis with libev
and libevent.

Connecting

Note: A redisAsyncContext is not thread-safe.

redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
    printf("Error: %s\n", c->errstr);
    // handle error
}

void(const redisAsyncContext *c, int status);

The context object is always freed after the disconnect callback fired. When a reconnect is needed,
the disconnect callback is a good point to do so.

Setting the disconnect callback can only be done once per context. For subsequent calls it will
return REDIS_ERR. The function to set the disconnect callback has the following prototype:

int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);

ac->data may be used to pass user data to this callback, the same can be done for redisConnectCallback.

Sending commands and their callbacks

void(redisAsyncContext *c, void *reply, void *privdata);

The privdata argument can be used to curry arbitrary data to the callback from the point where
the command is initially queued for execution.

The functions that can be used to issue commands in an asynchronous context are:

int redisAsyncCommand(
  redisAsyncContext *ac, redisCallbackFn *fn, void *privdata,
  const char *format, ...);
int redisAsyncCommandArgv(
  redisAsyncContext *ac, redisCallbackFn *fn, void *privdata,
  int argc, const char **argv, const size_t *argvlen);

All pending callbacks are called with a NULL reply when the context encountered an error.

Disconnecting

An asynchronous connection can be terminated using:

void redisAsyncDisconnect(redisAsyncContext *ac);

Hooking it up to event library X

There are a few hooks that need to be set on the context object after it is created.
See the adapters/ directory for bindings to libev and libevent.

Reply parsing API

Hiredis comes with a reply parsing API that makes it easy for writing higher
level language bindings.

The reply parsing API consists of the following functions:

redisReader *redisReaderCreate(void);
void redisReaderFree(redisReader *reader);
int redisReaderFeed(redisReader *reader, const char *buf, size_t len);
int redisReaderGetReply(redisReader *reader, void **reply);

The same set of functions are used internally by hiredis when creating a
normal Redis context, the above API just exposes it to the user for a direct
usage.

Usage

The function redisReaderCreate creates a redisReader structure that holds a
buffer with unparsed data and state for the protocol parser.

The parser limits the level of nesting for multi bulk payloads to 7. If the
multi bulk nesting level is higher than this, the parser returns an error.

Customizing replies

For example, hiredis-rb
uses customized reply object functions to create Ruby objects.

Reader max buffer

For instance if you have a normal Redis context you can set the maximum idle
buffer to zero (unlimited) just with:

context->reader->maxbuf = 0;

Reader max array elements

By default the hiredis reply parser sets the maximum number of multi-bulk elements
to 2^32 - 1 or 4,294,967,295 entries. If you need to process multi-bulk replies
with more than this many elements you can set the value higher or to zero, meaning
unlimited with:

context->reader->maxelements = 0;

SSL/TLS Support

Building

SSL/TLS support is not built by default and requires an explicit flag:

text

make USE_SSL=1

This requires OpenSSL development package (e.g. including header files to be
available.

When enabled, SSL/TLS support is built into extra libhiredis_ssl.a and
libhiredis_ssl.so static/dynamic libraries. This leaves the original libraries
unaffected so no additional dependencies are introduced.

Using it

First, you'll need to make sure you include the SSL header file:

#include "hiredis.h"
#include "hiredis_ssl.h"

You will also need to link against libhiredis_ssl, in addition to
libhiredis and add -lssl -lcrypto to satisfy its dependencies.

Hiredis implements SSL/TLS on top of its normal redisContext or
redisAsyncContext, so you will need to establish a connection first and then
initiate an SSL/TLS handshake.

Hiredis OpenSSL Wrappers

Before Hiredis can negotiate an SSL/TLS connection, it is necessary to
initialize OpenSSL and create a context. You can do that in two ways:

Work directly with the OpenSSL API to initialize the library's global context
and create SSL_CTX * and SSL * contexts. With an SSL * object you can
call redisInitiateSSL().
Work with a set of Hiredis-provided wrappers around OpenSSL, create a
redisSSLContext object to hold configuration and use
redisInitiateSSLWithContext() to initiate the SSL/TLS handshake.

/* An Hiredis SSL context. It holds SSL configuration and can be reused across
 * many contexts.
 */
redisSSLContext *ssl;

/* An error variable to indicate what went wrong, if the context fails to
 * initialize.
 */
redisSSLContextError ssl_error;

/* Initialize global OpenSSL state.
 *
 * You should call this only once when your app initializes, and only if
 * you don't explicitly or implicitly initialize OpenSSL it elsewhere.
 */
redisInitOpenSSL();

/* Create SSL context */
ssl = redisCreateSSLContext(
    "cacertbundle.crt",     /* File name of trusted CA/ca bundle file, optional */
    "/path/to/certs",       /* Path of trusted certificates, optional */
    "client_cert.pem",      /* File name of client certificate file, optional */
    "client_key.pem",       /* File name of client private key, optional */
    "redis.mydomain.com",   /* Server name to request (SNI), optional */
    &ssl_error
    ) != REDIS_OK) {
        printf("SSL error: %s\n", redisSSLContextGetError(ssl_error);
        /* Abort... */
    }

/* Create Redis context and establish connection */
c = redisConnect("localhost", 6443);
if (c == NULL || c->err) {
    /* Handle error and abort... */
}

/* Negotiate SSL/TLS */
if (redisInitiateSSLWithContext(c, ssl) != REDIS_OK) {
    /* Handle error, in c->err / c->errstr */
}

RESP3 PUSH replies

Redis 6.0 introduced PUSH replies with the reply-type >. These messages are generated spontaneously and can arrive at any time, so must be handled using callbacks.

Default behavior

Hiredis installs handlers on redisContext and redisAsyncContext by default, which will intercept and free any PUSH replies detected. This means existing code will work as-is after upgrading to Redis 6 and switching to RESP3.

Custom PUSH handler prototypes

The callback prototypes differ between redisContext and redisAsyncContext.

redisContext

void my_push_handler(void *privdata, void *reply) {
    /* Handle the reply */

    /* Note: We need to free the reply in our custom handler for
             blocking contexts.  This lets us keep the reply if
             we want. */
    freeReplyObject(reply);
}

redisAsyncContext

void my_async_push_handler(redisAsyncContext *ac, void *reply) {
    /* Handle the reply */

    /* Note:  Because async hiredis always frees replies, you should
              not call freeReplyObject in an async push callback. */
}

Installing a custom handler

There are two ways to set your own PUSH handlers.

Set push_cb or async_push_cb in the redisOptions struct and connect with redisConnectWithOptions or redisAsyncConnectWithOptions.

redisOptions = {0};
REDIS_OPTIONS_SET_TCP(&options, "127.0.0.1", 6379);
options->push_cb = my_push_handler;
redisContext *context = redisConnectWithOptions(&options);

Call redisSetPushCallback or redisAsyncSetPushCallback on a connected context.
c
```
redisContext *context = redisConnect("127.0.0.1", 6379);
redisSetPushCallback(context, my_push_handler);
```
Note redisSetPushCallback and redisAsyncSetPushCallback both return any currently configured handler, making it easy to override and then return to the old value.

Specifying no handler

If you have a unique use-case where you don't want hiredis to automatically intercept and free PUSH replies, you will want to configure no handler at all. This can be done in two ways.

Set the REDIS_OPT_NO_PUSH_AUTOFREE flag in redisOptions and leave the callback function pointer NULL.

redisOptions = {0};
REDIS_OPTIONS_SET_TCP(&options, "127.0.0.1", 6379);
options->options |= REDIS_OPT_NO_PUSH_AUTOFREE;
redisContext *context = redisConnectWithOptions(&options);

Call redisSetPushCallback with NULL once connected.
c
```
redisContext *context = redisConnect("127.0.0.1", 6379);
redisSetPushCallback(context, NULL);
```
Note: With no handler configured, calls to redisCommand may generate more than one reply, so this strategy is only applicable when there's some kind of blockingredisGetReply() loop (e.g. MONITOR or SUBSCRIBE workloads).

Allocator injection

Hiredis uses a pass-thru structure of function pointers defined in alloc.h that contain the currently configured allocation and deallocation functions. By default they just point to libc (malloc, calloc, realloc, etc).

Overriding

One can override the allocators like so:

hiredisAllocFuncs myfuncs = {
    .mallocFn = my_malloc,
    .callocFn = my_calloc,
    .reallocFn = my_realloc,
    .strdupFn = my_strdup,
    .freeFn = my_free,
};

// Override allocators (function returns current allocators if needed)
hiredisAllocFuncs orig = hiredisSetAllocators(&myfuncs);

To reset the allocators to their default libc function simply call:

hiredisResetAllocators();

AUTHORS

Salvatore Sanfilippo (antirez at gmail),
Pieter Noordhuis (pcnoordhuis at gmail)
Michael Grunder (michael dot grunder at gmail)

Hiredis is released under the BSD license.

Overview (`ext/http-parser/README.md`)

HTTP Parser

This is a parser for HTTP messages written in C. It parses both requests and
responses. The parser is designed to be used in performance HTTP
applications. It does not make any syscalls nor allocations, it does not
buffer data, it can be interrupted at anytime. Depending on your
architecture, it only requires about 40 bytes of data per message
stream (in a web server that is per connection).

Features:

No dependencies
Handles persistent streams (keep-alive).
Decodes chunked encoding.
Upgrade support
Defends against buffer overflow attacks.

The parser extracts the following information from HTTP messages:

Header fields and values
Content-Length
Request method
Response status code
Transfer-Encoding
HTTP version
Request URL
Message body

Usage

One http_parser object is used per TCP connection. Initialize the struct
using http_parser_init() and set the callbacks. That might look something
like this for a request parser:

http_parser_settings settings;
settings.on_url = my_url_callback;
settings.on_header_field = my_header_field_callback;
/* ... */

http_parser *parser = malloc(sizeof(http_parser));
http_parser_init(parser, HTTP_REQUEST);
parser->data = my_socket;

When data is received on the socket execute the parser and check for errors.

size_t len = 80*1024, nparsed;
char buf[len];
ssize_t recved;

recved = recv(fd, buf, len, 0);

if (recved < 0) {
  /* Handle error. */
}

/* Start up / continue the parser.
 * Note we pass recved==0 to signal that EOF has been received.
 */
nparsed = http_parser_execute(parser, &settings, buf, recved);

if (parser->upgrade) {
  /* handle new protocol */
} else if (nparsed != recved) {
  /* Handle error. Usually just close the connection. */
}

HTTP needs to know where the end of the stream is. For example, sometimes
servers send responses without Content-Length and expect the client to
consume input (for the body) until EOF. To tell http_parser about EOF, give
0 as the fourth parameter to http_parser_execute(). Callbacks and errors
can still be encountered during an EOF, so one must still be prepared
to receive them.

Scalar valued message information such as status_code, method, and the
HTTP version are stored in the parser structure. This data is only
temporally stored in http_parser and gets reset on each new message. If
this information is needed later, copy it out of the structure during the
headers_complete callback.

The parser decodes the transfer-encoding for both requests and responses
transparently. That is, a chunked encoding is decoded before being sent to
the on_body callback.

The Special Problem of Upgrade

HTTP supports upgrading the connection to a different protocol. An
increasingly common example of this is the WebSocket protocol which sends
a request like

text

GET /demo HTTP/1.1
    Upgrade: WebSocket
    Connection: Upgrade
    Host: example.com
    Origin: http://example.com
    WebSocket-Protocol: sample

followed by non-HTTP data.

(See RFC6455 for more information the
WebSocket protocol.)

To support this, the parser will treat this as a normal HTTP message without a
body, issuing both on_headers_complete and on_message_complete callbacks. However
http_parser_execute() will stop parsing at the end of the headers and return.

The user is expected to check if parser->upgrade has been set to 1 after
http_parser_execute() returns. Non-HTTP data begins at the buffer supplied
offset by the return value of http_parser_execute().

Callbacks

During the http_parser_execute() call, the callbacks set in
http_parser_settings will be executed. The parser maintains state and
never looks behind, so buffering the data is not necessary. If you need to
save certain data for later usage, you can do that from the callbacks.

There are two types of callbacks:

notification typedef int (*http_cb) (http_parser*);
Callbacks: on_message_begin, on_headers_complete, on_message_complete.
data typedef int (*http_data_cb) (http_parser*, const char *at, size_t length);
Callbacks: (requests only) on_url,
(common) on_header_field, on_header_value, on_body;

Callbacks must return 0 on success. Returning a non-zero value indicates
error to the parser, making it exit immediately.

For cases where it is necessary to pass local information to/from a callback,
the http_parser object's data field can be used.
An example of such a case is when using threads to handle a socket connection,
parse a request, and then give a response over that socket. By instantiation
of a thread-local struct containing relevant data (e.g. accepted socket,
allocated memory for callbacks to write into, etc), a parser's callbacks are
able to communicate data between the scope of the thread and the scope of the
callback in a threadsafe manner. This allows http-parser to be used in
multi-threaded contexts.

Example:

typedef struct {
  socket_t sock;
  void* buffer;
  int buf_len;
 } custom_data_t;


int my_url_callback(http_parser* parser, const char *at, size_t length) {
  /* access to thread local custom_data_t struct.
  Use this access save parsed data for later use into thread local
  buffer, or communicate over socket
  */
  parser->data;
  ...
  return 0;
}

...

void http_parser_thread(socket_t sock) {
 int nparsed = 0;
 /* allocate memory for user data */
 custom_data_t *my_data = malloc(sizeof(custom_data_t));

 /* some information for use by callbacks.
 * achieves thread -> callback information flow */
 my_data->sock = sock;

 /* instantiate a thread-local parser */
 http_parser *parser = malloc(sizeof(http_parser));
 http_parser_init(parser, HTTP_REQUEST); /* initialise parser */
 /* this custom data reference is accessible through the reference to the
 parser supplied to callback functions */
 parser->data = my_data;

 http_parser_settings settings; /* set up callbacks */
 settings.on_url = my_url_callback;

 /* execute parser */
 nparsed = http_parser_execute(parser, &settings, buf, recved);

 ...
 /* parsed information copied from callback.
 can now perform action on data copied into thread-local memory from callbacks.
 achieves callback -> thread information flow */
 my_data->buffer;
 ...
}

In case you parse HTTP message in chunks (i.e. read() request line
from socket, parse, read half headers, parse, etc) your data callbacks
may be called more than once. Http-parser guarantees that data pointer is only
valid for the lifetime of callback. You can also read() into a heap allocated
buffer to avoid copying memory around if this fits your application.

Reading headers may be a tricky task if you read/parse headers partially.
Basically, you need to remember whether last header callback was field or value
and apply the following logic:

text

(on_header_field and on_header_value shortened to on_h_*)
 ------------------------ ------------ --------------------------------------------
| State (prev. callback) | Callback   | Description/action                         |
 ------------------------ ------------ --------------------------------------------
| nothing (first call)   | on_h_field | Allocate new buffer and copy callback data |
|                        |            | into it                                    |
 ------------------------ ------------ --------------------------------------------
| value                  | on_h_field | New header started.                        |
|                        |            | Copy current name,value buffers to headers |
|                        |            | list and allocate new buffer for new name  |
 ------------------------ ------------ --------------------------------------------
| field                  | on_h_field | Previous name continues. Reallocate name   |
|                        |            | buffer and append callback data to it      |
 ------------------------ ------------ --------------------------------------------
| field                  | on_h_value | Value for current header started. Allocate |
|                        |            | new buffer and copy callback data to it    |
 ------------------------ ------------ --------------------------------------------
| value                  | on_h_value | Value continues. Reallocate value buffer   |
|                        |            | and append callback data to it             |
 ------------------------ ------------ --------------------------------------------

Parsing URLs

A simplistic zero-copy URL parser is provided as http_parser_parse_url().
Users of this library may wish to use it to parse URLs constructed from
consecutive on_url callbacks.

See examples of reading in headers:

Overview (`ext/inja/README.md`)

Inja is a template engine for modern C++, loosely inspired by jinja for python. It has an easy and yet powerful template syntax with all variables, loops, conditions, includes, callbacks, and comments you need, nested and combined as you like. Inja uses the wonderful json library by nlohmann for data input. Most importantly, inja needs only two header files, which is (nearly) as trivial as integration in C++ can get. Of course, everything is tested on all relevant compilers. Here is what it looks like:

.cpp

json data;
data["name"] = "world";

inja::render("Hello {{ name }}!", data); // Returns "Hello world!"

Integration

Inja is a headers only library, which can be downloaded from the releases or directly from the include/ or single_include/ folder. Inja uses nlohmann/json.hpp (>= v3.8.0) as its single dependency, so make sure it can be included from inja.hpp. json can be downloaded here. Then integration is as easy as:

.cpp

#include <inja.hpp>

// Just for convenience
using namespace inja;

If you are using the Meson Build System, then you can wrap this repository as a subproject.

If you are using Conan to manage your dependencies, have a look at this repository. Please file issues here if you experience problems with the packages.

You can also integrate inja in your project using Hunter, a package manager for C++.

If you are using vcpkg on your project for external dependencies, then you can use the inja package. Please see the vcpkg project for any issues regarding the packaging.

If you are using cget, you can install the latest development version with cget install pantor/inja. A specific version can be installed with cget install pantor/[email protected].

On macOS, you can install inja via Homebrew and brew install inja.

If you are using conda, you can install the latest version from conda-forge with conda install -c conda-forge inja.

Tutorial

This tutorial will give you an idea how to use inja. It will explain the most important concepts and give practical advices using examples and executable code. Beside this tutorial, you may check out the documentation.

Template Rendering

The basic template rendering takes a template as a std::string and a json object for all data. It returns the rendered template as an std::string.

.cpp

json data;
data["name"] = "world";

render("Hello {{ name }}!", data); // Returns std::string "Hello world!"
render_to(std::cout, "Hello {{ name }}!", data); // Writes "Hello world!" to stream

For more advanced usage, an environment is recommended.

.cpp

Environment env;

// Render a string with json data
std::string result = env.render("Hello {{ name }}!", data); // "Hello world!"

// Or directly read a template file
Template temp = env.parse_template("./templates/greeting.txt");
std::string result = env.render(temp, data); // "Hello world!"

data["name"] = "Inja";
std::string result = env.render(temp, data); // "Hello Inja!"

// Or read the template file (and/or the json file) directly from the environment
result = env.render_file("./templates/greeting.txt", data);
result = env.render_file_with_json_file("./templates/greeting.txt", "./data.json");

// Or write a rendered template file
env.write(temp, data, "./result.txt");
env.write_with_json_file("./templates/greeting.txt", "./data.json", "./result.txt");

The environment class can be configured to your needs.

.cpp

// With default settings
Environment env_default;

// With global path to template files and where files will be saved
Environment env_1 {"../path/templates/"};

// With separate input and output path
Environment env_2 {"../path/templates/", "../path/results/"};

// With other opening and closing strings (here the defaults)
env.set_expression("{{", "}}"); // Expressions
env.set_comment("{#", "#}"); // Comments
env.set_statement("{%", "%}"); // Statements {% %} for many things, see below
env.set_line_statement("##"); // Line statements ## (just an opener)

Variables

Variables are rendered within the {{ ... }} expressions.

.cpp

json data;
data["neighbour"] = "Peter";
data["guests"] = {"Jeff", "Tom", "Patrick"};
data["time"]["start"] = 16;
data["time"]["end"] = 22;

// Indexing in array
render("{{ guests.1 }}", data); // "Tom"

// Objects
render("{{ time.start }} to {{ time.end + 1 }}pm", data); // "16 to 23pm"

If no variable is found, valid JSON is printed directly, otherwise an inja::RenderError is thrown.

Statements

Statements can be written either with the {% ... %} syntax or the ## syntax for entire lines. Note that ## needs to start the line without indentation. The most important statements are loops, conditions and file includes. All statements can be nested.

Loops

.cpp

// Combining loops and line statements
render(R"(Guest List:
## for guest in guests
	{{ loop.index1 }}: {{ guest }}
## endfor )", data)

/* Guest List:
	1: Jeff
	2: Tom
	3: Patrick */

In a loop, the special variables loop.index (number), loop.index1 (number), loop.is_first (boolean) and loop.is_last (boolean) are defined. In nested loops, the parent loop variables are available e.g. via loop.parent.index. You can also iterate over objects like {% for key, value in time %}.

Conditions

Conditions support the typical if, else if and else statements. Following conditions are for example possible:

.cpp

// Standard comparisons with a variable
render("{% if time.hour >= 20 %}Serve{% else if time.hour >= 18 %}Make{% endif %} dinner.", data); // Serve dinner.

// Variable in list
render("{% if neighbour in guests %}Turn up the music!{% endif %}", data); // Turn up the music!

// Logical operations
render("{% if guest_count < (3+2) and all_tired %}Sleepy...{% else %}Keep going...{% endif %}", data); // Sleepy...

// Negations
render("{% if not guest_count %}The End{% endif %}", data); // The End

Includes

You can either include other in-memory templates or from the file system.

.cpp

// To include in-memory templates, add them to the environment first
inja::Template content_template = env.parse("Hello {{ neighbour }}!");
env.include_template("content", content_template);
env.render("Content: {% include \"content\" %}", data); // "Content: Hello Peter!"

// Other template files are included relative from the current file location
render("{% include \"footer.html\" %}", data);

If a corresponding template could not be found in the file system, the include callback is called:

.cpp

// The callback takes the current path and the wanted include name and returns a template
env.set_include_callback([&env](const std::string& path, const std::string& template_name) {
  return env.parse("Hello {{ neighbour }} from " + template_name);
});

// You can disable to search for templates in the file system via
env.set_search_included_templates_in_files(false);

Inja will throw an inja::RenderError if an included file is not found and no callback is specified. To disable this error, you can call env.set_throw_at_missing_includes(false).

Assignments

Variables can also be defined within the template using the set statment.

.cpp

render("{% set new_hour=23 %}{{ new_hour }}pm", data); // "23pm"
render("{% set time.start=18 %}{{ time.start }}pm", data); // using json pointers

Assignments only set the value within the rendering context; they do not modify the json object passed into the render call.

Functions

A few functions are implemented within the inja template syntax. They can be called with

text

/* Detailed source-code truncated for AI context efficiency. */

Callbacks

You can create your own and more complex functions with callbacks. These are implemented with std::function, so you can for example use C++ lambdas. Inja Arguments are a vector of json pointers.

.cpp

Environment env;

/*
 * Callbacks are defined by its:
 * - name,
 * - (optional) number of arguments,
 * - callback function.
 */
env.add_callback("double", 1, [](Arguments& args) {
	int number = args.at(0)->get<int>(); // Adapt the index and type of the argument
	return 2 * number;
});

// You can then use a callback like a regular function
env.render("{{ double(16) }}", data); // "32"

// Inja falls back to variadic callbacks if the number of expected arguments is omitted.
env.add_callback("argmax", [](Arguments& args) {
  auto result = std::max_element(args.begin(), args.end(), [](const json* a, const json* b) { return *a < *b;});
  return std::distance(args.begin(), result);
});
env.render("{{ argmax(4, 2, 6) }}", data); // "2"
env.render("{{ argmax(0, 2, 6, 8, 3) }}", data); // "3"

// A callback without argument can be used like a dynamic variable:
std::string greet = "Hello";
env.add_callback("double-greetings", 0, [greet](Arguments args) {
	return greet + " " + greet + "!";
});
env.render("{{ double-greetings }}", data); // "Hello Hello!"

You can also add a void callback without return variable, e.g. for debugging:

.cpp

env.add_void_callback("log", 1, [greet](Arguments args) {
	std::cout << "logging: " << args[0] << std::endl;
});
env.render("{{ log(neighbour) }}", data); // Prints nothing to result, only to cout...

Template Inheritance

Template inheritance allows you to build a base skeleton template that contains all the common elements and defines blocks that child templates can override. Lets show an example: The base template

.html

<!DOCTYPE html>
<html>
<head>
  {% block head %}
  <link rel="stylesheet" href="style.css" />
  <title>{% block title %}{% endblock %} - My Webpage</title>
  {% endblock %}
</head>
<body>
  <div id="content">{% block content %}{% endblock %}</div>
</body>
</html>

contains three blocks that child templates can fill in. The child template

.html

{% extends "base.html" %}
{% block title %}Index{% endblock %}
{% block head %}
  {{ super() }}
  <style type="text/css">
    .important { color: #336699; }
  </style>
{% endblock %}
{% block content %}
  <h1>Index</h1>
  <p class="important">
    Welcome to my blog!
  </p>
{% endblock %}

calls a parent template with the extends keyword; it should be the first element in the template. It is possible to render the contents of the parent block by calling super(). In the case of multiple levels of {% extends %}, super references may be called with an argument (e.g. super(2)) to skip levels in the inheritance tree.

Whitespace Control

In the default configuration, no whitespace is removed while rendering the file. To support a more readable template style, you can configure the environment to control whitespaces before and after a statement automatically. While enabling set_trim_blocks removes the first newline after a statement, set_lstrip_blocks strips tabs and spaces from the beginning of a line to the start of a block.

.cpp

Environment env;
env.set_trim_blocks(true);
env.set_lstrip_blocks(true);

With both trim_blocks and lstrip_blocks enabled, you can put statements on their own lines. Furthermore, you can also strip whitespaces for both statements and expressions by hand. If you add a minus sign (-) to the start or end, the whitespaces before or after that block will be removed:

.cpp

render("Hello       {{- name -}}     !", data); // "Hello Inja!"
render("{% if neighbour in guests -%}   I was there{% endif -%}   !", data); // Renders without any whitespaces

Stripping behind a statement or expression also removes any newlines.

Comments

Comments can be written with the {# ... #} syntax.

.cpp

render("Hello{# Todo #}!", data); // "Hello!"

Exceptions

Inja uses exceptions to handle ill-formed template input. However, exceptions can be switched off with either using the compiler flag -fno-exceptions or by defining the symbol INJA_NOEXCEPTION. In this case, exceptions are replaced by abort() calls.

Supported compilers

Inja uses the string_view feature of the C++17 STL. Currently, the following compilers are tested:

GCC 7 - 11 (and possibly later)
Clang 5 - 12 (and possibly later)
Microsoft Visual C++ 2017 15.0 - 2022 (and possibly later)

A list of supported compiler / os versions can be found in the CI definition.

Overview (`ext/libpqxx-7.7.3/README.md`)

libpqxx

Welcome to libpqxx, the C++ API to the PostgreSQL database management system.

Home page: http://pqxx.org/development/libpqxx/

Find libpqxx on Github: https://github.com/jtv/libpqxx

Documentation on Read The Docs: https://libpqxx.readthedocs.io

Compiling this package requires PostgreSQL to be installed -- or at least the C
headers and library for client development. The library builds on top of
PostgreSQL's standard C API, libpq, though your code won't notice.

If you're getting the code straight from the Git repo, the head of the master
branch represents the current development version. Releases are tags on
commits in master. For example, to get version 7.1.1:

text

git checkout 7.1.1

Upgrade notes

The 7.x versions require at least C++17. Make sure your compiler is up to
date. For libpqxx 8.x you will need at least C++20.

Also, 7.0 makes some breaking changes in rarely used APIs:

There is just a single connection class. It connects immediately.
Custom connection classes are no longer supported.
It's no longer possible to reactivate a connection once it's been closed.
The API for defining string conversions has changed.

If you're defining your own type conversions, 7.1 requires one additional
field in your nullness traits.

Building libpqxx

There are two different ways of building libpqxx from the command line:

Using CMake, on any system which supports it.
On Unix-like systems, using a configure script.

"Unix-like" systems include GNU/Linux, Apple macOS and the BSD family, AIX,
HP-UX, Irix, Solaris, etc. Even on Microsoft Windows, a Unix-like environment
such as WSL, Cygwin, or MinGW should work.

You'll find detailed build and install instructions in BUILDING-configure.md
and BUILDING-cmake.md, respectively.

And if you're working with Microsoft Visual Studio, have a look at Gordon
Elliott's

Easy-PQXX Build for Windows Visual Studio
project.

Documentation

Building the library, if you have the right tools installed, generates HTML
documentation in the doc/ directory. It is based on the headers in
include/pqxx/ and text in include/pqxx/doc/. This documentation is also
available online at readthedocs.

Programming with libpqxx

Your first program will involve the libpqxx classes connection (see the
pqxx/connection.hxx header), and work (a convenience alias for
transaction<> which conforms to the interface defined in
pqxx/transaction_base.hxx).

These *.hxx headers are not the ones you include in your program. Instead,
include the versions without filename suffix (e.g. pqxx/connection). Those
will include the actual .hxx files for you. This was done so that includes are
in standard C++ style (as in `` etc.), but an editor will still
recognize them as files containing C++ code.

Continuing the list of classes, you will most likely also need the result class
(pqxx/result.hxx). In a nutshell, you create a connection based on a
Postgres connection string (see below), create a work in the context of that
connection, and run one or more queries on the work which return result
objects. The results are containers of rows of data, each of which you can
treat as an array of strings: one for each field in the row. It's that simple.

Here is a simple example program to get you going, with full error handling:

c++

#include <iostream>
    #include <pqxx/pqxx>

    int main()
    {
        try
        {
            // Connect to the database.
            pqxx::connection C;
            std::cout << "Connected to " << C.dbname() << '\n';

            // Start a transaction.
            pqxx::work W{C};

            // Perform a query and retrieve all results.
            pqxx::result R{W.exec("SELECT name FROM employee")};

            // Iterate over results.
            std::cout << "Found " << R.size() << "employees:\n";
            for (auto row: R)
                std::cout << row[0].c_str() << '\n';

            // Perform a query and check that it returns no result.
            std::cout << "Doubling all employees' salaries...\n";
            W.exec0("UPDATE employee SET salary = salary*2");

            // Commit the transaction.
            std::cout << "Making changes definite: ";
            W.commit();
            std::cout << "OK.\n";
        }
        catch (std::exception const &e)
        {
            std::cerr << e.what() << '\n';
            return 1;
        }
        return 0;
    }

Connection strings

Postgres connection strings state which database server you wish to connect to,
under which username, using which password, and so on. Their format is defined
in the documentation for libpq, the C client interface for PostgreSQL.
Alternatively, these values may be defined by setting certain environment
variables as documented in e.g. the manual for psql, the command line interface
to PostgreSQL. Again the definitions are the same for libpqxx-based programs.

The connection strings and variables are not fully and definitively documented
here; this document will tell you just enough to get going. Check the
PostgreSQL documentation for authoritative information.

The connection string consists of attribute=value pairs separated by spaces,
e.g. "user=john password=1x2y3z4". The valid attributes include:

host —
Name of server to connect to, or the full file path (beginning with a
slash) to a Unix-domain socket on the local machine. Defaults to
"/tmp". Equivalent to (but overrides) environment variable PGHOST.
hostaddr —
IP address of a server to connect to; mutually exclusive with "host".
port —
Port number at the server host to connect to, or socket file name
extension for Unix-domain connections. Equivalent to (but overrides)
environment variable PGPORT.
dbname —
Name of the database to connect to. A single server may host multiple
databases. Defaults to the same name as the current user's name.
Equivalent to (but overrides) environment variable PGDATABASE.
user —
User name to connect under. This defaults to the name of the current
user, although PostgreSQL users are not necessarily the same thing as
system users.
requiressl —
If set to 1, demands an encrypted SSL connection (and fails if no SSL
connection can be created).

Settings in the connection strings override the environment variables, which in
turn override the default, on a variable-by-variable basis. You only need to
define those variables that require non-default values.

Linking with libpqxx

To link your final program, make sure you link to both the C-level libpq library
and the actual C++ library, libpqxx. With most Unix-style compilers, you'd do
this using the options

text

-lpqxx -lpq

while linking. Both libraries must be in your link path, so the linker knows
where to find them. Any dynamic libraries you use must also be in a place
where the loader can find them when loading your program at runtime.

Some users have reported problems using the above syntax, however, particularly
when multiple versions of libpqxx are partially or incorrectly installed on the
system. If you get massive link errors, try removing the "-lpqxx" argument from
the command line and replacing it with the name of the libpqxx library binary
instead. That's typically libpqxx.a, but you'll have to add the path to its
location as well, e.g. /usr/local/pqxx/lib/libpqxx.a. This will ensure that the
linker will use that exact version of the library rather than one found
elsewhere on the system, and eliminate worries about the exact right version of
the library being installed with your program..

Design goals

There are myriads of JSON libraries out there, and each may even have its reason to exist. Our class had these design goals:

Intuitive syntax. In languages such as Python, JSON feels like a first class data type. We used all the operator magic of modern C++ to achieve the same feeling in your code. Check out the examples below and you'll know what I mean.
Trivial integration. Our whole code consists of a single header file json.hpp. That's it. No library, no subproject, no dependencies, no complex build system. The class is written in vanilla C++11. All in all, everything should require no adjustment of your compiler flags or project settings.
Serious testing. Our class is heavily unit-tested and covers 100% of the code, including all exceptional behavior. Furthermore, we checked with Valgrind and the Clang Sanitizers that there are no memory leaks. Google OSS-Fuzz additionally runs fuzz tests agains all parsers 24/7, effectively executing billions of tests so far. To maintain high quality, the project is following the Core Infrastructure Initiative (CII) best practices.

Other aspects were not so important to us:

Memory efficiency. Each JSON object has an overhead of one pointer (the maximal size of a union) and one enumeration element (1 byte). The default generalization uses the following C++ data types: std::string for strings, int64_t, uint64_t or double for numbers, std::map for objects, std::vector for arrays, and bool for Booleans. However, you can template the generalized class basic_json to your needs.
Speed. There are certainly faster JSON libraries out there. However, if your goal is to speed up your development by adding JSON support with a single header, then this library is the way to go. If you know how to use a std::vector or std::map, you are already set.

See the contribution guidelines for more information.

Integration

json.hpp is the single required file in single_include/nlohmann or released here. You need to add

cpp

#include <nlohmann/json.hpp>

// for convenience
using json = nlohmann::json;

to the files you want to process JSON and set the necessary switches to enable C++11 (e.g., -std=c++11 for GCC and Clang).

You can further use file include/nlohmann/json_fwd.hpp for forward-declarations. The installation of json_fwd.hpp (as part of cmake's install step), can be achieved by setting -DJSON_MultipleHeaders=ON.

Package Managers

:beer: If you are using OS X and Homebrew, just type brew tap nlohmann/json and brew install nlohmann_json and you're set. If you want the bleeding edge rather than the latest release, use brew install nlohmann_json --HEAD.

If you are using the Meson Build System, then you can wrap this repository as a subproject.

If you are using Conan to manage your dependencies, merely add jsonformoderncpp/x.y.z@vthiery/stable to your conanfile.py's requires, where x.y.z is the release version you want to use. Please file issues here if you experience problems with the packages.

If you are using Spack to manage your dependencies, you can use the nlohmann_json package. Please see the spack project for any issues regarding the packaging.

If you are using hunter on your project for external dependencies, then you can use the nlohmann_json package. Please see the hunter project for any issues regarding the packaging.

If you are using Buckaroo, you can install this library's module with buckaroo install nlohmann/json. Please file issues here.

If you are using vcpkg on your project for external dependencies, then you can use the nlohmann-json package. Please see the vcpkg project for any issues regarding the packaging.

If you are using cget, you can install the latest development version with cget install nlohmann/json. A specific version can be installed with cget install nlohmann/[email protected]. Also, the multiple header version can be installed by adding the -DJSON_MultipleHeaders=ON flag (i.e., cget install nlohmann/json -DJSON_MultipleHeaders=ON).

If you are using CocoaPods, you can use the library by adding pod "nlohmann_json", '~>3.1.2' to your podfile (see an example). Please file issues here.

Examples

Beside the examples below, you may want to check the documentation where each function contains a separate code example (e.g., check out emplace()). All example files can be compiled and executed on their own (e.g., file emplace.cpp).

JSON as first-class data type

Here are some examples to give you an idea how to use the class.

Assume you want to create the JSON object

json

{
  "pi": 3.141,
  "happy": true,
  "name": "Niels",
  "nothing": null,
  "answer": {
    "everything": 42
  },
  "list": [1, 0, 2],
  "object": {
    "currency": "USD",
    "value": 42.99
  }
}

With this library, you could write:

cpp

// create an empty structure (null)
json j;

// add a number that is stored as double (note the implicit conversion of j to an object)
j["pi"] = 3.141;

// add a Boolean that is stored as bool
j["happy"] = true;

// add a string that is stored as std::string
j["name"] = "Niels";

// add another null object by passing nullptr
j["nothing"] = nullptr;

// add an object inside the object
j["answer"]["everything"] = 42;

// add an array that is stored as std::vector (using an initializer list)
j["list"] = { 1, 0, 2 };

// add another object (using an initializer list of pairs)
j["object"] = { {"currency", "USD"}, {"value", 42.99} };

// instead, you could also write (which looks very similar to the JSON above)
json j2 = {
  {"pi", 3.141},
  {"happy", true},
  {"name", "Niels"},
  {"nothing", nullptr},
  {"answer", {
    {"everything", 42}
  }},
  {"list", {1, 0, 2}},
  {"object", {
    {"currency", "USD"},
    {"value", 42.99}
  }}
};

Note that in all these cases, you never need to "tell" the compiler which JSON value type you want to use. If you want to be explicit or express some edge cases, the functions json::array and json::object will help:

cpp

// a way to express the empty array []
json empty_array_explicit = json::array();

// ways to express the empty object {}
json empty_object_implicit = json({});
json empty_object_explicit = json::object();

// a way to express an _array_ of key/value pairs [["currency", "USD"], ["value", 42.99]]
json array_not_object = json::array({ {"currency", "USD"}, {"value", 42.99} });

Serialization / Deserialization

To/from strings

You can create a JSON value (deserialization) by appending _json to a string literal:

cpp

// create object from string literal
json j = "{ \"happy\": true, \"pi\": 3.141 }"_json;

// or even nicer with a raw string literal
auto j2 = R"(
  {
    "happy": true,
    "pi": 3.141
  }
)"_json;

Note that without appending the _json suffix, the passed string literal is not parsed, but just used as JSON string value. That is, json j = "{ \"happy\": true, \"pi\": 3.141 }" would just store the string "{ "happy": true, "pi": 3.141 }" rather than parsing the actual object.

The above example can also be expressed explicitly using json::parse():

cpp

// parse explicitly
auto j3 = json::parse("{ \"happy\": true, \"pi\": 3.141 }");

You can also get a string representation of a JSON value (serialize):

cpp

// explicit conversion to string
std::string s = j.dump();    // {\"happy\":true,\"pi\":3.141}

// serialization with pretty printing
// pass in the amount of spaces to indent
std::cout << j.dump(4) << std::endl;
// {
//     "happy": true,
//     "pi": 3.141
// }

Note the difference between serialization and assignment:

cpp

// store a string in a JSON value
json j_string = "this is a string";

// retrieve the string value (implicit JSON to std::string conversion)
std::string cpp_string = j_string;
// retrieve the string value (explicit JSON to std::string conversion)
auto cpp_string2 = j_string.get<std::string>();

// retrieve the serialized value (explicit JSON serialization)
std::string serialized_string = j_string.dump();

// output of original string
std::cout << cpp_string << " == " << cpp_string2 << " == " << j_string.get<std::string>() << '\n';
// output of serialized value
std::cout << j_string << " == " << serialized_string << std::endl;

.dump() always returns the serialized value, and .get<std::string>() returns the originally stored string value.

Note the library only supports UTF-8. When you store strings with different encodings in the library, calling dump() may throw an exception.

To/from streams (e.g. files, string streams)

You can also use streams to serialize and deserialize:

cpp

// deserialize from standard input
json j;
std::cin >> j;

// serialize to standard output
std::cout << j;

// the setw manipulator was overloaded to set the indentation for pretty printing
std::cout << std::setw(4) << j << std::endl;

These operators work for any subclasses of std::istream or std::ostream. Here is the same example with files:

cpp

// read a JSON file
std::ifstream i("file.json");
json j;
i >> j;

// write prettified JSON to another file
std::ofstream o("pretty.json");
o << std::setw(4) << j << std::endl;

Please note that setting the exception bit for failbit is inappropriate for this use case. It will result in program termination due to the noexcept specifier in use.

Read from iterator range

You can also parse JSON from an iterator range; that is, from any container accessible by iterators whose content is stored as contiguous byte sequence, for instance a std::vector<std::uint8_t>:

cpp

std::vector<std::uint8_t> v = {'t', 'r', 'u', 'e'};
json j = json::parse(v.begin(), v.end());

You may leave the iterators for the range [begin, end):

cpp

std::vector<std::uint8_t> v = {'t', 'r', 'u', 'e'};
json j = json::parse(v);

SAX interface

The library uses a SAX-like interface with the following functions:

cpp

// called when null is parsed
bool null();

// called when a boolean is parsed; value is passed
bool boolean(bool val);

// called when a signed or unsigned integer number is parsed; value is passed
bool number_integer(number_integer_t val);
bool number_unsigned(number_unsigned_t val);

// called when a floating-point number is parsed; value and original string is passed
bool number_float(number_float_t val, const string_t& s);

// called when a string is parsed; value is passed and can be safely moved away
bool string(string_t& val);

// called when an object or array begins or ends, resp. The number of elements is passed (or -1 if not known)
bool start_object(std::size_t elements);
bool end_object();
bool start_array(std::size_t elements);
bool end_array();
// called when an object key is parsed; value is passed and can be safely moved away
bool key(string_t& val);

// called when a parse error occurs; byte position, the last token, and an exception is passed
bool parse_error(std::size_t position, const std::string& last_token, const detail::exception& ex);

The return value of each function determines whether parsing should proceed.

To implement your own SAX handler, proceed as follows:

Implement the SAX interface in a class. You can use class nlohmann::json_sax<json> as base class, but you can also use any class where the functions described above are implemented and public.
Create an object of your SAX interface class, e.g. my_sax.
Call bool json::sax_parse(input, &my_sax); where the first parameter can be any input like a string or an input stream and the second parameter is a pointer to your SAX interface.

Note the sax_parse function only returns a bool indicating the result of the last executed SAX event. It does not return a json value - it is up to you to decide what to do with the SAX events. Furthermore, no exceptions are thrown in case of a parse error - it is up to you what to do with the exception object passed to your parse_error implementation. Internally, the SAX interface is used for the DOM parser (class json_sax_dom_parser) as well as the acceptor (json_sax_acceptor), see file json_sax.hpp.

STL-like access

We designed the JSON class to behave just like an STL container. In fact, it satisfies the ReversibleContainer requirement.

cpp

// create an array using push_back
json j;
j.push_back("foo");
j.push_back(1);
j.push_back(true);

// also use emplace_back
j.emplace_back(1.78);

// iterate the array
for (json::iterator it = j.begin(); it != j.end(); ++it) {
  std::cout << *it << '\n';
}

// range-based for
for (auto& element : j) {
  std::cout << element << '\n';
}

// getter/setter
const std::string tmp = j[0];
j[1] = 42;
bool foo = j.at(2);

// comparison
j == "[\"foo\", 1, true]"_json;  // true

// other stuff
j.size();     // 3 entries
j.empty();    // false
j.type();     // json::value_t::array
j.clear();    // the array is empty again

// convenience type checkers
j.is_null();
j.is_boolean();
j.is_number();
j.is_object();
j.is_array();
j.is_string();

// create an object
json o;
o["foo"] = 23;
o["bar"] = false;
o["baz"] = 3.141;

// also use emplace
o.emplace("weather", "sunny");

// special iterator member functions for objects
for (json::iterator it = o.begin(); it != o.end(); ++it) {
  std::cout << it.key() << " : " << it.value() << "\n";
}

// find an entry
if (o.find("foo") != o.end()) {
  // there is an entry with key "foo"
}

// or simpler using count()
int foo_present = o.count("foo"); // 1
int fob_present = o.count("fob"); // 0

// delete an entry
o.erase("foo");

Conversion from STL containers

Any sequence container (std::array, std::vector, std::deque, std::forward_list, std::list) whose values can be used to construct JSON values (e.g., integers, floating point numbers, Booleans, string types, or again STL containers described in this section) can be used to create a JSON array. The same holds for similar associative containers (std::set, std::multiset, std::unordered_set, std::unordered_multiset), but in these cases the order of the elements of the array depends on how the elements are ordered in the respective STL container.

cpp

std::vector<int> c_vector {1, 2, 3, 4};
json j_vec(c_vector);
// [1, 2, 3, 4]

std::deque<double> c_deque {1.2, 2.3, 3.4, 5.6};
json j_deque(c_deque);
// [1.2, 2.3, 3.4, 5.6]

std::list<bool> c_list {true, true, false, true};
json j_list(c_list);
// [true, true, false, true]

std::forward_list<int64_t> c_flist {12345678909876, 23456789098765, 34567890987654, 45678909876543};
json j_flist(c_flist);
// [12345678909876, 23456789098765, 34567890987654, 45678909876543]

std::array<unsigned long, 4> c_array {{1, 2, 3, 4}};
json j_array(c_array);
// [1, 2, 3, 4]

std::set<std::string> c_set {"one", "two", "three", "four", "one"};
json j_set(c_set); // only one entry for "one" is used
// ["four", "one", "three", "two"]

std::unordered_set<std::string> c_uset {"one", "two", "three", "four", "one"};
json j_uset(c_uset); // only one entry for "one" is used
// maybe ["two", "three", "four", "one"]

std::multiset<std::string> c_mset {"one", "two", "one", "four"};
json j_mset(c_mset); // both entries for "one" are used
// maybe ["one", "two", "one", "four"]

std::unordered_multiset<std::string> c_umset {"one", "two", "one", "four"};
json j_umset(c_umset); // both entries for "one" are used
// maybe ["one", "two", "one", "four"]

Likewise, any associative key-value containers (std::map, std::multimap, std::unordered_map, std::unordered_multimap) whose keys can construct an std::string and whose values can be used to construct JSON values (see examples above) can be used to create a JSON object. Note that in case of multimaps only one key is used in the JSON object and the value depends on the internal order of the STL container.

cpp

std::map<std::string, int> c_map { {"one", 1}, {"two", 2}, {"three", 3} };
json j_map(c_map);
// {"one": 1, "three": 3, "two": 2 }

std::unordered_map<const char*, double> c_umap { {"one", 1.2}, {"two", 2.3}, {"three", 3.4} };
json j_umap(c_umap);
// {"one": 1.2, "two": 2.3, "three": 3.4}

std::multimap<std::string, bool> c_mmap { {"one", true}, {"two", true}, {"three", false}, {"three", true} };
json j_mmap(c_mmap); // only one entry for key "three" is used
// maybe {"one": true, "two": true, "three": true}

std::unordered_multimap<std::string, bool> c_ummap { {"one", true}, {"two", true}, {"three", false}, {"three", true} };
json j_ummap(c_ummap); // only one entry for key "three" is used
// maybe {"one": true, "two": true, "three": true}

JSON Pointer and JSON Patch

The library supports JSON Pointer (RFC 6901) as alternative means to address structured values. On top of this, JSON Patch (RFC 6902) allows to describe differences between two JSON values - effectively allowing patch and diff operations known from Unix.

cpp

// a JSON value
json j_original = R"({
  "baz": ["one", "two", "three"],
  "foo": "bar"
})"_json;

// access members with a JSON pointer (RFC 6901)
j_original["/baz/1"_json_pointer];
// "two"

// a JSON patch (RFC 6902)
json j_patch = R"([
  { "op": "replace", "path": "/baz", "value": "boo" },
  { "op": "add", "path": "/hello", "value": ["world"] },
  { "op": "remove", "path": "/foo"}
])"_json;

// apply the patch
json j_result = j_original.patch(j_patch);
// {
//    "baz": "boo",
//    "hello": ["world"]
// }

// calculate a JSON patch from two JSON values
json::diff(j_result, j_original);
// [
//   { "op":" replace", "path": "/baz", "value": ["one", "two", "three"] },
//   { "op": "remove","path": "/hello" },
//   { "op": "add", "path": "/foo", "value": "bar" }
// ]

JSON Merge Patch

The library supports JSON Merge Patch (RFC 7386) as a patch format. Instead of using JSON Pointer (see above) to specify values to be manipulated, it describes the changes using a syntax that closely mimics the document being modified.

cpp

// a JSON value
json j_document = R"({
  "a": "b",
  "c": {
    "d": "e",
    "f": "g"
  }
})"_json;

// a patch
json j_patch = R"({
  "a":"z",
  "c": {
    "f": null
  }
})"_json;

// apply the patch
j_original.merge_patch(j_patch);
// {
//  "a": "z",
//  "c": {
//    "d": "e"
//  }
// }

Implicit conversions

The type of the JSON object is determined automatically by the expression to store. Likewise, the stored value is implicitly converted.

cpp

// strings
std::string s1 = "Hello, world!";
json js = s1;
std::string s2 = js;

// Booleans
bool b1 = true;
json jb = b1;
bool b2 = jb;

// numbers
int i = 42;
json jn = i;
double f = jn;

// etc.

You can also explicitly ask for the value:

cpp

std::string vs = js.get<std::string>();
bool vb = jb.get<bool>();
int vi = jn.get<int>();

// etc.

Note that char types are not automatically converted to JSON strings, but to integer numbers. A conversion to a string must be specified explicitly:

cpp

char ch = 'A';                       // ASCII value 65
json j_default = ch;                 // stores integer number 65
json j_string = std::string(1, ch);  // stores string "A"

Arbitrary types conversions

Every type can be serialized in JSON, not just STL containers and scalar types. Usually, you would do something along those lines:

cpp

namespace ns {
    // a simple struct to model a person
    struct person {
        std::string name;
        std::string address;
        int age;
    };
}

ns::person p = {"Ned Flanders", "744 Evergreen Terrace", 60};

// convert to JSON: copy each value into the JSON object
json j;
j["name"] = p.name;
j["address"] = p.address;
j["age"] = p.age;

// ...

// convert from JSON: copy each value from the JSON object
ns::person p {
    j["name"].get<std::string>(),
    j["address"].get<std::string>(),
    j["age"].get<int>()
};

It works, but that's quite a lot of boilerplate... Fortunately, there's a better way:

cpp

// create a person
ns::person p {"Ned Flanders", "744 Evergreen Terrace", 60};

// conversion: person -> json
json j = p;

std::cout << j << std::endl;
// {"address":"744 Evergreen Terrace","age":60,"name":"Ned Flanders"}

// conversion: json -> person
ns::person p2 = j;

// that's it
assert(p == p2);

Basic usage

To make this work with one of your types, you only need to provide two functions:

cpp

using nlohmann::json;

namespace ns {
    void to_json(json& j, const person& p) {
        j = json{{"name", p.name}, {"address", p.address}, {"age", p.age}};
    }

    void from_json(const json& j, person& p) {
        p.name = j.at("name").get<std::string>();
        p.address = j.at("address").get<std::string>();
        p.age = j.at("age").get<int>();
    }
} // namespace ns

That's all! When calling the json constructor with your type, your custom to_json method will be automatically called.
Likewise, when calling get<your_type>(), the from_json method will be called.

Some important things:

Those methods MUST be in your type's namespace (which can be the global namespace), or the library will not be able to locate them (in this example, they are in namespace ns, where person is defined).
Those methods MUST be available (e.g., properly headers must be included) everywhere you use the implicit conversions. Look at issue 1108 for errors that may occur otherwise.
When using get<your_type>(), your_type MUST be DefaultConstructible. (There is a way to bypass this requirement described later.)
In function from_json, use function at() to access the object values rather than operator[]. In case a key does not exist, at throws an exception that you can handle, whereas operator[] exhibits undefined behavior.
In case your type contains several operator= definitions, code like your_variable = your_json; may not compile. You need to write your_variable = your_json.get<decltype your_variable>(); instead.
You do not need to add serializers or deserializers for STL types like std::vector: the library already implements these.
Be careful with the definition order of the from_json/to_json functions: If a type B has a member of type A, you MUST define to_json(A) before to_json(B). Look at issue 561 for more details.

How do I convert third-party types?

This requires a bit more advanced technique. But first, let's see how this conversion mechanism works:

The library uses JSON Serializers to convert types to json.
The default serializer for nlohmann::json is nlohmann::adl_serializer (ADL means Argument-Dependent Lookup).

It is implemented like this (simplified):

cpp

template <typename T>
struct adl_serializer {
    static void to_json(json& j, const T& value) {
        // calls the "to_json" method in T's namespace
    }

    static void from_json(const json& j, T& value) {
        // same thing, but with the "from_json" method
    }
};

This serializer works fine when you have control over the type's namespace. However, what about boost::optional or std::filesystem::path (C++17)? Hijacking the boost namespace is pretty bad, and it's illegal to add something other than template specializations to std...

To solve this, you need to add a specialization of adl_serializer to the nlohmann namespace, here's an example:

cpp

// partial specialization (full specialization works too)
namespace nlohmann {
    template <typename T>
    struct adl_serializer<boost::optional<T>> {
        static void to_json(json& j, const boost::optional<T>& opt) {
            if (opt == boost::none) {
                j = nullptr;
            } else {
              j = *opt; // this will call adl_serializer<T>::to_json which will
                        // find the free function to_json in T's namespace!
            }
        }

        static void from_json(const json& j, boost::optional<T>& opt) {
            if (j.is_null()) {
                opt = boost::none;
            } else {
                opt = j.get<T>(); // same as above, but with
                                  // adl_serializer<T>::from_json
            }
        }
    };
}

How can I use `get()` for non-default constructible/non-copyable types?

There is a way, if your type is MoveConstructible. You will need to specialize the adl_serializer as well, but with a special from_json overload:

cpp

struct move_only_type {
    move_only_type() = delete;
    move_only_type(int ii): i(ii) {}
    move_only_type(const move_only_type&) = delete;
    move_only_type(move_only_type&&) = default;

    int i;
};

namespace nlohmann {
    template <>
    struct adl_serializer<move_only_type> {
        // note: the return type is no longer 'void', and the method only takes
        // one argument
        static move_only_type from_json(const json& j) {
            return {j.get<int>()};
        }

        // Here's the catch! You must provide a to_json method! Otherwise you
        // will not be able to convert move_only_type to json, since you fully
        // specialized adl_serializer on that type
        static void to_json(json& j, move_only_type t) {
            j = t.i;
        }
    };
}

Can I write my own serializer? (Advanced use)

Yes. You might want to take a look at unit-udt.cpp in the test suite, to see a few examples.

If you write your own serializer, you'll need to do a few things:

use a different basic_json alias than nlohmann::json (the last template parameter of basic_json is the JSONSerializer)
use your basic_json alias (or a template parameter) in all your to_json/from_json methods
use nlohmann::to_json and nlohmann::from_json when you need ADL

Here is an example, without simplifications, that only accepts types with a size <= 32, and uses ADL.

cpp

// You should use void as a second template argument
// if you don't need compile-time checks on T
template<typename T, typename SFINAE = typename std::enable_if<sizeof(T) <= 32>::type>
struct less_than_32_serializer {
    template <typename BasicJsonType>
    static void to_json(BasicJsonType& j, T value) {
        // we want to use ADL, and call the correct to_json overload
        using nlohmann::to_json; // this method is called by adl_serializer,
                                 // this is where the magic happens
        to_json(j, value);
    }

    template <typename BasicJsonType>
    static void from_json(const BasicJsonType& j, T& value) {
        // same thing here
        using nlohmann::from_json;
        from_json(j, value);
    }
};

Be very careful when reimplementing your serializer, you can stack overflow if you don't pay attention:

cpp

template <typename T, void>
struct bad_serializer
{
    template <typename BasicJsonType>
    static void to_json(BasicJsonType& j, const T& value) {
      // this calls BasicJsonType::json_serializer<T>::to_json(j, value);
      // if BasicJsonType::json_serializer == bad_serializer ... oops!
      j = value;
    }

    template <typename BasicJsonType>
    static void to_json(const BasicJsonType& j, T& value) {
      // this calls BasicJsonType::json_serializer<T>::from_json(j, value);
      // if BasicJsonType::json_serializer == bad_serializer ... oops!
      value = j.template get<T>(); // oops!
    }
};

Binary formats (CBOR, MessagePack, and UBJSON)

Though JSON is a ubiquitous data format, it is not a very compact format suitable for data exchange, for instance over a network. Hence, the library supports CBOR (Concise Binary Object Representation), MessagePack, and UBJSON (Universal Binary JSON Specification) to efficiently encode JSON values to byte vectors and to decode such vectors.

cpp

// create a JSON value
json j = R"({"compact": true, "schema": 0})"_json;

// serialize to CBOR
std::vector<std::uint8_t> v_cbor = json::to_cbor(j);

// 0xA2, 0x67, 0x63, 0x6F, 0x6D, 0x70, 0x61, 0x63, 0x74, 0xF5, 0x66, 0x73, 0x63, 0x68, 0x65, 0x6D, 0x61, 0x00

// roundtrip
json j_from_cbor = json::from_cbor(v_cbor);

// serialize to MessagePack
std::vector<std::uint8_t> v_msgpack = json::to_msgpack(j);

// 0x82, 0xA7, 0x63, 0x6F, 0x6D, 0x70, 0x61, 0x63, 0x74, 0xC3, 0xA6, 0x73, 0x63, 0x68, 0x65, 0x6D, 0x61, 0x00

// roundtrip
json j_from_msgpack = json::from_msgpack(v_msgpack);

// serialize to UBJSON
std::vector<std::uint8_t> v_ubjson = json::to_ubjson(j);

// 0x7B, 0x69, 0x07, 0x63, 0x6F, 0x6D, 0x70, 0x61, 0x63, 0x74, 0x54, 0x69, 0x06, 0x73, 0x63, 0x68, 0x65, 0x6D, 0x61, 0x69, 0x00, 0x7D

// roundtrip
json j_from_ubjson = json::from_ubjson(v_ubjson);

Supported compilers

Though it's 2018 already, the support for C++11 is still a bit sparse. Currently, the following compilers are known to work:

GCC 4.9 - 8.2 (and possibly later)
Clang 3.4 - 6.1 (and possibly later)
Intel C++ Compiler 17.0.2 (and possibly later)
Microsoft Visual C++ 2015 / Build Tools 14.0.25123.0 (and possibly later)
Microsoft Visual C++ 2017 / Build Tools 15.5.180.51428 (and possibly later)

I would be happy to learn about other compilers/versions.

Please note:

GCC 4.8 does not work because of two bugs (55817 and 57824) in the C++11 support. Note there is a pull request to fix some of the issues.
Android defaults to using very old compilers and C++ libraries. To fix this, add the following to your Application.mk. This will switch to the LLVM C++ library, the Clang compiler, and enable C++11 and other features disabled by default.
text
```
APP_STL := c++_shared
NDK_TOOLCHAIN_VERSION := clang3.6
APP_CPPFLAGS += -frtti -fexceptions
```
The code compiles successfully with Android NDK, Revision 9 - 11 (and possibly later) and CrystaX's Android NDK version 10.
For GCC running on MinGW or Android SDK, the error 'to_string' is not a member of 'std' (or similarly, for strtod) may occur. Note this is not an issue with the code, but rather with the compiler itself. On Android, see above to build with a newer environment. For MinGW, please refer to this site and this discussion for information on how to fix this bug. For Android NDK using APP_STL := gnustl_static, please refer to this discussion.
Unsupported versions of GCC and Clang are rejected by #error directives. This can be switched off by defining JSON_SKIP_UNSUPPORTED_COMPILER_CHECK. Note that you can expect no support in this case.

The following compilers are currently used in continuous integration at Travis and AppVeyor:

Compiler	Operating System	Version String
GCC 4.9.4	Ubuntu 14.04.1 LTS	g++-4.9 (Ubuntu 4.9.4-2ubuntu1~14.04.1) 4.9.4
GCC 5.5.0	Ubuntu 14.04.1 LTS	g++-5 (Ubuntu 5.5.0-12ubuntu1~14.04) 5.5.0 20171010
GCC 6.4.0	Ubuntu 14.04.1 LTS	g++-6 (Ubuntu 6.4.0-17ubuntu1~14.04) 6.4.0 20180424
GCC 7.3.0	Ubuntu 14.04.1 LTS	g++-7 (Ubuntu 7.3.0-21ubuntu1~14.04) 7.3.0
GCC 7.3.0	Windows Server 2012 R2 (x64)	g++ (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 7.3.0
GCC 8.1.0	Ubuntu 14.04.1 LTS	g++-8 (Ubuntu 8.1.0-5ubuntu1~14.04) 8.1.0
Clang 3.5.0	Ubuntu 14.04.1 LTS	clang version 3.5.0-4ubuntu2~trusty2 (tags/RELEASE_350/final) (based on LLVM 3.5.0)
Clang 3.6.2	Ubuntu 14.04.1 LTS	clang version 3.6.2-svn240577-1~exp1 (branches/release_36) (based on LLVM 3.6.2)
Clang 3.7.1	Ubuntu 14.04.1 LTS	clang version 3.7.1-svn253571-1~exp1 (branches/release_37) (based on LLVM 3.7.1)
Clang 3.8.0	Ubuntu 14.04.1 LTS	clang version 3.8.0-2ubuntu3~trusty5 (tags/RELEASE_380/final)
Clang 3.9.1	Ubuntu 14.04.1 LTS	clang version 3.9.1-4ubuntu3~14.04.3 (tags/RELEASE_391/rc2)
Clang 4.0.1	Ubuntu 14.04.1 LTS	clang version 4.0.1-svn305264-1~exp1 (branches/release_40)
Clang 5.0.2	Ubuntu 14.04.1 LTS	clang version 5.0.2-svn328729-1~~exp1~~20180509123505.100 (branches/release_50)
Clang 6.0.1	Ubuntu 14.04.1 LTS	clang version 6.0.1-svn334776-1~~exp1~~20180726133705.85 (branches/release_60)
Clang Xcode 6.4	OSX 10.10.5	Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn)
Clang Xcode 7.3	OSX 10.11.6	Apple LLVM version 7.3.0 (clang-703.0.31)
Clang Xcode 8.0	OSX 10.11.6	Apple LLVM version 8.0.0 (clang-800.0.38)
Clang Xcode 8.1	OSX 10.12.6	Apple LLVM version 8.0.0 (clang-800.0.42.1)
Clang Xcode 8.2	OSX 10.12.6	Apple LLVM version 8.0.0 (clang-800.0.42.1)
Clang Xcode 8.3	OSX 10.11.6	Apple LLVM version 8.1.0 (clang-802.0.38)
Clang Xcode 9.0	OSX 10.12.6	Apple LLVM version 9.0.0 (clang-900.0.37)
Clang Xcode 9.1	OSX 10.12.6	Apple LLVM version 9.0.0 (clang-900.0.38)
Clang Xcode 9.2	OSX 10.13.3	Apple LLVM version 9.1.0 (clang-902.0.39.1)
Clang Xcode 9.3	OSX 10.13.3	Apple LLVM version 9.1.0 (clang-902.0.39.2)
Visual Studio 14 2015	Windows Server 2012 R2 (x64)	Microsoft (R) Build Engine version 14.0.25420.1, MSVC 19.0.24215.1
Visual Studio 2017	Windows Server 2016	Microsoft (R) Build Engine version 15.7.180.61344, MSVC 19.14.26433.0

License

The class is licensed under the MIT License:

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Contact

If you have questions regarding the library, I would like to invite you to open an issue at GitHub. Please describe your request, problem, or question as detailed as possible, and also mention the version of the library you are using as well as the version of your compiler and operating system. Opening an issue at GitHub allows other users and contributors to this library to collaborate. For instance, I have little experience with MSVC, and most issues in this regard have been solved by a growing community. If you have a look at the closed issues, you will see that we react quite timely in most cases.

Only if your request would contain confidential information, please send me an email. For encrypted messages, please use this key.

Security

Commits by Niels Lohmann and releases are signed with this PGP Key.

Thanks

I deeply appreciate the help of the following people.

Teemperor implemented CMake support and lcov integration, realized escape and Unicode handling in the string parser, and fixed the JSON serialization.
elliotgoodrich fixed an issue with double deletion in the iterator classes.
kirkshoop made the iterators of the class composable to other libraries.
wancw fixed a bug that hindered the class to compile with Clang.
Tomas Åblad found a bug in the iterator implementation.
Joshua C. Randall fixed a bug in the floating-point serialization.
Aaron Burghardt implemented code to parse streams incrementally. Furthermore, he greatly improved the parser class by allowing the definition of a filter function to discard undesired elements while parsing.
Daniel Kopeček fixed a bug in the compilation with GCC 5.0.
Florian Weber fixed a bug in and improved the performance of the comparison operators.
Eric Cornelius pointed out a bug in the handling with NaN and infinity values. He also improved the performance of the string escaping.
易思龙 implemented a conversion from anonymous enums.
kepkin patiently pushed forward the support for Microsoft Visual studio.
gregmarr simplified the implementation of reverse iterators and helped with numerous hints and improvements. In particular, he pushed forward the implementation of user-defined types.
Caio Luppi fixed a bug in the Unicode handling.
dariomt fixed some typos in the examples.
Daniel Frey cleaned up some pointers and implemented exception-safe memory allocation.
Colin Hirsch took care of a small namespace issue.
Huu Nguyen correct a variable name in the documentation.
Silverweed overloaded parse() to accept an rvalue reference.
dariomt fixed a subtlety in MSVC type support and implemented the get_ref() function to get a reference to stored values.
ZahlGraf added a workaround that allows compilation using Android NDK.
whackashoe replaced a function that was marked as unsafe by Visual Studio.
406345 fixed two small warnings.
Glen Fernandes noted a potential portability problem in the has_mapped_type function.
Corbin Hughes fixed some typos in the contribution guidelines.
twelsby fixed the array subscript operator, an issue that failed the MSVC build, and floating-point parsing/dumping. He further added support for unsigned integer numbers and implemented better roundtrip support for parsed numbers.
Volker Diels-Grabsch fixed a link in the README file.
msm- added support for American Fuzzy Lop.
Annihil fixed an example in the README file.
Themercee noted a wrong URL in the README file.
Lv Zheng fixed a namespace issue with int64_t and uint64_t.
abc100m analyzed the issues with GCC 4.8 and proposed a partial solution.
zewt added useful notes to the README file about Android.
Róbert Márki added a fix to use move iterators and improved the integration via CMake.
Chris Kitching cleaned up the CMake files.
Tom Needham fixed a subtle bug with MSVC 2015 which was also proposed by Michael K..
Mário Feroldi fixed a small typo.
duncanwerner found a really embarrassing performance regression in the 2.0.0 release.
Damien fixed one of the last conversion warnings.
Thomas Braun fixed a warning in a test case.
Théo DELRIEU patiently and constructively oversaw the long way toward iterator-range parsing. He also implemented the magic behind the serialization/deserialization of user-defined types and split the single header file into smaller chunks.
Stefan fixed a minor issue in the documentation.
Vasil Dimov fixed the documentation regarding conversions from std::multiset.
ChristophJud overworked the CMake files to ease project inclusion.
Vladimir Petrigo made a SFINAE hack more readable and added Visual Studio 17 to the build matrix.
Denis Andrejew fixed a grammar issue in the README file.
Pierre-Antoine Lacaze found a subtle bug in the dump() function.
TurpentineDistillery pointed to std::locale::classic() to avoid too much locale joggling, found some nice performance improvements in the parser, improved the benchmarking code, and realized locale-independent number parsing and printing.
cgzones had an idea how to fix the Coverity scan.
Jared Grubb silenced a nasty documentation warning.
Yixin Zhang fixed an integer overflow check.
Bosswestfalen merged two iterator classes into a smaller one.
Daniel599 helped to get Travis execute the tests with Clang's sanitizers.
Jonathan Lee fixed an example in the README file.
gnzlbg supported the implementation of user-defined types.
Alexej Harm helped to get the user-defined types working with Visual Studio.
Jared Grubb supported the implementation of user-defined types.
EnricoBilla noted a typo in an example.
Martin Hořeňovský found a way for a 2x speedup for the compilation time of the test suite.
ukhegg found proposed an improvement for the examples section.
rswanson-ihi noted a typo in the README.
Mihai Stan fixed a bug in the comparison with nullptrs.
Tushar Maheshwari added cotire support to speed up the compilation.
TedLyngmo noted a typo in the README, removed unnecessary bit arithmetic, and fixed some -Weffc++ warnings.
Krzysztof Woś made exceptions more visible.
ftillier fixed a compiler warning.
tinloaf made sure all pushed warnings are properly popped.
Fytch found a bug in the documentation.
Jay Sistar implemented a Meson build description.
Henry Lee fixed a warning in ICC and improved the iterator implementation.
Vincent Thiery maintains a package for the Conan package manager.
Steffen fixed a potential issue with MSVC and std::min.
Mike Tzou fixed some typos.
amrcode noted a misleading documentation about comparison of floats.
Oleg Endo reduced the memory consumption by replacing with.
dan-42 cleaned up the CMake files to simplify including/reusing of the library.
Nikita Ofitserov allowed for moving values from initializer lists.
Greg Hurrell fixed a typo.
Dmitry Kukovinets fixed a typo.
kbthomp1 fixed an issue related to the Intel OSX compiler.
Markus Werle fixed a typo.
WebProdPP fixed a subtle error in a precondition check.
Alex noted an error in a code sample.
Tom de Geus reported some warnings with ICC and helped fixing them.
Perry Kundert simplified reading from input streams.
Sonu Lohani fixed a small compilation error.
Jamie Seward fixed all MSVC warnings.
Nate Vargas added a Doxygen tag file.
pvleuven helped fixing a warning in ICC.
Pavel helped fixing some warnings in MSVC.
Jamie Seward avoided unnecessary string copies in find() and count().
Mitja fixed some typos.
Jorrit Wronski updated the Hunter package links.
Matthias Möller added a .natvis for the MSVC debug view.
bogemic fixed some C++17 deprecation warnings.
Eren Okka fixed some MSVC warnings.
abolz integrated the Grisu2 algorithm for proper floating-point formatting, allowing more roundtrip checks to succeed.
Vadim Evard fixed a Markdown issue in the README.
zerodefect fixed a compiler warning.
Kert allowed to template the string type in the serialization and added the possibility to override the exceptional behavior.
mark-99 helped fixing an ICC error.
Patrik Huber fixed links in the README file.
johnfb found a bug in the implementation of CBOR's indefinite length strings.
Paul Fultz II added a note on the cget package manager.
Wilson Lin made the integration section of the README more concise.
RalfBielig detected and fixed a memory leak in the parser callback.
agrianius allowed to dump JSON to an alternative string type.
Kevin Tonon overworked the C++11 compiler checks in CMake.
Axel Huebl simplified a CMake check and added support for the Spack package manager.
Carlos O'Ryan fixed a typo.
James Upjohn fixed a version number in the compilers section.
Chuck Atkins adjusted the CMake files to the CMake packaging guidelines
Jan Schöppach fixed a typo.
martin-mfg fixed a typo.
Matthias Möller removed the dependency from std::stringstream.
agrianius added code to use alternative string implementations.
Daniel599 allowed to use more algorithms with the items() function.
Julius Rakow fixed the Meson include directory and fixed the links to cppreference.com.
Sonu Lohani fixed the compilation with MSVC 2015 in debug mode.
grembo fixed the test suite and re-enabled several test cases.
Hyeon Kim introduced the macro JSON_INTERNAL_CATCH to control the exception handling inside the library.
thyu fixed a compiler warning.

Thanks a lot for helping out! Please let me know if I forgot someone.

Used third-party tools

The library itself consists of a single header file licensed under the MIT license. However, it is built, tested, documented, and whatnot using a lot of third-party tools and services. Thanks a lot!

amalgamate.py - Amalgamate C source and header files to create a single header file
American fuzzy lop for fuzz testing
AppVeyor for continuous integration on Windows
Artistic Style for automatic source code identation
Catch for the unit tests
Clang for compilation with code sanitizers
Cmake for build automation
Codacity for further code analysis
Coveralls to measure code coverage
Coverity Scan for static analysis
cppcheck for static analysis
Doxygen to generate documentation
git-update-ghpages to upload the documentation to gh-pages
GitHub Changelog Generator to generate the ChangeLog
Google Benchmark to implement the benchmarks
libFuzzer to implement fuzz testing for OSS-Fuzz
OSS-Fuzz for continuous fuzz testing of the library (project repository)
Probot for automating maintainer tasks such as closing stale issues, requesting missing information, or detecting toxic comments.
send_to_wandbox to send code examples to Wandbox
Travis for continuous integration on Linux and macOS
Valgrind to check for correct memory management
Wandbox for online examples

Projects using JSON for Modern C++

The library is currently used in Apple macOS Sierra and iOS 10. I am not sure what they are using the library for, but I am happy that it runs on so many devices.

Notes

The code contains numerous debug assertions which can be switched off by defining the preprocessor macro NDEBUG, see the documentation of assert. In particular, note operator[] implements unchecked access for const objects: If the given key is not present, the behavior is undefined (think of a dereferenced null pointer) and yields an assertion failure if assertions are switched on. If you are not sure whether an element in an object exists, use checked access with the at() function.
As the exact type of a number is not defined in the JSON specification, this library tries to choose the best fitting C++ number type automatically. As a result, the type double may be used to store numbers which may yield floating-point exceptions in certain rare situations if floating-point exceptions have been unmasked in the calling code. These exceptions are not caused by the library and need to be fixed in the calling code, such as by re-masking the exceptions prior to calling library functions.
The library supports Unicode input as follows:
- Only UTF-8 encoded input is supported which is the default encoding for JSON according to RFC 7159.
- Other encodings such as Latin-1, UTF-16, or UTF-32 are not supported and will yield parse or serialization errors.
- Unicode noncharacters will not be replaced by the library.
- Invalid surrogates (e.g., incomplete pairs such as \uDEAD) will yield parse errors.
- The strings stored in the library are UTF-8 encoded. When using the default string type (std::string), note that its length/size functions return the number of stored bytes rather than the number of characters or glyphs.
The code can be compiled without C++ runtime type identification features; that is, you can use the -fno-rtti compiler flag.
Exceptions are used widely within the library. They can, however, be switched off with either using the compiler flag -fno-exceptions or by defining the symbol JSON_NOEXCEPTION. In this case, exceptions are replaced by an abort() call.
By default, the library does not preserve the insertion order of object elements. This is standards-compliant, as the JSON standard defines objects as "an unordered collection of zero or more name/value pairs". If you do want to preserve the insertion order, you can specialize the object type with containers like tsl::ordered_map (integration) or nlohmann::fifo_map (integration).

Execute unit tests

To compile and run the tests, you need to execute

$ mkdir build
$ cd build
$ cmake ..
$ cmake --build .
$ ctest --output-on-failure

For more information, have a look at the file .travis.yml.

Overview (`ext/prometheus-cpp-lite-1.0/README.md`)

C++ Header-only Prometheus client library

It is a tool for quickly adding metrics (and profiling) functionality to C++ projects.

Advantages:

Written in pure C++,
Header-only,
Cross-platform,
Compiles with C ++ 11, C ++ 14, C ++ 17 standards,
Has no third-party dependencies,
Several APIs for use in your projects,
Saving metrics to a file (and then works with node_exporter) or sending via http (uses built-in header-only http-client-lite library),
Possiblity to use different types for storing metrics data (default is uint32_t, but you can use double or uint64_t types if you want),
Five types of metrics are supported: counter, gauge, summary, histogram and benchmark,
Has detailed examples of use (see examples folder)

How it differs from the [jupp0r/prometheus-cpp](https://github.com/jupp0r/prometheus-cpp) project:

I need a simple header only wariant library without dependencies to write metrics to a .prom file,
I need the fastest possible work using integer values of counters (original project use only floating pointer values),
The origianl project have problems on compilers that do not know how to do LTO optimization,
I did not like the python style of the original project and the large amount of extra code in it and I wanted to make it lighter and more c++ classic.

How to use it:

The library has two API:

Complex API for those who want to control everything,
Simple API for those who want to quickly add metrics to their C ++ (and it is actually just a wrapper around the complex API).

Let's start with a simple API because it's simple:

To add it to your C++ project add these lines to your CMakeLists.txt file:

text

add_subdirectory("prometheus-cpp-lite/core")
add_subdirectory("prometheus-cpp-lite/3rdparty/http-client-lite")
add_subdirectory("prometheus-cpp-lite/simpleapi")
target_link_libraries(your_target prometheus-cpp-simpleapi)

The simplest way to create a metric would be like this:

c++

prometheus::simpleapi::METRIC_metric_t metric1 { "metric1", "first simple metric without any tag" };
prometheus::simpleapi::METRIC_metric_t metric2 { "metric2", "second simple metric without any tag" };

where METRIC can be counter, gauge, summary, histogram or benchmark.

If you want to access an existing metric again elsewhere in the code, you can do this:

c++

prometheus::simpleapi::METRIC_metric_t metric2_yet_another_link { "metric2", "" };

this works because when adding a metric, it checks whether there is already a metric with the same name and, if there is one, a link to it is returned.

You can create a family of metrics (metrics with tags) as follows:

c++

prometheus::simpleapi::METRIC_family_t family  { "metric_family", "metric family" };
prometheus::simpleapi::METRIC_metric_t metric1 { family.Add({{"name", "metric1"}}) };
prometheus::simpleapi::METRIC_metric_t metric2 { family.Add({{"name", "metric2"}}) };

where METRIC can be counter, gauge, summary, histogram or benchmark.

Next, you can do the following things with metrics:

c++

metric++; // for increment it (only for counter and gauge metrics)
metric += value; // for add value to metric (only for gauge metric)
metric -= value; // for sub value from metric (only for gauge metric)
metric = value;  // save current value (only gauge metrics)
metric.start();  // start calculate time (only for benchmark metric)
metric.stop();   // stop calculate time (only for benchmark metric)

You can change the settings of save (or send) metrics data as follows:

c++

prometheus::simpleapi::saver.set_delay(period_in_seconds); // change the period of saving (or sending) metrics data in seconds (5 seconds by default)
prometheus::simpleapi::saver.set_out_file(filename);       // change the name of the output file (metrics.prom by default)
prometheus::simpleapi::saver.set_server_url(url);          // change the name of prometheus server (unset by default)

Simple API complex example 1 (examples/simpleapi_example.cpp):

c++

#include <prometheus/simpleapi.h>

void main() {

  using namespace prometheus::simpleapi;

  counter_family_t family  { "simple_family", "simple family example" };
  counter_metric_t metric1 { family.Add({{"name", "counter1"}}) };
  counter_metric_t metric2 { family.Add({{"name", "counter2"}}) };

  counter_metric_t metric3 { "simple_counter_1", "simple counter 1 without labels example" };
  counter_metric_t metric4 { "simple_counter_2", "simple counter 2 without labels example" };

  for (;; ) {
    std::this_thread::sleep_for(std::chrono::seconds(1));
    const int random_value = std::rand();
    if (random_value & 1) metric1++;
    if (random_value & 2) metric2++;
    if (random_value & 4) metric3++;
    if (random_value & 8) metric4++;
  }

}

Output in "metrics.prom" file (by default) will be:

text

# HELP simple_family simple family example
# TYPE simple_family counter
simple_family{name="counter1"} 10
simple_family{name="counter2"} 9
# HELP simple_counter_1 simple counter 1 without labels example
# TYPE simple_counter_1 counter
simple_counter_1 6
# HELP simple_counter_2 simple counter 2 without labels example
# TYPE simple_counter_2 counter
simple_counter_2 8

Simple API complex example 2 (examples/simpleapi_use_in_class_example.cpp):

c++

#include <prometheus/simpleapi.h>

using namespace prometheus::simpleapi;

class MyClass {

  counter_family_t metric_family { "simple_family", "simple family example" };
  counter_metric_t metric1 { metric_family.Add({{"name", "counter1"}}) };
  counter_metric_t metric2 { metric_family.Add({{"name", "counter2"}}) };

  counter_metric_t metric3 { "simple_counter_1", "simple counter 1 without labels example" };
  counter_metric_t metric4 { "simple_counter_2", "simple counter 2 without labels example" };

  benchmark_family_t benchmark_family { "simple_benchmark_family", "simple benchmark family example" };
  benchmark_metric_t benchmark1 { benchmark_family.Add({{"benchmark", "1"}}) };
  benchmark_metric_t benchmark2 { benchmark_family.Add({{"benchmark", "2"}}) };

public:

  MyClass() = default;

  void member_to_do_something() {

    benchmark1.start();
    const int random_value = std::rand();
    benchmark1.stop();

    benchmark2.start();
    if (random_value & 1)  metric1++;
    if (random_value & 2)  metric2++;
    if (random_value & 4)  metric3++;
    if (random_value & 8)  metric4++;
    benchmark2.stop();

  }

};

void main() {

  MyClass myClass;
  benchmark_metric_t benchmark { "simple_benchmark", "simple benchmark example" };

  for (;; ) {

    benchmark.start();
    std::this_thread::sleep_for(std::chrono::seconds(1));
    benchmark.stop();

    myClass.member_to_do_something();

  }

}

Output in "metrics.prom" file (by default) will be:

text

# HELP simple_family simple family example
# TYPE simple_family counter
simple_family{name="counter1"} 3
simple_family{name="counter2"} 2
# HELP simple_counter_1 simple counter 1 without labels example
# TYPE simple_counter_1 counter
simple_counter_1 3
# HELP simple_counter_2 simple counter 2 without labels example
# TYPE simple_counter_2 counter
simple_counter_2 3
# HELP simple_benchmark_family simple benchmark family example
# TYPE simple_benchmark_family counter
simple_benchmark_family{benchmark="1"} 0.0001088
simple_benchmark_family{benchmark="2"} 1.48e-05
# HELP simple_benchmark simple benchmark example
# TYPE simple_benchmark counter
simple_benchmark 6.0503248

Overview (`ext/redis-plus-plus-1.1.1/README.md`)

redis-plus-plus

Overview
- Features
Installation
Getting Started
API Reference
Author

Overview

This is a C++ client for Redis. It's based on hiredis, and written in C++ 11.

NOTE: I'm not a native speaker. So if the documentation is unclear, please feel free to open an issue or pull request. I'll response ASAP.

Features

Most commands for Redis.
Connection pool.
Redis scripting.
Thread safe unless otherwise stated.
Redis publish/subscribe.
Redis pipeline.
Redis transaction.
Redis Cluster.
Redis Sentinel.
STL-like interfaces.
Generic command interface.

Installation

Install hiredis

Since redis-plus-plus is based on hiredis, you should install hiredis first. The minimum version requirement for hiredis is v0.12.1, and you'd better use the latest release of hiredis.

text

git clone https://github.com/redis/hiredis.git

cd hiredis

make

make install

By default, hiredis is installed at /usr/local. If you want to install hiredis at non-default location, use the following commands to specify the installation path.

text

make PREFIX=/non/default/path

make PREFIX=/non/default/path install

Install redis-plus-plus

redis-plus-plus is built with CMAKE.

text

git clone https://github.com/sewenew/redis-plus-plus.git

cd redis-plus-plus

mkdir compile

cd compile

cmake -DCMAKE_BUILD_TYPE=Release ..

make

make install

cd ..

If hiredis is installed at non-default location, you should use CMAKE_PREFIX_PATH to specify the installation path of hiredis. By default, redis-plus-plus is installed at /usr/local. However, you can use CMAKE_INSTALL_PREFIX to install redis-plus-plus at non-default location.

text

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH=/path/to/hiredis -DCMAKE_INSTALL_PREFIX=/path/to/install/redis-plus-plus ..

Run Tests (Optional)

redis-plus-plus has been fully tested with the following compilers:

text

gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)
gcc version 5.5.0 20171010 (Ubuntu 5.5.0-12ubuntu1)
gcc version 6.5.0 20181026 (Ubuntu 6.5.0-2ubuntu1~18.04)
gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
gcc version 8.3.0 (Ubuntu 8.3.0-6ubuntu1~18.04.1)
clang version 3.9.1-19ubuntu1 (tags/RELEASE_391/rc2)
clang version 4.0.1-10 (tags/RELEASE_401/final)
clang version 5.0.1-4 (tags/RELEASE_501/final)
clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
clang version 7.0.0-3~ubuntu0.18.04.1 (tags/RELEASE_700/final)
Apple clang version 11.0.0 (clang-1100.0.33.8)

After compiling with cmake, you'll get a test program in compile/test directory: compile/test/test_redis++.

In order to run the tests, you need to set up a Redis instance, and a Redis Cluster. Since the test program will send most of Redis commands to the server and cluster, you need to set up Redis of the latest version (by now, it's 5.0). Otherwise, the tests might fail. For example, if you set up Redis 4.0 for testing, the test program will fail when it tries to send the ZPOPMAX command (a Redis 5.0 command) to the server. If you want to run the tests with other Redis versions, you have to comment out commands that haven't been supported by your Redis, from test source files in redis-plus-plus/test/src/sw/redis++/ directory. Sorry for the inconvenience, and I'll fix this problem to make the test program work with any version of Redis in the future.

NOTE: The latest version of Redis is only a requirement for running the tests. In fact, you can use redis-plus-plus with Redis of any version, e.g. Redis 2.0, Redis 3.0, Redis 4.0, Redis 5.0.

NEVER run the test program in production envronment, since the keys, which the test program reads or writes, might conflict with your application.

In order to run tests with both Redis and Redis Cluster, you can run the test program with the following command:

text

./compile/test/test_redis++ -h host -p port -a auth -n cluster_node -c cluster_port

host and port are the host and port number of the Redis instance.
cluster_node and cluster_port are the host and port number of Redis Cluster. You only need to set the host and port number of a single node in the cluster, redis-plus-plus will find other nodes automatically.
auth is the password of the Redis instance and Redis Cluster. The Redis instance and Redis Cluster must be configured with the same password. If there's no password configured, don't set this option.

If you only want to run tests with Redis, you only need to specify host, port and auth options:

text

./compile/test/test_redis++ -h host -p port -a auth

Similarly, if you only want to run tests with Redis Cluster, just specify cluster_node, cluster_port and auth options:

text

./compile/test/test_redis++ -a auth -n cluster_node -c cluster_port

The test program will test running redis-plus-plus in multi-threads environment, and this test will cost a long time. If you want to skip it (not recommended), just comment out the following lines in test/src/sw/redis++/test_main.cpp file.

c++

sw::redis::test::ThreadsTest threads_test(opts, cluster_node_opts);
threads_test.run();

If all tests have been passed, the test program will print the following message:

text

Pass all tests

Otherwise, it prints the error message.

Performance

redis-plus-plus runs as fast as hiredis, since it's a wrapper of hiredis. You can run test_redis++ in benchmark mode to check the performance in your environment.

text

./compile/test/test_redis++ -h host -p port -a auth -n cluster_node -c cluster_port -b -t thread_num -s connection_pool_size -r request_num -k key_len -v val_len

-b option turns the test program into benchmark mode.
thread_num specifies the number of worker threads. 10 by default.
connection_pool_size specifies the size of the connection pool. 5 by default.
request_num specifies the total number of requests sent to server for each test. 100000 by default.
key_len specifies the length of the key for each operation. 10 by default.
val_len specifies the length of the value. 10 by default.

The bechmark will generate 100 random binary keys for testing, and the size of these keys is specified by key_len. When the benchmark runs, it will read/write with these keys. So NEVER run the test program in your production environment, otherwise, it might inaccidently delete your data.

Use redis-plus-plus In Your Project

After compiling the code, you'll get both shared library and static library. Since redis-plus-plus depends on hiredis, you need to link both libraries to your Application. Also don't forget to specify the -std=c++11 and thread-related option.

Use Static Libraries

Take gcc as an example.

text

g++ -std=c++11 -o app app.cpp /path/to/libredis++.a /path/to/libhiredis.a -pthread

If hiredis and redis-plus-plus are installed at non-default location, you should use -I option to specify the header path.

text

g++ -std=c++11 -I/non-default/install/include/path -o app app.cpp /path/to/libredis++.a /path/to/libhiredis.a -pthread

Use Shared Libraries

text

g++ -std=c++11 -o app app.cpp -lredis++ -lhiredis -pthread

If hiredis and redis-plus-plus are installed at non-default location, you should use -I and -L options to specify the header and library paths.

text

g++ -std=c++11 -I/non-default/install/include/path -L/non-default/install/lib/path -o app app.cpp -lredis++ -lhiredis -pthread

When linking with shared libraries, and running your application, you might get the following error message:

text

error while loading shared libraries: xxx: cannot open shared object file: No such file or directory.

That's because the linker cannot find the shared libraries. In order to solve the problem, you can add the path where you installed hiredis and redis-plus-plus libraries, to LD_LIBRARY_PATH environment variable. For example:

text

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

Check this StackOverflow question for details on how to solve the problem.

Build With Cmake

If you're using cmake to build your application, you need to add hiredis and redis-plus-plus dependencies in your CMakeLists.txt:

cmake

# <------------ add hiredis dependency --------------->
find_path(HIREDIS_HEADER hiredis)
target_include_directories(target PUBLIC ${HIREDIS_HEADER})

find_library(HIREDIS_LIB hiredis)
target_link_libraries(target ${HIREDIS_LIB})

# <------------ add redis-plus-plus dependency -------------->
# NOTE: this should be *sw* NOT *redis++*
find_path(REDIS_PLUS_PLUS_HEADER sw)
target_include_directories(target PUBLIC ${REDIS_PLUS_PLUS_HEADER})

find_library(REDIS_PLUS_PLUS_LIB redis++)
target_link_libraries(target ${REDIS_PLUS_PLUS_LIB})

See this issue for a complete example of CMakeLists.txt.

Also, if you installed hiredis and redis-plus-plus at non-default location, you need to run cmake with CMAKE_PREFIX_PATH option to specify the installation path of these two libraries.

text

cmake -DCMAKE_PREFIX_PATH=/installation/path/to/the/two/libs ..

Getting Started

text

/* Detailed source-code truncated for AI context efficiency. */

API Reference

Connection

Redis class maintains a connection pool to Redis server. If the connection is broken, Redis reconnects to Redis server automatically.

You can initialize a Redis instance with ConnectionOptions and ConnectionPoolOptions. ConnectionOptions specifies options for connection to Redis server, and ConnectionPoolOptions specifies options for conneciton pool. ConnectionPoolOptions is optional. If not specified, Redis maintains a single connection to Redis server.

c++

ConnectionOptions connection_options;
connection_options.host = "127.0.0.1";  // Required.
connection_options.port = 6666; // Optional. The default port is 6379.
connection_options.password = "auth";   // Optional. No password by default.
connection_options.db = 1;  // Optional. Use the 0th database by default.

// Optional. Timeout before we successfully send request to or receive response from redis.
// By default, the timeout is 0ms, i.e. never timeout and block until we send or receive successfuly.
// NOTE: if any command is timed out, we throw a TimeoutError exception.
connection_options.socket_timeout = std::chrono::milliseconds(200);

// Connect to Redis server with a single connection.
Redis redis1(connection_options);

ConnectionPoolOptions pool_options;
pool_options.size = 3;  // Pool size, i.e. max number of connections.

// Connect to Redis server with a connection pool.
Redis redis2(connection_options, pool_options);

See ConnectionOptions and ConnectionPoolOptions for more options.

NOTE: Redis class is movable but NOT copyable.

c++

// auto redis3 = redis1;    // this won't compile.

// But it's movable.
auto redis3 = std::move(redis1);

redis-plus-plus also supports connecting to Redis server with Unix Domain Socket.

c++

ConnectionOptions options;
options.type = ConnectionType::UNIX;
options.path = "/path/to/socket";
Redis redis(options);

You can also connect to Redis server with a URI. However, in this case, you can only specify host and port, or Unix Domain Socket path. In order to specify other options, you need to use ConnectionOptions and ConnectionPoolOptions.

c++

// Single connection to the given host and port.
Redis redis1("tcp://127.0.0.1:6666");

// Use default port, i.e. 6379.
Redis redis2("tcp://127.0.0.1");

// Connect to Unix Domain Socket.
Redis redis3("unix://path/to/socket");

Lazily Create Connection

Connections in the pool are lazily created. When the connection pool is initialized, i.e. the constructor of Redis, Redis does NOT connect to the server. Instead, it connects to the server only when you try to send command. In this way, we can avoid unnecessary connections. So if the pool size is 5, but the number of max concurrent connections is 3, there will be only 3 connections in the pool.

Connection Failure

You don't need to check whether Redis object connects to server successfully. If Redis fails to create a connection to Redis server, or the connection is broken at some time, it throws an exception of type Error when you try to send command with Redis. Even when you get an exception, i.e. the connection is broken, you don't need to create a new Redis object. You can reuse the Redis object to send commands, and the Redis object will try to reconnect to server automatically. If it reconnects successfully, it sends command to server. Otherwise, it throws an exception again.

See the Exception section for details on exceptions.

Reuse Redis object As Much As Possible

It's NOT cheap to create a Redis object, since it will create new connections to Redis server. So you'd better reuse Redis object as much as possible. Also, it's safe to call Redis' member functions in multi-thread environment, and you can share Redis object in multiple threads.

c++

// This is GOOD practice.
auto redis = Redis("tcp://127.0.0.1");
for (auto idx = 0; idx < 100; ++idx) {
    // Reuse the Redis object in the loop.
    redis.set("key", "val");
}

// This is VERY BAD! It's very inefficient.
// NEVER DO IT!!!
for (auto idx = 0; idx < 100; ++idx) {
    // Create a new Redis object for each iteration.
    auto redis = Redis("tcp://127.0.0.1");
    redis.set("key", "val");
}

Send Command to Redis Server

You can send Redis commands through Redis object. Redis has one or more (overloaded) methods for each Redis command. The method has the same (lowercased) name as the corresponding command. For example, we have 3 overload methods for the DEL key [key ...] command:

c++

// Delete a single key.
long long Redis::del(const StringView &key);

// Delete a batch of keys: [first, last).
template <typename Input>
long long Redis::del(Input first, Input last);

// Delete keys in the initializer_list.
template <typename T>
long long Redis::del(std::initializer_list<T> il);

With input parameters, these methods build a Redis command based on Redis protocol, and send the command to Redis server. Then synchronously receive the reply, parse it, and return to the caller.

Let's take a closer look at these methods' parameters and return values.

Parameter Type

Most of these methods have the same parameters as the corresponding commands. The following is a list of parameter types:

Parameter Type	Explaination	Example	Note
StringView	Parameters of string type. Normally used for key, value, member name, field name and so on	bool Redis::hset(const StringView &key, const StringView &field, const StringView &val)	See the StringView section for details on `StringView`
long long	Parameters of integer type. Normally used for index (e.g. list commands) or integer	void ltrim(const StringView &key, long long start, long long stop) long long decrby(const StringView &key, long long decrement)
double	Parameters of floating-point type. Normally used for score (e.g. sorted set commands) or number of floating-point type	double incrbyfloat(const StringView &key, double increment)
std::chrono::duration std::chrono::time_point	Time-related parameters	bool expire(const StringView &key, const std::chrono::seconds &timeout) bool expireat(const StringView &key, const std::chrono::time_point<std::chrono::system_clock, std::chrono::seconds> &tp)
std::pair<StringView, StringView>	Used for Redis hash's (field, value) pair	bool hset(const StringView &key, const std::pair<StringView, StringView> &item)
std::pair<double, double>	Used for Redis geo's (longitude, latitude) pair	OptionalLongLong georadius(const StringView &key, const std::pair<double, double> &location, double radius, GeoUnit unit, const StringView &destination, bool store_dist, long long count)
pair of iterators	Use a pair of iterators to specify a range of input, so that we can pass the data in a STL container to these methods	template < typename Input > long long del(Input first, Input last)	Throw an exception, if it's an empty range, i.e. first == last
std::initializer_list< T >	Use an initializer list to specify a batch of input	template < typename T > long long del(std::initializer_list< T > il)
some options	Options for some commands	UpdateType, template < typename T > class BoundedInterval	See command_options.h for details

StringView

std::string_view is a good option for the type of string parameters. However, by now, not all compilers support std::string_view. So we wrote a simple version, i.e. StringView. Since there are conversions from std::string and c-style string to StringView, you can just pass std::string or c-style string to methods that need a StringView parameter.

c++

// bool Redis::hset(const StringView &key, const StringView &field, const StringView &val)

// Pass c-style string to StringView.
redis.hset("key", "field", "value");

// Pass std::string to StringView.
std::string key = "key";
std::string field = "field";
std::string val = "val";
redis.hset(key, field, val);

// Mix std::string and c-style string.
redis.hset(key, field, "value");

Return Type

Redis protocol defines 5 kinds of replies:

Status Reply: Also known as Simple String Reply. It's a non-binary string reply.
Bulk String Reply: Binary safe string reply.
Integer Reply: Signed integer reply. Large enough to hold long long.
Array Reply: (Nested) Array reply.
Error Reply: Non-binary string reply that gives error info.

Also these replies might be NULL. For instance, when you try to GET the value of a nonexistent key, Redis returns a NULL Bulk String Reply.

As we mentioned above, replies are parsed into return values of these methods. The following is a list of return types:

Return Type	Explaination	Example	Note
void	Status Reply that should always return a string of "OK"	RENAME, SETEX
std::string	Status Reply that NOT always return "OK", and Bulk String Reply	PING, INFO
bool	Integer Reply that always returns 0 or 1	EXPIRE, HSET	See the Boolean Return Value section for the meaning of a boolean return value
long long	Integer Reply that not always return 0 or 1	DEL, APPEND
double	Bulk String Reply that represents a double	INCRBYFLOAT, ZINCRBY
std::pair	Array Reply with exactly 2 elements. Since the return value is always an array of 2 elements, we return the 2 elements as a `std::pair`'s first and second elements	BLPOP
std::tuple	Array Reply with fixed length and has more than 2 elements. Since length of the returned array is fixed, we return the array as a `std::tuple`	BZPOPMAX
output iterator	General Array Reply with non-fixed/dynamic length. We use STL-like interface to return this kind of array replies, so that you can insert the return value into a STL container easily	MGET, LRANGE	Also, sometimes the type of output iterator decides which options to send with the command. See the Examples section for details
Optional< T >	For any reply of type `T` that might be NULL	GET, LPOP, BLPOP, BZPOPMAX	See the Optional section for details on `Optional<T>`

Boolean Return Value

The return type of some methods, e.g. EXPIRE, HSET, is bool. If the method returns false, it DOES NOT mean that Redis failed to send the command to Redis server. Instead, it means that Redis server returns an Integer Reply, and the value of the reply is 0. Accordingly, if the method returns true, it means that Redis server returns an Integer Reply, and the value of the reply is 1. You can
check Redis commands manual for what do 0 and 1 stand for.

For example, when we send EXPIRE command to Redis server, it returns 1 if the timeout was set, and it returns 0 if the key doesn't exist. Accordingly, if the timeout was set, Redis::expire returns true, and if the key doesn't exist, Redis::expire returns false.

So, never use the return value to check if the command has been successfully sent to Redis server. Instead, if Redis failed to send command to server, it throws an exception of type Error. See the Exception section for details on exceptions.

Optional

std::optional is a good option for return type, if Redis might return NULL REPLY. Again, since not all compilers support std::optional so far, we implement our own simple version, i.e. Optional<T>.

Take the GET and MGET commands for example:

c++

// Or just: auto val = redis.get("key");
Optional<std::string> val = redis.get("key");

// Optional<T> has a conversion to bool.
// If it's NOT a null Optional<T> object, it's converted to true.
// Otherwise, it's converted to false.
if (val) {
    // Key exists. Dereference val to get the string result.
    std::cout << *val << std::endl;
} else {
    // Redis server returns a NULL Bulk String Reply.
    // It's invalid to dereference a null Optional<T> object.
    std::cout << "key doesn't exist." << std::endl;
}

std::vector<Optional<std::string>> values;
redis.mget({"key1", "key2", "key3"}, std::back_inserter(values));
for (const auto &val : values) {
    if (val) {
        // Key exist, process the value.
    }
}

We also have some typedefs for some commonly used Optional<T>:

c++

using OptionalString = Optional<std::string>;

using OptionalLongLong = Optional<long long>;

using OptionalDouble = Optional<double>;

using OptionalStringPair = Optional<std::pair<std::string, std::string>>;

Exception

Redis throws exceptions if it receives an Error Reply or something bad happens, e.g. failed to create a connection to server, or connection to server is broken. All exceptions derived from Error class. See errors.h for details.

Error: Generic error. It's also the base class of other exceptions.
IoError: There's some IO error with the connection.
TimeoutError: Read or write operation was timed out. It's a derived class of IoError.
ClosedError: Redis server closed the connection.
ProtoError: The command or reply is invalid, and we cannot process it with Redis protocol.
OomError: hiredis library got an out-of-memory error.
ReplyError: Redis server returned an error reply, e.g. we try to call redis::lrange on a Redis hash.
WatchError: Watched key has been modified. See Watch section for details.

NOTE: NULL REPLYis not taken as an exception. For example, if we try toGETa non-existent key, we'll get a *NULL Bulk String Reply*. Instead of throwing an exception, we return the *NULL REPLY* as a nullOptional` object. Also see Optional section.

Examples

Let's see some examples on how to send commands to Redis server.

Various Parameter Types

text

/* Detailed source-code truncated for AI context efficiency. */

Various Return Types

text

/* Detailed source-code truncated for AI context efficiency. */

SCAN Commands

c++

auto cursor = 0LL;
auto pattern = "*pattern*";
auto count = 5;
std::vector<std::string> scan_vec;
while (true) {
    cursor = redis.scan(cursor, pattern, count, std::back_inserter(scan_vec));
    // Default pattern is "*", and default count is 10
    // cursor = redis.scan(cursor, std::back_inserter(scan_vec));

    if (cursor == 0) {
        break;
    }
}

Command Overloads

Sometimes the type of output iterator decides which options to send with the command.

text

/* Detailed source-code truncated for AI context efficiency. */

Please see redis.h for more API references, and see the tests for more examples.

Generic Command Interface

There're too many Redis commands, we haven't implemented all of them. However, you can use the generic Redis::command methods to send any commands to Redis. Unlike other client libraries, Redis::command doesn't use format string to combine command arguments into a command string. Instead, you can directly pass command arguments of StringView type or arithmetic type as parameters of Redis::command. For the reason why we don't use format string, please see this discussion.

text

/* Detailed source-code truncated for AI context efficiency. */

NOTE: The name of some Redis commands is composed with two strings, e.g. CLIENT SETNAME. In this case, you need to pass these two strings as two arguments for Redis::command.

c++

// This is GOOD.
redis.command<void>("client", "setname", "name");

// This is BAD, and will fail to send command to Redis server.
// redis.command<void>("client setname", "name");

As I mentioned in the comments, the SET command not always returns void. Because if you try to set a (key, value) pair with NX or XX option, you might fail, and Redis will return a NULL REPLY. Besides the SET command, there're other commands whose return value is NOT a fixed type, you need to parse it by yourself. For example, Redis::set method rewrite the reply of SET command, and make it return bool type, i.e. if no NX or XX option specified, Redis server will always return an "OK" string, and Redis::set returns true; if NX or XX specified, and Redis server returns a NULL REPLY, Redis::set returns false.

So Redis class also has other overloaded command methods, these methods return a ReplyUPtr, i.e. std::unique_ptr<redisReply, ReplyDeleter>, object. Normally you don't need to parse it manually. Instead, you only need to pass the reply to template <typename T> T reply::parse(redisReply &) to get a value of type T. Check the Return Type section for valid T types. If the command returns an array of elements, besides calling reply::parse to parse the reply to an STL container, you can also call template <typename Output> reply::to_array(redisReply &reply, Output output) to parse the result into an array or STL container with an output iterator.

Let's rewrite the above examples:

text

/* Detailed source-code truncated for AI context efficiency. */

In fact, there's one more Redis::command method:

c++

template <typename Cmd, typename ...Args>
auto command(Cmd cmd, Args &&...args)
    -> typename std::enable_if<!std::is_convertible<Cmd, StringView>::value, ReplyUPtr>::type;

However, this method exposes some implementation details, and is only for internal use. You should NOT use this method.

You can use Redis::publish to publish messages to channels. Redis randomly picks a connection from the underlying connection pool, and publishes message with that connection. So you might publish two messages with two different connections.

When you subscribe to a channel with a connection, all messages published to the channel are sent back to that connection. So there's NO Redis::subscribe method. Instead, you can call Redis::subscriber to create a Subscriber and the Subscriber maintains a connection to Redis. The underlying connection is a new connection, NOT picked from the connection pool. This new connection has the same ConnectionOptions as the Redis object.

With Subscriber, you can call Subscriber::subscribe, Subscriber::unsubscribe, Subscriber::psubscribe and Subscriber::punsubscribe to send SUBSCRIBE, UNSUBSCRIBE, PSUBSCRIBE and PUNSUBSCRIBE commands to Redis.

Thread Safety

Subscriber is NOT thread-safe. If you want to call its member functions in multi-thread environment, you need to synchronize between threads manually.

Subscriber Callbacks

There are 6 kinds of messages:

MESSAGE: message sent to a channel.
PMESSAGE: message sent to channels of a given pattern.
SUBSCRIBE: message sent when we successfully subscribe to a channel.
UNSUBSCRIBE: message sent when we successfully unsubscribe to a channel.
PSUBSCRIBE: message sent when we successfully subscribe to a channel pattern.
PUNSUBSCRIBE: message sent when we successfully unsubscribe to a channel pattern.

We call messages of SUBSCRIBE, UNSUBSCRIBE, PSUBSCRIBE and PUNSUBSCRIBE types as META MESSAGEs.

In order to process these messages, you can set callback functions on Subscriber:

Subscriber::on_message(MsgCallback): set callback function for messages of MESSAGE type, and the callback interface is: void (std::string channel, std::string msg).
Subscriber::on_pmessage(PatternMsgCallback): set the callback function for messages of PMESSAGE type, and the callback interface is: void (std::string pattern, std::string channel, std::string msg).
Subscriber::on_meta(MetaCallback): set callback function for messages of META MESSAGE type, and the callback interface is: void (Subscriber::MsgType type, OptionalString channel, long long num). type is an enum, it can be one of the following enum: Subscriber::MsgType::SUBSCRIBE, Subscriber::MsgType::UNSUBSCRIBE, Subscriber::MsgType::PSUBSCRIBE, Subscriber::MsgType::PUNSUBSCRIBE, Subscriber::MsgType::MESSAGE, and Subscriber::MsgType::PMESSAGE. If you haven't subscribe/psubscribe to any channel/pattern, and try to unsubscribe/punsubscribe without any parameter, i.e. unsubscribe/punsubscribe all channels/patterns, channel will be null. So the second parameter of meta callback is of type OptionalString.

All these callback interfaces pass std::string by value, and you can take their ownership (i.e. std::move) safely.

Consume Messages

You can call Subscriber::consume to consume messages published to channels/patterns that the Subscriber has been subscribed.

Subscriber::consume waits for message from the underlying connection. If the ConnectionOptions::socket_timeout is reached, and there's no message sent to this connection, Subscriber::consume throws a TimeoutError exception. If ConnectionOptions::socket_timeout is 0ms, Subscriber::consume blocks until it receives a message.

After receiving the message, Subscriber::consume calls the callback function to process the message based on message type. However, if you don't set callback for a specific kind of message, Subscriber::consume will ignore the received message, i.e. no callback will be called.

Examples

The following example is a common pattern for using Subscriber:

c++

// Create a Subscriber.
auto sub = redis.subscriber();

// Set callback functions.
sub.on_message([](std::string channel, std::string msg) {
            // Process message of MESSAGE type.
        });

sub.on_pmessage([](std::string pattern, std::string channel, std::string msg) {
            // Process message of PMESSAGE type.
        });

sub.on_meta([](Subscriber::MsgType type, OptionalString channel, long long num) {
            // Process message of META type.
        });

// Subscribe to channels and patterns.
sub.subscribe("channel1");
sub.subscribe({"channel2", "channel3"});

sub.psubscribe("pattern1*");

// Consume messages in a loop.
while (true) {
    try {
        sub.consume();
    } catch (const Error &err) {
        // Handle exceptions.
    }
}

If ConnectionOptions::socket_timeout is set, you might get TimeoutError exception before receiving a message:

c++

while (true) {
    try {
        sub.consume();
    } catch (const TimeoutError &e) {
        // Try again.
        continue;
    } catch (const Error &err) {
        // Handle other exceptions.
    }
}

The above examples use lambda as callback. If you're not familiar with lambda, you can also set a free function as callback. Check this issue for detail.

Pipeline

Pipeline is used to reduce RTT (Round Trip Time), and speed up Redis queries. redis-plus-plus supports pipeline with the Pipeline class.

Create Pipeline

You can create a pipeline with Redis::pipeline method, which returns a Pipeline object.

c++

ConnectionOptions connection_options;
ConnectionPoolOptions pool_options;

Redis redis(connection_options, pool_options);

auto pipe = redis.pipeline();

When creating a Pipeline object, Redis::pipeline method creates a new connection to Redis server. This connection is NOT picked from the connection pool, but a newly created connection. This connection has the same ConnectionOptions as other connections in the connection pool. Pipeline object maintains the new connection, and all piped commands are sent through this connection.

NOTE: Creating a Pipeline object is NOT cheap, since it creates a new connection. So you'd better reuse the Pipeline object as much as possible.

Send Commands

You can send Redis commands through the Pipeline object. Just like the Redis class, Pipeline has one or more (overloaded) methods for each Redis command. However, you CANNOT get the replies until you call Pipeline::exec. So these methods do NOT return the reply, instead they return the Pipeline object itself. And you can chain these methods calls.

c++

pipe.set("key", "val").incr("num").rpush("list", {0, 1, 2}).command("hset", "key", "field", "value");

Get Replies

Once you finish sending commands to Redis, you can call Pipeline::exec to get replies of these commands. You can also chain Pipeline::exec with other commands.

c++

pipe.set("key", "val").incr("num");
auto replies = pipe.exec();

// The same as:
replies = pipe.set("key", "val").incr("num).exec();

In fact, these commands won't be sent to Redis, until you call Pipeline::exec. So Pipeline::exec does 2 work in order: send all piped commands, then get all replies from Redis.

Also you can call Pipeline::discard to discard those piped commands.

c++

pipe.set("key", "val").incr("num");

pipe.discard();

Parse Replies

Pipeline::exec returns a QueuedReplies object, which contains replies of all commands that have been sent to Redis. You can use QueuedReplies::get method to get and parse the ith reply. It has 3 overloads:

template <typename Result> Result get(std::size_t idx): Return the ith reply as a return value, and you need to specify the return type as tempalte parameter.
template <typename Output> void get(std::size_t idx, Output output): If the reply is of type Array Reply, you can call this method to write the ith reply to an output iterator. Normally, compiler will deduce the type of the output iterator, and you don't need to specify the type parameter explicitly.
redisReply& get(std::size_t idx): If the reply is NOT a fixed type, call this method to get a reference to redisReply object. In this case, you need to call template <typename T> T reply::parse(redisReply &) to parse the reply manually.

Check the Return Type section for details on the return types of the result.

c++

auto replies = pipe.set("key", "val").incr("num").lrange("list", 0, -1).exec();

auto set_cmd_result = replies.get<bool>(0);

auto incr_cmd_result = replies.get<long long>(1);

std::vector<std::string> list_cmd_result;
replies.get(2, std::back_inserter(list_cmd_result));

Exception

If any of Pipeline's method throws an exception, the Pipeline object enters an invalid state. You CANNOT use it any more, but only destroy the object, and create a new one.

Thread Safety

Pipeline is NOT thread-safe. If you want to call its member functions in multi-thread environment, you need to synchronize between threads manually.

Transaction

Transaction is used to make multiple commands runs atomically.

Create Transaction

You can create a transaction with Redis::transaction method, which returns a Transaction object.

c++

ConnectionOptions connection_options;
ConnectionPoolOptions pool_options;

Redis redis(connection_options, pool_options);

auto tx = redis.transaction();

As the Pipeline class, Transaction maintains a newly created connection to Redis. This connection has the same ConnectionOptions as the Redis object.

NOTE: Creating a Transaction object is NOT cheap, since it creates a new connection. So you'd better reuse the Transaction as much as possible.

Also you don't need to send MULTI command to Redis. Transaction will do that for you automatically.

Send Commands

Transaction shares most of implementation with Pipeline. It has the same interfaces as Pipeline. You can send commands as what you do with Pipeline object.

c++

tx.set("key", "val").incr("num").lpush("list", {0, 1, 2}).command("hset", "key", "field", "val");

Execute Transaction

When you call Transaction::exec, you explicitly ask Redis to execute those queued commands, and return the replies. Otherwise, these commands won't be executed. Also, you can call Transaction::discard to discard the execution, i.e. no command will be executed. Both Transaction::exec and Transaction::discard can be chained with other commands.

c++

auto replies = tx.set("key", "val").incr("num").exec();

tx.set("key", "val").incr("num");

// Discard the transaction.
tx.discard();

Parse Replies

See Pipeline's Parse Replies section for how to parse the replies.

Piped Transaction

Normally, we always send multiple commnds in a transaction. In order to improve the performance, you can send these commands in a pipeline. You can create a piped transaction by passing true as parameter of Redis::transaction method.

c++

// Create a piped transaction
auto tx = redis.transaction(true);

With this piped transaction, all commands are sent to Redis in a pipeline.

Exception

If any of Transaction's method throws an exception other than WatchError, the Transaction object enters an invalid state. You CANNOT use it any more, but only destroy the object and create a new one.

Thread Safety

Transacation is NOT thread-safe. If you want to call its member functions in multi-thread environment, you need to synchronize between threads manually.

Watch

WATCH is used to provide a check-and-set(CAS) behavior to Redis transactions.

The WATCH command must be sent in the same connection as the transaction. And normally after the WATCH command, we also need to send some other commands to get data from Redis before executing the transaction. Take the following check-and-set case as an example:

text

WATCH key           // watch a key
val = GET key       // get value of the key
new_val = val + 1   // incr the value
MULTI               // begin the transaction
SET key new_val     // set value only if the value is NOT modified by others
EXEC                // try to execute the transaction.
                    // if val has been modified, the transaction won't be executed.

However, with Transaction object, you CANNOT get the result of commands until the whole transaction has been finished. Instead, you need to create a Redis object from the Transaction object. The created Redis object shares the connection with Transaction object. With this created Redis object, you can send WATCH command and any other Redis commands to Redis server, and get the result immediately.

Let's see how to implement the above example with redis-plus-plus:

c++

auto redis = Redis("tcp://127.0.0.1");

// Create a transaction.
auto tx = redis.transaction();

// Create a Redis object from the Transaction object. Both objects share the same connection.
auto r = tx.redis();

// If the watched key has been modified by other clients, the transaction might fail.
// So we need to retry the transaction in a loop.
while (true) {
    try {
        // Watch a key.
        r.watch("key");

        // Get the old value.
        auto val = r.get("key");
        auto num = 0;
        if (val) {
            num = std::stoi(*val);
        } // else use default value, i.e. 0.

        // Incr value.
        ++num;

        // Execute the transaction.
        auto replies = tx.set("key", std::to_string(num)).exec();

        // Transaction has been executed successfully. Check the result and break.

        assert(replies.size() == 1 && replies.get<bool>(0) == true);

        break;
    } catch (const WatchError &err) {
        // Key has been modified by other clients, retry.
        continue;
    } catch (const Error &err) {
        // Something bad happens, and the Transaction object is no longer valid.
        throw;
    }
}

Redis Cluster

redis-plus-plus supports Redis Cluster. You can use RedisCluster class to send commands to Redis Cluster. It has similar interfaces as Redis class.

Connection

RedisCluster connects to all master nodes in the cluster. For each master node, it maintains a connection pool. By now, it doesn't connect to slave nodes.

You can initialize a RedisCluster instance with ConnectionOptions and ConnectionPoolOptions. You only need to set one master node's host & port in ConnectionOptions, and RedisCluster will get other nodes' info automatically (with the CLUSTER SLOTS command). For each master node, it creates a connection pool with the specified ConnectionPoolOptions. If ConnectionPoolOptions is not specified, RedisCluster maintains a single connection to every master node.

c++

// Set a master node's host & port.
ConnectionOptions connection_options;
connection_options.host = "127.0.0.1";  // Required.
connection_options.port = 7000; // Optional. The default port is 6379.
connection_options.password = "auth"; // Optional. No password by default.

// Automatically get other nodes' info,
// and connect to every master node with a single connection.
RedisCluster cluster1(connection_options);

ConnectionPoolOptions pool_options;
pool_options.size = 3;

// For each master node, maintains a connection pool of size 3.
RedisCluster cluster2(connection_options, pool_options);

You can also specify connection option with an URI. However, in this way, you can only use default ConnectionPoolOptions, i.e. pool of size 1, and CANNOT specify password.

c++

// Specify a master node's host & port.
RedisCluster cluster3("tcp://127.0.0.1:7000");

// Use default port, i.e. 6379.
RedisCluster cluster4("tcp://127.0.0.1");

Note

RedisCluster only works with tcp connection. It CANNOT connect to Unix Domain Socket. If you specify Unix Domain Socket in ConnectionOptions, it throws an exception.
All nodes in the cluster should have the same password.
Since Redis Cluster does NOT support multiple databses, ConnectionOptions::db is ignored.

Interfaces

As we mentioned above, RedisCluster's interfaces are similar to Redis. It supports most of Redis' interfaces, including the generic command interface (see Redis' API Reference section for details), except the following:

Not support commands without key as argument, e.g. PING, INFO.
Not support Lua script without key parameters.

Since there's no key parameter, RedisCluster has no idea on to which node these commands should be sent. However there're 2 workarounds for this problem:

If you want to send these commands to a specific node, you can create a Redis object with that node's host and port, and use the Redis object to do the work.
Instead of host and port, you can also call Redis RedisCluster::redis(const StringView &hash_tag) to create a Redis object with a hash-tag specifying the node. In this case, the returned Redis object creates a new connection to Redis server.

Also you can use the hash tags to send multiple-key commands.

See the example section for details.

You can publish and subscribe messages with RedisCluster. The interfaces are exactly the same as Redis, i.e. use RedisCluster::publish to publish messages, and use RedisCluster::subscriber to create a subscriber to consume messages. See Publish/Subscribe section for details.

Pipeline and Transaction

You can also create Pipeline and Transaction objects with RedisCluster, but the interfaces are different from Redis. Since all commands in the pipeline and transaction should be sent to a single node in a single connection, we need to tell RedisCluster with which node the pipeline or transaction should be created.

Instead of specifing the node's IP and port, RedisCluster's pipeline and transaction interfaces allow you to specify the node with a hash tag. RedisCluster will calculate the slot number with the given hash tag, and create a pipeline or transaction with the node holding the slot.

c++

Pipeline RedisCluster::pipeline(const StringView &hash_tag);

Transaction RedisCluster::transaction(const StringView &hash_tag, bool piped = false);

With the created Pipeline or Transaction object, you can send commands with keys located on the same node as the given hash_tag. See Examples section for an example.

Examples

c++

#include <sw/redis++/redis++.h>

using namespace sw::redis;

auto redis_cluster = RedisCluster("tcp://127.0.0.1:7000");

redis_cluster.set("key", "value");
auto val = redis_cluster.get("key");
if (val) {
    std::cout << *val << std::endl;
}

// With hash-tag.
redis_cluster.set("key{tag}1", "val1");
redis_cluster.set("key{tag}2", "val2");
redis_cluster.set("key{tag}3", "val3");
std::vector<OptionalString> hash_tag_res;
redis_cluster.mget({"key{tag}1", "key{tag}2", "key{tag}3"},
        std::back_inserter(hash_tag_res));

redis_cluster.lpush("list", {"1", "2", "3"});
std::vector<std::string> list;
redis_cluster.lrange("list", 0, -1, std::back_inserter(list));

// Pipline.
auto pipe = redis_cluster.pipeline("counter");
auto replies = pipe.incr("{counter}:1").incr("{counter}:2").exec();

// Transaction.
auto tx = redis_cluster.transaction("key");
replies = tx.incr("key").get("key").exec();

// Create a Redis object with hash-tag.
// It connects to the Redis instance that holds the given key, i.e. hash-tag.
auto r = redis_cluster.redis("hash-tag");

// And send command without key parameter to the server.
r.command("client", "setname", "connection-name");

NOTE: When you use RedisCluster::redis(const StringView &hash_tag) to create a Redis object, instead of picking a connection from the underlying connection pool, it creates a new connection to the corresponding Redis server. So this is NOT a cheap operation, and you should try to reuse this newly created Redis object as much as possible.

c++

// This is BAD! It's very inefficient.
// NEVER DO IT!!!
// After sending PING command, the newly created Redis object will be destroied.
cluster.redis("key").ping();

// Then it creates a connection to Redis, and closes the connection after sending the command.
cluster.redis("key").command("client", "setname", "hello");

// Instead you should reuse the Redis object.
// This is GOOD!
auto redis = cluster.redis("key");

redis.ping();
redis.command("client", "setname", "hello");

Details

RedisCluster maintains the newest slot-node mapping, and sends command directly to the right node. Normally it works as fast as Redis. If the cluster reshards, RedisCluster will follow the redirection, and it will finally update the slot-node mapping. It can correctly handle the following resharding cases:

Data migration between exist nodes.
Add new node to the cluster.
Remove node from the cluster.

redis-plus-plus is able to handle both MOVED and ASK redirections, so it's a complete Redis Cluster client.

If master is down, the cluster will promote one of its replicas to be the new master. redis-plus-plus can also handle this case:

When the master is down, redis-plus-plus losts connection to it. In this case, if you try to send commands to this master, redis-plus-plus will try to update slot-node mapping from other nodes. If the mapping remains unchanged, i.e. new master hasn't been elected yet, it fails to send command to Redis Cluster and throws exception.
When the new master has been elected, the slot-node mapping will be updated by the cluster. In this case, if you send commands to the cluster, redis-plus-plus can get an update-to-date mapping, and sends commands to the new master.

Redis Sentinel

Redis Sentinel provides high availability for Redis. If Redis master is down, Redis Sentinels will elect a new master from slaves, i.e. failover. Besides, Redis Sentinel can also act like a configuration provider for clients, and clients can query master or slave address from Redis Sentinel. So that if a failover occurs, clients can ask the new master address from Redis Sentinel.

redis-plus-plus supports getting Redis master or slave's IP and port from Redis Sentinel. In order to use this feature, you only need to initialize Redis object with Redis Sentinel info, which is composed with 3 parts: std::shared_ptr<Sentinel>, master name and role (master or slave).

Before using Redis Sentinel with redis-plus-plus, ensure that you have read Redis Sentinel's doc.

Sentinel

You can create a std::shared_ptr<Sentinel> object with SentinelOptions.

c++

SentinelOptions sentinel_opts;
sentinel_opts.nodes = {{"127.0.0.1", 9000},
                        {"127.0.0.1", 9001},
                        {"127.0.0.1", 9002}};   // Required. List of Redis Sentinel nodes.

// Optional. Timeout before we successfully connect to Redis Sentinel.
// By default, the timeout is 100ms.
sentinel_opts.connect_timeout = std::chrono::milliseconds(200);

// Optional. Timeout before we successfully send request to or receive response from Redis Sentinel.
// By default, the timeout is 100ms.
sentinel_opts.socket_timeout = std::chrono::milliseconds(200);

auto sentinel = std::make_shared<Sentinel>(sentinel_opts);

SentinelOptions::connect_timeout and SentinelOptions::socket_timeout CANNOT be 0ms, i.e. no timeout and block forever. Otherwise, redis-plus-plus will throw an exception.

See SentinelOptions for more options.

Role

Besides std::shared_ptr<Sentinel> and master name, you also need to specify a role. There are two roles: Role::MASTER, and Role::SLAVE.

With Role::MASTER, redis-plus-plus will always connect to current master instance, even if a failover occurs. Each time when redis-plus-plus needs to create a new connection to master, or a connection is broken, and it needs to reconnect to master, redis-plus-plus will ask master address from Redis Sentinel, and connects to current master. If a failover occurs, redis-plus-plus can automatically get the address of the new master, and refresh all connections in the underlying connection pool.

Similarly, with Role::SLAVE, redis-plus-plus will always connect to a slave instance. A master might have several slaves, redis-plus-plus will randomly pick one, and connect to it, i.e. all connections in the underlying connection pool, connect to the same slave instance. If the connection is broken, while this slave instance is still an alive slave, redis-plus-plus will reconnect to this slave. However, if this slave instance is down, or it has been promoted to be the master, redis-plus-plus will randomly connect to another slave. If there's no slave alive, it throws an exception.

Create Redis With Sentinel

When creating a Redis object with sentinel, besides the sentinel info, you should also provide ConnectionOptions and ConnectionPoolOptions. These two options are used to connect to Redis instance. ConnectionPoolOptions is optional, if not specified, it creates a single connection the instance.

c++

ConnectionOptions connection_opts;
connection_opts.password = "auth";  // Optional. No password by default.
connection_opts.connect_timeout = std::chrono::milliseconds(100);   // Required.
connection_opts.socket_timeout = std::chrono::milliseconds(100);    // Required.

ConnectionPoolOptions pool_opts;
pool_opts.size = 3; // Optional. The default size is 1.

auto redis = Redis(sentinel, "master_name", Role::MASTER, connection_opts, pool_opts);

You might have noticed that we didn't specify the host and port fields for ConnectionOptions. Because, Redis will get these info from Redis Sentinel. Also, in this case, ConnectionOptions::connect_timeout and ConnectionOptions::socket_timeout CANNOT be 0ms, otherwise, it throws an exception. So you always need to specify these two timeouts manually.

After creating the Redis object with sentinel, you can send commands with it, just like an ordinary Redis object.

If you want to write to master, and scale read with slaves. You can use the following pattern:

c++

auto sentinel = std::make_shared<Sentinel>(sentinel_opts);

auto master = Redis(sentinel, "master_name", Role::MASTER, connection_opts, pool_opts);

auto slave = Redis(sentinel, "master_name", Role::SLAVE, connection_opts, pool_opts);

// Write to master.
master.set("key", "value");

// Read from slave.
slave.get("key");

Redis Stream

Since Redis 5.0, it introduces a new data type: Redis Stream. redis-plus-plus has built-in methods for all stream commands except the XINFO command (of course, you can use the Generic Command Interface to send XINFO command).

However, the replies of some streams commands, i.e. XPENDING, XREAD, are complex. So I'll give some examples to show you how to work with these built-in methods.

Examples

text

/* Detailed source-code truncated for AI context efficiency. */

If you have any problem on sending stream commands to Redis, please feel free to let me know.

Author

redis-plus-plus is written by sewenew, who is also active on StackOverflow.

Overview (`ext/redis-plus-plus-1.3.3/README.md`)

redis-plus-plus

Overview
- Features
- Branches
Installation
Getting Started
API Reference
Redis Recipes
- Redlock
Author

Overview

This is a C++ client library for Redis. It's based on hiredis, and is compatible with C++ 17, C++ 14, and C++ 11.

NOTE: I'm not a native speaker. So if the documentation is unclear, please feel free to open an issue or pull request. I'll response ASAP.

Features

Most commands for Redis.
Connection pool.
Redis scripting.
Thread safe unless otherwise stated.
Redis publish/subscribe.
Redis pipeline.
Redis transaction.
Redis Cluster.
Redis Sentinel.
STL-like interfaces.
Generic command interface.
Redis Stream.
Redlock.
Redis ACL.
TLS/SSL support.

Branches

The master branch is the stable branch, which passes all tests. The dev branch is unstable. If you want to contribute, please create pull request on dev branch.

Installation

Install hiredis

Since redis-plus-plus is based on hiredis, you should install hiredis first. The minimum version requirement for hiredis is v0.12.1. However, the latest stable release of hiredis is always recommended.

NOTE: You must ensure that there's only 1 version of hiredis is installed. Otherwise, you might get some wired problems. Check the following issues for example: issue 135, issue 140 and issue 158.

Normally, you can install hiredis with a C++ package manager, and that's the easiest way to do it, e.g. sudo apt-get install libhiredis-dev. However, if you want to install the latest code of hiredis, or a specified version (e.g. async support needs hiredis v1.0.0 or later), you can install it from source.

Note again: DO NOT INSTALL MULTIPLE VERSIONS OF HIREDIS.

text

git clone https://github.com/redis/hiredis.git

cd hiredis

make

make install

By default, hiredis is installed at /usr/local. If you want to install hiredis at non-default location, use the following commands to specify the installation path.

text

make PREFIX=/non/default/path

make PREFIX=/non/default/path install

Install redis-plus-plus

redis-plus-plus is built with CMAKE.

text

git clone https://github.com/sewenew/redis-plus-plus.git

cd redis-plus-plus

mkdir build

cd build

cmake -DREDIS_PLUS_PLUS_CXX_STANDARD=17 ..

make

make install

cd ..

text

cmake -DCMAKE_PREFIX_PATH=/path/to/hiredis -DCMAKE_INSTALL_PREFIX=/path/to/install/redis-plus-plus ..

Since version 1.3.0, by default, redis-plus-plus is built with the -std=c++17 standard. So that we can use the std::string_view and std::optional features. However, it can also be built with the -std=c++11 or -std=c++14 standard, and in that case, we have our own simple implementation of std::string_view and std::optional. In order to explicitly specify c++ standard, you can use the following cmake flag: -DREDIS_PLUS_PLUS_CXX_STANDARD=11.

text

cmake -DCMAKE_PREFIX_PATH=/path/to/hiredis -DCMAKE_INSTALL_PREFIX=/path/to/install/redis-plus-plus -DREDIS_PLUS_PLUS_CXX_STANDARD=11 ..

NOTE: You should build redis-plus-plus and your application with the same standard, e.g. if you build redis-plus-plus with C++17 standard, you MUST also build your application code with C++17 standard.

When compiling redis-plus-plus, it also compiles a test program, which might take a while. However, you can disable building test with the following cmake option: -DREDIS_PLUS_PLUS_BUILD_TEST=OFF.

text

cmake -DCMAKE_PREFIX_PATH=/path/to/hiredis -DCMAKE_INSTALL_PREFIX=/path/to/install/redis-plus-plus -DREDIS_PLUS_PLUS_BUILD_TEST=OFF ..

By default, redis-plus-plus builds both a static library and a shared library. If you only want to build one of them, you can disable the other with -DREDIS_PLUS_PLUS_BUILD_STATIC=OFF or -DREDIS_PLUS_PLUS_BUILD_SHARED=OFF.

redis-plus-plus builds static library with -fPIC option, i.e. Position Independent Code, by default. However, you can disable it with -DREDIS_PLUS_PLUS_BUILD_STATIC_WITH_PIC=OFF.

Windows Support

Now hiredis has Windows support, and since Visual Studio 2017, Visual Studio has built-in support for CMake. So redis-plus-plus also supports Windows platform now. It has been fully tested with Visual Studio 2017 and later on Win 10. I'm not familiar with Visual Studio environment, and the following doc might not be accurate. If you're familiar with the Windows platform, feel free to update this doc on how to install redis-plus-plus on Windows.

CMake Support On Visual Studio

The following are some links on how to build CMake project with Visual Studio 2017 or later. If you're not familiar with it, you'd better read these instructions first:

NOTE: IMHO, Visual Studio 2017's support for CMake project is not very mature, and I recommend you to build hiredis and *redis-plus-plus with Visual Studio 2019.

Build hiredis

First of all, you need to get the latest code of hiredis on master branch. Older version might not support Windows platform. hiredis' CMakeLists.txt uses add_compile_definitions method, which is only supported by cmake 3.12 or later. However, Visual Studio 2017's cmake version is older than that. So if you're using Visual Studio 2017, you need to comment the following line in the CMakeLists.txt file:

text

#IF(WIN32)
#    ADD_COMPILE_DEFINITIONS(_CRT_SECURE_NO_WARNINGS WIN32_LEAN_AND_MEAN)
#ENDIF()

You can use the Open Folder feature to open hiredis project, and build it with the instructions (links) mentioned above.

Build redis-plus-plus

Since redis-plus-plus depends on hiredis, we need to specify the installation paths of hiredis before building it. You can use the Open Folder feature to open redis-plus-plus project. You need to edit the CMakeSetting.json file (automatically generated by Visual Studio) to set HIREDIS_HEADER, HIREDIS_LIB and TEST_HIREDIS_LIB variables to specify the installation path of hiredis headers, installation path of hiredis dynamic library and installation path of hiredis static library. The following is an example of CMakeSetting.json file:

text

{
    "configurations": [
      {
        "name": "x64-Release",
        "generator": "Visual Studio 15 2017 Win64",
        "configurationType": "Release",
        "buildRoot": "${env.LOCALAPPDATA}\\CMakeBuild\\${workspaceHash}\\build\\${name}",
        "cmakeCommandArgs": "",
        "buildCommandArgs": "-m -v:minimal",
        "variables": [
          {
            "name": "HIREDIS_HEADER",
            "value": "installation path of hiredis header files",
            "type": "PATH"
          },
          {
            "name": "HIREDIS_LIB",
            "value": "installation path of dynamic library of hiredis",
            "type": "FILEPATH"
          },
          {
            "name": "TEST_HIREDIS_LIB",
            "value": "installation path of static library of hiredis",
            "type": "FILEPATH"
          }
        ]
      }
    ]
}

Then you can build it the instructions (links) mentioned above. If you're building with Visual Studio 2017 in debug mode, you might get /bigobj error when building the test. In this case, you can disable building test by setting -DREDIS_PLUS_PLUS_BUILD_TEST=OFF or build it in Release mode.

NOTE:

Since 1.3.0, redis-puls-plus is built with C++17 by default, and you should also set your application code to be built with C++17. If you still want to build the redis-plus-plus with C++11, you can set the REDIS_PLUS_PLUS_CXX_STANDARD cmake option to 11.
TLS/SSL support has not been tested on Windows yet.

The Order of Header Files

On Windows platform, if your application code also needs to include windows.h. You must ensure that sw/redis++/redis++.h is included before windows.h. Check this issue for detail.

Building a redis-plus-plus Debian Package (Optional)

Basic support for building a GNU/Debian package is supplied with the use of cmake.
The following example shows how to build the Debian package:

text

mkdir build; cd build
cmake ..
cpack -G DEB

The install prefix may be modified as follows:

text

mkdir build; cd build
cmake -DCMAKE_INSTALL_PREFIX=/usr ..
cpack -G DEB

Run Tests (Optional)

redis-plus-plus has been fully tested with the following compilers:

text

gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)
gcc version 5.5.0 20171010 (Ubuntu 5.5.0-12ubuntu1)
gcc version 6.5.0 20181026 (Ubuntu 6.5.0-2ubuntu1~18.04)
gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
gcc version 8.3.0 (Ubuntu 8.3.0-6ubuntu1~18.04.1)
gcc version 9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2)
gcc version 10.2.1 20210110 (Debian 10.2.1-6)
clang version 3.9.1-19ubuntu1 (tags/RELEASE_391/rc2)
clang version 4.0.1-10 (tags/RELEASE_401/final)
clang version 5.0.1-4 (tags/RELEASE_501/final)
clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
clang version 7.0.0-3~ubuntu0.18.04.1 (tags/RELEASE_700/final)
clang version 8.0.1-3build1 (tags/RELEASE_801/final)
Apple clang version 11.0.0 (clang-1100.0.33.12)
Visual Studio 2017 (Win 10)
Visual Studio 2019 (Win 10)

If you build redis-plus-plus with -DREDIS_PLUS_PLUS_BUILD_TEST=ON (the default behavior, and you can disable building test with -DREDIS_PLUS_PLUS_BUILD_TEST=OFF), you'll get a test program in compile/test directory: compile/test/test_redis++.

NOTE: The latest version of Redis is only a requirement for running the tests. In fact, you can use redis-plus-plus with Redis of any version, e.g. Redis 2.0, Redis 3.0, Redis 4.0, Redis 5.0.

NEVER run the test program in production envronment, since the keys, which the test program reads or writes, might conflict with your application.

In order to run tests with both Redis and Redis Cluster, you can run the test program with the following command:

text

./compile/test/test_redis++ -h host -p port -a auth -n cluster_node -c cluster_port

host and port are the host and port number of the Redis instance.
cluster_node and cluster_port are the host and port number of Redis Cluster. You only need to set the host and port number of a single node in the cluster, redis-plus-plus will find other nodes automatically.
auth is the password of the Redis instance and Redis Cluster. The Redis instance and Redis Cluster must be configured with the same password. If there's no password configured, don't set this option.

If you only want to run tests with Redis, you only need to specify host, port and auth options:

text

./compile/test/test_redis++ -h host -p port -a auth

Similarly, if you only want to run tests with Redis Cluster, just specify cluster_node, cluster_port and auth options:

text

./compile/test/test_redis++ -a auth -n cluster_node -c cluster_port

By default, the test program will not test running redis-plus-plus in multi-threads environment. If you want to do multi-threads test, which might cost a long time, you can specify the -m option:

text

./compile/test/test_redis++ -h host -p port -a auth -n cluster_node -c cluster_port -m

If all tests have been passed, the test program will print the following message:

text

Pass all tests

Otherwise, it prints the error message.

Performance

redis-plus-plus runs as fast as hiredis, since it's a wrapper of hiredis. You can run test_redis++ in benchmark mode to check the performance in your environment.

text

./compile/test/test_redis++ -h host -p port -a auth -n cluster_node -c cluster_port -b -t thread_num -s connection_pool_size -r request_num -k key_len -v val_len

-b option turns the test program into benchmark mode.
thread_num specifies the number of worker threads. 10 by default.
connection_pool_size specifies the size of the connection pool. 5 by default.
request_num specifies the total number of requests sent to server for each test. 100000 by default.
key_len specifies the length of the key for each operation. 10 by default.
val_len specifies the length of the value. 10 by default.

Use redis-plus-plus In Your Project

Use Static Libraries

Take gcc as an example.

text

g++ -std=c++17 -o app app.cpp /path/to/libredis++.a /path/to/libhiredis.a -pthread

If hiredis and redis-plus-plus are installed at non-default location, you should use -I option to specify the header path.

text

g++ -std=c++17 -I/non-default/install/include/path -o app app.cpp /path/to/libredis++.a /path/to/libhiredis.a -pthread

Use Shared Libraries

text

g++ -std=c++17 -o app app.cpp -lredis++ -lhiredis -pthread

If hiredis and redis-plus-plus are installed at non-default location, you should use -I and -L options to specify the header and library paths.

text

g++ -std=c++17 -I/non-default/install/include/path -L/non-default/install/lib/path -o app app.cpp -lredis++ -lhiredis -pthread

When linking with shared libraries, and running your application, you might get the following error message:

text

error while loading shared libraries: xxx: cannot open shared object file: No such file or directory.

text

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

Check this StackOverflow question for details on how to solve the problem.

Build With Cmake

If you're using cmake to build your application, you need to add hiredis and redis-plus-plus dependencies in your CMakeLists.txt:

cmake

# <---------- set c++ standard ------------->
# NOTE: you must build redis-plus-plus and your application code with the same standard.
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

# <------------ add hiredis dependency --------------->
find_path(HIREDIS_HEADER hiredis)
target_include_directories(target PUBLIC ${HIREDIS_HEADER})

find_library(HIREDIS_LIB hiredis)
target_link_libraries(target ${HIREDIS_LIB})

# <------------ add redis-plus-plus dependency -------------->
# NOTE: this should be *sw* NOT *redis++*
find_path(REDIS_PLUS_PLUS_HEADER sw)
target_include_directories(target PUBLIC ${REDIS_PLUS_PLUS_HEADER})

find_library(REDIS_PLUS_PLUS_LIB redis++)
target_link_libraries(target ${REDIS_PLUS_PLUS_LIB})

See this issue for a complete example of CMakeLists.txt.

Also, if you installed hiredis and redis-plus-plus at non-default location, you need to run cmake with CMAKE_PREFIX_PATH option to specify the installation path of these two libraries.

text

cmake -DCMAKE_PREFIX_PATH=/installation/path/to/the/two/libs ..

Getting Started

text

/* Detailed source-code truncated for AI context efficiency. */

API Reference

You can also see redis.h for doxygen style documentation.

Connection

Redis class maintains a connection pool to Redis server. If the connection is broken, Redis reconnects to Redis server automatically.

c++

ConnectionOptions connection_options;
connection_options.host = "127.0.0.1";  // Required.
connection_options.port = 6666; // Optional. The default port is 6379.
connection_options.password = "auth";   // Optional. No password by default.
connection_options.db = 1;  // Optional. Use the 0th database by default.

// Optional. Timeout before we successfully send request to or receive response from redis.
// By default, the timeout is 0ms, i.e. never timeout and block until we send or receive successfuly.
// NOTE: if any command is timed out, we throw a TimeoutError exception.
connection_options.socket_timeout = std::chrono::milliseconds(200);

// Connect to Redis server with a single connection.
Redis redis1(connection_options);

ConnectionPoolOptions pool_options;
pool_options.size = 3;  // Pool size, i.e. max number of connections.

// Optional. Max time to wait for a connection. 0ms by default, which means wait forever.
// Say, the pool size is 3, while 4 threds try to fetch the connection, one of them will be blocked.
pool_options.wait_timeout = std::chrono::milliseconds(100);

// Optional. Max lifetime of a connection. 0ms by default, which means never expire the connection.
// If the connection has been created for a long time, i.e. more than `connection_lifetime`,
// it will be expired and reconnected.
pool_options.connection_lifetime = std::chrono::minutes(10);

// Connect to Redis server with a connection pool.
Redis redis2(connection_options, pool_options);

NOTE: if you set ConnectionOptions::socket_timeout, and try to call blocking commands, e.g. Redis::brpop, Redis::blpop, Redis::bzpopmax, Redis::bzpopmin, you must ensure that ConnectionOptions::socket_timeout is larger than the timeout specified with these blocking commands. Otherwise, you might get TimeoutError, and lose message.

See ConnectionOptions and ConnectionPoolOptions for more options. Also see issue 80 for discussion on connection pool.

NOTE: Redis class is movable but NOT copyable.

c++

// auto redis3 = redis1;    // this won't compile.

// But it's movable.
auto redis3 = std::move(redis1);

redis-plus-plus also supports connecting to Redis server with Unix Domain Socket.

c++

ConnectionOptions options;
options.type = ConnectionType::UNIX;
options.path = "/path/to/socket";
Redis redis(options);

You can also connect to Redis server with a URI:

text

tcp://[[username:]password@]host[:port][/db]

unix://[[username:]password@]path-to-unix-domain-socket[/db]

The scheme and host parts are required, and others are optional. If you're connecting to Redis with Unix Domain Socket, you should use the unix scheme, otherwise, you should use tcp scheme. The following is a list of default values for those optional parts:

username: default
password: empty string, i.e. no password
port: 6379
db: 0

NOTE: If your password or username contains '@', or your username contains ':', you cannot construct Redis object with URI. Because redis-plus-plus will incorrectly parse the URI. In this case, you need to use ConnectionOptions to construct Redis object.

NOTE: Redis 6.0 supports ACL, and you can specify a username for the connection. However, before Redis 6.0, you cannot do that.

Also, the following connection options can be specified with the query string of URI, e.g. tcp://127.0.0.1?keep_alive=true&socket_timeout=100ms&connect_timeout=100ms:

ConnectionOptions::keep_alive: false by default.
ConnectionOptions::socket_timeout: 0ms by default.
ConnectionOptions::connect_timeout: 0ms by default.

NOTE: Options specified in query string are case-sensitive, i.e. all key-value pairs must be in lowercase.

So far, you cannot specify connection pool options with URI, e.g. ConnectionPoolOptions::size.

c++

// Single connection to the given host and port.
Redis redis1("tcp://127.0.0.1:6666");

// Use default port, i.e. 6379.
Redis redis2("tcp://127.0.0.1");

// Connect to Redis with password, and default port.
Redis redis3("tcp://[email protected]");

// Connect to Redis and select the 2nd (db number starts from 0) database.
Redis redis4("tcp://127.0.0.1:6379/2");

// Set keep_alive option to true with query string.
Redis redis5("tcp://127.0.0.1:6379/2?keep_alive=true");

// Set socket_timeout to 50 milliseconds, and connect_timeout to 1 second with query string.
Redis redis6("tcp://127.0.0.1?socket_timeout=50ms&connect_timeout=1s");

// Connect to Unix Domain Socket.
Redis redis7("unix://path/to/socket");

Lazily Create Connection

Connection Failure

See the Exception section for details on exceptions.

Reuse Redis object As Much As Possible

c++

// This is GOOD practice.
auto redis = Redis("tcp://127.0.0.1");
for (auto idx = 0; idx < 100; ++idx) {
    // Reuse the Redis object in the loop.
    redis.set("key", "val");
}

// This is VERY BAD! It's very inefficient.
// NEVER DO IT!!!
for (auto idx = 0; idx < 100; ++idx) {
    // Create a new Redis object for each iteration.
    auto redis = Redis("tcp://127.0.0.1");
    redis.set("key", "val");
}

TLS/SSL Support

redis-plus-plus also has TLS support. However, in order to use this feature, you need to enable it when building hiredis and redis-plus-plus.

NOTE: So far, TLS feature has not been tested on Windows platform. I'll fix it in the future.

Enable TLS/SSL support

When building hiredis with TLS support, you need to download hiredis of version v1.0.0 or latter, and specify USE_SSL=1 flag:

text

make PREFIX=/non/default/path USE_SSL=1

make PREFIX=/non/default/path USE_SSL=1 install

Then you can build redis-plus-plus to enable TLS support by specifying the -DREDIS_PLUS_PLUS_USE_TLS=ON option:

text

cmake -DREDIS_PLUS_PLUS_USE_TLS=ON ..

Connection Options

In order to connect to Redis with TLS support, you need to specify the following connection options:

text

ConnectionOptions opts;
opts.host = "127.0.0.1";
opts.port = 6379;

opts.tls.enabled = true;    // Required. `false` by default.
opts.tls.cert = "/path/to/client/certificate";  // Optional
opts.tls.key = "/path/to/private/key/file"; // Optional
opts.tls.cacert = "/path/to/CA/certificate/file";   // You can also set `opts.tls.cacertdir` instead.
opts.tls.sni = "server-name-indication";    // Optional

Although tls.cert and tls.key are optional, if you specify one of them, you must also specify the other. Instead of specifying tls.cacert, you can also specify tls.cacertdir to the directory where certificates are stored.

These options are the same as redis-cli's TLS related command line arguments, so you can also run redis-cli --help to get the detailed explanation of these options.

Then you can use this ConnectionOptions to create a Redis object to connect to Redis server with TLS support.

NOTE: When building your application code, you also need to link it with libhiredis.a, libhiredis_ssl.a, libredis++.a (or the corresponding shared libraries), -lssl and -lcrypto.

Automatically Initialize OpenSSL Library

By default, redis-plus-plus automatically initializes OpenSSL library, i.e. calls SSL_library_init and initializes locks if needed. However, your application code might already initialize OpenSSL library. In this case, you can call tls::disable_auto_init() to disable the initialization. You should call this function only once and call it before any other redis-plus-plus operation. Otherwise, the behavior is undefined.

Send Command to Redis Server

c++

// Delete a single key.
long long Redis::del(const StringView &key);

// Delete a batch of keys: [first, last).
template <typename Input>
long long Redis::del(Input first, Input last);

// Delete keys in the initializer_list.
template <typename T>
long long Redis::del(std::initializer_list<T> il);

With input parameters, these methods build a Redis command based on Redis protocol, and send the command to Redis server. Then synchronously receive the reply, parse it, and return to the caller.

Let's take a closer look at these methods' parameters and return values.

Parameter Type

Most of these methods have the same parameters as the corresponding commands. The following is a list of parameter types:

Parameter Type	Explaination	Example	Note
StringView	Parameters of string type. Normally used for key, value, member name, field name and so on	bool Redis::hset(const StringView &key, const StringView &field, const StringView &val)	See the StringView section for details on `StringView`
long long	Parameters of integer type. Normally used for index (e.g. list commands) or integer	void ltrim(const StringView &key, long long start, long long stop) long long decrby(const StringView &key, long long decrement)
double	Parameters of floating-point type. Normally used for score (e.g. sorted set commands) or number of floating-point type	double incrbyfloat(const StringView &key, double increment)
std::chrono::duration std::chrono::time_point	Time-related parameters	bool expire(const StringView &key, const std::chrono::seconds &timeout) bool expireat(const StringView &key, const std::chrono::time_point<std::chrono::system_clock, std::chrono::seconds> &tp)
std::pair<StringView, StringView>	Used for Redis hash's (field, value) pair	bool hset(const StringView &key, const std::pair<StringView, StringView> &item)
std::pair<double, double>	Used for Redis geo's (longitude, latitude) pair	OptionalLongLong georadius(const StringView &key, const std::pair<double, double> &location, double radius, GeoUnit unit, const StringView &destination, bool store_dist, long long count)
pair of iterators	Use a pair of iterators to specify a range of input, so that we can pass the data in a STL container to these methods	template < typename Input > long long del(Input first, Input last)	Throw an exception, if it's an empty range, i.e. first == last
std::initializer_list< T >	Use an initializer list to specify a batch of input	template < typename T > long long del(std::initializer_list< T > il)
some options	Options for some commands	UpdateType, template < typename T > class BoundedInterval	See command_options.h for details

StringView

std::string_view is a good choice for read-only string parameter types. std::string_view was however only introduced in the C++ 17 standard, so if you build redis-plus-plus with the -std=c++11 (i.e. by specifying -DREDIS_PLUS_PLUS_CXX_STANDARD=11 with cmake command) or the -std=c++14 standard, a simple implementation of std::string_view, called StringView, is available. You could build redis-plus-plus with the -std=c++17 standard (i.e. the default behavior), which will supply std::string_view natively. The StringView implementation will then be disregarded by aliasing it to std::string_view. This is done inside the redis-plus-plus library with: using StringView = std::string_view.

Since there are conversions from std::string and c-style string to StringView, you can just pass std::string or c-style string to methods that need a StringView parameter.

c++

// bool Redis::hset(const StringView &key, const StringView &field, const StringView &val)

// Pass c-style string to StringView.
redis.hset("key", "field", "value");

// Pass std::string to StringView.
std::string key = "key";
std::string field = "field";
std::string val = "val";
redis.hset(key, field, val);

// Mix std::string and c-style string.
redis.hset(key, field, "value");

Return Type

Redis protocol defines 5 kinds of replies:

Status Reply: Also known as Simple String Reply. It's a non-binary string reply.
Bulk String Reply: Binary safe string reply.
Integer Reply: Signed integer reply. Large enough to hold long long.
Array Reply: (Nested) Array reply.
Error Reply: Non-binary string reply that gives error info.

Also these replies might be NULL. For instance, when you try to GET the value of a nonexistent key, Redis returns a NULL Bulk String Reply.

As we mentioned above, replies are parsed into return values of these methods. The following is a list of return types:

Return Type	Explaination	Example	Note
void	Status Reply that should always return a string of "OK"	RENAME, SETEX
std::string	Status Reply that NOT always return "OK", and Bulk String Reply	PING, INFO
bool	Integer Reply that always returns 0 or 1	EXPIRE, HSET	See the Boolean Return Value section for the meaning of a boolean return value
long long	Integer Reply that not always return 0 or 1	DEL, APPEND
double	Bulk String Reply that represents a double	INCRBYFLOAT, ZINCRBY
std::pair	Array Reply with exactly 2 elements. Since the return value is always an array of 2 elements, we return the 2 elements as a `std::pair`'s first and second elements	BLPOP
std::tuple	Array Reply with fixed length and has more than 2 elements. Since length of the returned array is fixed, we return the array as a `std::tuple`	BZPOPMAX
output iterator	General Array Reply with non-fixed/dynamic length. We use STL-like interface to return this kind of array replies, so that you can insert the return value into a STL container easily	MGET, LRANGE	Also, sometimes the type of output iterator decides which options to send with the command. See the Examples section for details
Optional< T >	For any reply of type `T` that might be NULL	GET, LPOP, BLPOP, BZPOPMAX	See the Optional section for details on `Optional<T>`
Variant< Args... >	For reply that might be of serval different types	MEMORY STATS	NOTE: so far, this type is only supported when compiling redis-plus-plus with C++ 17 standard. This is normally used with generic command interface. See the Variant section for details on `Variant<Args...>`
STL container	General Array Reply	CONFIG GET	Both output iterator and STL container are used for array reply. The difference is that STL container is normally used with generic command interface. See the STL container section for example

Boolean Return Value

Optional

std::optional is a good option for return type, if Redis might return NULL REPLY. However, std::optional is introduced in C++ 17 standard, and if you build redis-plus-plus with -std=c++11 standard (i.e. by specifying -DREDIS_PLUS_PLUS_CXX_STANDARD=11 with cmake command), we implement our own simple version, i.e. template Optional<T>. Instead, if you build redis-plus-plus with -std=c++17 standard (i.e. the default behavior), you can use std::optional, and we have an alias for it: template <typename T> using Optional = std::optional<T>.

Take the GET and MGET commands for example:

c++

// Or just: auto val = redis.get("key");
Optional<std::string> val = redis.get("key");

// Optional<T> has a conversion to bool.
// If it's NOT a null Optional<T> object, it's converted to true.
// Otherwise, it's converted to false.
if (val) {
    // Key exists. Dereference val to get the string result.
    std::cout << *val << std::endl;
} else {
    // Redis server returns a NULL Bulk String Reply.
    // It's invalid to dereference a null Optional<T> object.
    std::cout << "key doesn't exist." << std::endl;
}

std::vector<Optional<std::string>> values;
redis.mget({"key1", "key2", "key3"}, std::back_inserter(values));
for (const auto &val : values) {
    if (val) {
        // Key exist, process the value.
    }
}

We also have some typedefs for some commonly used Optional<T>:

c++

using OptionalString = Optional<std::string>;

using OptionalLongLong = Optional<long long>;

using OptionalDouble = Optional<double>;

using OptionalStringPair = Optional<std::pair<std::string, std::string>>;

Variant

std::variant is a good option for return type, if the reply might be of different types. For example, the MEMORY STATS command returns an array reply, which is, in fact, a map of key-value pairs of configurations:

text

127.0.0.1:6379> memory stats
 1) "peak.allocated"
 2) (integer) 4471104
 ...
17) "db.0"
18) 1) "overhead.hashtable.main"
    2) (integer) 104
    3) "overhead.hashtable.expires"
    4) (integer) 32
...
27) "dataset.percentage"
28) "9.70208740234375"
...

However, as you can see, the value part of the result might be of type long long (key: peak.allocated), double (key: dataset.percentage) or even a map (key: db.0). So you cannot simply parse the result into a std::unordered_map<std::string, long long> or std::unordered_map<std::string, double>. A workaround is to parse the result into a tuple, however, this tuple solution is ugly and error-prone. Check this issue for detail.

In this case, Variant, which is a typedef of std::variant if you build redis-plus-plus with C++17 standard, is very helpful. You can parse the result into a std::unordered_map<std::string, Variant<double, long long, std::unordered_map<std::string, long long>>>.

text

using Var = Variant<double, long long, std::unordered_map<std::string, long long>>;
auto r = Redis("tcp://127.0.0.1");
auto v = r.command<std::unordered_map<std::string, Var>>("memory", "stats");

There're some limitations on Variant support:

The type arguments of Variant, cannot have duplicate items, e.g. Variant<double, long long, double> won't work.
double must be placed before std::string. Because double reply is, in fact, string reply, and when parsing variant, we try to parse the reply into the first matched type, specified with the type arguments from left to right. So if double is placed after std::string, i.e. on the right side of std::string, the reply will always be parsed into std::string.

Also check the generic command section for more examples on generic command interface.

STL container

When using generic command interface, instead of parsing the reply to output iterator, you can also parse it into a STL container.

text

auto r = Redis("tcp://127.0.0.1");
auto v = r.command<std::unordered_map<std::string, std::string>>("config", "get", "*");

Also check the generic command section for more examples on generic command interface.

Examples

Let's see some examples on how to send commands to Redis server.

Various Parameter Types

text

/* Detailed source-code truncated for AI context efficiency. */

Various Return Types

text

/* Detailed source-code truncated for AI context efficiency. */

SCAN Commands

c++

auto cursor = 0LL;
auto pattern = "*pattern*";
auto count = 5;
std::unordered_set<std::string> keys;
while (true) {
    cursor = redis.scan(cursor, pattern, count, std::inserter(keys, keys.begin()));
    // Default pattern is "*", and default count is 10
    // cursor = redis.scan(cursor, std::inserter(keys, keys.begin()));

    if (cursor == 0) {
        break;
    }
}

Command Overloads

Sometimes the type of output iterator decides which options to send with the command.

text

/* Detailed source-code truncated for AI context efficiency. */

Please see redis.h for doxygen style API references and examples, and see the tests for other examples.

Exception

Error: Generic error. It's derived from std::exception, and it's also the base class of other exceptions.
IoError: There's some IO error with the connection.
TimeoutError: Read or write operation was timed out. It's a derived class of IoError.
ClosedError: Redis server closed the connection.
ProtoError: The command or reply is invalid, and we cannot process it with Redis protocol.
OomError: hiredis library got an out-of-memory error.
ReplyError: Redis server returned an error reply, e.g. we try to call redis::lrange on a Redis hash.
WatchError: Watched key has been modified. See Watch section for details.

NOTE: NULL REPLY is not taken as an exception. For example, if we try to GET a non-existent key, we'll get a NULL Bulk String Reply. Instead of throwing an exception, we return the NULL REPLY as a null Optional<T> object. Also see Optional section.

Normally, when exception happens, you don't need to create a Redis object. It's exception safe, and you can reuse the Redis object. Even if the connection to Redis server is broken, and it throws some exception, say, IoError. The next time when you send command with the Redis object, it will try to reconnect to Redis server automatically. This rule also applies to RedisCluster. However, if Pipeline, Transcation and Subscriber throws exception, you need to destroy the object, and create a new one. See the corresponding documentation for details.

Examples

The following is an example on how to catch these exceptions:

text

try {
    redis.set("key", "value");

    // Wrong type error
    redis.lpush("key", {"a", "b", "c"});
} catch (const ReplyError &err) {
    // WRONGTYPE Operation against a key holding the wrong kind of value
    cout << err.what() << endl;
} catch (const TimeoutError &err) {
    // reading or writing timeout
} catch (const ClosedError &err) {
    // the connection has been closed.
} catch (const IoError &err) {
    // there's an IO error on the connection.
} catch (const Error &err) {
   // other errors
}

Generic Command Interface

text

/* Detailed source-code truncated for AI context efficiency. */

NOTE: The name of some Redis commands is composed with two strings, e.g. CLIENT SETNAME. In this case, you need to pass these two strings as two arguments for Redis::command.

c++

// This is GOOD.
redis.command<void>("client", "setname", "name");

// This is BAD, and will fail to send command to Redis server.
// redis.command<void>("client setname", "name");

Let's rewrite the above examples:

text

/* Detailed source-code truncated for AI context efficiency. */

In fact, there's one more Redis::command method:

c++

template <typename Cmd, typename ...Args>
auto command(Cmd cmd, Args &&...args)
    -> typename std::enable_if<!std::is_convertible<Cmd, StringView>::value, ReplyUPtr>::type;

However, this method exposes some implementation details, and is only for internal use. You should NOT use this method.

Thread Safety

Subscriber is NOT thread-safe. If you want to call its member functions in multi-thread environment, you need to synchronize between threads manually.

Exception

If any of the Subscriber's method throws an exception other than ReplyError or TimeoutError, you CANNOT use it any more. Instead, you have to destroy the Subscriber object, and create a new one.

Subscriber Callbacks

There are 6 kinds of messages:

MESSAGE: message sent to a channel.
PMESSAGE: message sent to channels of a given pattern.
SUBSCRIBE: message sent when we successfully subscribe to a channel.
UNSUBSCRIBE: message sent when we successfully unsubscribe to a channel.
PSUBSCRIBE: message sent when we successfully subscribe to a channel pattern.
PUNSUBSCRIBE: message sent when we successfully unsubscribe to a channel pattern.

We call messages of SUBSCRIBE, UNSUBSCRIBE, PSUBSCRIBE and PUNSUBSCRIBE types as META MESSAGEs.

In order to process these messages, you can set callback functions on Subscriber:

Subscriber::on_message(MsgCallback): set callback function for messages of MESSAGE type, and the callback interface is: void (std::string channel, std::string msg).
Subscriber::on_pmessage(PatternMsgCallback): set the callback function for messages of PMESSAGE type, and the callback interface is: void (std::string pattern, std::string channel, std::string msg).
Subscriber::on_meta(MetaCallback): set callback function for messages of META MESSAGE type, and the callback interface is: void (Subscriber::MsgType type, OptionalString channel, long long num). type is an enum, it can be one of the following enum: Subscriber::MsgType::SUBSCRIBE, Subscriber::MsgType::UNSUBSCRIBE, Subscriber::MsgType::PSUBSCRIBE, Subscriber::MsgType::PUNSUBSCRIBE, Subscriber::MsgType::MESSAGE, and Subscriber::MsgType::PMESSAGE. If you haven't subscribe/psubscribe to any channel/pattern, and try to unsubscribe/punsubscribe without any parameter, i.e. unsubscribe/punsubscribe all channels/patterns, channel will be null. So the second parameter of meta callback is of type OptionalString.

All these callback interfaces pass std::string by value, and you can take their ownership (i.e. std::move) safely.

Consume Messages

You can call Subscriber::consume to consume messages published to channels/patterns that the Subscriber has been subscribed.

After receiving the message, Subscriber::consume calls the callback function to process the message based on message type. However, if you don't set callback for a specific kind of message, Subscriber::consume will consume the received message and discard it, i.e. Subscriber::consume returns without running the callback.

Examples

The following example is a common pattern for using Subscriber:

c++

// Create a Subscriber.
auto sub = redis.subscriber();

// Set callback functions.
sub.on_message([](std::string channel, std::string msg) {
            // Process message of MESSAGE type.
        });

sub.on_pmessage([](std::string pattern, std::string channel, std::string msg) {
            // Process message of PMESSAGE type.
        });

sub.on_meta([](Subscriber::MsgType type, OptionalString channel, long long num) {
            // Process message of META type.
        });

// Subscribe to channels and patterns.
sub.subscribe("channel1");
sub.subscribe({"channel2", "channel3"});

sub.psubscribe("pattern1*");

// Consume messages in a loop.
while (true) {
    try {
        sub.consume();
    } catch (const Error &err) {
        // Handle exceptions.
    }
}

If ConnectionOptions::socket_timeout is set, you might get TimeoutError exception before receiving a message:

c++

while (true) {
    try {
        sub.consume();
    } catch (const TimeoutError &e) {
        // Try again.
        continue;
    } catch (const Error &err) {
        // Handle other exceptions.
    }
}

The above examples use lambda as callback. If you're not familiar with lambda, you can also set a free function as callback. Check this issue for detail.

Pipeline

Pipeline is used to reduce RTT (Round Trip Time), and speed up Redis queries. redis-plus-plus supports pipeline with the Pipeline class.

Create Pipeline

You can create a pipeline with Redis::pipeline method, which returns a Pipeline object.

c++

ConnectionOptions connection_options;
ConnectionPoolOptions pool_options;

Redis redis(connection_options, pool_options);

auto pipe = redis.pipeline();

When creating a Pipeline object, by default, Redis::pipeline method creates a new connection to Redis server. This connection is NOT picked from the connection pool, but a newly created connection. This connection has the same ConnectionOptions as other connections in the connection pool. Pipeline object maintains the new connection, and all piped commands are sent through this connection.

NOTE: By default, creating a Pipeline object is NOT cheap, since it creates a new connection. So you'd better reuse the Pipeline object as much as possible. Check this to see how to create a Pipeline object without creating a new connection.

Send Commands

c++

pipe.set("key", "val").incr("num").rpush("list", {0, 1, 2}).command("hset", "key", "field", "value");

Get Replies

Once you finish sending commands to Redis, you can call Pipeline::exec to get replies of these commands. You can also chain Pipeline::exec with other commands.

c++

pipe.set("key", "val").incr("num");
auto replies = pipe.exec();

// The same as:
replies = pipe.set("key", "val").incr("num).exec();

In fact, these commands won't be sent to Redis, until you call Pipeline::exec. So Pipeline::exec does 2 work in order: send all piped commands, then get all replies from Redis.

Also you can call Pipeline::discard to discard those piped commands.

c++

pipe.set("key", "val").incr("num");

pipe.discard();

Parse Replies

template <typename Result> Result get(std::size_t idx): Return the ith reply as a return value, and you need to specify the return type as tempalte parameter.
template <typename Output> void get(std::size_t idx, Output output): If the reply is of type Array Reply, you can call this method to write the ith reply to an output iterator. Normally, compiler will deduce the type of the output iterator, and you don't need to specify the type parameter explicitly.
redisReply& get(std::size_t idx): If the reply is NOT a fixed type, call this method to get a reference to redisReply object. In this case, you need to call template <typename T> T reply::parse(redisReply &) to parse the reply manually.

Check the Return Type section for details on the return types of the result.

c++

auto replies = pipe.set("key", "val").incr("num").lrange("list", 0, -1).exec();

auto set_cmd_result = replies.get<bool>(0);

auto incr_cmd_result = replies.get<long long>(1);

std::vector<std::string> list_cmd_result;
replies.get(2, std::back_inserter(list_cmd_result));

Exception

If any of Pipeline's method throws an exception other than ReplyError, the Pipeline object enters an invalid state. You CANNOT use it any more, but only destroy the object, and create a new one.

Thread Safety

Pipeline is NOT thread-safe. If you want to call its member functions in multi-thread environment, you need to synchronize between threads manually.

Create Pipeline Without Creating New Connection

YOU MUST CAREFULLY READ ALL WORDS IN THIS SECTION AND THE VERY IMPORTANT NOTES BEFORE USING THIS FEATURE!!!

In fact, you can also create a Pipeline object with a connection from the underlying connection pool, so that calling Redis::pipeline method can be much cheaper (since it doesn't need to create a new connection).

The prototype of Redis::pipeline is as follows: Pipeline pipeline(bool new_connection = true);. If new_connection is false, the Pipeline object will be created with a connection from the underlying pool.

text

ConnectionOptions connection_options;
ConnectionPoolOptions pool_options;

Redis redis(connection_options, pool_options);

// Create a Pipeline without creating a new connection.
auto pipe = redis.pipeline(false);

VERY IMPORTANT NOTES

However, in this case, you MUST be very careful, otherwise, you might get bad performance or even dead lock. Because when you run command with Pipeline object, it will hold the connection until Pipeline::exec, Pipeline::discard or Pipeline's destructor is called (the connection will also be released if any method of Pipeline throws Exception). If the Pipeline object holds the connection for a long time, other Redis methods might not be able to get a connection from the underlying pool.

Check the following dead lock example:

text

// By defaul, create a `Redis` object with only ONE connection in pool.
// Also by default, the `ConnectionPoolOptions::wait_timeout` is 0ms,
// which means if the pool is empty, `Redis` method will be blocked until
// the pool is not empty.
Redis redis("tcp://127.0.0.1");

// Create a `Pipeline` with a connection in the underlying pool.
// In fact, the connection hasn't been fetched from the pool
// until some method of `Pipeline` has been called.
auto pipe = redis.pipeline(false);

// Now the `Pipeline` object fetches a connection from the pool.
pipe.set("key1", "val");

// `Pipeline` object still holds the connection until `Pipeline::exec`,
// `Pipeline::discard` or the destructor is called.
pipe.set("key2", "val");

// Try to send a command with `Redis` object.
// However, the pool is empty, since the `Pipeline` object still holds
// the connection, and this call will be blocked forever.
// DEAD LOCK!!!
redis.get("key");

// NEVER goes here.
pipe.exec();

BEST PRACTICE:

When creating Pipeline without creating a new connection:

Always set ConnectionPoolOptions::wait_timeout larger than 0ms (i.e. when pool is empty, never block forever).
Avoid doing slow operation between Pipeline's methods.
Better chain Pipeline methods and the Pipeline::exec in one statements.
Better leave Pipeline related code in a block scope.

text

ConnectionOptions opts;
opts.host = "127.0.0.1";
opts.port = 6379;
opts.socket_timeout = std::chrono::milliseconds(50);

ConnectionPoolOptions pool_opts;
pool_opts.size = 3;

// Always set `wait_timeout` larger than 0ms.
pool_opts.wait_timeout = std::chrono::milliseconds(50);

auto redis = Redis(opts, pool_opts);

{
    // Better put `Pipeline` related code in a block scope.
    auto pipe = redis.pipeline(false);

    pipe.set("key1", "val");

    // DON'T run slow operations here, since `Pipeline` object still holds
    // the connection, other threads using this `Redis` object, might be blocked.

    pipe.set("key2", "val");

    // When `Pipeline::exec` finishes, `Pipeline` releases the connection, and returns it to pool.
    auto replies = pipe.exec();

    // This is even better, i.e. chain `Pipeline` methods with `Pipeline::exec`.
    replies = pipe.set("key1", "val").set("key2", "val").exec();
}

for (auto i = 0; i < 10; ++i) {
    // This operation, i.e. creating a `Pipeline` object with connection in pool, is cheap
    auto pipe = redis.pipeline(false);

    // Fetch a connection from the underlying pool, and hold it.
    pipe.set("key1", "val").set("key2", "val");

    // Although `Pipeline::exec` and `Pipeline::discard` haven't been called,
    // when `Pipeline`'s destructor is called, the connection will also be
    // returned to the pool.
}

Transaction

Transaction is used to make multiple commands runs atomically.

Create Transaction

You can create a transaction with Redis::transaction method, which returns a Transaction object.

c++

ConnectionOptions connection_options;
ConnectionPoolOptions pool_options;

Redis redis(connection_options, pool_options);

auto tx = redis.transaction();

As the Pipeline class, Transaction maintains a newly created connection to Redis. This connection has the same ConnectionOptions as the Redis object.

NOTE: Creating a Transaction object is NOT cheap, since it creates a new connection. So you'd better reuse the Transaction as much as possible. Check this to see how to create a Transaction object without creating a new connection.

Also you don't need to send MULTI command to Redis. Transaction will do that for you automatically.

Send Commands

Transaction shares most of implementation with Pipeline. It has the same interfaces as Pipeline. You can send commands as what you do with Pipeline object.

c++

tx.set("key", "val").incr("num").lpush("list", {0, 1, 2}).command("hset", "key", "field", "val");

Execute Transaction

c++

auto replies = tx.set("key", "val").incr("num").exec();

tx.set("key", "val").incr("num");

// Discard the transaction.
tx.discard();

Parse Replies

See Pipeline's Parse Replies section for how to parse the replies.

Piped Transaction

c++

// Create a piped transaction
auto tx = redis.transaction(true);

With this piped transaction, all commands are sent to Redis in a pipeline.

Exception

If any of Transaction's method throws an exception other than WatchError or ReplyError, the Transaction object enters an invalid state. You CANNOT use it any more, but only destroy the object and create a new one.

Thread Safety

Transacation is NOT thread-safe. If you want to call its member functions in multi-thread environment, you need to synchronize between threads manually.

Watch

WATCH is used to provide a check-and-set(CAS) behavior to Redis transactions.

text

WATCH key           // watch a key
val = GET key       // get value of the key
new_val = val + 1   // incr the value
MULTI               // begin the transaction
SET key new_val     // set value only if the value is NOT modified by others
EXEC                // try to execute the transaction.
                    // if val has been modified, the transaction won't be executed.

Let's see how to implement the above example with redis-plus-plus:

c++

auto redis = Redis("tcp://127.0.0.1");

// Create a transaction.
auto tx = redis.transaction();

// Create a Redis object from the Transaction object. Both objects share the same connection.
auto r = tx.redis();

// If the watched key has been modified by other clients, the transaction might fail.
// So we need to retry the transaction in a loop.
while (true) {
    try {
        // Watch a key.
        r.watch("key");

        // Get the old value.
        auto val = r.get("key");
        auto num = 0;
        if (val) {
            num = std::stoi(*val);
        } // else use default value, i.e. 0.

        // Incr value.
        ++num;

        // Execute the transaction.
        auto replies = tx.set("key", std::to_string(num)).exec();

        // Transaction has been executed successfully. Check the result and break.

        assert(replies.size() == 1 && replies.get<bool>(0) == true);

        break;
    } catch (const WatchError &err) {
        // Key has been modified by other clients, retry.
        continue;
    } catch (const Error &err) {
        // Something bad happens, and the Transaction object is no longer valid.
        throw;
    }
}

NOTE: in the example above, we create Transaction object outside the while loop, in order to avoid creating new connection again and again.

Create Transaction Without Creating New Connection

NOTE: YOU MUST CAREFULLY READ ALL WORDS AND THE VERY IMPORTANT NOTES LINK IN THIS SECTION BEFORE USING THIS FEATURE!!!

In fact, you can also create a transaction object with a connection from the underlying connection pool, so that calling Redis::transaction method can be much cheaper (since it doesn't need to create a new connection).

The prototype of Redis::transaction is as follows: Transaction transaction(bool piped = false, bool new_connection = true);. If new_connection is false, the Transaction object will be created with a connection from the underlying pool.

text

ConnectionOptions connection_options;
ConnectionPoolOptions pool_options;

Redis redis(connection_options, pool_options);

// Create a Transaction without creating a new connection.
auto tx = redis.transaction(false, false);

However, in this case, you MUST be very careful, otherwise, you might get bad performance or even dead lock. Please carefully check the similar pipeline's VERY IMPORTANT NOTES section, before you use it!

Besides those very important notes, there's another important note for Transaction:

Limit the scope of Redis object created by Transaction::Redis, i.e. destroy it ASAP.

Check the following example:

c++

auto redis = Redis(opts, pool_opts);

// Create a `Transaction` object without creating a new connection.
auto tx = redis.Transaction(false, false);

// Create a `Redis`, and this `Redis` object shares the same connection with the `Transaction` object.
auto r = tx.redis();

// Other code here...

// Execute the transaction.
auto replies = tx.set("key", "val").exec();

// Although `Transaction::exec` has been called, the connection has not been returned to pool.
// Because the `Redis` object, i.e. `r`, still holds the connection.

So the above watch example should be modified as follows:

c++

auto redis = Redis(opts, pool_opts);

// If the watched key has been modified by other clients, the transaction might fail.
// So we need to retry the transaction in a loop.
while (true) {
    try {
        // Create a transaction without creating a new connection.
        auto tx = redis.transaction(false, false);

        // Create a Redis object from the Transaction object. Both objects share the same connection.
        auto r = tx.redis();

        // Watch a key.
        r.watch("key");

        // Get the old value.
        auto val = r.get("key");
        auto num = 0;
        if (val) {
            num = std::stoi(*val);
        } // else use default value, i.e. 0.

        // Incr value.
        ++num;

        // Execute the transaction.
        auto replies = tx.set("key", std::to_string(num)).exec();

        // Transaction has been executed successfully. Check the result and break.

        assert(replies.size() == 1 && replies.get<bool>(0) == true);

        break;
    } catch (const WatchError &err) {
        // Key has been modified by other clients, retry.
        continue;
    } catch (const Error &err) {
        // Something bad happens, and the Transaction object is no longer valid.
        throw;
    }
}

NOTE: The difference is that we create the Transaction object in the while loop (it's cheap, since it doesn't need to create a new connection). When the Transaction object and the Redis object created by Transaction::redis have been destroyed, the connection will be return to pool.

Redis Cluster

redis-plus-plus supports Redis Cluster. You can use RedisCluster class to send commands to Redis Cluster. It has similar interfaces as Redis class.

Connection

By default, RedisCluster connects to all master nodes in the cluster. For each master node, it maintains a connection pool. If you want to read from slave nodes, you need to explicitly set an option (see below for reference).

c++

// Set a master node's host & port.
ConnectionOptions connection_options;
connection_options.host = "127.0.0.1";  // Required.
connection_options.port = 7000; // Optional. The default port is 6379.
connection_options.password = "auth"; // Optional. No password by default.

// Automatically get other nodes' info,
// and connect to every master node with a single connection.
RedisCluster cluster1(connection_options);

ConnectionPoolOptions pool_options;
pool_options.size = 3;

// For each master node, maintains a connection pool of size 3.
RedisCluster cluster2(connection_options, pool_options);

You can also specify connection option with an URI. However, in this way, you can only use default ConnectionPoolOptions, i.e. pool of size 1, and CANNOT specify password.

c++

// Specify a master node's host & port.
RedisCluster cluster3("tcp://127.0.0.1:7000");

// Use default port, i.e. 6379.
RedisCluster cluster4("tcp://127.0.0.1");

Read From Replica

If you want to scale read by reading (possible stale) data from slave nodes, you can specifiy Role::SLAVE as the third parameter of RedisCluster's constructor. In this case, redis-plus-plus will randomly pick a replica node for each master node of the cluster, and create a connection pool for the replica node.

c++

RedisCluster cluster(connection_options, pool_options, Role::SLAVE);

auto val = cluster.get("key");

In this case, you can only send readonly commands to Redis Cluster. If you try to send a write command, e.g. set, hset, redis-plus-plus will throw an exception. Currently, redis-plus-plus doesn't handle this case, i.e. sending write command in Role::SLAVE mode, elegantly, and you might get some performance problem. So, NEVER send write command in Role::SLAVE mode. I'll fix this issue in the future.

NOTE: In Role::SLAVE mode, you don't need to manually send READONLY command to slave nodes. Instead, redis-plus-plus will send READONLY command to slave nodes automatically.

Note

RedisCluster only works with tcp connection. It CANNOT connect to Unix Domain Socket. If you specify Unix Domain Socket in ConnectionOptions, it throws an exception.
All nodes in the cluster should have the same password.
Since Redis Cluster does NOT support multiple databses, ConnectionOptions::db is ignored.

Interfaces

Not support commands without key as argument, e.g. PING, INFO.
Not support Lua script without key parameters.

Since there's no key parameter, RedisCluster has no idea on to which node these commands should be sent. However there're 2 workarounds for this problem:

If you want to send these commands to a specific node, you can create a Redis object with that node's host and port, and use the Redis object to do the work.
Instead of host and port, you can also call Redis RedisCluster::redis(const StringView &hash_tag) to create a Redis object with a hash-tag specifying the node. In this case, the returned Redis object creates a new connection to Redis server. NOTE: the returned Redis object, IS NOT THREAD SAFE!. Also, when using the returned Redis object, if it throws exception, you need to destroy it, and create a new one with the RedisCluster::redis method.

Also you can use the hash tags to send multiple-key commands.

See the example section for details.

Pipeline and Transaction

Instead of specifying the node's IP and port, RedisCluster's pipeline and transaction interfaces allow you to specify the node with a hash tag. RedisCluster will calculate the slot number with the given hash tag, and create a pipeline or transaction with the node holding the slot.

c++

Pipeline RedisCluster::pipeline(const StringView &hash_tag, bool new_connection = true);

Transaction RedisCluster::transaction(const StringView &hash_tag, bool piped = false, bool new_connection = true);

With the created Pipeline or Transaction object, you can send commands with keys located on the same node as the given hash_tag. See Examples section for an example.

NOTE: By default, Pipeline and Transaction will be created with a new connection. In order to avoid creating new connection, you can pass false as the last parameter. However, in this case, you MUST be very careful, otherwise, you might get bad performance or even dead lock. Please carefully check the related pipeline section before using this feature.

Examples

c++

#include <sw/redis++/redis++.h>

using namespace sw::redis;

auto redis_cluster = RedisCluster("tcp://127.0.0.1:7000");

redis_cluster.set("key", "value");
auto val = redis_cluster.get("key");
if (val) {
    std::cout << *val << std::endl;
}

// With hash-tag.
redis_cluster.set("key{tag}1", "val1");
redis_cluster.set("key{tag}2", "val2");
redis_cluster.set("key{tag}3", "val3");
std::vector<OptionalString> hash_tag_res;
redis_cluster.mget({"key{tag}1", "key{tag}2", "key{tag}3"},
        std::back_inserter(hash_tag_res));

redis_cluster.lpush("list", {"1", "2", "3"});
std::vector<std::string> list;
redis_cluster.lrange("list", 0, -1, std::back_inserter(list));

// Pipeline.
auto pipe = redis_cluster.pipeline("counter");
auto replies = pipe.incr("{counter}:1").incr("{counter}:2").exec();

// Transaction.
auto tx = redis_cluster.transaction("key");
replies = tx.incr("key").get("key").exec();

// Create a Redis object with hash-tag.
// It connects to the Redis instance that holds the given key, i.e. hash-tag.
auto r = redis_cluster.redis("hash-tag");

// And send command without key parameter to the server.
r.command("client", "setname", "connection-name");

NOTE: By default, when you use RedisCluster::redis(const StringView &hash_tag, bool new_connection = true) to create a Redis object, instead of picking a connection from the underlying connection pool, it creates a new connection to the corresponding Redis server. So this is NOT a cheap operation, and you should try to reuse this newly created Redis object as much as possible. If you pass false as the second parameter, you can create a Redis object without creating a new connection. However, in this case, you should be very careful, otherwise, you might get bad performance or even dead lock. Please carefully check the related pipeline section before using this feature.

c++

// This is BAD! It's very inefficient.
// NEVER DO IT!!!
// After sending PING command, the newly created Redis object will be destroied.
cluster.redis("key").ping();

// Then it creates a connection to Redis, and closes the connection after sending the command.
cluster.redis("key").command("client", "setname", "hello");

// Instead you should reuse the Redis object.
// This is GOOD!
auto redis = cluster.redis("key");

redis.ping();
redis.command("client", "setname", "hello");

// This is GOOD! Create `Redis` object without creating a new connection. Use it, and destroy it ASAP.
cluster.redis("key", false).ping();

Details

Data migration between exist nodes.
Add new node to the cluster.
Remove node from the cluster.

redis-plus-plus is able to handle both MOVED and ASK redirections, so it's a complete Redis Cluster client.

If master is down, the cluster will promote one of its replicas to be the new master. redis-plus-plus can also handle this case:

When the master is down, redis-plus-plus losts connection to it. In this case, if you try to send commands to this master, redis-plus-plus will try to update slot-node mapping from other nodes. If the mapping remains unchanged, i.e. new master hasn't been elected yet, it fails to send command to Redis Cluster and throws exception.
When the new master has been elected, the slot-node mapping will be updated by the cluster. In this case, if you send commands to the cluster, redis-plus-plus can get an update-to-date mapping, and sends commands to the new master.

Redis Sentinel

Before using Redis Sentinel with redis-plus-plus, ensure that you have read Redis Sentinel's doc.

Sentinel

You can create a std::shared_ptr<Sentinel> object with SentinelOptions.

c++

SentinelOptions sentinel_opts;
sentinel_opts.nodes = {{"127.0.0.1", 9000},
                        {"127.0.0.1", 9001},
                        {"127.0.0.1", 9002}};   // Required. List of Redis Sentinel nodes.

// Optional. Timeout before we successfully connect to Redis Sentinel.
// By default, the timeout is 100ms.
sentinel_opts.connect_timeout = std::chrono::milliseconds(200);

// Optional. Timeout before we successfully send request to or receive response from Redis Sentinel.
// By default, the timeout is 100ms.
sentinel_opts.socket_timeout = std::chrono::milliseconds(200);

auto sentinel = std::make_shared<Sentinel>(sentinel_opts);

SentinelOptions::connect_timeout and SentinelOptions::socket_timeout CANNOT be 0ms, i.e. no timeout and block forever. Otherwise, redis-plus-plus will throw an exception.

See SentinelOptions for more options.

Role

Besides std::shared_ptr<Sentinel> and master name, you also need to specify a role. There are two roles: Role::MASTER, and Role::SLAVE.

Similarly, with Role::SLAVE, redis-plus-plus will always connect to a slave instance. A master might have several slaves, redis-plus-plus will randomly pick one, and connect to it, i.e. all connections in the underlying connection pool, connect to the same slave instance (check this discussion on why redis-plus-plus not connect to all slaves). If the connection is broken, while this slave instance is still an alive slave, redis-plus-plus will reconnect to this slave. However, if this slave instance is down, or it has been promoted to be the master, redis-plus-plus will randomly connect to another slave. If there's no slave alive, it throws an exception.

Create Redis With Sentinel

c++

ConnectionOptions connection_opts;
connection_opts.password = "auth";  // Optional. No password by default.
connection_opts.connect_timeout = std::chrono::milliseconds(100);   // Required.
connection_opts.socket_timeout = std::chrono::milliseconds(100);    // Required.

ConnectionPoolOptions pool_opts;
pool_opts.size = 3; // Optional. The default size is 1.

auto redis = Redis(sentinel, "master_name", Role::MASTER, connection_opts, pool_opts);

After creating the Redis object with sentinel, you can send commands with it, just like an ordinary Redis object.

If you want to write to master, and scale read with slaves. You can use the following pattern:

c++

auto sentinel = std::make_shared<Sentinel>(sentinel_opts);

auto master = Redis(sentinel, "master_name", Role::MASTER, connection_opts, pool_opts);

auto slave = Redis(sentinel, "master_name", Role::SLAVE, connection_opts, pool_opts);

// Write to master.
master.set("key", "value");

// Read from slave.
slave.get("key");

Redis Stream

However, the replies of some streams commands, i.e. XPENDING, XREAD, are complex. So I'll give some examples to show you how to work with these built-in methods.

Examples

text

/* Detailed source-code truncated for AI context efficiency. */

If you have any problem on sending stream commands to Redis, please feel free to let me know.

Redis Modules

Redis Modules enrich Redis. However, redis-plus-plus does not have built-in support/method for these modules, although you can use the generic interface to send commands related to these modules.

Fortunately, @wingunder did a great job to make the work easier. He wrote redis-plus-plus-modules, which is a header only project that has built-in support for some popular modules. If you need to work with Redis Modules, you should have a try.

@wingunder also contributes a lot to redis-plus-plus. Many thanks to @wingunder!

Async Interface

redis-plus-plus also supports async interface, however, async support for Transaction and Subscriber is still on the way.

The async interface depends on third-party event library, and so far, only libuv is supported.

Installation

You must install libuv(e.g. apt-get install libuv1-dev) before install hiredis and redis-plus-plus.

hiredis v1.0.0's async interface is different from older version, and redis-plus-plus only supports hiredis v1.0.0 or later. So you need to ensure you've installed the right version of hiredis before installing redis-plus-plus. Also, you should NEVER install multiple versions of hiredis, otherwise, you'll get some wired problems. If you already installed an older version, remove it, and install a newer version.

When installing redis-plus-plus, you should specify the following command line option: -DREDIS_PLUS_PLUS_BUILD_ASYNC=libuv.

text

cmake -DCMAKE_PREFIX_PATH=/installation/path/to/libuv/and/hiredis -DREDIS_PLUS_PLUS_BUILD_ASYNC=libuv ..

make

make install

Getting Started

The async interface is similar to sync interface, except that you should include sw/redis++/async_redis++.h, and define an object of sw::redis::AsyncRedis, and the related methods return Future object (so far, only std::future and boost::future are supported, support for other implementations of future is on the way).

NOTE: When building your application code, don't forget to link libuv.

text

#include <sw/redis++/async_redis++.h>

ConnectionOptions opts;
opts.host = "127.0.0.1";
opts.port = 6379;

ConnectionPoolOptions pool_opts;
pool_opts.size = 3;

auto async_redis = AsyncRedis(opts, pool_opts);

Future<string> ping_res = async_redis.ping();

Future<bool> set_res = async_redis.set("key", "val");

Future<Optional<string>> get_res = async_redis.get("key");

unordered_map<string, string> m = {{"a", "b"}, {"c", "d"}};
Future<void> hmset_res = async_redis.hmset("hash", m.begin(), m.end());

auto hgetall_res = async_redis.hgetall<vector<string>>("hash");

cout << ping_res.get() << endl;
cout << set_res.get() << endl;
auto val = get_res.get();
if (val)
    cout << *val << endl;
else
    cout << "not exist" << endl;

hmset_res.get();

for (const auto &ele : hgetall_res.get())
    cout << ele << endl;

// Generic interface.

// There's no *AsyncRedis::client_getname* interface.
// But you can use *Redis::command* to get the client name.
auto getname_res = redis.command<OptionalString>("client", "getname");
val = getname_res.get();
if (val) {
    std::cout << *val << std::endl;
}

Redis Sentinel

Aysnc interface also supports Redis Sentinel.

text

#include <sw/redis++/async_redis++.h>

SentinelOptions sentinel_opts;
sentinel_opts.nodes = {
    {"127.0.0.1", 8000},
    {"127.0.0.1", 8001},
    {"127.0.0.1", 8002}
};

sentinel_opts.connect_timeout = std::chrono::milliseconds(100);
sentinel_opts.socket_timeout = std::chrono::milliseconds(100);

auto sentinel = std::make_shared<AsyncSentinel>(sentinel_opts);

onnectionOptions connection_opts;
connection_opts.connect_timeout = std::chrono::milliseconds(100);   // Required.
connection_opts.socket_timeout = std::chrono::milliseconds(100);   // Required.

ConnectionPoolOptions pool_opts;
pool_opts.size = 3; // Optional. The default size is 1.

// Connect to master node.
AsyncRedis redis(sentinel, "mymaster", Role::MASTER, connection_opts, pool_opts);

// The following code randomly connects to one of the slave nodes.
// AsyncRedis redis(sentinel, "mymaster", Role::SLAVE, connection_opts, pool_opts);

redis.set("key", "value");

auto value = redis.get("key").get();

The async support for sentinel is similar with the sync one, except that you need to create an AsyncSentinel object instead of a Sentinel object. Check Redis Sentinel for more details on SentinelOptions, ConnectionOptions and Role.

Redis Cluster

Aysnc interface also supports Redis Cluster. Instead of AsyncRedis, you need to create an AsyncRedisCluster object.

text

ConnectionOptions opts;
opts.host = "127.0.0.1";
opts.port = 6379;

ConnectionPoolOptions pool_opts;
pool_opts.size = 3;

auto async_cluster = AsyncRedisCluster(opts, pool_opts);

Future<bool> set_res = async_cluster.set("key", "val");

Future<Optional<string>> get_res = async_cluster.get("key");

auto mget_res = async_cluster.mget<std::vector<OptionalString>>({"{hashtag}key1", "{hashhag}key2", "{hashtag}key3"});

unordered_map<string, string> m = {{"a", "b"}, {"c", "d"}};
Future<void> hmset_res = async_redis.hmset("hash", m.begin(), m.end());

Event Loop

By default, AsyncRedis and AsyncRedisCluster create a default event loop, and runs the loop in a dedicated thread to handle read and write operations. However, you can also share the underlying event loop with multiple AsyncRedis and AsyncRedisCluster objects. In order to do that, you need to create a std::shared_ptr<EventLoop>, and pass it to the constructors of AsyncRedis and AsyncRedisCluster.

text

auto event_loop = std::make_shared<EventLoop>();

auto redis = AsyncRedis(connection_opts, pool_opts, loop);

auto cluster = AsyncRedisCluster(connection_opts, pool_opts, Role::MASTER, loop);

Future with Continuation

Unfortunately, std::future doesn't support continuation so far, which is inconvenient. However, some other libraries, e.g. boost and folly, have continuation support.

By default, redis-plus-plus returns std::future for async interface. However, you can also make it return boost::future by specifying -DREDIS_PLUS_PLUS_ASYNC_FUTURE=boost when running cmake (folly and other libraries might be supported in the future). Of course, in this case, you need to install boost first.

text

cmake -DREDIS_PLUS_PLUS_BUILD_ASYNC=libuv -DREDIS_PLUS_PLUS_ASYNC_FUTURE=boost ..

NOTE: When building your application code, don't forget to link boost related libs, e.g. -lboost_thread, -lboost_system.

Then you can take advantage of boost::future's continuation support:

text

#include <sw/redis++/async_redis++.h>

ConnectionOptions opts;
opts.host = "127.0.0.1";
opts.port = 6379;
auto redis = AsyncRedis(opts);
auto fut = redis.get("key").then([](sw::redis::Future<sw::redis::Optional<std::string>> fut) {
                                    auto val = fut.get();
                                    if (val) cout << *val << endl;
                                });
// Do other things

// Wait for the continuation finishes.
fut.get();

You can also use a thread pool to run the continuation:

text

#define BOOST_THREAD_PROVIDES_EXECUTORS

// You might also need to `#define BOOST_THREAD_USES_MOVE` with some version of Boost.
// See [this issue](https://github.com/sewenew/redis-plus-plus/issues/272) for detail.

#include <sw/redis++/async_redis++.h>
#include <boost/thread/executors/basic_thread_pool.hpp>

boost::executors::basic_thread_pool pool(3);
auto fut = redis.get("key").then(pool,
        [](sw::redis::Future<sw::redis::Optional<std::string>> fut) {
            auto val = fut.get();
            if (val) cout << *val << endl;
        });

// Do other things

fut.get();

Redis Recipes

We can create many interesting data structures and algorithms based on Redis, such as Redlock. We call these data structures and algorithms as Redis Recipes. redis-plus-plus will support some of these recipes.

NOTE: These recipes will be first implemented on the recipes branch. I'd like to hear your feedback on the API of these recipes, and when these APIs become stable, I'll merge the code into the master branch. So APIs on the recipes branch are NOT stable, and might be changed in the future.

Redlock

Redlock is a distributed lock based on Redis. Thanks to @wingunder's suggestion, redis-plus-plus supports Redlock now. @wingunder and I made two different implementation of Redlock: one based on Lua script, and the other based on transaction. The Lua script version should be faster, and also it has many other parameters to control the behavior. However, if you are not allowed to, or don't want to run Lua scripts inside Redis, you could try using the transaction version.

Examples

text

auto redis1 = Redis("tcp://127.0.0.1:7000");
auto redis2 = Redis("tcp://127.0.0.1:7001");
auto redis3 = Redis("tcp://127.0.0.1:7002");

// Lua script version:
{
    RedLockMutex mtx({redis1, redis2, redis3}, "resource");

    // Not locked.
    RedLock<RedLockMutex> lock(mtx, std::defer_lock);

    // Try to get the lock, and keep 30 seconds.
    // It returns the validity time of the lock, i.e. the lock is only
    // valid in *validity_time*, after that the lock might be acquired by others.
    // If failed to acquire the lock, throw an exception of Error type.
    auto validity_time = lock.try_lock(std::chrono::seconds(30));

    // Extend the lock before the lock expired.
    validity_time = lock.extend_lock(std::chrono::seconds(10));

    // You can unlock explicitly.
    lock.unlock();
} // If unlock() is not called, the lock will be unlocked automatically when it's destroied.

// Transaction version:
{
    RedMutex mtx({redis1, redis2, redis3}, "resource");

    RedLock<RedMutex> lock(mtx, std::defer_lock);
    auto validity_time = lock.try_lock(std::chrono::seconds(30));
    validity_time = lock.extend_lock(std::chrono::seconds(30));

    // You can unlock explicitly.
    lock.unlock();
}

Please refer to the code for detail. I'll enhance the doc in the future.

Author

redis-plus-plus is written by sewenew, who is also active on StackOverflow.

Overview (`ext/x64-salsa2012-asm/README.md`)

Blazingly fast X64 ASM implementation of Salsa20/12

This is ripped from the cnacl source. The actual code is by Danial J. Bernstein and is in the public domain.

This is included on Linux and Mac 64-bit builds and is significantly faster than the SSE intrinsics or C versions. It's used for packet encode/decode only since its use differs a bit from the regular Salsa20 C++ class. Specifically it lacks the ability to be called on multiple blocks, preferring instead to take a key and a single stream to encrypt and that's it.

Overview (`nonfree/controller/README.md`)

Network Controller Microservice

Every ZeroTier virtual network has a network controller responsible for admitting members to the network, issuing certificates, and issuing default configuration information.

This is our reference controller implementation and is almost the same as the one we use to power our own hosted services at my.zerotier.com. The only difference is the database backend used.

Controller data is stored in JSON format under controller.d in the ZeroTier working directory. It can be copied, rsync'd, placed in git, etc. The files under controller.d should not be modified in place while the controller is running or data loss may result, and if they are edited directly take care not to save corrupt JSON since that can also lead to data loss when the controller is restarted. Going through the API is strongly preferred to directly modifying these files.

See the API section below for information about controlling the controller.

Scalability and Reliability

Controllers can in theory host up to 2^24 networks and serve many millions of devices (or more), but we recommend spreading large numbers of networks across many controllers for load balancing and fault tolerance reasons. Since the controller uses the filesystem as its data store we recommend fast filesystems and fast SSD drives for heavily loaded controllers.

Since ZeroTier nodes are mobile and do not need static IPs, implementing high availability fail-over for controllers is easy. Just replicate their working directories from master to backup and have something automatically fire up the backup if the master goes down. Modern orchestration tools like Nomad and Kubernetes can be of help here.

Dockerizing Controllers

ZeroTier network controllers can easily be run in Docker or other container systems. Since containers do not need to actually join networks, extra privilege options like "--device=/dev/net/tun --privileged" are not needed. You'll just need to map the local JSON API port of the running controller and allow it to access the Internet (over UDP/9993 at a minimum) so things can reach and query it.

Upgrading from Older (1.1.14 or earlier) Versions

Older versions of this code used a SQLite database instead of in-filesystem JSON. A migration utility called migrate-sqlite is included here and must be used to migrate this data to the new format. If the controller is started with an old controller.db in its working directory it will terminate after printing an error to stderr. This is done to prevent "surprises" for those running DIY controllers using the old code.

The migration tool is written in nodeJS and can be used like this:

text

cd migrate-sqlite
npm install
node migrate.js

Network Controller API

The controller API is hosted via the same JSON API endpoint that ZeroTier One uses for local control (usually at 127.0.0.1 port 9993). All controller options are routed under the /controller base path.

The controller microservice itself does not implement any fine-grained access control. Access control is via the ZeroTier control interface itself and authtoken.secret. This can be sent as the X-ZT1-Auth HTTP header field or appended to the URL as ?auth=<token>. Take care when doing the latter that request URLs are not being logged.

While networks with any valid ID can be added to the controller's database, it will only actually work to control networks whose first 10 hex digits correspond with the network controller's ZeroTier ID. See section 2.2.1 of the ZeroTier manual.

The controller JSON API is very sensitive about types. Integers must be integers and strings strings, etc. Incorrect types may be ignored, set to default values, or set to undefined values.

Full documentation of the Controller API can be found on our documentation site

Prometheus Metrics

Controller specific metrics are available from the /metrics endpoint.

Metric Name	Type	Description
controller_network_count	Gauge	number of networks the controller is serving
controller_member_count	Gauge	number of network members the controller is serving
controller_network_change_count	Counter	number of times a network configuration is changed
controller_member_change_count	Counter	number of times a network member configuration is changed
controller_member_auth_count	Counter	number of network member auths
controller_member_deauth_count	Counter	number of network member deauths

Overview (`pkg/synology/README.md`)

Package for Synology's DSM 6

Documentation and downloads: docs.zerotier.com/devices/synology

text

./build.sh build

Overview (`ext/libpqxx-7.7.3/config-tests/README.md`)

Configuration tests

Libpqxx comes with support for different build systems: the GNU autotools,
CMake, Visual Studio's "nmake", and raw GNU "make" on Windows.

For several of these build systems, we need to test things like "does this
compiler environment support std::to_chars for floating-point types?"

We test these things by trying to compile a particular snippet of code, and
seeing whether that succeeds.

To avoid duplicating those snippets for multiple build systems, we put them
here. Both the autotools configuration and the CMake configuration can refer to
them that way.

It took a bit of nasty magic to read a C++ source file into m4 and treat it as
a string literal, without macro expansion. There is every chance that I missed
something, so be prepared for tests failing for unexpected reasons! Some C++
syntax may end up having an unforeseen meaning in m4, and screw up the handling
of the code snippet. Re-configure, and read your logs carefully after editing
these snippets.

Overview (`pkg/synology/dsm7-docker/README.md`)

Docker image for Synology's DSM7

Documentation: docs.zerotier.com/devices/synology

Build & Push changes to DockerHub

shell

./build.sh build

Overview (`ext/prometheus-cpp-lite-1.0/3rdparty/http-client-lite/README.md`)

HTTP Client lite: C++ Cross-platform library only from single-file header-only

This is a lite, C++ cross-platform header-only client library for http request based
on csachs/picohttpclient project.

A Lightweight HTTP 1.1 client where to quickly do very simple HTTP requests,
without adding larger dependencies to a project.

License

http client lite is distributed under the MIT License.

Example usage

To see how this can be used see the examples folders.

Example:

c++

#include <jdl/httpclientlite.hpp>
...
using namespace jdl;
...
HTTPResponse response = HTTPClient::request(HTTPClient::GET, URI("http://example.com"));
cout << response.body << endl;
...

Ext Libpqxx 7.7.3 Include Pqxx Doc Getting Started (`ext/libpqxx-7.7.3/include/pqxx/doc/getting-started.md`)

Getting started {#getting-started}

The most basic three types in libpqxx are the connection, the transaction,
and the result.

They fit together as follows:

You connect to the database by creating a pqxx::connection object (see
@ref connections).
You create a transaction object (see @ref transactions) operating on that
connection. You'll usually want the pqxx::work variety.

Once you're done you call the transaction's commit function to make its
work final. If you don't call this, the work will be rolled back when the
transaction object is destroyed.
Until then, use the transaction's exec, query_value, and stream
functions (and variants) to execute SQL statements. You pass the statements
themselves in as simple strings. (See @ref streams for more about data
streaming).
Most of the exec functions return a pqxx::result object, which acts
as a standard container of rows: pqxx::row.

Each row in a result, in turn, acts as a container of fields: pqxx::field.
See @ref accessing-results for more about results, rows, and fields.
Each field's data is stored internally as a text string, in a format defined
by PostgreSQL. You can convert field or row values using their as() and
to() member functions.
After you've closed the transaction, the connection is free to run a next
transaction.

Here's a very basic example. It connects to the default database (you'll
need to have one set up), queries it for a very simple result, converts it to
an int, and prints it out. It also contains some basic error handling.

text

#include 
#include 

int main()
{
  try
  {
    // Connect to the database.  In practice we may have to pass some
    // arguments to say where the database server is, and so on.
    // The constructor parses options exactly like libpq's
    // PQconnectdb/PQconnect, see:
    // https://www.postgresql.org/docs/10/static/libpq-connect.html
    pqxx::connection c;

    // Start a transaction.  In libpqxx, you always work in one.
    pqxx::work w(c);

    // work::exec1() executes a query returning a single row of data.
    // We'll just ask the database to return the number 1 to us.
    pqxx::row r = w.exec1("SELECT 1");

    // Commit your transaction.  If an exception occurred before this
    // point, execution will have left the block, and the transaction will
    // have been destroyed along the way.  In that case, the failed
    // transaction would implicitly abort instead of getting to this point.
    w.commit();

    // Look at the first and only field in the row, parse it as an integer,
    // and print it.
    //
    // "r[0]" returns the first field, which has an "as<...>()" member
    // function template to convert its contents from their string format
    // to a type of your choice.
    std::cout << r[0].as() << std::endl;
  }
  catch (std::exception const &e)
  {
    std::cerr << e.what() << std::endl;
    return 1;
  }
}

This prints the number 1. Notice that you can keep the result object around
after you've closed the transaction or even the connection. There are
situations where you can't do it, but generally it's fine. If you're
interested: you can install your own callbacks for receiving error messages
from the database, and in that case you'll have to keep the connection object
alive. But otherwise, it's nice to be able to "fire and forget" your
connection and deal with the data.

You can also convert an entire row to a series of C++-side types in one go,
using the @c as member function on the row:

text

pqxx::connection c;
pqxx::work w(c);
pqxx::row r = w.exec1("SELECT 1, 2, 'Hello'");
auto [one, two, hello] = r.as();
std::cout << (one + two) << ' ' << std::strlen(hello) << std::endl;

Here's a slightly more complicated example. It takes an argument from the
command line and retrieves a string with that value. The interesting part is
that it uses the escaping-and-quoting function quote to embed this
string value in SQL safely. It also reads the result field's value as a
plain C-style string using its c_str function.

text

#include 
#include <stdexcept>
#include 

int main(int argc, char *argv[])
{
  try
  {
    if (!argv[1]) throw std::runtime_error("Give me a string!");

    pqxx::connection c;
    pqxx::work w(c);

    // work::exec() returns a full result set, which can consist of any
    // number of rows.
    pqxx::result r = w.exec("SELECT " + w.quote(argv[1]));

    // End our transaction here.  We can still use the result afterwards.
    w.commit();

    // Print the first field of the first row.  Read it as a C string,
    // just like std::string::c_str() does.
    std::cout << r[0][0].c_str() << std::endl;
  }
  catch (std::exception const &e)
  {
    std::cerr << e.what() << std::endl;
    return 1;
  }
}

You can find more about converting field values to native types, or
converting values to strings for use with libpqxx, under
@ref stringconversion. More about getting to the rows and fields of a
result is under @ref accessing-results.

If you want to handle exceptions thrown by libpqxx in more detail, for
example to print the SQL contents of a query that failed, see @ref exception.

Getting started {#getting-started}

The most basic three types in libpqxx are the connection, the transaction,
and the result.

They fit together as follows:

You connect to the database by creating a pqxx::connection object (see
@ref connections).
You create a transaction object (see @ref transactions) operating on that
connection. You'll usually want the pqxx::work variety.

Once you're done you call the transaction's commit function to make its
work final. If you don't call this, the work will be rolled back when the
transaction object is destroyed.
Until then, use the transaction's exec, query_value, and stream
functions (and variants) to execute SQL statements. You pass the statements
themselves in as simple strings. (See @ref streams for more about data
streaming).
Most of the exec functions return a pqxx::result object, which acts
as a standard container of rows: pqxx::row.

Each row in a result, in turn, acts as a container of fields: pqxx::field.
See @ref accessing-results for more about results, rows, and fields.
Each field's data is stored internally as a text string, in a format defined
by PostgreSQL. You can convert field or row values using their as() and
to() member functions.
After you've closed the transaction, the connection is free to run a next
transaction.

text

#include 
#include 

int main()
{
  try
  {
    // Connect to the database.  In practice we may have to pass some
    // arguments to say where the database server is, and so on.
    // The constructor parses options exactly like libpq's
    // PQconnectdb/PQconnect, see:
    // https://www.postgresql.org/docs/10/static/libpq-connect.html
    pqxx::connection c;

    // Start a transaction.  In libpqxx, you always work in one.
    pqxx::work w(c);

    // work::exec1() executes a query returning a single row of data.
    // We'll just ask the database to return the number 1 to us.
    pqxx::row r = w.exec1("SELECT 1");

    // Commit your transaction.  If an exception occurred before this
    // point, execution will have left the block, and the transaction will
    // have been destroyed along the way.  In that case, the failed
    // transaction would implicitly abort instead of getting to this point.
    w.commit();

    // Look at the first and only field in the row, parse it as an integer,
    // and print it.
    //
    // "r[0]" returns the first field, which has an "as<...>()" member
    // function template to convert its contents from their string format
    // to a type of your choice.
    std::cout << r[0].as() << std::endl;
  }
  catch (std::exception const &e)
  {
    std::cerr << e.what() << std::endl;
    return 1;
  }
}

You can also convert an entire row to a series of C++-side types in one go,
using the @c as member function on the row:

text

pqxx::connection c;
pqxx::work w(c);
pqxx::row r = w.exec1("SELECT 1, 2, 'Hello'");
auto [one, two, hello] = r.as();
std::cout << (one + two) << ' ' << std::strlen(hello) << std::endl;

text

#include 
#include <stdexcept>
#include 

int main(int argc, char *argv[])
{
  try
  {
    if (!argv[1]) throw std::runtime_error("Give me a string!");

    pqxx::connection c;
    pqxx::work w(c);

    // work::exec() returns a full result set, which can consist of any
    // number of rows.
    pqxx::result r = w.exec("SELECT " + w.quote(argv[1]));

    // End our transaction here.  We can still use the result afterwards.
    w.commit();

    // Print the first field of the first row.  Read it as a C string,
    // just like std::string::c_str() does.
    std::cout << r[0][0].c_str() << std::endl;
  }
  catch (std::exception const &e)
  {
    std::cerr << e.what() << std::endl;
    return 1;
  }
}

If you want to handle exceptions thrown by libpqxx in more detail, for
example to print the SQL contents of a query that failed, see @ref exception.

Getting started {#getting-started}

The most basic three types in libpqxx are the connection, the transaction,
and the result.

They fit together as follows:

You connect to the database by creating a pqxx::connection object (see
@ref connections).
You create a transaction object (see @ref transactions) operating on that
connection. You'll usually want the pqxx::work variety.

Once you're done you call the transaction's commit function to make its
work final. If you don't call this, the work will be rolled back when the
transaction object is destroyed.
Until then, use the transaction's exec, query_value, and stream
functions (and variants) to execute SQL statements. You pass the statements
themselves in as simple strings. (See @ref streams for more about data
streaming).
Most of the exec functions return a pqxx::result object, which acts
as a standard container of rows: pqxx::row.

Each row in a result, in turn, acts as a container of fields: pqxx::field.
See @ref accessing-results for more about results, rows, and fields.
Each field's data is stored internally as a text string, in a format defined
by PostgreSQL. You can convert field or row values using their as() and
to() member functions.
After you've closed the transaction, the connection is free to run a next
transaction.

text

#include 
#include 

int main()
{
  try
  {
    // Connect to the database.  In practice we may have to pass some
    // arguments to say where the database server is, and so on.
    // The constructor parses options exactly like libpq's
    // PQconnectdb/PQconnect, see:
    // https://www.postgresql.org/docs/10/static/libpq-connect.html
    pqxx::connection c;

    // Start a transaction.  In libpqxx, you always work in one.
    pqxx::work w(c);

    // work::exec1() executes a query returning a single row of data.
    // We'll just ask the database to return the number 1 to us.
    pqxx::row r = w.exec1("SELECT 1");

    // Commit your transaction.  If an exception occurred before this
    // point, execution will have left the block, and the transaction will
    // have been destroyed along the way.  In that case, the failed
    // transaction would implicitly abort instead of getting to this point.
    w.commit();

    // Look at the first and only field in the row, parse it as an integer,
    // and print it.
    //
    // "r[0]" returns the first field, which has an "as<...>()" member
    // function template to convert its contents from their string format
    // to a type of your choice.
    std::cout << r[0].as() << std::endl;
  }
  catch (std::exception const &e)
  {
    std::cerr << e.what() << std::endl;
    return 1;
  }
}

You can also convert an entire row to a series of C++-side types in one go,
using the @c as member function on the row:

text

pqxx::connection c;
pqxx::work w(c);
pqxx::row r = w.exec1("SELECT 1, 2, 'Hello'");
auto [one, two, hello] = r.as();
std::cout << (one + two) << ' ' << std::strlen(hello) << std::endl;

text

#include 
#include <stdexcept>
#include 

int main(int argc, char *argv[])
{
  try
  {
    if (!argv[1]) throw std::runtime_error("Give me a string!");

    pqxx::connection c;
    pqxx::work w(c);

    // work::exec() returns a full result set, which can consist of any
    // number of rows.
    pqxx::result r = w.exec("SELECT " + w.quote(argv[1]));

    // End our transaction here.  We can still use the result afterwards.
    w.commit();

    // Print the first field of the first row.  Read it as a C string,
    // just like std::string::c_str() does.
    std::cout << r[0][0].c_str() << std::endl;
  }
  catch (std::exception const &e)
  {
    std::cerr << e.what() << std::endl;
    return 1;
  }
}

If you want to handle exceptions thrown by libpqxx in more detail, for
example to print the SQL contents of a query that failed, see @ref exception.

Doc Zerotier Cli.1 (`doc/zerotier-cli.1.md`)

zerotier-cli(1) -- control local ZeroTier virtual network service

SYNOPSIS

zerotier-cli [-switches] [arguments]

DESCRIPTION

zerotier-cli provides a simple command line interface to the local JSON API of the ZeroTier virtual network endpoint service zerotier-one(8).

By default zerotier-cli must be run as root or with sudo. If you want to allow an unprivileged user to use zerotier-cli to control the system ZeroTier service, you can create a local copy of the ZeroTier service authorization token in the user's home directory:

text

sudo cp /var/lib/zerotier-one/authtoken.secret /home/user/.zeroTierOneAuthToken
chown user /home/user/.zeroTierOneAuthToken
chmod 0600 /home/user/.zeroTierOneAuthToken

(The location of ZeroTier's service home may differ by platform. See zerotier-one(8).)

Note that this gives the user the power to connect or disconnect the system to or from any virtual network, which is a significant permission.

zerotier-cli has several command line arguments that are visible in help output. The two most commonly used are -j for raw JSON output and -D to specify an alternative ZeroTier service working directory. Raw JSON output is easier to parse in scripts and also contains verbose details not present in the tabular output. The -D option specifies where the service's zerotier-one.port and authtoken.secret files are located if the service is not running at the default location for your system.

COMMANDS

help:
Displays zerotier-cli help.
info:
Shows information about this device including its 10-digit ZeroTier address and apparent connection status. Use -j for more verbose output.
listpeers:
This command lists the ZeroTier VL1 (virtual layer 1, the peer to peer network) peers this service knows about and has recently (within the past 30 minutes or so) communicated with. These are not necessarily all the devices on your virtual network(s), and may also include a few devices not on any virtual network you've joined. These are typically either root servers or network controllers.
listnetworks:
This lists the networks your system belongs to and some information about them, such as any ZeroTier-managed IP addresses you have been assigned. (IP addresses assigned manually to ZeroTier interfaces will not be listed here. Use the standard network interface commands to see these.)
join:
To join a network just use join and its 16-digit hex network ID. That's it. Then use listnetworks to see the status. You'll either get a reply from the network controller with a certificate and other info such as IP assignments, or you'll get "access denied." In this case you'll need the administrator of this network to authorize your device by its 10-digit device ID (visible with info) on the network's controller.
leave:
Leaving a network is as easy as joining it. This disconnects from the network and deletes its interface from the system. Note that peers on the network may hang around in listpeers for up to 30 minutes until they time out due to lack of traffic. But if they no longer share a network with you, they can't actually communicate with you in any meaningful way.

EXAMPLES

Join "Earth," ZeroTier's big public party line network:

text

$ sudo zerotier-cli join 8056c2e21c000001
$ sudo zerotier-cli listnetworks
( wait until you get an Earth IP )
$ ping earth.zerotier.net
( you should now be able to ping our Earth test IP )

Leave "Earth":

text

$ sudo zerotier-cli leave 8056c2e21c000001

List VL1 peers:

text

$ sudo zerotier-cli listpeers

COPYRIGHT

zerotier-idtool(1) -- tool for creating and manipulating ZeroTier identities

SYNOPSIS

zerotier-idtool [args]

DESCRIPTION

zerotier-idtool is a command line utility for doing things with ZeroTier identities. A ZeroTier identity consists of a public/private key pair (or just the public if it's only an identity.public) and a 10-digit hexadecimal ZeroTier address derived from the public key by way of a proof of work based hash function.

COMMANDS

When command arguments call for a public or secret (full) identity, the identity can be specified as a path to a file or directly on the command line.

help:
Display help. (Also running with no command does this.)
generate [secret file] [public file] [vanity]:
Generate a new ZeroTier identity. If a secret file is specified, the full identity including the private key will be written to this file. If the public file is specified, the public portion will be written there. If no file paths are specified the full secret identity is output to STDOUT. The vanity prefix is a series of hexadecimal digits that the generated identity's address should start with. Typically this isn't used, and if it's specified generation can take a very long time due to the intrinsic cost of generating identities with their proof of work function. Generating an identity with a known 16-bit (4 digit) prefix on a 2.8ghz Core i5 (using one core) takes an average of two hours.
validate :
Locally validate an identity's key and proof of work function correspondence.
getpublic :
Extract the public portion of an identity.secret and print to STDOUT.
sign :
Sign a file's contents with SHA512+ECC-256 (ed25519). The signature is output in hex to STDOUT.
verify :
Verify a signature created with sign.
mkcom [id,value,maxdelta] [...]:
Create and sign a network membership certificate. This is not generally useful since network controllers do this automatically and is included mostly for testing purposes.

EXAMPLES

Generate and dump a new identity:

text

$ zerotier-idtool generate

Generate and write a new identity, both secret and public parts:

text

$ zerotier-idtool generate identity.secret identity.public

Generate a vanity address that begins with the hex digits "beef" (this will take a while!):

text

$ zerotier-idtool generate beef.secret beef.public beef

Sign a file with an identity's secret key:

text

$ zerotier-idtool sign identity.secret last_will_and_testament.txt

Verify a file's signature with a public key:

text

$ zerotier-idtool verify identity.public last_will_and_testament.txt

COPYRIGHT

zerotier-one(8) -- ZeroTier virtual network endpoint service

SYNOPSIS

zerotier-one [-switches] [working directory]

DESCRIPTION

zerotier-one is the service/daemon responsible for connecting a Unix (Linux/BSD/OSX) system to one or more ZeroTier virtual networks and presenting those networks to the system as virtual network ports. You can think of it as a peer to peer VPN client.

It's typically run by init systems like systemd (Linux) or launchd (Mac) rather than directly by the user, and it must be run as root unless you give it the -U switch and don't plan on actually joining networks (e.g. to run a network controller microservice only).

The zerotier-one service keeps its state and other files in a working directory. If this directory is not specified at launch it defaults to "/var/lib/zerotier-one" on Linux, "/Library/Application Support/ZeroTier/One" on Mac, and "/var/db/zerotier-one" on FreeBSD and other similar BSDs. The working directory should persist. It shouldn't be automatically cleaned by system cleanup daemons or stored in a volatile location. Loss of its identity.secret file results in loss of this system's unique 10-digit ZeroTier address and key.

Multiple instances of zerotier-one can be run on the same system as long as they are run with different primary ports (see switches) and a different working directory. But since a single service can join any number of networks, typically there's no point in doing this.

The zerotier-one service is controlled via a JSON API available at 127.0.0.1: with the default primary port being 9993. Access to this API requires an authorization token normally found in the authtoken.secret file in the service's working directory. On some platforms access may be guarded by other measures such as socket peer UID/GID lookup if additional security options are enabled (this is not the default).

The first time the service is started in a fresh working directory, it generates a ZeroTier identity. On slow systems this process can take ten seconds or more due to an anti-DDOS/anti-counterfeit proof of work function used by ZeroTier in address generation. This only happens once, and once generated the result is saved in identity.secret in the working directory. This file represents and defines/claims your ZeroTier address and associated ECC-256 key pair.

SWITCHES

-h:
Display help.
-v:
Display ZeroTier One version.
-U:
Skip privilege check and allow to be run by non-privileged user. This is typically used when zerotier-one is built with the network controller option included. In this case the ZeroTier service might only be acting as a network controller and might never actually join networks, in which case it does not require elevated system permissions.
-p:
Specify a different primary port. If this is not given the default is 9993. If zero is given a random port is chosen each time.
-d:
Fork and run as a daemon.
-i:
Invoke the zerotier-idtool personality, in which case the binary behaves like zerotier-idtool(1). This happens automatically if the name of the binary (or a symlink to it) is zerotier-idtool.
-q:
Invoke the zerotier-cli personality, in which case the binary behaves like zerotier-cli(1). This happens automatically if the name of the binary (or a symlink to it) is zerotier-cli.

EXAMPLES

Run as daemon with OS default working directory and default port:

text

$ sudo zerotier-one -d

Run as daemon with a different working directory and port:

text

$ sudo zerotier-one -d -p12345 /tmp/zerotier-working-directory-test

FILES

These are found in the service's working directory.

identity.public:
The public portion of your ZeroTier identity, which is your 10-digit hex address and the associated public key.
identity.secret:
Your full ZeroTier identity including its private key. This file identifies the system on the network, which means you can move a ZeroTier address around by copying this file and you should back up this file if you want to save your system's static ZeroTier address. This file must be protected, since theft of its secret key will allow anyone to impersonate your device on any network and decrypt traffic. For network controllers this file is particularly sensitive since it constitutes the private key for a certificate authority for the controller's networks.
authtoken.secret:
The secret token used to authenticate requests to the service's local JSON API. If it does not exist it is generated from a secure random source on service start. To use, send it in the "X-ZT1-Auth" header with HTTP requests to 127.0.0.1:.
devicemap:
Remembers mappings of zt# interface numbers to ZeroTier networks so they'll persist across restarts. On some systems that support longer interface names that can encode the network ID (such as FreeBSD) this file may not be present.
zerotier-one.pid:
ZeroTier's PID. This file is deleted on normal shutdown.
zerotier-one.port:
ZeroTier's primary port, which is also where its JSON API is found at 127.0.0.1:. This file is created on startup and is read by zerotier-cli(1) to determine where it should find the control API.
controller.db:
If the ZeroTier One service is built with the network controller enabled, this file contains the controller's SQLite3 database.
controller.db.backup:
If the ZeroTier One service is built with the network controller enabled, it periodically backs up its controller.db database in this file (currently every 5 minutes if there have been changes). Since this file is not a currently in use SQLite3 database it's safer to back up without corruption. On new backups the file is rotated out rather than being rewritten in place.
iddb.d/ (directory):
Caches the public identity of every peer ZeroTier has spoken with in the last 60 days. This directory and its contents can be deleted, but this may result in slower connection initiations since it will require that we go out and re-fetch full identities for peers we're speaking to.
networks.d (directory):
This caches network configurations and certificate information for networks you belong to. ZeroTier scans this directory for .conf files on startup to recall its networks, so "touch"ing an empty .conf file in this directory is a way of pre-configuring ZeroTier to join a specific network on startup without using the API. If the config file is empty ZeroTIer will just fetch it from the network's controller.

COPYRIGHT

libpqxx

.. toctree::
:maxdepth: 2
:caption: Contents:

Indices and tables

:ref:genindex
:ref:modindex
:ref:search

Attic Historic Anode Docs Anode Protocol (`attic/historic/anode/docs/anode_protocol.txt`)

Anode Protocol Specification Draft
Version 0.8

Table of Contents

Introduction

Anode provides three components that work together to provide a global,
secure, and mobile addressing system for computer networks:

An addressing system based on public key cryptography enabling network
devices or applications to assign themselves secure, unique, and globally
reachable network addresses in a flat address space.
A system enabling network participants holding global addresses to locate
one another on local or global networks with "zero configuration."
A communications protocol for communication between addressed network
participants that requires no special operating system support and no
changes to existing network infrastructure.

Using Anode, both fixed and mobile applications and devices can communicate
directly as if they were all connected to the same VPN. Anode restores the
original vision of the Internet as a "flat" network where anything can talk
to anything, and adds the added benefits of address mobility and strong
protection against address spoofing and other protocol level attacks.

1.1. Design Philosophy

Anode's design philosophy is the classical "KISS" principle: "Keep It Simple
Stupid." Anode's design principles are:

#1: Do not try to solve too many problems at once, and stay in scope.

Anode does not attempt to solve too many problems at once. It attempts to
solve the problems of mobile addressing, address portability, and "flat"
addressing in the presence of NAT or other barriers.

It does not attempt to duplicate the full functionality of SSL, X.509, SSH,
XMPP, an enterprise service bus, a pub/sub architecture, BitTorrent, etc. All
of those protocols and services can be used over Anode if their functionality
is desired.

#2: Avoid state management.

State multiplies the complexity and failure modes of network protocols. State
also tends to get in the way of the achievement of new features implicitly
(see principle #4). Avoid state whenever possible.

#3: Avoid algorithm and dependency bloat.

Anode uses only elliptic curve Diffie-Hellman (EC-DH) and AES-256. No other
cryptographic algorithms or hash functions are presently necessary. This
yields implementations compact enough for embedded devices.

Anode also requires few or no dependencies, depending on whether the two
needed cryptographic algorithms are obtained through a library or included.
No other protocols or libraries are required in an implementation.

#4: Achieve features implicitly.

Use a simple stateless design that allows features to be achieved implicitly
rather than specified explicitly. For example, Anode can do multi-homing and
could be used to build a mesh network, but neither of these features is
explicitly specified.

Core Concepts and Algorithms

This section describes addresses, zones, common algorithms, and other core
concepts.

2.1. Zones

A zone is a 32-bit integer encoded into every Anode address. Zones serve to
assist in the location of peers by address on global IP networks. They are
not presently significant for local communications, though they could be
used to partition addresses into groups or link them with configuration
options.

Each zone has a corresponding zone file which can be fetched in a number of
ways (see below). A zone file is a flat text format dictionary of the format
"key=value" separated by carriage returns. Line feeds are ignored, and any
character may be escaped with a backslash () character. Blank lines are
ignored.

The following entries must appear in a zone file:

n=
d=
c=<zone contact, e-mail address of zone administrator>
r=<zone revision, monotonically increasing integer with each edit>
ttl=

Additional fields may appear as well, including fields specific to special
applications or protocols supported within the zone. Some of these are
defined in this document.

Zone file fetching mechanisms are described below. Multiple mechanisms are
specified to enable fallback in the event that one mechanism is not available.

2.1.1. Zone File Retrieval

Zone files are retrieved via HTTP, with the HTTP address being formed in one
of two ways.

The preferred DNS method:

To fetch a zone file via DNS, use the zone ID to generate a host name and URI
of the form:

http://a--XXXXXXXX.net/z

The XXXXXXXX field is the zone ID in hexadecimal.

The fallback IP method:

For fallback in the absence of DNS, the zone ID can be used directly as an
IPv4 or IPv4-mapped-to-IPv6 IP address. A URI is generated of the form:

http://ip_address/z

Support for this method requires that a zone ID be chosen to correspond to a
permanent IPv4 (preferably mappable to IPv6 space as well) IP address.

2.1.2. Zone ID Reservation

By convention, a zone ID is considered reserved when a domain of the form
"a--XXXXXXXX.net" (where XXXXXXXX is the ID in hex) is registered.

It is recommended that this be done even for zone IDs not used for global
address location in order to globally reserve them.

2.2. Addresses

Anode addresses are binary strings containing a 32-bit zone ID, a public key,
and possibly other fields. Only one address type is presently defined:

|---------------------------------------------------------------------------|

| Name | Type ID | Elliptic Curve Parameters | Total Length |
|---------------------------------------------------------------------------|

| ANODE-256-40 | 1 | NIST-P-256 | 40 |
|---------------------------------------------------------------------------|

|---------------------------------------------------------------------------|

| Name | Binary Layout |
|---------------------------------------------------------------------------|

| ANODE-256-40 | <type[1]><zone[4]> |
|---------------------------------------------------------------------------|

The public key is a "compressed" form elliptic curve public key as described
in RFC5480.

The unused section of the address must be zero. These bytes are reserved for
future use.

2.2.1. ASCII Format For Addresses

Addresses are encoded in ASCII using base-32, which provides a quotable and
printable encoding that is of manageable length and is case-insensitive. For
example, an ANODE-256-40 address is 64 characters long in base-32 encoding.

2.3. Relaying

An Anode peer may optionally relay packets to any other reachable peer.
Relaying is accomplished by sending a packet to a peer with the recipient set
to the final recipient. The receiving peer will, if relaying is allowed and if
it knows of or can reach the recipient, forward the packet.

No error is returned if relaying fails, so relay paths are treated as possible
paths for communication until a return is received in the same way as direct
paths.

Relaying can be used by peers to send messages indirectly, locate one
another, and determine network location information to facilitate the
establishment of direct communications.

Peers may refuse to relay or may limit the transmission rate at which packets
can be relayed.

2.3.1. Zone Relays

If a zone's addresses are globally reachable on global IP networks, it must
have one or more zone relays. These must have globally reachable public
static IP addresses.

Zone relays are specified in the zone file in the following format:

zr.

=[,]:::

The address checksum is the sum of the bytes in the Anode address modulus
the number of "zr" entries, in hexadecimal. For example, if a zone had four
global relays its zone file could contain the lines:

zr.0=1.2.3.4:4343:4344:klj4j3...
zr.1=2.3.4.5:4343:4344:00194j...
zr.2=3.4.5.6:4343:4344:1j42zz...
zr.3=4.5.6.7:4343:4344:z94j1q...

The relay would be chosen by taking the sum of the bytes in the address
modulo 4. For example, if the bytes of an address sum to 5081 then relay
zr.1 would be used to communicate with that address.

If more than one IP address is listed for a given relay, the peer must choose
at random from among the addresses of the desired type (IPv4 or IPv6).

Each relay must have one Anode address for every address type supported within
the zone. (At present there is only one address type defined.)

Peers should prefer UDP and fall back to TCP only if UDP is not available.

To make itself available, a peer must make itself known to its designated zone
relay. This is accomplished by sending a PING message.

2.4. Key Agreement and Derivation

Key agreement is performed using elliptic curve Diffie-Hellman. This yields
a raw key whose size depends on the elliptic curve parameters in use.

The following algorithm is used to derive a key of any length from a raw
key generated through key agreement:

Zero the derived key buffer.
Determine the largest of the original raw key or the derived key.
Loop from 0 to the largest length determined in step 2, XOR each byte of
the derived key buffer with the corresponding byte of the original key
buffer with each index being modulus the length of the respective buffer.

2.5. Message Authentication

For message authentication, CMAC-AES (with AES-256) is used. This is also
known in some literature as OMAC1-AES. The key is derived from key agreement
between the key pair of the sending peer and the address of the recipient.

2.6. AES-DIGEST

To maintain cryptographic algorithm frugality, a cryptographic hash function
is constructed from the AES-256 cipher. This hash function uses the common
Davis-Meyer construction with Merkle-Damgård length padding.

It is described by the following pseudocode:

byte previous_digest[16]
byte digest[16] = { 0,0,... }
byte block[32] = { 0,0,... }
integer block_counter = 0

; digest message
for each byte b of message
block[block_counter] = b
block_counter = block_counter + 1
if block_counter == 32 then
block_counter = 0
save digest[] in previous_digest[]
encrypt digest[] with aes-256 using block[] as 256-bit aes-256 key
xor digest[] with previous_digest[]
end if
next

; append end marker, do final block
block[block_counter] = 0x80
block_counter = block_counter + 1
zero rest of block[] from block_counter to 15
save digest[] in previous_digest[]
encrypt digest[] with aes-256 using block[] as 256-bit aes-256 key
xor digest[] with previous_digest[]

; Merkle-Damgård length padding
zero first 8 bytes of block[]
fill last 8 bytes of block[] w/64-bit length in big-endian order
save digest[] in previous_digest[]
encrypt digest[] with aes-256 using block[] as 256-bit aes-128 key
xor digest[] with previous_digest[]

; digest[] now contains 128-bit message digest

2.7. Short Address Identifiers (Address IDs)

A short 8-byte version of the Anode address is used in the protocol to reduce
transmission overhead when both sides are already aware of the other's full
address.

The short address identifier is formed by computing the AES-DIGEST of the
full address and then XORing the first 8 bytes of the digest with the last
8 bytes to yield an 8-byte shortened digest.

2.8. DNS Resolution of Anode Addresses

Anode addresses can be saved in DNS TXT records in the following format:

anode:

This permits Anode addresses to be resolved from normal DNS host name.

2.9. Packet Transmission Mechanisms

2.9.1. UDP Transmission

The recommended method of sending Anode packets is UDP. Each packet is simply
sent as a UDP packet.

2.9.2. TCP Transmission

To send packets over TCP, each packet is prefixed by its size as a 16-bit
integer.

2.9.3. HTTP Transmission

Anode packets may be submitted in HTTP POST transactions for transport over
networks where HTTP is the only available protocol.

Anode packets are simply prefixed with a 16-byte packet size and concatenated
together just as they are in a TCP stream. One or more packets may be sent
with each HTTP POST transaction for improved performance.

Since this method is intended for use in "hostile" or highly restricted
circumstances, no additional details such as special headers or MIME types
are specified to allow maximum flexibility. Peers should ignore anything
other than the payload.

2.10. Endpoints

An endpoint indicates a place where Anode packets may be sent. The following
endpoint types are specified:

|---------------------------------------------------------------------------|

| Endpoint Type | Description | Address Format |
|---------------------------------------------------------------------------|
| 0x00 | Unspecified | (none) |
| 0x01 | Ethernet | <mac[6]> |
| 0x02 | UDP/IPv4 | |
| 0x03 | TCP/IPv4 | |
| 0x04 | UDP/IPv6 | |
| 0x05 | TCP/IPv6 | |

| 0x06 | HTTP | |
|---------------------------------------------------------------------------|

Endpoints are encoded by beginning with a single byte indicating the endpoint
type followed by the address information required for the given type.

Note that IP ports bear no relationship to Anode protocol ports.

2.11. Notes

All integers in the protocol are transmitted in network (big endian) byte
order.

Common Packet Format

A common header is used for all Anode packets:

|---------------------------------------------------------------------------|

| Field | Length | Description |
|---------------------------------------------------------------------------|
| Hop Count | 1 | 8-bit hop count (not included in MAC) |
| Flags | 1 | 8-bit flags |
| MAC | 8 | 8 byte shortened CMAC-AES of packet |
| Sender Address | ? | Full address or short ID of sender |
| Recipient Address | ? | Full address or short ID of recipient |
| Peer IDs | 1 | Two 4-bit peer IDs: sender, recipient |
| Message Type | 1 | 8-bit message type |

| Message | ? | Message payload |
|---------------------------------------------------------------------------|

3.1. Hop Count

The hop count begins at zero and must be incremented by each peer that relays
the packet to another peer. The hop count must not wrap to zero at 255.

Because the hop count is modified in transit, it is not included in MAC
calculation or authentication.

The hop count is used to prioritize endpoints that are direct over endpoints
that involve relaying, or to prioritize closer routes over more distant
ones.

3.2. Flags and Flag Behavior

|---------------------------------------------------------------------------|

| Flag | Description |
|---------------------------------------------------------------------------|
| 0x01 | Sender address fully specified |
| 0x02 | Recipient address fully specified |

| 0x04 | Authentication error response |
|---------------------------------------------------------------------------|

If flag 0x01 is set, then the sender address will be the full address rather
than a short address identifier. The length of the address can be determined
from the first byte of the address, which always specifies the address type.
Flag 0x02 has the same meaning for the recipient address.

A peer must send fully specified sender addresses until it receives a response
from the recipient. At this point the sender may assume that the recipient
knows its address and use short a short sender address instead. This
assumption should time out, with a recommended timeout of 60 seconds.

There is presently no need to send fully specified recipient addresses, but
the flag is present in case it is needed and must be honored.

Flag 0x04 indicates that this is an error response containing a failed
authentication error. Since authentication failed, this packet may not have
a valid MAC. Packets with this flag must never have any effect other than
to inform of an error. This error, since it is unauthenticated, must never
have any side effects such as terminating a connection.

3.3. MAC

The MAC is calculated as follows:

Temporarily set the 64-bit/8-byte MAC field in the packet to the packet's
size as a 64-bit big-endian integer.
Calculate the MAC for the entire packet (excluding the first byte) using
the key agreed upon between the sender and the recipient, resulting in a
16 byte full CMAC-AES MAC.
Derive the 8 byte packet MAC by XORing the first 8 bytes of the full 16
byte CMAC-AES MAC with the last 8 bytes. Place this into the packet's MAC
field.

3.4. Peer IDs

Peer IDs provide a method for up to 15 different peers to share an address,
each with a unique ID allowing packets to be routed to them individually.

A peer ID of zero indicates "any" or "unspecified." Real peers must have a
nonzero peer ID. In the normal single peer per address case, any peer ID may
be used. If multiple peers are to share an address, some implementation-
dependent method must be used to ensure that each peer has a unique peer ID.

Relaying peers must follow these rules based on the recipient peer ID when
relaying messages:

IF the peer ID is zero or if the peer ID is not known, the message must
be forwarded to a random endpoint for the given recipient address.
IF the peer ID is nonzero and matches one or more known endpoints for the
given recipient address and peer ID, the message must only be sent to
a matching endpoint.

A receiving peer should process any message that it receives regardless of
whether its recipient peer ID is correct. The peer ID is primarily for relays.

Peers should typically send messages with a nonzero recipient peer ID when
responding to or involved in a conversation with a specific peer (e.g. a
streaming connection), and send zero recipient peer IDs otherwise.

3.5. Short Address Conflict Disambiguation

In the unlikely event of two Anode addresses with the same short identifier,
the recipient should use MAC validation to disambiguate. The peer ID must not
be relied upon for this purpose.

Basic Signaling and Transport Protocol

4.1. Message Types

|---------------------------------------------------------------------------|

| Type | ID | Description |
|---------------------------------------------------------------------------|
| ERROR | 0x00 | Error response |
| PING | 0x01 | Echo request |
| PONG | 0x02 | Echo response |
| EPC_REQ | 0x03 | Endpoint check request |
| EPC | 0x04 | Endpoint check response |
| EPI | 0x05 | Endpoint information |
| NAT_T | 0x06 | NAT traversal message |
| NETID_REQ | 0x07 | Request network address identification and/or test |
| NETID | 0x08 | Response to network address identification request |

| DGRAM | 0x09 | Simple UDP-like datagram |
|---------------------------------------------------------------------------|

4.2. Message Details

4.2.1. ERROR

|---------------------------------------------------------------------------|

| Field | Length | Description |
|---------------------------------------------------------------------------|
| Error Code | 2 | 16-bit error code |

| Error Arguments | ? | Error arguments, depending on error type |
|---------------------------------------------------------------------------|

Error arguments are empty unless otherwise stated below.

Error codes:

|---------------------------------------------------------------------------|

| Error Code | Description |
|---------------------------------------------------------------------------|
| 0x01 | Message not valid |
| 0x02 | Message authentication or decryption failed |
| 0x03 | Relaying and related features not authorized |

| 0x04 | Relay recipient not reachable |
|---------------------------------------------------------------------------|

Generation of errors is optional. A peer may choose to ignore invalid
messages or to throttle the sending of errors.

4.2.2. PING

(Payload unspecified.)

Request echo of payload as PONG message.

4.2.3. PONG

(Payload unspecified.)

Echoed payload of received PING message.

4.2.4. EPC_REQ

|---------------------------------------------------------------------------|

| Field | Length | Description |
|---------------------------------------------------------------------------|

| Request ID | 4 | 32-bit request ID |
|---------------------------------------------------------------------------|

Request echo of request ID in EPC message, used to check and learn endpoints.

To learn a network endpoint for a peer, CHECK_REQ is sent. If CHECK is
returned with a valid request ID, the endpoint is considered valid.

4.2.5. EPC

|---------------------------------------------------------------------------|

| Field | Length | Description |
|---------------------------------------------------------------------------|

| Request ID | 4 | 32-bit request ID echoed back |
|---------------------------------------------------------------------------|

Response to EPC_REQ containing request ID.

4.2.6. EPI

|---------------------------------------------------------------------------|

| Field | Length | Description |
|---------------------------------------------------------------------------|
| Flags | 1 | 8-bit flags |
| Endpoint | ? | Endpoint type and address |
| NAT-T mode | 1 | 8-bit NAT traversal mode |

| NAT-T options | ? | Options related to specified NAT-T mode |
|---------------------------------------------------------------------------|

EPI stands for EndPoint Identification, and is sent to notify another peer of
a network endpoint where the sending peer is reachable.

If the receiving peer is interested in communicating with the sending peer,
the receiving peer must send EPC_REQ to the sending peer at the specified
endpoint to check the validity of that endpoint. The endpoint is learned if a
valid EPC is returned.

If the endpoint in EPI is unspecified, the actual source of the EPI message
is the endpoint. This allows EPI messages to be broadcast on a local LAN
segment to advertise the presence of an address on a local network. EPI
broadcasts on local IP networks must be made to UDP port 8737.

Usually EPI is sent via relays (usually zone relays) to inform a peer of an
endpoint for direct communication.

There are presently no flags, so flags must be zero.

4.2.7. NAT_T

|---------------------------------------------------------------------------|

| Field | Length | Description |
|---------------------------------------------------------------------------|
| NAT-T mode | 1 | 8-bit NAT traversal mode |

| NAT-T options | ? | Options related to specified NAT-T mode |
|---------------------------------------------------------------------------|

NAT_T is used to send messages specific to certain NAT traversal modes.

4.2.8. NETID_REQ

|---------------------------------------------------------------------------|

| Field | Length | Description |
|---------------------------------------------------------------------------|
| Request ID | 4 | 32-bit request ID |

| Endpoint | ? | Endpoint type and address information |
|---------------------------------------------------------------------------|

When a NETID_REQ message is received, the recipient attempts to echo it back
as a NETID message to the specified endpoint address. If the endpoint is
unspecified, the recipient must fill it in with the actual origin of the
NETID_REQ message. This allows a peer to cooperate with another peer (usually
a zone relay) to empirically determine its externally visible network
address information.

A peer may ignore NETID_REQ or respond with an error if it does not allow
relaying.

4.2.9. NETID

|---------------------------------------------------------------------------|

| Field | Length | Description |
|---------------------------------------------------------------------------|
| Request ID | 4 | 32-bit request ID echoed back |
| Endpoint Type | 1 | 8-bit endpoint type |

| Endpoint Address | ? | Endpoint Address (size depends on type) |
|---------------------------------------------------------------------------|

NETID is sent in response to NETID_REQ to the specified endpoint address. It
always contains the endpoint address to which it was sent.

4.2.10. DGRAM

|---------------------------------------------------------------------------|

| Field | Length | Description |
|---------------------------------------------------------------------------|
| Source Port | 2 | 16-bit source port |
| Destination Port | 2 | 16-bit destination port |

| Payload | ? | Datagram packet payload |
|---------------------------------------------------------------------------|

A datagram is a UDP-like message without flow control or delivery assurance.

Stream Protocol

The stream protocol is very similar to TCP, though it omits some features
that are not required since they are taken care of by the encapsulating
protocol. SCTP was also an inspiration in the design.

5.1. Message Types

|---------------------------------------------------------------------------|

| Type | ID | Description |
|---------------------------------------------------------------------------|
| S_OPEN | 20 | Initiate a streaming connection (like TCP SYN) |
| S_CLOSE | 21 | Terminate a streaming connection (like TCP RST/FIN) |
| S_DATA | 22 | Data packet |
| S_ACK | 23 | Acknowedge receipt of one or more data packets |

| S_DACK | 24 | Combination of DATA and ACK |
|---------------------------------------------------------------------------|

5.2. Message Details

5.2.1. S_OPEN

|---------------------------------------------------------------------------|

| Field | Length | Description |
|---------------------------------------------------------------------------|
| Sender Link ID | 2 | 16-bit sender link ID |
| Destination Port | 2 | 16-bit destination port |
| Window Size | 2 | 16-bit window size in 1024-byte increments |
| Init. Seq. Number | 4 | 32-bit initial sequence number |

| Flags | 1 | 8-bit flags |
|---------------------------------------------------------------------------|

The OPEN message corresponds to TCP SYN, and initiates a connection. It
specifies the initial window size for the sender and the sender's initial
sequence number, which should be randomly chosen to prevent replay attacks.

If OPEN is successful, the recipient sends its own OPEN to establish the
connetion. If OPEN is unsuccessful, CLOSE is sent with its initial and current
sequence numbers equal and an appropriate reason such as "connection refused."

The sender link ID must be unique for a given recipient.

If flag 01 is set, the sender link ID is actually a source port where the
sender might be listening for connections as well. This exactly duplicates
the behavior of standard TCP. Otherwise, the sender link ID is simply an
arbitrary number that the sender uses to identify the connection with this
recipient and there is no port of origin. Ports of origin are optional for
Anode streaming connections to permit greater scalability.

5.2.2. S_CLOSE

|---------------------------------------------------------------------------|

| Field | Length | Description |
|---------------------------------------------------------------------------|
| Sender Link ID | 2 | 16-bit sender link ID |
| Destination Port | 2 | 16-bit destination port |
| Flags | 1 | 8-bit flags |
| Reason | 1 | 8-bit close reason |
| Init. Seq. Number | 4 | 32-bit initial sequence number |

| Sequence Number | 4 | 32-bit current sequence number |
|---------------------------------------------------------------------------|

The CLOSE message serves a function similar to TCP FIN. The initial sequence
number is the original starting sequence number sent with S_OPEN, while the
current sequence number is the sequence number corresponding to the close
and must be ACKed to complete the close operation. The use of the initial
sequence number helps to serve as a key to prevent replay attacks.

CLOSE is also used to indicate a failed OPEN attempt. In this case the current
sequence number will be equal to the initial sequence number and no ACK will
be expected.

There are currently no flags, so flags must be zero.

The reason field describes the reason for the close:

|---------------------------------------------------------------------------|

| Reason Code | Description |
|---------------------------------------------------------------------------|
| 00 | Application closed connection |
| 01 | Connection refused |
| 02 | Protocol error |

| 03 | Timed out |
|---------------------------------------------------------------------------|

Established connections will usually be closed with reason 00, while reason
01 is usually provided if an OPEN is received but the port is not bound.

5.2.3. S_DATA

|---------------------------------------------------------------------------|

| Payload | ? | Data payload |
|---------------------------------------------------------------------------|

The DATA message carries a packet of data, with the sequence number
determining order. The sequence number is monotonically incremented with
each data packet, and wraps at the maximum value of an unsigned 32-bit
integer.

5.2.4. S_ACK

|---------------------------------------------------------------------------|

| Acknowledgements | ? | One or more acknowledgements (see below) |
|---------------------------------------------------------------------------|

Each acknowledgement is a 32-bit integer followed by an 8-bit integer (5 bytes
total). The 32-bit integer is the first sequence number to acknowledge, and
the 8-bit integer is the number of sequential following sequence numbers to
acknowledge. For example "1, 4" would acknowledge sequence numbers 1, 2, 3,
and 4.

5.2.5. S_DACK

|---------------------------------------------------------------------------|

| Field | Length | Description |
|---------------------------------------------------------------------------|
| Sender Link ID | 2 | 16-bit sender link ID |
| Destination Port | 2 | 16-bit destination port |
| Window Size | 2 | 16-bit window size in 1024-byte increments |
| Num. Acks | 1 | 8-bit number of acknowledgements |
| Acknowledgements | ? | One or more acknowledgements |

| Payload | ? | Data payload |
|---------------------------------------------------------------------------|

The DACK message combines ACK and DATA, allowing two peers that are both
transmitting data to efficiently ACK without a separate packet.

Ext Libpqxx 7.7.3 Include Pqxx Doc Accessing Results (`ext/libpqxx-7.7.3/include/pqxx/doc/accessing-results.md`)

Accessing results and result rows {#accessing-results}

When you execute a query using one of the transaction exec functions, you
normally get a result object back. A result is a container of rows.

(There are exceptions. The exec1 functions expect exactly one row of data,
so they return just a row, not a full result.)

Result objects are an all-or-nothing affair. The exec function waits until
it's received all the result data, and then gives it to you in the form of the
result. (There is a faster, easier way of executing simple queries, so see
"streaming rows" below as well.)

For example, your code might do:

cxx

pqxx::result r = tx.exec("SELECT * FROM mytable");

Now, how do you access the data inside r?

Result sets act as standard C++ containers of rows. Rows act as standard
C++ containers of fields. So the easiest way to go through them is:

cxx

for (auto const &row: r)
    {
       for (auto const &field: row) std::cout << field.c_str() << '\t';
       std::cout << '\n';
    }

But results and rows also support other kinds of access. Array-style
indexing, for instance, such as r[rownum]:

cxx

std::size_t const num_rows = std::size(r);
    for (std::size_t rownum=0u; rownum < num_rows; ++rownum)
    {
      pqxx::row const row = r[rownum];
      std::size_t const num_cols = std::size(row);
      for (std::size_t colnum=0u; colnum < num_cols; ++colnum)
      {
        pqxx::field const field = row[colnum];
        std::cout << field.c_str() << '\t';
      }

      std::cout << '\n';
    }

Every row in the result has the same number of columns, so you don't need to
look up the number of fields again for each one:

cxx

std::size_t const num_rows = std::size(r);
    std::size_t const num_cols = r.columns();
    for (std::size_t rownum=0u; rownum < num_rows; ++rownum)
    {
      pqxx::row const row = r[rownum];
      for (std::size_t colnum=0u; colnum < num_cols; ++colnum)
      {
        pqxx::field const field = row[colnum];
        std::cout << field.c_str() << '\t';
      }

      std::cout << '\n';
    }

You can even address a field by indexing the row using the field's name:

cxx

std::cout << row["salary"] << '\n';

But try not to do that if speed matters, because looking up the column by name
takes time. At least you'd want to look up the column index before your loop
and then use numerical indexes inside the loop.

For C++23 or better, there's also a two-dimensional array access operator:

cxx

for (std::size_t rownum=0u; rownum < num_rows; ++rownum)
    {
        for (std::size_t colnum=0u; colnum < num_cols; ++colnum)
	    std::cout result[rownum, colnum].c_str() << '\t';
	std::cout << '\n';
    }

And of course you can use classic "begin/end" loops:

cxx

for (auto row = std::begin(r); row != std::end(r); row++)
    {
      for (auto field = std::begin(row); field != std::end(row); field++)
        std::cout << field->c_str() << '\t';
      std::cout << '\n';
    }

Result sets are immutable, so all iterators on results and rows are actually
const_iterators. There are also const_reverse_iterator types, which
iterate backwards from rbegin() to rend() exclusive.

All these iterator types provide one extra bit of convenience that you won't
normally find in C++ iterators: referential transparency. You don't need to
dereference them to get to the row or field they refer to. That is, instead
of row->end() you can also choose to say row.end(). Similarly, you
may prefer field.c_str() over field->c_str().

This becomes really helpful with the array-indexing operator. With regular
C++ iterators you would need ugly expressions like (*row)[0] or
row->operator[](0). With the iterator types defined by the result and
row classes you can simply say row[0].

Streaming rows

There's another way to go through the rows coming out of a query. It's
usually easier and faster, but there are drawbacks.

One, you start getting rows before all the data has come in from the
database. That speeds things up, but what happens if you lose your network
connection while transferring the data? Your application may already have
processed some of the data before finding out that the rest isn't coming. If
that is a problem for your application, streaming may not be the right choice.

Two, streaming only works for some types of query. The stream() function
wraps your query in a PostgreSQL COPY command, and COPY only supports a few
commands: SELECT, VALUES, or an INSERT, UPDATE, or DELETEwith aRETURNINGclause. See theCOPY` documentation here:
https://www.postgresql.org/docs/current/sql-copy.html

Three, when you convert a field to a "view" type (such as
std::string_view or std::basic_string_view<std::byte>), the view points to
underlying data which only stays valid until you iterate to the next row or
exit the loop. So if you want to use that data for longer than a single
iteration of the streaming loop, you'll have to store it somewhere yourself.

Now for the good news. Streaming does make it very easy to query data and loop
over it:

cxx

for (auto [id, name, x, y] :
        tx.stream<int, std::string_view, float, float>(
            "SELECT id, name, x, y FROM point"))
      process(id + 1, "point-" + name, x * 10.0, y * 10.0);

The conversion to C++ types (here int, std::string_view, and two floats)
is built into the function. You never even see row objects, field objects,
iterators, or conversion methods. You just put in your query and you receive
your data.

Ext Libpqxx 7.7.3 Include Pqxx Doc Binary Data (`ext/libpqxx-7.7.3/include/pqxx/doc/binary-data.md`)

Binary data {#binary}

The database has two ways of storing binary data: BYTEA is like a string, but
containing bytes rather than text characters. And large objects are more
like a separate table containing binary objects.

Generally you'll want to use BYTEA for reasonably-sized values, and large
objects for very large values.

That's the database side. On the C++ side, in libpqxx, all binary data must be
either std::basic_string<std::byte> or std::basic_string_view<std::byte>;
or if you're building in C++20 or better, anything that's a block of
contiguous std::byte in memory.

So for example, if you want to write a large object, you'd create a
pqxx::blob object. And you might use that to write data in the form of
std::basic_string_view<std::byte>.

Your particular binary data may look different though. You may have it in a
std::string, or a std::vector, or a pointer to char
accompanied by a size (which could be signed or unsigned, and of any of a few
different widths). Sometimes that's your choice, or sometimes some other
library will dictate what form it takes.

So long as it's basically still a block of bytes though, you can use
pqxx::binary_cast to construct a std::basic_string_view<std::byte> from it.

There are two forms of binary_cast. One takes a single argument that must
support std::data() and std::size():

text

std::string hi{"Hello binary world"};
my_blob.write(pqxx::binary_cast(hi);

The other takes a pointer and a size:

text

char const greeting[] = "Hello binary world";
char const *hi = greeting;
my_blob.write(pqxx::binary_cast(hi, sizeof(greeting)));

Caveats

There are some restrictions on binary_cast that you must be aware of.

First, your data must of a type that gives us bytes. So: char,
unsigned char, signed char, int8_t, uint8_t, or of course std::byte.
You can't feed in a vector of double, or anything like that.

Second, the data must be laid out as a contiguous block in memory. If there's
no std::data() implementation for your type, it's not suitable.

Third, binary_cast only constructs something like a std::string_view. It
does not make a copy of your actual data. So, make sure that your data remains
alive and in the same place while you're using it.

Ext Libpqxx 7.7.3 Include Pqxx Doc Datatypes (`ext/libpqxx-7.7.3/include/pqxx/doc/datatypes.md`)

Supporting additional data types {#datatypes}

Communication with the database mostly happens in a text format. When you
include an integer value in a query, you use to_string to convert it to that
text format. When you get a query result field "as a float," it converts from
the text format to a floating-point type. These conversions are everywhere in
libpqxx.

The conversion sydstem supports many built-in types, but it is also extensible.
You can "teach" libpqxx (in the scope of your own application) to convert
additional types of values to and from PostgreSQL's string format.

This is massively useful, but it's not for the faint of heart. You'll need to
specialise some templates. And, the API for doing this can change with any
major libpqxx release.

Converting types

In your application, a conversion is driven entirely by a C++ type you specify.
The value's SQL type has nothing to do with it, nor is there anything in the
string that would identify its type.

So, if you've SELECTed a 64-bit integer from the database, and you try to
convert it to a C++ "short," one of two things will happen: either the number
is small enough to fit in your short (and it just works), or else it throws a
conversion exception.

Or, your database table might have a text column, but a given field may contain
a string that looks just like a number. You can convert that value to an
integer type just fine. Or to a floating-point type. All that matters to the
conversion is the actual value, and the type.

In some cases the templates for these conversions can tell the type from the
arguments you pass them:

text

auto x = to_string(99);

In other cases you may need to instantiate template explicitly:

text

auto y = from_string("99");

Supporting a new type

Let's say you have some other SQL type which you want to be able to store in,
or retrieve from, the database. What would it take to support that?

Sometimes you do not need complete support. You might need a conversion to
a string but not from a string, for example. The conversion is defined at
compile time, so don't be too afraid to be incomplete. If you leave out one of
these steps, it's not going to crash at run time or mess up your data. The
worst that can happen is that your code won't build.

So what do you need for a complete conversion?

First off, of course, you need a C++ type. It may be your own, but it
doesn't have to be. It could be a type from a third-party library, or even one
from the standard library that libpqxx does not yet support.

You also specialise the pqxx::type_name variable to specify the type's name.
This is important for all code which mentions your type in human-readable text,
such as error messages.

Then, does your type have a built-in null value? You specialise the
pqxx::nullness template to specify the details.

Finally, you specialise the pqxx::string_traits template. This is where you
define the actual conversions.

Let's go through these steps one by one.

Your type

You'll need a type for which the conversions are not yet defined, because the
C++ type is what determines the right conversion. One type, one set of
conversions.

The type doesn't have to be one that you create. The conversion logic was
designed such that you can build it around any type. So you can just as
easily build a conversion for a type that's defined somewhere else. There's
no need to include any special methods or other members inside it. That's also
how libpqxx can support converting built-in types like int.

By the way, if the type is an enum, you don't need to do any of this. Just
invoke the preprocessor macro PQXX_DECLARE_ENUM_CONVERSION, from the global
namespace near the top of your translation unit, and pass the type as an
argument.

The library also provides specialisations for std::optional<T>,
std::shared_ptr<T>, and std::unique_ptr<T>. If you have conversions for
T, you'll also have conversions for those.

Specialise `type_name`

When errors happen during conversion, libpqxx will compose error messages for
the user. Sometimes these will include the name of the type that's being
converted.

To tell libpqxx the name of each type, there's a template variable called
pqxx::type_name. For any given type T, it should have a specialisation
that provides that T's human-readable name:

text

namespace pqxx
{
template<> std::string const type_name<T>{"T"};
}

(Yes, this means that you need to define something inside the pqxx namespace.
Future versions of libpqxx may move this into a separate namespace.)

Define this early on in your translation unit, before any code that might cause
libpqxx to need the name. That way, the libpqxx code which needs to know the
type's name can see your definition.

Specialise `nullness`

A struct template pqxx::nullness defines whether your type has a natural
"null value" built in. If so, it also provides member functions for producing
and recognising null values.

The simplest scenario is also the most common: most types don't have a null
value built in. In that case, derive your nullness traits from
pqxx::no_null:

text

namespace pqxx
{
template<> struct nullness<T> : pqxx::no_null<T> {};
}

(Here again you're defining this in the pqxx namespace.)

If your type does have a natural null value, the definition gets a little more
complex:

text

namespace pqxx
{
template<> struct nullness<T>
{
  static constexpr bool has_null{true};
  static constexpr bool always_null{false};

  static bool is_null(T const &value)
  {
    // Return whether "value" is null.
    return ...;
  }

  [[nodiscard]] static T null()
  {
    // Return a null value.
    return ...;
  }
};
}

You may be wondering why there's a function to produce a null value, but also a
function to check whether a value is null. Why not just compare the value to
the result of null()? Because two null values may not be equal. T may
have several different null values. Or it may override the comparison
operator, similar to SQL where NULL is not equal to NULL.

As a third case, your type may be one that always represents a null value.
This is the case for std::nullptr_t and std::nullopt_t. In that case, you
set nullness<TYPE>::always_null to true (as well as has_null of course),
and you won't need to define any actual conversions.

Specialise `string_traits`

This part is more work. (You can skip it for types that are always null,
but those will be rare.) Specialise the pqxx::string_traits template:

text

namespace pqxx
{
template<> struct string_traits<T>
{
  static T from_string(std::string_view text);
  static zview to_buf(char *begin, char *end, T const &value);
  static char *into_buf(char *begin, char *end, T const &value);
  static std::size_t size_buffer(T const &value) noexcept;
};
}

You'll also need to write those member functions, or as many of them as needed
to get your code to build.

`from_string`

We start off simple: from_string parses a string as a value of T, and
returns that value.

The string may not be zero-terminated; it's just the string_view from
beginning to end (exclusive). In your tests, cover cases where the string
does not end in a zero byte.

It's perfectly possible that the string isn't actually a T value. Mistakes
happen. In that case, throw a pqxx::conversion_error.

(Of course it's also possible that you run into some other error, so it's fine
to throw different exceptions as well. But when it's definitely "this is not
the right format for a T," throw conversion_error.)

`to_buf`

In this function, you convert a value of T into a string that the postgres
server will understand.

The caller will provide you with a buffer where you can write the string, if
you need it: from begin to end exclusive. It's a half-open interval, so
don't access *end.

If the buffer is insufficient for you to do the conversion, throw a
pqxx::conversion_overrun. It doesn't have to be exact: you can be a little
pessimistic and demand a bit more space than you need. Just be sure to throw
the exception if there's any risk of overrunning the buffer.

You don't have to use the buffer for this function though. For example,
pqxx::string_traits::to_buf returns a compile-time constant string and
ignores the buffer.

Even if you do use the buffer, your string does not have to start at the
beginning of the buffer. For example, the integer conversions start by writing
the least significant digit to the end of the buffer, and then writes the
more significant digits before it. It was just more convenient.

Return a pqxx::zview. This is basically a std::string_view, but with one
difference: a zview guarantees that there will be a valid zero byte right
after the string_view. The zero byte is not counted as part of its size, but
it will be there.

Expressed in code, this rule must hold:

text

void invariant(zview z)
{
  assert(z[std::size(z)] == 0);
}

Make sure you write your trailing zero before the end. If the trailing
zero doesn't fit in the buffer, then there's just not enough room to perform
the conversion.

Beware of locales when converting. If you use standard library features like
sprintf, they may obey whatever locale is currently set on the system. That
means that a simple integer like 1000000 may come out as "1000000" on your
system, but as "1,000,000" on mine, or as "1.000.000" for somebody else, and on
an Indian system it may be "1,00,000". Values coming from or going to the
database should be in non-localised formats. You can use libpqxx functions for
those conversions: pqxx::from_string, pqxx::to_string, pqxx::to_buf.

`into_buf`

This is a stricter version of to_buf. All the same requirements apply, but
in addition you must write your string into the buffer provided, starting
exactly at begin.

That's why this function returns just a simple pointer: the address right
behind the trailing zero. If the caller wants to use the string, they can
find it at begin. If they want to write a different value into the rest of
the buffer, they can start at the location you returned.

`size_buffer`

Here you estimate how much buffer space you need for converting a T to a
string. Be precise if you can, but pessimistic if you must. It's usually
better to waste a few unnecessary bytes than to spend a lot of time computing
the exact buffer space you need. And failing the conversion because you
under-budgeted the buffer is worst of all.

Include the trailing zero in the buffer size. If your to_buf takes more
space than just what's needed to store the result, include that too.

Make size_buffer a constexpr function if you can. It can allow the caller
to allocate the buffer on the stack, with a size known at compile time.

Optional: Specialise `is_unquoted_safe`

When converting arrays or composite values to strings, libpqxx may need to
quote values and escape any special characters. This takes time.

Some types though, such as integral or floating-point types, can never have
any special characters such as quotes, commas, or backslashes in their string
representations. In such cases, there's no need to quote or escape such values
in arrays or composite types.

If your type is like that, you can tell libpqxx about this by defining:

text

namespace pqxx
{
template<> inline constexpr bool is_unquoted_safe<MY_TYPE>{true};
}

The code that converts this type of field to strings in an array or a composite
type can then use a simpler, more efficient variant of the code. It's always
safe to leave this out; it's just an optimisation for when you're completely
sure that it's safe.

Do not do this if a string representation of your type may contain a comma;
semicolon; parenthesis; brace; quote; backslash; newline; or any other
character that might need escaping.

Optional: Specialise `param_format`

This one you don't generally need to worry about. Read on if you're writing a
type which represents raw binary data, or if you're writing a template where
some specialisations may contain raw binary data.

When you call parameterised statements, or prepared statements with parameters,
libpqxx needs to your parameters on to libpq, the underlying C-level PostgreSQL
client library.

There are two formats for doing that: text and binary. In the first, we
represent all values as strings, and the server then converts them into its own
internal binary representation. That's what the string conversions are all
about, and it's what we do for almost all types of parameters.

But we do it differently when the parameter is a contiguous series of raw bytes
and the corresponding SQL type is BYTEA. There is a text format for those,
but we bypass it for efficiency. The server can use the binary data in the
exact same form, without any conversion or extra processing. The binary data
is also twice as compact during transport.

(People sometimes ask why we can't just treat all types as binary. However the
general case isn't so clear-cut. The binary formats are not documented, there
are no guarantees that they will be platform-independent or that they will
remain stable, and there's no really solid way to detect when we might get the
format wrong. But also, the conversions aren't necessarily as straightforward
and efficient as they sound. So, for the general case, libpqxx sticks with the
text formats. Raw binary data alone stands out as a clear win.)

Long story short, the machinery for passing parameters needs to know: is this
parameter a binary string, or not? In the normal case it can assume "no," and
that's what it does. The text format is always a safe choice; we just try to
use the binary format where it's faster.

The param_format function template is what makes the decision. We specialise
it for types which may be binary strings, and use the default for all other
types.

"Types which may be binary"? You might think we know whether a type is a
binary type or not. But there are some complications with generic types.

Templates like std::shared_ptr, std::optional, and so on act like
"wrappers" for another type. A std::optional<T> is binary if T is binary.
Otherwise, it's not. If you're building support for a template of this nature,
you'll probably want to implement param_format for it.

The decision to use binary format is made based on a given object, not
necessarily based on the type in general. Look at std::variant. If you have
a std::variant type which can hold an int or a binary string, is that a
binary parameter? We can't decide without knowing the individual object.

Containers are another hard case. Should we pass std::vector<T> in binary?
Even when T is a binary type, we don't currently have any way to pass an
array in binary format, so we always pass it as text.

Ext Libpqxx 7.7.3 Include Pqxx Doc Escaping (`ext/libpqxx-7.7.3/include/pqxx/doc/escaping.md`)

String escaping {#escaping}

Writing queries as strings is easy. But sometimes you need a variable in
there: "SELECT id FROM user WHERE name = '" + name + "'".

This is dangerous. See the bug? If name can contain quotes, you may have
an SQL injection vulnerability there, where users can enter nasty stuff like
".'; DROP TABLE user". Or if you're lucky, it's just a nasty bug that you
discover when name happens to be "d'Arcy".

So, you'll need to escape the name before you insert it. This is where
quotes and other problematic characters are marked as "this is just a character
in the string, not the end of the string." There are
[several functions](@ref escaping-functions) in libpqxx to do this for you.

SQL injection

To understand what SQL injection vulnerabilities are and why they should be
prevented, imagine you use the following SQL statement somewhere in your
program:

text

TX.exec(
	"SELECT number,amount "
	"FROM accounts "
	"WHERE allowed_to_see('" + userid + "','" + password + "')");

This shows a logged-in user important information on all accounts he is
authorized to view. The userid and password strings are variables entered
by the user himself.

Now, if the user is actually an attacker who knows (or can guess) the
general shape of this SQL statement, imagine he enters the following
password:

text

x') OR ('x' = 'x

Does that make sense to you? Probably not. But if this is inserted into
the SQL string by the C++ code above, the query becomes:

text

SELECT number,amount
FROM accounts
WHERE allowed_to_see('user','x') OR ('x' = 'x')

Is this what you wanted to happen? Probably not! The neat allowed_to_see()
clause is completely circumvented by the "OR ('x' = 'x')" clause, which is
always true. Therefore, the attacker will get to see all accounts in the
database!

Using the esc functions

Here's how you can fix the problem in the example above:

text

TX.exec(
	"SELECT number,amount "
	"FROM accounts "
	"WHERE allowed_to_see('" + TX.esc(userid) + "', "
		"'" + TX.esc(password) + "')");

Now, the quotes embedded in the attacker's string will be neatly escaped so
they can't "break out" of the quoted SQL string they were meant to go into:

text

SELECT number,amount
FROM accounts
WHERE allowed_to_see('user', 'x'') OR (''x'' = ''x')

If you look carefully, you'll see that thanks to the added escape characters
(a single-quote is escaped in SQL by doubling it) all we get is a very
strange-looking password string--but not a change in the SQL statement.

Ext Libpqxx 7.7.3 Include Pqxx Doc Mainpage (`ext/libpqxx-7.7.3/include/pqxx/doc/mainpage.md`)

libpqxx {#mainpage}

@version 7.7.3
@author Jeroen T. Vermeulen
@see http://pqxx.org
@see https://github.com/jtv/libpqxx

Welcome to libpqxx, the C++ API to the PostgreSQL database management system.

Compiling this package requires PostgreSQL to be installed -- including the
C headers for client development. The library builds on top of PostgreSQL's
standard C API, libpq. The libpq headers are not needed to compile client
programs, however.

For a quick introduction to installing and using libpqxx, see the README.md
file. The latest information can be found at http://pqxx.org/

Some links that should help you find your bearings:

@ref getting-started
@ref thread-safety
@ref connections
@ref transactions
@ref escaping
@ref performance
@ref transactor
@ref datatypes

Ext Libpqxx 7.7.3 Include Pqxx Doc Parameters (`ext/libpqxx-7.7.3/include/pqxx/doc/parameters.md`)

Statement parameters {#parameters}

When you execute a prepared statement (see @ref prepared), or a parameterised
statement (using functions like pqxx::connection::exec_params), you may write
special placeholders in the query text. They look like $1, $2, and so
on.

If you execute the query and pass parameter values, the call will respectively
substitute the first where it finds $1, the second where it finds $2, et
cetera.

Doing this saves you work. If you don't use statement parameters, you'll need
to quote and escape your values (see connection::quote() and friends) as you
insert them into your query as literal values.

Or if you forget to do that, you leave yourself open to horrible
SQL injection attacks. Trust me, I was born in a town
whose name started with an apostrophe!

Statement parameters save you this work. With these parameters you can pass
your values as-is, and they will go across the wire to the database in a safe
format.

In some cases it may even be faster! When a parameter represents binary data
(as in the SQL BYTEA type), libpqxx will send it directly as binary, which is
a bit more efficient. If you insert the binary data directly in your query
text, your CPU will have some extra work to do, converting the data into a text
format, escaping it, and adding quotes.

Dynamic parameter lists

In rare cases you may just not know how many parameters you'll pass into your
statement when you call it.

For these situations, have a look at params. It lets you compose your
parameters list on the fly, even add whole ranges of parameters at a time.

You can pass a params into your statement as a normal parameter. It will
fill in all the parameter values it contains into that position of the
statement's overall parameter list.

So if you call your statement passing a regular parameter a, a
params containing just a parameter b, and another regular parameter c,
then your call will pass parameters a, b, and c. Or if the params object
is empty, it will pass just a and c. If the params object contains x and
y, your call will pass a, x, y, c.

You can mix static and dynamic parameters freely. Don't go overboard though:
complexity is where bugs happen!

Generating placeholders

If your code gets particularly complex, it may sometimes happen that it becomes
hard to track which parameter value belongs with which placeholder. Did you
intend to pass this numeric value as $7, or as $8? The answer may depend
on an if that happened earlier in a different function.

(Generally if things get that complex, it's a good idea to look for simpler
solutions. But especially when performance matters, sometimes you can't avoid
complexity like that.)

There's a little helper class called placeholders. You can use it as a
counter which produces those placeholder strings, $1, $2, $3, et cetera.
When you start generating a complex statement, you can create both a params
and a placeholders:

text

pqxx::params values;
pqxx::placeholders name;

Let's say you've got some complex code to generate the conditions for an SQL
"WHERE" clause. You'll generally want to do these things close together in
your, so that you don't accidentally update one part and forget another:

text

if (extra_clause)
{
  // Extend the query text, using the current placeholder.
  query += " AND x = " + name.get();
  // Add the parameter value.
  values.append(my_x);
  // Move on to the next placeholder value.
  name.next();
}

Depending on the starting value of name, this might add to query a fragment
like " AND x = $3" or " AND x = $5".

Ext Libpqxx 7.7.3 Include Pqxx Doc Performance (`ext/libpqxx-7.7.3/include/pqxx/doc/performance.md`)

Performance features {#performance}

If your program's database interaction is not as efficient as it needs to be,
the first place to look is usually the SQL you're executing. But libpqxx
has a few specialized features to help you squeeze more performance out
of how you issue commands and retrieve data:

@ref streams. Use these as a faster way to transfer data between your
code and the database.
std::string_view and pqxx::zview. In places where traditional C++ worked
with std::string, see whether std::string_view or pqxx::zview will
do. Of course that means that you'll have to look at the data's lifetime
more carefully, but it'll save the computer a lot of copying.
@ref prepared. These can be executed many times without the server
parsing and planning them anew each time. They also save you having to
escape string parameters.
pqxx::pipeline lets you send queries to the database in batches, and
continue other processing while they are executing.
pqxx::connecting lets you start setting up a database connection, but
without blocking the thread.

As always of course, don't risk the quality of your code for optimizations
that you don't need!

Ext Libpqxx 7.7.3 Include Pqxx Doc Prepared Statement (`ext/libpqxx-7.7.3/include/pqxx/doc/prepared-statement.md`)

Prepared statements {#prepared}

Prepared statements are SQL queries that you define once and then invoke
as many times as you like, typically with varying parameters. It's basically
a function that you can define ad hoc.

If you have an SQL statement that you're going to execute many times in
quick succession, it may be more efficient to prepare it once and reuse it.
This saves the database backend the effort of parsing complex SQL and
figuring out an efficient execution plan. Another nice side effect is that
you don't need to worry about escaping parameters. Some corporate coding
standards require all SQL parameters to be passed in this way, to reduce the
risk of programmer mistakes leaving room for SQL injections.

Preparing a statement

You create a prepared statement by preparing it on the connection (using the
pqxx::connection::prepare functions), passing an identifier and its SQL text.

The identifier is the name by which the prepared statement will be known; it
should consist of ASCII letters, digits, and underscores only, and start with
an ASCII letter. The name is case-sensitive.

cxx

void prepare_my_statement(pqxx::connection &c)
    {
      c.prepare(
          "my_statement",
          "SELECT * FROM Employee WHERE name = 'Xavier'");
    }

Once you've done this, you'll be able to call my_statement from any
transaction you execute on the same connection. For this, use the
pqxx::transaction_base::exec_prepared functions.

cxx

pqxx::result execute_my_statement(pqxx::transaction_base &t)
    {
      return t.exec_prepared("my_statement");
    }

Parameters

Did I mention that prepared statements can have parameters? The query text
can contain $1, $2 etc. as placeholders for parameter values that you
will provide when you invoke the prepared satement.

See @ref parameters for more about this. And here's a simple example of
preparing a statement and invoking it with parameters:

cxx

void prepare_find(pqxx::connection &c)
    {
      // Prepare a statement called "find" that looks for employees with a
      // given name (parameter 1) whose salary exceeds a given number
      // (parameter 2).
      c.prepare(
  	    "find",
  	    "SELECT * FROM Employee WHERE name = $1 AND salary > $2");
    }

This example looks up the prepared statement "find," passes name and
min_salary as parameters, and invokes the statement with those values:

cxx

pqxx::result execute_find(
      pqxx::transaction_base &t, std::string name, int min_salary)
    {
      return t.exec_prepared("find", name, min_salary);
    }

A special prepared statement

There is one special case: the nameless prepared statement. You may prepare
a statement without a name, i.e. whose name is an empty string. The unnamed
statement can be redefined at any time, without un-preparing it first.

Performance note

Don't assume that using prepared statements will speed up your application.
There are cases where prepared statements are actually slower than plain SQL.

The reason is that the backend can often produce a better execution plan when
it knows the statement's actual parameter values.

For example, say you've got a web application and you're querying for users
with status "inactive" who have email addresses in a given domain name X. If
X is a very popular provider, the best way for the database engine to plan the
query may be to list the inactive users first and then filter for the email
addresses you're looking for. But in other cases, it may be much faster to
find matching email addresses first and then see which of their owners are
"inactive." A prepared statement must be planned to fit either case, but a
direct query will be optimised based on table statistics, partial indexes, etc.

Zero bytes

@warning Beware of "nul" bytes!

Any string you pass as a parameter will end at the first char with value
zero. If you pass a string that contains a zero byte, the last byte in the
value will be the one just before the zero.

So, if you need a zero byte in a string, consider that it's really a binary
string, which is not the same thing as a text string. SQL represents binary
data as the BYTEA type, or in binary large objects ("blobs").

In libpqxx, you represent binary data as a range of std::byte. They must be
contiguous in memory, so that libpqxx can pass pointers to the underlying C
library. So you might use std::basic_string<std::byte>, or
std::basic_string_view<std::byte>, or std::vector<std::byte>.

Ext Libpqxx 7.7.3 Include Pqxx Doc Streams (`ext/libpqxx-7.7.3/include/pqxx/doc/streams.md`)

Streams {#streams}

Most of the time it's fine to retrieve data from the database using SELECT
queries, and store data using INSERT. But for those cases where efficiency
matters, there are two classes to help you do this better: stream_from and
stream_to. They're less flexible than SQL queries, and there's the risk of
losing your connection while you're in mid-stream, but you get some speed and
memory efficiencies in return.

Both stream classes do data conversion for you: stream_from receives values
from the database in PostgreSQL's text format, and converts them to the C++
types you specify. Likewise, stream_to converts C++ values you provide to
PostgreSQL's text format for transfer. (On its end, the database of course
converts values to and from their SQL types.)

Null values

So how do you deal with nulls? It depends on the C++ type you're using. Some
types may have a built-in null value. For instance, if you have a
char const * value and you convert it to an SQL string, then converting a
nullptr will produce a NULL SQL value.

But what do you do about C++ types which don't have a built-in null value, such
as int? The trick is to wrap it in std::optional. The difference between
int and std::optional is that the former always has an int value,
and the latter doesn't have to.

Actually it's not just std::optional. You can do the same thing with
std::unique_ptr or std::shared_ptr. A smart pointer is less efficient than
std::optional in most situations because they allocate their value on the
heap, but sometimes that's what you want in order to save moving or copying
large values around.

This part is not generic though. It won't work with just any smart-pointer
type, just the ones which are explicitly supported: shared_ptr and
unique_ptr. If you really need to, you can build support for additional
wrappers and smart pointers by copying the implementation patterns from the
existing smart-pointer support.

stream\_from

Use stream_from to read data directly from the database. It's faster than
the transaction's exec functions if the result contains enough rows. But
also, you won't need to keep your full result set in memory. That can really
matter with larger data sets.

And, you can start processing your data right after the first row of data comes
in from the server. With exec() you need to wait to receive all data, and
then you begin processing. With stream_from you can be processing data on
the client side while the server is still sending you the rest.

You don't actually need to create a stream_from object yourself, though you
can. Two shorthand functions, @ref pqxx::transaction_base::stream
and @ref pqxx::transaction_base::for_each, can create the streams for you with
a minimum of overhead.

Not all kinds of queries will work in a stream. Internally the streams make
use of PostgreSQL's COPY command, so see the PostgreSQL documentation for
COPY for the exact limitations. Basic SELECT and UPDATE ... RETURNING
queries should just work.

As you read a row, the stream converts its fields to a tuple type containing
the value types you ask for:

text

auto stream pqxx::stream_from::query(
    tx, "SELECT name, points FROM score");
std::tuple<std::string, int> row;
while (stream >> row)
  process(row);
stream.complete();

As the stream reads each row, it converts that row's data into your tuple,
goes through your loop body, and then promptly forgets that row's data. This
means you can easily process more data than will fit in memory.

stream\_to

Use stream_to to write data directly to a database table. This saves you
having to perform an INSERT for every row, and so it can be significantly
faster if you want to insert more than just one or two rows at a time.

As with stream_from, you can specify the table and the columns, and not much
else. You insert tuple-like objects of your choice:

text

pqxx::stream_to stream{
    tx,
    "score",
    std::vector<std::string>{"name", "points"}};
for (auto const &entry: scores)
    stream << entry;
stream.complete();

Each row is processed as you provide it, and not retained in memory after that.

The call to complete() is more important here than it is for stream_from.
It's a lot like a "commit" or "abort" at the end of a transaction. If you omit
it, it will be done automatically during the stream's destructor. But since
destructors can't throw exceptions, any failures at that stage won't be visible
in your code. So, always call complete() on a stream_to to close it off
properly!

Ext Libpqxx 7.7.3 Include Pqxx Doc Thread Safety (`ext/libpqxx-7.7.3/include/pqxx/doc/thread-safety.md`)

Thread safety {#thread-safety}

This library does not contain any locking code to protect objects against
simultaneous modification in multi-threaded programs. Therefore it is up
to you, the user of the library, to ensure that your threaded client
programs perform no conflicting operations concurrently.

Most of the time this isn't hard. Result sets are immutable, so you can
share them between threads without problem. The main rule is:

@li Treat a connection, together with any and all objects related to it, as
a "world" of its own. You should generally make sure that the same "world"
is never accessed by another thread while you're doing anything non-const
in there.

That means: don't issue a query on a transaction while you're also opening
a subtransaction, don't access a cursor while you may also be committing,
and so on.

In particular, cursors are tricky. It's easy to perform a non-const
operation without noticing. So, if you're going to share cursors or
cursor-related objects between threads, lock very conservatively!

Use pqxx::describe_thread_safety to find out at runtime what level of
thread safety is implemented in your build and version of libpqxx. It
returns a pqxx::thread_safety_model describing what you can and cannot rely
on. A command-line utility tools/pqxxthreadsafety prints out the same
information.

Accessing results and result rows {#accessing-results}

When you execute a query using one of the transaction exec functions, you
normally get a result object back. A result is a container of rows.

(There are exceptions. The exec1 functions expect exactly one row of data,
so they return just a row, not a full result.)

For example, your code might do:

cxx

pqxx::result r = tx.exec("SELECT * FROM mytable");

Now, how do you access the data inside r?

Result sets act as standard C++ containers of rows. Rows act as standard
C++ containers of fields. So the easiest way to go through them is:

cxx

for (auto const &row: r)
    {
       for (auto const &field: row) std::cout << field.c_str() << '\t';
       std::cout << '\n';
    }

But results and rows also support other kinds of access. Array-style
indexing, for instance, such as r[rownum]:

cxx

std::size_t const num_rows = std::size(r);
    for (std::size_t rownum=0u; rownum < num_rows; ++rownum)
    {
      pqxx::row const row = r[rownum];
      std::size_t const num_cols = std::size(row);
      for (std::size_t colnum=0u; colnum < num_cols; ++colnum)
      {
        pqxx::field const field = row[colnum];
        std::cout << field.c_str() << '\t';
      }

      std::cout << '\n';
    }

Every row in the result has the same number of columns, so you don't need to
look up the number of fields again for each one:

cxx

std::size_t const num_rows = std::size(r);
    std::size_t const num_cols = r.columns();
    for (std::size_t rownum=0u; rownum < num_rows; ++rownum)
    {
      pqxx::row const row = r[rownum];
      for (std::size_t colnum=0u; colnum < num_cols; ++colnum)
      {
        pqxx::field const field = row[colnum];
        std::cout << field.c_str() << '\t';
      }

      std::cout << '\n';
    }

You can even address a field by indexing the row using the field's name:

cxx

std::cout << row["salary"] << '\n';

For C++23 or better, there's also a two-dimensional array access operator:

cxx

for (std::size_t rownum=0u; rownum < num_rows; ++rownum)
    {
        for (std::size_t colnum=0u; colnum < num_cols; ++colnum)
	    std::cout result[rownum, colnum].c_str() << '\t';
	std::cout << '\n';
    }

And of course you can use classic "begin/end" loops:

cxx

for (auto row = std::begin(r); row != std::end(r); row++)
    {
      for (auto field = std::begin(row); field != std::end(row); field++)
        std::cout << field->c_str() << '\t';
      std::cout << '\n';
    }

Streaming rows

There's another way to go through the rows coming out of a query. It's
usually easier and faster, but there are drawbacks.

Now for the good news. Streaming does make it very easy to query data and loop
over it:

cxx

for (auto [id, name, x, y] :
        tx.stream<int, std::string_view, float, float>(
            "SELECT id, name, x, y FROM point"))
      process(id + 1, "point-" + name, x * 10.0, y * 10.0);

Binary data {#binary}

Generally you'll want to use BYTEA for reasonably-sized values, and large
objects for very large values.

So for example, if you want to write a large object, you'd create a
pqxx::blob object. And you might use that to write data in the form of
std::basic_string_view<std::byte>.

So long as it's basically still a block of bytes though, you can use
pqxx::binary_cast to construct a std::basic_string_view<std::byte> from it.

There are two forms of binary_cast. One takes a single argument that must
support std::data() and std::size():

text

std::string hi{"Hello binary world"};
my_blob.write(pqxx::binary_cast(hi);

The other takes a pointer and a size:

text

char const greeting[] = "Hello binary world";
char const *hi = greeting;
my_blob.write(pqxx::binary_cast(hi, sizeof(greeting)));

Caveats

There are some restrictions on binary_cast that you must be aware of.

Second, the data must be laid out as a contiguous block in memory. If there's
no std::data() implementation for your type, it's not suitable.

Supporting additional data types {#datatypes}

This is massively useful, but it's not for the faint of heart. You'll need to
specialise some templates. And, the API for doing this can change with any
major libpqxx release.

Converting types

In your application, a conversion is driven entirely by a C++ type you specify.
The value's SQL type has nothing to do with it, nor is there anything in the
string that would identify its type.

In some cases the templates for these conversions can tell the type from the
arguments you pass them:

text

auto x = to_string(99);

In other cases you may need to instantiate template explicitly:

text

auto y = from_string("99");

Supporting a new type

Let's say you have some other SQL type which you want to be able to store in,
or retrieve from, the database. What would it take to support that?

So what do you need for a complete conversion?

You also specialise the pqxx::type_name variable to specify the type's name.
This is important for all code which mentions your type in human-readable text,
such as error messages.

Then, does your type have a built-in null value? You specialise the
pqxx::nullness template to specify the details.

Finally, you specialise the pqxx::string_traits template. This is where you
define the actual conversions.

Let's go through these steps one by one.

Your type

You'll need a type for which the conversions are not yet defined, because the
C++ type is what determines the right conversion. One type, one set of
conversions.

The library also provides specialisations for std::optional<T>,
std::shared_ptr<T>, and std::unique_ptr<T>. If you have conversions for
T, you'll also have conversions for those.

Specialise `type_name`

When errors happen during conversion, libpqxx will compose error messages for
the user. Sometimes these will include the name of the type that's being
converted.

To tell libpqxx the name of each type, there's a template variable called
pqxx::type_name. For any given type T, it should have a specialisation
that provides that T's human-readable name:

text

namespace pqxx
{
template<> std::string const type_name<T>{"T"};
}

(Yes, this means that you need to define something inside the pqxx namespace.
Future versions of libpqxx may move this into a separate namespace.)

Define this early on in your translation unit, before any code that might cause
libpqxx to need the name. That way, the libpqxx code which needs to know the
type's name can see your definition.

Specialise `nullness`

A struct template pqxx::nullness defines whether your type has a natural
"null value" built in. If so, it also provides member functions for producing
and recognising null values.

The simplest scenario is also the most common: most types don't have a null
value built in. In that case, derive your nullness traits from
pqxx::no_null:

text

namespace pqxx
{
template<> struct nullness<T> : pqxx::no_null<T> {};
}

(Here again you're defining this in the pqxx namespace.)

If your type does have a natural null value, the definition gets a little more
complex:

text

namespace pqxx
{
template<> struct nullness<T>
{
  static constexpr bool has_null{true};
  static constexpr bool always_null{false};

  static bool is_null(T const &value)
  {
    // Return whether "value" is null.
    return ...;
  }

  [[nodiscard]] static T null()
  {
    // Return a null value.
    return ...;
  }
};
}

Specialise `string_traits`

This part is more work. (You can skip it for types that are always null,
but those will be rare.) Specialise the pqxx::string_traits template:

text

namespace pqxx
{
template<> struct string_traits<T>
{
  static T from_string(std::string_view text);
  static zview to_buf(char *begin, char *end, T const &value);
  static char *into_buf(char *begin, char *end, T const &value);
  static std::size_t size_buffer(T const &value) noexcept;
};
}

You'll also need to write those member functions, or as many of them as needed
to get your code to build.

`from_string`

We start off simple: from_string parses a string as a value of T, and
returns that value.

The string may not be zero-terminated; it's just the string_view from
beginning to end (exclusive). In your tests, cover cases where the string
does not end in a zero byte.

It's perfectly possible that the string isn't actually a T value. Mistakes
happen. In that case, throw a pqxx::conversion_error.

`to_buf`

In this function, you convert a value of T into a string that the postgres
server will understand.

The caller will provide you with a buffer where you can write the string, if
you need it: from begin to end exclusive. It's a half-open interval, so
don't access *end.

You don't have to use the buffer for this function though. For example,
pqxx::string_traits::to_buf returns a compile-time constant string and
ignores the buffer.

Expressed in code, this rule must hold:

text

void invariant(zview z)
{
  assert(z[std::size(z)] == 0);
}

Make sure you write your trailing zero before the end. If the trailing
zero doesn't fit in the buffer, then there's just not enough room to perform
the conversion.

`into_buf`

This is a stricter version of to_buf. All the same requirements apply, but
in addition you must write your string into the buffer provided, starting
exactly at begin.

`size_buffer`

Include the trailing zero in the buffer size. If your to_buf takes more
space than just what's needed to store the result, include that too.

Make size_buffer a constexpr function if you can. It can allow the caller
to allocate the buffer on the stack, with a size known at compile time.

Optional: Specialise `is_unquoted_safe`

When converting arrays or composite values to strings, libpqxx may need to
quote values and escape any special characters. This takes time.

If your type is like that, you can tell libpqxx about this by defining:

text

namespace pqxx
{
template<> inline constexpr bool is_unquoted_safe<MY_TYPE>{true};
}

Do not do this if a string representation of your type may contain a comma;
semicolon; parenthesis; brace; quote; backslash; newline; or any other
character that might need escaping.

Optional: Specialise `param_format`

When you call parameterised statements, or prepared statements with parameters,
libpqxx needs to your parameters on to libpq, the underlying C-level PostgreSQL
client library.

The param_format function template is what makes the decision. We specialise
it for types which may be binary strings, and use the default for all other
types.

"Types which may be binary"? You might think we know whether a type is a
binary type or not. But there are some complications with generic types.

String escaping {#escaping}

Writing queries as strings is easy. But sometimes you need a variable in
there: "SELECT id FROM user WHERE name = '" + name + "'".

SQL injection

To understand what SQL injection vulnerabilities are and why they should be
prevented, imagine you use the following SQL statement somewhere in your
program:

text

TX.exec(
	"SELECT number,amount "
	"FROM accounts "
	"WHERE allowed_to_see('" + userid + "','" + password + "')");

This shows a logged-in user important information on all accounts he is
authorized to view. The userid and password strings are variables entered
by the user himself.

Now, if the user is actually an attacker who knows (or can guess) the
general shape of this SQL statement, imagine he enters the following
password:

text

x') OR ('x' = 'x

Does that make sense to you? Probably not. But if this is inserted into
the SQL string by the C++ code above, the query becomes:

text

SELECT number,amount
FROM accounts
WHERE allowed_to_see('user','x') OR ('x' = 'x')

Using the esc functions

Here's how you can fix the problem in the example above:

text

TX.exec(
	"SELECT number,amount "
	"FROM accounts "
	"WHERE allowed_to_see('" + TX.esc(userid) + "', "
		"'" + TX.esc(password) + "')");

Now, the quotes embedded in the attacker's string will be neatly escaped so
they can't "break out" of the quoted SQL string they were meant to go into:

text

SELECT number,amount
FROM accounts
WHERE allowed_to_see('user', 'x'') OR (''x'' = ''x')

libpqxx {#mainpage}

@version 7.7.3
@author Jeroen T. Vermeulen
@see http://pqxx.org
@see https://github.com/jtv/libpqxx

Welcome to libpqxx, the C++ API to the PostgreSQL database management system.

For a quick introduction to installing and using libpqxx, see the README.md
file. The latest information can be found at http://pqxx.org/

Some links that should help you find your bearings:

@ref getting-started
@ref thread-safety
@ref connections
@ref transactions
@ref escaping
@ref performance
@ref transactor
@ref datatypes

Statement parameters {#parameters}

If you execute the query and pass parameter values, the call will respectively
substitute the first where it finds $1, the second where it finds $2, et
cetera.

Or if you forget to do that, you leave yourself open to horrible
SQL injection attacks. Trust me, I was born in a town
whose name started with an apostrophe!

Statement parameters save you this work. With these parameters you can pass
your values as-is, and they will go across the wire to the database in a safe
format.

Dynamic parameter lists

In rare cases you may just not know how many parameters you'll pass into your
statement when you call it.

For these situations, have a look at params. It lets you compose your
parameters list on the fly, even add whole ranges of parameters at a time.

You can pass a params into your statement as a normal parameter. It will
fill in all the parameter values it contains into that position of the
statement's overall parameter list.

You can mix static and dynamic parameters freely. Don't go overboard though:
complexity is where bugs happen!

Generating placeholders

(Generally if things get that complex, it's a good idea to look for simpler
solutions. But especially when performance matters, sometimes you can't avoid
complexity like that.)

text

pqxx::params values;
pqxx::placeholders name;

text

if (extra_clause)
{
  // Extend the query text, using the current placeholder.
  query += " AND x = " + name.get();
  // Add the parameter value.
  values.append(my_x);
  // Move on to the next placeholder value.
  name.next();
}

Depending on the starting value of name, this might add to query a fragment
like " AND x = $3" or " AND x = $5".

Performance features {#performance}

@ref streams. Use these as a faster way to transfer data between your
code and the database.
std::string_view and pqxx::zview. In places where traditional C++ worked
with std::string, see whether std::string_view or pqxx::zview will
do. Of course that means that you'll have to look at the data's lifetime
more carefully, but it'll save the computer a lot of copying.
@ref prepared. These can be executed many times without the server
parsing and planning them anew each time. They also save you having to
escape string parameters.
pqxx::pipeline lets you send queries to the database in batches, and
continue other processing while they are executing.
pqxx::connecting lets you start setting up a database connection, but
without blocking the thread.

As always of course, don't risk the quality of your code for optimizations
that you don't need!

Prepared statements {#prepared}

Prepared statements are SQL queries that you define once and then invoke
as many times as you like, typically with varying parameters. It's basically
a function that you can define ad hoc.

Preparing a statement

You create a prepared statement by preparing it on the connection (using the
pqxx::connection::prepare functions), passing an identifier and its SQL text.

The identifier is the name by which the prepared statement will be known; it
should consist of ASCII letters, digits, and underscores only, and start with
an ASCII letter. The name is case-sensitive.

cxx

void prepare_my_statement(pqxx::connection &c)
    {
      c.prepare(
          "my_statement",
          "SELECT * FROM Employee WHERE name = 'Xavier'");
    }

Once you've done this, you'll be able to call my_statement from any
transaction you execute on the same connection. For this, use the
pqxx::transaction_base::exec_prepared functions.

cxx

pqxx::result execute_my_statement(pqxx::transaction_base &t)
    {
      return t.exec_prepared("my_statement");
    }

Parameters

See @ref parameters for more about this. And here's a simple example of
preparing a statement and invoking it with parameters:

cxx

void prepare_find(pqxx::connection &c)
    {
      // Prepare a statement called "find" that looks for employees with a
      // given name (parameter 1) whose salary exceeds a given number
      // (parameter 2).
      c.prepare(
  	    "find",
  	    "SELECT * FROM Employee WHERE name = $1 AND salary > $2");
    }

This example looks up the prepared statement "find," passes name and
min_salary as parameters, and invokes the statement with those values:

cxx

pqxx::result execute_find(
      pqxx::transaction_base &t, std::string name, int min_salary)
    {
      return t.exec_prepared("find", name, min_salary);
    }

A special prepared statement

Performance note

Don't assume that using prepared statements will speed up your application.
There are cases where prepared statements are actually slower than plain SQL.

The reason is that the backend can often produce a better execution plan when
it knows the statement's actual parameter values.

Zero bytes

@warning Beware of "nul" bytes!

Any string you pass as a parameter will end at the first char with value
zero. If you pass a string that contains a zero byte, the last byte in the
value will be the one just before the zero.

Streams {#streams}

Null values

stream\_from

As you read a row, the stream converts its fields to a tuple type containing
the value types you ask for:

text

auto stream pqxx::stream_from::query(
    tx, "SELECT name, points FROM score");
std::tuple<std::string, int> row;
while (stream >> row)
  process(row);
stream.complete();

stream\_to

As with stream_from, you can specify the table and the columns, and not much
else. You insert tuple-like objects of your choice:

text

pqxx::stream_to stream{
    tx,
    "score",
    std::vector<std::string>{"name", "points"}};
for (auto const &entry: scores)
    stream << entry;
stream.complete();

Each row is processed as you provide it, and not retained in memory after that.

Thread safety {#thread-safety}

Most of the time this isn't hard. Result sets are immutable, so you can
share them between threads without problem. The main rule is:

That means: don't issue a query on a transaction while you're also opening
a subtransaction, don't access a cursor while you may also be committing,
and so on.

Accessing results and result rows {#accessing-results}

When you execute a query using one of the transaction exec functions, you
normally get a result object back. A result is a container of rows.

(There are exceptions. The exec1 functions expect exactly one row of data,
so they return just a row, not a full result.)

For example, your code might do:

cxx

pqxx::result r = tx.exec("SELECT * FROM mytable");

Now, how do you access the data inside r?

Result sets act as standard C++ containers of rows. Rows act as standard
C++ containers of fields. So the easiest way to go through them is:

cxx

for (auto const &row: r)
    {
       for (auto const &field: row) std::cout << field.c_str() << '\t';
       std::cout << '\n';
    }

But results and rows also support other kinds of access. Array-style
indexing, for instance, such as r[rownum]:

cxx

std::size_t const num_rows = std::size(r);
    for (std::size_t rownum=0u; rownum < num_rows; ++rownum)
    {
      pqxx::row const row = r[rownum];
      std::size_t const num_cols = std::size(row);
      for (std::size_t colnum=0u; colnum < num_cols; ++colnum)
      {
        pqxx::field const field = row[colnum];
        std::cout << field.c_str() << '\t';
      }

      std::cout << '\n';
    }

Every row in the result has the same number of columns, so you don't need to
look up the number of fields again for each one:

cxx

std::size_t const num_rows = std::size(r);
    std::size_t const num_cols = r.columns();
    for (std::size_t rownum=0u; rownum < num_rows; ++rownum)
    {
      pqxx::row const row = r[rownum];
      for (std::size_t colnum=0u; colnum < num_cols; ++colnum)
      {
        pqxx::field const field = row[colnum];
        std::cout << field.c_str() << '\t';
      }

      std::cout << '\n';
    }

You can even address a field by indexing the row using the field's name:

cxx

std::cout << row["salary"] << '\n';

For C++23 or better, there's also a two-dimensional array access operator:

cxx

for (std::size_t rownum=0u; rownum < num_rows; ++rownum)
    {
        for (std::size_t colnum=0u; colnum < num_cols; ++colnum)
	    std::cout result[rownum, colnum].c_str() << '\t';
	std::cout << '\n';
    }

And of course you can use classic "begin/end" loops:

cxx

for (auto row = std::begin(r); row != std::end(r); row++)
    {
      for (auto field = std::begin(row); field != std::end(row); field++)
        std::cout << field->c_str() << '\t';
      std::cout << '\n';
    }

Streaming rows

There's another way to go through the rows coming out of a query. It's
usually easier and faster, but there are drawbacks.

Now for the good news. Streaming does make it very easy to query data and loop
over it:

cxx

for (auto [id, name, x, y] :
        tx.stream<int, std::string_view, float, float>(
            "SELECT id, name, x, y FROM point"))
      process(id + 1, "point-" + name, x * 10.0, y * 10.0);

Binary data {#binary}

Generally you'll want to use BYTEA for reasonably-sized values, and large
objects for very large values.

So for example, if you want to write a large object, you'd create a
pqxx::blob object. And you might use that to write data in the form of
std::basic_string_view<std::byte>.

So long as it's basically still a block of bytes though, you can use
pqxx::binary_cast to construct a std::basic_string_view<std::byte> from it.

There are two forms of binary_cast. One takes a single argument that must
support std::data() and std::size():

text

std::string hi{"Hello binary world"};
my_blob.write(pqxx::binary_cast(hi);

The other takes a pointer and a size:

text

char const greeting[] = "Hello binary world";
char const *hi = greeting;
my_blob.write(pqxx::binary_cast(hi, sizeof(greeting)));

Caveats

There are some restrictions on binary_cast that you must be aware of.

Second, the data must be laid out as a contiguous block in memory. If there's
no std::data() implementation for your type, it's not suitable.

Supporting additional data types {#datatypes}

This is massively useful, but it's not for the faint of heart. You'll need to
specialise some templates. And, the API for doing this can change with any
major libpqxx release.

Converting types

In your application, a conversion is driven entirely by a C++ type you specify.
The value's SQL type has nothing to do with it, nor is there anything in the
string that would identify its type.

In some cases the templates for these conversions can tell the type from the
arguments you pass them:

text

auto x = to_string(99);

In other cases you may need to instantiate template explicitly:

text

auto y = from_string("99");

Supporting a new type

Let's say you have some other SQL type which you want to be able to store in,
or retrieve from, the database. What would it take to support that?

So what do you need for a complete conversion?

You also specialise the pqxx::type_name variable to specify the type's name.
This is important for all code which mentions your type in human-readable text,
such as error messages.

Then, does your type have a built-in null value? You specialise the
pqxx::nullness template to specify the details.

Finally, you specialise the pqxx::string_traits template. This is where you
define the actual conversions.

Let's go through these steps one by one.

Your type

You'll need a type for which the conversions are not yet defined, because the
C++ type is what determines the right conversion. One type, one set of
conversions.

The library also provides specialisations for std::optional<T>,
std::shared_ptr<T>, and std::unique_ptr<T>. If you have conversions for
T, you'll also have conversions for those.

Specialise `type_name`

When errors happen during conversion, libpqxx will compose error messages for
the user. Sometimes these will include the name of the type that's being
converted.

To tell libpqxx the name of each type, there's a template variable called
pqxx::type_name. For any given type T, it should have a specialisation
that provides that T's human-readable name:

text

namespace pqxx
{
template<> std::string const type_name<T>{"T"};
}

(Yes, this means that you need to define something inside the pqxx namespace.
Future versions of libpqxx may move this

Docker & Self-Hosted Deployment Verified OCI

Technical Documentation: zerotier/ZeroTierOne

Overview (`README.md`) Copy Section

ZeroTier - Global Area Networking

Quick Links Copy Chapter

About Copy Chapter

License Copy Chapter

Overview (`doc/README.md`) Copy Section

Manual Pages and Other Documentation

Overview (`ext/README.md`) Copy Section

Miscellaneous Stuff

Overview (`include/README.md`) Copy Section

ZeroTier Node API

Overview (`java/README.md`) Copy Section

ZeroTier One SDK - Android JNI Wrapper

Building Copy Chapter

Overview (`node/README.md`) Copy Section

ZeroTier Network Hypervisor Core

Overview (`osdep/README.md`) Copy Section

OS-Dependent and OS-Interface Things

Overview (`pkg/README.md`) Copy Section

Third-party packaging

Overview (`rule-compiler/README.md`) Copy Section

ZeroTier Rules Compiler

Overview (`service/README.md`) Copy Section

ZeroTier One Network Virtualization Service

Local Configuration File Copy Section

Network Virtualization Service API Copy Section

/status

/network

/network/\

/peer

/peer/\

Overview (`tcp-proxy/README.md`) Copy Section

TCP Proxy Server

How to run your own Copy Chapter

Build Copy Section

Point your node at it Copy Section

Option 1 - local.conf configuration

Option 2 - redirect 204.80.128.1 to your own IP

Overview (`windows/README.md`) Copy Section

Building Copy Chapter

Testing Copy Chapter

Overview (`ext/arm32-neon-salsa2012-asm/README.md`) Copy Section

ARM NEON (32-bit) ASM implementation of Salsa20/12

Overview (`ext/central-controller-docker/README.md`) Copy Section

ZeroTier Central Controller Docker Image

Overview (`ext/hiredis-0.14.1/README.md`) Copy Section

HIREDIS

IMPORTANT: Breaking changes when upgrading from 0.13.x -> 0.14.x Copy Chapter

Synchronous API Copy Chapter

Connecting Copy Section

Sending commands Copy Section

Using replies Copy Section

Cleaning up Copy Section

Sending commands (cont'd) Copy Section

Pipelining Copy Section

Errors Copy Section

Asynchronous API Copy Chapter

Connecting Copy Section

Sending commands and their callbacks Copy Section

Disconnecting Copy Section

Hooking it up to event library *X* Copy Section

Reply parsing API Copy Chapter

Usage Copy Section

Customizing replies Copy Section

Reader max buffer Copy Section

AUTHORS Copy Chapter

Overview (`ext/hiredis-1.0.2/README.md`) Copy Section

HIREDIS

Upgrading to `1.0.2` Copy Chapter

Upgrading to `1.0.0` Copy Chapter

IMPORTANT: Breaking changes from `0.14.1` -> `1.0.0` Copy Chapter

IMPORTANT: Breaking changes when upgrading from 0.13.x -> 0.14.x Copy Chapter

Upgrading from `<0.9.0` Copy Chapter

Synchronous API Copy Chapter

Connecting Copy Section

Sending commands Copy Section

Using replies Copy Section

RESP2 Copy Section

Overview (`README.md`)

Quick Links

About

License

Overview (`doc/README.md`)

Overview (`ext/README.md`)

Overview (`include/README.md`)

Overview (`java/README.md`)

Building

Overview (`node/README.md`)

Overview (`osdep/README.md`)

Overview (`pkg/README.md`)

Overview (`rule-compiler/README.md`)

Overview (`service/README.md`)

Local Configuration File

Network Virtualization Service API

Overview (`tcp-proxy/README.md`)

How to run your own

Build

Point your node at it

Overview (`windows/README.md`)

Building

Testing

Overview (`ext/arm32-neon-salsa2012-asm/README.md`)

Overview (`ext/central-controller-docker/README.md`)

Overview (`ext/hiredis-0.14.1/README.md`)

IMPORTANT: Breaking changes when upgrading from 0.13.x -> 0.14.x

Synchronous API

Connecting

Sending commands

Using replies

Cleaning up

Sending commands (cont'd)

Pipelining

Errors

Asynchronous API

Connecting

Sending commands and their callbacks

Disconnecting

Hooking it up to event library X

Reply parsing API

Usage

Customizing replies

Reader max buffer

AUTHORS

Overview (`ext/hiredis-1.0.2/README.md`)

Upgrading to `1.0.2`

Upgrading to `1.0.0`

IMPORTANT: Breaking changes from `0.14.1` -> `1.0.0`

IMPORTANT: Breaking changes when upgrading from 0.13.x -> 0.14.x

Upgrading from `<0.9.0`

Synchronous API

Connecting

Sending commands

Using replies

RESP2

RESP3

Cleaning up

Sending commands (cont'd)

Pipelining

Errors

Asynchronous API

Connecting

Sending commands and their callbacks

Disconnecting

Hooking it up to event library X

Reply parsing API

Usage

Customizing replies

Reader max buffer

Reader max array elements

SSL/TLS Support

Building

Using it

RESP3 PUSH replies

Default behavior

Custom PUSH handler prototypes

Installing a custom handler

Specifying no handler

Allocator injection