valkey Documentation

1. Project Overview & Quickstart (valkey-io/valkey) Copy Chapter

File: README.md Copy Chapter

This project was forked from the open source Redis project right before the transition to their new source available licenses.

This README is just a fast quick start document. More details can be found under valkey.io

What is Valkey?

Valkey is a high-performance data structure server that primarily serves key/value workloads.
It supports a wide range of native structures and an extensible plugin system for adding new data structures and access patterns.

Building Valkey using `Makefile`

Valkey can be compiled and used on Linux, macOS, OpenBSD, NetBSD, FreeBSD.
We support big endian and little endian architectures, and both 32 bit
and 64 bit systems.

It may compile on Solaris derived systems (for instance SmartOS) but our
support for this platform is best effort and Valkey is not guaranteed to
work as well as in Linux, macOS, and *BSD.

It is as simple as:

% make

To build with TLS support, you'll need OpenSSL development libraries (e.g.
libssl-dev on Debian/Ubuntu).

To build TLS support as Valkey built-in:

% make BUILD_TLS=yes

To build TLS as Valkey module:

% make BUILD_TLS=module

Note that sentinel mode does not support TLS module.

To build with experimental RDMA support you'll need RDMA development libraries
(e.g. librdmacm-dev and libibverbs-dev on Debian/Ubuntu).

To build RDMA support as Valkey built-in:

% make BUILD_RDMA=yes

To build RDMA as Valkey module:

% make BUILD_RDMA=module

To build with systemd support, you'll need systemd development libraries (such
as libsystemd-dev on Debian/Ubuntu or systemd-devel on CentOS) and run:

% make USE_SYSTEMD=yes

To build with enhanced stack traces that include file names and line numbers
for all functions (including static functions), use libbacktrace:

% make USE_LIBBACKTRACE=yes

To build Valkey without the Lua engine:

% make BUILD_LUA=no

To append a suffix to Valkey program names, use:

% make PROG_SUFFIX="-alt"

You can build a 32 bit Valkey binary using:

% make 32bit

After building Valkey, it is a good idea to test it using:

% make test

The above runs the main integration tests. Additional tests are started using:

% make test-unit     # Unit tests
% make test-modules  # Tests of the module API
% make test-sentinel # Valkey Sentinel integration tests
% make test-cluster  # Valkey Cluster integration tests

More about running the integration tests can be found in
tests/README.md and for unit tests, see
src/unit/README.md.

Performance monitoring Copy Chapter

Valkey Performance Dashboards provide a consolidated view of throughput trends across versions, helping contributors validate improvements and identify regressions quickly.

• Performance Overview - Compare throughput across Valkey versions
• Unstable Branch Dashboard - Track performance of all commits in the unstable branch

Fixing build problems with dependencies or cached build options Copy Chapter

Valkey has some dependencies which are included in the deps directory.
make does not automatically rebuild dependencies even if something in
the source code of dependencies changes.

When you update the source code with git pull or when code inside the
dependencies tree is modified in any other way, make sure to use the following
command in order to really clean everything and rebuild from scratch:

% make distclean

This will clean: jemalloc, lua, libvalkey, linenoise and other dependencies.

Also if you force certain build options like 32bit target, no C compiler
optimizations (for debugging purposes), and other similar build time options,
those options are cached indefinitely until you issue a make distclean
command.

Fixing problems building 32 bit binaries Copy Chapter

If after building Valkey with a 32 bit target you need to rebuild it
with a 64 bit target, or the other way around, you need to perform a
make distclean in the root directory of the Valkey distribution.

In case of build errors when trying to build a 32 bit binary of Valkey, try
the following steps:

• Install the package libc6-dev-i386 (also try g++-multilib).
• Try using the following command line instead of make 32bit:
  make CFLAGS="-m32 -march=native" LDFLAGS="-m32"

Allocator Copy Chapter

Selecting a non-default memory allocator when building Valkey is done by setting
the MALLOC environment variable. Valkey is compiled and linked against libc
malloc by default, with the exception of jemalloc being the default on Linux
systems. This default was picked because jemalloc has proven to have fewer
fragmentation problems than libc malloc.

To force compiling against libc malloc, use:

% make MALLOC=libc

To compile against jemalloc on Mac OS X systems, use:

% make MALLOC=jemalloc

Monotonic clock Copy Chapter

By default, Valkey uses the processor's internal instruction clock (TSC on x86,
CNTVCT on ARM) for monotonic time tracking, which provides approximately 3x
faster time access compared to POSIX clock_gettime (~10-30ns vs ~100ns).
This is enabled by default on supported architectures (x86_64 Linux and aarch64)
and automatically falls back to POSIX clock_gettime on unsupported systems.

For more information about processor clock usage, see:
http://oliveryang.net/2015/09/pitfalls-of-TSC-usage/

To disable the processor clock and force POSIX clock_gettime, use:

% make CFLAGS="-DNO_PROCESSOR_CLOCK"

Verbose build Copy Chapter

Valkey will build with a user-friendly colorized output by default.
If you want to see a more verbose output, use the following:

% make V=1

Running Valkey

To run Valkey with the default configuration, just type:

% cd src
% ./valkey-server

If you want to provide your valkey.conf, you have to run it using an additional
parameter (the path of the configuration file):

% cd src
% ./valkey-server /path/to/valkey.conf

It is possible to alter the Valkey configuration by passing parameters directly
as options using the command line. Examples:

% ./valkey-server --port 9999 --replicaof 127.0.0.1 6379
% ./valkey-server /etc/valkey/6379.conf --loglevel debug

All the options in valkey.conf are also supported as options using the command
line, with exactly the same name.

Running Valkey with TLS:

Running manually Copy Chapter

To manually run a Valkey server with TLS mode (assuming ./utils/gen-test-certs.sh
was invoked so sample certificates/keys are available):

• TLS built-in mode:

  text Copy

  ./src/valkey-server --tls-port 6379 --port 0 \
      --tls-cert-file ./tests/tls/valkey.crt \
      --tls-key-file ./tests/tls/valkey.key \
      --tls-ca-cert-file ./tests/tls/ca.crt

• TLS module mode:

  text Copy

  ./src/valkey-server --tls-port 6379 --port 0 \
      --tls-cert-file ./tests/tls/valkey.crt \
      --tls-key-file ./tests/tls/valkey.key \
      --tls-ca-cert-file ./tests/tls/ca.crt \
      --loadmodule src/valkey-tls.so

Note that you can disable TCP by specifying --port 0 explicitly.
It's also possible to have both TCP and TLS available at the same time,
but you'll have to assign different ports.

Use valkey-cli to connect to the Valkey server:

./src/valkey-cli --tls \
    --cert ./tests/tls/valkey.crt \
    --key ./tests/tls/valkey.key \
    --cacert ./tests/tls/ca.crt

Specifying --tls-replication yes makes a replica connect to the primary.

Using --tls-cluster yes makes Valkey Cluster use TLS across nodes.

Running Valkey with RDMA:

Note that Valkey Over RDMA is an experimental feature.
It may be changed or removed in any minor or major version.
Currently, it is only supported on Linux.

• RDMA built-in mode:

  text Copy

  ./src/valkey-server --protected-mode no \
       --rdma-bind 192.168.122.100 --rdma-port 6379

• RDMA module mode:

  text Copy

  ./src/valkey-server --protected-mode no \
       --loadmodule src/valkey-rdma.so --rdma-bind 192.168.122.100 --rdma-port 6379

It's possible to change bind address/port of RDMA by runtime command:

192.168.122.100:6379> CONFIG SET rdma-port 6380

It's also possible to have both RDMA and TCP available, and there is no
conflict of TCP(6379) and RDMA(6379), Ex:

% ./src/valkey-server --protected-mode no \
     --loadmodule src/valkey-rdma.so --rdma-bind 192.168.122.100 --rdma-port 6379 \
     --port 6379

Note that the network card (192.168.122.100 of this example) should support
RDMA. To test a server supports RDMA or not:

% rdma res show (a new version iproute2 package)

Or:

% ibv_devices

Playing with Valkey

You can use valkey-cli to play with Valkey. Start a valkey-server instance,
then in another terminal try the following:

% cd src
% ./valkey-cli
valkey> ping
PONG
valkey> set foo bar
OK
valkey> get foo
"bar"
valkey> incr mycounter
(integer) 1
valkey> incr mycounter
(integer) 2
valkey>

Installing Valkey

In order to install Valkey binaries into /usr/local/bin, just use:

% make install

You can use make PREFIX=/some/other/directory install if you wish to use a
different destination.

Note: For compatibility with Redis, we create symlinks from the Redis names (redis-server, redis-cli, etc.) to the Valkey binaries installed by make install.
The symlinks are created in same directory as the Valkey binaries.
The symlinks are removed when using make uninstall.
The creation of the symlinks can be skipped by setting the makefile variable USE_REDIS_SYMLINKS=no.

make install will just install binaries in your system, but will not configure
init scripts and configuration files in the appropriate place. This is not
needed if you just want to play a bit with Valkey, but if you are installing
it the proper way for a production system, we have a script that does this
for Ubuntu and Debian systems:

% cd utils
% ./install_server.sh

Note: install_server.sh will not work on macOS; it is built for Linux only.

The script will ask you a few questions and will setup everything you need
to run Valkey properly as a background daemon that will start again on
system reboots.

You'll be able to stop and start Valkey using the script named
/etc/init.d/valkey_, for instance /etc/init.d/valkey_6379.

Building using `CMake`

In addition to the traditional Makefile build, Valkey supports an alternative, experimental, build system using CMake.

To build and install Valkey, in Release mode (an optimized build), type this into your terminal:

mkdir build-release
cd $_
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/valkey
sudo make install
# Valkey is now installed under /opt/valkey

Other options supported by Valkey's CMake build system:

Special build flags Copy Chapter

• -DBUILD_TLS=<yes|no> enable TLS build for Valkey. Default: no
• -DBUILD_RDMA=<no|module> enable RDMA module build (only module mode supported). Default: no
• -DBUILD_MALLOC=<libc|jemalloc|tcmalloc|tcmalloc_minimal> choose the allocator to use. Default on Linux: jemalloc, for other OS: libc
• -DBUILD_SANITIZER=<address|thread|undefined> build with address sanitizer enabled. Default: disabled (no sanitizer)
• -DBUILD_UNIT_GTESTS=[yes|no] when set, the build will produce unit tests executable valkey-unit-gtests. Default: no
• -DBUILD_TEST_MODULES=[yes|no] when set, the build will include the modules located under the tests/modules folder. Default: no
• -DBUILD_EXAMPLE_MODULES=[yes|no] when set, the build will include the example modules located under the src/modules folder. Default: no

Common flags Copy Chapter

• -DCMAKE_BUILD_TYPE=<Debug|Release...> define the build type, see CMake manual for more details
• -DCMAKE_INSTALL_PREFIX=/installation/path override this value to define a custom install prefix. Default: /usr/local
• -G"<Generator Name>" generate build files for "Generator Name". By default, CMake will generate Makefiles.

Verbose build Copy Chapter

CMake generates a user-friendly colorized output by default.
If you want to see a more verbose output, use the following:

make VERBOSE=1

Troubleshooting Copy Chapter

During the CMake stage, CMake caches variables in a local file named CMakeCache.txt. All variables generated by Valkey
are removed from the cache once consumed (this is done by calling to unset(VAR-NAME CACHE)). However, some variables,
like the compiler path, are kept in cache. To start a fresh build either remove the cache file CMakeCache.txt from the
build folder, or delete the build folder completely.

It is important to re-run CMake when adding new source files.

Integration with IDE Copy Chapter

During the CMake stage of the build, CMake generates a JSON file named compile_commands.json and places it under the
build folder. This file is used by many IDEs and text editors for providing code completion (via clangd).

A small caveat is that these tools will look for compile_commands.json under the Valkey's top folder.
A common workaround is to create a symbolic link to it:

cd /path/to/valkey/
# We assume here that your build folder is `build-release`
ln -sf $(pwd)/build-release/compile_commands.json $(pwd)/compile_commands.json

Restart your IDE and voila

Code contributions

Please see the CONTRIBUTING.md. For security bugs and vulnerabilities, please see SECURITY.md.

Valkey is an open community project under LF Projects

Valkey a Series of LF Projects, LLC
2810 N Church St, PMB 57274
Wilmington, Delaware 19802-4447

File: deps/fpconv/README.md Copy Chapter

libfpconv

Fast and accurate double to string conversion based on Florian Loitsch's Grisu-algorithm1.

This port contains a subset of the 'C' version of Fast and accurate double to string conversion based on Florian Loitsch's Grisu-algorithm available at github.com/night-shift/fpconv).

1 https://www.cs.tufts.edu/~nr/cs257/archive/florian-loitsch/printf.pdf

File: deps/hdr_histogram/README.md Copy Chapter

HdrHistogram_c: 'C' port of High Dynamic Range (HDR) Histogram

HdrHistogram Copy Chapter

This port contains a subset of the functionality supported by the Java
implementation. The current supported features are:

• Standard histogram with 64 bit counts (32/16 bit counts not supported)
• All iterator types (all values, recorded, percentiles, linear, logarithmic)
• Histogram serialisation (encoding version 1.2, decoding 1.0-1.2)
• Reader/writer phaser and interval recorder

Features not supported, but planned

• Auto-resizing of histograms

Features unlikely to be implemented

• Double histograms
• Atomic/Concurrent histograms
• 16/32 bit histograms

Simple Tutorial

Recording values Copy Chapter

#include <hdr_histogram.h>

struct hdr_histogram* histogram;

// Initialise the histogram
hdr_init(
    1,  // Minimum value
    INT64_C(3600000000),  // Maximum value
    3,  // Number of significant figures
    &histogram)  // Pointer to initialise

// Record value
hdr_record_value(
    histogram,  // Histogram to record to
    value)  // Value to record

// Record value n times
hdr_record_values(
    histogram,  // Histogram to record to
    value,  // Value to record
    10)  // Record value 10 times

// Record value with correction for co-ordinated omission.
hdr_record_corrected_value(
    histogram,  // Histogram to record to
    value,  // Value to record
    1000)  // Record with expected interval of 1000.

// Print out the values of the histogram
hdr_percentiles_print(
    histogram,
    stdout,  // File to write to
    5,  // Granularity of printed values
    1.0,  // Multiplier for results
    CLASSIC);  // Format CLASSIC/CSV supported.

More examples Copy Chapter

For more detailed examples of recording and logging results look at the
hdr_decoder
and hiccup
examples. You can run hiccup and decoder
and pipe the results of one into the other.

$ ./examples/hiccup | ./examples/hdr_decoder

File: deps/libvalkey/docs/cluster.md Copy Chapter

Cluster API documentation

This document describes using libvalkey in cluster mode, including an overview of the synchronous and asynchronous APIs.
It is not intended as a complete reference. For that it's always best to refer to the source code.

Table of Contents Copy Chapter

• Synchronous API
  • Connecting
  • Connection options
  • Executing commands
  • Executing commands on a specific node
  • Disconnecting/cleanup
  • Pipelining
  • Events
• Asynchronous API
  • Connecting
  • Connection options
  • Executing commands
  • Executing commands on a specific node
  • Disconnecting/cleanup
  • Events
• Miscellaneous
  • TLS support
  • Cluster node iterator
  • Extend the list of supported commands
  • Random number generator

Synchronous API Copy Chapter

Connecting Copy Section

There are a few alternative ways to setup and connect to a cluster.
The basic alternatives lacks most options, but can be enough for some use cases.

valkeyClusterContext *valkeyClusterConnect(const char *addrs);
valkeyClusterContext *valkeyClusterConnectWithTimeout(const char *addrs,
                                                      const struct timeval tv);

There is also a convenience struct to specify various options.

valkeyClusterContext *valkeyClusterConnectWithOptions(const valkeyClusterOptions *options);

When connecting to a cluster, NULL is returned when the context can't be allocated, or err and errstr are set in the returned allocated context when there are issues.
So when connecting it's simple to handle error states.

valkeyClusterContext *cc = valkeyClusterConnect("127.0.0.1:6379,127.0.0.1:6380");
if (cc == NULL || cc->err) {
    fprintf(stderr, "Error: %s\n", cc ? cc->errstr : "OOM");
}

Connection options Copy Section

There are a variety of options you can specify using the valkeyClusterOptions struct when connecting to a cluster.
This includes information about how to connect to the cluster and defining optional callbacks and other options.
See include/valkey/cluster.h for more details.

valkeyClusterOptions opt = {
   .initial_nodes = "127.0.0.1:6379,127.0.0.1:6380"; // Addresses to initially connect to.
   .options = VALKEY_OPT_USE_CLUSTER_NODES;          // See available option flags below.
   .password = "password";                           // Authenticate connections using the `AUTH` command.
};

valkeyClusterContext *cc = valkeyClusterConnectWithOptions(&opt);
if (cc == NULL || cc->err) {
    fprintf(stderr, "Error: %s\n", cc ? cc->errstr : "OOM");
}

There are also several flags you can specify in valkeyClusterOptions.options. It's a bitwise OR of the following flags:

Executing commands Copy Section

The primary command interface is a printf-like function that takes a format string along with a variable number of arguments.
This will construct a RESP command and deliver it to the correct node in the cluster.

valkeyReply *reply = valkeyClusterCommand(cc, "SET %s %d", "counter", 42);

if (reply == NULL) {
    fprintf(stderr, "Communication error: %s\n", cc->err ? cc->errstr : "Unknown error");
} else if (reply->type == VALKEY_REPLY_ERROR) {
    fprintf(stderr, "Error response from node: %s\n", reply->str);
} else {
    // Handle reply..
}
freeReplyObject(reply);

Commands will be sent to the cluster node that the client perceives handling the given key.
If the cluster topology has changed the Valkey node might respond with a redirection error which the client will handle, update its slot map and resend the command to correct node.
The reply will in this case arrive from the correct node.

If a node is unreachable, for example if the command times out or if the connect times out, it can indicate that there has been a failover and the node is no longer part of the cluster.
In this case, valkeyClusterCommand returns NULL and sets err and errstr on the cluster context, but additionally, libvalkey schedules a slot map update to be performed when the next command is sent.
That means that if you try the same command again, there is a good chance the command will be sent to another node and the command may succeed.

Executing commands on a specific node Copy Section

When there is a need to send commands to a specific node, the following low-level API can be used.

valkeyReply *reply = valkeyClusterCommandToNode(cc, node, "DBSIZE");

This function handles printf-like arguments similar to valkeyClusterCommand, but will only attempt to send the command to the given node and will not perform redirects or retries.

If the command times out or the connection to the node fails, a slot map update is scheduled to be performed when the next command is sent.
valkeyClusterCommandToNode also performs a slot map update if it has previously been scheduled.

Disconnecting/cleanup Copy Section

To disconnect and free the context the following function can be used:

valkeyClusterFree(cc);

This function closes the sockets and deallocates the context.

Pipelining Copy Section

For pipelined commands, every command is simply appended to the output buffer but not delivered to the server, until you attempt to read the first response, at which point the entire buffer will be delivered.

The valkeyClusterAppendCommand function can be used to append a command, which is identical to the valkeyClusterCommand family, apart from not returning a reply.
After calling an append function valkeyClusterGetReply can be used to receive the subsequent replies.

The following example shows a simple cluster pipeline.

if (valkeyClusterAppendCommand(cc, "SET foo bar") != VALKEY_OK) {
    fprintf(stderr, "Error appending command: %s\n", cc->errstr);
    exit(1);
}

if (valkeyClusterAppendCommand(cc, "GET foo") != VALKEY_OK) {
    fprintf(stderr, "Error appending command: %s\n", cc->errstr);
    exit(1);
}

valkeyReply *reply;
if (valkeyClusterGetReply(cc,&reply) != VALKEY_OK) {
    fprintf(stderr, "Error reading reply %zu: %s\n", i, c->errstr);
    exit(1);
}
// Handle the reply for SET here.
freeReplyObject(reply);

if (valkeyClusterGetReply(cc,&reply) != VALKEY_OK) {
    fprintf(stderr, "Error reading reply %zu: %s\n", i, c->errstr);
    exit(1);
}
// Handle the reply for GET here.
freeReplyObject(reply);

Events Copy Section

Events per cluster context

There is a hook to get notified when certain events occur.

/* Function to be called when events occur. */
void event_cb(const valkeyClusterContext *cc, int event, void *privdata) {
    switch (event) {
       // Handle event
    }
}

valkeyClusterOptions opt = {
   .event_callback = event_cb;
   .event_privdata = my_privdata; // User defined data can be provided to the callback.
};
valkeyClusterContext *cc = valkeyClusterConnectWithOptions(&opt);

The callback is called with event set to one of the following values:

• VALKEYCLUSTER_EVENT_SLOTMAP_UPDATED when the slot mapping has been updated;
• VALKEYCLUSTER_EVENT_READY when the slot mapping has been fetched for the first
  time and the client is ready to accept commands, useful when initiating the
  client using valkeyClusterAsyncConnectWithOptions without enabling the option
  VALKEY_OPT_BLOCKING_INITIAL_UPDATE where a client is not immediately ready
  after a successful call;
• VALKEYCLUSTER_EVENT_FREE_CONTEXT when the cluster context is being freed, so
  that the user can free the event privdata.

Events per connection

There is a hook to get notified about connect and reconnect attempts.
This is useful for applying socket options or access endpoint information for a connection to a particular node.
The callback is registered using an option.

void connect_cb(const valkeyContext *c, int status) {
   // Perform desired action
}

valkeyClusterOptions opt = {
   .connect_callback = connect_cb;
};
valkeyClusterContext *cc = valkeyClusterConnectWithOptions(&opt);

The callback is called just after connect, before TLS handshake and authentication.

On successful connection, status is set to VALKEY_OK and the valkeyContext can be used, for example,
to see which IP and port it's connected to or to set socket options directly on the file descriptor which can be accessed as c->fd.

On failed connection attempt, this callback is called with status set to VALKEY_ERR.
The err field in the valkeyContext can be used to find out the cause of the error.

Asynchronous API Copy Chapter

The asynchronous API supports a wide range of event libraries and uses adapters to attach to a specific event library.
Each adapter provide a convenience function that configures which event loop instance the created context will be attached to.

Connecting Copy Section

To asynchronously connect to a cluster a valkeyClusterOptions should first be initiated with initial nodes and more,
but it's also important to configure which event library to use before calling valkeyClusterAsyncConnectWithOptions.

valkeyClusterOptions options = {
   .initial_nodes = "127.0.0.1:7000";
};

// Use convenience function to set which event library to use.
valkeyClusterOptionsUseLibev(&options, EV_DEFAULT);

// Initiate the context and start connecting to the initial nodes.
valkeyClusterAsyncContext *acc = valkeyClusterAsyncConnectWithOptions(&options);

Since an initial slot map update is performed asynchronously any command sent directly after valkeyClusterAsyncConnectWithOptions may fail
because the initial slot map has not yet been retrieved and the client doesn't know which cluster node to send the command to.
You may use the eventCallback to be notified when the slot map is updated and the client is ready to accept commands.
A crude example of using the eventCallback can be found in this test case.

Another option is to enable blocking initial slot map updates using the option VALKEY_OPT_BLOCKING_INITIAL_UPDATE.
When enabled valkeyClusterAsyncConnectWithOptions will initially connect to the cluster in a blocking fashion and wait for the slot map before returning.
Any command sent by the user thereafter will create a new non-blocking connection, unless a non-blocking connection already exists to the destination.
The function returns a pointer to a newly created valkeyClusterAsyncContext struct and its err field should be checked to make sure the initial slot map update was successful.

Connection options Copy Section

There is a variety of options you can specify using the valkeyClusterOptions struct when connecting to a cluster.

One asynchronous API specific option is VALKEY_OPT_BLOCKING_INITIAL_UPDATE which enables the initial slot map update to be performed in a blocking fashion.
The connect function will wait for a slot map update before returning so that the returned context is immediately ready to accept commands.

See previous Connection options section for common options.

Executing commands Copy Section

Executing commands in an asynchronous context work similarly to the synchronous context, except that you can pass a callback that will be invoked when the reply is received.

int status = valkeyClusterAsyncCommand(cc, commandCallback, privdata,
                                       "SET %s %s", "key", "value");

The return value is VALKEY_OK when the command was successfully added to the output buffer and VALKEY_ERR otherwise.
When the connection is being disconnected per user-request, no new commands may be added to the output buffer and VALKEY_ERR is returned.

Commands will be sent to the cluster node that the client perceives handling the given key.
If the cluster topology has changed the Valkey node might respond with a redirection error which the client will handle, update its slot map and resend the command to correct node.
The reply will in this case arrive from the correct node.

The reply callback, that is called when the reply is received, should have the following prototype:

void(valkeyClusterAsyncContext *acc, void *reply, void *privdata);

The privdata argument can be used to carry arbitrary data to the callback.

All pending callbacks are called with a NULL reply when the context encountered an error.

Executing commands on a specific node Copy Section

When there is a need to send commands to a specific node, the following low-level API can be used.

status = valkeyClusterAsyncCommandToNode(acc, node, commandCallback, privdata, "DBSIZE");

This functions will only attempt to send the command to a specific node and will not perform redirects or retries, but communication errors will trigger a slot map update just like the commonly used API.

Disconnecting/cleanup Copy Section

Asynchronous cluster connections can be terminated using:

valkeyClusterAsyncDisconnect(acc);

When this function is called, connections are not immediately terminated.
Instead, new commands are no longer accepted and connections are only terminated when all pending commands have been written to a socket, their respective replies have been read and their respective callbacks have been executed.
After this, the disconnection callback is executed with the VALKEY_OK status and the context object is freed.

Events Copy Section

Events per cluster context

Use event_callback in valkeyClusterOptions to get notified when certain events occur.

When the callback function requires the current valkeyClusterAsyncContext, it can typecast the given valkeyClusterContext to a valkeyClusterAsyncContext.
The valkeyClusterAsyncContext struct is an extension of the valkeyClusterContext struct.

void eventCallback(const valkeyClusterContext *cc, int event, void *privdata) {
   valkeyClusterAsyncContext *acc = (valkeyClusterAsyncContext *)cc;
}

Events per connection

Because the connections that will be created are non-blocking, the kernel is not able to instantly return if the specified host and port is able to accept a connection.
Instead, use a connect callback to be notified when a connection is established or failed.
Similarly, a disconnect callback can be used to be notified about a disconnected connection (either because of an error or per user request).
The callbacks can be enabled using the following options when calling valkeyClusterAsyncConnectWithOptions:

valkeyClusterOptions opt = {
   .async_connect_callback = connect_cb;
   .async_disconnect_callback = disconnect_cb;
}

The connect callback function should have the following prototype, aliased to valkeyConnectCallback:

void(valkeyAsyncContext *ac, int status);

On a connection attempt, the status argument is set to VALKEY_OK when the connection was successful.
The file description of the connection socket can be retrieved from a valkeyAsyncContext as ac->c->fd.

The disconnect callback function should have the following prototype, aliased to valkeyDisconnectCallback:

void(const valkeyAsyncContext *ac, int status);

On a disconnection the status argument is set to VALKEY_OK if it was initiated by the user, or to VALKEY_ERR when it was caused by an error.
When caused by an error the err field in the context can be accessed to get the error cause.
You don't need to reconnect in the disconnect callback since libvalkey will reconnect by itself when the next command is handled.

Miscellaneous Copy Chapter

TLS support Copy Section

TLS support is not enabled by default and requires an explicit build flag as described in README.md.

When support is enabled, TLS can be enabled on a cluster context using a prepared valkeyTLSContext and the options valkeyClusterOptions.tls and valkeyClusterOptions.tls_init_fn.

// Initialize the OpenSSL library.
valkeyInitOpenSSL();

// Initialize a valkeyTLSContext, which holds an OpenSSL context.
valkeyTLSContext *tls = valkeyCreateTLSContext("ca.crt", NULL, "client.crt",
                                               "client.key", NULL, NULL);
// Set options to enable TLS on context.
valkeyClusterOptions opt = {
   .tls = tls;
   .tls_init_fn = &valkeyInitiateTLSWithContext;
};

valkeyClusterContext *cc = valkeyClusterConnectWithOptions(&opt);
if (cc == NULL || cc->err) {
    fprintf(stderr, "Error: %s\n", cc ? cc->errstr : "OOM");
}

Cluster node iterator Copy Section

A valkeyClusterNodeIterator can be used to iterate on all known master nodes in a cluster context.
First it needs to be initiated using valkeyClusterInitNodeIterator and then you can repeatedly call valkeyClusterNodeNext to get the next node from the iterator.

valkeyClusterNodeIterator ni;
valkeyClusterInitNodeIterator(&ni, cc);

valkeyClusterNode *node;
while ((node = valkeyClusterNodeNext(&ni)) != NULL) {
   valkeyReply *reply = valkeyClusterCommandToNode(cc, node, "DBSIZE");
   // Handle reply..
}

The iterator will handle changes due to slot map updates by restarting the iteration, but on the new set of master nodes.
There is no bookkeeping for already iterated nodes when a restart is triggered, which means that a node can be iterated over more than once depending on when the slot map update happened and the change of cluster nodes.

Note that when valkeyClusterCommandToNode is called, a slot map update can happen if it has been scheduled by the previous command, for example if the previous call to valkeyClusterCommandToNode timed out or the node wasn't reachable.

To detect when the slot map has been updated, you can check if the slot map version (iter.route_version) is equal to the current cluster context's slot map version (cc->route_version).
If it isn't, it means that the slot map has been updated and the iterator will restart itself at the next call to valkeyClusterNodeNext.

Another way to detect that the slot map has been updated is to register an event callback and look for the event VALKEYCLUSTER_EVENT_SLOTMAP_UPDATED.

Extend the list of supported commands Copy Section

The list of commands and the position of the first key in the command line is defined in src/cmddef.h which is included in this repository.
It has been generated using the JSON files describing the syntax of each command in the Valkey repository, which makes sure we support all commands in Valkey, at least in terms of cluster routing.
To add support for custom commands defined in modules, you can regenerate cmddef.h using the script gencommands.py.
Use the JSON files from Valkey and any additional files on the same format as arguments to the script.
For details, see the comments inside gencommands.py.

Random number generator Copy Section

This library uses random() while selecting a node used for requesting the cluster topology (slot map).
A user should seed the random number generator using srandom() to get less predictability in the node selection.

File: deps/libvalkey/docs/migration-guide.md Copy Chapter

Migration guide

Libvalkey can replace both libraries hiredis and hiredis-cluster.
This guide highlights which APIs that have changed and what you need to do when migrating to libvalkey.

The general actions needed are:

• Replace the prefix redis with valkey in API usages.
• Replace the term SSL with TLS in API usages for secure communication.
• Update include paths depending on your previous installation.
  All libvalkey headers are now found under include/valkey/.
• Update used build options, e.g. USE_TLS replaces USE_SSL.

Migrating from `hiredis` v1.2.0 Copy Chapter

The type sds is removed from the public API.

Renamed API functions Copy Section

• redisAsyncSetConnectCallbackNC is renamed to valkeyAsyncSetConnectCallback.

Removed API functions Copy Section

• redisFormatSdsCommandArgv removed from API. Can be replaced with valkeyFormatCommandArgv.
• redisFreeSdsCommand removed since the sds type is for internal use only.
• redisAsyncSetConnectCallback is removed, but can be replaced with valkeyAsyncSetConnectCallback which accepts the non-const callback function prototype.

Renamed API defines Copy Section

• HIREDIS_MAJOR is renamed to LIBVALKEY_VERSION_MAJOR.
• HIREDIS_MINOR is renamed to LIBVALKEY_VERSION_MINOR.
• HIREDIS_PATCH is renamed to LIBVALKEY_VERSION_PATCH.

Removed API defines Copy Section

• HIREDIS_SONAME removed.

Migrating from `hiredis-cluster` 0.14.0 Copy Chapter

• The cluster client initiation procedure is changed and valkeyClusterOptions
  should be used to specify options when creating a context.
  See documentation for configuration examples when using the
  Synchronous API or the
  Asynchronous API.
  The examples directory also contains some common client
  initiation examples that might be helpful.
• The default command to update the internal slot map is changed to CLUSTER SLOTS.
  CLUSTER NODES can be re-enabled through options using VALKEY_OPT_USE_CLUSTER_NODES.
• A valkeyClusterAsyncContext now embeds a valkeyClusterContext instead of
  holding a pointer to it. Replace any use of acc->cc with &acc->cc or similar.

Renamed API functions Copy Section

• ctx_get_by_node is renamed to valkeyClusterGetValkeyContext.
• actx_get_by_node is renamed to valkeyClusterGetValkeyAsyncContext.

Renamed API defines Copy Section

• REDIS_ROLE_NULL is renamed to VALKEY_ROLE_UNKNOWN.
• REDIS_ROLE_MASTER is renamed to VALKEY_ROLE_PRIMARY.
• REDIS_ROLE_SLAVE is renamed to VALKEY_ROLE_REPLICA.

Removed API functions Copy Section

• redisClusterConnect2 removed, use valkeyClusterConnectWithOptions.
• redisClusterContextInit removed, use valkeyClusterConnectWithOptions.
• redisClusterSetConnectCallback removed, use valkeyClusterOptions.connect_callback.
• redisClusterSetEventCallback removed, use valkeyClusterOptions.event_callback.
• redisClusterSetMaxRedirect removed, use valkeyClusterOptions.max_retry.
• redisClusterSetOptionAddNode removed, use valkeyClusterOptions.initial_nodes.
• redisClusterSetOptionAddNodes removed, use valkeyClusterOptions.initial_nodes.
• redisClusterSetOptionConnectBlock removed since it was deprecated.
• redisClusterSetOptionConnectNonBlock removed since it was deprecated.
• redisClusterSetOptionConnectTimeout removed, use valkeyClusterOptions.connect_timeout.
• redisClusterSetOptionMaxRetry removed, use valkeyClusterOptions.max_retry.
• redisClusterSetOptionParseSlaves removed, use valkeyClusterOptions.options and VALKEY_OPT_USE_REPLICAS.
• redisClusterSetOptionPassword removed, use valkeyClusterOptions.password.
• redisClusterSetOptionRouteUseSlots removed, CLUSTER SLOTS is used by default.
• redisClusterSetOptionUsername removed, use valkeyClusterOptions.username.
• redisClusterAsyncConnect removed, use valkeyClusterAsyncConnectWithOptions with options flag VALKEY_OPT_BLOCKING_INITIAL_UPDATE.
• redisClusterAsyncConnect2 removed, use valkeyClusterAsyncConnectWithOptions.
• redisClusterAsyncContextInit removed, valkeyClusterAsyncConnectWithOptions will initiate the context.
• redisClusterAsyncSetConnectCallback removed, but valkeyClusterOptions.async_connect_callback can be used which accepts a non-const callback function prototype.
• redisClusterAsyncSetConnectCallbackNC removed, use valkeyClusterOptions.async_connect_callback.
• redisClusterAsyncSetDisconnectCallback removed, use valkeyClusterOptions.async_disconnect_callback.
• parse_cluster_nodes removed from API, for internal use only.
• parse_cluster_slots removed from API, for internal use only.

Removed API defines Copy Section

• HIRCLUSTER_FLAG_NULL removed.
• HIRCLUSTER_FLAG_ADD_SLAVE removed, flag can be replaced with an option, see VALKEY_OPT_USE_REPLICAS.
• HIRCLUSTER_FLAG_ROUTE_USE_SLOTS removed, the use of CLUSTER SLOTS is enabled by default.

Removed support for splitting multi-key commands per slot Copy Section

Since old days (from hiredis-vip) there has been support for sending some commands with multiple keys that covers multiple slots.
The client would split the command into multiple commands and send to each node handling each slot.
This was unnecessary complex and broke any expectations of atomicity.
Commands affected are DEL, EXISTS, MGET and MSET.

Proposed action:

Partition the keys by slot using valkeyClusterGetSlotByKey before sending affected commands.
Construct new commands when needed and send them using multiple calls to valkeyClusterCommand or equivalent.

File: deps/libvalkey/docs/standalone.md Copy Chapter

Standalone API documentation

This document describes using libvalkey in standalone (non-cluster) mode, including an overview of the synchronous and asynchronous APIs. It is not intended as a complete reference. For that it's always best to refer to the source code.

Table of Contents Copy Chapter

• Synchronous API
  • Connecting
  • Connection options
  • Executing commands
  • Using replies
  • Reply types
  • Disconnecting/cleanup
  • Pipelining
  • Errors
  • Thread safety
  • Reader configuration
    • Input buffer size
    • Maximum array elements
    • RESP3 Push Replies
    • Allocator injection
• Asynchronous API
  • Connecting
  • Executing commands
  • Disconnecting/cleanup
• TLS support

Synchronous API Copy Chapter

The synchronous API has a pretty small surface area, with only a few commands to use. In general they are very similar to the way printf works, except they construct RESP commands.

Connecting Copy Section

There are several convenience functions to connect in various ways (e.g. host and port, Unix socket, etc). See include/valkey/valkey.h for more details.

valkeyContext *valkeyConnect(const char *host, int port);
valkeyContext *valkeyConnectUnix(const char *path);

// There is also a convenience struct to specify various options.
valkeyContext *valkeyConnectWithOptions(valkeyOptions *opt);

When connecting to a server, libvalkey will return NULL in the event that we can't allocate the context, and set the err member if we can connect but there are issues. So when connecting it's simple to handle error states.

When a hostname resolves to multiple addresses, libvalkey will try each address in order until one succeeds or all have failed. Note that the connect_timeout applies per address, so the total connection time may be up to N × timeout when multiple addresses are unreachable.

valkeyContext *ctx = valkeyConnect("localhost", 6379);
if (ctx == NULL || ctx->err) {
    fprintf(stderr, "Error connecting: %s\n", ctx ? ctx->errstr : "OOM");
}

Connection options Copy Section

There are a variety of options you can specify when connecting to the server, which are delivered via the valkeyOptions helper struct. This includes information to connect to the server as well as other flags.

valkeyOptions opt = {0};

// You can set primary connection info
if (tcp)  {
    VALKEY_OPTIONS_SET_TCP(&opt, "localhost", 6379);
} else {
    VALKEY_OPTIONS_SET_UNIX(&opt, "/tmp/valkey.sock");
}

// You may attach any arbitrary data to the context
VALKEY_OPTIONS_SET_PRIVDATA(&opt, my_data);

There are also several flags you can specify when using the valkeyOptions helper struct.

Executing commands Copy Section

The primary command interface is a printf-like function that takes a format string along with a variable number of arguments. This will construct a RESP command and deliver it to the server.

valkeyReply *reply = valkeyCommand(ctx, "INCRBY %s %d", "counter", 42);

if (reply == NULL) {
    fprintf(stderr, "Communication error: %s\n", c->err ? c->errstr : "Unknown error");
} else if (reply->type == VALKEY_REPLY_ERROR) {
    fprintf(stderr, "Error response from server: %s\n", reply->str);
} else if (reply->type != VALKEY_REPLY_INTEGER) {
    // Very unlikely but should be checked.
    fprintf(stderr, "Error:  Non-integer reply to INCRBY?\n");
}

printf("New value of 'counter' is %lld\n", reply->integer);
freeReplyObject(reply);

If you need to deliver binary safe strings to the server, you can use the %b format specifier which requires you to pass the length as well.

struct binary { int x; int y; } = {0xdeadbeef, 0xcafebabe};
valkeyReply *reply = valkeyCommand(ctx, "SET %s %b", "some-key", &binary, sizeof(binary));

Commands may also be constructed by sending an array of arguments along with an optional array of their lengths. If lengths are not provided, libvalkey will execute strlen on each argument.

const char *argv[] = {"SET", "captain", "James Kirk"};
sonst size_t argvlens[] = {3, 7, 10};

valkeyReply *reply = valkeyCommandArgv(ctx, 3, argv, argvlens);
// Handle error conditions similarly to `valkeyCommand`

Using replies Copy Section

The valkeyCommand and valkeyCommandArgv functions return a valkeyReply on success and NULL in the event of a severe error (e.g. a communication failure with the server, out of memory condition, etc).

If the reply is NULL you can inspect the nature of the error by querying valkeyContext->err for the error code and valkeyContext->errstr for a human readable error string.

When a valkeyReply is returned, you should test the valkeyReply->type field to determine which kind of reply was received from the server. If for example there was an error in the command, this reply can be VALKEY_REPLY_ERROR and the specific error string will be in the reply->str member.

Reply types Copy Section

• VALKEY_REPLY_ERROR - An error reply. The error string is in reply->str.
• VALKEY_REPLY_STATUS - A status reply which will be in reply->str.
• VALKEY_REPLY_INTEGER - An integer reply, which will be in reply->integer.
• VALKEY_REPLY_DOUBLE - A double reply which will be in reply->dval as well as reply->str.
• VALKEY_REPLY_NIL - a nil reply.
• VALKEY_REPLY_BOOL - A boolean reply which will be in reply->integer.
• VALKEY_REPLY_BIGNUM - As of yet unused, but the string would be in reply->str.
• VALKEY_REPLY_STRING - A string reply which will be in reply->str.
• VALKEY_REPLY_VERB - A verbatim string reply which will be in reply->str and who's type will be in reply->vtype.
• VALKEY_REPLY_ARRAY - An array reply where each element is in reply->element with the number of elements in reply->elements.
• VALKEY_REPLY_MAP - A map reply, which structurally looks just like VALKEY_REPLY_ARRAY only is meant to represent keys and values. As with an array reply you can access the elements with reply->element and reply->elements.
• VALKEY_REPLY_SET - Another array-like reply representing a set (e.g. a reply from SMEMBERS). Access via reply->element and reply->elements.
• VALKEY_REPLY_ATTR - An attribute reply. As of yet unused by valkey-server.
• VALKEY_REPLY_PUSH - An out of band push reply. This is also array-like in nature.

Disconnecting/cleanup Copy Section

When libvalkey returns non-null valkeyReply struts you are responsible for freeing them with freeReplyObject. In order to disconnect and free the context simply call valkeyFree.

valkeyReply *reply = valkeyCommand(ctx, "set %s %s", "foo", "bar");
// Error handling ...
freeReplyObject(reply);

// Disconnect and free context
valkeyFree(ctx);

Pipelining Copy Section

valkeyCommand and valkeyCommandArgv each make a round-trip to the server, by sending the command and then waiting for a reply. Alternatively commands may be pipelined with the valkeyAppendCommand and valkeyAppendCommandArgv functions.

When you use valkeyAppendCommand the command is simply appended to the output buffer of valkeyContext but not delivered to the server, until you attempt to read the first response, at which point the entire buffer will be delivered.

// No data will be delivered to the server while these commands are being appended.
for (size_t i = 0; i < 100000; i++) {
    if (valkeyAppendCommand(c, "INCRBY key:%zu %zu", i, i) != VALKEY_OK) {
        fprintf(stderr, "Error appending command: %s\n", c->errstr);
        exit(1);
    }
}

// The entire output buffer will be delivered on the first call to `valkeyGetReply`.
for (size_t i = 0; i < 100000; i++) {
    if (valkeyGetReply(c, (void**)&reply) != VALKEY_OK) {
        fprintf(stderr, "Error reading reply %zu: %s\n", i, c->errstr);
        exit(1);
    } else if (reply->type != VALKEY_REPLY_INTEGER) {
        fprintf(stderr, "Error:  Non-integer reply to INCRBY?\n");
        exit(1);
    }

    printf("INCRBY key:%zu => %lld\n", i, reply->integer);
    freeReplyObject(reply);
}

valkeyGetReply can also be used in other contexts than pipeline, for example when you want to continuously block for commands for example in a subscribe context.

valkeyReply *reply = valkeyCommand(c, "SUBSCRIBE channel");
assert(reply != NULL && !c->err);

while (valkeyGetReply(c, (void**)&reply) == VALKEY_OK) {
    // Do something with the message...
    freeReplyObject(reply);
}

Errors Copy Section

As previously mentioned, when there is a communication error libvalkey will return NULL and set the err and errstr members with the nature of the problem. The specific error types are as follows.

• VALKEY_ERR_IO - A problem with the connection.
• VALKEY_ERR_EOF - The server closed the connection.
• VALKEY_ERR_PROTOCOL - There was an error parsing the reply.
• VALKEY_ERR_TIMEOUT - A connect, read, or write timeout.
• VALKEY_ERR_OOM - Out of memory.
• VALKEY_ERR_OTHER - Some other error (check c->errstr for details).

Thread safety Copy Section

Libvalkey context structs are not thread safe. You should not attempt to share them between threads, unless you really know what you're doing.

Reader configuration Copy Section

Libvalkey contexts have a few more mechanisms you can customize to your needs.

Maximum input buffer size

Libvalkey uses a buffer to hold incoming bytes, which is typically restored to the configurable max buffer size (16KB) when it is empty. To avoid continually reallocating this buffer you can set the value higher, or to zero which means "no limit".

context->reader->maxbuf = 0;

Maximum array elements

By default, libvalkey will refuse to parse array-like replies if they have more than 2^32-1 or 4,294,967,295 elements. This value can be set to any arbitrary 64-bit value or zero which just means "no limit".

context->reader->maxelements = 0;

RESP3 Push Replies

The RESP protocol introduced out-of-band "push" replies in the third version of the specification. These replies may come at any point in the data stream. By default, libvalkey will simply process these messages and discard them.

If your application needs to perform specific actions on PUSH messages you can install your own handler which will be called as they are received. It is also possible to set the push handler to NULL, in which case the messages will be delivered "in-band". This can be useful for example in a blocking subscribe loop.

NOTE: You may also specify a push handler in the valkeyOptions struct and set it on initialization .

Synchronous context

void my_push_handler(void *privdata, void *reply) {
    // In a synchronous context, you are expected to free the reply after you're done with it.
}

// Initialization, etc.
valkeySetPushCallback(c, my_push_handler);

Asynchronous context

void my_async_push_handler(valkeyAsyncContext *ac, void *reply) {
    // As with other async replies, libvalkey will free it for you, unless you have
    // configured the context with `VALKEY_OPT_NOAUTOFREE`.
}

// Initialization, etc
valkeyAsyncSetPushCallback(ac, my_async_push_handler);

Allocator injection

Internally libvalkey uses a layer of indirection from the standard allocation functions, by keeping a global structure with function pointers to the allocators we are going to use. By default they are just set to malloc, calloc, realloc, etc.

These can be overridden like so

valkeyAllocFuncs my_allocators = {
    .mallocFn = my_malloc,
    .callocFn = my_calloc,
    .reallocFn = my_realloc,
    .strdupFn = my_strdup,
    .freeFn = my_free,
};

// libvalkey will return the previously set allocators.
valkeyAllocFuncs old = valkeySetAllocators(&my_allocators);

They can also be reset to the glibc or musl defaults

valkeyResetAllocators();

NOTE: The vk_calloc function handles the case where nmemb * size would overflow a size_t and returns NULL in that case.

Asynchronous API Copy Chapter

Libvalkey also has an asynchronous API which supports a great many different event libraries. See the examples directory for specific information about each individual event library.

Connecting Copy Section

Libvalkey provides an valkeyAsyncContext to manage asynchronous connections which works similarly to the synchronous context.

valkeyAsyncContext *ac = valkeyAsyncConnect("localhost", 6379);
if (ac == NULL) {
    fprintf(stderr, "Error:  Out of memory trying to allocate valkeyAsyncContext\n");
    exit(1);
} else if (ac->err) {
    fprintf(stderr, "Error: %s (%d)\n", ac->errstr, ac->err);
    exit(1);
}

// If we're using libev
valkeyLibevAttach(EV_DEFAULT_ ac);

valkeySetConnectCallback(ac, my_connect_callback);
valkeySetDisconnectCallback(ac, my_disconnect_callback);

ev_run(EV_DEFAULT_ 0);

The asynchronous context should hold a connect callback function that is called when the connection attempt completes, either successfully or with an error.

It can also hold a disconnect callback function that is called when the connection is disconnected (either because of an error or per user request).
The context object is always freed after the disconnect callback fired.

Executing commands Copy Section

Executing commands in an asynchronous context work similarly to the synchronous context, except that you can pass a callback that will be invoked when the reply is received.

struct my_app_data {
    size_t incrby_replies;
    size_t get_replies;
};

void my_incrby_callback(valkeyAsyncContext *ac, void *r, void *privdata) {
    struct my_app_data *data = privdata;
    valkeyReply *reply = r;

    assert(reply != NULL && reply->type == VALKEY_REPLY_INTEGER);

    printf("Incremented value: %lld\n", reply->integer);
    data->incrby_replies++;
}

void my_get_callback(valkeyAsyncContext *ac, void *r, void *privdata) {
    struct my_app_data *data = privdata;
    valkeyReply *reply = r;

    assert(reply != NULL && reply->type == VALKEY_REPLY_STRING);

    printf("Key value: %s\n", reply->str);
    data->get_replies++;
}

int exec_some_commands(struct my_app_data *data) {
    valkeyAsyncCommand(ac, my_incrby_callback, data, "INCRBY mykey %d", 42);
    valkeyAsyncCommand(ac, my_get_callback, data, "GET %s", "mykey");
}

Disconnecting/cleanup Copy Section

For a graceful disconnect use valkeyAsyncDisconnect which will block new commands from being issued.
The connection is only terminated when all pending commands have been sent, their respective replies have been read, and their respective callbacks have been executed.
After this, the disconnection callback is called with the status, and the context object is freed.

To terminate the connection forcefully use valkeyAsyncFree which also will block new commands from being issued.
There will be no more data sent on the socket and all pending callbacks will be called with a NULL reply.
After this, the disconnection callback is called with the VALKEY_OK status, and the context object is freed.

TLS support Copy Chapter

TLS support is not enabled by default and requires an explicit build flag as described in README.md.

Libvalkey implements TLS on top of its valkeyContext and valkeyAsyncContext, so you will need to establish a connection first and then initiate a TLS handshake.
See the examples directory for how to create the TLS context and initiate the handshake.

File: deps/libvalkey/README.md Copy Chapter

Libvalkey

Libvalkey is the official C client for the Valkey database. It also supports any server that uses the RESP protocol (version 2 or 3). This project supports both standalone and cluster modes.

Table of Contents Copy Chapter

• Features
• Supported platforms
• Building
  • Building with make
  • Building with CMake
• Contributing
• Using the library
  • Standalone mode
  • Cluster mode
  • Migration guide

Features Copy Chapter

• Commands are executed in a generic way, with printf-like invocation.
• Supports both RESP2 and RESP3 protocol versions.
• Supports both synchronous and asynchronous operation.
• Optional support for MPTCP, TLS and RDMA connections.
• Asynchronous API with several event libraries to choose from.
• Supports both standalone and cluster mode operation.
• Can be compiled with either make or CMake.

Supported platforms Copy Chapter

This library supports and is tested against Linux, FreeBSD, macOS, and Windows. It should build and run on various proprietary Unixes although we can't run CI for those platforms. If you encounter any issues, please open an issue.

Building Copy Chapter

Libvalkey is written in C targeting C99. Unfortunately we have no plans on supporting C89 or earlier. The project does use a few widely supported compiler extensions, specifically for the bundled sds string library, although we have plans to remove this from the library.

We support plain GNU make and CMake. Following is information on how to build the library.

Building with make Copy Section

# Build and install the default library
sudo make install

# With all options
sudo USE_TLS=1 USE_RDMA=1 make install

# If your openssl is in a non-default location
sudo USE_TLS=1 OPENSSL_PREFIX=/path/to/openssl make install

Building with CMake Copy Section

See CMakeLists.txt for all available options.

# Build and install
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..
sudo make install

# Build with TLS and RDMA support
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_TLS=1 -DENABLE_RDMA=1 ..
sudo make install

Contributing Copy Chapter

Please see CONTRIBUTING.md.

File: deps/README.md Copy Chapter

This directory contains all Valkey dependencies, except for the libc that
should be provided by the operating system.

• Jemalloc is our memory allocator, used as replacement for libc malloc on Linux by default. It has good performances and excellent fragmentation behavior. This component is upgraded from time to time.
• libvalkey is the official C client library for Valkey. It is used by valkey-cli, valkey-benchmark and Valkey Sentinel. It is managed in a separate project and updated as needed.
• linenoise is a readline replacement. It is developed by the same authors of Valkey but is managed as a separated project and updated as needed.
• lua is Lua 5.1 with minor changes for security and additional libraries.
• LZ4 is the v1.10.0 streaming compression library.
• hdr_histogram Used for per-command latency tracking histograms.
• ffc.h is a C99 port of the fast_float library, used as a replacement for strtod to convert strings to floats efficiently.
• gtest-parallel is a script for running googletest tests in parallel.

How to upgrade the above dependencies

Jemalloc Copy Chapter

Jemalloc is used mostly unmodified from upstream. Active defragmentation relies
on jemalloc's native API rather than custom source patches (those were removed).
The only source-level modification is in include/jemalloc/jemalloc.sh, which
defines the VALKEY_VENDORED_JEMALLOC macro so Valkey can detect the vendored
copy at build time. A CMakeLists.txt is added for the Valkey build system.

We change Jemalloc settings via the configure script using the
--with-lg-quantum option, setting it to the value of 3 instead of 4. This
provides us with more size classes that better suit the Valkey data structures,
in order to gain memory efficiency.

Updating/upgrading jemalloc

The jemalloc directory is pulled as a subtree from the upstream jemalloc github repo. To update it you should run from the project root:

1. git subtree pull --prefix deps/jemalloc https://github.com/jemalloc/jemalloc.git <version-tag> --squash
   This should hopefully merge the local changes into the new version.
2. In case any conflicts arise you'll need to resolve them and commit.
3. Reconfigure jemalloc:

rm deps/jemalloc/VERSION deps/jemalloc/configure
cd deps/jemalloc
./autogen.sh --with-version=<version-tag>-0-g0

4. Update jemalloc's version in deps/Makefile: search for "--with-version=<old-version-tag>-0-g0" and update it accordingly.
5. Commit the changes (VERSION,configure,Makefile).

Libvalkey Copy Chapter

Libvalkey is used by Sentinel, valkey-cli and valkey-benchmark.
The library is built without its own version of the sds and dict type and uses the Valkey provided variant instead.

1. git subtree pull --prefix deps/libvalkey https://github.com/valkey-io/libvalkey.git <version-tag> --squash
   This should hopefully merge the local changes into the new version.
2. Commit the changes.

Linenoise Copy Chapter

Linenoise is rarely upgraded as needed. The upgrade process is trivial since
Valkey uses a non modified version of linenoise, so to upgrade just do the
following:

1. Remove the linenoise directory.
2. Substitute it with the new linenoise source tree.

LZ4 Copy Chapter

LZ4 is imported from the upstream release archive. The vendored version is
defined by the LZ4_VERSION_MAJOR, LZ4_VERSION_MINOR, and
LZ4_VERSION_RELEASE macros in lz4.h. Makefile and CMakeLists.txt are
maintained locally for the Valkey build, and xxHash symbols are namespaced to
avoid conflicts with modules loaded by Valkey.

To upgrade LZ4:

1. Download the new release archive from https://github.com/lz4/lz4/releases.
2. Replace lz4.c, lz4.h, lz4hc.c, lz4hc.h, lz4frame.c,
   lz4frame.h, xxhash.c, xxhash.h, and LICENSE with the versions from
   the release's lib directory.
3. Preserve the local Makefile and CMakeLists.txt, including the
   XXH_NAMESPACE definition.
4. Verify that the LZ4_VERSION_* macros in lz4.h match the imported release
   and update the version in the dependency summary above.

Lua Copy Chapter

We use Lua 5.1 and no upgrade is planned currently, since we don't want to break
Lua scripts for new Lua features: in the context of Valkey Lua scripts the
capabilities of 5.1 are usually more than enough, the release is rock solid,
and we definitely don't want to break old scripts.

So upgrading of Lua is up to the Valkey project maintainers and should be a
manual procedure performed by taking a diff between the different versions.

Currently we have at least the following differences between official Lua 5.1
and our version:

1. Makefile and CMakeLists.txt are modified for the Valkey build system.
2. We have the implementation source code, and directly link to the following external libraries: lua_cjson.o, lua_struct.o, lua_cmsgpack.o and lua_bit.o.
3. There is a security fix in ldo.c: The check for LUA_SIGNATURE[0] is removed in order to avoid direct bytecode execution.
4. In lstring.c, the luaS_newlstr function's hash calculation has been upgraded from a simple hash function to MurmurHash3, to enhance performance particularly for large strings.
5. Support for readonly tables (lua_enablereadonlytable API) is added across lapi.c, lvm.c, lobject.h, and ltable.c.
6. Protection of tables reachable from globals, with a globals whitelist.
7. Multiple CVE security patches (CVE-2022-24834, CVE-2024-31449, CVE-2024-31227, CVE-2024-31228) in cjson, cmsgpack, and the parser.

Hdr_Histogram Copy Chapter

Updated source can be found here: https://github.com/HdrHistogram/HdrHistogram_c
We use a customized version based on master branch commit e4448cf6d1cd08fff519812d3b1e58bd5a94ac42.

1. Compare all changes under /hdr_histogram directory to upstream master commit e4448cf6d1cd08fff519812d3b1e58bd5a94ac42
2. Copy updated files from newer version onto files in /hdr_histogram.
3. Apply the changes from 1 above to the updated files.

ffc.h Copy Chapter

ffc.h is a pure C99 port of the fast_float library, providing fast string-to-double
conversion without requiring a C++ compiler.

To upgrade the library,

1. Download the latest ffc.h from https://github.com/kolemannix/ffc.h/releases
2. Copy ffc.h to "deps/fast_float/".

gtest-parallel Copy Chapter

The deps/gtest-parallel directory is imported from the upstream
https://github.com/google/gtest-parallel repository as a subtree snapshot (not a real Git subtree).

Current upstream version:

• Upstream commit: cd488bd (from google/gtest-parallel)

Updating gtest-parallel

Run the following from the repository root.

1. Add the remote and fetch upstream:

   sh Copy

   git remote add gtest-parallel https://github.com/google/gtest-parallel.git
   git fetch gtest-parallel master

2. Remove any previous import and commit (commit A):

   sh Copy

   rm -rf deps/gtest-parallel

3. Update the subtree from upstream:

   sh Copy

   git subtree add --prefix=deps/gtest-parallel gtest-parallel master --squash

4. Reset back to commit A with proper sign-off:

   sh Copy

   git reset --soft <commit-A-hash>

5. Commit the changes.

File: src/commands/README.md Copy Chapter

Command JSON files

This directory contains JSON files, one for each command.

Each JSON file contains all the information about the command itself. It is the
"single source of truth" (SSOT) for the command's metadata.

These JSON files were originally not intended to be used directly, since they
contain internals and some fields like "acl_categories" that are not the final
ACL categories. (Valkey will apply some implicit rules to compute the final ACL
categories.) However, people see JSON files and use them directly anyway.

Any third party who needs access to command information were originally supposed
to get it from COMMAND INFO and COMMAND DOCS. These commands can be combined
into a JSON file by the script utils/generate-commands-json.py. Confusingly
enough, this JSON file as a slightly different format!

Structure Copy Chapter

Each JSON file contains an object with a single key. The key is the command name
in uppercase, e.g. "HSCAN" (hscan.json). The value is a JSON object with the
following keys. To be safe, assume all of them are optional.

• "summary": a string with a short description of the command. One sentence.
• "complexity": a string like "O(1)" or longer, like "O(1) for every call. O(N) for a complete iteration, including enough command calls for the cursor to return back to 0. N is the number of elements inside the collection.".
• "group": a string used for categorization in documentation. One of these:
  • "bitmap"
  • "cluster"
  • "connection"
  • "generic"
  • "geo"
  • "hash"
  • "hyperloglog"
  • "list"
  • "pubsub"
  • "scripting"
  • "sentinel"
  • "server"
  • "set"
  • "sorted_set"
  • "stream"
  • "string"
  • "transactions"
• "since": a string with a version number, like "7.0.0". It's the version
  (Redis OSS or Valkey) where the command was introduced.
• "arity": The number of arguments, including the command name itself. A
  negative number means "at least", e.g. -3 means at least 3.
• "container": Only present for subcommands. See below.
• "history": An array of changes, each change represented by a 2-element array
  on the form [VERSION, DESCRIPTION]. Omit if empty. Don't add an empty array.
• "function": The name of the C function in Valkey's source code implementing
  the command. (Don't use it for anything else.)
• "command_flags": An array of flags represented as strings. Command flags:
  • "ADMIN"
  • "ALL_DBS"
  • "ALLOW_BUSY"
  • "ASKING"
  • "BLOCKING"
  • "DENYOOM"
  • "FAST"
  • "LOADING"
  • "MAY_REPLICATE"
  • "NO_ASYNC_LOADING"
  • "NO_AUTH"
  • "NO_MANDATORY_KEYS"
  • "NO_MULTI"
  • "NOSCRIPT"
  • "ONLY_SENTINEL"
  • "PROTECTED"
  • "PUBSUB"
  • "READONLY"
  • "SENTINEL"
  • "SKIP_MONITOR"
  • "SKIP_SLOWLOG"
  • "STALE"
  • "TOUCHES_ARBITRARY_KEYS"
  • "WRITE"
• "acl_categories": A list of ACL categories in uppercase. Note that the
  effective ACL categories include "implicit ACL categories" explained below.
  • "ADMIN"
  • "BITMAP"
  • "CONNECTION"
  • "DANGEROUS"
  • "GEO"
  • "HASH"
  • "HYPERLOGLOG"
  • "KEYSPACE"
  • "LIST"
  • "SCRIPTING"
  • "SET"
  • "SORTEDSET"
  • "STREAM"
  • "STRING"
  • "TRANSACTION"
• "get_dbid_args": The name of the C function in Valkey's source code
  implementing retrieval of database ID arguments from commands that accept
  database ID as an argument.
• "command_tips": Optional. A list of one or more of these strings:
  • "NONDETERMINISTIC_OUTPUT"
  • "NONDETERMINISTIC_OUTPUT_ORDER"
  • "REQUEST_POLICY:ALL_NODES"
  • "REQUEST_POLICY:ALL_SHARDS"
  • "REQUEST_POLICY:MULTI_SHARD"
  • "REQUEST_POLICY:SPECIAL"
  • "RESPONSE_POLICY:AGG_LOGICAL_AND"
  • "RESPONSE_POLICY:AGG_MIN"
  • "RESPONSE_POLICY:AGG_SUM"
  • "RESPONSE_POLICY:ALL_SUCCEEDED"
  • "RESPONSE_POLICY:ONE_SUCCEEDED"
  • "RESPONSE_POLICY:SPECIAL"
• "key_specs": An array of key specifications. See below.
• "reply_schema": A JSON Schema that describes the
  reply of the command. This isn't complete. For example, JSON Schema can't
  distinguish arrays from sets, commands returning a set are declared to return
  an array.
• "arguments": An array of arguments. Each argument is an object with the following keys:
  • "name": A string identifying the argument. It's unique among the arguments.
  • "type": The type of the argument.
    • "block": A group of arguments. The elements are in the key "arguments".
    • "double": A number, not necessarily an integer.
    • "integer": An integer.
    • "key": A string representing a key in the database.
    • "oneof": One of a list of alternatives. The alternatives are in the key "arguments".
    • "pattern": A string representing a glob-style pattern.
    • "pure-token": A fixed string. The string is in the key "token".
    • "string": A string.
    • "unix-time": An integer representing a unix time in either seconds or milliseconds.
  • "arguments": A list with the same structure as its parent. Present if type is "block" or "oneof".
  • "display": ("entries-read", "key" or "pattern")
  • "key_spec_index": An index into the "key_specs" array. Only if "type" is "key".
  • "multiple": true if the argument can be repeated multiple times. Omitted means false.
  • "multiple_token": Unclear meaning. Maybe meaningless.
  • "optional": True if the argument is optional. Omitted means false.
  • "since": Version (string) when the argument was introduced.
  • "token": A string indicating a fixed string value. This is always present
    if type is "pure-token". If type is anything else, then "token" indicates
    the argument is preceded by an extra (fixed string) argument.

ACL categories Copy Chapter

The ACL categories specified as "acl_categories" are the ones that are actually used.
Note: commands categories should follow specific rules that are checked in utils/generate-command-code.py

ACL Category Rules:

• Command flag WRITE implies ACL category WRITE.
• Command flag READONLY and not ACL category SCRIPTING implies ACL category READ.
  "Exclude scripting commands from the RO category."
• Command flag ADMIN implies ACL categories ADMIN and DANGEROUS.
• Command flag PUBSUB implies ACL category PUBSUB.
• Command flag FAST implies ACL category FAST.
• Command flag BLOCKING implies ACL category BLOCKING.
• Not ACL category FAST implies ACL category SLOW. "If it's not fast, it's slow."

Key specs Copy Chapter

Key specifications are specified in the array "key_specs" key of a command.

Each element in this array is an object with the following keys:

• "flags": An array of strings indicating what kind of access is the command does on the key.
  • "ACCESS"
  • "DELETE"
  • "INCOMPLETE"
  • "INSERT"
  • "NOT_KEY"
  • "OW"
  • "RM"
  • "RO"
  • "RW"
  • "UPDATE"
  • "VARIABLE_FLAGS"
• "begin_search": How to find the first key used by this key spec. It's an
  object with only one key. The key determines the method for finding the first
  key. Here are the possible forms of the "begin_search" object:
  • {"index": {"pos": N}}: The first key is at position N in the command line,
    where 0 is the command name.
  • {"keyword": KEYWORD, "startfrom": N}: The first key is found by searching
    for an argument with the exact value KEYWORD starting from index N in the
    command line. The first key is the argument after the keyword.
  • {"unknown": null}: Finding the keys of this command is too complicated to
    explain.
• "find_keys": How to find the remaining keys of this key spec. It's an object
  on one of these forms:
  • {"range": {"lastkey": LAST, "step": STEP, "limit": LIMIT}}: A range of keys.
    • LAST: If LAST is positive, it's the index of the last key relative to the
      first key. If last is negative, then -1 is the end of the whole command
      line, -2 is the penultimate argument of the command, and so on.
    • STEP: The number of arguments to skip to find the next one. Typically 1.
    • LIMIT: If LAST is -1, we use the limit to stop the search by a factor. 0
      and 1 mean no limit. 2 means half of the remaining arguments, 3 means a
      third, and so on.

Commands with subcommands Copy Chapter

Commands with subcommands are special. Examples of commands with subcommands are
CLUSTER and ACL. Their first argument is a subcommand which determines the
syntax of the rest the command, which is stored in its own JSON file.

For example CLUSTER INFO is stored in a file called cluster-info.json. The
toplevel key is called "INFO". Within the body, there's a key called
"container" with the value "CLUSTER". The file cluster.json exists by it
doesn't have an "arguments" key.

Appendix Copy Chapter

How to list all the group, command_flags and acl_categories, etc. used in all these files:

cat *.json | jq '.[].group'             | grep -F '"' | sed 's/^ *//;s/, *$//;s/^/  * `/;s/$/`/' | sort | uniq
cat *.json | jq '.[].command_flags'     | grep -F '"' | sed 's/^ *//;s/, *$//;s/^/  * `/;s/$/`/' | sort | uniq
cat *.json | jq '.[].acl_categories'    | grep -F '"' | sed 's/^ *//;s/, *$//;s/^/  * `/;s/$/`/' | sort | uniq
cat *.json | jq '.[].arguments[]?.type' | grep -F '"' | sed 's/^ *//;s/, *$//;s/^/  * `/;s/$/`/' | sort | uniq

File: src/modules/lua/README.md Copy Chapter

Lua 5.1 Scripting Module

This module provides a Lua 5.1 scripting engine for Valkey, enabling server-side script execution with atomic operations and enhanced performance.

Overview Copy Chapter

The Lua module enables execution of Lua 5.1 scripts within Valkey using the EVAL and FCALL commands. This allows for:

• Atomic execution of complex operations
• Reduced network round trips
• Server-side data processing and transformation
• Custom command implementations

Scripts are executed in a sandboxed environment with access to Valkey's data structures and a subset of Lua's standard library.

For comprehensive information about using Lua scripts in Valkey, please refer to the Extending Valkey section of the official documentation site.

Building Copy Chapter

This module is automatically built as part of the Valkey core build process. However, you can also build it independently for development or testing purposes:

cd src/modules/lua
make

The independent build will create the module binary that can be loaded into Valkey using the MODULE LOAD command.

Dependencies Copy Chapter

This Lua module is designed to be mostly self-contained with minimal external dependencies.

Currently, the module has only two dependencies on Valkey core files:

• src/sha1.c - for SHA1 hash computation
• src/random.c - for random number generation

These dependencies were chosen to leverage existing Valkey functionality while maintaining modularity. Should the need arise to relocate this Lua module to a separate repository, these files can be easily copied alongside the module without introducing additional transitive dependencies.

When adding new dependencies in the future, please ensure they are lightweight and self-contained to preserve the module's portability and minimize complexity.

2. Official Technical Reference & Guides (valkey-io/valkey-io.github.io) Copy Chapter

Valkey.io website

This repo contains the source for the valkey.io website (build scripts, template, blog posts, stylesheets, etc.).
The build integrates content from valkey-io/valkey-doc and the commands definitions from valkey-io/valkey, valkey-io/valkey-bloom, and valkey-io/valkey-json (see Build Locally below for more details)

Contributing Copy Chapter

We welcome contributions! Please see our CONTRIBUTING page to learn more about how to contribute to the website.

Security Copy Chapter

If you discover potential security issues, see the reporting instructions on our CONTRIBUTING page for more information.

Build Locally Copy Chapter

This site is built with Zola.

Follow these steps to build the site locally:

1. Install Zola.
2. Switch to the directory with your fork of this repo.
3. Run zola serve

Open your browser to http://127.0.0.1:1111/

Zola will automatically rebuild on changes to the template or any content stored in this repo.
Changes to external content (command reference, documentation topics) require a restart of the Zola server process (ctrl-c then zola serve again, a browser refresh may also be needed).

Building additional content Copy Chapter

By default, the site will build without documentation topics, command reference, or the clients page.
The content for documentation topics and the clients page are stored within the valkey-io/valkey-doc repo.
The content for the command reference page is in the valkey-io/valkey repo.

If you want to build the site with this content, you'll need to have a local copy of valkey-io/valkey-doc and valkey-io/valkey outside of this repo.
Then follow the instructions to build the documentation topics and clients and/or build the command reference.
The instructions show how to use scripts that create symbolic links to the valkey-io/valkey-doc and valkey-io/valkey repos as well as create a series of empty stub files that tell Zola to create pages.

Building the documentation topics and clients page Copy Section

Documentation 'topics' (i.e. /topics/keyspace/, /topics/encryption/, /topics/transactions/) and the client libraries' data (i.e. /client-page-clients/nodejs/valkey-glide, /client-page-clients/python/valkey-py) sources content from valkey-io/valkey-doc.

flowchart TD
    A[Webpage: /topics/keyspace/ ]
    A --> B[Template: valkey-io/valkey-website]
    B --> H[Repo: valkey-io/valkey-doc ] --> I[File: /topics/keyspace.md ] --> Y[Topic content]

Let's say that this repo and your local copy of valkey-io/valkey-doc reside in the same directory.
First, stop the zola serve process if you're running it.
From the root directory of this repo run:

# You should only need to run this once or when you add a new topic/client.
./build/init-topics-and-clients.sh ../valkey-doc/topics ../valkey-doc/clients

Then, restart Zola.
Point your browser at http://127.0.0.1:1111/topics/ and you should see the fully populated list of topics and clients.
All files created in this process are ignored by git.
Commit your changes to your local copy of valkey-io/valkey-doc.

Building the command reference Copy Section

The command reference (i.e. /commands/set/, /commands/get/, /commands/lolwut/) sources information from valkey-io/valkey, valkey-io/valkey-bloom, and valkey-io/valkey-doc.
valkey-io/valkey, valkey-io/valkey-bloom, valkey-io/valkey-json and valkey-io/valkey-search provides the command metadata (items like computational complexity, version history, arguments, etc)
whilst valkey-io/valkey-doc provides the command description and the command reply.

flowchart TD
    A[Webpage: valkey.io/commands/set]
    A --> B[Template: valkey-io/valkey-website]
    B --> F[Repo: valkey-io/valkey ] --> G[File: /src/commands/set.json ] --> X[Command Metadata]
    B --> H[Repo: valkey-io/valkey-doc ] --> I[File: /commands/set.md ] --> Y[Command Description]
    H --> J[Files: /resp2_replies.json,/resp3_replies.json] --> Z[Command Reply]

Let's say that this repo and your local copy of valkey-io/valkey-doc, valkey-io/valkey-bloom, valkey-io/valkey-json, valkey-io/valkey-search
and valkey-io/valkey reside in the same directories.

First, stop the zola serve process if you're running it.
From the root directory of this repo run:

# You should only need to run this once or when you add a new command.
./build/init-commands.sh ../valkey-doc/commands ../valkey/src/commands \
    ../valkey-bloom/src/commands ../valkey-json/src/commands ../valkey-search/src/commands

Then, restart Zola.
Point your browser at http://127.0.0.1:1111/commands/ and you should see the fully populated list of topics.
All files created in this process are ignored by git.
Commit your changes to your local copy of valkey-io/valkey-doc for description changes and valkey-io/valkey for command JSON changes (if you have any).

License Copy Chapter

This project is licensed under the BSD-3-Clause License.