## 1. Project Overview & Quickstart (valkey-io/valkey) ## File: README.md [](https://codecov.io/gh/valkey-io/valkey) [](https://securityscorecards.dev/viewer/?uri=github.com/valkey-io/valkey) This project was forked from the open source Redis project right before the transition to their new source available licenses. This README is just a fast *quick start* document. More details can be found under [valkey.io](https://valkey.io/) # What is Valkey? Valkey is a high-performance data structure server that primarily serves key/value workloads. It supports a wide range of native structures and an extensible plugin system for adding new data structures and access patterns. # Building Valkey using `Makefile` Valkey can be compiled and used on Linux, macOS, OpenBSD, NetBSD, FreeBSD. We support big endian and little endian architectures, and both 32 bit and 64 bit systems. It may compile on Solaris derived systems (for instance SmartOS) but our support for this platform is *best effort* and Valkey is not guaranteed to work as well as in Linux, macOS, and \*BSD. It is as simple as: % make To build with TLS support, you'll need OpenSSL development libraries (e.g. libssl-dev on Debian/Ubuntu). To build TLS support as Valkey built-in: % make BUILD_TLS=yes To build TLS as Valkey module: % make BUILD_TLS=module Note that sentinel mode does not support TLS module. To build with experimental RDMA support you'll need RDMA development libraries (e.g. librdmacm-dev and libibverbs-dev on Debian/Ubuntu). To build RDMA support as Valkey built-in: % make BUILD_RDMA=yes To build RDMA as Valkey module: % make BUILD_RDMA=module To build with systemd support, you'll need systemd development libraries (such as libsystemd-dev on Debian/Ubuntu or systemd-devel on CentOS) and run: % make USE_SYSTEMD=yes To build with enhanced stack traces that include file names and line numbers for all functions (including static functions), use libbacktrace: % make USE_LIBBACKTRACE=yes To build Valkey without the Lua engine: % make BUILD_LUA=no To append a suffix to Valkey program names, use: % make PROG_SUFFIX="-alt" You can build a 32 bit Valkey binary using: % make 32bit After building Valkey, it is a good idea to test it using: % make test The above runs the main integration tests. Additional tests are started using: % make test-unit # Unit tests % make test-modules # Tests of the module API % make test-sentinel # Valkey Sentinel integration tests % make test-cluster # Valkey Cluster integration tests More about running the integration tests can be found in [tests/README.md](tests/README.md) and for unit tests, see [src/unit/README.md](src/unit/README.md). ## Performance monitoring Valkey Performance Dashboards provide a consolidated view of throughput trends across versions, helping contributors validate improvements and identify regressions quickly. - [Performance Overview](https://valkey.io/performance/) - Compare throughput across Valkey versions - [Unstable Branch Dashboard](https://perf-dashboard.valkey.io/public-dashboards/3e45bf8ded3043edaa941331cd1a94e2) - Track performance of all commits in the unstable branch ## Fixing build problems with dependencies or cached build options Valkey has some dependencies which are included in the `deps` directory. `make` does not automatically rebuild dependencies even if something in the source code of dependencies changes. When you update the source code with `git pull` or when code inside the dependencies tree is modified in any other way, make sure to use the following command in order to really clean everything and rebuild from scratch: % make distclean This will clean: jemalloc, lua, libvalkey, linenoise and other dependencies. Also if you force certain build options like 32bit target, no C compiler optimizations (for debugging purposes), and other similar build time options, those options are cached indefinitely until you issue a `make distclean` command. ## Fixing problems building 32 bit binaries If after building Valkey with a 32 bit target you need to rebuild it with a 64 bit target, or the other way around, you need to perform a `make distclean` in the root directory of the Valkey distribution. In case of build errors when trying to build a 32 bit binary of Valkey, try the following steps: * Install the package libc6-dev-i386 (also try g++-multilib). * Try using the following command line instead of `make 32bit`: `make CFLAGS="-m32 -march=native" LDFLAGS="-m32"` ## Allocator Selecting a non-default memory allocator when building Valkey is done by setting the `MALLOC` environment variable. Valkey is compiled and linked against libc malloc by default, with the exception of jemalloc being the default on Linux systems. This default was picked because jemalloc has proven to have fewer fragmentation problems than libc malloc. To force compiling against libc malloc, use: % make MALLOC=libc To compile against jemalloc on Mac OS X systems, use: % make MALLOC=jemalloc ## Monotonic clock By default, Valkey uses the processor's internal instruction clock (TSC on x86, CNTVCT on ARM) for monotonic time tracking, which provides approximately 3x faster time access compared to POSIX clock_gettime (~10-30ns vs ~100ns). This is enabled by default on supported architectures (x86_64 Linux and aarch64) and automatically falls back to POSIX clock_gettime on unsupported systems. For more information about processor clock usage, see: http://oliveryang.net/2015/09/pitfalls-of-TSC-usage/ To disable the processor clock and force POSIX clock_gettime, use: % make CFLAGS="-DNO_PROCESSOR_CLOCK" ## Verbose build Valkey will build with a user-friendly colorized output by default. If you want to see a more verbose output, use the following: % make V=1 # Running Valkey To run Valkey with the default configuration, just type: % cd src % ./valkey-server If you want to provide your valkey.conf, you have to run it using an additional parameter (the path of the configuration file): % cd src % ./valkey-server /path/to/valkey.conf It is possible to alter the Valkey configuration by passing parameters directly as options using the command line. Examples: % ./valkey-server --port 9999 --replicaof 127.0.0.1 6379 % ./valkey-server /etc/valkey/6379.conf --loglevel debug All the options in valkey.conf are also supported as options using the command line, with exactly the same name. # Running Valkey with TLS: ## Running manually To manually run a Valkey server with TLS mode (assuming `./utils/gen-test-certs.sh` was invoked so sample certificates/keys are available): * TLS built-in mode: ``` ./src/valkey-server --tls-port 6379 --port 0 \ --tls-cert-file ./tests/tls/valkey.crt \ --tls-key-file ./tests/tls/valkey.key \ --tls-ca-cert-file ./tests/tls/ca.crt ``` * TLS module mode: ``` ./src/valkey-server --tls-port 6379 --port 0 \ --tls-cert-file ./tests/tls/valkey.crt \ --tls-key-file ./tests/tls/valkey.key \ --tls-ca-cert-file ./tests/tls/ca.crt \ --loadmodule src/valkey-tls.so ``` Note that you can disable TCP by specifying `--port 0` explicitly. It's also possible to have both TCP and TLS available at the same time, but you'll have to assign different ports. Use `valkey-cli` to connect to the Valkey server: ``` ./src/valkey-cli --tls \ --cert ./tests/tls/valkey.crt \ --key ./tests/tls/valkey.key \ --cacert ./tests/tls/ca.crt ``` Specifying `--tls-replication yes` makes a replica connect to the primary. Using `--tls-cluster yes` makes Valkey Cluster use TLS across nodes. # Running Valkey with RDMA: Note that Valkey Over RDMA is an experimental feature. It may be changed or removed in any minor or major version. Currently, it is only supported on Linux. * RDMA built-in mode: ``` ./src/valkey-server --protected-mode no \ --rdma-bind 192.168.122.100 --rdma-port 6379 ``` * RDMA module mode: ``` ./src/valkey-server --protected-mode no \ --loadmodule src/valkey-rdma.so --rdma-bind 192.168.122.100 --rdma-port 6379 ``` It's possible to change bind address/port of RDMA by runtime command: 192.168.122.100:6379> CONFIG SET rdma-port 6380 It's also possible to have both RDMA and TCP available, and there is no conflict of TCP(6379) and RDMA(6379), Ex: % ./src/valkey-server --protected-mode no \ --loadmodule src/valkey-rdma.so --rdma-bind 192.168.122.100 --rdma-port 6379 \ --port 6379 Note that the network card (192.168.122.100 of this example) should support RDMA. To test a server supports RDMA or not: % rdma res show (a new version iproute2 package) Or: % ibv_devices # Playing with Valkey You can use valkey-cli to play with Valkey. Start a valkey-server instance, then in another terminal try the following: % cd src % ./valkey-cli valkey> ping PONG valkey> set foo bar OK valkey> get foo "bar" valkey> incr mycounter (integer) 1 valkey> incr mycounter (integer) 2 valkey> # Installing Valkey In order to install Valkey binaries into /usr/local/bin, just use: % make install You can use `make PREFIX=/some/other/directory install` if you wish to use a different destination. _Note_: For compatibility with Redis, we create symlinks from the Redis names (`redis-server`, `redis-cli`, etc.) to the Valkey binaries installed by `make install`. The symlinks are created in same directory as the Valkey binaries. The symlinks are removed when using `make uninstall`. The creation of the symlinks can be skipped by setting the makefile variable `USE_REDIS_SYMLINKS=no`. `make install` will just install binaries in your system, but will not configure init scripts and configuration files in the appropriate place. This is not needed if you just want to play a bit with Valkey, but if you are installing it the proper way for a production system, we have a script that does this for Ubuntu and Debian systems: % cd utils % ./install_server.sh _Note_: `install_server.sh` will not work on macOS; it is built for Linux only. The script will ask you a few questions and will setup everything you need to run Valkey properly as a background daemon that will start again on system reboots. You'll be able to stop and start Valkey using the script named `/etc/init.d/valkey_`, for instance `/etc/init.d/valkey_6379`. # Building using `CMake` In addition to the traditional `Makefile` build, Valkey supports an alternative, **experimental**, build system using `CMake`. To build and install `Valkey`, in `Release` mode (an optimized build), type this into your terminal: ```bash mkdir build-release cd $_ cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/valkey sudo make install # Valkey is now installed under /opt/valkey ``` Other options supported by Valkey's `CMake` build system: ## Special build flags - `-DBUILD_TLS=` enable TLS build for Valkey. Default: `no` - `-DBUILD_RDMA=` enable RDMA module build (only module mode supported). Default: `no` - `-DBUILD_MALLOC=` choose the allocator to use. Default on Linux: `jemalloc`, for other OS: `libc` - `-DBUILD_SANITIZER=` build with address sanitizer enabled. Default: disabled (no sanitizer) - `-DBUILD_UNIT_GTESTS=[yes|no]` when set, the build will produce unit tests executable `valkey-unit-gtests`. Default: `no` - `-DBUILD_TEST_MODULES=[yes|no]` when set, the build will include the modules located under the `tests/modules` folder. Default: `no` - `-DBUILD_EXAMPLE_MODULES=[yes|no]` when set, the build will include the example modules located under the `src/modules` folder. Default: `no` ## Common flags - `-DCMAKE_BUILD_TYPE=` define the build type, see CMake manual for more details - `-DCMAKE_INSTALL_PREFIX=/installation/path` override this value to define a custom install prefix. Default: `/usr/local` - `-G""` generate build files for "Generator Name". By default, CMake will generate `Makefile`s. ## Verbose build `CMake` generates a user-friendly colorized output by default. If you want to see a more verbose output, use the following: ```bash make VERBOSE=1 ``` ## Troubleshooting During the `CMake` stage, `CMake` caches variables in a local file named `CMakeCache.txt`. All variables generated by Valkey are removed from the cache once consumed (this is done by calling to `unset(VAR-NAME CACHE)`). However, some variables, like the compiler path, are kept in cache. To start a fresh build either remove the cache file `CMakeCache.txt` from the build folder, or delete the build folder completely. **It is important to re-run `CMake` when adding new source files.** ## Integration with IDE During the `CMake` stage of the build, `CMake` generates a JSON file named `compile_commands.json` and places it under the build folder. This file is used by many IDEs and text editors for providing code completion (via `clangd`). A small caveat is that these tools will look for `compile_commands.json` under the Valkey's top folder. A common workaround is to create a symbolic link to it: ```bash cd /path/to/valkey/ # We assume here that your build folder is `build-release` ln -sf $(pwd)/build-release/compile_commands.json $(pwd)/compile_commands.json ``` Restart your IDE and voila # Code contributions Please see the [CONTRIBUTING.md][2]. For security bugs and vulnerabilities, please see [SECURITY.md][3]. # Valkey is an open community project under LF Projects Valkey a Series of LF Projects, LLC 2810 N Church St, PMB 57274 Wilmington, Delaware 19802-4447 [1]: https://github.com/valkey-io/valkey/blob/unstable/COPYING [2]: https://github.com/valkey-io/valkey/blob/unstable/CONTRIBUTING.md [3]: https://github.com/valkey-io/valkey/blob/unstable/SECURITY.md --- ## File: deps/fpconv/README.md libfpconv ---------------------------------------------- Fast and accurate double to string conversion based on Florian Loitsch's Grisu-algorithm[1]. This port contains a subset of the 'C' version of Fast and accurate double to string conversion based on Florian Loitsch's Grisu-algorithm available at [github.com/night-shift/fpconv](https://github.com/night-shift/fpconv)). [1] https://www.cs.tufts.edu/~nr/cs257/archive/florian-loitsch/printf.pdf --- ## File: deps/hdr_histogram/README.md HdrHistogram_c: 'C' port of High Dynamic Range (HDR) Histogram HdrHistogram ---------------------------------------------- [](https://gitter.im/HdrHistogram/HdrHistogram) This port contains a subset of the functionality supported by the Java implementation. The current supported features are: * Standard histogram with 64 bit counts (32/16 bit counts not supported) * All iterator types (all values, recorded, percentiles, linear, logarithmic) * Histogram serialisation (encoding version 1.2, decoding 1.0-1.2) * Reader/writer phaser and interval recorder Features not supported, but planned * Auto-resizing of histograms Features unlikely to be implemented * Double histograms * Atomic/Concurrent histograms * 16/32 bit histograms # Simple Tutorial ## Recording values ```C #include struct hdr_histogram* histogram; // Initialise the histogram hdr_init( 1, // Minimum value INT64_C(3600000000), // Maximum value 3, // Number of significant figures &histogram) // Pointer to initialise // Record value hdr_record_value( histogram, // Histogram to record to value) // Value to record // Record value n times hdr_record_values( histogram, // Histogram to record to value, // Value to record 10) // Record value 10 times // Record value with correction for co-ordinated omission. hdr_record_corrected_value( histogram, // Histogram to record to value, // Value to record 1000) // Record with expected interval of 1000. // Print out the values of the histogram hdr_percentiles_print( histogram, stdout, // File to write to 5, // Granularity of printed values 1.0, // Multiplier for results CLASSIC); // Format CLASSIC/CSV supported. ``` ## More examples For more detailed examples of recording and logging results look at the [hdr_decoder](examples/hdr_decoder.c) and [hiccup](examples/hiccup.c) examples. You can run hiccup and decoder and pipe the results of one into the other. ``` $ ./examples/hiccup | ./examples/hdr_decoder ``` --- ## File: deps/libvalkey/docs/cluster.md # Cluster API documentation This document describes using `libvalkey` in cluster mode, including an overview of the synchronous and asynchronous APIs. It is not intended as a complete reference. For that it's always best to refer to the source code. ## Table of Contents - [Synchronous API](#synchronous-api) - [Connecting](#connecting) - [Connection options](#connection-options) - [Executing commands](#executing-commands) - [Executing commands on a specific node](#executing-commands-on-a-specific-node) - [Disconnecting/cleanup](#disconnecting-cleanup) - [Pipelining](#pipelining) - [Events](#events) - [Asynchronous API](#asynchronous-api) - [Connecting](#connecting-1) - [Connection options](#connection-options-1) - [Executing commands](#executing-commands-1) - [Executing commands on a specific node](#executing-commands-on-a-specific-node-1) - [Disconnecting/cleanup](#disconnecting-cleanup-1) - [Events](#events-1) - [Miscellaneous](#miscellaneous) - [TLS support](#tls-support) - [Cluster node iterator](#cluster-node-iterator) - [Extend the list of supported commands](#extend-the-list-of-supported-commands) - [Random number generator](#random-number-generator) ## Synchronous API ### Connecting There are a few alternative ways to setup and connect to a cluster. The basic alternatives lacks most options, but can be enough for some use cases. ```c valkeyClusterContext *valkeyClusterConnect(const char *addrs); valkeyClusterContext *valkeyClusterConnectWithTimeout(const char *addrs, const struct timeval tv); ``` There is also a convenience struct to specify various options. ```c valkeyClusterContext *valkeyClusterConnectWithOptions(const valkeyClusterOptions *options); ``` When connecting to a cluster, `NULL` is returned when the context can't be allocated, or `err` and `errstr` are set in the returned allocated context when there are issues. So when connecting it's simple to handle error states. ```c valkeyClusterContext *cc = valkeyClusterConnect("127.0.0.1:6379,127.0.0.1:6380"); if (cc == NULL || cc->err) { fprintf(stderr, "Error: %s\n", cc ? cc->errstr : "OOM"); } ``` ### Connection options There are a variety of options you can specify using the `valkeyClusterOptions` struct when connecting to a cluster. This includes information about how to connect to the cluster and defining optional callbacks and other options. See [include/valkey/cluster.h](../include/valkey/cluster.h) for more details. ```c valkeyClusterOptions opt = { .initial_nodes = "127.0.0.1:6379,127.0.0.1:6380"; // Addresses to initially connect to. .options = VALKEY_OPT_USE_CLUSTER_NODES; // See available option flags below. .password = "password"; // Authenticate connections using the `AUTH` command. }; valkeyClusterContext *cc = valkeyClusterConnectWithOptions(&opt); if (cc == NULL || cc->err) { fprintf(stderr, "Error: %s\n", cc ? cc->errstr : "OOM"); } ``` There are also several flags you can specify in `valkeyClusterOptions.options`. It's a bitwise OR of the following flags: | Flag | Description | | --- | --- | | `VALKEY_OPT_USE_CLUSTER_NODES` | Tells libvalkey to use the command `CLUSTER NODES` when updating its slot map (cluster topology).Libvalkey uses `CLUSTER SLOTS` by default. | | `VALKEY_OPT_USE_REPLICAS` | Tells libvalkey to keep parsed information of replica nodes. | | `VALKEY_OPT_BLOCKING_INITIAL_UPDATE` | **ASYNC**: Tells libvalkey to perform the initial slot map update in a blocking fashion. The function call will wait for a slot map update before returning so that the returned context is immediately ready to accept commands. | | `VALKEY_OPT_REUSEADDR` | Tells libvalkey to set the [SO_REUSEADDR](https://man7.org/linux/man-pages/man7/socket.7.html) socket option | | `VALKEY_OPT_PREFER_IPV4``VALKEY_OPT_PREFER_IPV6``VALKEY_OPT_PREFER_IP_UNSPEC` | Informs libvalkey to either prefer IPv4 or IPv6 when invoking [getaddrinfo](https://man7.org/linux/man-pages/man3/gai_strerror.3.html). `VALKEY_OPT_PREFER_IP_UNSPEC` will cause libvalkey to specify `AF_UNSPEC` in the getaddrinfo call, which means both IPv4 and IPv6 addresses will be searched simultaneously.Libvalkey prefers IPv4 by default. | | `VALKEY_OPT_MPTCP` | Tells libvalkey to use multipath TCP (MPTCP). Note that only when both the server and client are using MPTCP do they establish an MPTCP connection between them; otherwise, they use a regular TCP connection instead. | ### Executing commands The primary command interface is a `printf`-like function that takes a format string along with a variable number of arguments. This will construct a `RESP` command and deliver it to the correct node in the cluster. ```c valkeyReply *reply = valkeyClusterCommand(cc, "SET %s %d", "counter", 42); if (reply == NULL) { fprintf(stderr, "Communication error: %s\n", cc->err ? cc->errstr : "Unknown error"); } else if (reply->type == VALKEY_REPLY_ERROR) { fprintf(stderr, "Error response from node: %s\n", reply->str); } else { // Handle reply.. } freeReplyObject(reply); ``` Commands will be sent to the cluster node that the client perceives handling the given key. If the cluster topology has changed the Valkey node might respond with a redirection error which the client will handle, update its slot map and resend the command to correct node. The reply will in this case arrive from the correct node. If a node is unreachable, for example if the command times out or if the connect times out, it can indicate that there has been a failover and the node is no longer part of the cluster. In this case, `valkeyClusterCommand` returns NULL and sets `err` and `errstr` on the cluster context, but additionally, libvalkey schedules a slot map update to be performed when the next command is sent. That means that if you try the same command again, there is a good chance the command will be sent to another node and the command may succeed. ### Executing commands on a specific node When there is a need to send commands to a specific node, the following low-level API can be used. ```c valkeyReply *reply = valkeyClusterCommandToNode(cc, node, "DBSIZE"); ``` This function handles `printf`-like arguments similar to `valkeyClusterCommand`, but will only attempt to send the command to the given node and will not perform redirects or retries. If the command times out or the connection to the node fails, a slot map update is scheduled to be performed when the next command is sent. `valkeyClusterCommandToNode` also performs a slot map update if it has previously been scheduled. ### Disconnecting/cleanup To disconnect and free the context the following function can be used: ```c valkeyClusterFree(cc); ``` This function closes the sockets and deallocates the context. ### Pipelining For pipelined commands, every command is simply appended to the output buffer but not delivered to the server, until you attempt to read the first response, at which point the entire buffer will be delivered. The `valkeyClusterAppendCommand` function can be used to append a command, which is identical to the `valkeyClusterCommand` family, apart from not returning a reply. After calling an append function `valkeyClusterGetReply` can be used to receive the subsequent replies. The following example shows a simple cluster pipeline. ```c if (valkeyClusterAppendCommand(cc, "SET foo bar") != VALKEY_OK) { fprintf(stderr, "Error appending command: %s\n", cc->errstr); exit(1); } if (valkeyClusterAppendCommand(cc, "GET foo") != VALKEY_OK) { fprintf(stderr, "Error appending command: %s\n", cc->errstr); exit(1); } valkeyReply *reply; if (valkeyClusterGetReply(cc,&reply) != VALKEY_OK) { fprintf(stderr, "Error reading reply %zu: %s\n", i, c->errstr); exit(1); } // Handle the reply for SET here. freeReplyObject(reply); if (valkeyClusterGetReply(cc,&reply) != VALKEY_OK) { fprintf(stderr, "Error reading reply %zu: %s\n", i, c->errstr); exit(1); } // Handle the reply for GET here. freeReplyObject(reply); ``` ### Events #### Events per cluster context There is a hook to get notified when certain events occur. ```c /* Function to be called when events occur. */ void event_cb(const valkeyClusterContext *cc, int event, void *privdata) { switch (event) { // Handle event } } valkeyClusterOptions opt = { .event_callback = event_cb; .event_privdata = my_privdata; // User defined data can be provided to the callback. }; valkeyClusterContext *cc = valkeyClusterConnectWithOptions(&opt); ``` The callback is called with `event` set to one of the following values: * `VALKEYCLUSTER_EVENT_SLOTMAP_UPDATED` when the slot mapping has been updated; * `VALKEYCLUSTER_EVENT_READY` when the slot mapping has been fetched for the first time and the client is ready to accept commands, useful when initiating the client using `valkeyClusterAsyncConnectWithOptions` without enabling the option `VALKEY_OPT_BLOCKING_INITIAL_UPDATE` where a client is not immediately ready after a successful call; * `VALKEYCLUSTER_EVENT_FREE_CONTEXT` when the cluster context is being freed, so that the user can free the event `privdata`. #### Events per connection There is a hook to get notified about connect and reconnect attempts. This is useful for applying socket options or access endpoint information for a connection to a particular node. The callback is registered using an option. ```c void connect_cb(const valkeyContext *c, int status) { // Perform desired action } valkeyClusterOptions opt = { .connect_callback = connect_cb; }; valkeyClusterContext *cc = valkeyClusterConnectWithOptions(&opt); ``` The callback is called just after connect, before TLS handshake and authentication. On successful connection, `status` is set to `VALKEY_OK` and the `valkeyContext` can be used, for example, to see which IP and port it's connected to or to set socket options directly on the file descriptor which can be accessed as `c->fd`. On failed connection attempt, this callback is called with `status` set to `VALKEY_ERR`. The `err` field in the `valkeyContext` can be used to find out the cause of the error. ## Asynchronous API The asynchronous API supports a wide range of event libraries and uses [adapters](../include/valkey/adapters/) to attach to a specific event library. Each adapter provide a convenience function that configures which event loop instance the created context will be attached to. ### Connecting To asynchronously connect to a cluster a `valkeyClusterOptions` should first be initiated with initial nodes and more, but it's also important to configure which event library to use before calling `valkeyClusterAsyncConnectWithOptions`. ```c valkeyClusterOptions options = { .initial_nodes = "127.0.0.1:7000"; }; // Use convenience function to set which event library to use. valkeyClusterOptionsUseLibev(&options, EV_DEFAULT); // Initiate the context and start connecting to the initial nodes. valkeyClusterAsyncContext *acc = valkeyClusterAsyncConnectWithOptions(&options); ``` Since an initial slot map update is performed asynchronously any command sent directly after `valkeyClusterAsyncConnectWithOptions` may fail because the initial slot map has not yet been retrieved and the client doesn't know which cluster node to send the command to. You may use the [`eventCallback`](#events-per-cluster-context-1) to be notified when the slot map is updated and the client is ready to accept commands. A crude example of using the `eventCallback` can be found in [this test case](../tests/ct_async.c). Another option is to enable blocking initial slot map updates using the option `VALKEY_OPT_BLOCKING_INITIAL_UPDATE`. When enabled `valkeyClusterAsyncConnectWithOptions` will initially connect to the cluster in a blocking fashion and wait for the slot map before returning. Any command sent by the user thereafter will create a new non-blocking connection, unless a non-blocking connection already exists to the destination. The function returns a pointer to a newly created `valkeyClusterAsyncContext` struct and its `err` field should be checked to make sure the initial slot map update was successful. ### Connection options There is a variety of options you can specify using the `valkeyClusterOptions` struct when connecting to a cluster. One asynchronous API specific option is `VALKEY_OPT_BLOCKING_INITIAL_UPDATE` which enables the initial slot map update to be performed in a blocking fashion. The connect function will wait for a slot map update before returning so that the returned context is immediately ready to accept commands. See previous [Connection options](#connection-options) section for common options. ### Executing commands Executing commands in an asynchronous context work similarly to the synchronous context, except that you can pass a callback that will be invoked when the reply is received. ```c int status = valkeyClusterAsyncCommand(cc, commandCallback, privdata, "SET %s %s", "key", "value"); ``` The return value is `VALKEY_OK` when the command was successfully added to the output buffer and `VALKEY_ERR` otherwise. When the connection is being disconnected per user-request, no new commands may be added to the output buffer and `VALKEY_ERR` is returned. Commands will be sent to the cluster node that the client perceives handling the given key. If the cluster topology has changed the Valkey node might respond with a redirection error which the client will handle, update its slot map and resend the command to correct node. The reply will in this case arrive from the correct node. The reply callback, that is called when the reply is received, should have the following prototype: ```c void(valkeyClusterAsyncContext *acc, void *reply, void *privdata); ``` The `privdata` argument can be used to carry arbitrary data to the callback. All pending callbacks are called with a `NULL` reply when the context encountered an error. ### Executing commands on a specific node When there is a need to send commands to a specific node, the following low-level API can be used. ```c status = valkeyClusterAsyncCommandToNode(acc, node, commandCallback, privdata, "DBSIZE"); ``` This functions will only attempt to send the command to a specific node and will not perform redirects or retries, but communication errors will trigger a slot map update just like the commonly used API. ### Disconnecting/cleanup Asynchronous cluster connections can be terminated using: ```c valkeyClusterAsyncDisconnect(acc); ``` When this function is called, connections are **not** immediately terminated. Instead, new commands are no longer accepted and connections are only terminated when all pending commands have been written to a socket, their respective replies have been read and their respective callbacks have been executed. After this, the disconnection callback is executed with the `VALKEY_OK` status and the context object is freed. ### Events #### Events per cluster context Use [`event_callback` in `valkeyClusterOptions`](#events-per-cluster-context) to get notified when certain events occur. When the callback function requires the current `valkeyClusterAsyncContext`, it can typecast the given `valkeyClusterContext` to a `valkeyClusterAsyncContext`. The `valkeyClusterAsyncContext` struct is an extension of the `valkeyClusterContext` struct. ```c void eventCallback(const valkeyClusterContext *cc, int event, void *privdata) { valkeyClusterAsyncContext *acc = (valkeyClusterAsyncContext *)cc; } ``` #### Events per connection Because the connections that will be created are non-blocking, the kernel is not able to instantly return if the specified host and port is able to accept a connection. Instead, use a connect callback to be notified when a connection is established or failed. Similarly, a disconnect callback can be used to be notified about a disconnected connection (either because of an error or per user request). The callbacks can be enabled using the following options when calling `valkeyClusterAsyncConnectWithOptions`: ```c valkeyClusterOptions opt = { .async_connect_callback = connect_cb; .async_disconnect_callback = disconnect_cb; } ``` The connect callback function should have the following prototype, aliased to `valkeyConnectCallback`: ```c void(valkeyAsyncContext *ac, int status); ``` On a connection attempt, the `status` argument is set to `VALKEY_OK` when the connection was successful. The file description of the connection socket can be retrieved from a `valkeyAsyncContext` as `ac->c->fd`. The disconnect callback function should have the following prototype, aliased to `valkeyDisconnectCallback`: ```c void(const valkeyAsyncContext *ac, int status); ``` On a disconnection the `status` argument is set to `VALKEY_OK` if it was initiated by the user, or to `VALKEY_ERR` when it was caused by an error. When caused by an error the `err` field in the context can be accessed to get the error cause. You don't need to reconnect in the disconnect callback since libvalkey will reconnect by itself when the next command is handled. ## Miscellaneous ### TLS support TLS support is not enabled by default and requires an explicit build flag as described in [`README.md`](../README.md#building). When support is enabled, TLS can be enabled on a cluster context using a prepared `valkeyTLSContext` and the options `valkeyClusterOptions.tls` and `valkeyClusterOptions.tls_init_fn`. ```c // Initialize the OpenSSL library. valkeyInitOpenSSL(); // Initialize a valkeyTLSContext, which holds an OpenSSL context. valkeyTLSContext *tls = valkeyCreateTLSContext("ca.crt", NULL, "client.crt", "client.key", NULL, NULL); // Set options to enable TLS on context. valkeyClusterOptions opt = { .tls = tls; .tls_init_fn = &valkeyInitiateTLSWithContext; }; valkeyClusterContext *cc = valkeyClusterConnectWithOptions(&opt); if (cc == NULL || cc->err) { fprintf(stderr, "Error: %s\n", cc ? cc->errstr : "OOM"); } ``` ### Cluster node iterator A `valkeyClusterNodeIterator` can be used to iterate on all known master nodes in a cluster context. First it needs to be initiated using `valkeyClusterInitNodeIterator` and then you can repeatedly call `valkeyClusterNodeNext` to get the next node from the iterator. ```c valkeyClusterNodeIterator ni; valkeyClusterInitNodeIterator(&ni, cc); valkeyClusterNode *node; while ((node = valkeyClusterNodeNext(&ni)) != NULL) { valkeyReply *reply = valkeyClusterCommandToNode(cc, node, "DBSIZE"); // Handle reply.. } ``` The iterator will handle changes due to slot map updates by restarting the iteration, but on the new set of master nodes. There is no bookkeeping for already iterated nodes when a restart is triggered, which means that a node can be iterated over more than once depending on when the slot map update happened and the change of cluster nodes. Note that when `valkeyClusterCommandToNode` is called, a slot map update can happen if it has been scheduled by the previous command, for example if the previous call to `valkeyClusterCommandToNode` timed out or the node wasn't reachable. To detect when the slot map has been updated, you can check if the slot map version (`iter.route_version`) is equal to the current cluster context's slot map version (`cc->route_version`). If it isn't, it means that the slot map has been updated and the iterator will restart itself at the next call to `valkeyClusterNodeNext`. Another way to detect that the slot map has been updated is to [register an event callback](#events-per-cluster-context) and look for the event `VALKEYCLUSTER_EVENT_SLOTMAP_UPDATED`. ### Extend the list of supported commands The list of commands and the position of the first key in the command line is defined in [`src/cmddef.h`](../src/cmddef.h) which is included in this repository. It has been generated using the `JSON` files describing the syntax of each command in the Valkey repository, which makes sure we support all commands in Valkey, at least in terms of cluster routing. To add support for custom commands defined in modules, you can regenerate `cmddef.h` using the script [`gencommands.py`](../script/gencommands.py). Use the `JSON` files from Valkey and any additional files on the same format as arguments to the script. For details, see the comments inside `gencommands.py`. ### Random number generator This library uses [random()](https://linux.die.net/man/3/random) while selecting a node used for requesting the cluster topology (slot map). A user should seed the random number generator using [`srandom()`](https://linux.die.net/man/3/srandom) to get less predictability in the node selection. --- ## File: deps/libvalkey/docs/migration-guide.md # Migration guide Libvalkey can replace both libraries `hiredis` and `hiredis-cluster`. This guide highlights which APIs that have changed and what you need to do when migrating to libvalkey. The general actions needed are: * Replace the prefix `redis` with `valkey` in API usages. * Replace the term `SSL` with `TLS` in API usages for secure communication. * Update include paths depending on your previous installation. All `libvalkey` headers are now found under `include/valkey/`. * Update used build options, e.g. `USE_TLS` replaces `USE_SSL`. ## Migrating from `hiredis` v1.2.0 The type `sds` is removed from the public API. ### Renamed API functions * `redisAsyncSetConnectCallbackNC` is renamed to `valkeyAsyncSetConnectCallback`. ### Removed API functions * `redisFormatSdsCommandArgv` removed from API. Can be replaced with `valkeyFormatCommandArgv`. * `redisFreeSdsCommand` removed since the `sds` type is for internal use only. * `redisAsyncSetConnectCallback` is removed, but can be replaced with `valkeyAsyncSetConnectCallback` which accepts the non-const callback function prototype. ### Renamed API defines * `HIREDIS_MAJOR` is renamed to `LIBVALKEY_VERSION_MAJOR`. * `HIREDIS_MINOR` is renamed to `LIBVALKEY_VERSION_MINOR`. * `HIREDIS_PATCH` is renamed to `LIBVALKEY_VERSION_PATCH`. ### Removed API defines * `HIREDIS_SONAME` removed. ## Migrating from `hiredis-cluster` 0.14.0 * The cluster client initiation procedure is changed and `valkeyClusterOptions` should be used to specify options when creating a context. See documentation for configuration examples when using the [Synchronous API](cluster.md#synchronous-api) or the [Asynchronous API](cluster.md#asynchronous-api). The [examples](../examples/) directory also contains some common client initiation examples that might be helpful. * The default command to update the internal slot map is changed to `CLUSTER SLOTS`. `CLUSTER NODES` can be re-enabled through options using `VALKEY_OPT_USE_CLUSTER_NODES`. * A `valkeyClusterAsyncContext` now embeds a `valkeyClusterContext` instead of holding a pointer to it. Replace any use of `acc->cc` with `&acc->cc` or similar. ### Renamed API functions * `ctx_get_by_node` is renamed to `valkeyClusterGetValkeyContext`. * `actx_get_by_node` is renamed to `valkeyClusterGetValkeyAsyncContext`. ### Renamed API defines * `REDIS_ROLE_NULL` is renamed to `VALKEY_ROLE_UNKNOWN`. * `REDIS_ROLE_MASTER` is renamed to `VALKEY_ROLE_PRIMARY`. * `REDIS_ROLE_SLAVE` is renamed to `VALKEY_ROLE_REPLICA`. ### Removed API functions * `redisClusterConnect2` removed, use `valkeyClusterConnectWithOptions`. * `redisClusterContextInit` removed, use `valkeyClusterConnectWithOptions`. * `redisClusterSetConnectCallback` removed, use `valkeyClusterOptions.connect_callback`. * `redisClusterSetEventCallback` removed, use `valkeyClusterOptions.event_callback`. * `redisClusterSetMaxRedirect` removed, use `valkeyClusterOptions.max_retry`. * `redisClusterSetOptionAddNode` removed, use `valkeyClusterOptions.initial_nodes`. * `redisClusterSetOptionAddNodes` removed, use `valkeyClusterOptions.initial_nodes`. * `redisClusterSetOptionConnectBlock` removed since it was deprecated. * `redisClusterSetOptionConnectNonBlock` removed since it was deprecated. * `redisClusterSetOptionConnectTimeout` removed, use `valkeyClusterOptions.connect_timeout`. * `redisClusterSetOptionMaxRetry` removed, use `valkeyClusterOptions.max_retry`. * `redisClusterSetOptionParseSlaves` removed, use `valkeyClusterOptions.options` and `VALKEY_OPT_USE_REPLICAS`. * `redisClusterSetOptionPassword` removed, use `valkeyClusterOptions.password`. * `redisClusterSetOptionRouteUseSlots` removed, `CLUSTER SLOTS` is used by default. * `redisClusterSetOptionUsername` removed, use `valkeyClusterOptions.username`. * `redisClusterAsyncConnect` removed, use `valkeyClusterAsyncConnectWithOptions` with options flag `VALKEY_OPT_BLOCKING_INITIAL_UPDATE`. * `redisClusterAsyncConnect2` removed, use `valkeyClusterAsyncConnectWithOptions`. * `redisClusterAsyncContextInit` removed, `valkeyClusterAsyncConnectWithOptions` will initiate the context. * `redisClusterAsyncSetConnectCallback` removed, but `valkeyClusterOptions.async_connect_callback` can be used which accepts a non-const callback function prototype. * `redisClusterAsyncSetConnectCallbackNC` removed, use `valkeyClusterOptions.async_connect_callback`. * `redisClusterAsyncSetDisconnectCallback` removed, use `valkeyClusterOptions.async_disconnect_callback`. * `parse_cluster_nodes` removed from API, for internal use only. * `parse_cluster_slots` removed from API, for internal use only. ### Removed API defines * `HIRCLUSTER_FLAG_NULL` removed. * `HIRCLUSTER_FLAG_ADD_SLAVE` removed, flag can be replaced with an option, see `VALKEY_OPT_USE_REPLICAS`. * `HIRCLUSTER_FLAG_ROUTE_USE_SLOTS` removed, the use of `CLUSTER SLOTS` is enabled by default. ### Removed support for splitting multi-key commands per slot Since old days (from `hiredis-vip`) there has been support for sending some commands with multiple keys that covers multiple slots. The client would split the command into multiple commands and send to each node handling each slot. This was unnecessary complex and broke any expectations of atomicity. Commands affected are `DEL`, `EXISTS`, `MGET` and `MSET`. _Proposed action:_ Partition the keys by slot using `valkeyClusterGetSlotByKey` before sending affected commands. Construct new commands when needed and send them using multiple calls to `valkeyClusterCommand` or equivalent. --- ## File: deps/libvalkey/docs/standalone.md # Standalone API documentation This document describes using `libvalkey` in standalone (non-cluster) mode, including an overview of the synchronous and asynchronous APIs. It is not intended as a complete reference. For that it's always best to refer to the source code. ## Table of Contents - [Synchronous API](#synchronous-api) - [Connecting](#connecting) - [Connection options](#connection-options) - [Executing commands](#executing-commands) - [Using replies](#using-replies) - [Reply types](#reply-types) - [Disconnecting/cleanup](#disconnecting-cleanup) - [Pipelining](#pipelining) - [Errors](#errors) - [Thread safety](#thread-safety) - [Reader configuration](#reader-configuration) - [Input buffer size](#maximum-input-buffer-size) - [Maximum array elements](#maximum-array-elements) - [RESP3 Push Replies](#resp3-push-replies) - [Allocator injection](#allocator-injection) - [Asynchronous API](#asynchronous-api) - [Connecting](#connecting-1) - [Executing commands](#executing-commands-1) - [Disconnecting/cleanup](#disconnecting-cleanup-1) - [TLS support](#tls-support) ## Synchronous API The synchronous API has a pretty small surface area, with only a few commands to use. In general they are very similar to the way printf works, except they construct `RESP` commands. ### Connecting There are several convenience functions to connect in various ways (e.g. host and port, Unix socket, etc). See [include/valkey/valkey.h](../include/valkey/valkey.h) for more details. ```c valkeyContext *valkeyConnect(const char *host, int port); valkeyContext *valkeyConnectUnix(const char *path); // There is also a convenience struct to specify various options. valkeyContext *valkeyConnectWithOptions(valkeyOptions *opt); ``` When connecting to a server, libvalkey will return `NULL` in the event that we can't allocate the context, and set the `err` member if we can connect but there are issues. So when connecting it's simple to handle error states. When a hostname resolves to multiple addresses, libvalkey will try each address in order until one succeeds or all have failed. Note that the `connect_timeout` applies per address, so the total connection time may be up to N × timeout when multiple addresses are unreachable. ```c valkeyContext *ctx = valkeyConnect("localhost", 6379); if (ctx == NULL || ctx->err) { fprintf(stderr, "Error connecting: %s\n", ctx ? ctx->errstr : "OOM"); } ``` ### Connection options There are a variety of options you can specify when connecting to the server, which are delivered via the `valkeyOptions` helper struct. This includes information to connect to the server as well as other flags. ```c valkeyOptions opt = {0}; // You can set primary connection info if (tcp) { VALKEY_OPTIONS_SET_TCP(&opt, "localhost", 6379); } else { VALKEY_OPTIONS_SET_UNIX(&opt, "/tmp/valkey.sock"); } // You may attach any arbitrary data to the context VALKEY_OPTIONS_SET_PRIVDATA(&opt, my_data); ``` There are also several flags you can specify when using the `valkeyOptions` helper struct. | Flag | Description | | --- | --- | | `VALKEY_OPT_NONBLOCK` | Tells libvalkey to make a non-blocking connection. | | `VALKEY_OPT_REUSEADDR` | Tells libvalkey to set the [SO_REUSEADDR](https://man7.org/linux/man-pages/man7/socket.7.html) socket option | | `VALKEY_OPT_PREFER_IPV4``VALKEY_OPT_PREFER_IPV6``VALKEY_OPT_PREFER_IP_UNSPEC` | Informs libvalkey to either prefer IPv4 or IPv6 when invoking [getaddrinfo](https://man7.org/linux/man-pages/man3/gai_strerror.3.html). `VALKEY_OPT_PREFER_IP_UNSPEC` will cause libvalkey to specify `AF_UNSPEC` in the getaddrinfo call, which means both IPv4 and IPv6 addresses will be searched simultaneously.Libvalkey prefers IPv4 by default. | | `VALKEY_OPT_NO_PUSH_AUTOFREE` | Tells libvalkey to not install the default RESP3 PUSH handler (which just intercepts and frees the replies). This is useful in situations where you want to process these messages in-band. | | `VALKEY_OPT_NOAUTOFREEREPLIES` | **ASYNC**: tells libvalkey not to automatically invoke `freeReplyObject` after executing the reply callback. | | `VALKEY_OPT_NOAUTOFREE` | **ASYNC**: Tells libvalkey not to automatically free the `valkeyAsyncContext` on connection/communication failure, but only if the user makes an explicit call to `valkeyAsyncDisconnect` or `valkeyAsyncFree` | | `VALKEY_OPT_MPTCP` | Tells libvalkey to use multipath TCP (MPTCP). Note that only when both the server and client are using MPTCP do they establish an MPTCP connection between them; otherwise, they use a regular TCP connection instead. | ### Executing commands The primary command interface is a `printf`-like function that takes a format string along with a variable number of arguments. This will construct a `RESP` command and deliver it to the server. ```c valkeyReply *reply = valkeyCommand(ctx, "INCRBY %s %d", "counter", 42); if (reply == NULL) { fprintf(stderr, "Communication error: %s\n", c->err ? c->errstr : "Unknown error"); } else if (reply->type == VALKEY_REPLY_ERROR) { fprintf(stderr, "Error response from server: %s\n", reply->str); } else if (reply->type != VALKEY_REPLY_INTEGER) { // Very unlikely but should be checked. fprintf(stderr, "Error: Non-integer reply to INCRBY?\n"); } printf("New value of 'counter' is %lld\n", reply->integer); freeReplyObject(reply); ``` If you need to deliver binary safe strings to the server, you can use the `%b` format specifier which requires you to pass the length as well. ```c struct binary { int x; int y; } = {0xdeadbeef, 0xcafebabe}; valkeyReply *reply = valkeyCommand(ctx, "SET %s %b", "some-key", &binary, sizeof(binary)); ``` Commands may also be constructed by sending an array of arguments along with an optional array of their lengths. If lengths are not provided, libvalkey will execute `strlen` on each argument. ```c const char *argv[] = {"SET", "captain", "James Kirk"}; sonst size_t argvlens[] = {3, 7, 10}; valkeyReply *reply = valkeyCommandArgv(ctx, 3, argv, argvlens); // Handle error conditions similarly to `valkeyCommand` ``` ### Using replies The `valkeyCommand` and `valkeyCommandArgv` functions return a `valkeyReply` on success and `NULL` in the event of a severe error (e.g. a communication failure with the server, out of memory condition, etc). If the reply is `NULL` you can inspect the nature of the error by querying `valkeyContext->err` for the error code and `valkeyContext->errstr` for a human readable error string. When a `valkeyReply` is returned, you should test the `valkeyReply->type` field to determine which kind of reply was received from the server. If for example there was an error in the command, this reply can be `VALKEY_REPLY_ERROR` and the specific error string will be in the `reply->str` member. ### Reply types - `VALKEY_REPLY_ERROR` - An error reply. The error string is in `reply->str`. - `VALKEY_REPLY_STATUS` - A status reply which will be in `reply->str`. - `VALKEY_REPLY_INTEGER` - An integer reply, which will be in `reply->integer`. - `VALKEY_REPLY_DOUBLE` - A double reply which will be in `reply->dval` as well as `reply->str`. - `VALKEY_REPLY_NIL` - a nil reply. - `VALKEY_REPLY_BOOL` - A boolean reply which will be in `reply->integer`. - `VALKEY_REPLY_BIGNUM` - As of yet unused, but the string would be in `reply->str`. - `VALKEY_REPLY_STRING` - A string reply which will be in `reply->str`. - `VALKEY_REPLY_VERB` - A verbatim string reply which will be in `reply->str` and who's type will be in `reply->vtype`. - `VALKEY_REPLY_ARRAY` - An array reply where each element is in `reply->element` with the number of elements in `reply->elements`. - `VALKEY_REPLY_MAP` - A map reply, which structurally looks just like `VALKEY_REPLY_ARRAY` only is meant to represent keys and values. As with an array reply you can access the elements with `reply->element` and `reply->elements`. - `VALKEY_REPLY_SET` - Another array-like reply representing a set (e.g. a reply from `SMEMBERS`). Access via `reply->element` and `reply->elements`. - `VALKEY_REPLY_ATTR` - An attribute reply. As of yet unused by valkey-server. - `VALKEY_REPLY_PUSH` - An out of band push reply. This is also array-like in nature. ### Disconnecting/cleanup When libvalkey returns non-null `valkeyReply` struts you are responsible for freeing them with `freeReplyObject`. In order to disconnect and free the context simply call `valkeyFree`. ```c valkeyReply *reply = valkeyCommand(ctx, "set %s %s", "foo", "bar"); // Error handling ... freeReplyObject(reply); // Disconnect and free context valkeyFree(ctx); ``` ### Pipelining `valkeyCommand` and `valkeyCommandArgv` each make a round-trip to the server, by sending the command and then waiting for a reply. Alternatively commands may be pipelined with the `valkeyAppendCommand` and `valkeyAppendCommandArgv` functions. When you use `valkeyAppendCommand` the command is simply appended to the output buffer of `valkeyContext` but not delivered to the server, until you attempt to read the first response, at which point the entire buffer will be delivered. ```c // No data will be delivered to the server while these commands are being appended. for (size_t i = 0; i < 100000; i++) { if (valkeyAppendCommand(c, "INCRBY key:%zu %zu", i, i) != VALKEY_OK) { fprintf(stderr, "Error appending command: %s\n", c->errstr); exit(1); } } // The entire output buffer will be delivered on the first call to `valkeyGetReply`. for (size_t i = 0; i < 100000; i++) { if (valkeyGetReply(c, (void**)&reply) != VALKEY_OK) { fprintf(stderr, "Error reading reply %zu: %s\n", i, c->errstr); exit(1); } else if (reply->type != VALKEY_REPLY_INTEGER) { fprintf(stderr, "Error: Non-integer reply to INCRBY?\n"); exit(1); } printf("INCRBY key:%zu => %lld\n", i, reply->integer); freeReplyObject(reply); } ``` `valkeyGetReply` can also be used in other contexts than pipeline, for example when you want to continuously block for commands for example in a subscribe context. ```c valkeyReply *reply = valkeyCommand(c, "SUBSCRIBE channel"); assert(reply != NULL && !c->err); while (valkeyGetReply(c, (void**)&reply) == VALKEY_OK) { // Do something with the message... freeReplyObject(reply); } ``` ### Errors As previously mentioned, when there is a communication error libvalkey will return `NULL` and set the `err` and `errstr` members with the nature of the problem. The specific error types are as follows. - `VALKEY_ERR_IO` - A problem with the connection. - `VALKEY_ERR_EOF` - The server closed the connection. - `VALKEY_ERR_PROTOCOL` - There was an error parsing the reply. - `VALKEY_ERR_TIMEOUT` - A connect, read, or write timeout. - `VALKEY_ERR_OOM` - Out of memory. - `VALKEY_ERR_OTHER` - Some other error (check `c->errstr` for details). ### Thread safety Libvalkey context structs are **not** thread safe. You should not attempt to share them between threads, unless you really know what you're doing. ### Reader configuration Libvalkey contexts have a few more mechanisms you can customize to your needs. #### Maximum input buffer size Libvalkey uses a buffer to hold incoming bytes, which is typically restored to the configurable max buffer size (`16KB`) when it is empty. To avoid continually reallocating this buffer you can set the value higher, or to zero which means "no limit". ```c context->reader->maxbuf = 0; ``` #### Maximum array elements By default, libvalkey will refuse to parse array-like replies if they have more than 2^32-1 or 4,294,967,295 elements. This value can be set to any arbitrary 64-bit value or zero which just means "no limit". ```c context->reader->maxelements = 0; ``` #### RESP3 Push Replies The `RESP` protocol introduced out-of-band "push" replies in the third version of the specification. These replies may come at any point in the data stream. By default, libvalkey will simply process these messages and discard them. If your application needs to perform specific actions on PUSH messages you can install your own handler which will be called as they are received. It is also possible to set the push handler to NULL, in which case the messages will be delivered "in-band". This can be useful for example in a blocking subscribe loop. **NOTE**: You may also specify a push handler in the `valkeyOptions` struct and set it on initialization . #### Synchronous context ```c void my_push_handler(void *privdata, void *reply) { // In a synchronous context, you are expected to free the reply after you're done with it. } // Initialization, etc. valkeySetPushCallback(c, my_push_handler); ``` #### Asynchronous context ```c void my_async_push_handler(valkeyAsyncContext *ac, void *reply) { // As with other async replies, libvalkey will free it for you, unless you have // configured the context with `VALKEY_OPT_NOAUTOFREE`. } // Initialization, etc valkeyAsyncSetPushCallback(ac, my_async_push_handler); ``` #### Allocator injection Internally libvalkey uses a layer of indirection from the standard allocation functions, by keeping a global structure with function pointers to the allocators we are going to use. By default they are just set to `malloc`, `calloc`, `realloc`, etc. These can be overridden like so ```c valkeyAllocFuncs my_allocators = { .mallocFn = my_malloc, .callocFn = my_calloc, .reallocFn = my_realloc, .strdupFn = my_strdup, .freeFn = my_free, }; // libvalkey will return the previously set allocators. valkeyAllocFuncs old = valkeySetAllocators(&my_allocators); ``` They can also be reset to the glibc or musl defaults ```c valkeyResetAllocators(); ``` **NOTE**: The `vk_calloc` function handles the case where `nmemb` * `size` would overflow a `size_t` and returns `NULL` in that case. ## Asynchronous API Libvalkey also has an asynchronous API which supports a great many different event libraries. See the [examples](../examples) directory for specific information about each individual event library. ### Connecting Libvalkey provides an `valkeyAsyncContext` to manage asynchronous connections which works similarly to the synchronous context. ```c valkeyAsyncContext *ac = valkeyAsyncConnect("localhost", 6379); if (ac == NULL) { fprintf(stderr, "Error: Out of memory trying to allocate valkeyAsyncContext\n"); exit(1); } else if (ac->err) { fprintf(stderr, "Error: %s (%d)\n", ac->errstr, ac->err); exit(1); } // If we're using libev valkeyLibevAttach(EV_DEFAULT_ ac); valkeySetConnectCallback(ac, my_connect_callback); valkeySetDisconnectCallback(ac, my_disconnect_callback); ev_run(EV_DEFAULT_ 0); ``` The asynchronous context _should_ hold a connect callback function that is called when the connection attempt completes, either successfully or with an error. It _can_ also hold a disconnect callback function that is called when the connection is disconnected (either because of an error or per user request). The context object is always freed after the disconnect callback fired. ### Executing commands Executing commands in an asynchronous context work similarly to the synchronous context, except that you can pass a callback that will be invoked when the reply is received. ```c struct my_app_data { size_t incrby_replies; size_t get_replies; }; void my_incrby_callback(valkeyAsyncContext *ac, void *r, void *privdata) { struct my_app_data *data = privdata; valkeyReply *reply = r; assert(reply != NULL && reply->type == VALKEY_REPLY_INTEGER); printf("Incremented value: %lld\n", reply->integer); data->incrby_replies++; } void my_get_callback(valkeyAsyncContext *ac, void *r, void *privdata) { struct my_app_data *data = privdata; valkeyReply *reply = r; assert(reply != NULL && reply->type == VALKEY_REPLY_STRING); printf("Key value: %s\n", reply->str); data->get_replies++; } int exec_some_commands(struct my_app_data *data) { valkeyAsyncCommand(ac, my_incrby_callback, data, "INCRBY mykey %d", 42); valkeyAsyncCommand(ac, my_get_callback, data, "GET %s", "mykey"); } ``` ### Disconnecting/cleanup For a graceful disconnect use `valkeyAsyncDisconnect` which will block new commands from being issued. The connection is only terminated when all pending commands have been sent, their respective replies have been read, and their respective callbacks have been executed. After this, the disconnection callback is called with the status, and the context object is freed. To terminate the connection forcefully use `valkeyAsyncFree` which also will block new commands from being issued. There will be no more data sent on the socket and all pending callbacks will be called with a `NULL` reply. After this, the disconnection callback is called with the `VALKEY_OK` status, and the context object is freed. ## TLS support TLS support is not enabled by default and requires an explicit build flag as described in [`README.md`](../README.md#building). Libvalkey implements TLS on top of its `valkeyContext` and `valkeyAsyncContext`, so you will need to establish a connection first and then initiate a TLS handshake. See the [examples](../examples) directory for how to create the TLS context and initiate the handshake. --- ## File: deps/libvalkey/README.md # Libvalkey Libvalkey is the official C client for the [Valkey](https://valkey.io) database. It also supports any server that uses the `RESP` protocol (version 2 or 3). This project supports both standalone and cluster modes. ## Table of Contents - [Features](#features) - [Supported platforms](#supported-platforms) - [Building](#building) - [Building with make](#building-with-make) - [Building with CMake](#building-with-cmake) - [Contributing](#contributing) - Using the library - [Standalone mode](docs/standalone.md) - [Cluster mode](docs/cluster.md) - [Migration guide](docs/migration-guide.md) ## Features - Commands are executed in a generic way, with printf-like invocation. - Supports both `RESP2` and `RESP3` protocol versions. - Supports both synchronous and asynchronous operation. - Optional support for `MPTCP`, `TLS` and `RDMA` connections. - Asynchronous API with several event libraries to choose from. - Supports both standalone and cluster mode operation. - Can be compiled with either `make` or `CMake`. ## Supported platforms This library supports and is tested against `Linux`, `FreeBSD`, `macOS`, and `Windows`. It should build and run on various proprietary Unixes although we can't run CI for those platforms. If you encounter any issues, please open an issue. ## Building Libvalkey is written in C targeting C99. Unfortunately we have no plans on supporting C89 or earlier. The project does use a few widely supported compiler extensions, specifically for the bundled `sds` string library, although we have plans to remove this from the library. We support plain GNU make and CMake. Following is information on how to build the library. ### Building with make ```bash # Build and install the default library sudo make install # With all options sudo USE_TLS=1 USE_RDMA=1 make install # If your openssl is in a non-default location sudo USE_TLS=1 OPENSSL_PREFIX=/path/to/openssl make install ``` ### Building with CMake See [CMakeLists.txt](CMakeLists.txt) for all available options. ```bash # Build and install mkdir build && cd build cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo .. sudo make install # Build with TLS and RDMA support mkdir build && cd build cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_TLS=1 -DENABLE_RDMA=1 .. sudo make install ``` ## Contributing Please see [`CONTRIBUTING.md`](https://github.com/valkey-io/libvalkey/blob/main/CONTRIBUTING.md). --- ## File: deps/README.md This directory contains all Valkey dependencies, except for the libc that should be provided by the operating system. * **Jemalloc** is our memory allocator, used as replacement for libc malloc on Linux by default. It has good performances and excellent fragmentation behavior. This component is upgraded from time to time. * **libvalkey** is the official C client library for Valkey. It is used by valkey-cli, valkey-benchmark and Valkey Sentinel. It is managed in a separate project and updated as needed. * **linenoise** is a readline replacement. It is developed by the same authors of Valkey but is managed as a separated project and updated as needed. * **lua** is Lua 5.1 with minor changes for security and additional libraries. * **LZ4** is the v1.10.0 streaming compression library. * **hdr_histogram** Used for per-command latency tracking histograms. * **ffc.h** is a C99 port of the fast_float library, used as a replacement for strtod to convert strings to floats efficiently. * **gtest-parallel** is a script for running googletest tests in parallel. How to upgrade the above dependencies === Jemalloc --- Jemalloc is used mostly unmodified from upstream. Active defragmentation relies on jemalloc's native API rather than custom source patches (those were removed). The only source-level modification is in `include/jemalloc/jemalloc.sh`, which defines the `VALKEY_VENDORED_JEMALLOC` macro so Valkey can detect the vendored copy at build time. A `CMakeLists.txt` is added for the Valkey build system. We change Jemalloc settings via the `configure` script using the `--with-lg-quantum` option, setting it to the value of 3 instead of 4. This provides us with more size classes that better suit the Valkey data structures, in order to gain memory efficiency. #### Updating/upgrading jemalloc The jemalloc directory is pulled as a subtree from the upstream jemalloc github repo. To update it you should run from the project root: 1. `git subtree pull --prefix deps/jemalloc https://github.com/jemalloc/jemalloc.git --squash` This should hopefully merge the local changes into the new version. 2. In case any conflicts arise you'll need to resolve them and commit. 3. Reconfigure jemalloc: ```sh rm deps/jemalloc/VERSION deps/jemalloc/configure cd deps/jemalloc ./autogen.sh --with-version=-0-g0 ``` 4. Update jemalloc's version in `deps/Makefile`: search for "`--with-version=-0-g0`" and update it accordingly. 5. Commit the changes (VERSION,configure,Makefile). Libvalkey --- Libvalkey is used by Sentinel, `valkey-cli` and `valkey-benchmark`. The library is built without its own version of the sds and dict type and uses the Valkey provided variant instead. 1. `git subtree pull --prefix deps/libvalkey https://github.com/valkey-io/libvalkey.git --squash` This should hopefully merge the local changes into the new version. 2. Commit the changes. Linenoise --- Linenoise is rarely upgraded as needed. The upgrade process is trivial since Valkey uses a non modified version of linenoise, so to upgrade just do the following: 1. Remove the linenoise directory. 2. Substitute it with the new linenoise source tree. LZ4 --- LZ4 is imported from the upstream release archive. The vendored version is defined by the `LZ4_VERSION_MAJOR`, `LZ4_VERSION_MINOR`, and `LZ4_VERSION_RELEASE` macros in `lz4.h`. `Makefile` and `CMakeLists.txt` are maintained locally for the Valkey build, and xxHash symbols are namespaced to avoid conflicts with modules loaded by Valkey. To upgrade LZ4: 1. Download the new release archive from https://github.com/lz4/lz4/releases. 2. Replace `lz4.c`, `lz4.h`, `lz4hc.c`, `lz4hc.h`, `lz4frame.c`, `lz4frame.h`, `xxhash.c`, `xxhash.h`, and `LICENSE` with the versions from the release's `lib` directory. 3. Preserve the local `Makefile` and `CMakeLists.txt`, including the `XXH_NAMESPACE` definition. 4. Verify that the `LZ4_VERSION_*` macros in `lz4.h` match the imported release and update the version in the dependency summary above. Lua --- We use Lua 5.1 and no upgrade is planned currently, since we don't want to break Lua scripts for new Lua features: in the context of Valkey Lua scripts the capabilities of 5.1 are usually more than enough, the release is rock solid, and we definitely don't want to break old scripts. So upgrading of Lua is up to the Valkey project maintainers and should be a manual procedure performed by taking a diff between the different versions. Currently we have at least the following differences between official Lua 5.1 and our version: 1. Makefile and CMakeLists.txt are modified for the Valkey build system. 2. We have the implementation source code, and directly link to the following external libraries: `lua_cjson.o`, `lua_struct.o`, `lua_cmsgpack.o` and `lua_bit.o`. 3. There is a security fix in `ldo.c`: The check for `LUA_SIGNATURE[0]` is removed in order to avoid direct bytecode execution. 4. In `lstring.c`, the `luaS_newlstr` function's hash calculation has been upgraded from a simple hash function to MurmurHash3, to enhance performance particularly for large strings. 5. Support for readonly tables (`lua_enablereadonlytable` API) is added across `lapi.c`, `lvm.c`, `lobject.h`, and `ltable.c`. 6. Protection of tables reachable from globals, with a globals whitelist. 7. Multiple CVE security patches (CVE-2022-24834, CVE-2024-31449, CVE-2024-31227, CVE-2024-31228) in cjson, cmsgpack, and the parser. Hdr_Histogram --- Updated source can be found here: https://github.com/HdrHistogram/HdrHistogram_c We use a customized version based on master branch commit e4448cf6d1cd08fff519812d3b1e58bd5a94ac42. 1. Compare all changes under /hdr_histogram directory to upstream master commit e4448cf6d1cd08fff519812d3b1e58bd5a94ac42 2. Copy updated files from newer version onto files in /hdr_histogram. 3. Apply the changes from 1 above to the updated files. ffc.h --- ffc.h is a pure C99 port of the fast_float library, providing fast string-to-double conversion without requiring a C++ compiler. To upgrade the library, 1. Download the latest ffc.h from https://github.com/kolemannix/ffc.h/releases 2. Copy ffc.h to "deps/fast_float/". gtest-parallel --- The `deps/gtest-parallel` directory is imported from the upstream https://github.com/google/gtest-parallel repository as a subtree snapshot (not a real Git subtree). Current upstream version: - Upstream commit: `cd488bd` (from google/gtest-parallel) Updating gtest-parallel Run the following from the repository root. 1. Add the remote and fetch upstream: ```sh git remote add gtest-parallel https://github.com/google/gtest-parallel.git git fetch gtest-parallel master ``` 2. Remove any previous import and commit (commit A): ```sh rm -rf deps/gtest-parallel ``` 3. Update the subtree from upstream: ```sh git subtree add --prefix=deps/gtest-parallel gtest-parallel master --squash ``` 4. Reset back to commit A with proper sign-off: ```sh git reset --soft ``` 5. Commit the changes. --- ## File: src/commands/README.md Command JSON files ================== This directory contains JSON files, one for each command. Each JSON file contains all the information about the command itself. It is the "single source of truth" (SSOT) for the command's metadata. These JSON files were originally not intended to be used directly, since they contain internals and some fields like "acl_categories" that are not the final ACL categories. (Valkey will apply some implicit rules to compute the final ACL categories.) However, people see JSON files and use them directly anyway. Any third party who needs access to command information were originally supposed to get it from `COMMAND INFO` and `COMMAND DOCS`. These commands can be combined into a JSON file by the script `utils/generate-commands-json.py`. Confusingly enough, this JSON file as a slightly different format! Structure --------- Each JSON file contains an object with a single key. The key is the command name in uppercase, e.g. "HSCAN" (hscan.json). The value is a JSON object with the following keys. To be safe, assume all of them are optional. * `"summary"`: a string with a short description of the command. One sentence. * `"complexity"`: a string like `"O(1)"` or longer, like `"O(1) for every call. O(N) for a complete iteration, including enough command calls for the cursor to return back to 0. N is the number of elements inside the collection."`. * `"group"`: a string used for categorization in documentation. One of these: * `"bitmap"` * `"cluster"` * `"connection"` * `"generic"` * `"geo"` * `"hash"` * `"hyperloglog"` * `"list"` * `"pubsub"` * `"scripting"` * `"sentinel"` * `"server"` * `"set"` * `"sorted_set"` * `"stream"` * `"string"` * `"transactions"` * `"since"`: a string with a version number, like "7.0.0". It's the version (Redis OSS or Valkey) where the command was introduced. * `"arity"`: The number of arguments, including the command name itself. A negative number means "at least", e.g. -3 means at least 3. * `"container"`: Only present for subcommands. See below. * `"history"`: An array of changes, each change represented by a 2-element array on the form `[VERSION, DESCRIPTION]`. Omit if empty. Don't add an empty array. * `"function"`: The name of the C function in Valkey's source code implementing the command. (Don't use it for anything else.) * `"command_flags"`: An array of flags represented as strings. Command flags: * `"ADMIN"` * `"ALL_DBS"` * `"ALLOW_BUSY"` * `"ASKING"` * `"BLOCKING"` * `"DENYOOM"` * `"FAST"` * `"LOADING"` * `"MAY_REPLICATE"` * `"NO_ASYNC_LOADING"` * `"NO_AUTH"` * `"NO_MANDATORY_KEYS"` * `"NO_MULTI"` * `"NOSCRIPT"` * `"ONLY_SENTINEL"` * `"PROTECTED"` * `"PUBSUB"` * `"READONLY"` * `"SENTINEL"` * `"SKIP_MONITOR"` * `"SKIP_SLOWLOG"` * `"STALE"` * `"TOUCHES_ARBITRARY_KEYS"` * `"WRITE"` * `"acl_categories"`: A list of ACL categories in uppercase. Note that the effective ACL categories include "implicit ACL categories" explained below. * `"ADMIN"` * `"BITMAP"` * `"CONNECTION"` * `"DANGEROUS"` * `"GEO"` * `"HASH"` * `"HYPERLOGLOG"` * `"KEYSPACE"` * `"LIST"` * `"SCRIPTING"` * `"SET"` * `"SORTEDSET"` * `"STREAM"` * `"STRING"` * `"TRANSACTION"` * `"get_dbid_args"`: The name of the C function in Valkey's source code implementing retrieval of database ID arguments from commands that accept database ID as an argument. * `"command_tips"`: Optional. A list of one or more of these strings: * `"NONDETERMINISTIC_OUTPUT"` * `"NONDETERMINISTIC_OUTPUT_ORDER"` * `"REQUEST_POLICY:ALL_NODES"` * `"REQUEST_POLICY:ALL_SHARDS"` * `"REQUEST_POLICY:MULTI_SHARD"` * `"REQUEST_POLICY:SPECIAL"` * `"RESPONSE_POLICY:AGG_LOGICAL_AND"` * `"RESPONSE_POLICY:AGG_MIN"` * `"RESPONSE_POLICY:AGG_SUM"` * `"RESPONSE_POLICY:ALL_SUCCEEDED"` * `"RESPONSE_POLICY:ONE_SUCCEEDED"` * `"RESPONSE_POLICY:SPECIAL"` * `"key_specs"`: An array of key specifications. See below. * `"reply_schema"`: A [JSON Schema](https://json-schema.org/) that describes the reply of the command. This isn't complete. For example, JSON Schema can't distinguish arrays from sets, commands returning a set are declared to return an array. * `"arguments"`: An array of arguments. Each argument is an object with the following keys: * `"name"`: A string identifying the argument. It's unique among the arguments. * `"type"`: The type of the argument. * `"block"`: A group of arguments. The elements are in the key `"arguments"`. * `"double"`: A number, not necessarily an integer. * `"integer"`: An integer. * `"key"`: A string representing a key in the database. * `"oneof"`: One of a list of alternatives. The alternatives are in the key `"arguments"`. * `"pattern"`: A string representing a glob-style pattern. * `"pure-token"`: A fixed string. The string is in the key `"token"`. * `"string"`: A string. * `"unix-time"`: An integer representing a unix time in either seconds or milliseconds. * `"arguments"`: A list with the same structure as its parent. Present if type is "block" or "oneof". * `"display"`: ("entries-read", "key" or "pattern") * `"key_spec_index"`: An index into the `"key_specs"` array. Only if `"type"` is `"key"`. * `"multiple":` true if the argument can be repeated multiple times. Omitted means false. * `"multiple_token"`: Unclear meaning. Maybe meaningless. * `"optional":` True if the argument is optional. Omitted means false. * `"since"`: Version (string) when the argument was introduced. * `"token"`: A string indicating a fixed string value. This is always present if type is "pure-token". If type is anything else, then `"token"` indicates the argument is preceded by an extra (fixed string) argument. ACL categories ----------------------- The ACL categories specified as `"acl_categories"` are the ones that are actually used. **Note:** commands categories should follow specific rules that are checked in `utils/generate-command-code.py` **ACL Category Rules:** * Command flag WRITE implies ACL category WRITE. * Command flag READONLY and not ACL category SCRIPTING implies ACL category READ. "Exclude scripting commands from the RO category." * Command flag ADMIN implies ACL categories ADMIN and DANGEROUS. * Command flag PUBSUB implies ACL category PUBSUB. * Command flag FAST implies ACL category FAST. * Command flag BLOCKING implies ACL category BLOCKING. * Not ACL category FAST implies ACL category SLOW. "If it's not fast, it's slow." Key specs --------- Key specifications are specified in the array `"key_specs"` key of a command. Each element in this array is an object with the following keys: * `"flags"`: An array of strings indicating what kind of access is the command does on the key. * `"ACCESS"` * `"DELETE"` * `"INCOMPLETE"` * `"INSERT"` * `"NOT_KEY"` * `"OW"` * `"RM"` * `"RO"` * `"RW"` * `"UPDATE"` * `"VARIABLE_FLAGS"` * `"begin_search"`: How to find the first key used by this key spec. It's an object with only one key. The key determines the method for finding the first key. Here are the possible forms of the `"begin_search"` object: * `{"index": {"pos": N}}`: The first key is at position N in the command line, where 0 is the command name. * `{"keyword": KEYWORD, "startfrom": N}`: The first key is found by searching for an argument with the exact value KEYWORD starting from index N in the command line. The first key is the argument after the keyword. * `{"unknown": null}`: Finding the keys of this command is too complicated to explain. * `"find_keys"`: How to find the remaining keys of this key spec. It's an object on one of these forms: * `{"range": {"lastkey": LAST, "step": STEP, "limit": LIMIT}}`: A range of keys. * LAST: If LAST is positive, it's the index of the last key relative to the first key. If last is negative, then -1 is the end of the whole command line, -2 is the penultimate argument of the command, and so on. * STEP: The number of arguments to skip to find the next one. Typically 1. * LIMIT: If LAST is -1, we use the limit to stop the search by a factor. 0 and 1 mean no limit. 2 means half of the remaining arguments, 3 means a third, and so on. Commands with subcommands ------------------------- Commands with subcommands are special. Examples of commands with subcommands are `CLUSTER` and `ACL`. Their first argument is a subcommand which determines the syntax of the rest the command, which is stored in its own JSON file. For example `CLUSTER INFO` is stored in a file called `cluster-info.json`. The toplevel key is called `"INFO"`. Within the body, there's a key called `"container"` with the value `"CLUSTER"`. The file `cluster.json` exists by it doesn't have an `"arguments"` key. Appendix -------- How to list all the `group`, `command_flags` and `acl_categories`, etc. used in all these files: cat *.json | jq '.[].group' | grep -F '"' | sed 's/^ *//;s/, *$//;s/^/ * `/;s/$/`/' | sort | uniq cat *.json | jq '.[].command_flags' | grep -F '"' | sed 's/^ *//;s/, *$//;s/^/ * `/;s/$/`/' | sort | uniq cat *.json | jq '.[].acl_categories' | grep -F '"' | sed 's/^ *//;s/, *$//;s/^/ * `/;s/$/`/' | sort | uniq cat *.json | jq '.[].arguments[]?.type' | grep -F '"' | sed 's/^ *//;s/, *$//;s/^/ * `/;s/$/`/' | sort | uniq --- ## File: src/modules/lua/README.md # Lua 5.1 Scripting Module This module provides a Lua 5.1 scripting engine for Valkey, enabling server-side script execution with atomic operations and enhanced performance. ## Overview The Lua module enables execution of Lua 5.1 scripts within Valkey using the `EVAL` and `FCALL` commands. This allows for: - Atomic execution of complex operations - Reduced network round trips - Server-side data processing and transformation - Custom command implementations Scripts are executed in a sandboxed environment with access to Valkey's data structures and a subset of Lua's standard library. For comprehensive information about using Lua scripts in Valkey, please refer to the **Extending Valkey** section of the official [documentation site](https://valkey.io/topics/). ## Building This module is automatically built as part of the Valkey core build process. However, you can also build it independently for development or testing purposes: ```bash cd src/modules/lua make ``` The independent build will create the module binary that can be loaded into Valkey using the `MODULE LOAD` command. ## Dependencies This Lua module is designed to be mostly self-contained with minimal external dependencies. Currently, the module has only two dependencies on Valkey core files: - `src/sha1.c` - for SHA1 hash computation - `src/random.c` - for random number generation These dependencies were chosen to leverage existing Valkey functionality while maintaining modularity. Should the need arise to relocate this Lua module to a separate repository, these files can be easily copied alongside the module without introducing additional transitive dependencies. When adding new dependencies in the future, please ensure they are lightweight and self-contained to preserve the module's portability and minimize complexity. ## 2. Official Technical Reference & Guides (valkey-io/valkey-io.github.io) # Valkey.io website This repo contains the source for the valkey.io website (build scripts, template, blog posts, stylesheets, etc.). The build integrates content from [`valkey-io/valkey-doc`](https://github.com/valkey-io/valkey-doc) and the commands definitions from [`valkey-io/valkey`](https://github.com/valkey-io/valkey), [`valkey-io/valkey-bloom`](https://github.com/valkey-io/valkey-bloom), and [`valkey-io/valkey-json`](https://github.com/valkey-io/valkey-json) (see [Build Locally](#build-locally) below for more details) ## Contributing We welcome contributions! Please see our [CONTRIBUTING](CONTRIBUTING.md) page to learn more about how to contribute to the website. ## Security If you discover potential security issues, see the reporting instructions on our [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) page for more information. ## Build Locally This site is built with [Zola](https://www.getzola.org/). Follow these steps to build the site locally: 1. [Install Zola](https://www.getzola.org/documentation/getting-started/installation/). 2. Switch to the directory with your fork of this repo. 3. Run `zola serve` Open your browser to `http://127.0.0.1:1111/` Zola will automatically rebuild on changes to the template or any content stored in this repo. Changes to external content (command reference, documentation topics) require a restart of the Zola server process (`ctrl-c` then `zola serve` again, a browser refresh may also be needed). ## Building additional content **By default, the site will build without documentation topics, command reference, or the clients page.** The content for documentation topics and the clients page are stored within the `valkey-io/valkey-doc` repo. The content for the command reference page is in the `valkey-io/valkey` repo. If you want to build the site with this content, you'll need to have a local copy of `valkey-io/valkey-doc` and `valkey-io/valkey` _outside_ of this repo. Then follow the instructions to [build the documentation topics and clients](#building-the-documentation-topics-and-clients-page) and/or [build the command reference](#building-the-command-reference). The instructions show how to use scripts that create symbolic links to the `valkey-io/valkey-doc` and `valkey-io/valkey` repos as well as create a series of empty stub files that tell Zola to create pages. ### Building the documentation topics and clients page Documentation 'topics' (i.e. `/topics/keyspace/`, `/topics/encryption/`, `/topics/transactions/`) and the client libraries' data (i.e. `/client-page-clients/nodejs/valkey-glide`, `/client-page-clients/python/valkey-py`) sources content from `valkey-io/valkey-doc`. ```mermaid flowchart TD A[Webpage: /topics/keyspace/ ] A --> B[Template: valkey-io/valkey-website] B --> H[Repo: valkey-io/valkey-doc ] --> I[File: /topics/keyspace.md ] --> Y[Topic content] ``` Let's say that this repo and your local copy of `valkey-io/valkey-doc` reside in the same directory. First, stop the `zola serve` process if you're running it. From the root directory of this repo run: ```shell # You should only need to run this once or when you add a new topic/client. ./build/init-topics-and-clients.sh ../valkey-doc/topics ../valkey-doc/clients ``` Then, restart Zola. Point your browser at `http://127.0.0.1:1111/topics/` and you should see the fully populated list of topics and clients. All files created in this process are ignored by git. Commit your changes to your local copy of `valkey-io/valkey-doc`. ### Building the command reference The command reference (i.e. `/commands/set/`, `/commands/get/`, `/commands/lolwut/`) sources information from `valkey-io/valkey`, `valkey-io/valkey-bloom`, and `valkey-io/valkey-doc`. `valkey-io/valkey`, `valkey-io/valkey-bloom`, `valkey-io/valkey-json` and `valkey-io/valkey-search` provides the command metadata (items like computational complexity, version history, arguments, etc) whilst `valkey-io/valkey-doc` provides the command description and the command reply. ```mermaid flowchart TD A[Webpage: valkey.io/commands/set] A --> B[Template: valkey-io/valkey-website] B --> F[Repo: valkey-io/valkey ] --> G[File: /src/commands/set.json ] --> X[Command Metadata] B --> H[Repo: valkey-io/valkey-doc ] --> I[File: /commands/set.md ] --> Y[Command Description] H --> J[Files: /resp2_replies.json,/resp3_replies.json] --> Z[Command Reply] ``` Let's say that this repo and your local copy of `valkey-io/valkey-doc`, `valkey-io/valkey-bloom`, `valkey-io/valkey-json`, `valkey-io/valkey-search` and `valkey-io/valkey` reside in the same directories. First, stop the `zola serve` process if you're running it. From the root directory of this repo run: ```shell # You should only need to run this once or when you add a new command. ./build/init-commands.sh ../valkey-doc/commands ../valkey/src/commands \ ../valkey-bloom/src/commands ../valkey-json/src/commands ../valkey-search/src/commands ``` Then, restart Zola. Point your browser at `http://127.0.0.1:1111/commands/` and you should see the fully populated list of topics. All files created in this process are ignored by git. Commit your changes to your local copy of `valkey-io/valkey-doc` for description changes and `valkey-io/valkey` for command JSON changes (if you have any). ## License This project is licensed under the BSD-3-Clause License.