## 1. Project Overview & Quickstart (redis/lettuce) ## File: README.md Lettuce - Advanced Java Redis client =============================== [](https://github.com/redis/lettuce/actions/workflows/integration.yml) [](https://codecov.io/gh/redis/lettuce) [](./LICENSE.txt) [![Maven Central](https://img.shields.io/maven-central/v/io.lettuce/lettuce-core?versionSuffix=RELEASE&logo=redis )](https://maven-badges.herokuapp.com/maven-central/io.lettuce/lettuce-core) [](https://www.javadoc.io/doc/io.lettuce/lettuce-core) [](https://discord.gg/redis) [](https://www.twitch.tv/redisinc) [](https://www.youtube.com/redisinc) [](https://twitter.com/redisinc) [](https://stackoverflow.com/questions/tagged/lettuce) Lettuce is a scalable thread-safe Redis client for synchronous, asynchronous and reactive usage. Multiple threads may share one connection if they avoid blocking and transactional operations such as `BLPOP` and `MULTI`/`EXEC`. Lettuce is built with [netty](https://github.com/netty/netty). Supports advanced Redis features such as Sentinel, Cluster, Pipelining, Auto-Reconnect and Redis data models. This version of Lettuce has been tested against the latest Redis source-build. * [synchronous](https://redis.github.io/lettuce/user-guide/connecting-redis/#basic-usage), [asynchronous](https://redis.github.io/lettuce/user-guide/async-api/) and [reactive](https://redis.github.io/lettuce/user-guide/reactive-api/) usage * [Redis Sentinel](https://redis.github.io/lettuce/ha-sharding/#redis-sentinel_1) * [Redis Cluster](https://redis.github.io/lettuce/ha-sharding/#redis-cluster) * [SSL](https://redis.github.io/lettuce/advanced-usage/#ssl-connections) and [Unix Domain Socket](https://redis.github.io/lettuce/advanced-usage/#unix-domain-sockets) connections * [Streaming API](https://redis.github.io/lettuce/advanced-usage/streaming-api) * [Codecs](https://redis.github.io/lettuce/integration-extension/#codecss) (for UTF8/bit/JSON etc. representation of your data) * multiple [Command Interfaces](https://github.com/redis/lettuce/wiki/Command-Interfaces-%284.0%29) * Support for [Native Transports](https://redis.github.io/lettuce/advanced-usage/native-transports) * Support [RediSearch](https://redis.github.io/lettuce/user-guide/redis-search/), [RedisJSON](https://redis.github.io/lettuce/user-guide/redis-json/) and [Redis Vector Sets](https://redis.github.io/lettuce/user-guide/vector-sets/) * Compatible with Java 8++ (implicit automatic module w/o descriptors) See the [reference documentation](https://redis.github.io/lettuce/) and [API Reference](https://www.javadoc.io/doc/io.lettuce/lettuce-core/latest/index.html) for more details. ## How do I Redis? [Learn for free at Redis University](https://university.redis.io/academy) [Try the Redis Cloud](https://redis.io/try-free/) [Dive in developer tutorials](https://redis.io/learn/) [Join the Redis community](https://redis.io/community/) [Work at Redis](https://redis.io/careers/jobs/) Documentation --------------- * [Reference documentation](https://redis.github.io/lettuce/) * [Javadoc](https://www.javadoc.io/doc/io.lettuce/lettuce-core/latest/index.html) Binaries/Download ---------------- Binaries and dependency information for Maven, Ivy, Gradle and others can be found at http://search.maven.org. Releases of lettuce are available in the Maven Central repository. Take also a look at the [Releases](https://github.com/redis/lettuce/releases). Example for Maven: ```xml io.lettuce lettuce-core x.y.z ``` If you'd rather like the latest snapshots of the upcoming major version, use our Maven snapshot repository and declare the appropriate dependency version. ```xml io.lettuce lettuce-core x.y.z.BUILD-SNAPSHOT sonatype-snapshots Sonatype Snapshot Repository https://oss.sonatype.org/content/repositories/snapshots/ true ``` Basic Usage ----------- ```java RedisClient client = RedisClient.create("redis://localhost"); StatefulRedisConnection connection = client.connect(); RedisStringCommands sync = connection.sync(); String value = sync.get("key"); ``` Each Redis command is implemented by one or more methods with names identical to the lowercase Redis command name. Complex commands with multiple modifiers that change the result type include the CamelCased modifier as part of the command name, e.g. zrangebyscore and zrangebyscoreWithScores. See [Basic usage](https://redis.github.io/lettuce/user-guide/connecting-redis/#basic-usage) for further details. Asynchronous API ------------------------ ```java StatefulRedisConnection connection = client.connect(); RedisStringAsyncCommands async = connection.async(); RedisFuture set = async.set("key", "value"); RedisFuture get = async.get("key"); LettuceFutures.awaitAll(set, get) == true set.get() == "OK" get.get() == "value" ``` See [Asynchronous API](https://redis.github.io/lettuce/user-guide/async-api/) for further details. Reactive API ------------------------ ```java StatefulRedisConnection connection = client.connect(); RedisStringReactiveCommands reactive = connection.reactive(); Mono set = reactive.set("key", "value"); Mono get = reactive.get("key"); set.subscribe(); get.block() == "value" ``` See [Reactive API](https://redis.github.io/lettuce/user-guide/reactive-api/) for further details. Pub/Sub ------- ```java RedisPubSubCommands connection = client.connectPubSub().sync(); connection.getStatefulConnection().addListener(new RedisPubSubListener() { ... }) connection.subscribe("channel"); ``` Building ----------- Lettuce is built with Apache Maven. The tests require multiple running Redis instances for different test cases which are configured using a ```Makefile```. Tests run by default against Redis `latest`. To build: ``` $ git clone https://github.com/redis/lettuce.git $ cd lettuce/ $ make start ``` * Run the build: ```make test``` * Start Redis (manually): ```make start``` * Stop Redis (manually): ```make stop``` * Clean up: ```make clean``` Bugs and Feedback ----------- For bugs, questions and discussions please use the [GitHub Issues](https://github.com/redis/lettuce/issues). License ------- * This repository is licensed under the "MIT" license. See [LICENSE](LICENSE). * Fork of https://github.com/wg/lettuce Contributing ------- Github is for social coding: if you want to write code, I encourage contributions through pull requests from forks of this repository. Create Github tickets for bugs and new features and comment on the ones that you are interested in and take a look into [CONTRIBUTING.md](https://github.com/redis/lettuce/blob/main/.github/CONTRIBUTING.md) --- ## File: docs/user-guide/async-api.md ## Asynchronous API This guide will give you an impression how and when to use the asynchronous API provided by Lettuce 4.x. ### Motivation Asynchronous methodologies allow you to utilize better system resources, instead of wasting threads waiting for network or disk I/O. Threads can be fully utilized to perform other work instead. Lettuce facilitates asynchronicity from building the client on top of [netty](http://netty.io) that is a multithreaded, event-driven I/O framework. All communication is handled asynchronously. Once the foundation is able to processes commands concurrently, it is convenient to take advantage from the asynchronicity. It is way harder to turn a blocking and synchronous working software into a concurrently processing system. #### Understanding Asynchronicity Asynchronicity permits other processing to continue before the transmission has finished and the response of the transmission is processed. This means, in the context of Lettuce and especially Redis, that multiple commands can be issued serially without the need of waiting to finish the preceding command. This mode of operation is also known as [Pipelining](https://redis.io/docs/latest/develop/use/pipelining/). The following example should give you an impression of the mode of operation: - Given client *A* and client *B* - Client *A* triggers command `SET A=B` - Client *B* triggers at the same time of Client *A* command `SET C=D` - Redis receives command from Client *A* - Redis receives command from Client *B* - Redis processes `SET A=B` and responds `OK` to Client *A* - Client *A* receives the response and stores the response in the response handle - Redis processes `SET C=D` and responds `OK` to Client *B* - Client *B* receives the response and stores the response in the response handle Both clients from the example above can be either two threads or connections within an application or two physically separated clients. Clients can operate concurrently to each other by either being separate processes, threads, event-loops, actors, fibers, etc. Redis processes incoming commands serially and operates mostly single-threaded. This means, commands are processed in the order they are received with some characteristic that we’ll cover later. Let’s take the simplified example and enhance it by some program flow details: - Given client *A* - Client *A* triggers command `SET A=B` - Client *A* uses the asynchronous API and can perform other processing - Redis receives command from Client *A* - Redis processes `SET A=B` and responds `OK` to Client *A* - Client *A* receives the response and stores the response in the response handle - Client *A* can access now the response to its command without waiting (non-blocking) The Client *A* takes advantage from not waiting on the result of the command so it can process computational work or issue another Redis command. The client can work with the command result as soon as the response is available. #### Impact of asynchronicity to the synchronous API While this guide helps you to understand the asynchronous API it is worthwhile to learn the impact on the synchronous API. The general approach of the synchronous API is no different than the asynchronous API. In both cases, the same facilities are used to invoke and transport commands to the Redis server. The only difference is a blocking behavior of the caller that is using the synchronous API. Blocking happens on command level and affects only the command completion part, meaning multiple clients using the synchronous API can invoke commands on the same connection and at the same time without blocking each other. A call on the synchronous API is unblocked at the moment a command response was processed. - Given client *A* and client *B* - Client *A* triggers command `SET A=B` on the synchronous API and waits for the result - Client *B* triggers at the same time of Client *A* command `SET C=D` on the synchronous API and waits for the result - Redis receives command from Client *A* - Redis receives command from Client *B* - Redis processes `SET A=B` and responds `OK` to Client *A* - Client *A* receives the response and unblocks the program flow of Client *A* - Redis processes `SET C=D` and responds `OK` to Client *B* - Client *B* receives the response and unblocks the program flow of Client *B* However, there are some cases you should not share a connection among threads to avoid side-effects. The cases are: - Disabling flush-after-command to improve performance - The use of blocking operations like `BLPOP`. Blocking operations are queued on Redis until they can be executed. While one connection is blocked, other connections can issue commands to Redis. Once a command unblocks the blocking command (that said an `LPUSH` or `RPUSH` hits the list), the blocked connection is unblocked and can proceed after that. - Transactions - Using multiple databases #### Result handles Every command invocation on the asynchronous API creates a `RedisFuture` that can be canceled, awaited and subscribed (listener). A `CompleteableFuture` or `RedisFuture` is a pointer to the result that is initially unknown since the computation of its value is yet incomplete. A `RedisFuture` provides operations for synchronization and chaining. ``` java CompletableFuture future = new CompletableFuture<>(); System.out.println("Current state: " + future.isDone()); future.complete("my value"); System.out.println("Current state: " + future.isDone()); System.out.println("Got value: " + future.get()); ``` The example prints the following lines: Current state: false Current state: true Got value: my value Attaching a listener to a future allows chaining. Promises can be used synonymous to futures, but not every future is a promise. A promise guarantees a callback/notification and thus it has come to its name. A simple listener that gets called once the future completes: ``` java final CompletableFuture future = new CompletableFuture<>(); future.thenRun(new Runnable() { @Override public void run() { try { System.out.println("Got value: " + future.get()); } catch (Exception e) { e.printStackTrace(); } } }); System.out.println("Current state: " + future.isDone()); future.complete("my value"); System.out.println("Current state: " + future.isDone()); ``` The value processing moves from the caller into a listener that is then called by whoever completes the future. The example prints the following lines: Current state: false Got value: my value Current state: true The code from above requires exception handling since calls to the `get()` method can lead to exceptions. Exceptions raised during the computation of the `Future` are transported within an `ExecutionException`. Another exception that may be thrown is the `InterruptedException`. This is because calls to `get()` are blocking calls and the blocked thread can be interrupted at any time. Just think about a system shutdown. The `CompletionStage` type allows since Java 8 a much more sophisticated handling of futures. A `CompletionStage` can consume, transform and build a chain of value processing. The code from above can be rewritten in Java 8 in the following style: ``` java CompletableFuture future = new CompletableFuture<>(); future.thenAccept(new Consumer() { @Override public void accept(String value) { System.out.println("Got value: " + value); } }); System.out.println("Current state: " + future.isDone()); future.complete("my value"); System.out.println("Current state: " + future.isDone()); ``` The example prints the following lines: Current state: false Got value: my value Current state: true You can find the full reference for the `CompletionStage` type in the [Java 8 API documentation](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html). ### Creating futures using Lettuce Lettuce futures can be used for initial and chaining operations. When using Lettuce futures, you will notice the non-blocking behavior. This is because all I/O and command processing are handled asynchronously using the netty EventLoop. The Lettuce `RedisFuture` extends a `CompletionStage` so all methods of the base type are available. Lettuce exposes its futures on the Standalone, Sentinel, Publish/Subscribe and Cluster APIs. Connecting to Redis is insanely simple: ``` java RedisClient client = RedisClient.create("redis://localhost"); RedisAsyncCommands commands = client.connect().async(); ``` In the next step, obtaining a value from a key requires the `GET` operation: ``` java RedisFuture future = commands.get("key"); ``` ### Consuming futures The first thing you want to do when working with futures is to consume them. Consuming a futures means obtaining the value. Here is an example that blocks the calling thread and prints the value: ``` java RedisFuture future = commands.get("key"); String value = future.get(); System.out.println(value); ``` Invocations to the `get()` method (pull-style) block the calling thread at least until the value is computed but in the worst case indefinitely. Using timeouts is always a good idea to not exhaust your threads. ``` java try { RedisFuture future = commands.get("key"); String value = future.get(1, TimeUnit.MINUTES); System.out.println(value); } catch (Exception e) { e.printStackTrace(); } ``` The example will wait at most 1 minute for the future to complete. If the timeout exceeds, a `TimeoutException` is thrown to signal the timeout. Futures can also be consumed in a push style, meaning when the `RedisFuture` is completed, a follow-up action is triggered: ``` java RedisFuture future = commands.get("key"); future.thenAccept(new Consumer() { @Override public void accept(String value) { System.out.println(value); } }); ``` Alternatively, written in Java 8 lambdas: ``` java RedisFuture future = commands.get("key"); future.thenAccept(System.out::println); ``` Lettuce futures are completed on the netty EventLoop. Consuming and chaining futures on the default thread is always a good idea except for one case: Blocking/long-running operations. As a rule of thumb, never block the event loop. If you need to chain futures using blocking calls, use the `thenAcceptAsync()`/`thenRunAsync()` methods to fork the processing to another thread. The `…​async()` methods need a threading infrastructure for execution, by default the `ForkJoinPool.commonPool()` is used. The `ForkJoinPool` is statically constructed and does not grow with increasing load. Using default `Executor`s is almost always the better idea. ``` java Executor sharedExecutor = ... RedisFuture future = commands.get("key"); future.thenAcceptAsync(new Consumer() { @Override public void accept(String value) { System.out.println(value); } }, sharedExecutor); ``` ### Synchronizing futures A key point when using futures is the synchronization. Futures are usually used to: 1. Trigger multiple invocations without the urge to wait for the predecessors (Batching) 2. Invoking a command without awaiting the result at all (Fire&Forget) 3. Invoking a command and perform other computing in the meantime (Decoupling) 4. Adding concurrency to certain computational efforts (Concurrency) There are several ways how to wait or get notified in case a future completes. Certain synchronization techniques apply to some motivations why you want to use futures. #### Blocking synchronization Blocking synchronization comes handy if you perform batching/add concurrency to certain parts of your system. An example to batching can be setting/retrieving multiple values and awaiting the results before a certain point within processing. ``` java List> futures = new ArrayList>(); for (int i = 0; i < 10; i++) { futures.add(commands.set("key-" + i, "value-" + i)); } LettuceFutures.awaitAll(1, TimeUnit.MINUTES, futures.toArray(new RedisFuture[futures.size()])); ``` The code from above does not wait until a certain command completes before it issues another one. The synchronization is done after all commands are issued. The example code can easily be turned into a Fire&Forget pattern by omitting the call to `LettuceFutures.awaitAll()`. A single future execution can be also awaited, meaning an opt-in to wait for a certain time but without raising an exception: ``` java RedisFuture future = commands.get("key"); if(!future.await(1, TimeUnit.MINUTES)) { System.out.println("Could not complete within the timeout"); } ``` Calling `await()` is friendlier to call since it throws only an `InterruptedException` in case the blocked thread is interrupted. You are already familiar with the `get()` method for synchronization, so we will not bother you with this one. At last, there is another way to synchronize futures in a blocking way. The major caveat is that you will become responsible to handle thread interruptions. If you do not handle that aspect, you will not be able to shut down your system properly if it is in a running state. ``` java RedisFuture future = commands.get("key"); while (!future.isDone()) { // do something ... } ``` While the `isDone()` method does not aim primarily for synchronization use, it might come handy to perform other computational efforts while the command is executed. #### Chaining synchronization Futures can be synchronized/chained in a non-blocking style to improve thread utilization. Chaining works very well in systems relying on event-driven characteristics. Future chaining builds up a chain of one or more futures that are executed serially, and every chain member handles a part in the computation. The `CompletionStage` API offers various methods to chain and transform futures. A simple transformation of the value can be done using the `thenApply()` method: ``` java future.thenApply(new Function() { @Override public Integer apply(String value) { return value.length(); } }).thenAccept(new Consumer() { @Override public void accept(Integer integer) { System.out.println("Got value: " + integer); } }); ``` Alternatively, written in Java 8 lambdas: ``` java future.thenApply(String::length) .thenAccept(integer -> System.out.println("Got value: " + integer)); ``` The `thenApply()` method accepts a function that transforms the value into another one. The final `thenAccept()` method consumes the value for final processing. You have already seen the `thenRun()` method from previous examples. The `thenRun()` method can be used to handle future completions in case the data is not crucial to your flow: ``` java future.thenRun(new Runnable() { @Override public void run() { System.out.println("Finished the future."); } }); ``` Keep in mind to execute the `Runnable` on a custom `Executor` if you are doing blocking calls within the `Runnable`. Another chaining method worth mentioning is the either-or chaining. A couple of `…​Either()` methods are available on a `CompletionStage`, see the [Java 8 API docs](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html) for the full reference. The either-or pattern consumes the value from the first future that is completed. A good example might be two services returning the same data, for instance, a Master-Replica scenario, but you want to return the data as fast as possible: ``` java RedisStringAsyncCommands master = masterClient.connect().async(); RedisStringAsyncCommands replica = replicaClient.connect().async(); RedisFuture future = master.get("key"); future.acceptEither(replica.get("key"), new Consumer() { @Override public void accept(String value) { System.out.println("Got value: " + value); } }); ``` ### Error handling Error handling is an indispensable component of every real world application and should to be considered from the beginning on. Futures provide some mechanisms to deal with errors. In general, you want to react in the following ways: - Return a default value instead - Use a backup future - Retry the future `RedisFuture`s transport exceptions if any occurred. Calls to the `get()` method throw the occurred exception wrapped within an `ExecutionException` (this is different to Lettuce 3.x). You can find more details within the Javadoc on [CompletionStage](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html). The following code falls back to a default value after it runs to an exception by using the `handle()` method: ``` java future.handle(new BiFunction() { @Override public Integer apply(String value, Throwable throwable) { if(throwable != null) { return "default value"; } return value; } }).thenAccept(new Consumer() { @Override public void accept(String value) { System.out.println("Got value: " + value); } }); ``` More sophisticated code could decide on behalf of the throwable type that value to return, as the shortcut example using the `exceptionally()` method: ``` java future.exceptionally(new Function() { @Override public String apply(Throwable throwable) { if (throwable instanceof IllegalStateException) { return "default value"; } return "other default value"; } }); ``` Retrying futures and recovery using futures is not part of the Java 8 `CompleteableFuture`. See the [Reactive API](reactive-api.md) for comfortable ways handling with exceptions. ### Examples ``` java RedisAsyncCommands async = client.connect().async(); RedisFuture set = async.set("key", "value"); RedisFuture get = async.get("key"); set.get() == "OK" get.get() == "value" ``` ``` java RedisAsyncCommands async = client.connect().async(); RedisFuture set = async.set("key", "value"); RedisFuture get = async.get("key"); set.await(1, SECONDS) == true set.get() == "OK" get.get(1, TimeUnit.MINUTES) == "value" ``` ``` java RedisStringAsyncCommands async = client.connect().async(); RedisFuture set = async.set("key", "value"); Runnable listener = new Runnable() { @Override public void run() { ...; } }; set.thenRun(listener); ``` --- ## File: docs/user-guide/connecting-redis.md # Connecting Redis Connections to a Redis Standalone, Sentinel, or Cluster require a specification of the connection details. The unified form is `RedisURI`. You can provide the database, password and timeouts within the `RedisURI`. You have following possibilities to create a `RedisURI`: 1. Use an URI: ``` java RedisURI.create("redis://localhost/"); ``` 2. Use the Builder ``` java RedisURI.Builder.redis("localhost", 6379).auth("password").database(1).build(); ``` 3. Set directly the values in `RedisURI` ``` java new RedisURI("localhost", 6379, 60, TimeUnit.SECONDS); ``` ## URI syntax **Redis Standalone** redis :// [[username :] password@] host [:port][/database] [?[timeout=timeout[d|h|m|s|ms|us|ns]] [&clientName=clientName] [&libraryName=libraryName] [&libraryVersion=libraryVersion] ] **Redis Standalone (SSL)** rediss :// [[username :] password@] host [: port][/database] [?[timeout=timeout[d|h|m|s|ms|us|ns]] [&clientName=clientName] [&libraryName=libraryName] [&libraryVersion=libraryVersion] ] **Redis Standalone (Unix Domain Sockets)** redis-socket :// [[username :] password@]path [?[timeout=timeout[d|h|m|s|ms|us|ns]] [&database=database] [&clientName=clientName] [&libraryName=libraryName] [&libraryVersion=libraryVersion] ] **Redis Sentinel** redis-sentinel :// [[username :] password@] host1[:port1] [, host2[:port2]] [, hostN[:portN]] [/database] [?[timeout=timeout[d|h|m|s|ms|us|ns]] [&sentinelMasterId=sentinelMasterId] [&clientName=clientName] [&libraryName=libraryName] [&libraryVersion=libraryVersion] ] **Schemes** - `redis` Redis Standalone - `rediss` Redis Standalone SSL - `redis-socket` Redis Standalone Unix Domain Socket - `redis-sentinel` Redis Sentinel **Timeout units** - `d` Days - `h` Hours - `m` Minutes - `s` Seconds - `ms` Milliseconds - `us` Microseconds - `ns` Nanoseconds Hint: The database parameter within the query part has higher precedence than the database in the path. RedisURI supports Redis Standalone, Redis Sentinel and Redis Cluster with plain, SSL, TLS and unix domain socket connections. Hint: The database parameter within the query part has higher precedence than the database in the path. RedisURI supports Redis Standalone, Redis Sentinel and Redis Cluster with plain, SSL, TLS and unix domain socket connections. ## Authentication Redis URIs may contain authentication details that effectively lead to usernames with passwords, password-only, or no authentication. Connections are authenticated by using the information provided through `RedisCredentials`. Credentials are obtained at connection time from `RedisCredentialsProvider`. When configuring username/password on the URI statically, then a `StaticCredentialsProvider` holds the configured information. **Notes** - When using Redis Sentinel, the password from the URI applies to the data nodes only. Sentinel authentication must be configured for each sentinel node. - Usernames are supported as of Redis 6. - Library name and library version are automatically set on Redis 7.2 or greater. ## Basic Usage ``` java RedisClient client = RedisClient.create("redis://localhost"); StatefulRedisConnection connection = client.connect(); RedisCommands commands = connection.sync(); String value = commands.get("foo"); ... connection.close(); client.shutdown(); ``` - Create the `RedisClient` instance and provide a Redis URI pointing to localhost, Port 6379 (default port). - Open a Redis Standalone connection. The endpoint is used from the initialized `RedisClient` - Obtain the command API for synchronous execution. Lettuce supports asynchronous and reactive execution models, too. - Issue a `GET` command to get the key `foo`. - Close the connection when you’re done. This happens usually at the very end of your application. Connections are designed to be long-lived. - Shut down the client instance to free threads and resources. This happens usually at the very end of your application. Each Redis command is implemented by one or more methods with names identical to the lowercase Redis command name. Complex commands with multiple modifiers that change the result type include the CamelCased modifier as part of the command name, e.g. `zrangebyscore` and `zrangebyscoreWithScores`. Redis connections are designed to be long-lived and thread-safe, and if the connection is lost will reconnect until `close()` is called. Pending commands that have not timed out will be (re)sent after successful reconnection. All connections inherit a default timeout from their RedisClient and and will throw a `RedisException` when non-blocking commands fail to return a result before the timeout expires. The timeout defaults to 60 seconds and may be changed in the RedisClient or for each connection. Synchronous methods will throw a `RedisCommandExecutionException` in case Redis responds with an error. Asynchronous connections do not throw exceptions when Redis responds with an error. ### RedisURI The RedisURI contains the host/port and can carry authentication/database details. On a successful connect you get authenticated, and the database is selected afterward. This applies also after re-establishing a connection after a connection loss. A Redis URI can also be created from an URI string. Supported formats are: - `redis://[password@]host[:port][/databaseNumber]` Plaintext Redis connection - `rediss://[password@]host[:port][/databaseNumber]` [SSL Connections](../advanced-usage/ssl-connections.md) Redis connection - `redis-sentinel://[password@]host[:port][,host2[:port2]][/databaseNumber]#sentinelMasterId` for using Redis Sentinel - `redis-socket:///path/to/socket` [Unix Domain Sockets](../advanced-usage/native-transports.md#unix-domain-sockets) connection to Redis ### Exceptions In the case of an exception/error response from Redis, you’ll receive a `RedisException` containing the error message. `RedisException` is a `RuntimeException`. ### Examples ``` java RedisURI uri = new RedisURI("localhost", 6379, Duration.ofSeconds(20)); RedisClient client = RedisClient.create(uri); // … client.shutdown(); ``` ``` java RedisURI redisUri = RedisURI.Builder.redis("localhost") .withPassword("authentication") .withDatabase(2) .build(); RedisClient client = RedisClient.create(redisUri); // … client.shutdown(); ``` ``` java RedisURI redisUri = RedisURI.Builder.redis("localhost") .withSsl(true) .withPassword("authentication") .withDatabase(2) .build(); RedisClient client = RedisClient.create(redisUri); // … client.shutdown(); ``` ``` java RedisURI redisUri = RedisURI.create("redis://authentication@localhost/2"); RedisClient client = RedisClient.create(redisUri); // … client.shutdown(); ``` ## Streaming Credentials Provider [Lettuce 6.6.0](https://github.com/redis/lettuce/releases/tag/6.6.0.RELEASE) extends `RedisCredentialsProvider` to support streaming credentials. It is useful when you need to refresh credentials periodically. Example use cases include: token expiration, rotating credentials, etc. Connection configured with `RedisCredentialsProvider` supporting streaming will be re-authenticated automatically when new credentials are emitted and `ReauthenticateBehavior` is set to `ON_NEW_CREDENTIALS`. ### Step 1 - Create a Streaming Credentials Provider A simple example of a streaming credentials provider that emits new credentials. ```java public class MyStreamingRedisCredentialsProvider implements RedisCredentialsProvider { private final Sinks.Many credentialsSink = Sinks.many().replay().latest(); @Override public boolean supportsStreaming() { return true; } @Override public Mono resolveCredentials() { return credentialsSink.asFlux().next(); } @Override // Provide a continuous stream of credentials public Flux credentials() { return credentialsSink.asFlux().onBackpressureLatest(); } public void close() { credentialsSink.tryEmitComplete(); } // Emit new credentials when needed public void emitCredentials(String username, char[] password) { credentialsSink.tryEmitNext(new StaticRedisCredentials(username, password)); } } ``` ### Step 2 - Create a RedisURI with streaming credentials provider ```java // Create a streaming credentials provider MyStreamingRedisCredentialsProvider streamingCredentialsProvider = new MyStreamingRedisCredentialsProvider(); // Emit initial credentials streamingCredentialsProvider.emitCredentials("testuser", "testpass".toCharArray()); // Enable automatic re-authentication ClientOptions clientOptions = ClientOptions.builder() // enable automatic re-authentication .reauthenticateBehavior(ClientOptions.ReauthenticateBehavior.ON_NEW_CREDENTIALS) .build(); // Create a RedisURI with streaming credentials provider RedisURI redisURI = RedisURI.builder().withHost(HOST).withPort(PORT) .withAuthentication(streamingCredentialsProvider) .build(); // RedisClient RedisClient redisClient = RedisClient.create(redisURI); rediscClient.connect().sync().ping(); // ... // Emit new credentials when needed streamingCredentialsProvider.emitCredentials("testuser", "password-rotated".toCharArray()); ``` ## Microsoft Entra ID Authentication [Lettuce 6.6.0](https://github.com/redis/lettuce/releases/tag/6.6.0.RELEASE) introduces built-in support for authentication with [Azure Managed Redis](https://azure.microsoft.com/en-us/products/managed-redis) and Azure Cache for Redis using Microsoft Entra ID (formerly Azure Active Directory). It enables seamless integration with Azure's Redis services by fetching authentication tokens and managing the token renewal in the background. Integration is built on top of [redis-authx](https://github.com/redis/jvm-redis-authx-entraid) library, and provides support for: - System-assigned managed identity - User-assigned managed identity - Service principal You can learn more about managed identities in the [Microsoft Entra ID documentation](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview). ### Basic Usage #### Pre-requisites * [register an application and create a service principal](https://learn.microsoft.com/en-us/entra/identity-platform/app-objects-and-service-principals?tabs=browser) in Azure. * Create a Redis cache in Azure and grant your service principal access: [AMR](https://learn.microsoft.com/en-us/azure/azure-cache-for-redis/managed-redis/managed-redis-entra-for-authentication) or [ACR](https://learn.microsoft.com/en-us/azure/azure-cache-for-redis/cache-azure-active-directory-for-authentication) documentation. #### Step 1 - Add the dependencies Lettuce requires [redis-authx-entraid](https://github.com/redis/jvm-redis-authx-entraid/) dependency to provide Microsoft Entra ID authentication support. Make sure to include that dependency on your classpath. If using Maven, add the following dependency to your `pom.xml`: ``` xml redis.clients.authentication redis-authx-entraid 0.1.1-beta1 ``` ### Step 2 - Create Entra ID enabled credentials provider The lifecycle of the credentials provider is not managed by the Lettuce client. You can create it once and reuse it across multiple clients\connections. When no longer needed, you should close the provider to release resources `TokenBasedRedisCredentialsProvider#close`. #### Create Microsoft Entra ID enabled credentials provider ```java // Entra ID enabled credentials provider for Service Principle Identity with Client Secret TokenBasedRedisCredentialsProvider credentialsSP; try ( EntraIDTokenAuthConfigBuilder builder = EntraIDTokenAuthConfigBuilder.builder()) { builder.clientId(CLIENT_ID) .secret(CLIENT_SECRET) .authority(AUTHORITY) // "https://login.microsoftonline.com/{YOUR_TENANT_ID}"; .scopes(SCOPES); // "https://redis.azure.com/.default"; credentialsSP = TokenBasedRedisCredentialsProvider.create(builder.build()); } ``` You can test the credentials provider by obtaining a token. ```java // Test Entra ID credentials provider can resolve credentials credentialsSP.resolveCredentials() .doOnNext(c-> System.out.println(c.getUsername())) .block(); ``` ### Step 3 - Enable automatic re-authentication Microsoft Entra ID tokens have a limited lifetime. Lettuce provides a mechanism to automatically re-authenticate when new credentials are emitted by a `RedisCredentialsProvider`. ```java // Enable automatic re-authentication ClientOptions clientOptions = ClientOptions.builder() .reauthenticateBehavior(ClientOptions.ReauthenticateBehavior.ON_NEW_CREDENTIALS) .build(); ``` ### Step 4 - Connect with Entra ID enabled credentials provider ```java // Use Entra ID credentials provider RedisURI redisURI = RedisURI.builder() .withHost(HOST) .withPort(PORT) .withSsl(true) .withAuthentication(credentialsSP).build(); // RedisClient RedisClient redisClient = RedisClient.create(redisURI); redisClient.setOptions(clientOptions); try { redisClient.setOptions(clientOptions); // Connect with Entra ID credentials provider try (StatefulRedisConnection user1 = redisClient.connect(StringCodec.UTF8)) { System.out.println("Connected to redis as :" + user1.sync().aclWhoami()); System.out.println("Db size :" + user1.sync().dbsize()); } } finally { redisClient.shutdown(); // Shutdown Redis client and close connections credentialsSP.close(); // Shutdown Entra ID Credentials provider } ``` --- ## File: docs/user-guide/kotlin-api.md ## Kotlin API Kotlin Coroutines are using Kotlin lightweight threads allowing to write non-blocking code in an imperative way. On language side, suspending functions provides an abstraction for asynchronous operations while on library side kotlinx.coroutines provides functions like `async { }` and types like `Flow`. Lettuce ships with extensions to provide support for idiomatic Kotlin use. ### Dependencies Coroutines support is available when `kotlinx-coroutines-core` and `kotlinx-coroutines-reactive` dependencies are on the classpath: ``` xml org.jetbrains.kotlinx kotlinx-coroutines-core ${kotlinx-coroutines.version} org.jetbrains.kotlinx kotlinx-coroutines-reactive ${kotlinx-coroutines.version} ``` ### How does Reactive translate to Coroutines? `Flow` is an equivalent to `Flux` in Coroutines world, suitable for hot or cold streams, finite or infinite streams, with the following main differences: - `Flow` is push-based while `Flux` is a push-pull hybrid - Backpressure is implemented via suspending functions - `Flow` has only a single suspending collect method and operators are implemented as extensions - Operators are easy to implement thanks to Coroutines - Extensions allow to add custom operators to Flow - Collect operations are suspending functions - `map` operator supports asynchronous operations (no need for `flatMap`) since it takes a suspending function parameter ### Coroutines API based on reactive operations Example for retrieving commands and using it: ``` kotlin val api: RedisCoroutinesCommands = connection.coroutines() val foo1 = api.set("foo", "bar") val foo2 = api.keys("fo*") ``` !!! NOTE Coroutine Extensions are experimental and require opt-in using `@ExperimentalLettuceCoroutinesApi`. The API ships with a reduced feature set. Deprecated methods and `StreamingChannel` are left out intentionally. Expect evolution towards a `Flow`-based API to consume large Redis responses. ### Extensions for existing APIs #### Transactions DSL Example for the synchronous API: ``` kotlin val result: TransactionResult = connection.sync().multi { set("foo", "bar") get("foo") } ``` Example for async with coroutines: ``` kotlin val result: TransactionResult = connection.async().multi { set("foo", "bar") get("foo") } ``` --- ## File: docs/user-guide/lua-scripting.md ### Lua Scripting [Lua](https://redis.io/topics/lua-api) is a powerful scripting language that is supported at the core of Redis. Lua scripts can be invoked dynamically by providing the script contents to Redis or used as stored procedure by loading the script into Redis and using its digest to invoke it. ``` java String helloWorld = redis.eval("return ARGV[1]", STATUS, new String[0], "Hello World"); ``` Using Lua scripts is straightforward. Consuming results in Java requires additional details to consume the result through a matching type. As we do not know what your script will return, the API uses call-site generics for you to specify the result type. Additionally, you must provide a `ScriptOutputType` hint to `EVAL` so that the driver uses the appropriate output parser. See [Output Formats](redis-functions.md#output-formats) for further details. Lua scripts can be stored on the server for repeated execution. Dynamically-generated scripts are an anti-pattern as each script is stored in Redis' script cache. Generating scripts during the application runtime may, and probably will, exhaust the host’s memory resources for caching them. Instead, scripts should be as generic as possible and provide customized execution via their arguments. You can register a script through `SCRIPT LOAD` and use its SHA digest to invoke it later: ``` java String digest = redis.scriptLoad("return ARGV[1]", STATUS, new String[0], "Hello World"); // later String helloWorld = redis.evalsha(digest, STATUS, new String[0], "Hello World"); ``` --- ## File: docs/user-guide/pubsub.md ## Publish/Subscribe Lettuce provides support for Publish/Subscribe on Redis Standalone and Redis Cluster connections. The connection is notified on message/subscribed/unsubscribed events after subscribing to channels or patterns. [Synchronous](connecting-redis.md#basic-usage), [asynchronous](async-api.md) and [reactive](reactive-api.md) APIs are provided to interact with Redis Publish/Subscribe features. ### Subscribing A connection can notify multiple listeners that implement `RedisPubSubListener` (Lettuce provides a `RedisPubSubAdapter` for convenience). All listener registrations are kept within the `StatefulRedisPubSubConnection`/`StatefulRedisClusterConnection`. ``` java StatefulRedisPubSubConnection connection = client.connectPubSub() connection.addListener(new RedisPubSubListener() { ... }) RedisPubSubCommands sync = connection.sync(); sync.subscribe("channel"); // application flow continues ``` !!! NOTE Don’t issue blocking calls (includes synchronous API calls to Lettuce) from inside of Pub/Sub callbacks as this would block the EventLoop. If you need to fetch data from Redis from inside a callback, please use the asynchronous API. ``` java StatefulRedisPubSubConnection connection = client.connectPubSub() connection.addListener(new RedisPubSubListener() { ... }) RedisPubSubAsyncCommands async = connection.async(); RedisFuture future = async.subscribe("channel"); // application flow continues ``` ### Reactive API The reactive API provides hot `Observable`s to listen on `ChannelMessage`s and `PatternMessage`s. The `Observable`s receive all inbound messages. You can do filtering using the observable chain if you need to filter out the interesting ones, The `Observable` stops triggering events when the subscriber unsubscribes from it. ``` java StatefulRedisPubSubConnection connection = client.connectPubSub() RedisPubSubReactiveCommands reactive = connection.reactive(); reactive.subscribe("channel").subscribe(); reactive.observeChannels().doOnNext(patternMessage -> {...}).subscribe() // application flow continues ``` ### Redis Cluster Redis Cluster support Publish/Subscribe but requires some attention in general. User-space Pub/Sub messages (Calling `PUBLISH`) are broadcasted across the whole cluster regardless of subscriptions to particular channels/patterns. This behavior allows connecting to an arbitrary cluster node and registering a subscription. The client isn’t required to connect to the node where messages were published. A cluster-aware Pub/Sub connection is provided by `RedisClusterClient.connectPubSub()` allowing to listen for cluster reconfiguration and reconnect if the topology changes. ``` java StatefulRedisClusterPubSubConnection connection = clusterClient.connectPubSub() connection.addListener(new RedisPubSubListener() { ... }) RedisPubSubCommands sync = connection.sync(); sync.subscribe("channel"); ``` Redis Cluster also makes a distinction between user-space and key-space messages. Key-space notifications (Pub/Sub messages for key-activity) stay node-local and are not broadcasted across the Redis Cluster. A notification about, e.g. an expiring key, stays local to the node on which the key expired. Clients that are interested in keyspace notifications must subscribe to the appropriate node (or nodes) to receive these notifications. You can either use `RedisClient.connectPubSub()` to establish Pub/Sub connections to the individual nodes or use `RedisClusterClient`'s message propagation and NodeSelection API to get a managed set of connections. ``` java StatefulRedisClusterPubSubConnection connection = clusterClient.connectPubSub() connection.addListener(new RedisClusterPubSubListener() { ... }) connection.setNodeMessagePropagation(true); RedisPubSubCommands sync = connection.sync(); sync.masters().commands().subscribe("__keyspace@0__:*"); ``` There are two things to pay special attention to: 1. Replication: Keys replicated to replica nodes, especially considering expiry, generate keyspace events on all nodes holding the key. If a key expires and it is replicated, it will expire on the master and all replicas. Each Redis server will emit keyspace events. Subscribing to non-master nodes, therefore, will let your application see multiple events of the same type for the same key because of Redis distributed nature. 2. Topology Changes: Subscriptions are issued either by using the NodeSelection API or by calling `subscribe(…)` on the individual cluster node connections. Subscription registrations are not propagated to new nodes that are added on a topology change. --- ## File: docs/user-guide/reactive-api.md ## Reactive API This guide helps you to understand the Reactive Stream pattern and aims to give you a general understanding of how to build reactive applications. ### Motivation Asynchronous and reactive methodologies allow you to utilize better system resources, instead of wasting threads waiting for network or disk I/O. Threads can be fully utilized to perform other work instead. A broad range of technologies exists to facilitate this style of programming, ranging from the very limited and less usable `java.util.concurrent.Future` to complete libraries and runtimes like Akka. [Project Reactor](http://projectreactor.io/), has a very rich set of operators to compose asynchronous workflows, it has no further dependencies to other frameworks and supports the very mature Reactive Streams model. ### Understanding Reactive Streams Reactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments (JVM and JavaScript) as well as network protocols. The scope of Reactive Streams is to find a minimal set of interfaces, methods, and protocols that will describe the necessary operations and entities to achieve the goal—asynchronous streams of data with non-blocking back pressure. It is an interoperability standard between multiple reactive composition libraries that allow interaction without the need of bridging between libraries in application code. The integration of Reactive Streams is usually accompanied with the use of a composition library that hides the complexity of bare `Publisher` and `Subscriber` types behind an easy-to-use API. Lettuce uses [Project Reactor](http://projectreactor.io/) that exposes its publishers as `Mono` and `Flux`. For more information about Reactive Streams see . ### Understanding Publishers Asynchronous processing decouples I/O or computation from the thread that invoked the operation. A handle to the result is given back, usually a `java.util.concurrent.Future` or similar, that returns either a single object, a collection or an exception. Retrieving a result, that was fetched asynchronously is usually not the end of processing one flow. Once data is obtained, further requests can be issued, either always or conditionally. With Java 8 or the Promise pattern, linear chaining of futures can be set up so that subsequent asynchronous requests are issued. Once conditional processing is needed, the asynchronous flow has to be interrupted and synchronized. While this approach is possible, it does not fully utilize the advantage of asynchronous processing. In contrast to the preceding examples, `Publisher` objects answer the multiplicity and asynchronous questions in a different fashion: By inverting the `Pull` pattern into a `Push` pattern. **A Publisher is the asynchronous/push “dual” to the synchronous/pull Iterable** | event | Iterable (pull) | Publisher (push) | |----------------|------------------|--------------------| | retrieve data | T next() | onNext(T) | | discover error | throws Exception | onError(Exception) | | complete | !hasNext() | onCompleted() | An `Publisher` supports emission sequences of values or even infinite streams, not just the emission of single scalar values (as Futures do). You will very much appreciate this fact once you start to work on streams instead of single values. Project Reactor uses two types in its vocabulary: `Mono` and `Flux` that are both publishers. A `Mono` can emit `0` to `1` events while a `Flux` can emit `0` to `N` events. A `Publisher` is not biased toward some particular source of concurrency or asynchronicity and how the underlying code is executed - synchronous or asynchronous, running within a `ThreadPool`. As a consumer of a `Publisher`, you leave the actual implementation to the supplier, who can change it later on without you having to adapt your code. The last key point of a `Publisher` is that the underlying processing is not started at the time the `Publisher` is obtained, rather its started at the moment an observer subscribes or signals demand to the `Publisher`. This is a crucial difference to a `java.util.concurrent.Future`, which is started somewhere at the time it is created/obtained. So if no observer ever subscribes to the `Publisher`, nothing ever will happen. ### A word on the lettuce Reactive API All commands return a `Flux`, `Mono` or `Mono` to which a `Subscriber` can subscribe to. That subscriber reacts to whatever item or sequence of items the `Publisher` emits. This pattern facilitates concurrent operations because it does not need to block while waiting for the `Publisher` to emit objects. Instead, it creates a sentry in the form of a `Subscriber` that stands ready to react appropriately at whatever future time the `Publisher` does so. ### Consuming `Publisher` The first thing you want to do when working with publishers is to consume them. Consuming a publisher means subscribing to it. Here is an example that subscribes and prints out all the items emitted: ``` java Flux.just("Ben", "Michael", "Mark").subscribe(new Subscriber() { public void onSubscribe(Subscription s) { s.request(3); } public void onNext(String s) { System.out.println("Hello " + s + "!"); } public void onError(Throwable t) { } public void onComplete() { System.out.println("Completed"); } }); ``` The example prints the following lines: Hello Ben Hello Michael Hello Mark Completed You can see that the Subscriber (or Observer) gets notified of every event and also receives the completed event. A `Publisher` emits items until either an exception is raised or the `Publisher` finishes the emission calling `onCompleted`. No further elements are emitted after that time. A call to the `subscribe` registers a `Subscription` that allows to cancel and, therefore, do not receive further events. Publishers can interoperate with the un-subscription and free resources once a subscriber unsubscribed from the `Publisher`. Implementing a `Subscriber` requires implementing numerous methods, so lets rewrite the code to a simpler form: ``` java Flux.just("Ben", "Michael", "Mark").doOnNext(new Consumer() { public void accept(String s) { System.out.println("Hello " + s + "!"); } }).doOnComplete(new Runnable() { public void run() { System.out.println("Completed"); } }).subscribe(); ``` alternatively, even simpler by using Java 8 Lambdas: ``` java Flux.just("Ben", "Michael", "Mark") .doOnNext(s -> System.out.println("Hello " + s + "!")) .doOnComplete(() -> System.out.println("Completed")) .subscribe(); ``` You can control the elements that are processed by your `Subscriber` using operators. The `take()` operator limits the number of emitted items if you are interested in the first `N` elements only. ``` java Flux.just("Ben", "Michael", "Mark") // .doOnNext(s -> System.out.println("Hello " + s + "!")) .doOnComplete(() -> System.out.println("Completed")) .take(2) .subscribe(); ``` The example prints the following lines: Hello Ben Hello Michael Completed Note that the `take` operator implicitly cancels its subscription from the `Publisher` once the expected count of elements was emitted. A subscription to a `Publisher` can be done either by another `Flux` or a `Subscriber`. Unless you are implementing a custom `Publisher`, always use `Subscriber`. The used subscriber `Consumer` from the example above does not handle `Exception`s so once an `Exception` is thrown you will see a stack trace like this: Exception in thread "main" reactor.core.Exceptions$BubblingException: java.lang.RuntimeException: Example exception at reactor.core.Exceptions.bubble(Exceptions.java:96) at reactor.core.publisher.Operators.onErrorDropped(Operators.java:296) at reactor.core.publisher.LambdaSubscriber.onError(LambdaSubscriber.java:117) ... Caused by: java.lang.RuntimeException: Example exception at demos.lambda$example3Lambda$4(demos.java:87) at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:157) ... 23 more It is always recommended to implement an error handler right from the beginning. At a certain point, things can and will go wrong. A fully implemented subscriber declares the `onCompleted` and `onError` methods allowing you to react to these events: ``` java Flux.just("Ben", "Michael", "Mark").subscribe(new Subscriber() { public void onSubscribe(Subscription s) { s.request(3); } public void onNext(String s) { System.out.println("Hello " + s + "!"); } public void onError(Throwable t) { System.out.println("onError: " + t); } public void onComplete() { System.out.println("Completed"); } }); ``` ### From push to pull The examples from above illustrated how publishers can be set up in a not-opinionated style about blocking or non-blocking execution. A `Flux` can be converted explicitly into an `Iterable` or synchronized with `block()`. Avoid calling `block()` in your code as you start expressing the nature of execution inside your code. Calling `block()` removes all non-blocking advantages of the reactive chain to your application. ``` java String last = Flux.just("Ben", "Michael", "Mark").last().block(); System.out.println(last); ``` The example prints the following line: Mark A blocking call can be used to synchronize the publisher chain and find back a way into the plain and well-known `Pull` pattern. ``` java List list = Flux.just("Ben", "Michael", "Mark").collectList().block(); System.out.println(list); ``` The `toList` operator collects all emitted elements and passes the list through the `BlockingPublisher`. The example prints the following line: [Ben, Michael, Mark] ### Creating `Flux` and `Mono` using Lettuce There are many ways to establish publishers. You have already seen `just()`, `take()` and `collectList()`. Refer to the [Project Reactor documentation](http://projectreactor.io/docs/) for many more methods that you can use to create `Flux` and `Mono`. Lettuce publishers can be used for initial and chaining operations. When using Lettuce publishers, you will notice the non-blocking behavior. This is because all I/O and command processing are handled asynchronously using the netty EventLoop. Connecting to Redis is insanely simple: ``` java RedisClient client = RedisClient.create("redis://localhost"); RedisStringReactiveCommands commands = client.connect().reactive(); ``` In the next step, obtaining a value from a key requires the `GET` operation: ``` java commands.get("key").subscribe(new Consumer() { public void accept(String value) { System.out.println(value); } }); ``` Alternatively, written in Java 8 lambdas: ``` java commands .get("key") .subscribe(value -> System.out.println(value)); ``` The execution is handled asynchronously, and the invoking Thread can be used to processed in processing while the operation is completed on the Netty EventLoop threads. Due to its decoupled nature, the calling method can be left before the execution of the `Publisher` is finished. Lettuce publishers can be used within the context of chaining to load multiple keys asynchronously: ``` java Flux.just("Ben", "Michael", "Mark"). flatMap(key -> commands.get(key)). subscribe(value -> System.out.println("Got value: " + value)); ``` ### Hot and Cold Publishers There is a distinction between Publishers that was not covered yet: - A cold Publishers waits for a subscription until it emits values and does this freshly for every subscriber. - A hot Publishers begins emitting values upfront and presents them to every subscriber subsequently. All Publishers returned from the Redis Standalone, Redis Cluster, and Redis Sentinel API are cold, meaning that no I/O happens until they are subscribed to. As such a subscriber is guaranteed to see the whole sequence from the beginning. So just creating a Publisher will not cause any network I/O thus creating and discarding Publishers is cheap. Publishers created for a Publish/Subscribe emit `PatternMessage`s and `ChannelMessage`s once they are subscribed to. Publishers guarantee however to emit all items from the beginning until their end. While this is true for Publish/Subscribe publishers, the nature of subscribing to a Channel/Pattern allows missed messages due to its subscription nature and less to the Hot/Cold distinction of publishers. ### Transforming publishers Publishers can transform the emitted values in various ways. One of the most basic transformations is `flatMap()` which you have seen from the examples above that converts the incoming value into a different one. Another one is `map()`. The difference between `map()` and `flatMap()` is that `flatMap()` allows you to do those transformations with `Publisher` calls. ``` java Flux.just("Ben", "Michael", "Mark") .flatMap(commands::get) .flatMap(value -> commands.rpush("result", value)) .subscribe(); ``` The first `flatMap()` function is used to retrieve a value and the second `flatMap()` function appends the value to a Redis list named `result`. The `flatMap()` function returns a Publisher whereas the normal map just returns ``. You will use `flatMap()` a lot when dealing with flows like this, you’ll become good friends. An aggregation of values can be achieved using the `reduce()` transformation. It applies a function to each value emitted by a `Publisher`, sequentially and emits each successive value. We can use it to aggregate values, to count the number of elements in multiple Redis sets: ``` java Flux.just("Ben", "Michael", "Mark") .flatMap(commands::scard) .reduce((sum, current) -> sum + current) .subscribe(result -> System.out.println("Number of elements in sets: " + result)); ``` The aggregation function of `reduce()` is applied on each emitted value, so three times in the example above. If you want to get the last value, which denotes the final result containing the number of elements in all Redis sets, apply the `last()` transformation: ``` java Flux.just("Ben", "Michael", "Mark") .flatMap(commands::scard) .reduce((sum, current) -> sum + current) .last() .subscribe(result -> System.out.println("Number of elements in sets: " + result)); ``` Now let’s take a look at grouping emitted items. The following example emits three items and groups them by the beginning character. ``` java Flux.just("Ben", "Michael", "Mark") .groupBy(key -> key.substring(0, 1)) .subscribe( groupedFlux -> { groupedFlux.collectList().subscribe(list -> { System.out.println("First character: " + groupedFlux.key() + ", elements: " + list); }); } ); ``` The example prints the following lines: First character: B, elements: [Ben] First character: M, elements: [Michael, Mark] ### Absent values The presence and absence of values is an essential part of reactive programming. Traditional approaches consider `null` as an absence of a particular value. With Java 8, `Optional` was introduced to encapsulate nullability. Reactive Streams prohibits the use of `null` values. In the scope of Redis, an absent value is an empty list, a non-existent key or any other empty data structure. Reactive programming discourages the use of `null` as value. The reactive answer to absent values is just not emitting any value that is possible due the `0` to `N` nature of `Publisher`. Suppose we have the keys `Ben` and `Michael` set each to the value `value`. We query those and another, absent key with the following code: ``` java Flux.just("Ben", "Michael", "Mark") .flatMap(commands::get) .doOnNext(value -> System.out.println(value)) .subscribe(); ``` The example prints the following lines: value value The output is just two values. The `GET` to the absent key `Mark` does not emit a value. The reactive API provides operators to work with empty results when you require a value. You can use one of the following operators: - `defaultIfEmpty`: Emit a default value if the `Publisher` did not emit any value at all - `switchIfEmpty`: Switch to a fallback `Publisher` to emit values - `Flux.hasElements`/`Flux.hasElement`: Emit a `Mono` that contains a flag whether the original `Publisher` is empty - `next`/`last`/`elementAt`: Positional operators to retrieve the first/last/`N`th element or emit a default value ### Filtering items The values emitted by a `Publisher` can be filtered in case you need only specific results. Filtering does not change the emitted values itself. Filters affect how many items and at which point (and if at all) they are emitted. ``` java Flux.just("Ben", "Michael", "Mark") .filter(s -> s.startsWith("M")) .flatMap(commands::get) .subscribe(value -> System.out.println("Got value: " + value)); ``` The code will fetch only the keys `Michael` and `Mark` but not `Ben`. The filter criteria are whether the `key` starts with a `M`. You already met the `last()` filter to retrieve the last value: ``` java Flux.just("Ben", "Michael", "Mark") .last() .subscribe(value -> System.out.println("Got value: " + value)); ``` the extended variant of `last()` allows you to take the last `N` values: ``` java Flux.just("Ben", "Michael", "Mark") .takeLast(3) .subscribe(value -> System.out.println("Got value: " + value)); ``` The example from above takes the last `2` values. The opposite to `next()` is the `first()` filter that is used to retrieve the next value: ``` java Flux.just("Ben", "Michael", "Mark") .next() .subscribe(value -> System.out.println("Got value: " + value)); ``` ### Error handling Error handling is an indispensable component of every real world application and should to be considered from the beginning on. Project Reactor provides several mechanisms to deal with errors. In general, you want to react in the following ways: - Return a default value instead - Use a backup publisher - Retry the Publisher (immediately or with delay) The following code falls back to a default value after it throws an exception at the first emitted item: ``` java Flux.just("Ben", "Michael", "Mark") .doOnNext(value -> { throw new IllegalStateException("Takes way too long"); }) .onErrorReturn("Default value") .subscribe(); ``` You can use a backup `Publisher` which will be called if the first one fails. ``` java Flux.just("Ben", "Michael", "Mark") .doOnNext(value -> { throw new IllegalStateException("Takes way too long"); }) .switchOnError(commands.get("Default Key")) .subscribe(); ``` It is possible to retry the publisher by re-subscribing. Re-subscribing can be done as soon as possible, or with a wait interval, which is preferred when external resources are involved. ``` java Flux.just("Ben", "Michael", "Mark") .flatMap(commands::get) .retry() .subscribe(); ``` Use the following code if you want to retry with backoff: ``` java Flux.just("Ben", "Michael", "Mark") .doOnNext(v -> { if (new Random().nextInt(10) + 1 == 5) { throw new RuntimeException("Boo!"); } }) .doOnSubscribe(subscription -> { System.out.println(subscription); }) .retryWhen(throwableFlux -> Flux.range(1, 5) .flatMap(i -> { System.out.println(i); return Flux.just(i) .delay(Duration.of(i, ChronoUnit.SECONDS)); })) .blockLast(); ``` The attempts get passed into the `retryWhen()` method delayed with the number of seconds to wait. The delay method is used to complete once its timer is done. ### Schedulers and threads Schedulers in Project Reactor are used to instruct multi-threading. Some operators have variants that take a Scheduler as a parameter. These instruct the operator to do some or all of its work on a particular Scheduler. Project Reactor ships with a set of preconfigured Schedulers, which are all accessible through the `Schedulers` class: - Schedulers.parallel(): Executes the computational work such as event-loops and callback processing. - Schedulers.immediate(): Executes the work immediately in the current thread - Schedulers.elastic(): Executes the I/O-bound work such as asynchronous performance of blocking I/O, this scheduler is backed by a thread-pool that will grow as needed - Schedulers.newSingle(): Executes the work on a new thread - Schedulers.fromExecutor(): Create a scheduler from a `java.util.concurrent.Executor` - Schedulers.timer(): Create or reuse a hash-wheel based TimedScheduler with a resolution of 50ms. Do not use the computational scheduler for I/O. Publishers can be executed by a scheduler in the following different ways: - Using an operator that makes use of a scheduler - Explicitly by passing the Scheduler to such an operator - By using `subscribeOn(Scheduler)` - By using `publishOn(Scheduler)` Operators like `buffer`, `replay`, `skip`, `delay`, `parallel`, and so forth use a Scheduler by default if not instructed otherwise. All of the listed operators allow you to pass in a custom scheduler if needed. Sticking most of the time with the defaults is a good idea. If you want the subscribe chain to be executed on a specific scheduler, you use the `subscribeOn()` operator. The code is executed on the main thread without a scheduler set: ``` java Flux.just("Ben", "Michael", "Mark").flatMap(key -> { System.out.println("Map 1: " + key + " (" + Thread.currentThread().getName() + ")"); return Flux.just(key); } ).flatMap(value -> { System.out.println("Map 2: " + value + " (" + Thread.currentThread().getName() + ")"); return Flux.just(value); } ).subscribe(); ``` The example prints the following lines: Map 1: Ben (main) Map 2: Ben (main) Map 1: Michael (main) Map 2: Michael (main) Map 1: Mark (main) Map 2: Mark (main) This example shows the `subscribeOn()` method added to the flow (it does not matter where you add it): ``` java Flux.just("Ben", "Michael", "Mark").flatMap(key -> { System.out.println("Map 1: " + key + " (" + Thread.currentThread().getName() + ")"); return Flux.just(key); } ).flatMap(value -> { System.out.println("Map 2: " + value + " (" + Thread.currentThread().getName() + ")"); return Flux.just(value); } ).subscribeOn(Schedulers.parallel()).subscribe(); ``` The output of the example shows the effect of `subscribeOn()`. You can see that the Publisher is executed on the same thread, but on the computation thread pool: Map 1: Ben (parallel-1) Map 2: Ben (parallel-1) Map 1: Michael (parallel-1) Map 2: Michael (parallel-1) Map 1: Mark (parallel-1) Map 2: Mark (parallel-1) If you apply the same code to Lettuce, you will notice a difference in the threads on which the second `flatMap()` is executed: ``` java Flux.just("Ben", "Michael", "Mark").flatMap(key -> { System.out.println("Map 1: " + key + " (" + Thread.currentThread().getName() + ")"); return commands.set(key, key); }).flatMap(value -> { System.out.println("Map 2: " + value + " (" + Thread.currentThread().getName() + ")"); return Flux.just(value); }).subscribeOn(Schedulers.parallel()).subscribe(); ``` The example prints the following lines: Map 1: Ben (parallel-1) Map 1: Michael (parallel-1) Map 1: Mark (parallel-1) Map 2: OK (lettuce-nioEventLoop-3-1) Map 2: OK (lettuce-nioEventLoop-3-1) Map 2: OK (lettuce-nioEventLoop-3-1) Two things differ from the standalone examples: 1. The values are set rather concurrently than sequentially 2. The second `flatMap()` transformation prints the netty EventLoop thread name This is because Lettuce publishers are executed and completed on the netty EventLoop threads by default. `publishOn` instructs an Publisher to call its observer’s `onNext`, `onError`, and `onCompleted` methods on a particular Scheduler. Here, the order matters: ``` java Flux.just("Ben", "Michael", "Mark").flatMap(key -> { System.out.println("Map 1: " + key + " (" + Thread.currentThread().getName() + ")"); return commands.set(key, key); }).publishOn(Schedulers.parallel()).flatMap(value -> { System.out.println("Map 2: " + value + " (" + Thread.currentThread().getName() + ")"); return Flux.just(value); }).subscribe(); ``` Everything before the `publishOn()` call is executed in main, everything below in the scheduler: Map 1: Ben (main) Map 1: Michael (main) Map 1: Mark (main) Map 2: OK (parallel-1) Map 2: OK (parallel-1) Map 2: OK (parallel-1) Schedulers allow direct scheduling of operations. Refer to the [Project Reactor documentation](https://projectreactor.io/core/docs/api/reactor/core/scheduler/Schedulers.html) for further information. ### Redis Transactions Lettuce provides a convenient way to use Redis Transactions in a reactive way. Commands that should be executed within a transaction can be executed after the `MULTI` command was executed. Functional chaining allows to execute commands within a closure, and each command receives its appropriate response. A cumulative response is also returned with `TransactionResult` in response to `EXEC`. See [Transactions](transactions-multi.md#transactions-using-the-reactive-api) for further details. #### Other examples **Blocking example** ``` java RedisStringReactiveCommands reactive = client.connect().reactive(); Mono set = reactive.set("key", "value"); set.block(); ``` **Non-blocking example** ``` java RedisStringReactiveCommands reactive = client.connect().reactive(); Mono set = reactive.set("key", "value"); set.subscribe(); ``` **Functional chaining** ``` java RedisStringReactiveCommands reactive = client.connect().reactive(); Flux.just("Ben", "Michael", "Mark") .flatMap(key -> commands.sadd("seen", key)) .flatMap(value -> commands.randomkey()) .flatMap(commands::type) .doOnNext(System.out::println).subscribe(); ``` **Redis Transaction** ``` java RedisReactiveCommands reactive = client.connect().reactive(); reactive.multi().doOnSuccess(s -> { reactive.set("key", "1").doOnNext(s1 -> System.out.println(s1)).subscribe(); reactive.incr("key").doOnNext(s1 -> System.out.println(s1)).subscribe(); }).flatMap(s -> reactive.exec()) .doOnNext(transactionResults -> System.out.println(transactionResults.wasRolledBack())) .subscribe(); ``` --- ## File: docs/user-guide/redis-functions.md ## Redis Functions [Redis Functions](https://redis.io/topics/functions-intro) is an evolution of the scripting API to provide extensibility beyond Lua. Functions can leverage different engines and follow a model where a function library registers functionality to be invoked later with the `FCALL` command. ``` java redis.functionLoad("FUNCTION LOAD "#!lua name=mylib\nredis.register_function('knockknock', function() return 'Who\\'s there?' end)"); String response = redis.fcall("knockknock", STATUS); ``` Using Functions is straightforward. Consuming results in Java requires additional details to consume the result through a matching type. As we do not know what your function will return, the API uses call-site generics for you to specify the result type. Additionally, you must provide a `ScriptOutputType` hint to `EVAL` so that the driver uses the appropriate output parser. See [Output Formats](#output-formats) for further details. ### Output Formats You can choose from one of the following: - `BOOLEAN`: Boolean output, expects a number `0` or `1` to be converted to a boolean value. - `INTEGER`: 64-bit Integer output, represented as Java `Long`. - `MULTI`: List of flat arrays. - `STATUS`: Simple status value such as `OK`. The Redis response is parsed as ASCII. - `VALUE`: Value return type decoded through `RedisCodec`. - `OBJECT`: RESP3-defined object output supporting all Redis response structures. ### Leveraging Scripting and Functions through Command Interfaces Using dynamic functionality without a documented response structure can impose quite some complexity on your application. If you consider using scripting or functions, then you can use [Command Interfaces](../redis-command-interfaces.md) to declare an interface along with methods that represent your scripting or function landscape. Declaring a method with input arguments and a response type not only makes it obvious how the script or function is supposed to be called, but also how the response structure should look like. Let’s take a look at a simple function call first: ``` lua local function my_hlastmodified(keys, args) local hash = keys[1] return redis.call('HGET', hash, '_last_modified_') end ``` ``` java Long lastModified = redis.fcall("my_hlastmodified", INTEGER, "my_hash"); ``` This example calls the `my_hlastmodified` function expecting some `Long` response an input argument. Calling a function from a single place in your code isn’t an issue on its own. The arrangement becomes problematic once the number of functions grows or you start calling the functions with different arguments from various places in your code. Without the function code, it becomes impossible to investigate how the response mechanics work or determine the argument semantics, as there is no single place to document the function behavior. Let’s apply the Command Interface pattern to see how the the declaration and call sites change: ``` java interface MyCustomCommands extends Commands { /** * Retrieve the last modified value from the hash key. * @param hashKey the key of the hash. * @return the last modified timestamp, can be {@code null}. */ @Command("FCALL my_hlastmodified 1 :hashKey") Long getLastModified(@Param("my_hash") String hashKey); } MyCustomCommands myCommands = …; Long lastModified = myCommands.getLastModified("my_hash"); ``` By declaring a command method, you create a place that allows for storing additional documentation. The method declaration makes clear what the function call expects and what you get in return. --- ## File: docs/user-guide/redis-json.md # RedisJSON support in Lettuce Lettuce supports [RedisJSON](https://oss.redis.com/redisjson/) starting from [Lettuce 6.5.0.RELEASE](https://github.com/redis/lettuce/releases/tag/6.5.0.RELEASE). The driver generally allows three distinct ways of working with the RedisJSON module: * (Default mode) - default JSON parsing using Jackson behind the scenes * (Advanced mode) - custom JSON parsing using a user-provided JSON parser * (Power-user mode) - unprocessed JSON documents that have not gone through any process of deserialization or serialization !!! INFO In all the above modes, the driver would refrain from processing the JSON document in the main event loop and instead delegate this to the user thread. This behaviour is consistent when both receiving and sending JSON documents - when receiving the parsing is done lazily whenever a method is called that requires the JSON to be parsed; when sending the JSON is serialized immediately after it is passed to any of the commands, but before dispatching the command to the event loop. !!! WARNING Unless you are using a custom JSON parser you would need to add a dependency to the [jackson-databind](https://github.com/FasterXML/jackson-databind) library in your project. This is because the default JSON parser uses Jackson to parse JSON documents to and from string representations. ## Default mode Best for: * Most typical use-cases where the JSON document is parsed and processed ### Example usage ```java RedisURI redisURI = RedisURI.Builder.redis("acme.com").build(); RedisClient redisClient = RedisClient.create(redisURI); try (StatefulRedisConnection connect = redisClient.connect()){ redis = connect.async(); JsonPath path = JsonPath.of("$..mountain_bikes[0:2].model"); JsonParser parser = redis.getJsonParser(); JsonObject bikeRecord = parser.createJsonObject(); JsonObject bikeSpecs = parser.createJsonObject(); JsonArray bikeColors = parser.createJsonArray(); bikeSpecs.put("material", parser.createJsonValue("\"wood\"")); bikeSpecs.put("weight", parser.createJsonValue("19")); bikeColors.add(parser.createJsonValue("\"walnut\"")); bikeColors.add(parser.createJsonValue("\"chestnut\"")); bikeRecord.put("id", parser.createJsonValue("\"bike:13\"")); bikeRecord.put("model", parser.createJsonValue("\"Woody\"")); bikeRecord.put("description", parser.createJsonValue("\"The Woody is an environmentally-friendly wooden bike\"")); bikeRecord.put("price", parser.createJsonValue("\"1112\"")); bikeRecord.put("specs", bikeSpecs); bikeRecord.put("colors", bikeColors); String result = redis.jsonSet("bikes:inventory", path, bikeRecord).get(); } ``` ## Advanced mode Best for: * Applications that want to handle parsing manually - either by using another library or by implementing their own parser ### Example usage ```java RedisURI redisURI = RedisURI.Builder.redis("127.0.0.1").withPort(16379).build(); try (RedisClient client = RedisClient.create(redisURI)) { client.setOptions(ClientOptions.builder().jsonParser(Mono.just(new CustomParser())).build()); StatefulRedisConnection connection = client.connect(StringCodec.UTF8); RedisCommands redis = connection.sync(); } ``` ## Power-user mode Best for: * Applications that do little to no processing on the Java layer ### Example usage ```java JsonPath myPath = JsonPath.of("$..mountain_bikes"); RedisURI redisURI = RedisURI.Builder.redis("127.0.0.1").withPort(16379).build(); try (RedisClient client = RedisClient.create(redisURI)) { RedisAsyncCommands redis = client.connect().async(); RedisFuture> bikes = redis.jsonGet("bikes:inventory", myPath); CompletionStage> stage = bikes.thenApply( fetchedBikes -> redis.jsonSet("service_bikes", JsonPath.ROOT_PATH, fetchedBikes.get(0))); String result = stage.toCompletableFuture().get().get(); } ``` !!! NOTE The power-user mode is not exclusive to using a custom parser (Advanced mode), as long as the custom parser follows the API contract of the `JsonParser`, `JsonValue`, `JsonArray` and `JsonObject` interfaces. --- ## File: docs/user-guide/transactions-multi.md ## Transactions/Multi Transactions allow the execution of a group of commands in a single step. Transactions can be controlled using `WATCH`, `UNWATCH`, `EXEC`, `MULTI` and `DISCARD` commands. Synchronous, asynchronous, and reactive APIs allow the use of transactions. !!! note Transactional use requires external synchronization when a single connection is used by multiple threads/processes. This can be achieved either by serializing transactions or by providing a dedicated connection to each concurrent process. Lettuce itself does not synchronize transactional/non-transactional invocations regardless of the used API facade. Redis responds to commands invoked during a transaction with a `QUEUED` response. The response related to the execution of the command is received at the moment the `EXEC` command is processed, and the transaction is executed. The particular APIs behave in different ways: - Synchronous: Invocations to the commands return `null` while they are invoked within a transaction. The `MULTI` command carries the response of the particular commands. - Asynchronous: The futures receive their response at the moment the `EXEC` command is processed. This happens while the `EXEC` response is received. - Reactive: An `Obvervable` triggers `onNext`/`onCompleted` at the moment the `EXEC` command is processed. This happens while the `EXEC` response is received. As soon as you’re within a transaction, you won’t receive any responses on triggering the commands ``` java redis.multi() == "OK" redis.set(key, value) == null redis.exec() == list("OK") ``` You’ll receive the transactional response when calling `exec()` on the end of your transaction. ``` java redis.multi() == "OK" redis.set(key1, value) == null redis.set(key2, value) == null redis.exec() == list("OK", "OK") ``` ### Transactions using the asynchronous API Asynchronous use of Redis transactions is very similar to non-transactional use. The asynchronous API returns `RedisFuture` instances that eventually complete and they are handles to a future result. Regular commands complete as soon as Redis sends a response. Transactional commands complete as soon as the `EXEC` result is received. Each command is completed individually with its own result so users of `RedisFuture` will see no difference between transactional and non-transactional `RedisFuture` completion. That said, transactional command results are available twice: Once via `RedisFuture` of the command and once through `List` (`TransactionResult` since Lettuce 5) of the `EXEC` command future. ``` java RedisAsyncCommands async = client.connect().async(); RedisFuture multi = async.multi(); RedisFuture set = async.set("key", "value"); RedisFuture> exec = async.exec(); List objects = exec.get(); String setResult = set.get(); objects.get(0) == setResult ``` ### Transactions using the reactive API The reactive API can be used to execute multiple commands in a single step. The nature of the reactive API encourages nesting of commands. It is essential to understand the time at which an `Observable` emits a value when working with transactions. Redis responds with `QUEUED` to commands invoked during a transaction. The response related to the execution of the command is received at the moment the `EXEC` command is processed, and the transaction is executed. Subsequent calls in the processing chain are executed after the transactional end. The following code starts a transaction, executes two commands within the transaction and finally executes the transaction. ``` java RedisReactiveCommands reactive = client.connect().reactive(); reactive.multi().subscribe(multiResponse -> { reactive.set("key", "1").subscribe(); reactive.incr("key").subscribe(); reactive.exec().subscribe(); }); ``` ### Transactions on clustered connections Clustered connections perform a routing by default. This means, that you can’t be really sure, on which host your command is executed. So if you are working in a clustered environment, use rather a regular connection to your node, since then you’ll bound to that node knowing which hash slots are handled by it. ### Examples **Multi with executing multiple commands** ``` java redis.multi(); redis.set("one", "1"); redis.set("two", "2"); redis.mget("one", "two"); redis.llen(key); redis.exec(); // result: list("OK", "OK", list("1", "2"), 0L) ``` **Mult executing multiple asynchronous commands** ``` java redis.multi(); RedisFuture set1 = redis.set("one", "1"); RedisFuture set2 = redis.set("two", "2"); RedisFuture mget = redis.mget("one", "two"); RedisFuture llen = mgetredis.llen(key); set1.thenAccept(value -> …); // OK set2.thenAccept(value -> …); // OK RedisFuture> exec = redis.exec(); // result: list("OK", "OK", list("1", "2"), 0L) mget.get(); // list("1", "2") llen.thenAccept(value -> …); // 0L ``` **Using WATCH** ``` java redis.watch(key); RedisConnection redis2 = client.connect(); redis2.set(key, value + "X"); redis2.close(); redis.multi(); redis.append(key, "foo"); redis.exec(); // result is an empty list because of the changed key ``` ## Scripting and Functions Redis functionality can be extended through many ways, of which [Lua Scripting](https://redis.io/topics/eval-intro) and [Functions](https://redis.io/topics/functions-intro) are two approaches that do not require specific pre-requisites on the server. ## 2. Official Technical Reference & Guides (redis/docs) # Redis Docs > Note: if you are an AI agent, please see [AI_AGENT_DEVELOPER_GUIDE.md](AI_AGENT_DEVELOPER_GUIDE.md) for more information. PRs are merged to the `main` branch of this repo. Merging to `main` automatically builds and publishes your changes to the live docs site (under `/latest`) — there is no separate manual publish step, so allow a little time for the build and CDN cache to refresh after your PR is merged. Before your PR is merged, you can preview your changes on a per-branch staging URL; the docs team's bot posts the staging links as a comment on your PR. ## Site template files and folders * **/archetypes**: A Markdown file needs to have some front matter. An archetype defines which front matter is used when using `hugo new content`. Right now, the only supported archetype is the default one. **Note:** We might want to add additional archetypes in the future because most of our pages contain additional meta data properties like `linkTitle`. * **/content**: This folder contains the markdown files. We will have the subfolders like `/develop`, `/integrate`, and `/operate` * **/assets**: CSS files, site-wide icons, and images. * **/data**: Data files that are accessed by Hugo and rendered with the help of short codes or partials. * **/layouts/partials**: HTML templates that are used across sites. Examples are TOCs, breadcrumbs, or headers. * **/layouts/$type**: Each page type has at least the following templates to implement `single.html` and `list.html`. The `single` template is used to render a discrete page. The `list` template is used to render a collection of related pages (e.g., all sub-pages). * **/layouts/home.html**: The home page of the site, that is, the page that is displayed when you open the root path. * **/layouts/404.html**: The default 404 page. * **/layouts/shortcodes** HTML template snippets that you want to leverage within Markdown files. Examples are tabbed code examples or notification blocks. * **/public**: This is the folder that is generated by Hugo. This content needs to be published by site maintainers. * **/static**: Any static files that need to be accessed by the site, e.g., CSS or JavaScript. * **/package.json**: Node.js dependencies. Tailwind, for example, is installed via the Node package manager (`npm`). * **/config.toml**: Hugo's site configuration, like the root path and menu items. Hugo can access configuration elements when rendering the site. So you can define custom configuration settings here. * **/Makefile**: We use make to wrap some Hugo commands and to add additional build steps. * **/tailwind.config.js**: This is the Tailwind CSS framwork's configuration file. * **/postcss.config.js**: Needed to make Tailwind statically accessible to the site. ## Requirements You will need the following tools to build the site locally: - `python3`: Python >= 3.8. - `node` and `npm`: Node.js >= 19.7.0, and the Node.js package manager >= 9.5.0. - `hugo`: Hugo site generator, v0.143.0; the extended edition. - `make`: To run the makefile. - `git`: To manage the source code of this repo. ## Build script and data files The site can be built using the command `make all`. Here is what's executed: 1. Clean the current folder by deleting the generated output of a previous build. 1. Install the necessary Node.js and Python dependencies. 1. Build the components that were fetched from external Github repositories (e.g., client examples). This fetches source code files and generates data files. 1. Execute the `hugo` command to generate the HTML pages. The build result can be found within the folder `/public`. The command `make serve` performs the same steps, but serves the web pages on the local host for development purposes. The `Makefile` contains details about the executed commands. N.B. once you've built the site once, you can continuing working without rebuilding every time you make a change. Use `hugo serve` to serve the docs locally. The only time you really need to rebuild the site locally is when new code samples need to be pulled from the client sites. ## Build pipeline The build pipeline that is defined within `.github/workflows/main.yml` builds the documentation and uploads it into a GCS bucket: 1. Install Hugo 1. Check the branch out 1. Perform the build as stated before by using `make all` 1. Install the Google Cloud CLI 1. Authenticate to the GCS bucket 1. Validate the branch name 1. Sync the branch with a GCS folder ## Hugo specifics ### Relative links We are using the following syntax for Hugo relrefs: ``` [Link title]({{< relref "link relative to the site's base url" >}}) ``` Here is an example: ``` [Data structure store]({{< relref "/develop/get-started/data-store" >}}) ``` It's strongly advised to use `relref` because it provides the following advantages: 1. Links are checked at build time. Any broken references within the site are reported, stopping the build. 2. References are prefixed with the site's base URL, which means that they work in builds with a different base URL. The following needs to be taken into account when using `relref`: The reference `/develop/get-started/data-store` and `/develop/get-started/data-store/` aren't the same. You must use the trailing slash if the referenced article is an `_index.md` file within a folder (e.g., `.../data-store/` for `.../data-store/_index.md`). Otherwise, you should not use the trailing slash (e.g., `.../get-started/data-store.md`). ### Images The image shortcode doesn't need to be closed anymore. Here is an example; ``` {{< image filename="/images/rc/icon-database-update-status-pending.png" alt="Database update" >}} ``` The `filename` property value can be any file name path which is relative to the site's base path. We added a new property `class` which allows you to override the CSS class of the image. Images have by default a `block` display in Tailwind. You can change this by setting the class property to `inline`. The following example shows two images that are in a single line: ``` {{< image filename="/images/rc/icon-database-update-status-pending.png#no-click" alt="Pending database status" class="inline" >}}   {{< image filename="/images/rc/icon-database-status-active.png#no-click" alt="Active database status" class="inline" >}} ``` ### Templating Variables are scoped in the context of their block. Overriding a variable within an if block doesn't change the variable value as soon as you leave that block. This is a specific limitation of the Go templating language. The following will give you the `/` outside of the condition block if the path was initially set to `/`: ``` {{ $path := $parsed.Path }} {{ if (eq $path "/") }} {{ $path = "" }} {{ end }} {{- $path -}} ``` The solution is to use a scratchpad for storing the 'global' state: ``` {{ $path := $parsed.Path }} {{ if (eq $path "/") }} {{ .Scratch.Set "path" ""}} {{ else }} {{ .Scratch.Set "path" $path}} {{ end }} {{- .Scratch.Get "path" -}} ``` ### Child pages The redis.io template adds a list of links to child pages at the bottom of each section page. You can prevent this by adding the following front matter property: ``` hideListLinks: true ``` ## Tailwind specifics ### Fonts The fonts are located in the folder `static/fonts`. This folder contains typically file of the formats `ttf`, `woff`, or `woff2`. You can perform the following steps to add a font: * Copy your font files to the fonts folder `static/fonts`. * Ensure that the font is loaded by adding it to the partial `layouts/partials/fonts.html`. This partial defines the font using a style sheet and ensures that the font resources are pre-loaded. Here is an example that shows how to define the font face 'Geist Mono' with a regular font weight: ``` @font-face { font-family: 'Geist Mono'; src: url('{{ $basePath }}/fonts/GeistMono-Regular.woff2') format('woff2'); font-weight: 400; font-style: normal; font-display: swap; } ``` * Make your font available via Tailwind by adding it to the `tailwind.config.js` file. ``` module.exports = { content: ["content/**/*.md", "layouts/**/*.html", "./tailwindcss.whitelist.txt"], theme: { extend: { fontFamily: { sans: [ 'Space Grotesk', ...defaultTheme.fontFamily.sans ], mono: [ 'Space Mono', 'SF Mono', ...defaultTheme.fontFamily.mono ], monogeist: ['Geist Mono', ...defaultTheme.fontFamily.mono ], }, ... ``` * Use a `font-*` CSS class within HTML or your style sheets, whereby the `*` needs to be substituted with one of the Tailwind fonts that you defined in the Tailwind configuration file (`tailwind.config.js`). Here is an example how to use `font-monogeist` in the global CSS file `assets/css/index.css` to change the font of code blocks: ``` .prose pre > code { @apply bg-none font-monogeist; } ``` ## Troubleshooting ### NIL pointer error when rendering a section You might sometimes see the following error message: ``` nil pointer evaluating page.Page.Params ``` This happened for us when Hugo executed the `bannerText` logic, which inspected all parent pages by checking if they have a banner text set. The problem was that one of the parent folders was missing an `_index.md` file, which meant the folder couldn't be identified as a page. The solution was to add an `_index.md` file to the folder that didn't have it yet.