Technical Documentation: redis/redis-py
βΉοΈ Provenance: Multi-Tier Ingestion Β· CodeWiki Reference
Overview (`README.md`)
redis-py
The Python interface to the Redis key-value store.
Installation | Usage | Advanced Topics | Contributing
Note: redis-py 5.0 is the last version of redis-py that supports Python 3.7, as it has reached end of life. redis-py 5.1 supports Python 3.8+.
Note: redis-py 6.1.0 is the last version of redis-py that supports Python 3.8, as it has reached end of life. redis-py 6.2.0 supports Python 3.9+.
How do I Redis?
Learn for free at Redis University
Installation
Start a redis via docker (for Redis versions >= 8.0):
docker run -p 6379:6379 -it redis:latestStart a redis via docker (for Redis versions < 8.0):
docker run -p 6379:6379 -it redis/redis-stack:latestTo install redis-py, simply:
$ pip install redisFor faster performance, install redis with hiredis support, this provides a compiled response parser, and for most cases requires zero code changes.
By default, if hiredis >= 1.0 is available, redis-py will attempt to use it for response parsing.
$ pip install "redis[hiredis]"Looking for a high-level library to handle object mapping? See redis-om-python!
Supported Redis Versions
The most recent version of this library supports Redis version 7.2, 7.4, 8.0, 8.2, 8.4, 8.6 and 8.8.
The table below highlights version compatibility of the most-recent library versions and redis versions.
| Library version | Supported redis versions |
|---|---|
| 3.5.3 | <= 6.2 Family of releases |
| >= 4.5.0 | Version 5.0 to 7.0 |
| >= 5.0.0 | Version 5.0 to 7.4 |
| >= 6.0.0 | Version 7.2 to current |
Usage
Basic Example
>>> import redis
>>> r = redis.Redis(host='localhost', port=6379, db=0)
>>> r.set('foo', 'bar')
True
>>> r.get('foo')
b'bar'The above code connects to localhost on port 6379, sets a value in Redis, and retrieves it. All responses are returned as bytes in Python, to receive decoded strings, set decode_responses=True. For this, and more connection options, see these examples.
RESP3 Support
redis-py supports RESP3 starting with version 5.0. Starting with redis-py 8.0,
clients use RESP3 on the wire by default while preserving legacy
RESP2-compatible Python response shapes for existing applications.
Set protocol=3 explicitly when your application should receive
RESP3-specific response shapes or when you want the wire protocol choice to be
visible in code. To force RESP2 on the wire, set protocol=2. To opt in to
protocol-independent response shapes, set legacy_responses=False.
For new projects, we recommend opting out of legacy response compatibility by
setting legacy_responses=False. This makes redis-py return unified Python
response shapes for affected commands whether the connection uses RESP2 or
RESP3. Existing applications can keep the default legacy-compatible behavior
while they migrate response handling at their own pace.
>>> import redis
>>> r = redis.Redis(host='localhost', port=6379, db=0, legacy_responses=False)See the unified responses migration guide
for activation instructions and command-by-command response differences. For
RESP3-specific behavior, see the RESP3 features guide.
Connection Pools
By default, redis-py uses a connection pool to manage connections. Each instance of a Redis class receives its own connection pool. You can however define your own redis.ConnectionPool.
>>> pool = redis.ConnectionPool(host='localhost', port=6379, db=0)
>>> r = redis.Redis(connection_pool=pool)Alternatively, you might want to look at Async connections, or Cluster connections, or even Async Cluster connections.
Redis Commands
There is built-in support for all of the out-of-the-box Redis commands. They are exposed using the raw Redis command names (HSET, HGETALL, etc.) except where a word (i.e. del) is reserved by the language. The complete set of commands can be found here, or the documentation.
Advanced Topics
The official Redis command documentation
does a great job of explaining each command in detail. redis-py attempts
to adhere to the official command syntax. There are a few exceptions:
MULTI/EXEC: These are implemented as part of the Pipeline class.
The pipeline is wrapped with the MULTI and EXEC statements by
default when it is executed, which can be disabled by specifying
transaction=False. See more about Pipelines below.SUBSCRIBE/LISTEN: Similar to pipelines, PubSub is implemented as
a separate class as it places the underlying connection in a state
where it can't execute non-pubsub commands. Calling the pubsub
method from the Redis client will return a PubSub instance where you
can subscribe to channels and listen for messages. You can only call
PUBLISH from the Redis client (see this comment on issue
#151
for details).
For more details, please see the documentation on advanced topics page.
Pipelines
The following is a basic example of a Redis pipeline, a method to optimize round-trip calls, by batching Redis commands, and receiving their results as a list.
>>> pipe = r.pipeline()
>>> pipe.set('foo', 5)
>>> pipe.set('bar', 18.5)
>>> pipe.set('blee', "hello world!")
>>> pipe.execute()
[True, True, True]PubSub
The following example shows how to utilize Redis Pub/Sub to subscribe to specific channels.
>>> r = redis.Redis(...)
>>> p = r.pubsub()
>>> p.subscribe('my-first-channel', 'my-second-channel', ...)
>>> p.get_message()
{'pattern': None, 'type': 'subscribe', 'channel': b'my-second-channel', 'data': 1}Redisβ search and query capabilities default dialect
Release 6.0.0 introduces a client-side default dialect for Redisβ search and query capabilities.
By default, the client now overrides the server-side dialect with version 2, automatically appending DIALECT 2 to commands like FT.AGGREGATE and FT.SEARCH.
Important: Be aware that the query dialect may impact the results returned. If needed, you can revert to a different dialect version by configuring the client accordingly.
>>> from redis.commands.search.field import TextField
>>> from redis.commands.search.query import Query
>>> from redis.commands.search.index_definition import IndexDefinition
>>> import redis
>>> r = redis.Redis(host='localhost', port=6379, db=0)
>>> r.ft().create_index(
>>> (TextField("name"), TextField("lastname")),
>>> definition=IndexDefinition(prefix=["test:"]),
>>> )
>>> r.hset("test:1", "name", "James")
>>> r.hset("test:1", "lastname", "Brown")
>>> # Query with default DIALECT 2
>>> query = "@name: James Brown"
>>> q = Query(query)
>>> res = r.ft().search(q)
>>> # Query with explicit DIALECT 1
>>> query = "@name: James Brown"
>>> q = Query(query).dialect(1)
>>> res = r.ft().search(q)You can find further details in the query dialect documentation.
Multi-database client (Active-Active)
The multi-database client allows your application to connect to multiple Redis databases, which are typically replicas of each other. It is designed to work with Redis Software and Redis Cloud Active-Active setups. The client continuously monitors database health, detects failures, and automatically fails over to the next healthy database using a configurable strategy. When the original database becomes healthy again, the client can automatically switch back to it.
This is useful when:
- You have more than one Redis deployment. This might include two independent Redis servers or two or more Redis databases replicated across multiple active-active Redis Enterprise clusters.
- You want your application to connect to one deployment at a time and to fail over to the next available deployment if the first deployment becomes unavailable.
For the complete failover configuration options and examples, see the Multi-database client docs.
Bulk hash ingestion (HIMPORT)
Redis 8.10 adds the HIMPORT command family for loading many hashes that share
the same set of field names: register the field names once with himport_prepare,
then create each hash by sending only its values. Keys written this way are regular
hashes β every hash command works on them.
r.himport_prepare("users", ["name", "email", "age"])
r.himport_set("user:1", "users", ["alice", "[email protected]", "25"])
r.himport_set("user:2", "users", ["bob", "[email protected]", "30"])
r.himport_discard("users") # => 1Values pair positionally with the prepared fields. Hash enumeration order
(HGETALL, HKEYS) is not guaranteed to match the prepare order.
Fieldsets are connection state. A prepared fieldset lives in the server-side
session of the physical connection that prepared it: it is invisible to other
connections and destroyed by a disconnect or RESET. redis-py handles this for you βhimport_prepare records the fieldset in a client-level registry, and the PREPARE
is applied lazily on whatever pooled connection serves each himport_set (and
re-applied automatically after a reconnect, RESET, or Sentinel/cluster failover).
You declare each fieldset once per client with himport_prepare; there is no
constructor argument for it.
For the highest ingestion throughput, send the PREPARE and its SETs in one
pipeline β a single batch always executes on one connection:
with r.pipeline(transaction=False) as pipe:
pipe.himport_prepare("users", ["name", "email", "age"])
for uid, row in rows:
pipe.himport_set(f"user:{uid}", "users", row)
pipe.execute()The automatic re-prepare applies to direct calls only, not to commands insidepipeline/transaction blocks: a batched himport_set relies on the single
pre-flight PREPARE in that batch.
With RedisCluster, himport_prepare / himport_discard / himport_discard_all
update the client's shared, cluster-wide registry and return immediately β like the
standalone API, they perform no server I/O of their own. The server-side PREPARE
(and, after a discard, DISCARD) is applied lazily on each node's connection the
next time it serves an himport_set, and re-applied after reconnects or failover;himport_set itself routes by the key's hash slot. A discard is therefore not
removed from every server session at once: each connection drops the fieldset on its
next himport_set (or on disconnect). With Sentinel, callhimport_prepare on the long-lived client returned by master_for(...); the fieldset
survives failover automatically. HIMPORT is not supported on the multi-database
(Active-Active) client.
The async client mirrors this exactly:
await r.himport_prepare("users", ["name", "email", "age"])
await r.himport_set("user:1", "users", ["alice", "[email protected]", "25"])Author
redis-py is developed and maintained by Redis Inc. It can be found here, or downloaded from pypi.
Special thanks to:
- Andy McCurdy ([email protected]) the original author of redis-py.
- Ludovico Magnocavallo, author of the original Python Redis client,
from which some of the socket code is still used. - Alexander Solovyov for ideas on the generic response callback
system. - Paul Hubbard for initial packaging support.
Overview (`benchmarks/README.md`)
Benchmarks
This directory contains local benchmark scripts for redis-py. They are not part
of the normal CI test matrix.
OpenTelemetry Benchmarks
otel_benchmark.py measures redis-py operation throughput and latency with
different OpenTelemetry configurations:
baseline: redis-py from a git tag, without the OpenTelemetry code path.otel_disabled: current working tree with observability not initialized.otel_noop: observability initialized with a no-op meter provider.otel_inmemory: observability initialized with an in-memory metric reader.otel_enabled_http: metrics exported over OTLP/HTTP.otel_enabled_grpc: metrics exported over OTLP/gRPC.
Run the commands below from the repository root.
The regular development requirements install the OTLP/HTTP exporter only. Install
the gRPC exporter separately when you want to run the otel_enabled_grpc
scenario:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r dev_requirements.txt
python -m pip install -r benchmarks/requirements.txtStart Redis before running any scenario. The OpenTelemetry example stack also
starts an OTLP collector on ports 4317 and 4318, which is useful for the HTTP
and gRPC exporter scenarios:
docker compose -f docs/examples/opentelemetry/docker-compose.yml up -dChoose one way to run the benchmark and use the matching examples below.
Run as a module
If you prefer to run the benchmark with python -m, use these commands.
Run one scenario at a time:
python -m benchmarks.otel_benchmark --scenario otel_disabled
python -m benchmarks.otel_benchmark --scenario otel_noop
python -m benchmarks.otel_benchmark --scenario otel_inmemory
python -m benchmarks.otel_benchmark --scenario otel_enabled_http
python -m benchmarks.otel_benchmark --scenario otel_enabled_grpcFor a shorter smoke run while checking setup, lower the duration and iteration
count:
python -m benchmarks.otel_benchmark --scenario otel_enabled_grpc --duration 10 --warmup 2 --iterations 1Run the baseline scenario with a redis-py release tag:
python -m benchmarks.otel_benchmark --scenario baseline --baseline-tag v5.2.1Use --async to run the async client variant:
python -m benchmarks.otel_benchmark --scenario otel_enabled_http --asyncBy default the benchmark sends exporter traffic to localhost. SetOTEL_COLLECTOR_HOST when the collector is on another host:
OTEL_COLLECTOR_HOST=collector.example.com python -m benchmarks.otel_benchmark --scenario otel_enabled_grpcRun the file directly
If you prefer to run the Python file directly, use these commands instead.
Run one scenario at a time:
python benchmarks/otel_benchmark.py --scenario otel_disabled
python benchmarks/otel_benchmark.py --scenario otel_noop
python benchmarks/otel_benchmark.py --scenario otel_inmemory
python benchmarks/otel_benchmark.py --scenario otel_enabled_http
python benchmarks/otel_benchmark.py --scenario otel_enabled_grpcFor a shorter smoke run while checking setup, lower the duration and iteration
count:
python benchmarks/otel_benchmark.py --scenario otel_enabled_grpc --duration 10 --warmup 2 --iterations 1Run the baseline scenario with a redis-py release tag:
python benchmarks/otel_benchmark.py --scenario baseline --baseline-tag v5.2.1Use --async to run the async client variant:
python benchmarks/otel_benchmark.py --scenario otel_enabled_http --asyncBy default the benchmark sends exporter traffic to localhost. SetOTEL_COLLECTOR_HOST when the collector is on another host:
OTEL_COLLECTOR_HOST=collector.example.com python benchmarks/otel_benchmark.py --scenario otel_enabled_grpcStop the example stack when finished:
docker compose -f docs/examples/opentelemetry/docker-compose.yml downOverview (`doctests/README.md`)
Command examples for redis.io
How to add an example
Create regular python file in the current folder with meaningful name. It makes sense prefix example files with
command category (e.g. string, set, list, hash, etc) to make navigation in the folder easier. Files ending in .py
are automatically run by the test suite.
Special markup
See https://github.com/redis-stack/redis-stack-website#readme for more details.
How to test examples
Examples are standalone python scripts, committed to the doctests directory. These scripts assume that thedoctests/requirements.txt and dev_requirements.txt from this repository have been installed, as per below.
pip install -r dev_requirements.txt
pip uninstall -y redis # uninstall Redis package installed via redis-entraid
pip install -r doctests/requirements.txtNote - the CI process, runs linters against the examples. Assuming
the requirements above have been installed you can run ruff check yourfile.py and ruff format yourfile.py
locally to validate the linting, prior to CI.
Just include necessary assertions in the example file and run
sh doctests/run_examples.shto test all examples in the current folder.
Overview (`docs/examples/README.md`)
Examples
Examples of redis-py usage go here. They're being linked to the generated documentation.
Overview (`docs/examples/opentelemetry/README.md`)
Example for redis-py OpenTelemetry instrumentation
This example demonstrates how to monitor Redis using OpenTelemetry and
Uptrace. It requires Docker to start Redis Server and Uptrace.
See
Monitoring redis-py performance with OpenTelemetry
for details.
Step 1. Download the example using Git:
git clone https://github.com/redis/redis-py.git
cd example/opentelemetryStep 2. Optionally, create a virtualenv:
python3 -m venv .venv
source .venv/bin/activeStep 3. Install dependencies:
pip install -e .Step 4. Start the services using Docker and make sure Uptrace is running:
docker-compose up -d
docker-compose logs uptraceStep 5. Run the Redis client example and follow the link from the CLI to view the trace:
python3 main.py
trace: http://localhost:14318/traces/ee029d8782242c8ed38b16d961093b35You can also open Uptrace UI at http://localhost:14318 to view available
spans, logs, and metrics.
Advanced Features (`docs/advanced_features.rst`)
Advanced Features
A note about threading
Redis client instances can safely be shared between threads. Internally,
connection instances are only retrieved from the connection pool during
command execution, and returned to the pool directly after. Command
execution never modifies state on the client instance.
However, there is one caveat: the Redis SELECT command. The SELECT
command allows you to switch the database currently in use by the
connection. That database remains selected until another is selected or
until the connection is closed. This creates an issue in that
connections could be returned to the pool that are connected to a
different database.
As a result, redis-py does not implement the SELECT command on client
instances. If you use multiple Redis databases within the same
application, you should create a separate client instance (and possibly
a separate connection pool) for each database.
It is not safe to pass PubSub or Pipeline objects between threads.
Pipelines
Default pipelines
Pipelines are a subclass of the base Redis class that provide support
for buffering multiple commands to the server in a single request. They
can be used to dramatically increase the performance of groups of
commands by reducing the number of back-and-forth TCP packets between
the client and server.
Pipelines are quite simple to use:
.. code:: python
>>> r = redis.Redis(...)
>>> r.set('bing', 'baz')
>>> # Use the pipeline() method to create a pipeline instance
>>> pipe = r.pipeline()
>>> # The following SET commands are buffered
>>> pipe.set('foo', 'bar')
>>> pipe.get('bing')
>>> # the EXECUTE call sends all buffered commands to the server, returning
>>> # a list of responses, one for each command.
>>> pipe.execute()
[True, b'baz']
For ease of use, all commands being buffered into the pipeline return
the pipeline object itself. Therefore calls can be chained like:
.. code:: python
>>> pipe.set('foo', 'bar').sadd('faz', 'baz').incr('auto_number').execute()
[True, True, 6]
In addition, pipelines can also ensure the buffered commands are
executed atomically as a group. This happens by default. If you want to
disable the atomic nature of a pipeline but still want to buffer
commands, you can turn off transactions.
.. code:: python
>>> pipe = r.pipeline(transaction=False)
A common issue occurs when requiring atomic transactions but needing to
retrieve values in Redis prior for use within the transaction. For
instance, let's assume that the INCR command didn't exist and we need to
build an atomic version of INCR in Python.
The completely naive implementation could GET the value, increment it in
Python, and SET the new value back. However, this is not atomic because
multiple clients could be doing this at the same time, each getting the
same value from GET.
Enter the WATCH command. WATCH provides the ability to monitor one or
more keys prior to starting a transaction. If any of those keys change
prior the execution of that transaction, the entire transaction will be
canceled and a WatchError will be raised. To implement our own
client-side INCR command, we could do something like this:
.. code:: python
>>> with r.pipeline() as pipe:
... while True:
... try:
... # put a WATCH on the key that holds our sequence value
... pipe.watch('OUR-SEQUENCE-KEY')
... # after WATCHing, the pipeline is put into immediate execution
... # mode until we tell it to start buffering commands again.
... # this allows us to get the current value of our sequence
... current_value = pipe.get('OUR-SEQUENCE-KEY')
... next_value = int(current_value) + 1
... # now we can put the pipeline back into buffered mode with MULTI
... pipe.multi()
... pipe.set('OUR-SEQUENCE-KEY', next_value)
... # and finally, execute the pipeline (the set command)
... pipe.execute()
... # if a WatchError wasn't raised during execution, everything
... # we just did happened atomically.
... break
... except WatchError:
... # another client must have changed 'OUR-SEQUENCE-KEY' between
... # the time we started WATCHing it and the pipeline's execution.
... # our best bet is to just retry.
... continue
Note that, because the Pipeline must bind to a single connection for the
duration of a WATCH, care must be taken to ensure that the connection is
returned to the connection pool by calling the reset() method. If the
Pipeline is used as a context manager (as in the example above) reset()
will be called automatically. Of course you can do this the manual way
by explicitly calling reset():
.. code:: python
>>> pipe = r.pipeline()
>>> while True:
... try:
... pipe.watch('OUR-SEQUENCE-KEY')
... ...
... pipe.execute()
... break
... except WatchError:
... continue
... finally:
... pipe.reset()
A convenience method named "transaction" exists for handling all the
boilerplate of handling and retrying watch errors. It takes a callable
that should expect a single parameter, a pipeline object, and any number
of keys to be WATCHed. Our client-side INCR command above can be written
like this, which is much easier to read:
.. code:: python
>>> def client_side_incr(pipe):
... current_value = pipe.get('OUR-SEQUENCE-KEY')
... next_value = int(current_value) + 1
... pipe.multi()
... pipe.set('OUR-SEQUENCE-KEY', next_value)
>>>
>>> r.transaction(client_side_incr, 'OUR-SEQUENCE-KEY')
[True]
Be sure to call pipe.multi() in the callable passed to Redis.transaction
prior to any write commands.
Pipelines in clustersClusterPipeline is a subclass of RedisCluster that provides support for
Redis pipelines in cluster mode. When calling the execute() command, all
the commands are grouped by the node on which they will be executed, and
are then executed by the respective nodes in parallel. The pipeline
instance will wait for all the nodes to respond before returning the
result to the caller. Command responses are returned as a list sorted in
the same order in which they were sent. Pipelines can be used to
dramatically increase the throughput of Redis Cluster by significantly
reducing the number of network round trips between the client and
the server.
.. code:: python
rc = RedisCluster()
with rc.pipeline() as pipe:
... pipe.set('foo', 'value1')
... pipe.set('bar', 'value2')
... pipe.get('foo')
... pipe.get('bar')
... print(pipe.execute())
[True, True, b'value1', b'value2']
... pipe.set('foo1', 'bar1').get('foo1').execute()
[True, b'bar1']
Please note:
- RedisCluster pipelines currently only support key-based commands.
- The pipeline gets its βload_balancing_strategyβ value from the
clusterβs parameter. Thus, if read from replications is enabled in
the cluster instance, the pipeline will also direct read commands to
replicas.
Transactions in clusters
Transactions are supported in cluster-mode with one caveat: all keys of
all commands issued on a transaction pipeline must reside on the
same slot. This is similar to the limitation of multikey commands in
cluster. The reason behind this is that the Redis engine does not offer
a mechanism to block or exchange key data across nodes on the fly. A
client may add some logic to abstract engine limitations when running
on a cluster, such as the pipeline behavior explained on the previous
block, but there is no simple way that a client can enforce atomicity
across nodes on a distributed system.
The compromise of limiting the transaction pipeline to same-slot keys
is exactly that: a compromise. While this behavior is different from
non-transactional cluster pipelines, it simplifies migration of clients
from standalone to cluster under some circumstances. Note that application
code that issues multi/exec commands on a standalone client without
embedding them within a pipeline would eventually get βAttributeErrorβ.
With this approach, if the application uses βclient.pipeline(transaction=True)β,
then switching the client with a cluster-aware instance would simplify
code changes (to some extent). This may be true for application code that
makes use of hash keys, since its transactions may already be
mapping all commands to the same slot.
An alternative is some kind of two-step commit solution, where a slot
validation is run before the actual commands are run. This could work
with controlled node maintenance but does not cover single node failures.
Given the cluster limitations for transactions, by default pipeline isn't in
transactional mode. To enable transactional context set:
.. code:: python
>>> p = rc.pipeline(transaction=True)
After entering the transactional context you can add commands to a transactional
context, by one of the following ways:
.. code:: python
>>> p = rc.pipeline(transaction=True) # Chaining commands
>>> p.set("key", "value")
>>> p.get("key")
>>> response = p.execute()
Or
.. code:: python
>>> with rc.pipeline(transaction=True) as pipe: # Using context manager
... pipe.set("key", "value")
... pipe.get("key")
... response = pipe.execute()
As you see there's no need to explicitly send `MULTI/EXEC` commands to control context start/end
`ClusterPipeline` will take care of it.
To ensure that different keys will be mapped to a same hash slot on the server side
prepend your keys with the same hash tag, the technique that allows you to control
keys distribution.
More information `here <https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/#hash-tags>`_
.. code:: python
>>> with rc.pipeline(transaction=True) as pipe:
... pipe.set("{tag}foo", "bar")
... pipe.set("{tag}bar", "foo")
... pipe.get("{tag}foo")
... pipe.get("{tag}bar")
... response = pipe.execute()
CAS TransactionsIf you want to apply optimistic locking for certain keys, you have to executeWATCH command in transactional context. WATCH command follows the same limitations
as any other multi key command - all keys should be mapped to the same hash slot.
However, the difference between CAS transaction and normal one is that you have to
explicitly call MULTI command to indicate the start of transactional context, WATCH
command itself and any subsequent commands before MULTI will be immediately executed
on the server side so you can apply optimistic locking and get necessary data before
transaction execution.
.. code:: python
with rc.pipeline(transaction=True) as pipe:
... pipe.watch("mykey") # Apply locking by immediately executing command
... val = pipe.get("mykey") # Immediately retrieves value
... val = val + 1 # Increment value
... pipe.multi() # Starting transaction context
... pipe.set("mykey", val) # Command will be pipelined
... response = pipe.execute() # Returns OK or None if key was modified in the meantime
Publish / Subscribe
redis-py includes a PubSub object that subscribes to channels and
listens for new messages. Creating a PubSub object is easy.
.. code:: python
r = redis.Redis(...)
p = r.pubsub()
Once a PubSub instance is created, channels and patterns can be
subscribed to.
.. code:: python
p.subscribe('my-first-channel', 'my-second-channel', ...)
p.psubscribe('my-*', ...)
The PubSub instance is now subscribed to those channels/patterns. The
subscription confirmations can be seen by reading messages from the
PubSub instance.
.. code:: python
p.get_message()
{'pattern': None, 'type': 'subscribe', 'channel': b'my-second-channel', 'data': 1}
p.get_message()
{'pattern': None, 'type': 'subscribe', 'channel': b'my-first-channel', 'data': 2}
p.get_message()
{'pattern': None, 'type': 'psubscribe', 'channel': b'my-*', 'data': 3}
Every message read from a PubSub instance will be a dictionary with the
following keys.
- type: One of the following: 'subscribe', 'unsubscribe',
'psubscribe', 'punsubscribe', 'message', 'pmessage' - channel: The channel [un]subscribed to or the channel a message
was published to - pattern: The pattern that matched a published message's channel.
Will be None in all cases except for 'pmessage' types. - data: The message data. With [un]subscribe messages, this value
will be the number of channels and patterns the connection is
currently subscribed to. With [p]message messages, this value will be
the actual published message.
Let's send a message now.
.. code:: python
the publish method returns the number matching channel and pattern
subscriptions. 'my-first-channel' matches both the 'my-first-channel'
subscription and the 'my-*' pattern subscription, so this message will
be delivered to 2 channels/patterns
r.publish('my-first-channel', 'some data')
2
p.get_message()
{'channel': b'my-first-channel', 'data': b'some data', 'pattern': None, 'type': 'message'}
p.get_message()
{'channel': b'my-first-channel', 'data': b'some data', 'pattern': b'my-*', 'type': 'pmessage'}
Unsubscribing works just like subscribing. If no arguments are passed to
[p]unsubscribe, all channels or patterns will be unsubscribed from.
.. code:: python
p.unsubscribe()
p.punsubscribe('my-')
p.get_message()
{'channel': b'my-second-channel', 'data': 2, 'pattern': None, 'type': 'unsubscribe'}
p.get_message()
{'channel': b'my-first-channel', 'data': 1, 'pattern': None, 'type': 'unsubscribe'}
p.get_message()
{'channel': b'my-', 'data': 0, 'pattern': None, 'type': 'punsubscribe'}
redis-py also allows you to register callback functions to handle
published messages. Message handlers take a single argument, the
message, which is a dictionary just like the examples above. To
subscribe to a channel or pattern with a message handler, pass the
channel or pattern name as a keyword argument with its value being the
callback function. For channel or pattern names that cannot be used as
Python keyword arguments, including arbitrary binary names, pass aredis.Subscription object as a positional argument instead.
Subscription forms can be mixed in the same call. Assumingmy_handler is a callback, a plain channel name can be supplied
alongside a Subscription object, and Subscription objects can
also be used together with the existing keyword-argument handler style.
.. code:: python
p.subscribe('logs', redis.Subscription(b'\x80\x81', my_handler))
p.subscribe(redis.Subscription(b'\x82\x83', my_handler), **{'my-channel': my_handler})
When a message is read on a channel or pattern with a message handler,
the message dictionary is created and passed to the message handler. In
this case, a None value is returned from get_message() since the message
was already handled.
.. code:: python
def my_handler(message):
... print('MY HANDLER: ', message['data'])
p.subscribe(**{'my-channel': my_handler})
p.subscribe(redis.Subscription(b'\x80\x81', my_handler))
read the subscribe confirmation message
p.get_message()
{'pattern': None, 'type': 'subscribe', 'channel': b'my-channel', 'data': 1}
r.publish('my-channel', 'awesome data')
1
for the message handler to work, we need tell the instance to read data.
this can be done in several ways (read more below). we'll just use
the familiar get_message() function for now
message = p.get_message()
MY HANDLER: awesome data
note here that the my_handler callback printed the string above.
`message` is None because the message was handled by our handler.
print(message)
None
If your application is not interested in the (sometimes noisy)
subscribe/unsubscribe confirmation messages, you can ignore them by
passing ignore_subscribe_messages=True to r.pubsub(). This will cause
all subscribe/unsubscribe messages to be read, but they won't bubble up
to your application.
.. code:: python
p = r.pubsub(ignore_subscribe_messages=True)
p.subscribe('my-channel')
p.get_message() # hides the subscribe message and returns None
r.publish('my-channel', 'my data')
1
p.get_message()
{'channel': b'my-channel', 'data': b'my data', 'pattern': None, 'type': 'message'}
There are three different strategies for reading messages.
The examples above have been using pubsub.get_message(). Behind the
scenes, get_message() uses the system's 'select' module to quickly poll
the connection's socket. If there's data available to be read,
get_message() will read it, format the message and return it or pass it
to a message handler. If there's no data to be read, get_message() will
immediately return None. This makes it trivial to integrate into an
existing event loop inside your application.
.. code:: python
while True:
message = p.get_message()
if message:
# do something with the message
time.sleep(0.001) # be nice to the system :)
Older versions of redis-py only read messages with pubsub.listen().
listen() is a generator that blocks until a message is available. If
your application doesn't need to do anything else but receive and act on
messages received from redis, listen() is an easy way to get up an
running.
.. code:: python
for message in p.listen():
... # do something with the message
The third option runs an event loop in a separate thread.
pubsub.run_in_thread() creates a new thread and starts the event loop.
The thread object is returned to the caller of run_in_thread(). The
caller can use the thread.stop() method to shut down the event loop and
thread. Behind the scenes, this is simply a wrapper around get_message()
that runs in a separate thread, essentially creating a tiny non-blocking
event loop for you. run_in_thread() takes an optional sleep_time
argument. If specified, the event loop will call time.sleep() with the
value in each iteration of the loop.
Note: Since we're running in a separate thread, there's no way to handle
messages that aren't automatically handled with registered message
handlers. Therefore, redis-py prevents you from calling run_in_thread()
if you're subscribed to patterns or channels that don't have message
handlers attached.
.. code:: python
p.subscribe(**{'my-channel': my_handler})
thread = p.run_in_thread(sleep_time=0.001)
the event loop is now running in the background processing messages
when it's time to shut it down...
thread.stop()
run_in_thread also supports an optional exception handler, which lets
you catch exceptions that occur within the worker thread and handle them
appropriately. The exception handler will take as arguments the
exception itself, the pubsub object, and the worker thread returned by
run_in_thread.
.. code:: python
p.subscribe(**{'my-channel': my_handler})
def exception_handler(ex, pubsub, thread):
print(ex)
thread.stop()
thread = p.run_in_thread(exception_handler=exception_handler)
A PubSub object adheres to the same encoding semantics as the client
instance it was created from. Any channel or pattern that's unicode will
be encoded using the encoding specified on the client before being sent
to Redis. If the client's decode_responses flag is set the False (the
default), the 'channel', 'pattern' and 'data' values in message
dictionaries will be byte strings (str on Python 2, bytes on Python 3).
If the client's decode_responses is True, then the 'channel', 'pattern'
and 'data' values will be automatically decoded to unicode strings using
the client's encoding.
PubSub objects remember what channels and patterns they are subscribed
to. In the event of a disconnection such as a network error or timeout,
the PubSub object will re-subscribe to all prior channels and patterns
when reconnecting. Messages that were published while the client was
disconnected cannot be delivered. When you're finished with a PubSub
object, call its .close() method to shutdown the connection.
.. code:: python
p = r.pubsub()
...
p.close()
The PUBSUB set of subcommands CHANNELS, NUMSUB and NUMPAT are also
supported:
.. code:: python
r.pubsub_channels()
[b'foo', b'bar']
r.pubsub_numsub('foo', 'bar')
[(b'foo', 9001), (b'bar', 42)]
r.pubsub_numsub('baz')
[(b'baz', 0)]
r.pubsub_numpat()
1204
Sharded pubsub
`Sharded pubsub <https://redis.io/docs/interact/pubsub/#:~:text=Sharded%20Pub%2FSub%20helps%20to,the%20shard%20of%20a%20cluster.>`_ is a feature introduced with Redis 7.0, and fully supported by redis-py as of 5.0. It helps scale the usage of pub/sub in cluster mode, by having the cluster shard messages to nodes that own a slot for a shard channel. Here, the cluster ensures the published shard messages are forwarded to the appropriate nodes. Clients subscribe to a channel by connecting to either the master responsible for the slot, or any of its replicas.
This makes use of the `SSUBSCRIBE <https://redis.io/commands/ssubscribe>`_ and `SPUBLISH <https://redis.io/commands/spublish>`_ commands within Redis.
The following, is a simplified example:
.. code:: python
>>> from redis.cluster import RedisCluster, ClusterNode
>>> r = RedisCluster(startup_nodes=[ClusterNode('localhost', 6379), ClusterNode('localhost', 6380)])
>>> p = r.pubsub()
>>> p.ssubscribe('foo')
>>> # assume someone sends a message along the channel via a publish
>>> message = p.get_sharded_message()
Similarly, the same process can be used to acquire sharded pubsub messages, that have already been sent to a specific node, by passing the node to get_sharded_message:
.. code:: python
>>> from redis.cluster import RedisCluster, ClusterNode
>>> first_node = ClusterNode['localhost', 6379]
>>> second_node = ClusterNode['localhost', 6380]
>>> r = RedisCluster(startup_nodes=[first_node, second_node])
>>> p = r.pubsub()
>>> p.ssubscribe('foo')
>>> # assume someone sends a message along the channel via a publish
>>> message = p.get_sharded_message(target_node=second_node)
The client keeps shard subscriptions attached to the node that currently owns
their slot, so a slot migration or a lost node needs no handling in application
code. When a poll on one node fails, or a node replies ``MOVED`` because it no
longer owns the slot, the client re-routes the affected shard channels to their
new owner and keeps polling the remaining nodes.
**Important**: this affects what ``get_sharded_message()`` raises.
- A connection failure on a single node is no longer surfaced while other nodes
can still deliver. ``get_sharded_message()`` returns the messages those nodes
hold, and raises only when every node polled in the pass failed.
- A ``MOVED`` reply is handled internally and no longer raised. The channels the
former owner no longer serves are re-subscribed on the new owner.
- Code that caught an exception from ``get_sharded_message()`` only to tear the
subscription down and re-subscribe is no longer needed.
- ``get_sharded_message(target_node=...)`` is unchanged: a caller that names a
single node still gets its connection errors raised.
Monitor
~~~~~~~
redis-py includes a Monitor object that streams every command processed
by the Redis server. Use listen() on the Monitor object to block until a
command is received.
.. code:: python
>>> r = redis.Redis(...)
>>> with r.monitor() as m:
>>> for command in m.listen():
>>> print(command)
---
### Backoff (`docs/backoff.rst`)
.. _backoff-label:
Backoff
#############
.. automodule:: redis.backoff
:members:
---
### Clustering (`docs/clustering.rst`)
Clustering
==========
redis-py now supports cluster mode and provides a client for `Redis
Cluster <https://redis.io/topics/cluster-tutorial>`__.
The cluster client is based on Grokzenβs
`redis-py-cluster <https://github.com/Grokzen/redis-py-cluster>`__, has
added bug fixes, and now supersedes that library. Support for these
changes is thanks to his contributions.
To learn more about Redis Cluster, see `Redis Cluster
specifications <https://redis.io/topics/cluster-spec>`__.
`Creating clusters <#creating-clusters>`__ \| `Specifying Target
Nodes <#specifying-target-nodes>`__ \| `Multi-key
Commands <#multi-key-commands>`__ \| `Known PubSub
Limitations <#known-pubsub-limitations>`__
Connecting to cluster
---------------------
Connecting redis-py to a Redis Cluster instance(s) requires at a minimum
a single node for cluster discovery. There are multiple ways in which a
cluster instance can be created:
- Using βhostβ and βportβ arguments:
.. code:: python
>>> from redis.cluster import RedisCluster as Redis
>>> rc = Redis(host='localhost', port=6379)
>>> print(rc.get_nodes())
[[host=127.0.0.1,port=6379,name=127.0.0.1:6379,server_type=primary,redis_connection=Redis<ConnectionPool<Connection<host=127.0.0.1,port=6379,db=0>>>], [host=127.0.0.1,port=6378,name=127.0.0.1:6378,server_type=primary,redis_connection=Redis<ConnectionPool<Connection<host=127.0.0.1,port=6378,db=0>>>], [host=127.0.0.1,port=6377,name=127.0.0.1:6377,server_type=replica,redis_connection=Redis<ConnectionPool<Connection<host=127.0.0.1,port=6377,db=0>>>]]
- Using the Redis URL specification:
.. code:: python
>>> from redis.cluster import RedisCluster as Redis
>>> rc = Redis.from_url("redis://localhost:6379/0")
- Directly, via the ClusterNode class:
.. code:: python
>>> from redis.cluster import RedisCluster as Redis
>>> from redis.cluster import ClusterNode
>>> nodes = [ClusterNode('localhost', 6379), ClusterNode('localhost', 6378)]
>>> rc = Redis(startup_nodes=nodes)
When a RedisCluster instance is being created it first attempts to
establish a connection to one of the provided startup nodes. If none of
the startup nodes are reachable, a βRedisClusterExceptionβ will be
thrown. After a connection to the one of the clusterβs nodes is
established, the RedisCluster instance will be initialized with 3
caches: a slots cache which maps each of the 16384 slots to the node/s
handling them, a nodes cache that contains ClusterNode objects (name,
host, port, redis connection) for all of the clusterβs nodes, and a
commands cache contains all the server supported commands that were
retrieved using the Redis βCOMMANDβ output. See *RedisCluster specific
options* below for more.
RedisCluster instance can be directly used to execute Redis commands.
When a command is being executed through the cluster instance, the
target node(s) will be internally determined. When using a key-based
command, the target node will be the node that holds the keyβs slot.
Cluster management commands and other commands that are not key-based
have a parameter called βtarget_nodesβ where you can specify which nodes
to execute the command on. In the absence of target_nodes, the command
will be executed on the default cluster node. As part of cluster
instance initialization, the clusterβs default node is randomly selected
from the clusterβs primaries, and will be updated upon reinitialization.
Using r.get_default_node(), you can get the clusterβs default node, or
you can change it using the βset_default_nodeβ method.
The βtarget_nodesβ parameter is explained in the following section,
βSpecifying Target Nodesβ.
.. code:: python
>>> # target-nodes: the node that holds 'foo1's key slot
>>> rc.set('foo1', 'bar1')
>>> # target-nodes: the node that holds 'foo2's key slot
>>> rc.set('foo2', 'bar2')
>>> # target-nodes: the node that holds 'foo1's key slot
>>> print(rc.get('foo1'))
b'bar'
>>> # target-node: default-node
>>> print(rc.keys())
[b'foo1']
>>> # target-node: default-node
>>> rc.ping()
Specifying Target Nodes
-----------------------
As mentioned above, all non key-based RedisCluster commands accept the
kwarg parameter βtarget_nodesβ that specifies the node/nodes that the
command should be executed on. The best practice is to specify target
nodes using RedisCluster classβs node flags: PRIMARIES, REPLICAS,
ALL_NODES, RANDOM. When a nodes flag is passed along with a command, it
will be internally resolved to the relevant node/s. If the nodes
topology of the cluster changes during the execution of a command, the
client will be able to resolve the nodes flag again with the new
topology and attempt to retry executing the command.
.. code:: python
>>> from redis.cluster import RedisCluster as Redis
>>> # run cluster-meet command on all of the cluster's nodes
>>> rc.cluster_meet('127.0.0.1', 6379, target_nodes=Redis.ALL_NODES)
>>> # ping all replicas
>>> rc.ping(target_nodes=Redis.REPLICAS)
>>> # ping a random node
>>> rc.ping(target_nodes=Redis.RANDOM)
>>> # get the keys from all cluster nodes
>>> rc.keys(target_nodes=Redis.ALL_NODES)
[b'foo1', b'foo2']
>>> # execute bgsave in all primaries
>>> rc.bgsave(Redis.PRIMARIES)
You could also pass ClusterNodes directly if you want to execute a
command on a specific node / node group that isnβt addressed by the
nodes flag. However, if the command execution fails due to cluster
topology changes, a retry attempt will not be made, since the passed
target node/s may no longer be valid, and the relevant cluster or
connection error will be returned.
.. code:: python
>>> node = rc.get_node('localhost', 6379)
>>> # Get the keys only for that specific node
>>> rc.keys(target_nodes=node)
>>> # get Redis info from a subset of primaries
>>> subset_primaries = [node for node in rc.get_primaries() if node.port > 6378]
>>> rc.info(target_nodes=subset_primaries)
In addition, the RedisCluster instance can query the Redis instance of a
specific node and execute commands on that node directly. The Redis
client, however, does not handle cluster failures and retries.
.. code:: python
>>> cluster_node = rc.get_node(host='localhost', port=6379)
>>> print(cluster_node)
[host=127.0.0.1,port=6379,name=127.0.0.1:6379,server_type=primary,redis_connection=Redis<ConnectionPool<Connection<host=127.0.0.1,port=6379,db=0>>>]
>>> r = cluster_node.redis_connection
>>> r.client_list()
[{'id': '276', 'addr': '127.0.0.1:64108', 'fd': '16', 'name': '', 'age': '0', 'idle': '0', 'flags': 'N', 'db': '0', 'sub': '0', 'psub': '0', 'multi': '-1', 'qbuf': '26', 'qbuf-free': '32742', 'argv-mem': '10', 'obl': '0', 'oll': '0', 'omem': '0', 'tot-mem': '54298', 'events': 'r', 'cmd': 'client', 'user': 'default'}]
>>> # Get the keys only for that specific node
>>> r.keys()
[b'foo1']
Multi-key Commands
------------------
Redis supports multi-key commands in Cluster Mode, such as Set type
unions or intersections, mset and mget, as long as the keys all hash to
the same slot. By using RedisCluster client, you can use the known
functions (e.g.Β mget, mset) to perform an atomic multi-key operation.
However, you must ensure all keys are mapped to the same slot, otherwise
a RedisClusterException will be thrown. Redis Cluster implements a
concept called hash tags that can be used in order to force certain keys
to be stored in the same hash slot, see `Keys hash
tag <https://redis.io/topics/cluster-spec#keys-hash-tags>`__. You can
also use nonatomic for some of the multikey operations, and pass keys
that arenβt mapped to the same slot. The client will then map the keys
to the relevant slots, sending the commands to the slotsβ node owners.
Non-atomic operations batch the keys according to their hash value, and
then each batch is sent separately to the slotβs owner.
.. code:: python
# Atomic operations can be used when all keys are mapped to the same slot
>>> rc.mset({'{foo}1': 'bar1', '{foo}2': 'bar2'})
>>> rc.mget('{foo}1', '{foo}2')
[b'bar1', b'bar2']
# Non-atomic multi-key operations splits the keys into different slots
>>> rc.mset_nonatomic({'foo': 'value1', 'bar': 'value2', 'zzz': 'value3')
>>> rc.mget_nonatomic('foo', 'bar', 'zzz')
[b'value1', b'value2', b'value3']
**Cluster PubSub:**
When a ClusterPubSub instance is created without specifying a node, a
single node will be transparently chosen for the pubsub connection on
the first command execution. The node will be determined by: 1. Hashing
the channel name in the request to find its keyslot 2. Selecting a node
that handles the keyslot: If read_from_replicas is set to true or
load_balancing_strategy is provided, a replica can be selected.
For sharded pubsub, the client instead keeps each shard channel on the node
that currently owns its slot and re-routes it when the topology changes; see
the sharded pubsub section of :doc:`advanced_features` for that behavior and
what it means for the errors ``get_sharded_message()`` raises.
Known PubSub Limitations
------------------------
Pattern subscribe and publish do not currently work properly due to key
slots. If we hash a pattern like fo\* we will receive a keyslot for that
string but there are endless possibilities for channel names based on
this pattern - unknowable in advance. This feature is not disabled but
the commands are not currently recommended for use. See
`redis-py-cluster
documentation <https://redis-py-cluster.readthedocs.io/en/stable/pubsub.html>`__
for more.
.. code:: python
>>> p1 = rc.pubsub()
# p1 connection will be set to the node that holds 'foo' keyslot
>>> p1.subscribe('foo')
# p2 connection will be set to node 'localhost:6379'
>>> p2 = rc.pubsub(rc.get_node('localhost', 6379))
**Read Only Mode**
By default, Redis Cluster always returns MOVE redirection response on
accessing a replica node. You can overcome this limitation and scale
read commands by triggering READONLY mode.
To enable READONLY mode pass read_from_replicas=True or define
a load_balancing_strategy to RedisCluster constructor.
When read_from_replicas is set to true read commands will be assigned between
the primary and its replications in a Round-Robin manner.
With load_balancing_strategy you can define a custom strategy for
assigning read commands to the replicas and primary nodes.
READONLY mode can be set at runtime by calling the readonly() method
with target_nodes=βreplicasβ, and read-write access can be restored by
calling the readwrite() method.
.. code:: python
>>> from cluster import RedisCluster as Redis
# Use 'debug' log level to print the node that the command is executed on
>>> rc_readonly = Redis(startup_nodes=startup_nodes,
... read_from_replicas=True)
>>> rc_readonly.set('{foo}1', 'bar1')
>>> for i in range(0, 4):
... # Assigns read command to the slot's hosts in a Round-Robin manner
... rc_readonly.get('{foo}1')
# set command would be directed only to the slot's primary node
>>> rc_readonly.set('{foo}2', 'bar2')
# reset READONLY flag
>>> rc_readonly.readwrite(target_nodes='replicas')
# now the get command would be directed only to the slot's primary node
>>> rc_readonly.get('{foo}1')
---
### Commands (`docs/commands.rst`)
Redis Commands
##############
Core Commands
*************
The following functions can be used to replicate their equivalent `Redis command <https://redis.io/commands>`_. Generally they can be used as functions on your redis connection. For the simplest example, see below:
Getting and settings data in redis::
import redis
r = redis.Redis(decode_responses=True)
r.set('mykey', 'thevalueofmykey')
r.get('mykey')
.. autoclass:: redis.commands.core.CoreCommands
:inherited-members:
Sentinel Commands
*****************
.. autoclass:: redis.commands.sentinel.SentinelCommands
:inherited-members:
Redis Cluster Commands
**********************
The following `Redis commands <https://redis.io/commands>`_ are available within a `Redis Cluster <https://redis.io/topics/cluster-tutorial>`_. Generally they can be used as functions on your redis connection.
.. autoclass:: redis.commands.cluster.RedisClusterCommands
:inherited-members:
---
### Connections (`docs/connections.rst`)
Connecting to Redis
###################
Generic Client
**************
This is the client used to connect directly to a standard Redis node.
.. autoclass:: redis.Redis
:members:
Sentinel Client
***************
Redis `Sentinel <https://redis.io/topics/sentinel>`_ provides high availability for Redis. There are commands that can only be executed against a Redis node running in sentinel mode. Connecting to those nodes, and executing commands against them requires a Sentinel connection.
Connection example (assumes Redis exists on the ports listed below):
>>> from redis import Sentinel
>>> sentinel = Sentinel([('localhost', 26379)], socket_timeout=0.1)
>>> sentinel.discover_master('mymaster')
('127.0.0.1', 6379)
>>> sentinel.discover_slaves('mymaster')
[('127.0.0.1', 6380)]
Sentinel
========
.. autoclass:: redis.sentinel.Sentinel
:members:
SentinelConnectionPool
======================
.. autoclass:: redis.sentinel.SentinelConnectionPool
:members:
Cluster Client
**************
This client is used for connecting to a Redis Cluster.
RedisCluster
============
.. autoclass:: redis.cluster.RedisCluster
:members:
ClusterNode
===========
.. autoclass:: redis.cluster.ClusterNode
:members:
Async Client
************
See complete example: `here <examples/asyncio_examples.html>`__
This client is used for communicating with Redis, asynchronously.
.. autoclass:: redis.asyncio.client.Redis
:members:
Async Cluster Client
********************
RedisCluster (Async)
====================
.. autoclass:: redis.asyncio.cluster.RedisCluster
:members:
:member-order: bysource
ClusterNode (Async)
===================
.. autoclass:: redis.asyncio.cluster.ClusterNode
:members:
:member-order: bysource
ClusterPipeline (Async)
=======================
.. autoclass:: redis.asyncio.cluster.ClusterPipeline
:members: execute_command, execute
:member-order: bysource
Connection
**********
See complete example: `here <examples/connection_examples.html>`__
Connection
==========
.. autoclass:: redis.connection.Connection
:members:
Connection (Async)
==================
.. autoclass:: redis.asyncio.connection.Connection
:members:
Connection Pools
****************
See complete example: `here <examples/connection_examples.html>`__
ConnectionPool
==============
.. autoclass:: redis.connection.ConnectionPool
:members:
ConnectionPool (Async)
======================
.. autoclass:: redis.asyncio.connection.ConnectionPool
:members:
---
### Examples (`docs/examples.rst`)
Examples
########
.. toctree::
:maxdepth: 3
:glob:
examples/connection_examples
examples/ssl_connection_examples
examples/asyncio_examples
examples/search_json_examples
examples/set_and_get_examples
examples/search_vector_similarity_examples
examples/pipeline_examples
examples/timeseries_examples
examples/redis-stream-example
examples/opentelemetry_api_examples
examples/hash_examples
examples/pubsub_examples
---
### Exceptions (`docs/exceptions.rst`)
.. _exceptions-label:
Exceptions
##########
.. automodule:: redis.exceptions
:members:
---
### Geographic Failover (`docs/geographic_failover.rst`)
Client-side geographic failover (Active-Active)
=====================================
The multi-database client allows your application to connect to multiple Redis databases, which are typically replicas of each other.
It is designed to work with Redis Software and Redis Cloud Active-Active setups.
The client continuously monitors database health, detects failures, and automatically fails over to the next healthy database using a configurable strategy.
When the original database becomes healthy again, the client can automatically switch back to it.
Key concepts
------------
- Database and weight:
Each database has a weight indicating its priority. The failover strategy chooses the highest-weight
healthy database as the active one.
- Circuit breaker:
Each database is guarded by a circuit breaker with states CLOSED (healthy), OPEN (unhealthy),
and HALF_OPEN (probing). Health checks toggle these states to avoid hammering a downed database.
- Health checks:
A set of checks determines whether a database is healthy in proactive manner.
By default, an "PING" check runs against the database (all cluster nodes must
pass for a cluster). You can provide your own set of health checks or add an
additional health check on top of the default one. A Redis Enterprise specific
"lag-aware" health check is also available.
- Failure detector:
A detector observes command failures over a moving window (reactive monitoring).
You can specify an exact number of failures and failures rate to have more
fine-grain tuned configuration of triggering fail over based on organic traffic.
You can provide your own set of custom failure detectors or add an additional
detector on top of the default one.
- Failover strategy:
The default strategy is based on statically configured weights. It prefers the highest weighted healthy database.
- Command retry:
Command execution supports retry with backoff. Low-level client retries are disabled and a global retry
setting is applied at the multi-database layer.
- Auto fallback:
If configured with a positive interval, the client periodically attempts to fall back to a higher-weighted
healthy database.
- Events:
The client emits events like "active database changed" and "commands failed". In addition it resubscribes to Pub/Sub channels automatically.
Synchronous usage
-----------------
Minimal example
^^^^^^^^^^^^^^^
.. code-block:: python
from redis.multidb.client import MultiDBClient
from redis.multidb.config import MultiDbConfig, DatabaseConfig
# Two databases. The first has higher weight -> preferred when healthy.
cfg = MultiDbConfig(
databases_config=[
DatabaseConfig(from_url="redis://db-primary:6379/0", weight=1.0),
DatabaseConfig(from_url="redis://db-secondary:6379/0", weight=0.5),
]
)
client = MultiDBClient(cfg)
# First call triggers initialization and health checks.
client.set("key", "value")
print(client.get("key"))
# Pipeline
with client.pipeline() as pipe:
pipe.set("a", 1)
pipe.incrby("a", 2)
values = pipe.execute()
print(values)
# Transaction
def txn(pipe):
current = pipe.get("balance")
current = int(current or 0)
pipe.multi() # mark transaction
pipe.set("balance", current + 100)
client.transaction(txn)
# Pub/Sub usage - will automatically re-subscribe on database switch
pubsub = client.pubsub()
pubsub.subscribe("events")
# In your loop:
message = pubsub.get_message(timeout=1.0)
if message:
print(message)
Asyncio usage
-------------
The asyncio API mirrors the synchronous one and provides async/await semantics.
.. code-block:: python
import asyncio
from redis.asyncio.multidb.client import MultiDBClient
from redis.asyncio.multidb.config import MultiDbConfig, DatabaseConfig
async def main():
cfg = MultiDbConfig(
databases_config=[
DatabaseConfig(from_url="redis://db-primary:6379/0", weight=1.0),
DatabaseConfig(from_url="redis://db-secondary:6379/0", weight=0.5),
]
)
# Context-manager approach for graceful client termination when exits.
# client = MultiDBClient(cfg) could be used instead
async with MultiDBClient(cfg) as client:
await client.set("key", "value")
print(await client.get("key"))
# Pipeline
async with client.pipeline() as pipe:
pipe.set("a", 1)
pipe.incrby("a", 2)
values = await pipe.execute()
print(values)
# Transaction
async def txn(pipe):
current = await pipe.get("balance")
current = int(current or 0)
await pipe.multi()
await pipe.set("balance", current + 100)
await client.transaction(txn)
# Pub/Sub
pubsub = client.pubsub()
await pubsub.subscribe("events")
message = await pubsub.get_message(timeout=1.0)
if message:
print(message)
asyncio.run(main())
MultiDBClient
^^^^^^^^^^^^^
The client exposes the same API as the `Redis` or `RedisCluster` client, making it fully interchangeable and ensuring a simple migration of your application code. Additionally, it supports runtime reconfiguration, allowing you to add features such as health checks, failure detectors, or even new databases without restarting.
Configuration
-------------
MultiDbConfig
^^^^^^^^^^^^^
.. code-block:: python
from redis.multidb.config import (
MultiDbConfig, DatabaseConfig,
DEFAULT_HEALTH_CHECK_INTERVAL, DEFAULT_GRACE_PERIOD
)
from redis.retry import Retry
from redis.backoff import ExponentialWithJitterBackoff
cfg = MultiDbConfig(
databases_config=[
# Construct via URL
DatabaseConfig(
from_url="redis://db-a:6379/0",
weight=1.0,
# Optional: use a custom circuit breaker grace period
grace_period=DEFAULT_GRACE_PERIOD,
# Optional: Redis Enterprise cluster FQDN for REST health checks
health_check_url="https://cluster.example.com",
# Optional: Underlying Redis client related configuration
client_kwargs={"socket_timeout": 5}
),
# Or construct via ConnectionPool
# DatabaseConfig(from_pool=my_pool, weight=1.0),
],
# Global command retry policy (applied at multi-db layer)
command_retry=Retry(
retries=10,
backoff=ExponentialWithJitterBackoff(base=0.01, cap=1),
),
# Health checks
health_check_interval: float = DEFAULT_HEALTH_CHECK_INTERVAL # seconds
health_check_probes: int = DEFAULT_HEALTH_CHECK_PROBES
health_check_delay: float = DEFAULT_HEALTH_CHECK_DELAY # seconds
health_check_timeout: float = DEFAULT_HEALTH_CHECK_TIMEOUT # seconds
health_check_policy: HealthCheckPolicies = DEFAULT_HEALTH_CHECK_POLICY,
# Failure detector
min_num_failures: int = DEFAULT_MIN_NUM_FAILURES
failure_rate_threshold: float = DEFAULT_FAILURE_RATE_THRESHOLD
failures_detection_window: float = DEFAULT_FAILURES_DETECTION_WINDOW # seconds
# Failover behavior
failover_attempts: int = DEFAULT_FAILOVER_ATTEMPTS
failover_delay: float = DEFAULT_FAILOVER_DELAY # seconds
)
Notes:
- Low-level client retries are disabled automatically per database. The multi-database layer handles retries.
- For clusters, health checks validate all nodes.
DatabaseConfig
^^^^^^^^^^^^^^
Each database needs a `DatabaseConfig` that specifies how to connect.
Method 1: Using client_kwargs (most flexible)There's an underlying instance of Redis or RedisCluster client for each database,
so you can pass all the arguments related to them via client_kwargs argument:
.. code:: python
database_config = DatabaseConfig(
weight=1.0,
client_kwargs={
'host': 'localhost',
'port': 6379,
'username': "username",
'password': "password",
}
)
Method 2: Using Redis URL
.. code:: python
database_config1 = DatabaseConfig(
weight=1.0,
from_url="redis://host1:port1",
client_kwargs={
'username': "username",
'password': "password",
}
)
Method 3: Using Custom Connection Pool.. code:: python
database_config2 = DatabaseConfig(
weight=0.9,
from_pool=connection_pool,
)
Important: Don't pass Retry objects in client_kwargs. MultiDBClient
handles all retries at the top level through the command_retry configuration.
Health Monitoring
The MultiDBClient uses two complementary mechanisms to ensure database availability:
- Health Checks (Proactive Monitoring)
- Failure Detection (Reactive Monitoring)
Health Checks (Proactive Monitoring)
These checks run continuously in the background at configured intervals to proactively
detect database issues. They run in the background with a given interval and
configuration defined in the `MultiDBConfig` class.
To avoid false positives, you can configure amount of health check probes and also
define one of the health check policies to evaluate probes result.
**HealthCheckPolicies.HEALTHY_ALL** - (default) All probes should be successful.
**HealthCheckPolicies.HEALTHY_MAJORITY** - Majority of probes should be successful.
**HealthCheckPolicies.HEALTHY_ANY** - Any of probes should be successful.
The whole health check - all of its probes - runs under `health_check_timeout`
(3 seconds by default), and a single check running out of it marks the database
unhealthy regardless of the failure detector's threshold. See
`Planned maintenance (maintenance notifications)`_ for how this budget behaves
while a database undergoes a planned maintenance.
PingHealthCheck (default)
^^^^^^^^^^^^^^^^^^^^^^^^^
The default health check sends the [PING](https://redis.io/docs/latest/commands/ping/) command
to the database (and to all nodes for clusters).
Lag-Aware Healthcheck (Redis Enterprise Only)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This is a special type of healthcheck available for Redis (Enterprise) Software
that utilizes a REST API endpoint to obtain information about the synchronization
lag between a given database and all other databases in an Active-Active setup.
To use this healthcheck, first you need to adjust your `DatabaseConfig`
to expose `health_check_url` used by your deployment. By default, your
Cluster FQDN should be used as URL, unless you have some kind of
reverse proxy behind an actual REST API endpoint.
.. code-block:: python
from redis.multidb.client import MultiDBClient
from redis.multidb.config import MultiDbConfig, DatabaseConfig
from redis.multidb.healthcheck import PingHealthCheck, LagAwareHealthCheck
from redis.retry import Retry
from redis.backoff import ExponentialWithJitterBackoff
cfg = MultiDbConfig(
databases_config=[
DatabaseConfig(
from_url="redis://db-primary:6379/0",
weight=1.0,
health_check_url="https://cluster.example.com", # optional for LagAware
),
DatabaseConfig(
from_url="redis://db-secondary:6379/0",
weight=0.5,
health_check_url="https://cluster.example.com",
),
],
# Add custom health check to replace the default
health_checks=[
# Redis Enterprise REST-based lag-aware check
LagAwareHealthCheck(
# Customize REST port, lag tolerance, TLS, etc.
rest_api_port=9443,
lag_aware_tolerance=100, # ms
verify_tls=True,
# auth_basic=("user", "pass"),
# ca_file="/path/ca.pem",
# client_cert_file="/path/cert.pem",
# client_key_file="/path/key.pem",
),
],
)
client = MultiDBClient(cfg)
**Custom Health Checks**
~~~~~~~~~~~~~~~~~~~~~
You can add custom health checks for specific requirements. Please notice that all health checks are executed within
asyncio event loop, so please ensure that `check_health` method is async:
.. code-block:: python
from redis.asyncio.multidb.healthcheck import AbstractHealthCheck, AsyncRedisClientT
class EchoHealthCheck(AbstractHealthCheck):
"""
Health check based on ECHO command.
"""
async def check_health(self, database, hc_client: AsyncRedisClientT) -> bool:
await connection.send_command("ECHO", "healthcheck")
response = await connection.read_response()
return response in (b"healthcheck", "healthcheck")
Failure Detection (Reactive Monitoring)
-----------------
The failure detector monitors command failures and marks a database as unhealthy when its failure rate exceeds a defined threshold within a sliding time window.
Under real traffic conditions, this reactive detection mechanism likely triggers earlier than proactive health checks.
You can extend the set of failure detectors by implementing your own and configuring it through the `MultiDBConfig` class.
By default the failure detector is configured for 1000 failures and a 10% failure rate
threshold within a 2 seconds sliding window. This could be adjusted regarding
your application specifics and traffic pattern.
.. code-block:: python
from redis.multidb.config import MultiDbConfig, DatabaseConfig
from redis.multidb.client import MultiDBClient
cfg = MultiDbConfig(
databases_config=[
DatabaseConfig(from_url="redis://db-a:6379/0", weight=1.0),
DatabaseConfig(from_url="redis://db-b:6379/0", weight=0.5),
],
# Default detector also created from config values
)
client = MultiDBClient(cfg)
# Add an additional detector, optionally limited to specific exception types:
client.add_failure_detector(
CustomFailureDetector()
)
Failover and automatic fallback
--------------------------
Weight-based failover chooses the highest-weighted database with a CLOSED circuit. If no database is
healthy it returns `TemporaryUnavailableException`. This exception indicates that the application should
retry sending requests for a configurable period of time (the configuration (`failover_attempts` * `failover_delay`)
defaults to 120 seconds). If none of the databases became available, then a `NoValidDatabaseException` is thrown.
To enable periodic fallback to a higher-priority healthy database, set `auto_fallback_interval` (seconds):
.. code-block:: python
from redis.multidb.config import MultiDbConfig, DatabaseConfig
cfg = MultiDbConfig(
databases_config=[
DatabaseConfig(from_url="redis://db-primary:6379/0", weight=1.0),
DatabaseConfig(from_url="redis://db-secondary:6379/0", weight=0.5),
],
# Try to fallback to higher-weight healthy database every 30 seconds
auto_fallback_interval=30.0,
)
client = MultiDBClient(cfg)
Planned maintenance (maintenance notifications)
-----------------------------------------------
A planned maintenance on one of the databases - a shard migration, a shard failover or
an endpoint rebind during an upgrade or a rebalancing - is not an outage, and should not
turn into a geo failover: the database stays available, and the operation is resolved
inside its own cluster within seconds. Redis Enterprise announces such operations to the
connected clients through push notifications (`MIGRATING` / `MIGRATED`,
`FAILING_OVER` / `FAILED_OVER`, `MOVING`), and redis-py reacts to them on the
connections of the cluster they concern. What follows describes what the
`MultiDBClient` does with them, how to enable them, and what to expect without them.
Enabling the notifications
^^^^^^^^^^^^^^^^^^^^^^^^^^
`MultiDbConfig` disables maintenance notifications in the underlying clients unless a
`MaintNotificationsConfig` is passed through `client_kwargs`; a database configured
with `from_pool` keeps whatever its pool was built with. The notifications require
RESP3, and the relaxed timeout they apply only means something when the connections
have a finite socket timeout to relax:
.. code-block:: python
from redis.maint_notifications import MaintNotificationsConfig
from redis.multidb.config import MultiDbConfig, DatabaseConfig
maint_config = MaintNotificationsConfig(
enabled=True,
relaxed_timeout=30, # seconds; the socket timeout during a maintenance
)
cfg = MultiDbConfig(
databases_config=[
DatabaseConfig(
from_url="redis://db-primary:6379/0",
weight=1.0,
client_kwargs={
"protocol": 3,
"socket_timeout": 5,
"maint_notifications_config": maint_config,
},
),
DatabaseConfig(
from_url="redis://db-secondary:6379/0",
weight=0.5,
client_kwargs={
"protocol": 3,
"socket_timeout": 5,
"maint_notifications_config": maint_config,
},
),
],
)
`enabled=True` makes the client require the server-side support (the cluster setting
`client_maint_notifications`) and fail the connection otherwise; `enabled="auto"`
attempts to enable the notifications and continues without them if the server rejects
the command.
What the client does during a maintenance
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
With the notifications enabled, a planned maintenance is handled by the underlying
client of the affected database, and the `MultiDBClient` keeps that database active:
- **Connections.** A `MIGRATING` or `FAILING_OVER` notification relaxes the socket
timeout of the connection it arrives on to `relaxed_timeout`, so a command that the
shard switch stalls waits instead of timing out; the matching completion
notification restores the timeout. A `MOVING` notification announces the endpoint
moving to another node: the pool relaxes all of its connections and reconnects them
to the new node, transparently to the commands in flight.
- **Health checks.** The health check client of each database is built from that
database's `client_kwargs`, so it receives the same notifications, and the health
check budget follows the connection timeout they relax: while the database is under
a maintenance, `health_check_timeout` becomes
`health_check_probes * (relaxed_timeout + health_check_delay)` - the time every probe
needs when each may take as long as the relaxed socket timeout - and it returns to
its configured value once the maintenance completes. A stall with no notification
behind it, an unplanned failure, keeps the configured budget and fails the database
over as before. `relaxed_timeout=-1` disables the relaxation of both the connections
and the budget; `relaxed_timeout=None` (blocking sockets) relaxes the budget up to
the notification's time-to-live. The notifications reach the health checks through
the probe client's Redis connections, which only a check that talks to the
database keeps reading - `PingHealthCheck` does, `LagAwareHealthCheck` uses the
REST API alone. Combined with a `PingHealthCheck` (the default), the budget of a
`LagAwareHealthCheck` is relaxed the same way, with each REST request still bounded
by its own `http_timeout`; on its own it keeps the configured budget.
- **Idle connections.** A connection that is idle through the maintenance has the
notifications, and possibly the server's close after an endpoint rebind, waiting in
its buffer. When the pool hands it out again it reads them first. If the connection
is still alive the maintenance is still in progress and the notifications are
applied as usual. If the server has closed it, the maintenance is over: the stale
notifications are discarded and the connection is reconnected, through the
configured address, before the command is sent - so resuming traffic after an idle
period does not count as a failure towards a failover.
The failure detector and the circuit breakers are not involved in any of this: a
maintenance that the notifications cover produces neither the command failures nor
the failed health checks that would open a circuit.
Without the notifications
^^^^^^^^^^^^^^^^^^^^^^^^^
Without the notifications the client learns of a maintenance only through its effects:
connections closed by an endpoint rebind, and commands or health check probes stalled
by a shard switch for longer than their timeouts. Whether that turns into a geo
failover depends on the operation and on the configuration - the size of the shard,
`socket_timeout`, `health_check_timeout`, `min_num_failures` and the failure rate
threshold. A failover to a healthy database is the intended fallback when it does, and
the client falls back to the higher-weighted database once it is healthy again (see
`Failover and automatic fallback`_), but it costs a reconnect of every command in
flight and a switch of the database the application talks to, for an operation the
cluster could have resolved on its own. Enabling the notifications is the way to avoid
that.
Observing a maintenance
^^^^^^^^^^^^^^^^^^^^^^^
The phases the underlying clients go through are observable: the handlers dispatch a
`MaintenanceStartedEvent` when a notification starts relaxing timeouts and a
`MaintenanceCompletedEvent` when the relaxation is reverted, to the `event_dispatcher`
of the `MaintNotificationsConfig` passed in `client_kwargs`. Both events carry the
`state` (`MaintenanceState.MOVING` for a pool-level handoff,
`MaintenanceState.MAINTENANCE` for a connection-level migration or failover), the
affected `connection_pool` and `connection`, the `notification` and the `config` acted
with; `notification` is `None` when the relaxation ended because a connection still
under maintenance was closed. The events are dispatched synchronously from the
connection's read path, so listeners must not block.
.. code-block:: python
from redis.event import (
EventDispatcher,
EventListenerInterface,
MaintenanceCompletedEvent,
MaintenanceStartedEvent,
)
from redis.maint_notifications import MaintNotificationsConfig
class LogMaintenanceListener(EventListenerInterface):
def listen(self, event):
print(f"{type(event).__name__}: {event.state} {event.notification}")
dispatcher = EventDispatcher()
listener = LogMaintenanceListener()
dispatcher.register_listeners(
{MaintenanceStartedEvent: [listener], MaintenanceCompletedEvent: [listener]}
)
maint_config = MaintNotificationsConfig(enabled=True, event_dispatcher=dispatcher)
A geo failover, should one happen anyway, is reported through the
`ActiveDatabaseChanged` event described in the next section.
Custom failover callbacks
-------------------------
You may want to activate custom actions when failover happens. For example, you may want to collect some metrics,
logs or externally persist a connection state.
You can register your own event listener for the `ActiveDatabaseChanged` event (which is emitted when a failover happens) using
the `EventDispatcher`.
.. code-block:: python
class LogFailoverEventListener(EventListenerInterface):
def __init__(self, logger: Logger):
self.logger = logger
def listen(self, event: ActiveDatabaseChanged):
self.logger.warning(
f"Failover happened. Active database switched from {event.old_database} to {event.new_database}"
)
event_dispatcher = EventDispatcher()
listener = LogFailoverEventListener(logging.getLogger(__name__))
# Register custom listener
event_dispatcher.register_listeners(
{
ActiveDatabaseChanged: [listener],
}
)
config = MultiDbConfig(
client_class=client_class,
databases_config=db_configs,
command_retry=command_retry,
min_num_failures=min_num_failures,
health_check_probes=3,
health_check_interval=health_check_interval,
event_dispatcher=event_dispatcher,
health_check_probes_delay=health_check_delay,
)
client = MultiDBClient(config)
Managing databases at runtime
-----------------------------
You can manually add/remove databases, update weights, and promote a database if itβs healthy.
.. code-block:: python
from redis.multidb.client import MultiDBClient
from redis.multidb.config import MultiDbConfig, DatabaseConfig
from redis.multidb.database import Database
from redis.multidb.circuit import PBCircuitBreakerAdapter
import pybreaker
from redis import Redis
cfg = MultiDbConfig(
databases_config=[DatabaseConfig(from_url="redis://db-a:6379/0", weight=1.0)]
)
client = MultiDBClient(cfg)
# Add a database programmatically
other = Database(
client=Redis.from_url("redis://db-b:6379/0"),
circuit=PBCircuitBreakerAdapter(pybreaker.CircuitBreaker(reset_timeout=5.0)),
weight=0.5,
health_check_url=None,
)
client.add_database(other)
# Update weight; if it becomes the highest and healthy, it may become active
client.update_database_weight(other, 0.9)
# Promote a specific healthy database to active
client.set_active_database(other)
# Remove a database
client.remove_database(other)
Pub/Sub and re-subscription
--------------------------
The MultiDBClient offers Pub/Sub functionality with automatic re-subscription
to channels during failover events. For optimal failover handling,
both publishers and subscribers should use MultiDBClient instances.
1. **Subscriber failover**: Automatically reconnects to an alternative database
and re-subscribes to the same channels
2. **Publisher failover**: Seamlessly switches to an alternative database and
continues publishing to the same channels
**Note**: Message loss may occur if failover events happen in reverse order
(publisher fails before subscriber).
.. code-block:: python
pubsub = client.pubsub()
pubsub.subscribe("news", "alerts")
# If failover happens here, subscriptions are re-established on the new active DB.
msg = pubsub.get_message(timeout=1.0)
if msg:
print(msg)
Pipelines and transactions
--------------------------
Pipelines and transactions are executed against the active database at execution time. The client ensures
the active database is healthy and up-to-date before running the stack.
.. code-block:: python
with client.pipeline() as pipe:
pipe.set("x", 1)
pipe.incr("x")
results = pipe.execute()
def txn(pipe):
pipe.multi()
pipe.set("y", "42")
client.transaction(txn)
Best practices
--------------
- Assign the highest weight to your primary database and lower weights to replicas or disaster recovery sites.
- Keep `health_check_interval` short enough to promptly detect failures but avoid excessive load.
- Tune `command_retry` and failover attempts to your SLA and workload profile.
- Use `auto_fallback_interval` if you want the client to fail over back to your primary automatically.
- Handle `TemporaryUnavailableException` to be able to recover before giving up. In the meantime, you
can switch the data source (e.g. cache). `NoValidDatabaseException` indicates that there are no healthy
databases to operate.
Troubleshooting
---------------
- NoValidDatabaseException:
Indicates no healthy database is available. Check circuit breaker states and health checks.
- TemporaryUnavailableException
Indicates that currently there are no healthy databases, but you can still send requests until
`NoValidDatabaseException` is thrown. Probe interval is configured with `failure_attemtps`
- Health checks always failing:
Verify connectivity and, for clusters, that all nodes are reachable. For `LagAwareHealthCheck`,
ensure `health_check_url` points to your Redis Enterprise endpoint and authentication/TLS options
are configured properly.
- Pub/Sub not receiving messages after failover:
Ensure you are using the clientβs Pub/Sub helper. The client re-subscribes automatically on switch.
---
### Index (`docs/index.rst`)
.. redis-py documentation master file, created by
sphinx-quickstart on Thu Jul 28 13:55:57 2011.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
redis-py - Python Client for Redis
====================================
Getting Started
****************
`redis-py <https://pypi.org/project/redis>`_ requires a running Redis server, and Python 3.7+. See the `Redis
quickstart <https://redis.io/topics/quickstart>`_ for Redis installation instructions.
redis-py can be installed using pip via ``pip install redis``.
Quickly connecting to redis
***************************
There are two quick ways to connect to Redis.
**Assuming you run Redis on localhost:6379 (the default)**
.. code-block:: python
import redis
r = redis.Redis()
r.ping()
**Running redis on foo.bar.com, port 12345**
.. code-block:: python
import redis
r = redis.Redis(host='foo.bar.com', port=12345)
r.ping()
**Another example with foo.bar.com, port 12345**
.. code-block:: python
import redis
r = redis.from_url('redis://foo.bar.com:12345')
r.ping()
After that, you probably want to `run redis commands <commands.html>`_.
.. toctree::
:hidden:
genindex
Redis Command Functions
***********************
.. toctree::
:maxdepth: 2
commands
redismodules
Module Documentation
********************
.. toctree::
:maxdepth: 1
connections
clustering
geographic_failover
exceptions
backoff
lock
retry
lua_scripting
opentelemetry
resp3_features
unified_responses
advanced_features
examples
Contributing
*************
- `How to contribute <https://github.com/redis/redis-py/blob/master/CONTRIBUTING.md>`_
- `Issue Tracker <https://github.com/redis/redis-py/issues>`_
- `Source Code <https://github.com/redis/redis-py/>`_
- `Release History <https://github.com/redis/redis-py/releases/>`_
License
*******
This project is licensed under the `MIT license <https://github.com/redis/redis-py/blob/master/LICENSE>`_.
---
### Lock (`docs/lock.rst`)
Lock
#########
.. automodule:: redis.lock
:members:
---
### Lua Scripting (`docs/lua_scripting.rst`)
Lua Scripting
===
`Lua Scripting <#lua-scripting-in-default-connections>`__ \|
`Pipelines <#pipelines>`__ \| `Cluster mode <#cluster-mode>`__
--------------
Lua Scripting in default connections
------------------------------------
redis-py supports the EVAL, EVALSHA, and SCRIPT commands. However, there
are a number of edge cases that make these commands tedious to use in
real world scenarios. Therefore, redis-py exposes a Script object that
makes scripting much easier to use. (RedisClusters have limited support
for scripting.)
To create a Script instance, use the register_script function on a
client instance passing the Lua code as the first argument.
register_script returns a Script instance that you can use throughout
your code.
The following trivial Lua script accepts two parameters: the name of a
key and a multiplier value. The script fetches the value stored in the
key, multiplies it with the multiplier value and returns the result.
.. code:: python
>>> r = redis.Redis()
>>> lua = """
... local value = redis.call('GET', KEYS[1])
... value = tonumber(value)
... return value * ARGV[1]"""
>>> multiply = r.register_script(lua)
multiply is now a Script instance that is invoked by calling it like a
function. Script instances accept the following optional arguments:
- **keys**: A list of key names that the script will access. This
becomes the KEYS list in Lua.
- **args**: A list of argument values. This becomes the ARGV list in
Lua.
- **client**: A redis-py Client or Pipeline instance that will invoke
the script. If client isn't specified, the client that initially
created the Script instance (the one that register_script was invoked
from) will be used.
Continuing the example from above:
.. code:: python
>>> r.set('foo', 2)
>>> multiply(keys=['foo'], args=[5])
10
The value of key 'foo' is set to 2. When multiply is invoked, the 'foo'
key is passed to the script along with the multiplier value of 5. Lua
executes the script and returns the result, 10.
Script instances can be executed using a different client instance, even
one that points to a completely different Redis server.
.. code:: python
>>> r2 = redis.Redis('redis2.example.com')
>>> r2.set('foo', 3)
>>> multiply(keys=['foo'], args=[5], client=r2)
15
The Script object ensures that the Lua script is loaded into Redis's
script cache. In the event of a NOSCRIPT error, it will load the script
and retry executing it.
Pipelines
---------
Script objects can also be used in pipelines. The pipeline instance
should be passed as the client argument when calling the script. Care is
taken to ensure that the script is registered in Redis's script cache
just prior to pipeline execution.
.. code:: python
>>> pipe = r.pipeline()
>>> pipe.set('foo', 5)
>>> multiply(keys=['foo'], args=[5], client=pipe)
>>> pipe.execute()
[True, 25]
Cluster Mode
------------
Cluster mode has limited support for lua scripting.
The following commands are supported, with caveats:
- ``EVAL`` and ``EVALSHA``: The command is sent to the relevant node,
depending on the keys (i.e., in ``EVAL "<script>" num_keys key_1 ...
key_n ...``). The keys *must* all be on the same node. If the script
requires 0 keys, *the command is sent to a random (primary) node*.
- ``SCRIPT EXISTS``: The command is sent to all primaries. The result
is a list of booleans corresponding to the input SHA hashes. Each
boolean is an AND of βdoes the script exist on each node?β. In other
words, each boolean is True iff the script exists on all nodes.
- ``SCRIPT FLUSH``: The command is sent to all primaries. The result
is a bool AND over all nodesβ responses.
- ``SCRIPT LOAD``: The command is sent to all primaries. The result
is the SHA1 digest.
The following commands are not supported:
- ``EVAL_RO``
- ``EVALSHA_RO``
``EVALSHA`` can be used inside a ``ClusterPipeline``. Keys must map to the
same hash slot (or ``numkeys`` may be ``0``, in which case the command is
routed to a random primary, exactly as in the non-pipelined case above). In a
transactional pipeline (``pipeline(transaction=True)``), zero-key ``EVALSHA``
reuses the transaction's existing slot when one is already chosen, so multiple
zero-key scripts (or a mix with keyed commands) stay single-slot.
``load_scripts`` and ``script_load_for_pipeline`` remain **not supported**
on cluster pipelines. On the sync client, ``ClusterPipeline.eval()`` is also
blocked; the async cluster pipeline has no ``eval`` override.
Important caveats when using ``EVALSHA`` in a cluster pipeline:
- The Lua script cache in Redis is **per-node and is not replicated**.
``SCRIPT LOAD`` loads the script onto the *current* primaries only, at the
moment it is called.
- Any topology change -- a failover that promotes a replica, a rolling
upgrade that replaces nodes, resharding, or adding a new shard -- can route
an ``EVALSHA`` to a node whose cache does not contain the script, which
fails with ``NOSCRIPT`` (``redis.exceptions.NoScriptError``).
- Unlike the non-pipelined ``Script`` object, **a cluster pipeline performs no
automatic reload or retry** on ``NOSCRIPT``. Recovery is the caller's
responsibility: catch ``NoScriptError``, re-run ``SCRIPT LOAD``, then retry
only when replay is safe (for example an idempotent or single-command
pipeline). Blindly re-executing a multi-command pipeline can duplicate
side effects: Redis still runs the rest of a non-transactional batch when
one ``EVALSHA`` returns ``NOSCRIPT``, and the client raises only after
reading every response. Because zero-key ``EVALSHA`` is routed to a random
primary, the script must be present on **all** primaries.
- In a **transactional** pipeline, a ``NOSCRIPT`` from ``EVALSHA`` is raised at
``EXEC`` time and does **not** roll back the other commands (this follows
Redis ``MULTI``/``EXEC`` semantics), so partial application is possible.
.. code:: python
>>> from redis.exceptions import NoScriptError
>>> sha = rc.script_load(lua) # loads on all current primaries
>>> def run():
... with rc.pipeline() as pipe:
... pipe.evalsha(sha, 1, "{user}:1")
... return pipe.execute()
>>> try:
... result = run()
... except NoScriptError:
... # single-command pipeline: safe to reload and retry
... sha = rc.script_load(lua) # reload on current primaries
... result = run()
---
### Opentelemetry (`docs/opentelemetry.rst`)
Integrating OpenTelemetry
=========================
What is OpenTelemetry?
----------------------
`OpenTelemetry <https://opentelemetry.io>`__ is an open-source observability framework for traces, metrics, and logs. It is a merger of OpenCensus and OpenTracing projects hosted by Cloud Native Computing Foundation.
OpenTelemetry allows developers to collect and export telemetry data in a vendor agnostic way. With OpenTelemetry, you can instrument your application once and then add or change vendors without changing the instrumentation, for example, here is a list of `popular DataDog competitors <https://uptrace.dev/get/compare/datadog-competitors.html>`_ that support OpenTelemetry.
What is tracing?
----------------
`OpenTelemetry tracing <https://uptrace.dev/opentelemetry/distributed-tracing.html>`_ allows you to see how a request progresses through different services and systems, timings of each operation, any logs and errors as they occur.
In a distributed environment, tracing also helps you understand relationships and interactions between microservices. Distributed tracing gives an insight into how a particular microservice is performing and how that service affects other microservices.
.. image:: images/opentelemetry/distributed-tracing.png
:alt: Trace
Using tracing, you can break down requests into spans. **Span** is an operation (unit of work) your app performs handling a request, for example, a database query or a network call.
**Trace** is a tree of spans that shows the path that a request makes through an app. Root span is the first span in a trace.
.. image:: images/opentelemetry/tree-of-spans.png
:alt: Trace
To learn more about tracing, see `Distributed Tracing using OpenTelemetry <https://uptrace.dev/opentelemetry/distributed-tracing.html>`_.
Native OpenTelemetry Integration (Recommended)
----------------------------------------------
redis-py includes built-in support for OpenTelemetry metrics collection. This native integration is the **recommended approach** as it provides comprehensive metrics without requiring external instrumentation packages.
Installation
^^^^^^^^^^^^
To use native OpenTelemetry support, install the required dependencies:
.. code-block:: shell
pip install redis[otel]
Basic Setup
^^^^^^^^^^^
Initialize OpenTelemetry observability once at application startup. All Redis clients will automatically collect metrics:
.. code-block:: python
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
# 1. Set up OpenTelemetry MeterProvider
exporter = OTLPMetricExporter(endpoint="http://localhost:4318/v1/metrics")
reader = PeriodicExportingMetricReader(exporter=exporter, export_interval_millis=10000)
provider = MeterProvider(metric_readers=[reader])
metrics.set_meter_provider(provider)
# 2. Initialize redis-py observability
from redis.observability import get_observability_instance, OTelConfig
otel = get_observability_instance()
otel.init(OTelConfig())
# 3. Use Redis as usual - metrics are collected automatically
import redis
r = redis.Redis(host='localhost', port=6379)
r.set('key', 'value') # Metrics collected automatically
r.get('key')
# 4. Shutdown observability at application exit
otel.shutdown()
Configuration Options
^^^^^^^^^^^^^^^^^^^^^
The ``OTelConfig`` class provides fine-grained control over metrics collection:
.. code-block:: python
from redis.observability import OTelConfig, MetricGroup
config = OTelConfig(
# Metric groups to enable (default: CONNECTION_BASIC | RESILIENCY)
metric_groups=[
MetricGroup.CONNECTION_BASIC, # Connection creation time, relaxed timeout
MetricGroup.CONNECTION_ADVANCED, # Connection wait time, timeouts, closed connections
MetricGroup.COMMAND, # Command execution duration
MetricGroup.RESILIENCY, # Error counts, maintenance notifications
MetricGroup.PUBSUB, # PubSub message counts
MetricGroup.STREAMING, # Stream message lag
MetricGroup.CSC, # Client Side Caching metrics
],
# Filter which commands to track
include_commands=['GET', 'SET', 'HGET'], # Only track these commands
# OR
exclude_commands=['DEBUG', 'SLOWLOG'], # Track all except these
# Privacy controls
hide_pubsub_channel_names=True, # Hide channel names in PubSub metrics
hide_stream_names=True, # Hide stream names in streaming metrics
)
otel = get_observability_instance()
otel.init(config)
Available Metric Groups
^^^^^^^^^^^^^^^^^^^^^^^
+------------------------+----------------------------------------------------------+
| Metric Group | Description |
+========================+==========================================================+
| ``CONNECTION_BASIC`` | Connection creation time, relaxed timeout, handoff |
+------------------------+----------------------------------------------------------+
| ``CONNECTION_ADVANCED``| Connection wait time, timeouts, closed connections |
+------------------------+----------------------------------------------------------+
| ``COMMAND`` | Command execution duration |
+------------------------+----------------------------------------------------------+
| ``RESILIENCY`` | Error counts, maintenance notifications |
+------------------------+----------------------------------------------------------+
| ``PUBSUB`` | PubSub message counts (publish/receive) |
+------------------------+----------------------------------------------------------+
| ``STREAMING`` | Stream message lag (XREAD/XREADGROUP) |
+------------------------+----------------------------------------------------------+
| ``CSC`` | Client Side Caching (requests, evictions, bytes saved) |
+------------------------+----------------------------------------------------------+
Available Metrics
^^^^^^^^^^^^^^^^^
The following metrics are collected based on enabled metric groups:
**Connection Metrics:**
- ``db.client.connection.create_time`` - Time to create a new connection (histogram)
- ``db.client.connection.timeouts`` - Number of connection timeouts (counter)
- ``db.client.connection.wait_time`` - Time to obtain a connection from pool (histogram)
- ``db.client.connection.count`` - Current number of connections (observable gauge)
- ``redis.client.connection.closed`` - Total closed connections (counter)
- ``redis.client.connection.relaxed_timeout`` - Relaxed timeout events (up/down counter)
- ``redis.client.connection.handoff`` - Connection handoff events (counter)
**Command Metrics:**
- ``db.client.operation.duration`` - Command execution duration (histogram)
**Resiliency Metrics:**
- ``redis.client.errors`` - Error counts with error type (counter)
- ``redis.client.maintenance.notifications`` - Server maintenance notifications (counter)
**PubSub Metrics:**
- ``redis.client.pubsub.messages`` - Published and received messages (counter)
**Streaming Metrics:**
- ``redis.client.stream.lag`` - End-to-end message lag (histogram)
**Client Side Caching (CSC) Metrics:**
- ``redis.client.csc.requests`` - Cache requests with hit/miss result (counter)
- ``redis.client.csc.evictions`` - Cache evictions (counter)
- ``redis.client.csc.network_saved`` - Bytes saved by caching (counter)
- ``redis.client.csc.items`` - Current cache size (observable gauge)
Custom Histogram Buckets
^^^^^^^^^^^^^^^^^^^^^^^^
You can customize histogram bucket boundaries for better granularity:
.. code-block:: python
config = OTelConfig(
buckets_operation_duration=[0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1],
buckets_connection_create_time=[0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5],
buckets_connection_wait_time=[0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1],
buckets_stream_processing_duration=[0.001, 0.01, 0.1, 1, 10],
)
Context Manager Usage
^^^^^^^^^^^^^^^^^^^^^
For automatic cleanup, use the context manager pattern:
.. code-block:: python
from redis.observability import get_observability_instance, OTelConfig
otel = get_observability_instance()
with otel.get_provider_manager():
# Redis operations here
r = redis.Redis()
r.set('key', 'value')
# Metrics are automatically flushed on exit
Error Handling
^^^^^^^^^^^^^^
The native integration is designed to be non-intrusive. All metric recording functions are wrapped with try-except blocks, ensuring that any errors during metric collection will not affect your Redis operations.
External OpenTelemetry Instrumentation (Alternative)
----------------------------------------------------
As an alternative to the native integration, you can use the external ``opentelemetry-instrumentation-redis`` package. This approach uses monkey-patching to instrument redis-py.
Instrumentations are plugins for popular frameworks and libraries that use OpenTelemetry API to record important operations, for example, HTTP requests, DB queries, logs, errors, and more.
To install OpenTelemetry `instrumentation <https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/redis/redis.html>`_ for redis-py:
.. code-block:: shell
pip install opentelemetry-instrumentation-redis
You can then use it to instrument code like this:
.. code-block:: python
from opentelemetry.instrumentation.redis import RedisInstrumentor
RedisInstrumentor().instrument()
Once the code is patched, you can use redis-py as usual:
.. code-block:: python
# Sync client
client = redis.Redis()
client.get("my-key")
# Async client
client = redis.asyncio.Redis()
await client.get("my-key")
OpenTelemetry API
-----------------
`OpenTelemetry API <https://uptrace.dev/opentelemetry/>`__ is a programming interface that you can use to instrument code and collect telemetry data such as traces, metrics, and logs.
You can use OpenTelemetry API to measure important operations:
.. code-block:: python
from opentelemetry import trace
tracer = trace.get_tracer("app_or_package_name", "1.0.0")
# Create a span with name "operation-name" and kind="server".
with tracer.start_as_current_span("operation-name", kind=trace.SpanKind.CLIENT) as span:
do_some_work()
Record contextual information using attributes:
.. code-block:: python
if span.is_recording():
span.set_attribute("http.method", "GET")
span.set_attribute("http.route", "/projects/:id")
And monitor exceptions:
.. code-block:: python
except ValueError as exc:
# Record the exception and update the span status.
span.record_exception(exc)
span.set_status(trace.Status(trace.StatusCode.ERROR, str(exc)))
See `OpenTelemetry Python Tracing API <https://uptrace.dev/opentelemetry/python-tracing.html>`_ for details.
Uptrace
-------
Uptrace is an `open source APM <https://uptrace.dev/get/open-source-apm.html>`_ that supports distributed tracing, metrics, and logs. You can use it to monitor applications and set up automatic alerts to receive notifications via email, Slack, Telegram, and more.
You can use Uptrace to monitor redis-py using this `GitHub example <https://github.com/redis/redis-py/tree/master/docs/examples/opentelemetry>`_ as a starting point.
.. image:: images/opentelemetry/redis-py-trace.png
:alt: Redis-py trace
You can `install Uptrace <https://uptrace.dev/get/install.html>`_ by downloading a DEB/RPM package or a pre-compiled binary.
Monitoring Redis Server performance
-----------------------------------
In addition to monitoring redis-py client, you can also monitor Redis Server performance using OpenTelemetry Collector Agent.
OpenTelemetry Collector is a proxy/middleman between your application and a `distributed tracing tool <https://uptrace.dev/blog/distributed-tracing-tools.html>`_ such as Uptrace or Jaeger. Collector receives telemetry data, processes it, and then exports the data to APM tools that can store it permanently.
For example, you can use the `OpenTelemetry Redis receiver <https://uptrace.dev/get/monitor/opentelemetry-redis.html>` provided by Otel Collector to monitor Redis performance:
.. image:: images/opentelemetry/redis-metrics.png
:alt: Redis metrics
See introduction to `OpenTelemetry Collector <https://uptrace.dev/opentelemetry/collector.html>`_ for details.
Alerting and notifications
--------------------------
Uptrace also allows you to monitor `OpenTelemetry metrics <https://uptrace.dev/opentelemetry/metrics.html>`_ using alerting rules. For example, the following monitor uses the group by node expression to create an alert whenever an individual Redis shard is down:
.. code-block:: yaml
monitors:
- name: Redis shard is down
metrics:
- redis_up as $redis_up
query:
- group by cluster # monitor each cluster,
- group by bdb # each database,
- group by node # and each shard
- $redis_up
min_allowed_value: 1
# shard should be down for 5 minutes to trigger an alert
for_duration: 5m
You can also create queries with more complex expressions. For example, the following rule creates an alert when the keyspace hit rate is lower than 75%:
.. code-block:: yaml
monitors:
- name: Redis read hit rate < 75%
metrics:
- redis_keyspace_read_hits as $hits
- redis_keyspace_read_misses as $misses
query:
- group by cluster
- group by bdb
- group by node
- $hits / ($hits + $misses) as hit_rate
min_allowed_value: 0.75
for_duration: 5m
See `Alerting and Notifications <https://uptrace.dev/get/alerting.html>`_ for details.
What's next?
------------
Next, you can learn how to configure `uptrace-python <https://uptrace.dev/get/opentelemetry-python.html>`_ to export spans, metrics, and logs to Uptrace.
You may also be interested in the following guides:
- `OpenTelemetry Django <https://uptrace.dev/get/instrument/opentelemetry-django.html>`_
- `OpenTelemetry Flask <https://uptrace.dev/get/instrument/opentelemetry-flask.html>`_
- `OpenTelemetry FastAPI <https://uptrace.dev/get/instrument/opentelemetry-fastapi.html>`_
- `OpenTelemetry SQLAlchemy <https://uptrace.dev/get/instrument/opentelemetry-sqlalchemy.html>`_
---
### Redismodules (`docs/redismodules.rst`)
Redis Modules Commands
######################
Accessing redis module commands requires the installation of the supported `Redis module <https://docs.redis.com/latest/modules/>`_. For a quick start with redis modules, try the `Redismod docker <https://hub.docker.com/r/redislabs/redismod>`_.
RedisBloom Commands
*******************
These are the commands for interacting with the `RedisBloom module <https://redisbloom.io>`_. Below is a brief example, as well as documentation on the commands themselves.
**Create and add to a bloom filter**
.. code-block:: python
import redis
r = redis.Redis()
r.bf().create("bloom", 0.01, 1000)
r.bf().add("bloom", "foo")
**Create and add to a cuckoo filter**
.. code-block:: python
import redis
r = redis.Redis()
r.cf().create("cuckoo", 1000)
r.cf().add("cuckoo", "filter")
**Create Count-Min Sketch and get information**
.. code-block:: python
import redis
r = redis.Redis()
r.cms().initbydim("dim", 1000, 5)
r.cms().incrby("dim", ["foo"], [5])
r.cms().info("dim")
**Create a topk list, and access the results**
.. code-block:: python
import redis
r = redis.Redis()
r.topk().reserve("mytopk", 3, 50, 4, 0.9)
r.topk().info("mytopk")
.. automodule:: redis.commands.bf.commands
:members: BFCommands, CFCommands, CMSCommands, TOPKCommands
------
RedisJSON Commands
******************
These are the commands for interacting with the `RedisJSON module <https://redisjson.io>`_. Below is a brief example, as well as documentation on the commands themselves.
**Create a json object**
.. code-block:: python
import redis
r = redis.Redis()
r.json().set("mykey", ".", {"hello": "world", "i am": ["a", "json", "object!"]})
Examples of how to combine search and json can be found `here <examples/search_json_examples.html>`_.
.. automodule:: redis.commands.json.commands
:members: JSONCommands
-----
RediSearch Commands
*******************
These are the commands for interacting with the `RediSearch module <https://redisearch.io>`_. Below is a brief example, as well as documentation on the commands themselves. In the example
below, an index named *my_index* is being created. When an index name is not specified, an index named *idx* is created.
**Create a search index, and display its information**
.. code-block:: python
import redis
from redis.commands.search.field import TextField
r = redis.Redis()
index_name = "my_index"
schema = (
TextField("play", weight=5.0),
TextField("ball"),
)
r.ft(index_name).create_index(schema)
print(r.ft(index_name).info())
.. automodule:: redis.commands.search.commands
:members: SearchCommands
-----
RedisTimeSeries Commands
************************
These are the commands for interacting with the `RedisTimeSeries module <https://redistimeseries.io>`_. Below is a brief example, as well as documentation on the commands themselves.
**Create a timeseries object with 5 second retention**
.. code-block:: python
import redis
r = redis.Redis()
r.ts().create(2, retention_msecs=5000)
.. automodule:: redis.commands.timeseries.commands
:members: TimeSeriesCommands
---
### Requirements (`docs/requirements.txt`)
sphinx>=5.0,<7.0
docutils<0.18
nbsphinx
sphinx_gallery
ipython
sphinx-autodoc-typehints
furo
pandoc
---
### Resp3 Features (`docs/resp3_features.rst`)
RESP 3 Features
===============
As of version 5.0, redis-py supports the `RESP 3 standard <https://github.com/redis/redis-specifications/blob/master/protocol/RESP3.md>`_. Starting with redis-py 8.0, clients use RESP3 on the wire by default.
By default, redis-py keeps legacy RESP2-compatible Python response shapes for
existing applications. Set ``protocol=3`` explicitly when your application
should receive RESP3-specific Python response shapes or when you want the wire
protocol choice to be visible in code. Set ``protocol=2`` to force RESP2 on the
wire. Set
``legacy_responses=False`` to opt in to protocol-independent unified response
shapes; see :doc:`unified_responses`.
Connecting
-----------
The default connection already uses RESP3 on the wire in redis-py 8.0 and
later while preserving legacy RESP2-compatible Python response shapes. The
following examples show how to set ``protocol=3`` explicitly when you want
RESP3-specific response shapes or visible protocol configuration for standard,
async, and cluster clients.
Connect with a standard connection, explicitly specifying RESP3:
.. code:: python
>>> import redis
>>> r = redis.Redis(host='localhost', port=6379, protocol=3)
>>> r.ping()
Or using the URL scheme:
.. code:: python
>>> import redis
>>> r = redis.from_url("redis://localhost:6379?protocol=3")
>>> r.ping()
Connect with async, explicitly specifying RESP3:
.. code:: python
>>> import redis.asyncio as redis
>>> r = redis.Redis(host='localhost', port=6379, protocol=3)
>>> await r.ping()
The URL scheme with the async client
.. code:: python
>>> import redis.asyncio as Redis
>>> r = redis.from_url("redis://localhost:6379?protocol=3")
>>> await r.ping()
Connecting to an OSS Redis Cluster with RESP 3
.. code:: python
>>> from redis.cluster import RedisCluster, ClusterNode
>>> r = RedisCluster(startup_nodes=[ClusterNode('localhost', 6379), ClusterNode('localhost', 6380)], protocol=3)
>>> r.ping()
Push notifications
------------------
Push notifications are a way that redis sends out of band data. The RESP 3 protocol includes a `push type <https://github.com/redis/redis-specifications/blob/master/protocol/RESP3.md#push-type>`_ that allows our client to intercept these out of band messages. By default, clients will log simple messages, but redis-py includes the ability to bring your own function processor.
This means that should you want to perform something, on a given push notification, you specify a function during the connection, as per this examples:
.. code:: python
>> from redis import Redis
>>
>> def our_func(message):
>> if message.find("This special thing happened"):
>> raise IOError("This was the message: \n" + message)
>>
>> r = Redis(protocol=3)
>> p = r.pubsub(push_handler_func=our_func)
In the example above, upon receipt of a push notification, rather than log the message, in the case where specific text occurs, an IOError is raised. This example, highlights how one could start implementing a customized message handler.
Client-side caching
-------------------
Client-side caching is a technique used to create high performance services.
It utilizes the memory on application servers, typically separate from the database nodes, to cache a subset of the data directly on the application side.
For more information please check the `Redis client-side caching documentation <https://redis.io/docs/latest/develop/use/client-side-caching/>`_.
Please notice that this feature is available only with RESP3 protocol enabled
in sync clients. redis-py 8.0 and later use RESP3 on the wire by default, and
the examples below pass ``protocol=3`` explicitly to make the requirement clear.
Supported in standalone, Cluster, and Sentinel clients.
Basic usage:
Enable caching with default configuration:
.. code:: python
>>> import redis
>>> from redis.cache import CacheConfig
>>> r = redis.Redis(host='localhost', port=6379, protocol=3, cache_config=CacheConfig())
The same interface applies to Redis Cluster and Sentinel.
Enable caching with custom cache implementation:
.. code:: python
>>> import redis
>>> from foo.bar import CacheImpl
>>> r = redis.Redis(host='localhost', port=6379, protocol=3, cache=CacheImpl())
CacheImpl should implement a `CacheInterface` specified in `redis.cache` package.
Which commands are cached
~~~~~~~~~~~~~~~~~~~~~~~~~
A reply is cached only when both of the following hold:
1. The command's metadata says its reply may be cached at all: it is ``readonly``, is not
``blocking``, takes at least one key name argument, and carries none of the
``nondeterministic_output``, ``script_runner`` or ``dont_cache`` markers.
2. The client can identify the key arguments of that particular invocation. A command whose
keys the client does not yet extract executes normally with caching skipped - it never
fails and never changes the reply.
The first question is answered by a ``redis.commands.metadata.MetadataResolver``, which is
also what the Cluster client resolves its routing by. By default the client resolves the
command metadata this library ships, so enabling caching adds no ``COMMAND`` round trips.
An unknown command is never cached.
A different resolver can be supplied as ``metadata_resolver``, which is the seam to implement
if you need eligibility decided some other way - it is a small public ABC, and
``StaticMetadataResolver`` shows what a resolver has to answer. Resolvers chain through
``with_fallback``, first match wins, so a resolver placed in front of the static one overrides
the commands it carries and the static records answer for everything else.
Eligibility can also be decided from the connected server, by building a
``DynamicMetadataResolver`` from a live ``COMMAND`` reply. Use it with care: reading that reply
relies on ``CommandsParser``, which lives in the private ``redis._parsers`` package and is not
part of the public API.
.. code:: python
>>> import redis
>>> from redis._parsers.commands import CommandsParser
>>> from redis.cache import CacheConfig
>>> from redis.commands.metadata import DynamicMetadataResolver, StaticMetadataResolver
>>> records = CommandsParser(redis.Redis(host='localhost', port=6379)).get_commands_metadata_cache()
>>> resolver = StaticMetadataResolver(fallback=DynamicMetadataResolver(records))
>>> r = redis.Redis(host='localhost', port=6379, protocol=3,
... cache_config=CacheConfig(), metadata_resolver=resolver)
Chained this way the server answers only for the commands the shipped table does not carry;
swap the order to let the server override it.
The static table stays the more trustworthy source of the two. It carries the commands
whose server metadata is incomplete or wrong - ``TOUCH`` and ``VRANDMEMBER``, and the
read-only script runners on servers older than 8.10 - and it withholds the routing policies of
the ``movablekeys`` reads (``SINTERCARD``, ``ZDIFF``, ``ZINTER``, ``ZINTERCARD``, ``ZUNION``,
``XREAD``) so the Cluster client keeps resolving their keys itself instead of routing them by
derived keyless policies.
The same argument is accepted by ``ConnectionPool`` (configure it there when you supply your
own ``connection_pool=``) and by ``RedisCluster``, which shares the one resolver with every
node's client. On the Cluster client it also supersedes ``policy_resolver``: given only
``metadata_resolver``, routing is derived from it, so one object serves both. An explicit
``policy_resolver`` still decides routing, for backwards compatibility.
``CacheConfig.DEFAULT_ALLOW_LIST`` is deprecated and no longer consulted.
Tracking modes
~~~~~~~~~~~~~~
Eligibility decides what *may* be cached. ``tracking_mode`` and ``cache_predicate`` decide what
the application *wants* cached, and with it how much the server has to remember.
By default every cache-managed connection sends ``CLIENT TRACKING ON``, so the server remembers
the keys of every read-only, keyed command that connection performs - whether or not the reply
was stored. ``redis.cache.TrackingMode`` selects a narrower contract:
- ``TrackingMode.PLAIN`` - default behaviour. Every trackable read is tracked, and every
eligible reply is stored.
- ``TrackingMode.OPTIN`` - ``CLIENT TRACKING ON OPTIN``. The server remembers nothing unless
``CLIENT CACHING YES`` comes right before the read, which the client sends for exactly the
misses it is about to store. Pick it when you cache a small, chosen subset.
- ``TrackingMode.OPTOUT`` - ``CLIENT TRACKING ON OPTOUT``. The server remembers every trackable
read unless ``CLIENT CACHING NO`` comes right before it, which the client sends for the reads
it will not store. Pick it when you cache almost everything.
``cache_predicate`` is the intent decision. It is called with the command name and the keys of
the invocation, and is consulted under ``optin`` and ``optout`` only - passing it with ``plain``
warns and is ignored. The keys arrive as the
invocation supplied them, so a predicate that inspects them should not assume ``str`` or
``bytes`` - ``redis.utils.str_if_bytes`` normalises both, as in the examples below.
.. code:: python
>>> import redis
>>> from redis.cache import CacheConfig, TrackingMode
>>> from redis.utils import str_if_bytes
>>> r = redis.Redis(host='localhost', port=6379, protocol=3,
... cache_config=CacheConfig(
... max_size=10_000,
... tracking_mode=TrackingMode.OPTOUT,
... cache_predicate=lambda command, keys: not str_if_bytes(keys[0]).startswith('counter:'),
... ))
>>> r.get('user:42') # sent alone; stored, and tracked by default
>>> r.get('counter:hits') # CLIENT CACHING NO + GET, one write; not stored, not tracked
Under ``optin`` the predicate selects what to cache instead:
.. code:: python
>>> r = redis.Redis(host='localhost', port=6379, protocol=3,
... cache_config=CacheConfig(
... tracking_mode=TrackingMode.OPTIN,
... cache_predicate=lambda command, keys: str_if_bytes(keys[0]).startswith('user:'),
... ))
>>> r.get('user:42') # miss: CLIENT CACHING YES + GET, one write; stored, tracked
>>> r.get('user:42') # hit: nothing on the wire
>>> r.get('session:9') # sent alone; not stored, not tracked
``optin`` with no ``cache_predicate`` caches **nothing** - every read is sent alone and left
untracked. That configuration is inert, and the client warns about it when the config is built.
Some things work the same in every mode:
- A cache hit sends nothing to the server.
- Pipelines, transactions and ``send_packed_command`` bypass the cache. Under ``optout`` the
server still tracks the keys they read.
- A command redirected with ``ASK`` is sent without ``CLIENT CACHING``, and its reply is not
stored. ``ASKING`` and ``CLIENT CACHING`` each apply only to the command that immediately
follows them, so one command cannot carry both. Pairing the redirected read with
``CLIENT CACHING`` would strip the ``ASK`` allowance, and the read would be redirected
again. The reply is not stored because it belongs to a slot that is migrating. Under
``optout`` the server tracks the read by default anyway.
- Under ``optout``, the client sends ``CLIENT CACHING NO`` only for commands that the metadata
table marks ``readonly``. Any other read stays tracked. The worst case is an unused entry in
the server's invalidation table, never a stale reply.
The mode is sent in the tracking handshake, because the server refuses to switch a live
connection between ``OPTIN`` and ``OPTOUT``. A configuration change therefore applies to new
connections only.
Commands the cache owns
~~~~~~~~~~~~~~~~~~~~~~~
The cache depends on the tracking state of each connection it manages. Three commands change
that state, so a client that has a cache refuses them with a ``RedisError``, on every
connection and in every mode. This includes pipelines and transactions:
- ``CLIENT CACHING``. The server applies the flag to the next command on that socket, and a
pooled client cannot promise which command that is. A stray ``CLIENT CACHING NO`` under
``optout`` would leave the next cached read untracked, so its stored reply would never be
invalidated. The client sends ``CLIENT CACHING`` itself, together with the read it applies
to, whenever the tracking mode needs it.
- ``CLIENT TRACKING``, in every form, including ``client_tracking_on()`` and
``client_tracking_off()``. Turning tracking off or redirecting it would leave the cache
storing replies that no invalidation reaches. The client enables tracking itself when it
opens the connection.
- ``RESET``, including ``reset()``. It turns off tracking together with the rest of the
connection state.
``CLIENT TRACKINGINFO`` only reads the state, so it stays available. A client without a cache
sends all of these commands to the server as before.
Client-side caching is not yet implemented in the async clients, so
``redis.asyncio.Redis`` takes no ``metadata_resolver`` argument. However,
``redis.asyncio.RedisCluster`` accepts ``metadata_resolver`` for dynamic replica routing.
More comprehensive documentation soon will be available at the `Redis documentation site <https://redis.io/docs/latest/>`_.
---
### Retry (`docs/retry.rst`)
Retry Helpers
#############
.. automodule:: redis.retry
:members:
Retry in Redis Standalone
**************************
>>> from redis.backoff import ExponentialBackoff
>>> from redis.retry import Retry
>>> from redis.client import Redis
>>> from redis.exceptions import (
>>> BusyLoadingError,
>>> RedisError,
>>> )
>>>
>>> # Run 3 retries with exponential backoff strategy
>>> retry = Retry(ExponentialBackoff(), 3)
>>> # Redis client with retries on custom errors in addition to the errors
>>> # that are already retried by default
>>> r = Redis(host='localhost', port=6379, retry=retry, retry_on_error=[BusyLoadingError, RedisError])
As you can see from the example above, Redis client supports 2 parameters to configure the retry behaviour:
* ``retry``: :class:`~.Retry` instance with a :ref:`backoff-label` strategy and the max number of retries
* The :class:`~.Retry` instance has default set of :ref:`exceptions-label` to retry on,
which can be overridden by passing a tuple with :ref:`exceptions-label` to the ``supported_errors`` parameter.
* ``retry_on_error``: list of additional :ref:`exceptions-label` to retry on
``retry_on_error`` can also be set in a connection URL as a comma-separated
list of names from :mod:`redis.exceptions` (for example
``redis://localhost?retry_on_error=ConnectionError,TimeoutError``).
If no ``retry`` is provided, a default one is created with :class:`~.ExponentialWithJitterBackoff` as backoff strategy
and 3 retries.
Retry with asyncio
******************
Async clients use ``redis.asyncio.retry.Retry`` so retry callbacks and backoff
delays can be awaited:
>>> from redis.asyncio import Redis
>>> from redis.asyncio.retry import Retry
>>> from redis.backoff import ExponentialBackoff
>>>
>>> retry = Retry(ExponentialBackoff(), 3)
>>> r = Redis(host='localhost', port=6379, retry=retry)
Passing :class:`redis.retry.Retry` to an async client is supported for
compatibility, but it is converted to ``redis.asyncio.retry.Retry`` and emits a
warning. Custom synchronous ``call_with_retry`` implementations cannot be
preserved by that conversion, so async clients should be configured with an
async retry policy directly.
Retry in Redis Cluster
**************************
>>> from redis.backoff import ExponentialBackoff
>>> from redis.retry import Retry
>>> from redis.cluster import RedisCluster
>>>
>>> # Run 3 retries with exponential backoff strategy
>>> retry = Retry(ExponentialBackoff(), 3)
>>> # Redis Cluster client with retries
>>> rc = RedisCluster(host='localhost', port=6379, retry=retry)
Retry behaviour in Redis Cluster is a little bit different from Standalone:
* ``retry``: :class:`~.Retry` instance with a :ref:`backoff-label` strategy and the max number of retries, default value is ``Retry(ExponentialWithJitterBackoff(base=0.01, cap=1), cluster_error_retry_attempts)``
* ``cluster_error_retry_attempts``: number of times to retry before raising an error when :class:`~.TimeoutError`, :class:`~.ConnectionError`, :class:`~.ClusterDownError` or :class:`~.SlotNotCoveredError` are encountered, default value is ``10``
* This argument is deprecated - it is used to initialize the number of retries for the retry object,
only in the case when the ``retry`` object is not provided.
When the ``retry`` argument is provided, the ``cluster_error_retry_attempts`` argument is ignored!
* The retry object is not yet fully utilized in the cluster client.
The retry object is used only to determine the number of retries for the cluster level calls.
Let's consider the following example:
>>> from redis.backoff import ExponentialBackoff
>>> from redis.retry import Retry
>>> from redis.cluster import RedisCluster
>>>
>>> rc = RedisCluster(host='localhost', port=6379, retry=Retry(ExponentialBackoff(), 6))
>>> rc.set('foo', 'bar')
#. the client library calculates the hash slot for key 'foo'.
#. given the hash slot, it then determines which node to connect to, in order to execute the command.
#. during the connection, a :class:`~.ConnectionError` is raised.
#. because we set ``retry=Retry(ExponentialBackoff(), 6)``, the cluster client starts a cluster update, removes the failed node from the startup nodes, and re-initializes the cluster.
#. the cluster client retries the command until it either succeeds or the max number of retries is reached.
---
### Unified Responses (`docs/unified_responses.rst`)
Migrate to Unified Responses
============================
redis-py can return protocol-independent Python response shapes. When unified
responses are enabled, redis-py applies the same public response structure for
the affected commands whether the connection uses RESP2 or RESP3.
Unified responses are selected with ``legacy_responses=False``. The protocol
still controls the wire format; ``legacy_responses`` controls the Python shape
returned by redis-py.
Unified responses are the recommended mode for new projects and for
applications that can update their response handling. They provide the most
portable application-facing API because command results keep the same Python
shape when you change the Redis serialization protocol.
Enable Unified Responses
------------------------
Use ``legacy_responses=False`` when constructing a client.
.. code:: python
import redis
# Default wire protocol, unified Python responses.
r = redis.Redis(legacy_responses=False)
# RESP2 wire, unified Python responses.
r_resp2 = redis.Redis(protocol=2, legacy_responses=False)
# RESP3 wire, unified Python responses.
r_resp3 = redis.Redis(protocol=3, legacy_responses=False)
The same option is available for asyncio and cluster clients.
.. code:: python
import redis.asyncio as redis_async
from redis.cluster import RedisCluster
async_r = redis_async.Redis(legacy_responses=False)
cluster = RedisCluster(host="localhost", port=6379, legacy_responses=False)
Connection URLs can also select unified responses.
.. code:: python
r = redis.from_url("redis://localhost:6379?legacy_responses=false")
r = redis.from_url(
"redis://localhost:6379?protocol=2&legacy_responses=false"
)
Response Modes
--------------
.. list-table::
:header-rows: 1
:widths: 30 25 45
* - Client options
- Wire protocol
- Python response shape
* - ``Redis()``
- Default RESP3 wire
- Legacy RESP2-compatible shape
* - ``Redis(protocol=2)``
- RESP2
- Legacy RESP2 shape
* - ``Redis(protocol=3)``
- RESP3
- Native RESP3 shape
* - ``Redis(legacy_responses=False)``
- Default RESP3 wire
- Unified shape
* - ``Redis(protocol=2, legacy_responses=False)``
- RESP2
- Unified shape
* - ``Redis(protocol=3, legacy_responses=False)``
- RESP3
- Unified shape
``decode_responses`` is independent from unified responses. It still controls
bulk-string decoding where the command parser does not otherwise normalize a
structural key or value.
Migration Checklist
-------------------
1. Find the redis-py client construction points in your application, including
background workers, admin scripts, asyncio clients, and cluster clients.
2. Enable ``legacy_responses=False`` for one environment or service path first.
The server protocol can stay unchanged unless you also want to pin
``protocol=2`` or ``protocol=3``.
3. Check the Redis commands and ``execute_command()`` calls your application
depends on against the tables below. Update application API type hints,
response models, serializers, and assertions to reflect the unified
response shapes.
4. Review module command responses such as JSON, TimeSeries, RediSearch, and
probabilistic commands separately; several of them return richer objects or
nested containers in unified mode.
5. Keep ``decode_responses`` decisions separate. Unified responses normalize
response structures, while ``decode_responses`` still controls how bulk
string data is decoded.
6. Roll the change out gradually and monitor application paths that parse Redis
responses manually.
RESP2 Legacy to Unified
-----------------------
The following tables summarize the main differences when moving from
``protocol=2`` with legacy responses to unified responses.
Core Commands
^^^^^^^^^^^^^
.. list-table::
:header-rows: 1
:widths: 24 32 22 22
* - Command
- Change
- RESP2 legacy example
- Unified example
* - ``ZDIFF`` with scores
- Flat or tuple score pairs normalize to list pairs with float scores.
- ``[b"a", b"1"]`` or ``[(b"a", b"1")]``
- ``[[b"a", 1.0]]``
* - ``ZINTER``, ``ZRANGE``, ``ZRANGEBYSCORE``, ``ZREVRANGE``, ``ZREVRANGEBYSCORE``, ``ZUNION`` with scores
- Tuple pairs become list pairs; scores are floats.
- ``[(b"a", b"1")]``
- ``[[b"a", 1.0]]``
* - ``ZPOPMAX``, ``ZPOPMIN``
- Tuple pairs become list pairs; scores are floats.
- ``[(b"a", 3)]``
- ``[[b"a", 3.0]]``
* - ``BZPOPMAX``, ``BZPOPMIN``
- Tuple result becomes list result; score is a float.
- ``(b"key", b"a", 3)``
- ``[b"key", b"a", 3.0]``
* - ``ZRANK``, ``ZREVRANK`` with score
- Rank response score is a float.
- ``[2, b"3"]``
- ``[2, 3.0]``
* - ``ZSCAN``
- Score pairs are lists; ``score_cast_func`` receives a float.
- ``(0, [(b"a", b"1")])``
- ``(0, [[b"a", 1.0]])``
* - ``ZRANDMEMBER`` with scores
- Flat interleaved values become nested score pairs.
- ``[b"a", b"1", b"b", b"2"]``
- ``[[b"a", 1.0], [b"b", 2.0]]``
* - ``HRANDFIELD`` with values
- Flat interleaved values become nested field/value pairs.
- ``[b"f1", b"v1", b"f2", b"v2"]``
- ``[[b"f1", b"v1"], [b"f2", b"v2"]]``
* - ``BLPOP``, ``BRPOP``
- Tuple result becomes list result.
- ``(b"key", b"value")``
- ``[b"key", b"value"]``
* - ``ZMPOP``, ``BZMPOP``
- Score values are floats inside nested list pairs.
- ``[b"key", [(b"a", b"1")]]``
- ``[b"key", [[b"a", 1.0]]]``
* - ``XREAD``, ``XREADGROUP``
- List of stream entries becomes a dict keyed by stream.
- ``[[b"s", [(b"1-0", {})]]]``
- ``{b"s": [(b"1-0", {})]}``
* - ``LCS`` with ``IDX``
- Flat key/value list becomes a dict with string keys.
- ``[b"matches", [...], b"len", 3]``
- ``{"matches": [...], "len": 3}``
* - ``STRALGO ... IDX``
- Match ranges use lists and string keys.
- ``{"matches": [((0, 2), (0, 2))], "len": 3}``
- ``{"matches": [[[0, 2], [0, 2]]], "len": 3}``
* - ``CLIENT TRACKINGINFO``
- Flat list becomes dict; structural strings are decoded.
- ``[b"flags", [b"on"], b"redirect", -1]``
- ``{"flags": ["on"], "redirect": -1}``
* - ``COMMAND``
- Flags and ACL categories are sets of strings.
- ``{"get": {"flags": [b"readonly"]}}``
- ``{"get": {"flags": {"readonly"}, "acl_categories": {...}}}``
* - ``ACL GETUSER``
- Selector lists become selector dicts.
- ``{"selectors": [[b"~*", b"+get"]]}``
- ``{"selectors": [{"keys": "~*", "commands": "+get"}]}``
* - ``ACL LOG``
- ``age-seconds`` is float and ``client-info`` is parsed.
- ``{"age-seconds": b"0.5"}``
- ``{"age-seconds": 0.5, "client-info": {...}}``
* - ``SENTINEL`` state commands
- ``flags`` becomes a set and derived booleans are present.
- ``{"flags": "master,odown"}``
- ``{"flags": {"master", "odown"}, "is_master": True}``
* - ``CLUSTER LINKS``
- Link dictionaries use string keys.
- ``[{b"direction": b"to"}]``
- ``[{"direction": b"to"}]``
* - ``CLUSTER SHARDS``
- Shard and node dictionaries use string keys.
- ``{b"nodes": [{b"id": b"abc"}]}``
- ``{"nodes": [{"id": b"abc"}]}``
* - ``GEOPOS``
- Coordinates are lists.
- ``[(1.0, 2.0), None]``
- ``[[1.0, 2.0], None]``
* - ``GEOSEARCH``, ``GEORADIUS``, ``GEORADIUSBYMEMBER`` with coordinates
- RESP2 and RESP3 unified responses use tuple coordinates.
- ``[b"place", (1.0, 2.0)]``
- ``[b"place", (1.0, 2.0)]``
* - ``FUNCTION LIST``
- Flat function data becomes nested dictionaries.
- ``[[b"library_name", b"lib", ...]]``
- ``[{b"library_name": b"lib", b"functions": [...]}]``
* - ``MEMORY STATS``
- Structural keys are decoded and numeric values use native types.
- ``{b"peak.allocated": b"1024"}``
- ``{"peak.allocated": 1024}``
JSON
^^^^
.. list-table::
:header-rows: 1
:widths: 24 32 22 22
* - Command
- Change
- RESP2 legacy example
- Unified example
* - ``JSON.NUMINCRBY``, ``JSON.NUMMULTBY``
- Legacy scalar paths are normalized with JSONPath array behavior.
- ``5``
- ``[5]``
* - ``JSON.RESP``
- Numeric string leaves that represent floats become Python floats.
- ``[b"{", b"price", b"-19.5"]``
- ``[b"{", b"price", -19.5]``
* - ``JSON.OBJKEYS``
- Key values respect ``decode_responses`` instead of being forced to str.
- ``["a", "b"]``
- ``[b"a", b"b"]`` when ``decode_responses=False``
TimeSeries
^^^^^^^^^^
.. list-table::
:header-rows: 1
:widths: 24 32 22 22
* - Command
- Change
- RESP2 legacy example
- Unified example
* - ``TS.GET``
- Tuple becomes list.
- ``(1, 2.0)``
- ``[1, 2.0]``
* - ``TS.RANGE``, ``TS.REVRANGE``
- Sample tuples become sample lists.
- ``[(1, 2.0)]``
- ``[[1, 2.0]]``
* - ``TS.MGET``
- Sorted list of dicts becomes a key/value dict.
- ``[{b"k": [{}, None, None]}]``
- ``{b"k": [{}, [1, 2.0]]}``
* - ``TS.MRANGE``, ``TS.MREVRANGE``
- Sorted list of dicts becomes a dict with metadata slot.
- ``[{b"k": [labels, samples]}]``
- ``{b"k": [labels, metadata, samples]}``
* - ``TS.QUERYINDEX``
- Key strings are preserved instead of coerced to numbers.
- ``2``
- ``"2"``
RediSearch Commands
^^^^^^^^^^^^^^^^^^^
.. list-table::
:header-rows: 1
:widths: 24 32 22 22
* - Command
- Change
- RESP2 legacy example
- Unified example
* - ``FT.INFO``
- Attribute sublists become structured dictionaries.
- ``[["identifier", "title", "SORTABLE"]]``
- ``[{"identifier": "title", "flags": ["SORTABLE"]}]``
* - ``FT.CONFIG GET``
- Keys and values are strings.
- ``{b"TIMEOUT": b"500"}``
- ``{"TIMEOUT": "500"}``
* - ``FT.SEARCH``
- ``Result`` includes response warnings.
- ``result.total``, ``result.docs``
- ``result.total``, ``result.docs``, ``result.warnings``
* - ``FT.AGGREGATE``
- ``AggregateResult`` includes ``total`` and ``warnings``.
- ``result.rows``
- ``result.total``, ``result.rows``, ``result.warnings``
* - ``FT.PROFILE``
- Returns parsed result plus ``ProfileInformation``.
- ``(result, ProfileInformation(list_data))``
- ``(result, ProfileInformation(profile_data))``
* - ``FT.HYBRID``
- Returns ``HybridResult``; result field values and warnings stay bytes by default.
- ``HybridResult(..., results=[{"field": b"value"}])``
- ``HybridResult(..., results=[{"field": b"value"}])``
Probabilistic
^^^^^^^^^^^^^
.. list-table::
:header-rows: 1
:widths: 24 32 22 22
* - Command
- Change
- RESP2 legacy example
- Unified example
* - ``TOPK.ADD``, ``TOPK.INCRBY``, ``TOPK.LIST``
- Item names are preserved instead of coerced through numeric parsing.
- ``"42"`` could become ``42``
- ``"42"`` stays ``"42"``
Vector Sets
^^^^^^^^^^^
.. list-table::
:header-rows: 1
:widths: 24 32 22 22
* - Command
- Change
- RESP2 legacy example
- Unified example
* - ``VEMB``
- Vector components are always floats, as on RESP3.
- ``[1, 0.5, 0]``
- ``[1.0, 0.5, 0.0]``
RESP3 Legacy to Unified
-----------------------
The following tables summarize the main differences when moving from
``protocol=3`` with legacy responses to unified responses.
Core Commands
^^^^^^^^^^^^^
.. list-table::
:header-rows: 1
:widths: 24 32 22 22
* - Command
- Change
- RESP3 legacy example
- Unified example
* - Sorted-set score commands
- Scores pass through the same callback normalization as RESP2 unified.
- ``[[b"a", 1.0]]``
- ``[[b"a", 1.0]]``
* - ``ZSCAN``
- Score pairs are lists and ``score_cast_func`` receives floats.
- ``(0, [[b"a", 1.0]])``
- ``(0, [[b"a", 1.0]])``
* - ``ACL CAT``, ``ACL HELP``, ``ACL LIST``, ``ACL USERS``
- Byte strings are decoded to strings.
- ``[b"default"]``
- ``["default"]``
* - ``ACL GENPASS``, ``ACL WHOAMI``, ``CLIENT GETNAME``, ``RESET``
- Byte scalar becomes string scalar.
- ``b"default"``
- ``"default"``
* - ``ACL LOG``
- ``age-seconds`` is float, ``client-info`` is parsed, and structural strings are decoded.
- ``{"age-seconds": "0.5"}``
- ``{"age-seconds": 0.5, "client-info": {...}}``
* - ``CLIENT TRACKINGINFO``
- Structural keys and string lists are decoded.
- ``{b"flags": [b"on"]}``
- ``{"flags": ["on"]}``
* - ``COMMAND``
- Flags and ACL categories are sets of strings.
- ``{"get": {"flags": [b"readonly"]}}``
- ``{"get": {"flags": {"readonly"}, "acl_categories": {...}}}``
* - ``CLUSTER LINKS``, ``CLUSTER SHARDS``
- Dict keys are strings at the exposed structural levels.
- ``[{b"direction": b"to"}]``
- ``[{"direction": b"to"}]``
* - ``GEOHASH``
- Hash strings are decoded.
- ``[b"sqc8b49rny0"]``
- ``["sqc8b49rny0"]``
* - ``GEOPOS``
- Keeps RESP3-style list coordinates.
- ``[[1.0, 2.0]]``
- ``[[1.0, 2.0]]``
* - ``GEOSEARCH``, ``GEORADIUS``, ``GEORADIUSBYMEMBER`` with coordinates
- Coordinates normalize to tuples.
- ``[b"place", [1.0, 2.0]]``
- ``[b"place", (1.0, 2.0)]``
* - ``LCS`` and ``STRALGO ... IDX``
- Dict keys are strings and match ranges use the unified list shape.
- ``{b"matches": [...]}``
- ``{"matches": [...], "len": 3}``
* - ``SENTINEL`` state commands
- Native RESP3 state maps normalize to the unified state dictionaries.
- ``{b"flags": b"master"}``
- ``{"flags": {"master"}, "is_master": True}``
Module Commands
^^^^^^^^^^^^^^^
.. list-table::
:header-rows: 1
:widths: 24 32 22 22
* - Command
- Change
- RESP3 legacy example
- Unified example
* - ``BF.INFO``, ``CF.INFO``, ``CMS.INFO``, ``TOPK.INFO``, ``TDIGEST.INFO``
- Raw module info maps become rich info objects.
- ``{b"Capacity": 100}``
- ``info.capacity == 100``
* - ``TDIGEST.BYRANK``, ``TDIGEST.BYREVRANK``, ``TDIGEST.CDF``, ``TDIGEST.QUANTILE``
- Raw lists are parsed with numeric and special-value handling.
- ``[0.5, b"inf"]``
- ``[0.5, "inf"]``
* - ``TS.INFO``
- Raw TimeSeries info map becomes ``TSInfo``.
- ``{b"totalSamples": 10}``
- ``info.total_samples == 10``
* - ``TS.MRANGE``, ``TS.MREVRANGE``
- Keeps the metadata slot in the unified three-element value.
- ``{b"k": [labels, samples]}``
- ``{b"k": [labels, metadata, samples]}``
* - ``JSON.TYPE`` for missing keys
- Wrapped missing value becomes bare ``None``.
- ``[None]``
- ``None``
* - ``JSON.RESP``
- Numeric float leaves are normalized consistently.
- ``[b"{", b"price", b"-19.5"]``
- ``[b"{", b"price", -19.5]``
* - ``FT.SEARCH``
- Raw RESP3 result map becomes ``Result``.
- ``{"total_results": 2, "results": [...]}``
- ``result.total == 2``
* - ``FT.AGGREGATE``
- Raw RESP3 aggregate map becomes ``AggregateResult``.
- ``{"total_results": 2, "results": [...]}``
- ``result.total == 2``
* - ``FT.PROFILE``
- Single ``ProfileInformation`` wrapper becomes parsed result plus profile info.
- ``ProfileInformation(raw_profile_response)``
- ``(result, ProfileInformation(profile_data))``
* - ``FT.SPELLCHECK``
- Native nested RESP3 spellcheck map becomes normalized term suggestions.
- ``{"results": {"term": [{"fix": 0.0}]}}``
- ``{"term": [{"score": "0", "suggestion": "fix"}]}``
* - ``FT.INFO``, ``FT.CONFIG GET``, ``FT.SYNDUMP``
- Structural keys are strings.
- ``{b"index_name": b"idx"}``
- ``{"index_name": "idx"}``
* - ``FT.HYBRID``
- Raw native response becomes ``HybridResult``; field values remain bytes by default.
- ``{"total_results": 1, "results": [...]}``
- ``HybridResult(total_results=1, results=[...])``
HYBRID Command Loaded Fields
----------------------------
``FT.HYBRID`` is experimental. Unified responses for the HYBRID command
preserve loaded field values as bytes by default to keep binary data, such as
vector fields, intact.
Use ``decode_field=True`` only for fields that are known text values.
.. code:: python
post = HybridPostProcessingConfig()
post.load("@title", decode_field=True)
post.load("@embedding", decode_field=False)
---
### Examples Opentelemetry Docker Compose.Yml (`docs/examples/opentelemetry/docker-compose.yml`)
version: "3"
services:
clickhouse:
image: clickhouse/clickhouse-server:22.7
restart: on-failure
environment:
CLICKHOUSE_DB: uptrace
healthcheck:
test: ["CMD", "wget", "--spider", "-q", "localhost:8123/ping"]
interval: 1s
timeout: 1s
retries: 30
volumes:
- ch_data:/var/lib/clickhouse
ports:
- "8123:8123"
- "9000:9000"
uptrace:
image: "uptrace/uptrace:1.2.0"
#image: 'uptrace/uptrace-dev:latest'
restart: on-failure
volumes:
- uptrace_data:/var/lib/uptrace
- ./uptrace.yml:/etc/uptrace/uptrace.yml
#environment:
# - DEBUG=2
ports:
- "14317:14317"
- "14318:14318"
depends_on:
clickhouse:
condition: service_healthy
otel-collector:
image: otel/opentelemetry-collector-contrib:0.58.0
restart: on-failure
volumes:
- ./config/otel-collector.yaml:/etc/otelcol-contrib/config.yaml
ports:
- "4317:4317"
- "4318:4318"
vector:
image: timberio/vector:0.24.X-alpine
volumes:
- ./config/vector.toml:/etc/vector/vector.toml:ro
alertmanager:
image: prom/alertmanager:v0.24.0
restart: on-failure
volumes:
- ./config/alertmanager.yml:/etc/alertmanager/config.yml
- alertmanager_data:/alertmanager
ports:
- 9093:9093
command:
- "--config.file=/etc/alertmanager/config.yml"
- "--storage.path=/alertmanager"
mailhog:
image: mailhog/mailhog:v1.0.1
restart: on-failure
ports:
- "8025:8025"
redis-server:
image: redis
ports:
- "6379:6379"
redis-cli:
image: redis
volumes:
uptrace_data:
driver: local
ch_data:
driver: local
alertmanager_data:
driver: local
---
### Examples Opentelemetry Requirements (`docs/examples/opentelemetry/requirements.txt`)
redis==4.3.4
uptrace==1.14.0
opentelemetry-instrumentation-redis==0.35b0
---
### Examples Opentelemetry Uptrace.Yml (`docs/examples/opentelemetry/uptrace.yml`)
##
## Uptrace configuration file.
## See https://uptrace.dev/get/config.html for details.
##
## You can use environment variables anywhere in this file, for example:
##
## foo: $FOO
## bar: ${BAR}
## baz: ${BAZ:default}
##
## To escape `$`, use `$$`, for example:
##
## foo: $$FOO_BAR
##
##
## ClickHouse database credentials.
##
ch:
# Connection string for ClickHouse database. For example:
# clickhouse://:@<host>:/<database>?sslmode=disable
#
# See https://clickhouse.uptrace.dev/guide/golang-clickhouse.html#options
dsn: "clickhouse://default:@clickhouse:9000/uptrace?sslmode=disable"
##
## A list of pre-configured projects. Each project is fully isolated.
##
projects:
# Conventionally, the first project is used to monitor Uptrace itself.
- id: 1
name: Uptrace
# Token grants write access to the project. Keep a secret.
token: project1_secret_token
pinned_attrs:
- service.name
- host.name
- deployment.environment
# Group spans by deployment.environment attribute.
group_by_env: false
# Group funcs spans by service.name attribute.
group_funcs_by_service: false
# Other projects can be used to monitor your applications.
# To monitor micro-services or multiple related services, use a single project.
- id: 2
name: My project
token: project2_secret_token
pinned_attrs:
- service.name
- host.name
- deployment.environment
# Group spans by deployment.environment attribute.
group_by_env: false
# Group funcs spans by service.name attribute.
group_funcs_by_service: false
##
## Create metrics from spans and events.
##
metrics_from_spans:
- name: uptrace.tracing.spans_duration
instrument: histogram
unit: microseconds
value: span.duration / 1000
attrs:
- span.system as system
- service.name as service
- host.name as host
- span.status_code as status
where: not span.is_event
- name: uptrace.tracing.spans
instrument: counter
unit: 1
value: span.count
attrs:
- span.system as system
- service.name as service
- host.name as host
- span.status_code as status
where: not span.is_event
- name: uptrace.tracing.events
instrument: counter
unit: 1
value: span.count
attrs:
- span.system as system
- service.name as service
- host.name as host
where: span.is_event
##
## To require authentication, uncomment the following section.
##
auth:
# users:
# - username: uptrace
# password: uptrace
# - username: admin
# password: admin
# # Cloudflare user provider: uses Cloudflare Zero Trust Access (Identity)
# # See https://developers.cloudflare.com/cloudflare-one/identity/ for more info.
# cloudflare:
# # The base URL of the Cloudflare Zero Trust team.
# - team_url: https://myteam.cloudflareaccess.com
# # The Application Audience (AUD) Tag for this application.
# # You can retrieve this from the Cloudflare Zero Trust 'Access' Dashboard.
# audience: bea6df23b944e4a0cd178609ba1bb64dc98dfe1f66ae7b918e563f6cf28b37e0
# # OpenID Connect (Single Sign-On)
# oidc:
# # The ID is used in API endpoints, for example, in redirect URL
# # `http:///api/v1/sso/<oidc-id>/callback`.
# - id: keycloak
# # Display name for the button in the login form.
# # Default to 'OpenID Connect'
# display_name: Keycloak
# # The base URL for the OIDC provider.
# issuer_url: http://localhost:8080/realms/uptrace
# # The OAuth 2.0 Client ID
# client_id: uptrace
# # The OAuth 2.0 Client Secret
# client_secret: ogbhd8Q0X0e5AZFGSG3m9oirPvnetqkA
# # Additional OAuth 2.0 scopes to request from the OIDC provider.
# # Defaults to 'profile'. 'openid' is requested by default and need not be specified.
# scopes:
# - profile
# # The OIDC UserInfo claim to use as the user's username.
# # Defaults to 'preferred_username'.
# claim: preferred_username
##
## Alerting rules for monitoring metrics.
##
## See https://uptrace.dev/get/alerting.html for details.
##
alerting:
rules:
- name: Network errors
metrics:
- system.network.errors as $net_errors
query:
- $net_errors > 0 group by host.name
# for the last 5 minutes
for: 5m
annotations:
summary: "{{ $labels.host_name }} has high number of net errors: {{ $values.net_errors }}"
- name: Filesystem usage >= 90%
metrics:
- system.filesystem.usage as $fs_usage
query:
- group by host.name
- group by device
- where device !~ "loop"
- $fs_usage{state="used"} / $fs_usage >= 0.9
for: 5m
annotations:
summary: "{{ $labels.host_name }} has high FS usage: {{ $values.fs_usage }}"
- name: Uptrace is dropping spans
metrics:
- uptrace.projects.spans as $spans
query:
- $spans{type=dropped} > 0
for: 1m
annotations:
summary: "Uptrace has dropped {{ $values.spans }} spans"
- name: Always firing (for fun and testing)
metrics:
- process.runtime.go.goroutines as $goroutines
query:
- $goroutines >= 0 group by host.name
for: 1m
annotations:
summary: "{{ $labels.host_name }} has high number of goroutines: {{ $values.goroutines }}"
# Create alerts from error logs and span events.
create_alerts_from_spans:
enabled: true
labels:
alert_kind: error
##
## AlertManager client configuration.
## See https://uptrace.dev/get/alerting.html for details.
##
## Note that this is NOT an AlertManager config and you need to configure AlertManager separately.
## See https://prometheus.io/docs/alerting/latest/configuration/ for details.
##
alertmanager_client:
# AlertManager API endpoints that Uptrace uses to manage alerts.
urls:
- "http://alertmanager:9093/api/v2/alerts"
##
## Various options to tweak ClickHouse schema.
## For changes to take effect, you need reset the ClickHouse database with `ch reset`.
##
ch_schema:
# Compression codec, for example, LZ4, ZSTD(3), or Default.
compression: ZSTD(3)
# Whether to use ReplicatedMergeTree instead of MergeTree.
replicated: false
# Cluster name for Distributed tables and ON CLUSTER clause.
#cluster: uptrace1
spans:
storage_policy: "default"
# Delete spans data after 30 days.
ttl_delete: 30 DAY
metrics:
storage_policy: "default"
# Delete metrics data after 90 days.
ttl_delete: 90 DAY
##
## Addresses on which Uptrace receives gRPC and HTTP requests.
##
listen:
# OTLP/gRPC API.
grpc:
addr: ":14317"
# tls:
# cert_file: config/tls/uptrace.crt
# key_file: config/tls/uptrace.key
# OTLP/HTTP API and Uptrace API with UI.
http:
addr: ":14318"
# tls:
# cert_file: config/tls/uptrace.crt
# key_file: config/tls/uptrace.key
##
## Various options for Uptrace UI.
##
site:
# Overrides public URL for Vue-powered UI in case you put Uptrace behind a proxy.
#addr: 'https://uptrace.mydomain.com'
##
## Spans processing options.
##
spans:
# The size of the Go chan used to buffer incoming spans.
# If the buffer is full, Uptrace starts to drop spans.
#buffer_size: 100000
# The number of spans to insert in a single query.
#batch_size: 10000
##
## Metrics processing options.
##
metrics:
# List of attributes to drop for being noisy.
drop_attrs:
- telemetry.sdk.language
- telemetry.sdk.name
- telemetry.sdk.version
# The size of the Go chan used to buffer incoming measures.
# If the buffer is full, Uptrace starts to drop measures.
#buffer_size: 100000
# The number of measures to insert in a single query.
#batch_size: 10000
##
## SQLite/PostgreSQL db that is used to store metadata such us metric names, dashboards, alerts,
## and so on.
##
db:
# Either sqlite or postgres.
driver: sqlite
# Database connection string.
#
# Uptrace automatically creates SQLite database file in the current working directory.
# Make sure the directory is writable by Uptrace process.
dsn: "file:uptrace.sqlite3?_pragma=foreign_keys(1)&_pragma=busy_timeout(1000)"
# Secret key that is used to sign JWT tokens etc.
secret_key: 102c1a557c314fc28198acd017960843
# Enable to log HTTP requests and database queries.
debug: false
---
### Examples Opentelemetry Config Alertmanager.Yml (`docs/examples/opentelemetry/config/alertmanager.yml`)
# See https://prometheus.io/docs/alerting/latest/configuration/ for details.
global:
# The smarthost and SMTP sender used for mail notifications.
smtp_smarthost: "mailhog:1025"
smtp_from: "[email protected]"
smtp_require_tls: false
receivers:
- name: "team-X"
email_configs:
- to: "[email protected]"
send_resolved: true
# The root route on which each incoming alert enters.
route:
# The labels by which incoming alerts are grouped together. For example,
# multiple alerts coming in for cluster=A and alertname=LatencyHigh would
# be batched into a single group.
group_by: ["alertname", "cluster", "service"]
# When a new group of alerts is created by an incoming alert, wait at
# least 'group_wait' to send the initial notification.
# This way ensures that you get multiple alerts for the same group that start
# firing shortly after another are batched together on the first
# notification.
group_wait: 30s
# When the first notification was sent, wait 'group_interval' to send a batch
# of new alerts that started firing for that group.
group_interval: 5m
# If an alert has successfully been sent, wait 'repeat_interval' to
# resend them.
repeat_interval: 3h
# A default receiver
receiver: team-X
# All the above attributes are inherited by all child routes and can
# overwritten on each.
# The child route trees.
routes:
# This route matches error alerts created from spans or logs.
- matchers:
- alert_kind="error"
group_interval: 24h
receiver: team-X
# The directory from which notification templates are read.
templates:
- "/etc/alertmanager/template/*.tmpl"
---
### Examples Opentelemetry Config Otel Collector.Yaml (`docs/examples/opentelemetry/config/otel-collector.yaml`)
extensions:
health_check:
pprof:
endpoint: 0.0.0.0:1777
zpages:
endpoint: 0.0.0.0:55679
receivers:
otlp:
protocols:
grpc:
http:
hostmetrics:
collection_interval: 10s
scrapers:
cpu:
disk:
load:
filesystem:
memory:
network:
paging:
redis:
endpoint: "redis-server:6379"
collection_interval: 10s
jaeger:
protocols:
grpc:
processors:
resourcedetection:
detectors: ["system"]
batch:
send_batch_size: 10000
timeout: 10s
exporters:
logging:
logLevel: debug
otlp:
endpoint: uptrace:14317
tls:
insecure: true
headers: { "uptrace-dsn": "http://project2_secret_token@localhost:14317/2" }
service:
# telemetry:
# logs:
# level: DEBUG
pipelines:
traces:
receivers: [otlp, jaeger]
processors: [batch]
exporters: [otlp, logging]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
metrics/hostmetrics:
receivers: [hostmetrics, redis]
processors: [batch, resourcedetection]
exporters: [otlp]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
extensions: [health_check, pprof, zpages]
---
--- METRICS ---
- Files Extracted: 27
- Estimated Token Budget: ~41152 tokens
- Canonical Reference: https://codewiki.google/github.com/redis/redis-py