seastar Documentation

Technical Documentation: scylladb/seastar

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Seastar

Introduction Copy Chapter

Seastar is an event-driven framework that allows you to write non-blocking,
asynchronous code in a relatively straightforward manner (once understood).
It is based on futures.

Building Seastar Copy Chapter

For more details and alternative workflows, read HACKING.md.

Assuming that you would like to use system packages (RPMs or DEBs) for Seastar's dependencies, first install them:

$ sudo ./install-dependencies.sh

Then configure (in "release" mode):

$ ./configure.py --mode=release

Then compile:

$ ninja -C build/release

If compilation fails with an error such as g++: internal compiler error: Killed (program cc1plus),
try giving GCC more memory. You can limit the number of parallel jobs with -j1 and/or allocate at least 4 GiB of RAM to your
machine.

If you're missing a dependency of Seastar, then it is possible to have the configuration process fetch a version of the dependency locally for development.

For example, to fetch fmt locally, configure Seastar like this:

$ ./configure.py --mode=dev --cook fmt

--cook can be repeated many times for selecting multiple dependencies.

Build modes Copy Chapter

The configure.py script is a wrapper around CMake. The --mode argument
maps to CMAKE_BUILD_TYPE and supports the following modes:

Note that Seastar is more sensitive to allocators and optimizations than
usual. As a rough rule of thumb, release is twice as fast as dev, 150
times as fast as sanitize, and 300 times as fast as debug.

Using Seastar from its build directory (without installation) Copy Chapter

It's possible to consume Seastar directly from its build directory with CMake or pkg-config.

We'll assume that the Seastar repository is located in a directory at $seastar_dir.

Via pkg-config:

$ g++ my_app.cc $(pkg-config --libs --cflags --static $seastar_dir/build/release/seastar.pc) -o my_app

and with CMake using the Seastar package:

CMakeLists.txt for my_app:

set (CMAKE_CXX_STANDARD 23)

find_package (Seastar REQUIRED)

add_executable (my_app
  my_app.cc)

target_link_libraries (my_app
  Seastar::seastar)

$ mkdir $my_app_dir/build
$ cd $my_app_dir/build
$ cmake -DCMAKE_PREFIX_PATH="$seastar_dir/build/release;$seastar_dir/build/release/_cooking/installed" -DCMAKE_MODULE_PATH=$seastar_dir/cmake $my_app_dir

The CMAKE_PREFIX_PATH values ensure that CMake can locate Seastar and its compiled submodules. The CMAKE_MODULE_PATH value ensures that CMake can use Seastar's CMake scripts to locate its dependencies.

Using an installed Seastar Copy Chapter

You can also consume Seastar after it has been installed to the filesystem.

Important:

• Seastar works with a customized version of DPDK, so by default builds and installs the DPDK submodule to $build_dir/_cooking/installed

First, configure the installation path:

$ ./configure.py --mode=release --prefix=/usr/local

Then run the install target:

$ ninja -C build/release install

Then consume it from pkg-config:

$ g++ my_app.cc $(pkg-config --libs --cflags --static seastar) -o my_app

or consume it with the same CMakeLists.txt as before but with a simpler CMake invocation:

$ cmake ..

(If Seastar has not been installed to a "standard" location like /usr or /usr/local, then you can invoke CMake with -DCMAKE_PREFIX_PATH=$my_install_root.)

There are also instructions for building on any host that supports Docker.

Use of DPDK is optional.

Seastar's C++ standard: C++23 or C++26

Seastar supports both C++23 and C++26. The build defaults to the latest
standard supported by your compiler, but can be explicitly selected with
the --c++-standard configure option, e.g., --c++-standard=23,
or, if using CMake directly, by setting the CMAKE_CXX_STANDARD CMake
variable.

See the compatibility statement for more information.

Getting started Copy Chapter

There is a mini tutorial and a more comprehensive one.

The documentation is available on the web.

Resources Copy Chapter

• Seastar Development Mailing List: Discuss challenges, propose improvements, send
  code contributions (patches), and get help from experienced developers.
  Subscribe or browse archives: here
  (or email [email protected]).
• GitHub Discussions: For more casual conversations and quick questions, consider
  using the Seastar project's discussions on GitHub.
• Issue Tracker: File bug reports on the project's issue tracker.

Learn more about Seastar on the main project website.

The Native TCP/IP Stack Copy Chapter

Seastar comes with its own userspace TCP/IP stack for better performance.

Recommended hardware configuration for Seastar Copy Chapter

• CPUs - As many as you need. Seastar is well suited to multicore and NUMA systems.
• NICs - As fast as possible; we recommend 10G or 40G cards. It is possible to use
  1G cards too, but you may be limited by their capacity.
  In addition, the more hardware queues per CPU, the better for Seastar.
  Otherwise we have to emulate that in software.
• Disks - Fast SSDs with a high number of IOPS.
• Client machines - Usually a single client machine cannot fully load our servers.
  Both memaslap (memcached) and wrk (httpd) may be unable to overload their matching
  server counterparts. We recommend running clients on machines other than the servers
  and using several of them.

Projects using Seastar Copy Chapter

• cpv-cql-driver: C++ driver for Cassandra/Scylla based on the Seastar framework
• cpv-framework: A web framework written in C++ based on the Seastar framework
• Redpanda: A Kafka-compatible streaming data platform for mission-critical systems
• Scylla: A fast and reliable NoSQL data store compatible with Cassandra and DynamoDB
• smf: The fastest RPC in the West
• Ceph - Crimson: Next-generation OSD (Object Storage Daemon) implementation based on the Seastar framework

Overview (`kvm/README.md`) Copy Section

seastar-in-kvm

Create a VM for Seastar development environment

Why we need this

SeaStar scores muximum performance with DPDK, but it cannot live with existing NIC driver/Linux kernel network stack.
Also it directly accesses NIC device, it's bit hard to try it on remote node.

seastar-in-kvm offers Fedora VM with SeaStar + DPDK without setup, it's easiest way to begin SeaStar application development.

Prerequire Copy Section

On Fedora 21:

yum install @virtualization
systemctl enable libvirtd
systemctl start libvirtd
yum install libguestfs-tools-c virt-install

How to build & run Copy Section

./build.sh
./register.sh
virsh start seastar-dev && virsh console seastar-dev
(Try login as 'seastar' after firstboot.sh finished, Fedora will ask new password for the user)

Usage of the VM Copy Section

Wait until finish running setup script on first startup.
Then login as 'seastar', login prompt will ask for entering new password.

After login to seastar, initialize DPDK module by following instruction:

sudo su - # entering root user
resize    # extend console to actual terminal window size
export TERM=xterm-256color # set terminal type
cd ~/dpdk
./tools/setup.sh

# input numbers by following order:
(type 9 to re-compile DPDK)
(type 12 to insert IGB UIO module)
(type 15, then input "64" to setup hugepage mappings)
(type 18, then input PCI device id something like "0000:xx:yy.z",
which is shown at 'Network devices using DPDK-compatible driver')
(type 30 to exit)

cd ~/seastar
# httpd example
env LD_LIBRARY_PATH=~/dpdk/x86_64-native-linuxapp-gcc/lib/ \
./build/release/apps/httpd/httpd --network-stack native --dpdk-pmd --csum-offload off

Doc CMakeLists (`doc/CMakeLists.txt`) Copy Section

find_program (Seastar_DOXYGEN_EXECUTABLE doxygen)
if (NOT Seastar_DOXYGEN_EXECUTABLE)
message (FATAL_ERROR "doxygen is required for building document!")
endif ()

configure_file (
${CMAKE_CURRENT_SOURCE_DIR}/Doxyfile.in
${CMAKE_CURRENT_BINARY_DIR}/Doxyfile
@ONLY)

configure_file (
${CMAKE_CURRENT_SOURCE_DIR}/DoxygenLayout.xml
${CMAKE_CURRENT_BINARY_DIR}/DoxygenLayout.xml
COPYONLY)

add_custom_target (doc_api
COMMAND ${Seastar_DOXYGEN_EXECUTABLE} ${CMAKE_CURRENT_BINARY_DIR}/Doxyfile)

add_custom_command (
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/html/tutorial.html
DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/tutorial.md
COMMAND
${CMAKE_CURRENT_SOURCE_DIR}/md2html
${CMAKE_CURRENT_SOURCE_DIR}/tutorial.md
${CMAKE_CURRENT_BINARY_DIR}/html/tutorial.html)

add_custom_target (doc_tutorial_html
DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/html/tutorial.html)

add_custom_command (
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/html/split
DEPENDS
# Necessary because file-level dependencies are not propagated for custom targets.
${CMAKE_CURRENT_BINARY_DIR}/html/tutorial.html
doc_tutorial_html
COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_CURRENT_BINARY_DIR}/html/split
COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/htmlsplit.py
--input ${CMAKE_CURRENT_BINARY_DIR}/html/tutorial.html
--output-dir ${CMAKE_CURRENT_BINARY_DIR}/html/split)

add_custom_target (doc_tutorial_html_split
DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/html/split)

add_custom_command (
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/tutorial.pdf
DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/tutorial.md
COMMAND
${CMAKE_CURRENT_SOURCE_DIR}/md2pdf
${CMAKE_CURRENT_SOURCE_DIR}/tutorial.md
${CMAKE_CURRENT_BINARY_DIR}/tutorial.pdf)

add_custom_target (doc_tutorial_pdf
DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/tutorial.pdf)

Logical target for all documentation.

add_custom_target (docs
DEPENDS
doc_api
doc_tutorial_html
doc_tutorial_html_split
doc_tutorial_pdf)

Doc Building Docker (`doc/building-docker.md`) Copy Section

Building Seastar in a Docker container Copy Chapter

To build a Docker image:

docker build -t seastar-dev -f docker/dev/Dockerfile .

Building is done with two commands:

$ ./configure.py
$ ninja -C build/release

You can run them inside a container, for example:

$ seabuild() { docker run -v $HOME/seastar/:/seastar -u $(id -u):$(id -g) -w /seastar -t seastar-dev "$@"; }
$ seabuild ./configure.py
$ seabuild ninja -C build/release

Doc Building Dpdk (`doc/building-dpdk.md`) Copy Section

Building with a DPDK network backend Copy Chapter

1. Set up the host to compile DPDK:
   • Ubuntu
     sudo apt-get install -y build-essential linux-image-extra-$(uname -r)
2. Configure the project with DPDK enabled: ./configure.py --mode=release --enable-dpdk
3. Run ninja -C build/release.

To run a Seastar application with the DPDK backend for the native stack, pass the --dpdk-pmd option.

You can also configure DPDK as an external package.

Doc Compatibility (`doc/compatibility.md`) Copy Section

Compatibility

As a library, Seastar aims to maintain backward compatibility
in terms of the source (application code should continue to
build with newer versions of Seastar) and any binary protocols
that Seastar exposes (for example, RPC).

Link compatibility is not maintained - you cannot link an
application built with one version of Seastar with another
version of Seastar.

Language standards

Seastar will support the last two standards approved by the
ISO C++ committee. Each time a new standard is approved, support
for the older of the two is retired. See
Seastar's C++ standard
in the README for the pair currently supported.

Some features may only be enabled for newer dialects.

Platforms

Seastar supports Linux. There is no known minimum kernel version,
but very old kernels might not work. Performance can be significantly
better for newer kernels.

Filesystem implementation quality can have a significant effect on
file I/O performance. XFS is known to be working, ext4 may work well
too. Test your filesystem and kernel versions to be sure.

Patches for new platforms (for example, Windows) are welcome.

Compilers

Seastar supports GCC and Clang. Ports to other compilers are
welcome.

The last two major releases of a compiler are supported (for example,
GCC 13 and GCC 14). Patches to support older versions are welcome,
as long as they don't require onerous compromises.

Deprecation

Occasionally, we discover that we took the wrong approach with
an API. In these cases we will offer a new API and tag the old
API with the [[deprecated]] attribute. The deprecated API will
be removed after a transition period (which can vary depending on
how central the deprecated API is).

Breaking changes

Rarely, we have to make breaking changes. We try to limit those,
but sometimes there is no choice.

To support a transition period for breaking changes, Seastar
offers the Seastar_API_LEVEL CMake variable (and the corresponding
--api-level option to configure.py). An API level selects different
versions of the API. For example:

• Seastar_API_LEVEL=1 selects an old version of the
  server_socket::accept() API that returns a variadic
  future (which is deprecated)
• Seastar_API_LEVEL=2 selects a new version of the
  server_socket::accept() API that returns a non-variadic
  future
• Seastar_API_LEVEL=6 makes futures non-variadic
• Seastar_API_LEVEL=7 unifies CPU scheduling groups and IO priority classes
  "while at it" file_impl API is forced to accept io_intent argument
• Seastar_API_LEVEL=8 changes json_return_type to hold a noncopyable function
  and become a move-only type
• Seastar_API_LEVEL=9 defines the data_sink_impl::put(span)
  as the new and only method to be implemented
• Seastar_API_LEVEL=10 makes co_return and promise.set_value() semantics
  closer to those of plain C++ return regarding how they convert
  expressions used as their arguments to the declared type.

Applications can use an old API_LEVEL during a transition
period, fix their code, and move to the new API_LEVEL.

Old API levels only live for a transition period, so if
you are using an API level below the latest, you should
upgrade quickly.

Note that applications should not refer to the api_vN
sub-namespaces that Seastar defines as part of the API_LEVEL
mechanism; these are internal.

Internal namespace

Identifiers in the seastar::internal namespace are not subject
to source-level compatibility and are subject to change or removal
without notice. In addition, the api_vN sub-namespaces are also
internal.

Accidentally exposed internal identifiers

Some identifiers predate the internal namespace, and are only
exposed accidentally. These can also be removed or changed. Exposed
identifiers are documented using Doxygen, but not all exposed
APIs are documented. In case of doubt, ask on the mailing list.

API Level History

Note: The "Mandatory" column indicates when backward-compatibility
support for the API preceding the new level was removed.

Implementation notes for API levels

API levels are implemented by defining internal sub-namespaces
for each API level: seastar::api_v1, seastar::api_v2, etc. #ifdefs
are used to inline the user-selected API level namespace into the
main seastar namespace, making it visible.

Usually, the old API is implemented in terms of the new API to
avoid code duplication.

Here is an example of the transition from API_LEVEL 1 to 2. The
transition from 2 to 3, and similar transitions, are analogous.

Unconditionally:

• the new API is defined in sub-namespace api_v2

If API_LEVEL is 2:

• api_v2 namespace is inlined into the seastar namespace

If API_LEVEL is 1:

• the old API is defined in sub-namespace api_v1
• api_v1 is implemented in terms of api_v2 to prevent code duplication
• api_v1 namespace is inlined into the seastar namespace

After a transition period:

• everything in api_v1 is dropped
• api_v2 is removed, and its contents are placed in the parent namespace

Doc Contributing (`doc/contributing.md`) Copy Section

Contributing to Seastar

Sending Patches

Seastar follows a patch-submission workflow similar to that of the Linux kernel. Send patches to seastar-dev with a DCO sign-off. Use git send-email to send your patch.

Example:

1. When you commit, use -s in your Git command. This adds a sign-off for the Developer Certificate of Origin.

   You can prefix the commit message with a tag identifying the area of the codebase that the patch addresses:

   text Copy

   git commit -s -m "core: some descriptive commit message"

2. Then send an email to the Google Group:

   text Copy

   git send-email <revision>..<final_revision> --to [email protected]

When replying to patches, use --in-reply-to with the message ID of the original message. If you are sending a new version of a change, use git rebase and then git send-email with an option such as -v2 to indicate that it is the second version.

Testing and Approval

Run test.py and ensure that tests pass at least as well as they did before the patch.

Doc Generator (`doc/generator.md`) Copy Section

Coroutine Generator

Overview Copy Chapter

The generator implementation is based on C++23 proposal P2502R2. It provides
asynchronous coroutine-based value generation with two variants:

• Unbuffered: Zero-copy, one suspension per element
• Buffered: Batches elements, amortizes suspension overhead

Unlike std::generator, operator() is a coroutine returning
std::optional<reference_type>.

Object Relationships and Control Flow Copy Chapter

The generator implementation uses awaiters to manage control flow between producer and consumer coroutines. Each awaiter has multiple entry/exit points:

yield_awaiter lifecycle (4 entry/exit points): Copy Section

1. Created by co_yield value → promise.yield_value() creates awaiter
2. co_await yield_awaiter → calls await_suspend(producer_handle)
3. await_suspend() branches:
   • Direct path: returns consumer_handle (symmetric transfer)
   • Scheduler path: schedules consumer, returns noop_coroutine()
4. await_resume() called:
   • By scheduler (after scheduler path)
   • After symmetric transfer back from consumer

next_awaiter/call_awaiter lifecycle (4 entry/exit points): Copy Section

1. Created by co_await gen() → operator() creates awaiter
2. co_await call_awaiter → calls await_suspend(consumer_handle)
3. await_suspend() branches:
   • Direct path: returns producer_handle (symmetric transfer)
   • Scheduler path: schedules producer, returns noop_coroutine()
4. await_resume() called:
   • By scheduler (after scheduler path)
   • After symmetric transfer back from producer

/* Detailed source-code truncated for AI context efficiency. */

Control Flow Copy Chapter

Producer and consumer transfer control via symmetric transfer when
!need_preempt(). When need_preempt() returns true, the target coroutine
is scheduled via seastar::schedule() and the current coroutine suspends
to noop_coroutine().

The sequence diagrams below show the complete lifecycle of awaiters as
participants. Awaiters are shown with their creation and destruction
points via activate/deactivate markers. Three separate diagrams cover
the different awaiter types and their behaviors.

Unbuffered Generator: yield_awaiter (Zero-Copy) Copy Section

This is the most common case where co_yield passes an rvalue reference.
The promise stores only a pointer to the yielded value (zero-copy).

/* Detailed source-code truncated for AI context efficiency. */

Unbuffered Generator: copy_awaiter (Type Conversion) Copy Section

Used when co_yield passes a const lvalue reference requiring type conversion.
The awaiter creates and stores a copy of the converted value.

/* Detailed source-code truncated for AI context efficiency. */

Buffered Generator: yield_awaiter (Batching) Copy Section

The buffered generator accumulates elements and conditionally suspends.
await_ready() can return true when the buffer has space and no preemption is needed.

/* Detailed source-code truncated for AI context efficiency. */

Unbuffered Generator Copy Chapter

Characteristics Copy Section

• Stores pointer to value in producer's stack frame
• No copies or moves
• One suspension per yielded element

Usage Copy Section

generator<const T&> produce() {
    T value;
    co_yield value;  // Zero-copy: stores pointer only
}

Use when element moves are expensive or latency is critical.

Buffered Generator Copy Chapter

Characteristics Copy Section

• Accumulates elements in a container
• Suspends when buffer full or need_preempt() returns true
• Consumer drains buffer without suspensions

Usage Copy Section

generator<const T&, T, circular_buffer_fixed_capacity<T, 128>> produce() {
    co_yield element;           // Individual elements
    co_yield std::span(data);   // Ranges
}

Use when throughput matters and element moves are cheap.

Buffer Measurement Copy Section

The buffered variant uses a customization point object to check buffer capacity:

// Priority 1: Member function
struct MemoryBuffer {
    bool can_push_more() const {
        return memory_used < memory_limit;
    }
};

// Priority 2: ADL free function
namespace my_ns {
    bool can_push_more(const MyContainer& c);
}

// Priority 3: Default
return container.size() < container.capacity();

Lifetime Guarantees Copy Chapter

Unbuffered Copy Section

When co_yield evaluates an expression producing a glvalue, the object lives
until the coroutine resumes. The promise stores only a pointer.

Buffered Copy Section

Values are moved into the buffer and have independent lifetime.

Exception Handling Copy Chapter

Exceptions in the producer are caught by promise_type::unhandled_exception()
and stored. On the next consumer resumption, await_resume() rethrows the
exception.

Template Parameters Copy Chapter

Single-parameter form Copy Section

generator<int>              // value_type=int, reference=int&&
generator<const string&>    // value_type=string, reference=const string&

Two-parameter form Copy Section

generator<string_view, string>  // Return string_view, store string

Allows proxy reference pattern: producer yields string, consumer receives
string_view.

Performance Copy Chapter

Performance characteristics measured using benchmarks in tests/perf/coroutine_perf.cc.

Benchmark Results Copy Section

Each test generates 100 integers per iteration (Release mode with -O2):

Unbuffered generator:

• One suspension per element (100 suspensions per iteration)
• Zero-copy: stores pointer to value in producer's stack frame
• ~3.1ns per element (308ns / 100)

Buffered generator (using circular_buffer_fixed_capacity):

• Amortized suspensions (~6-7 suspensions per iteration with buffer size 16)
• Moves elements into fixed-capacity buffer (zero heap allocations)
• 35% faster than unbuffered due to amortized suspension overhead
• ~2.0ns per element (200ns / 100)

Important: Container choice is critical for buffered generator performance!

Using std::vector instead of circular_buffer_fixed_capacity results in:

• 2.6x slower performance (524ns vs 200ns)
• 2x more allocations (4 vs 2) due to dynamic memory
• 2x more instructions (13,088 vs 6,259)

The heap allocation overhead completely negates the batching benefit. Always use fixed-capacity containers like circular_buffer_fixed_capacity for best performance.

Choosing Between Variants Copy Section

The choice depends on multiple factors:

Use unbuffered when:

• Element moves are expensive (large objects, non-trivial move constructors)
• Latency is critical (need first element ASAP)
• Memory pressure is a concern (no buffering overhead)
• Elements are naturally references to existing data

Use buffered when:

• Throughput matters more than latency
• Elements are cheap to move (integers, small PODs)
• Producer can generate elements in batches
• Using a fixed-capacity container (avoids heap allocations)

Note: The buffered variant's performance advantage becomes more pronounced with:

• Larger element counts (more amortization)
• Fixed-capacity containers like circular_buffer_fixed_capacity (no allocations)
• Higher element generation cost in the producer
• Natural batching in the data source

Doc Io Properties File (`doc/io-properties-file.md`) Copy Section

Specifying the I/O properties of a system

The I/O properties of a system can be specified as a YAML string, by
using the --io-properties option, or as a YAML file with the
--io-properties-file option.

The expected format starts with a map of sections at the top level.
Currently only disks is supported.

The disks section Copy Chapter

Inside the disks section, the user can specify a list of mount points.

For each mount point, four properties have to be specified (none are
optional):

• read_iops: read IOPS speed of the device
• read_bandwidth: read bandwidth speed of the device
• write_iops: write IOPS speed of the device
• write_bandwidth: write bandwidth speed of the device

Additionally, the following optional properties can be added:

• read_saturation_length: read buffer length to saturate the device throughput
• write_saturation_length: write buffer length to saturate the device throughput
• physical_block_size: override for the physical block size of the device (in bytes).
  This is used as the write alignment to avoid hardware-level read-modify-write operations.
  Some devices misreport their physical block size, so this override can be used to
  specify the correct value

Those quantities can be specified in raw form, or followed with a
suffix (k, M, G, or T).

Example:

disks:
  - mountpoint: /var/lib/some_seastar_app
    read_iops: 95000
    read_bandwidth: 545M
    write_iops: 85000
    write_bandwidth: 510M
    write_saturation_length: 64k

Instead of mountpoint, a list element can contain a mountpoints list with more
than one path. As a result, all the listed mount points share the corresponding
internal I/O queue.

Example:

disks:
  - mountpoints:
      - /var/lib/some_seastar_app/sub_one
      - /var/lib/some_seastar_app/sub_two
    read_iops: 95000
    ...

This configuration is useful, for example, for a set of LVM volumes from one disk,
each mounted at its own path. In that case, different mount points have different
virtual block devices but share the same physical disk, so they need to use one
shared I/O queue.

Doc Io Scheduler (`doc/io-scheduler.md`) Copy Section

The I/O scheduler uses a rate limiter to throttle the amount of data it dispatches to the disk.

Basic math

The scheduler's main equation that models the disk behavior is

$$ \frac {bandwidth_r} {bandwidth_{r_{max}}} +
\frac {bandwidth_w} {bandwidth_{w_{max}}} +
\frac {iops_r} {iops_{r_{max}}} +
\frac {iops_w} {iops_{w_{max}}} \le 1.0 $$

Let's say that

$$ m_b = \frac {bandwidth_{r_{max}}} {bandwidth_{w_{max}}} \ \ m_o = \frac {iops_{r_{max}}} {iops_{w_{max}}} $$

Since

$$ bandwidth_x = \frac {d}{dt} bytes_x \ \ iops_x = \frac {d}{dt} ios_x $$

The main equation turns into

$$ \frac {d}{dt} ( \frac {bytes_r + m_b \times bytes_w} {bandwidth_{r_{max}}} + \frac {ios_r + m_o \times ios_w} {iops_{r_{max}}} ) \le 1.0 $$

Requests are assigned a two-dimensional value called a "ticket": {1, bytes} for reads and {m_o, m_b * bytes} for writes.

The "normalization" operation is defined as

$$ N(ticket) = \frac {ticket_0}{iops_{r_{max}}} + \frac {ticket_1}{bandwidth_{r_{max}}} $$

With that the main equation turns into

$$ \frac {d}{dt} \sum_{ticket} N(ticket) \le 1.0 $$

The time-derivative limitation is then implemented with the token bucket algorithm

Token bucket

The algorithm creates a token bucket with a refill rate of 1.0. Each request carries the fractional token value tokens = N(ticket), with ticket defined above.

The bucket also requires the limit parameter, which is the maximum number of tokens it may hold. This value is calculated using the io_latency_goal parameter and is the number of tokens accumulated during the io_latency_goal duration.

Slowing down the flow

Let's assume we need to reduce the rate at which requests run. This means that we want to

$$ \frac {d}{dt} ( \frac {bytes_r + m_b \times bytes_w} {bandwidth_{r_{max}} \times \alpha} + \frac {ios_r + m_o \times ios_w} {iops_{r_{max}} \times \beta} ) \le 1.0 \ \ \alpha \le 1.0 \ \beta \le 1.0 $$

There are many ways of selecting which of IOPS or bandwidth to reduce, the "general" slowing down may assume that

$$ \alpha = \beta = \frac {1}{\gamma} , \ \ \ \gamma \ge 1.0 $$

The main equation then turns into

$$ \frac {d}{dt} \sum_{ticket} \gamma \times N(ticket) \le 1.0 $$

This means that we can multiply the request cost (in tokens) by a number greater than 1.0.

Detecting the slowdown

Let's say that

d_i -- the number of requests dispatched at tick i
p_i -- the number of requests processed by the disk at tick i
c_i -- the number of requests completed by the reactor loop at tick i

We can observe d_i and c_i in the dispatcher, but not the p_i, because we don't have direct access to disks' queues

After n ticks we have

D_n -- total number of requests dispatched,
P_n -- total number of requests processed,
C_n -- total number of requests completed,

$$ D_n = \sum_{i=0}^n d_i $$

$$ P_n = \sum_{i=0}^n p_i $$

$$ C_n = \sum_{i=0}^n c_i $$

• Disk cannot process more than it was dispatched, but it can process less "accumulating" a queue

$$ d_i \ge p_i \Rightarrow D_n \ge P_n$$

• Reactor cannot complete more than it was processed by disk either

$$ p_i \ge c_i \Rightarrow P_n \ge C_n $$

Note that D_n > P_n means that disk is falling behind. Our goal is to make sure the disk doesn't do it and doesn't accumulate the queue, i.e. D_n = P_n, but we cannot observe P_n directly, only C_n.

Next

$$ D_n - P_n = Qd_n \ge 0 $$

$$ P_n - C_n = Qc_n \ge 0 $$

$$ D_n - C_N = Qd_n + Qc_n \ge 0 $$

Qd_n is the queue accumulated in disk. We try to avoid it. Qc_n is the accumulated delta between processed (by disk) and completed (by reactor) requests. We cannot reliably say that it goes to zero over time, because there's always some unknown amount of processed but not yet completed requests. So the above formula contains two unknowns and we cannot solve it. Let's try the other way

$$ \frac {D_n} {P_n} = Rd_n \ge 1 $$

$$ \frac {P_n} {C_n} = Rc_n \ge 1 $$

$$ \frac {D_n} {C_n} = Rd_n \times Rc_n \ge 1 $$

Rd_n is the ratio of dispatched to processed requests. If it's 1, we're OK, if it's greater, disk is accumulating the queue, we try to avoid it. Rc_n is the ratio between processed (by disk) and completed (by reactor) requests. It has a not immediately apparent, but very pretty property

$$ \lim_{n\to\infty} Rc_n = \lim_{n\to\infty} \frac {P_n} {C_n} = \lim_{n\to\infty} \frac {C_n + Qc_n} { C_n} = \lim_{n\to\infty} (1 + \frac {Qc_n} {C_n} ) = 1 + \lim_{n\to\infty} \frac {Qc_n} {C_n}$$

Qc_n does not grow over time. It is nonzero but bounded above by some value. Therefore,

$$ \lim_{n\to\infty} Rc_n = 1 $$

$$ \lim_{n\to\infty} \frac {D_n} {C_n} = \lim_{n\to\infty} ( Rd_n \times Rc_n ) = \lim_{n\to\infty} Rd_n $$

In other words, we can determine whether the disk is accumulating a queue by observing the dispatched-to-completed ratio (using completed, not processed) for long enough.

Doc Io Tester (`doc/io-tester.md`) Copy Section

I/O Tester utility Copy Chapter

The I/O Tester utility, io_tester, generates a user-defined I/O pattern
spanning one or more shards and is designed to simulate the I/O behavior
of a complex Seastar application.

Running I/O tester

I/O tester takes the same options as Seastar, and those options may be used
to test the behavior of I/O under the circumstances established by those
options. For instance, one may adjust the --task-quota-ms option to see
if that affects higher percentile latencies.

Aside from the usual Seastar options, I/O tester accepts the following options:

• duration: for how long to run the evaluation,
• storage: a directory, a block device, or /dev/null where to execute the test (a directory must be on XFS),
• conf: the path to a YAML file describing the evaluation,
• keep-files: a flag that preserves test files so the next run may reuse them,
• sched-groups: optional path to a YAML file describing the scheduling group hierarchy (see Scheduling groups below).

When storage points at /dev/null, no real file is created and reads/writes complete immediately; useful to measure framework overhead. When storage is a block device, the device is partitioned equally between shards and each job is given a non-overlapping range within its shard's area.

Describing the evaluation

The evaluation is described in a YAML file that contains multiple classes.
Each class spans jobs of similar characteristics in different shards and (for now)
all jobs run concurrently.

The YAML file contains a list of maps where each element of the list describes a class.
A class has some properties that are common to all elements of the class, and a nested map
that contains properties of a job (a class instance on a shard).

For example:

- name: big_writes
  type: seqread
  shards: all
  shard_info:
    parallelism: 10
    reqsize: 256kB
    shares: 10
    think_time: 0

- name: vectorized_writes
  type: randwrite
  shards: all
  shard_info:
    parallelism: 10
    reqsize: 64kB
    vectorized: true
    iov_count: 4        # 4 segments of 16kB each
    shares: 10

• name: mandatory property, a string that identifies jobs of this class
• type: mandatory property, one of seqread, overwrite, randread, randwrite, append, cpu, or unlink
• shards: mandatory property, either the string "all" or a list of shards where this class should place jobs.
• data_size: optional property, used to divide the available disk space between workloads. Each shard inside the workload uses its portion of the assigned space. If not specified 1GB is used (0 for append jobs, since the file is grown by the workload itself).
• files_count: property required only for the unlink job class. It specifies, per shard, the number of files to create during startup and unlink during evaluation.

NOTE: the actual file size is always aligned to 1MB.
NOTE: if not properly aligned, the extent allocation size hint is aligned to 128 kB by Seastar.

The properties under shard_info represent properties of the job that will
be replicated to each shard. All properties under shard_info are optional; defaults are used when they are not specified.

• parallelism: the number of parallel requests this job will generate on a specific shard. Requests can be either active or thinking (see think_time)
• limit: the maximum number of requests to send in this job. If not set, job sends requests throughout the whole duration
• rps: the requests-per-second rate to apply to sending fibers. If unset, each fiber sends a new request as soon as the previous one completes
• batch: the number of requests sent at a time when the rps mode is used. It's 1 by default
• reqsize : (I/O loads only) the size of requests generated by this job
• bandwidth: (I/O loads only) per-class I/O bandwidth cap, in bytes per second. When set, the scheduling group's IO bandwidth limit is updated on shard 0 to this value.
• shares : how many shares requests in this job will have in the scheduler
• class: name of the scheduling class this job should run in (used instead of shares). The named class is either taken from the --sched-groups configuration or implicitly created from another job's name.
• think_time: how long to wait before submitting another request in this job once one finishes.
• think_after: if non-zero, alternate between active and thinking phases. The job starts in active mode for think_after, then pauses for think_time, then becomes active again for think_after, and so on. When think_after is 0 (the default), think_time is applied after every request as a fixed inter-request pause.
• execution_time: (cpu loads only) for how long to execute a CPU loop
• vectorized: (I/O loads only) if true, use scatter-gather I/O with multiple iovec segments per request
• iov_count: (I/O loads only) number of iovec segments when vectorized is true. The reqsize is split into iov_count uniform segments. reqsize must be evenly divisible by iov_count. Each segment must be aligned to (and no smaller than) 4096 bytes when reqsize is at least 4096, otherwise to 512 bytes.

In addition to shard_info, a class may carry an options map. All keys in it are optional:

• dsync: if true, the test file is opened with O_DSYNC.
• sleep_type: how rps-mode fibers wait between request submissions. One of:
  • lowres (default) - sleep on the lowres clock,
  • steady - sleep on the steady clock (more precise, more expensive),
  • busyloop - busy-loop with yield() until the deadline.
• pause_distribution: distribution of the inter-request pause in rps mode. One of uniform (default, fixed 1s/rps pause) or poisson (exponential distribution with the same mean).
• extent_allocation_size_hint: hint for the allocation size of file extents. If not specified, the file size is used as the hint. If not properly aligned, the hint is aligned to 128 kB by Seastar.
• sloppy_size_hint: when set, opens the file with sloppy_size enabled and uses this value as the size hint.
• fallocate: if true, the test file is fallocate()-d after being truncated to its target size, so blocks are pre-allocated before the workload starts.

Scheduling groups

By default, each class implicitly creates a scheduling group named after its name, with shares taken from shard_info. A more elaborate hierarchy can be described in a separate YAML file passed via --sched-groups. That file is a list of maps with these properties:

• name: the name of the scheduling group.
• shares: the share weight of this group.
• parent: optional. If present, the group identified by name becomes a child of the group named here, and the parent is automatically promoted to a scheduling supergroup.

A class can be attached to one of these groups via the class property in shard_info (in which case shares in the class is ignored). Two classes referencing the same class share a scheduling group; classes that do not set class get their own implicit group.

For example:

- name: parent_group
  shares: 1000

- name: child_a
  parent: parent_group
  shares: 100

- name: child_b
  parent: parent_group
  shares: 200

Example output

The evaluation result is emitted to standard output as a YAML document: a sequence of per-shard maps, each containing one entry per class plus a statistics block.

---
- shard: 0
  big_writes:
    throughput: 436556 # kB/s
    IOPS: 1665
    latencies: # usec
      average: 5847
      p0.5: 2678
      p0.95: 13029
      p0.99: 20835
      p0.999: 246090
      max: 450785
    stats:
      total_requests: 16650
      io_queue_total_exec_sec: 95.1
      io_queue_total_delay_sec: 2.3
      io_queue_total_operations: 16650
      io_queue_starvation_time_sec: 0.01
      io_queue_consumption: 4259840000
      io_queue_adjusted_consumption: 4259840000
      io_queue_activations: 1
      reactor_aio_outsizes: 0
      disk_queue_length:
        p0.5: 8
        p0.9: 12
        max: 24
      file_size_overflows: 4
  statistics:
    aio_retries: 0
...

For cpu jobs only throughput is reported; for unlink jobs only IOPS, latencies (average and max) and total_requests are reported.

Future

Some ideas for extending I/O tester:

• allow properties like think time, request size, etc, to be specified as distributions instead of a fixed number
• allow classes to have class-wide properties. For instance, we could define a class with parallelism of 100, and distribute those 100 requests over all shards in which this class is placed
• allow some jobs to be executed sequentially in relationship to others, so we can have preparation jobs.
• support other types, like sync, etc.
• provide functionality similar to diskplorer.

Doc Io Tracing (`doc/io-tracing.md`) Copy Section

I/O Tracing

Seastar includes optional LTTng-UST tracepoints in the I/O scheduler that
allow low-overhead recording of I/O request lifecycle events. The
tracepoints must be explicitly enabled at build time via
-DSeastar_LTTNG=ON, but have near-zero cost (~3 ns per tracepoint) when
tracing is not active.

Prerequisites Copy Chapter

Install the LTTng userspace tracer:

# Debian/Ubuntu
sudo apt install lttng-tools liblttng-ust-dev

# Fedora/RHEL
sudo dnf install lttng-tools lttng-ust-devel

Build Seastar with LTTng support enabled:

cmake -DSeastar_LTTNG=ON ...

Verify that the tracepoints are present:

readelf -n ./your_app | grep "Provider: seastar_io"

You should see entries for io_queue_queued, io_queue_dispatched,
io_queue_completed, and io_queue_cancelled.

Available Tracepoints Copy Chapter

All tracepoints belong to the seastar_io provider.

Field Descriptions Copy Section

• dev_id — I/O queue device identifier (corresponds to a mount point)
• req_id — Unique request identifier (pointer value), stable across
  queued → dispatched → completed lifecycle
• direction — 0 = write, 1 = read
• pclass — Priority/scheduling class identifier
• offset — File offset in bytes
• length — Request size in bytes

Recording a Trace Copy Chapter

Using LTTng (recommended for production recording) Copy Section

# Create a tracing session
lttng create io-session --output=/tmp/io-trace

# Enable all Seastar I/O tracepoints
lttng enable-event -u 'seastar_io:*'

# Start recording
lttng start

# ... run your Seastar application ...

# Stop recording
lttng stop

# View the trace as text
lttng view

# Or use babeltrace2 for more control
babeltrace2 /tmp/io-trace

# Destroy the session
lttng destroy io-session

Attaching to a Running Application Copy Section

LTTng-UST can attach to an already running Seastar application
without restarting it:

# Assuming your app is already running...
lttng create live-session
lttng enable-event -u 'seastar_io:*'
lttng start

# Events are now being recorded from the running app.
# No restart, no signal, no interruption.

lttng stop
lttng view
lttng destroy live-session

The only requirement is that lttng-sessiond must have been running
before the application started (so the lttng-ust library could
register the tracepoints at startup). If lttng-sessiond was not
running when the app launched, the tracepoints will not be
discoverable. In practice, ensure lttng-sessiond --daemonize is
part of your system startup.

Example Output Copy Section

[17:33:21.510589059] (+?.?????????) host seastar_io:io_queue_queued: { cpu_id = 0 }, { dev_id = 0, req_id = 0x601000175C80, direction = 0, pclass = 0, offset = 0, length = 4096 }
[17:33:21.510592245] (+0.000003186) host seastar_io:io_queue_dispatched: { cpu_id = 0 }, { dev_id = 0, req_id = 0x601000175C80 }
[17:33:21.518178549] (+0.007586304) host seastar_io:io_queue_completed: { cpu_id = 0 }, { dev_id = 0, req_id = 0x601000175C80 }
[17:33:21.521191175] (+0.003012626) host seastar_io:io_queue_queued: { cpu_id = 0 }, { dev_id = 0, req_id = 0x601000175C80, direction = 1, pclass = 0, offset = 0, length = 4096 }
[17:33:21.521194281] (+0.000003106) host seastar_io:io_queue_dispatched: { cpu_id = 0 }, { dev_id = 0, req_id = 0x601000175C80 }
[17:33:21.521265876] (+0.000071595) host seastar_io:io_queue_completed: { cpu_id = 0 }, { dev_id = 0, req_id = 0x601000175C80 }

Here you can see a write (direction = 0) at offset 0 taking 7.5 ms
(queued → completed), followed by a read (direction = 1) at the same
offset taking only 71 µs. The delta in parentheses shows the time since
the previous event.

Using perf (quick ad-hoc inspection) Copy Section

The tracepoints are also exposed as SDT probes:

# List available probes
perf list sdt | grep seastar_io

# Record all I/O events
perf record -e 'sdt_seastar_io:*' ./your_app

# View results
perf script

Using bpftrace (live analysis / filtering) Copy Section

# Count I/O requests per second
bpftrace -e 'usdt:./your_app:seastar_io:io_queue_queued { @ops = count(); }
             interval:s:1 { print(@ops); clear(@ops); }'

# Histogram of I/O request sizes
bpftrace -e 'usdt:./your_app:seastar_io:io_queue_queued {
    @sizes = hist(arg6);
}'

# Measure queue latency (time from queued to dispatched)
bpftrace -e '
usdt:./your_app:seastar_io:io_queue_queued { @start[arg1] = nsecs; }
usdt:./your_app:seastar_io:io_queue_dispatched /@start[arg1]/ {
    @queue_lat = hist(nsecs - @start[arg1]);
    delete(@start[arg1]);
}'

Per-Shard Event Filtering Copy Chapter

Seastar runs one shard per thread, and LTTng-UST records events into
per-thread ring buffers. Each event carries the virtual thread ID
(vtid) in its context, which maps 1:1 to a shard. Seastar names
its reactor threads reactor-N where N is the shard number, so
resolving the mapping is straightforward.

Adding Thread Context Copy Section

To include thread identifiers in the trace output:

lttng add-context -u -t vtid -t procname

After this, every event will carry the OS thread ID and the thread
name (e.g., reactor-0, reactor-3). Example output:

[17:33:21.510589059] host seastar_io:io_queue_queued: { cpu_id = 0 }, { vtid = 12345, procname = "reactor-0" }, { dev_id = 0, req_id = 0x601000175C80, direction = 0, ... }
[17:33:21.510592245] host seastar_io:io_queue_dispatched: { cpu_id = 1 }, { vtid = 12346, procname = "reactor-1" }, { dev_id = 0, req_id = 0x601000180A00 }

Filtering Events by Shard Copy Section

With babeltrace2, filter to a specific shard by thread name:

# Show only shard 0 events
babeltrace2 /tmp/io-trace --filter 'procname == "reactor-0"'

# Show only shard 2 events (using vtid if known)
babeltrace2 /tmp/io-trace --filter 'vtid == 12347'

With lttng enable-event you can restrict recording to specific threads
from the start (reduces trace volume in production):

# Only trace shard 0 (by thread name, requires lttng 2.13+)
lttng enable-event -u 'seastar_io:*' --filter 'procname == "reactor-0"'

Resolving Shard ↔ Thread Mapping Copy Section

If you didn't add procname context, the mapping can be reconstructed:

# While the Seastar app is running, dump thread names:
ls /proc/<pid>/task/*/comm
# Output: reactor-0, reactor-1, ...

# Or from within the trace itself: vtid values are consistent, so
# a single lookup at capture time is enough.

No explicit shard_id field is emitted in tracepoint payloads — this
avoids bloating every event with redundant data since the thread→shard
mapping is stable for the lifetime of the process.

Note: Shard 0 retains the original process name (possibly truncated
to 15 characters), while shards 1 and above are named reactor-1,
reactor-2, etc. When filtering, account for this by using vtid
rather than procname for shard 0, or simply note that the "non-reactor"
thread name is shard 0.

Analyzing Traces Copy Chapter

With babeltrace2 Copy Section

# Print all events with timestamps (same format as lttng view)
babeltrace2 /tmp/io-trace

# Filter for a specific device
babeltrace2 /tmp/io-trace --filter 'dev_id == 0'

# Convert to JSON for scripting
babeltrace2 --output-format=json /tmp/io-trace > trace.json

Example babeltrace2 output (same format as lttng view):

[17:33:21.521875455] (+0.000609579) host seastar_io:io_queue_queued: { cpu_id = 0 }, { dev_id = 0, req_id = 0x601000175C80, direction = 0, pclass = 0, offset = 0, length = 4096 }
[17:33:21.521879041] (+0.000003586) host seastar_io:io_queue_dispatched: { cpu_id = 0 }, { dev_id = 0, req_id = 0x601000175C80 }
[17:33:21.521960435] (+0.000081394) host seastar_io:io_queue_completed: { cpu_id = 0 }, { dev_id = 0, req_id = 0x601000175C80 }

Computing I/O Latencies Copy Section

Since LTTng records precise timestamps for each event, latencies are
computed by matching req_id values:

• Queue latency = dispatched.timestamp - queued.timestamp
• Execution latency = completed.timestamp - dispatched.timestamp
• Total latency = completed.timestamp - queued.timestamp

With Trace Compass (GUI) Copy Section

Trace Compass can open CTF
traces produced by LTTng and provides timeline visualization, statistics,
and custom analysis views.

Overhead Copy Chapter

• When not tracing: ~3 ns per tracepoint (single branch, predicted not-taken)
• When actively recording: ~100–300 ns per event (userspace ring buffer
  write, no kernel transition)
• At 100K IOPS with 3 events per I/O: ~30–90 ms of CPU per second (<1% of one core)

Building Without LTTng Copy Chapter

By default (-DSeastar_LTTNG=OFF), Seastar builds without tracepoints
(zero overhead). To enable tracing, pass -DSeastar_LTTNG=ON — this
requires lttng-ust-devel to be installed and will fail at configure
time if the library is not found. The SEASTAR_HAVE_LTTNG_UST
preprocessor macro indicates whether tracing is compiled in.

Doc Lambda Coroutine Fiasco (`doc/lambda-coroutine-fiasco.md`) Copy Section

The Lambda Coroutine Fiasco

Lambda coroutines and Seastar APIs that accept continuations interact badly. This
document explains the bad interaction and how it is mitigated.

Lambda coroutines revisited Copy Chapter

A lambda coroutine is a lambda function that is also a coroutine due
to the use of the coroutine keywords (typically co_await). A lambda
coroutine is notionally translated by the compiler into a struct with a
function call operator:

[captures] (arguments) -> seastar::future<> {
    body
    co_return result
}

becomes (more or less)

struct lambda {
    captures;
    seastar::future<> operator()(arguments) const {
        body
    }
};

Lambda coroutines and coroutine argument capture Copy Chapter

Coroutines, like lambdas, can capture variables from their enclosing
scope. Additionally, coroutines capture their arguments, which can occur
by value or by reference, depending on the argument's declaration.

The lambda's captures, however, are captured by reference. To understand why,
consider that the coroutine translation process notionally transforms a member function
(lambda::operator()) to a free function:

// before
seastar::future<> lambda::operator()(arguments) const;

// after
seastar::future<> lambda_call_operator(const lambda& self, arguments);

This transform means that the lambda structure, which contains all the captured variables,
is itself captured by the coroutine by reference.

Interaction with Seastar APIs accepting continuations Copy Chapter

Consider a Seastar API that accepts a continuation, such as
seastar::future::then(Func continuation). The behavior
is that continuation is moved or copied into a private memory
area managed by then(). Sometime later, the continuation is
executed (Func::operator()) and the memory area is freed.
Crucially, the memory area is freed as soon as Func::operator()
returns, which can be before the future returned by it becomes
ready. However, the coroutine can access the lambda captures
stored in this memory area after the future is returned and before
it becomes ready. This is a use-after-free.

Solution (C++23 and up) Copy Chapter

With "deducing this", the value category of the implicit object
parameter to the lambda, which contains the captures, can be changed.

A lambda of the form

[captures...] (this auto, arguments...)

is transformed to a function call operator of the form

seastar::future<> lambda::operator()(this auto, arguments...);

which in turn is equivalent to the free function

seastar::future<> lambda_call_operator(lambda, arguments...);

Since the function accepts the lambda struct containing the
capture groups by value, the capture groups are moved to the
coroutine frame and are not tied to the lifetime of the expression
that created the lambda.

Solution (pre C++23) Copy Chapter

The solution is to avoid copying or moving the lambda into
the memory area managed by seastar::future::then(). Instead,
the lambda spends its life as a temporary. We then rely on C++
temporary lifetime extension rules to extend its life until the
returned future is ready, at which point the captures can no longer
be accessed.

co_await seastar::yield().then(seastar::coroutine::lambda([captures] () -> future<> {
        co_await seastar::coroutine::maybe_yield();
        // Can use `captures` here safely.
    }));

seastar::coroutine::lambda is very similar to std::reference_wrapper (the
only difference is that it works with temporaries); it can be safely moved to
the memory area managed by seastar::future::then() since it's only used
to call the real lambda, and then is safe to discard.

Alternative solution (pre-C++23) when lifetime extension cannot be used Copy Chapter

If the lambda coroutine is not awaited immediately, we cannot rely on
lifetime extension and so we must name the coroutine and use std::ref() to
refer to it without copying it from the coroutine frame:

auto a_lambda = [captures] () -> future<> {
        co_await seastar::coroutine::maybe_yield();
        // Can use `captures` here safely.
    };
    auto f = seastar::yield().then(std::ref(a_lambda));
    co_await std::move(f);

Doc Mini Tutorial (`doc/mini-tutorial.md`) Copy Section

Futures and promises Copy Chapter

A future is a result of a computation that may not be available yet.
Examples include:

• a data buffer that we are reading from the network
• the expiration of a timer
• the completion of a disk write
• the result computation that requires the values from
  one or more other futures.

A promise is an object or function that provides you with a future,
with the expectation that it will fulfill the future.

Promises and futures simplify asynchronous programming since they decouple
the event producer (the promise) and the event consumer (whoever uses the
future). Whether the promise is fulfilled before the future is consumed,
or vice versa, does not change the outcome of the code.

Consuming a future Copy Chapter

You consume a future by using its then() method, providing it with a
callback (typically a lambda). For example, consider the following
operation:

future<int> get();   // promises an int will be produced eventually
future<> put(int);   // promises to store an int

future<> f() {
    return get().then([] (int value) {
        return put(value + 1).then([] {
            std::cout << "value stored successfully\n";
        });
    });
}

Here, we initiate a get() operation, requesting that when it completes, a
put() operation will be scheduled with an incremented value. We also
request that when the put() completes, some text will be printed out.

Chaining futures Copy Chapter

If a then() lambda returns a future (call it x), then that then()
will return a future (call it y) that will receive the same value. This
removes the need for nesting lambda blocks; for example the code above
could be rewritten as:

future<int> get();   // promises an int will be produced eventually
future<> put(int);   // promises to store an int

future<> f() {
    return get().then([] (int value) {
        return put(value + 1);
    }).then([] {
        std::cout << "value stored successfully\n";
    });
}

Loops Copy Chapter

Loops are achieved with a tail call; for example:

future<int> get();   // promises an int will be produced eventually
future<> put(int);   // promises to store an int

future<> loop_to(int end) {
    return get().then([end] (int value) {
        if (value == end) {
            return make_ready_future<>();
        }
        return put(value + 1).then([end] {
            return loop_to(end);
        });
    });
}

The make_ready_future() function returns a future that is already
available --- corresponding to the loop termination condition, where
no further I/O needs to take place.

Under the hood Copy Chapter

When the loop above runs, both then method calls execute immediately
--- but without executing the bodies. What happens is the following:

1. get() is called, initiates the I/O operation, and allocates a
   temporary structure (call it f1).
2. The first then() call chains its body to f1 and allocates
   another temporary structure, f2.
3. The second then() call chains its body to f2.

Again, all this runs immediately without waiting for anything.

After the I/O operation initiated by get() completes, it calls the
continuation stored in f1, calls it, and frees f1. The continuation
calls put(), which initiates the I/O operation required to perform
the store, and allocates a temporary object f12, and chains some glue
code to it.

After the I/O operation initiated by put() completes, it calls the
continuation associated with f12, which simply tells it to call the
continuation associated with f2. This continuation simply calls
loop_to(). Both f12 and f2 are freed. loop_to() then calls
get(), which starts the process all over again, allocating new versions
of f1 and f2.

Handling exceptions Copy Chapter

If a .then() clause throws an exception, the scheduler will catch it
and cancel any dependent .then() clauses. If you want to trap the
exception, add a .then_wrapped() clause at the end:

future<buffer> receive();
request parse(buffer buf);
future<response> process(request req);
future<> send(response resp);

future<> f() {
    return receive().then([] (buffer buf) {
        return process(parse(std::move(buf)));
    }).then([] (response resp) {
        return send(std::move(resp));
    }).then([] {
        f();
    }).then_wrapped([] (auto&& f) {
        try {
            f.get();
        } catch (std::exception& e) {
            // your handler goes here
        }
    });
}

The previous future is passed as a parameter to the lambda, and its value can
be inspected with f.get(). When the future's get() method is called,
it will rethrow the exception that aborted processing, and you can
then apply any needed error handling. It is essentially a transformation of

buffer receive();
request parse(buffer buf);
response process(request req);
void send(response resp);

void f() {
    try {
        while (true) {
            auto req = parse(receive());
            auto resp = process(std::move(req));
            send(std::move(resp));
        }
    } catch (std::exception& e) {
        // your handler goes here
    }
}

Note, however, that the .then_wrapped() clause will be scheduled whether or
not an exception occurs. Therefore, the mere fact that .then_wrapped() is
executed does not mean that an exception was thrown. Only the execution of the
catch block can guarantee that.

This is shown below:

future<my_type> receive();

void f() {
    receive().then_wrapped([] (future<my_type> f) {
        try {
            my_type x = f.get();
            return do_something(x);
        } catch (std::exception& e) {
            // your handler goes here
        }
    });
}

Setup notes Copy Section

Seastar is a high-performance framework tuned for the best
performance by default. As such, it favors polling over interrupt-driven
operation. The assumption is that applications written for Seastar will be
busy handling 100,000 IOPS and beyond. Polling means that each of our
cores will consume 100% CPU even when they have no work.

Doc Native Stack (`doc/native-stack.md`) Copy Section

Seastar Native TCP/IP Stack Copy Chapter

Seastar comes with a native, sharded TCP/IP stack. It is usually used with the DPDK environment, but vhost drivers are also available for testing in a development environment.

To enable the native network stack, pass the --network-stack native option to a Seastar application.

To test the native stack without DPDK, install and start the libvirt daemon. This creates a bridge device named virbr0, to which Seastar will connect.

Seastar's vhost driver needs a tap device. The scripts/tap.sh script sets up a tap device and binds it to virbr0:

$ sh ./scripts/tap.sh
Set 'tap0' nonpersistent
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.5254008be729	no		tap0
							virbr0-nic
virbr0: flags=4163  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:8b:e7:29  txqueuelen 1000  (Ethernet)
        RX packets 384938  bytes 21866184 (20.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 547098  bytes 2508723098 (2.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

You can now run a Seastar application; for example, the HTTP server:

$ ./build/release/apps/httpd/httpd --network-stack native
DHCP sending discover
DHCP Got offer for 192.168.122.18
DHCP sending request for 192.168.122.18
DHCP Got ack on request
DHCP  ip: 192.168.122.18
DHCP  nm: 255.255.255.0
DHCP  gw: 192.168.122.1
Seastar HTTP server listening on port 10000 ...

You can now ping the IP address shown (192.168.122.18) or connect to it:

$ ping 192.168.122.18
PING 192.168.122.18 (192.168.122.18) 56(84) bytes of data.
64 bytes from 192.168.122.18: icmp_seq=1 ttl=64 time=0.160 ms
64 bytes from 192.168.122.18: icmp_seq=2 ttl=64 time=0.110 ms
64 bytes from 192.168.122.18: icmp_seq=3 ttl=64 time=0.116 ms
64 bytes from 192.168.122.18: icmp_seq=4 ttl=64 time=0.112 ms
64 bytes from 192.168.122.18: icmp_seq=5 ttl=64 time=0.093 ms
64 bytes from 192.168.122.18: icmp_seq=6 ttl=64 time=0.108 ms
^C
--- 192.168.122.18 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 4999ms
rtt min/avg/max/mdev = 0.093/0.116/0.160/0.023 ms

$ curl http://192.168.122.18:10000/
"hello"

Doc Network Configuration (`doc/network-configuration.md`) Copy Section

Network Configuration Copy Chapter

To support multiple network devices, Seastar uses a YAML-based network configuration
format. It contains a list of network devices with their IP parameters and, when DPDK
is used, optional hardware parameters. Provide the configuration on the command line
with --net-config or in a file with --net-config-file.

DPDK access Copy Section

A network device (called a port in DPDK) can be accessed by either its port index (the
zero-based device index shown by dpdk-setup.sh) or its PCI address (shown by tools
such as lspci and lshw).

Example configuration with a PCI address:

eth0: {pci_address: 0000:06:00.0, ip: 192.168.100.10, gateway: 192.168.100.1, netmask: 255.255.255.0 }

Example configuration with a port index:

eth0: {port_index: 0, ip: 192.168.100.10, gateway: 192.168.100.1, netmask: 255.255.255.0 }

The device name (eth0 above) is not used by DPDK itself; it is retained only for
configuration consistency. Hardware configuration must use the same identifier for
all network devices. For example, if pci_address is specified for one device,
port_index cannot be specified for any other device.

Non-DPDK access Copy Chapter

When neither pci_address nor port_index is defined, Seastar assumes non-DPDK access
provided by the libvirt daemon (see the native stack documentation):

virbr0: { ip: 192.168.100.10, gateway: 192.168.100.1, netmask: 255.255.255.0 }

Other hardware related options Copy Chapter

Other optional hardware-related settings are listed below. Some apply to both DPDK and non-DPDK modes.

• lro (large receive offload), Boolean, default: true
• tso (TCP segmentation offload), Boolean, default: true
• ufo (UDP fragmentation offload), Boolean, default: true
• hw-fc (hardware flow control), Boolean, default: true
• csum-offload (IP checksum offload), Boolean, default: true
• ring-size (device ring buffer size), unsigned integer, default: 256; libvirt only
• event-index (VIRTIO_RING_F_EVENT_IDX support), Boolean, default: true; libvirt only

DHCP Copy Chapter

IP configuration can use either ip, gateway, and netmask (as shown above) or DHCP with dhcp: true:

eth0: {pci_address: 0000:06:00.0, dhcp: true}

DHCP can be selected per network device. For example, you can use DHCP for eth0 and specify ip, netmask, and gateway for eth1.

Multiple devices Copy Chapter

To configure multiple devices, provide a comma-separated YAML map:

{virbr0: { ip: 192.168.100.10, gateway: 192.168.100.1, netmask: 255.255.255.0 } , virbr1: { dhcp: true } }

Doc Network Connection Load Balancing (`doc/network-connection-load-balancing.md`) Copy Section

Motivation

In sharded systems like Seastar, it is important for work to be
distributed equally between all shards to achieve maximum performance
from the system. Networking subsystem has its part in distributing work
equally. For instance, if all connections on a server are served by a
single shard, the system will run at the speed of that
one shard and all other shards will be underutilized.

Common ways to distribute work received over the network between shards

Two common ways to distribute work between shards are:

• do the work at a shard that received it
• choose the shard that does the actual work based on the data being processed
  (one way to do it is to hash(data) % smp_count = shard,
  another way is to bind shards to different server addresses)

Load Balancing

These two approaches require different strategies for distributing connections
between shards. The first works best if each CPU has the
same number of connections (assuming each connection gets the same amount of
work). The second works best if data arrives at the shard where
it is going to be processed and actual connection distribution does
not matter.

Seastar's POSIX stack supports both strategies. Choose one by specifying the load-balancing algorithm in
the listen_options provided to the reactor::listen() call. Available options
are:

• load_balancing_algorithm::connection_distribution

  Ensures that a new connection is placed on the shard with the fewest
  connections of the same type.

• load_balancing_algorithm::port

  Destination shard is chosen as a function of client's local port:
  shard = port_number % num_shards. This allows a client to make sure that
  a connection will be processed by a specific shard by choosing its local
  port accordingly (the knowledge about amount of shards in the server is
  needed and can be negotiated by different channel).

• load_balancing_algorithm::fixed

  Destination shard is statically configured in listen_options::fixed_cpu. This
  allows a client to make sure that a connection to a server address will be
  established in a specific shard, without any further negotiations.

Doc Prometheus (`doc/prometheus.md`) Copy Section

The Prometheus Protocol

Seastar supports the Prometheus protocol for metrics reporting.
Supported exposition formats are the 0.0.4 text and protocol buffer formats.

For more information about these formats, see the Prometheus documentation.

By default, Seastar listens on port 9180 on localhost.

See the Seastar configuration documentation on how to change the default configuration.

Seastar uses the protocol buffer format only when it is enabled in the server
configuration and the request's Accept header contains a media type beginning with
application/vnd.google.protobuf;; otherwise, it uses the text format. Therefore,
opening http://localhost:9180/metrics/ in a browser returns a text representation
of the metrics with their documentation.

Starting with Prometheus 2.0, the binary protocol is no longer supported.
While Seastar still supports the binary protocol, that support may be deprecated in a future release.

Querying a subset of the metrics Copy Chapter

Seastar supports querying for a subset of the metrics by their names and labels.

Filtering is recommended when you only need a subset of the available metrics, especially
in systems with many metrics. Filtering reduces the processing overhead on the Seastar
application, results in smaller HTTP responses, and decreases the load on your Prometheus
server when scraping and storing metrics.

Filtering by a metric name Copy Section

Use the __name__ query parameter to select according to a metric name or a prefix.

For example, to get all HTTP metrics, open
http://localhost:9180/metrics?__name__=http*. Note the asterisk following http.
Filtering by name only supports prefix matching.

To query for only the HTTP requests-served metric, open http://localhost:9180/metrics?__name__=httpd_requests_served.

You can use either the full metric name as it appears in the output (e.g., seastar_httpd_requests_served)
or the name without the prefix (e.g., httpd_requests_served). Both forms work identically.

Multiple name filters

You can specify multiple __name__ parameters to query for several specific metrics at once.
A metric is included if it matches any of the specified names.

For example, to get both the HTTP requests and connections metrics:
http://localhost:9180/metrics?__name__=httpd_requests_served&__name__=httpd_connections_total

This also works with prefix matching:
http://localhost:9180/metrics?__name__=httpd_requests*&__name__=httpd_connections*

Filtering by a label value Copy Section

The Prometheus protocol uses labels to differentiate the characteristics of the thing that is being measured.
For example, in Seastar, it is common to report each metric per shard and add a shard label to the metric.

You can filter by any label using regular expressions. If you use multiple labels in your query, all conditions must be met.
A missing label is considered an empty string. The expression should match the entire label value;
to match a missing label, you can use label= or label=^$.

Here are a few examples:

To return all metrics from shard 1 or shard 0:
http://localhost:9180/metrics?shard=1|0

To get all metrics without a service label:
http://localhost:9180/metrics?service=

To get all metrics whose service label equals prometheus and that come from shard 0:
http://localhost:9180/metrics?service=prometheus&shard=0

Remove the help lines Copy Chapter

Sending the help associated with each metric on every request adds overhead.
Prometheus itself does not use those help lines.
Seastar supports an option to remove those lines from the metrics output using the __help__ query parameter.
To remove the help lines, set __help__=false. For example:
http://localhost:9180/metrics?__help__=false

Aggregation Copy Chapter

In Seastar, metrics can be defined with implicit aggregation by specific labels,
which occurs at query time. This feature is useful, for instance to define metrics
per shard or even more finely grained per an application-defined entity while reporting
them in a more aggregated manner, such as sum or histogram per server.

However, there are times when it is necessary to inspect the fine-grained metrics.
This can be achieved by adding __aggregate__=false to the query string. For example:
http://localhost:9180/metrics?__aggregate__=false

Configuring the Prometheus server to select specific metrics Copy Section

The Prometheus configuration documentation describes the general configuration format.

To specify one or more metrics, add a metrics_path to the scrape configuration in the prometheus.yml file.

For example, the following scrape configuration queries all HTTP metrics:

scrape_configs:
    - job_name: http
      honor_labels: true
      metrics_path: /metrics
      params:
        __name__: ['http*']

To query for multiple specific metrics, list them in the array:

scrape_configs:
    - job_name: selected_metrics
      honor_labels: true
      metrics_path: /metrics
      params:
        __name__: ['httpd_requests_served', 'httpd_connections_total', 'scheduler*']

Doc Reactor Backend Asymmetric Uring (`doc/reactor_backend_asymmetric_uring.md`) Copy Section

Asymmetric io_uring Reactor Backend

The asymmetric_io_uring reactor backend in Seastar is designed to optimize application performance by offloading kernel-side networking and disk I/O processing from compute-bound application shards onto a dedicated set of CPU cores, referred to as worker cores.

Overview Copy Chapter

In traditional Seastar reactor backends, each shard handles its own I/O processing (such as executing system calls) directly on its assigned application core. While this model is highly efficient for many workloads, it forces shards to split processing time between computations and I/O.

The asymmetric_io_uring backend addresses this by dedicating specific CPUs exclusively to running io_uring kernel worker threads and Submission Queue (SQ) polling threads. Shards submit I/O requests directly to io_uring instances instead of performing synchronous system calls inline, offloading application cores from this responsibility.

This backend is expected to benefit compute-intensive workloads by offloading work from application shards. However, it is unlikely to improve I/O-bound workloads, as multiple shards would delegate their I/O operations to the same worker core, which could become a bottleneck.

Key Concepts Copy Chapter

Speculation-Free Execution (No Fast-Track) Copy Section

Traditional Seastar backends use a heuristic-based fast track for networking I/O. If a socket is speculated to be ready, the backend attempts to execute the operations immediately using synchronous system calls on the application core.

The asymmetric_io_uring backend removes the speculative fast path entirely. All networking and disk I/O requests are consistently submitted via io_uring and processed off-shard.

Dedicated Worker Cores Copy Section

The io_uring asynchronous worker threads and polling threads are pinned to a user-defined set of CPUs (--async-workers-cpuset). Typically, these cores are aligned with the system's networking cores (cores configured to handle network interface card IRQs using the perftune.py script) and are ideally excluded from the main Seastar application cpuset. However, exclusion is not strictly enforced; if the worker cpuset overlaps with the application shard cpuset, Seastar will log a configuration warning due to the risks of CPU contention.

Master and Proxy Shards Copy Section

To avoid excessive thread creation and CPU contention on the worker cores, multiple shards share a single worker core. This is implemented via a Master/Proxy grouping model:

• Grouping: Shards are partitioned into groups, and each group is assigned to a specific worker core.
• Master Shard: One shard in each group is designated the master shard. The master shard initializes its io_uring instance with the IORING_SETUP_SQ_AFF flag and sets sq_thread_cpu to the assigned worker core, which spawns the SQ polling thread on that core.
• Proxy Shards: The remaining shards in the group act as proxy shards. They initialize their io_uring instances using the IORING_SETUP_ATTACH_WQ flag, passing the master shard's io_uring file descriptor. This attaches the proxy instances to the master's instance, allowing them to share the same worker thread pool and SQ polling thread.

flowchart TB

subgraph "I/O CPUs"
    direction LR
    N0[I/O CPU 0]
    N1[I/O CPU 1]
end

subgraph "Application CPUs"
    direction LR
    subgraph Group1["Group 1"]
        direction LR
        S3[App CPU 3]
        S2[App CPU 2]
        S2 ~~~S3
    end
    subgraph Group0["Group 0"]
        direction LR
        S1[App CPU 1]
        S0[App CPU 0]
        S0 ~~~ S1
    end
end

N0 -. work on I/O from .-> Group0
N1 -. work on I/O from .-> Group1

Worker-to-Shard Assignment Algorithm Copy Section

The assignment of worker cores to application shards is topology-aware:

1. Topology Detection: The backend uses the Portable Hardware Locality (hwloc) library to detect NUMA nodes, cores, and SMT sibling threads. If hwloc is not available, the backend falls back to a topology-agnostic greedy assignment (effectively treating the system as a single NUMA node and disabling SMT sibling evaluation).
2. Greedy Allocation: based on the detected topology, the backend iterates over the application shards in three passes:
   • Pass 1 (SMT Siblings): Assigns worker cores to shards running on application cores that are SMT siblings of those worker cores.
   • Pass 2 (NUMA Locality): Assigns worker cores to shards on application cores in the same NUMA node. The algorithm greedily assigns shards to the least loaded worker core in that NUMA node.
   • Pass 3 (Cross-NUMA Node): For shards that do not have any worker cores in their NUMA node, the algorithm greedily assigns them to the least loaded worker core in other nodes.

Configuration & Usage Copy Chapter

Compilation Copy Section

The backend must be compiled in by enabling io_uring support. Pass the --enable-io_uring flag when running the Seastar configuration script:

./configure.py --enable-io_uring [other options]

Running Seastar Copy Section

To run an application with the asymmetric backend, specify the reactor backend and the cpuset reserved for the async workers:

./your_seastar_app --reactor-backend=asymmetric_io_uring --async-workers-cpuset=14-15

• --reactor-backend=asymmetric_io_uring: Selects the asymmetric backend.
• --async-workers-cpuset: Specifies the cores dedicated to the io_uring worker and polling threads. This parameter is mandatory when using this backend.

Options Interaction Behavior Copy Section

The following table summarizes the behavior of Seastar depending on the combinations of CPU configuration options:

• Note: taskset refers to the external Linux OS utility/affinity mask constraint applied to the process, rather than a native Seastar command-line option.

Doc Rpc Compression (`doc/rpc-compression.md`) Copy Section

RPC provided compression infrastructure

Compression algorithm negotiation Copy Chapter

The RPC protocol defines the COMPRESS feature bit but not the format of its data.
If an application supports multiple compression algorithms, it may use this data for
negotiation. RPC provides the multi_algo_compressor_factory convenience class so that
applications do not have to reimplement the same logic. The class accepts a list of supported
compression algorithms and sends them as a comma-separated list in the client's COMPRESS
feature payload. On receiving the list, it finds algorithms supported by both the client and server.
If there is more than one, their order in the client's list acts as a tiebreaker (the first algorithm
wins). Once the server chooses a compressor, it puts that compressor's identifier in the returned
COMPRESS feature payload, informing the client which algorithm to use
for the connection.

Compression algorithms Copy Chapter

`LZ4` compressor Copy Section

This compressor uses LZ4 to compress and decompress RPC messages. It requires all memory buffers to be contiguous, which may force it to temporarily linearise fragmented messages. LZ4 is fast enough to often make the cost of those copies not negligible compared to the cost of the whole compression or decompression routine. Therefore, this algorithm is best suited if there is an upper bound of the message size and they are expected to fit in a single fragment of input and output memory buffers.

`LZ4_FRAGMENTED` compressor Copy Section

This compressor uses LZ4 streaming interface to compress and decompress even large messages without linearising them. The LZ4 streaming routines tend to be slower than the basic ones and the general logic for handling buffers is more complex, so this compressor is best suited only when there is no clear upper bound on the message size or if the messages are expected to be fragmented.

Internally, the compressor processes data in 32 kB chunks and tries to avoid unnecessary copies. Applications should therefore use memory-buffer fragment sizes that are an integral multiple of 32 kB.

Doc Rpc Streaming (`doc/rpc-streaming.md`) Copy Section

RPC streaming

Streaming API Copy Chapter

Sink and Source Copy Section

The basic elements of the streaming API are rpc::sink and rpc::source. The former
is used to send data and the latter to receive it. The client and server
each have their own pair of sinks and sources. rpc::sink and rpc::source are
templated classes where template parameters describe a type of the data
that is sent/received. For instance the sink that is used to send messages
containing int and long will be of a type rpc::sink. The
opposite end of the stream will have a source of the type rpc::source
which will be used to receive those messages. Messages are received at a
source as a std::optional containing the actual message as a std::tuple. A disengaged
optional means EOS (end of stream): the stream was closed by the peer. If an
error happens before EOS is received, the receiver cannot be sure that it received all
the data.

To send data using rpc::sink, one can write the following (assuming a seastar::async context):

while (has_data()) {
          int data1 = get_data1();
          long data2 = get_data2();
          sink(data1, data2).get(); // sends data
      }
      sink.close().get(); // closes stream

To receive:

while (true) {
          std::optional<std::tuple<int, long>> data = source().get();
          if (!data) {
             // unengaged optional means EOS
             break;
          } else {
             auto [data1, data2] = *data;
             // process data
          }
      }

Creating a stream Copy Section

To open an RPC stream, an RPC client must already exist. The stream
will be associated with the client, and closing the client aborts any stream
still open on it. Aborting a stream this way is not a substitute for closing
it, see Closing a stream below. Given RPC client rc, and a serializer class that models the Serializer concept (as explained in the rpc::protocol class), one creates rpc::sink as follows
(again assuming seastar::async context):

rpc::sink<int, long> sink = rc.make_stream_sink<serializer, int, long>().get();

Now the client has the sink that can be used for streaming data to
a server, but how the server will get a corresponding rpc::source to
read it? For that the sink should be passed to the server by an RPC
call. To receive a sink a server should register an RPC handler that will
be used to receive it along with any auxiliary information deemed necessary.
To receive the sink above one may register an RPC handler like that:

rpc_proto.register_handler(1, [] (int aux_data, rpc::source<int, long> source) {
    });

Notice that rpc::sink is received as an rpc::source since at the server
side it will be used to receive data. All that remains is for the client to
invoke this RPC handler with aux_data and the sink.

For communication in the other direction, from server to client, the server
must have a sink and the client must have a source. Because messages in this
direction may have a different type from client-to-server messages, the sink
and source may have a different
type as well.

The server initiates creation of a communication channel in the other direction.
It does this by creating a sink from the source it receives and returning the sink
from the RPC handler, causing it to be received as a source by the client. The following
full example shows a server sending a message containing an sstring to a client.

Server handler will look like that:

rpc_proto.register_handler(1, [] (int aux_data, rpc::source<int, long> source) {
        rpc::sink<sstring> sink = source.make_sink<serializer, sstring>();
        // use sink and source asynchronously
        return sink;
    });

Client code will be:

auto rpc_call = rpc_proto.make_client<rpc::source<sstring> (int, rpc::sink<int, long>)>(1);
   rpc::sink<int, long> sink = rc.make_stream_sink<serializer, int, long>().get();
   rpc::source<sstring> source = rpc_call(rc, aux_data, sink).get();
   // use sink and source here

Closing a stream Copy Section

Both halves of a stream must be closed by the application:

• Close the sink with rpc::sink::close() and wait for the returned
  future. Dropping an unclosed sink is an error.
• Read the source until eof or error. rpc::source deliberately has no
  close(): reading it until it yields an unengaged optional (eof) or throws
  is what closes it. If you no longer want the data, tell the peer at the
  application level and then drain what is still in flight.

The stream's connection is shut down only once both halves are closed.

This applies on error too. The calls may fail, and the failures are expected,
but they still have to be made.

Implementation notes Copy Chapter

RPC stream creation Copy Section

An RPC stream is implemented as a separate TCP connection. The RPC server knows that a connection
will be used for streaming if during RPC negotiation Stream parent feature is present.
The feature will contain ID of an RPC client that was used to create the stream.

So in the example from previous chapter:

rpc::sink<int, long> sink = rc.make_stream_sink<serializer, int, long>().get();

the call will initiate a new TCP connection to the same server rc is connected to. During RPC
protocol negotiation this connection will have Stream parent feature with rc's ID as a value.

Passing sink/source over RPC call Copy Section

When rpc::sink is sent over an RPC call, it is serialized as its connection ID. The server's RPC handler
then looks up the connection and creates an rpc::source from it. When an RPC handler returns rpc::sink,
the same happens in the other direction.

Doc Rpc (`doc/rpc.md`) Copy Section

RPC protocol

Data encoding Copy Chapter

All integral data is encoded in little-endian format.

Protocol negotiation Copy Chapter

Negotiation works by exchanging a negotiation frame immediately after the connection is established. The negotiation frame format is:

uint8_t magic[8] = SSTARRPC
uint32_t len
uint8_t data[len]

The negotiation frame data is itself composed of multiple records, one for each feature number present. Feature numbers begin at zero and will be defined by later versions of this document.

struct negotiation_frame_feature_record {
     uint32_t feature_number;
     uint32_t len;
     uint8_t data[len];
 }

A negotiation_frame_feature_record signals that an optional feature is present in the client, and can contain additional feature-specific data. The feature number will be omitted in a server response if an optional feature is declined by the server.

Actual negotiation looks like this:

Client                                  Server
--------------------------------------------------------------------------------------------------
send negotiation frame
                                    recv frame
                                    check magic (disconnect if magic is not SSTARRPC)
                                    send negotiation frame back
recv frame
check magic (disconnect if magic is not SSTARRPC)

Supported features Copy Section

Compression

feature_number:  0
data          :  opaque data that is passed to a compressor factory
                 provided by an application. Compressor factory is
                 responsible for negotiation of compression algorithm.

If compression is negotiated, request and response frames are encapsulated in a compressed frame.

Timeout propagation

feature_number:  1
data          :  none

If timeout propagation is negotiated, the request frame has an additional 8 bytes that hold the
timeout value in milliseconds. A value of zero means that no timeout was specified.
If a timeout is specified and the server cannot handle the request within that time, it may choose
not to send a reply (sending one is not an error either).

Connection ID

feature_number: 2
uint64_t connection_id  : RPC connection ID

The server assigns a unique connection ID to each connection and sends it to the client using
this feature.

Stream parent

feature_number: 3
uint64_t connection_id : RPC connection ID representing a parent of the stream

If this feature is present, the connection is a stream connection rather than a regular RPC
connection. If the parent connection is closed or aborted, all streams belonging
to it will be closed as well.

A stream connection allows a bidirectional flow of bytes that may carry one or more messages in
each direction. A stream connection should be explicitly closed by both the client and the server.
Closing is done by sending a special EOS frame (described below).

Isolation

feature number: 4
uint32_t isolation_cookie_len
uint8_t isolation_cookie[len]

The `isolation_cookie` field is used by the server to select a
`seastar::scheduling_group` (or equivalent in another implementation) that
will run this connection. In the future it will also be used for rpc buffer
isolation, to avoid rpc traffic in one isolation group from starving another.

The server does not directly assign meaning to values of `isolation_cookie`;
instead, the interpretation is left to user code.

Handler duration

feature number: 5
data: none

Asks server to send "extended" response that includes the handler duration time. See
the response frame description for more details

Compressed frame format

uint32_t len
uint8_t compressed_data[len]

After compressed_data is uncompressed, it becomes a regular request, response or streaming frame.

As a special case, it is allowed to send a compressed frame of size 0 (pre-compression).
Such a frame will be a no-op on the receiver.
(This can be used as a means of communication between the compressors themselves.
If a compressor wants to send some metadata to its peer, it can send a no-op frame,
and prepend the metadata as a compressor-specific header.)

Request frame format Copy Chapter

uint64_t timeout_in_ms - only present if timeout propagation is negotiated
uint64_t verb_type
int64_t msg_id
uint32_t len
uint8_t data[len]

msg_id must be positive and must never be reused.
data is transparent to the protocol and is serialized/deserialized by the user.

Response frame format Copy Chapter

int64_t msg_id
uint32_t len
uint32_t handler_duration - present if handler duration is negotiated
uint8_t data[len]

If msg_id < 0, the enclosed response contains an exception returned for message ID abs(msg_id).
data is transparent to the protocol and is serialized/deserialized by the user.

handler_duration is in microseconds. A value of 0xffffffff means that it was not measured
and should be disregarded by the client.

Stream frame format Copy Chapter

uint32_t len
uint8_t data[len]

len == 0xffffffff signals the end of the stream.
data is transparent to the protocol and is serialized/deserialized by the user.

Exception encoding Copy Chapter

uint32_t type
uint32_t len
uint8_t data[len]

Known exception types Copy Section

USER = 0
UNKNOWN_VERB = 1

USER exception encoding

uint32_t len
char[len]

This exception is sent as a reply if an RPC handler throws an exception.
It is delivered to the caller as rpc::remote_verb_error(char[len]).

UNKNOWN_VERB exception encoding

uint64_t verb_id

This exception is sent in response to a request with an unknown verb_id; the verb ID is returned as part of the exception payload.

More formal protocol description Copy Chapter

request_stream = negotiation_frame, { request | compressed_request }
request = verb_type, msg_id, len, { byte }*len
compressed_request = len, { bytes }*len
response_stream = negotiation_frame, { response | compressed_response }
response = reply | exception
compressed_response = len, { byte }*len
    streaming_stream = negotiation_frame, { streaming_frame | compressed_streaming_frame }
    streaming_frame = len, { byte }*len
    compressed_streaming_frame = len, { byte }*len
reply = msg_id, len, { byte }*len
exception = exception_header, serialized_exception
exception_header = -msg_id, len
serialized_exception = (user|unknown_verb)
user = len, {byte}*len
unknown_verb = verb_type
verb_type = uint64_t
msg_id = int64_t
len = uint32_t
byte = uint8_t
negotiation_frame = 'SSTARRPC' len32(negotiation_frame_data) negotiation_frame_data
negotiation_frame_data = negotiation_frame_feature_record*
negotiation_frame_feature_record = feature_number len {byte}*len
feature_number = uint32_t

Note that replies can come in order different from requests, and some requests may not have a reply at all.

Doc Shared Token Bucket (`doc/shared-token-bucket.md`) Copy Section

Shared token bucket

Intro Copy Chapter

The classical token bucket has two parameters -- rate and limit. The rate
is the number of tokens put into the bucket per unit of time
(for example, per second); the limit is the maximum number of tokens that can be
accumulated in the bucket. The process of regeneration of tokens this way
is called "replenishing" below. When a request needs to be served it should
try to get a certain amount of tokens from the bucket.

The shared token bucket implements the above model for Seastar's sharded
architecture. The implementation doesn't use locks and is built on atomic
arithmetic operations.

Theory Copy Chapter

Rovers Copy Section

The bucket is implemented as a pair of increasing counters -- tail and
head rovers. The consumer of tokens advances the tail rover. To replenish
tokens into the bucket the head rover is advanced.

+--------------------------------------------------------------------->
^             ^
tail          head

grab N tokens:

+--------------------------------------------------------------------->
.---> ^       ^
      tail    head

replenish N tokens:

+--------------------------------------------------------------------->
      ^       .---> ^
      tail          head

After a grab, the tail may overrun the head and end up
in front of it. This would mean that there's not enough tokens in the bucket
and that some amount of tokens were claimed from it.

grab a lot of tokens:

+--------------------------------------------------------------------->
      .------------ ^ --> ^
                    head  tail

To check if the tokens were grabbed the caller needs to check if the head is
(still) in front of the tail. This approach adds the ability for the consumers
to line up in the queue when they all try to grab tokens from a contended
bucket. The "ticket lock" model works the same way.

Capped release Copy Section

The implementation additionally supports a "capped release." In this mode,
tokens are not created during replenishment. Instead, they move into the main
bucket from another bucket that the caller must explicitly refill. This mode
can be useful when the token bucket guards entry into a location
that can temporarily (or constantly, which would indicate
a bucket misconfiguration) slow down and stop handling tokens at the given
rate. To prevent token bucket from over-subscribing the guardee at those times,
the second bucket can be refilled with the completions coming from the latter.

In terms of rovers this is implemented with the help of a third rover called
ceiling (or ceil in the code). This ceil rover actually defines the upper
limit at which the head rover may point. Respectively, putting tokens into
the second bucket (it's called releasing below) means advancing the ceil.

Practice Copy Chapter

API Copy Section

To work with the token bucket there are 4 calls:

• grab(N) -- grabs a certain amount of tokens from the bucket and returns
  the resulting "tail" rover value. The value is useless per se and is
  only needed to call the deficiency() method

• replenish(time) -- tries to replenish tokens into the bucket. The amount
  of replenished tokens is how many had accumulated since last replenish
  till the time parameter

• release(N) -- releases the given number of tokens, making them available
  for replenishment. Only works if capped release is turned on by the
  template parameter, otherwise asserts

• deficiency(tail) -- returns back the number of tokens that were claimed
  from the bucket but that are not yet there. Non-zero number means that the
  bucket is contended and request dispatch should be delayed

Example Copy Section

For example, the simple dispatch loop may look like this

while (true) {
    request = get_next_request()
    tail = token_bucket.grab(request.cost())
    while (token_bucket.deficiency(tail)) {
        yield
    }
    request.dispatch()
}

And in the background there should run a timer calling token_bucket.replenish(now())

Additionally, if there's a need to cap the token bucket with the real request serving
rate, upon request completion one should call token_bucket.release(request.cost())

Doc Signal (`doc/signal.md`) Copy Section

Signals

Seastar provides an interface to handle signals natively and safely as asynchronous tasks.

Default Signal Handlers Copy Chapter

By default, Seastar sets signal handlers for SIGINT and SIGTERM that call reactor::stop(). The reactor then executes callbacks installed by reactor::at_exit().

You can disable this behavior by setting app_template::config::auto_handle_sigint_sigterm to false. This flag is provided in seastar/core/app-template.hh. Seastar will then leave these signal handlers unset, preserving the default Linux behavior of terminating the program.

Examples Copy Section

#include <seastar/core/app-template.hh>
#include <seastar/core/future.hh>
#include <iostream>
#include <utility>

int main(int argc, char** argv) {
    seastar::app_template::config cfg;
    cfg.auto_handle_sigint_sigterm = false;
    seastar::app_template app(std::move(cfg));

    return app.run(argc, argv, [] {
        std::cout << "SIGINT/SIGTERM will terminate the program\n";
        return seastar::make_ready_future<>();
    });
}

Custom Signal Handler Copy Chapter

To set a custom signal handler, use seastar::handle_signal from seastar/core/signal.hh. It registers a custom handler for the specified signal.

The function must be called inside the app.run() lambda; calling it elsewhere causes undefined behavior.

Examples Copy Section

#include <seastar/core/app-template.hh>
#include <seastar/core/future.hh>
#include <seastar/core/signal.hh>
#include <csignal>
#include <iostream>

int main(int argc, char** argv) {
    seastar::app_template app;
    return app.run(argc, argv, [] {
        seastar::handle_signal(SIGINT, [] {
            std::cout << "caught sigint\n";
        }, true);
        return seastar::make_ready_future<>();
    });
}

• tests/unit/signal_test.cc
• apps/lib/stop_signal.hh

Doc Testing (`doc/testing.md`) Copy Section

Testing

Seastar leverages Boost.Test and provides facilities for developers to implement tests in coroutines.

Test Declarations Copy Chapter

There are three categories of boost-based tests in our system:

• Boost.Test Native: Tests defined using BOOST_AUTO_TEST_CASE and related macros from Boost.Test. For more information, see the Boost.Test documentation.
• Coroutine: Tests defined using SEASTAR_TEST_CASE. The test body returns a future, allowing implementation as a coroutine.
• Coroutine with seastar::thread: Tests defined using SEASTAR_THREAD_TEST_CASE. The test body is launched in a Seastar thread, allowing the use of Seastar coroutines. These tests should return void.

Choosing the Appropriate Macro Copy Chapter

• Use SEASTAR_TEST_CASE or SEASTAR_THREAD_TEST_CASE if you need to run tests in a Seastar reactor thread, typically for Seastar coroutines. SEASTAR_THREAD_TEST_CASE can be more convenient in some cases.
• For Seastar facilities that don't depend on coroutines, consider using BOOST_AUTO_TEST_CASE.

Tests' organization Copy Chapter

• Tests defined with BOOST_AUTO_TEST_CASE are driven by Boost.Test's built-in test runner. They require defining BOOST_TEST_MODULE to include the runner in the executable. These tests support additional features like fixtures and data-driven testing.
• SEASTAR_TEST_CASE and SEASTAR_THREAD_TEST_CASE tests only support decorators and share a common test runner, allowing them to be co-located in the same test suite.
• Seastar tests can be co-located with "native" Boost tests. The SEASTAR_TEST_CASE / SEASTAR_THREAD_TEST_CASE restrictions above apply to the entire test binary, and the BOOST_TEST_MODULE macro must not be defined.

Adding Tests in `CMakeLists.txt` Copy Chapter

For Seastar tests (and binaries that contain both Seastar and Boost tests):

seastar_add_test(meow_test
  KIND SEASTAR
  SOURCES meow.cc)

or simply:

seastar_add_test(meow_test
  SOURCES meow.cc)

The KIND parameter of seastar_add_test() defaults to SEASTAR, which uses the Seastar test runner by defining the SEASTAR_TESTING_MAIN macro.

For "native" Boost(-only) tests:

seastar_add_test(woof_test
  KIND BOOST
  SOURCES woof.cc)

Fuzz Testing Copy Chapter

Seastar supports fuzz testing using libFuzzer for testing non-reactor code. Fuzz tests are built only in fuzz mode.

Building Fuzz Tests Copy Section

Configure with fuzz mode (requires Clang):

./configure.py --mode fuzz --compiler clang++ [other options]

Build all fuzz tests:

ninja -C build/fuzz fuzz_tests

Running Fuzz Tests Copy Section

Fuzz test executables are located in build/fuzz/tests/fuzz/. Run a fuzzer:

# Basic run (runs indefinitely until stopped or crash found)
./build/fuzz/tests/fuzz/sstring_fuzz

# With a corpus directory (recommended for persistent fuzzing)
mkdir -p corpus/sstring
./build/fuzz/tests/fuzz/sstring_fuzz corpus/sstring

# Limit iterations
./build/fuzz/tests/fuzz/sstring_fuzz -runs=10000

# Limit time (in seconds)
./build/fuzz/tests/fuzz/sstring_fuzz -max_total_time=60

For more libFuzzer options, see the libFuzzer documentation.

Adding Fuzz Tests Copy Section

Fuzz tests are added in tests/fuzz/CMakeLists.txt:

seastar_add_fuzz_test(mycomponent
  SOURCES mycomponent_fuzz.cc)

The fuzz test source must define LLVMFuzzerTestOneInput:

extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
    // Test code here using data[0..size-1] as input
    return 0;
}

Doc Tutorial (`doc/tutorial.md`) Copy Section

% Asynchronous Programming with Seastar
% Nadav Har'El - [email protected]
Avi Kivity - [email protected]

Introduction

Seastar, which we introduce in this document, is a C++ library for writing highly efficient complex server applications on modern multi-core machines.

Traditionally, the programming languages, libraries, and frameworks used for writing server applications have been divided into two distinct camps: those focusing on efficiency and those focusing on complexity. Some frameworks are extremely efficient yet allow building only simple applications (e.g., DPDK allows applications that process packets individually), while other frameworks allow building extremely complex applications at the cost of runtime efficiency. Seastar is our attempt to get the best of both worlds: to create a library that allows building highly complex server applications while achieving optimal performance.

The inspiration and first use case for Seastar was Scylla, a rewrite of Apache Cassandra. Cassandra is a very complex application, yet with Seastar we were able to reimplement it with up to a tenfold throughput increase, as well as significantly lower and more consistent latencies.

Seastar offers a complete asynchronous programming framework, which uses two concepts—futures and continuations—to uniformly represent and handle every type of asynchronous event, including network I/O, disk I/O, and complex combinations of other events.

Since modern multi-core and multi-socket machines have steep penalties for sharing data between cores (atomic instructions, cache line bouncing and memory fences), Seastar programs use the share-nothing programming model, i.e., the available memory is divided between the cores, each core works on data in its own part of memory, and communication between cores happens via explicit message passing (which itself happens using the SMP's shared memory hardware, of course).

Asynchronous programming Copy Chapter

A server for a network protocol, such as the classic HTTP (Web) or SMTP (email) servers, inherently deals with parallelism: multiple clients send requests in parallel, and we cannot finish handling one request before starting to handle the next. A request may, and often does, need to block for various reasons—a full TCP window (i.e., a slow connection), disk I/O, or even the client holding on to an inactive connection—and the server needs to handle other connections as well.

The most straightforward way to handle such parallel connections, employed by classic network servers such as Inetd, Apache Httpd and Sendmail, is to use a separate operating-system process per connection. This technique evolved over the years to improve its performance: At first, a new process was spawned to handle each new connection; Later, a pool of existing processes was kept and each new connection was assigned to an unemployed process from the pool; Finally, the processes were replaced by threads. However, the common idea behind all these implementations is that at each moment, each process handles exclusively a single connection. Therefore, the server code is free to use blocking system calls, such as reading or writing to a connection, or reading from disk, and if this process blocks, all is well because we have many additional processes ready to handle other connections.

Programming a server which uses a process (or a thread) per connection is known as synchronous programming, because the code is written linearly, and one line of code starts to run after the previous line finished. For example, the code may read a request from a socket, parse the request, and then piecemeal read a file from disk and write it back to the socket. Such code is easy to write, almost like traditional non-parallel programs. In fact, it's even possible to run an external non-parallel program to handle each request --- this is for example how Apache HTTPd ran "CGI" programs, the first implementation of dynamic Web-page generation.

NOTE: although the synchronous server application is written in a linear, non-parallel, fashion, behind the scenes the kernel helps ensure that everything happens in parallel and the machine's resources --- CPUs, disk and network --- are fully utilized. Beyond the process parallelism (we have multiple processes handling multiple connections in parallel), the kernel may even parallelize the work of one individual connection --- for example process an outstanding disk request (e.g., read from a disk file) in parallel with handling the network connection (send buffered-but-yet-unsent data, and buffer newly-received data until the application is ready to read it).

But synchronous, process-per-connection server programming came with disadvantages and costs. Slowly but surely, server authors realized that starting a new process is slow, context switching is slow, and each process has significant overhead—most notably the size of its stack. Server and kernel authors worked hard to mitigate these overheads: they switched from processes to threads and from creating new threads to thread pools, lowered each thread's default stack size, and increased the virtual memory size to allow more partially utilized stacks. Still, servers with synchronous designs had unsatisfactory performance and scaled poorly as the number of concurrent connections grew. In 1999, Dan Kegel popularized "the C10K problem": the need for a single server to efficiently handle 10,000 concurrent connections, most of them slow or even inactive.

The solution, which became popular in the following decade, was to abandon the cozy but inefficient synchronous server design, and switch to a new type of server design --- the asynchronous, or event-driven, server. An event-driven server has just one thread, or more accurately, one thread per CPU. This single thread runs a tight loop which, at each iteration, checks, using poll() (or the more efficient epoll) for new events on many open file descriptors, e.g., sockets. For example, an event can be a socket becoming readable (new data has arrived from the remote end) or becoming writable (we can send more data on this connection). The application handles this event by doing some non-blocking operations, modifying one or more of the file descriptors, and maintaining its knowledge of the state of this connection.

However, writers of asynchronous server applications faced, and still face today, two significant challenges:

• Complexity: Writing a simple asynchronous server is straightforward. But writing a complex asynchronous server is notoriously difficult. The handling of a single connection, instead of being a simple easy-to-read function call, now involves a large number of small callback functions, and a complex state machine to remember which function needs to be called when each event occurs.

• Non-blocking: Having just one thread per core is important for the performance of the server application, because context switches are slow. However, if we only have one thread per core, the event-handling functions must never block, or the core will remain idle. But some existing programming languages and frameworks leave the server author no choice but to use blocking functions, and therefore multiple threads.
  For example, Cassandra was written as an asynchronous server application; But because disk I/O was implemented with mmaped files, which can uncontrollably block the whole thread when accessed, they are forced to run multiple threads per CPU.

Moreover, when the best possible performance is desired, the server application, and its programming framework, has no choice but to also take the following into account:

• Modern Machines: Modern machines are very different from those of just 10 years ago. They have many cores and deep memory hierarchies (from L1 caches to NUMA) which reward certain programming practices and penalize others. Unscalable programming practices (such as taking locks) can devastate performance on many cores. Shared-memory and lock-free synchronization primitives are available (i.e., atomic operations and memory-ordering fences), but they are dramatically slower than operations involving only data in a single core's cache and also prevent the application from scaling to many cores.

• Programming Language: High-level languages such as Java, JavaScript, and similar "modern" languages are convenient, but each comes with its own set of assumptions which conflict with the requirements listed above. These languages, aiming to be portable, also give the programmer less control over the performance of critical code. For optimal performance, we need a programming language which gives the programmer full control and zero runtime overhead while still providing sophisticated compile-time code generation and optimization.

Seastar is a framework for writing asynchronous server applications which aims to solve all four of the above challenges: It is a framework for writing complex asynchronous applications involving both network and disk I/O. The framework's fast path is entirely single-threaded (per core), scales to many cores, and minimizes costly memory sharing between cores. It supports C++23 and C++26, giving the user sophisticated compile-time features and full control over performance without runtime overhead.

Seastar Copy Chapter

Seastar is an event-driven framework allowing you to write non-blocking, asynchronous code in a relatively straightforward manner (once understood). Its APIs are based on futures. Seastar utilizes the following concepts to achieve extreme performance:

• Cooperative micro-task scheduler: instead of running threads, each core runs a cooperative task scheduler. Each task is typically very lightweight -- only running for as long as it takes to process the last I/O operation's result and to submit a new one.
• Share-nothing SMP architecture: each core runs independently of other cores in an SMP system. Memory, data structures, and CPU time are not shared; instead, inter-core communication uses explicit message passing. A Seastar core is often termed a shard. See the SMP documentation for more information.
• Future-based APIs: futures allow you to submit an I/O operation and chain tasks to be executed when the I/O operation completes. It is easy to run multiple I/O operations in parallel—for example, in response to a request from a TCP connection, you can issue multiple disk I/O requests, send messages to other cores on the same system, or send requests to other nodes in the cluster, wait for some or all of the results, aggregate them, and send a response.
• Share-nothing TCP stack: while Seastar can use the host operating system's TCP stack, it also provides its own high-performance TCP/IP stack built on top of the task scheduler and the share-nothing architecture. The stack provides zero-copy in both directions: you can process data directly from the TCP stack's buffers, and send the contents of your own data structures as part of a message without incurring a copy. See the networking documentation for more information.
• DMA-based storage APIs: as with the networking stack, Seastar provides zero-copy storage APIs, allowing you to DMA your data to and from your storage devices.

This tutorial is intended for developers already familiar with the C++ language, and will cover how to use Seastar to create a new application.

Getting started

The simplest Seastar program is this:

#include <seastar/core/app-template.hh>
#include <seastar/core/reactor.hh>
#include <iostream>

int main(int argc, char** argv) {
    seastar::app_template app;
    return app.run(argc, argv, [] {
            std::cout << "Hello world\n";
            return seastar::make_ready_future<>();
    });
}

As in this example, each Seastar program must define and run an app_template object. This object starts the main event loop (the Seastar engine) on one or more CPUs and then runs the given function—in this case an unnamed function, or lambda—once.

The return make_ready_future<>(); causes the event loop, and the whole application, to exit immediately after printing the "Hello World" message. In a more typical Seastar application, we want the event loop to remain alive and process incoming packets, for example, until explicitly stopped. Such applications return a future which determines when to exit the application. We introduce futures and how to use them below. In any case, the regular C exit() should not be used because it prevents Seastar or the application from cleaning up appropriately.

As shown in this example, all Seastar functions and types live in the "seastar" namespace. A user can either type this namespace prefix every time, or use shortcuts like "using seastar::app_template" or even "using namespace seastar" to avoid typing this prefix. We generally recommend using the namespace prefixes seastar and std explicitly and follow this style in all the examples below.

To compile this program (it's present in the demos/hello-world.cc file) you can just use Docker.

$ docker build -t seastar-dev  -f ./docker/dev/Dockerfile .
$ docker run -it --rm -v $(pwd):/seastar seastar-dev /seastar/configure.py --mode=dev --cook=c-ares
$ docker run -it --rm -v $(pwd):/seastar seastar-dev ninja -C /seastar/build/dev
$ docker run -it --rm -v $(pwd):/seastar seastar-dev /seastar/build/dev/demos/hello-world_demo -c1

Without the docker help, first make sure you have downloaded, built, and optionally installed Seastar, and put the above program in a source file anywhere you want, let's call the file getting-started.cc.

Linux's pkg-config is one way to determine the compilation and linking parameters needed to use libraries such as Seastar. For example, if Seastar was built in the directory $SEASTAR but not installed, one can compile getting-started.cc with it using the command:

c++ getting-started.cc `pkg-config --cflags --libs --static $SEASTAR/build/release/seastar.pc`

The "--static" is needed because currently, Seastar is built as a static library, so we need to tell pkg-config to include its dependencies in the link command (whereas, had Seastar been a shared library, it could have pulled in its own dependencies).

If Seastar was installed, the pkg-config command line is even shorter:

c++ getting-started.cc `pkg-config --cflags --libs --static seastar`

Alternatively, one can easily build a Seastar program with CMake. Given the following CMakeLists.txt

cmake_minimum_required (VERSION 3.5)

project (SeastarExample)

find_package (Seastar REQUIRED)

add_executable (example
  getting-started.cc)

target_link_libraries (example
  PRIVATE Seastar::seastar)

you can compile the example with the following commands:

$ mkdir build
$ cd build
$ cmake ..
$ make

The program now runs as expected:

$ ./example
Hello world
$

Threads and memory

Seastar threads Copy Chapter

As explained in the introduction, Seastar-based programs run a single thread on each CPU. Each of these threads runs its own event loop, known as the engine in Seastar nomenclature. By default, the Seastar application will take over all the available cores, starting one thread per core. We can see this with the following program, printing seastar::smp::count which is the number of started threads:

#include <seastar/core/app-template.hh>
#include <seastar/core/reactor.hh>
#include <iostream>

int main(int argc, char** argv) {
    seastar::app_template app;
    return app.run(argc, argv, [] {
            std::cout << seastar::smp::count << "\n";
            return seastar::make_ready_future<>();
    });
}

On a machine with 4 hardware threads (two cores, and hyperthreading enabled), Seastar will by default start 4 engine threads:

$ ./a.out
4

Each of these 4 engine threads will be pinned (à la taskset(1)) to a different hardware thread. Note how, as we mentioned above, the app's initialization function is run only on one thread, so we see the output "4" only once. Later in the tutorial we'll see how to make use of all threads.

The user can pass a command line parameter, -c, to tell Seastar to start fewer threads than the available number of hardware threads. For example, to start Seastar on only 2 threads, the user can do:

$ ./a.out -c2
2

When the machine is configured as in the example above - two cores with two hyperthreads on each - and only two threads are requested, Seastar ensures that each thread is pinned to a different core, and we don't get the two threads competing as hyperthreads of the same core (which would, of course, damage performance).

We cannot start more threads than the number of hardware threads, as allowing this will be grossly inefficient. Trying it will result in an error:

$ ./a.out -c5
Could not initialize seastar: std::runtime_error (insufficient processing units)

The error is an exception thrown from app.run, which was caught by seastar itself and turned into a non-zero exit code. Note that catching the exceptions this way does not catch exceptions thrown in the application's actual asynchronous code. We will discuss these later in this tutorial.

Seastar memory Copy Chapter

As explained in the introduction, Seastar applications shard their memory. Each thread is preallocated with a large piece of memory (on the same NUMA node it is running on), and uses only that memory for its allocations (such as malloc() or new).

By default, the machine's entire memory except a certain reservation left for the OS (defaulting to the maximum of 1.5G or 7% of total memory) is pre-allocated for the application in this manner. This default can be changed by either changing the amount reserved for the OS (not used by Seastar) with the --reserve-memory option, or by explicitly giving the amount of memory given to the Seastar application, with the -m option. This amount of memory can be in bytes, or using the units "k", "M", "G" or "T". These units use the power-of-two values: "M" is a mebibyte, 2^20 (=1,048,576) bytes, not a megabyte (10^6 or 1,000,000 bytes).

Trying to give Seastar more memory than physical memory immediately fails:

$ ./a.out -m10T
Couldn't start application: std::runtime_error (insufficient physical memory)

Introducing futures and continuations

Futures and continuations, which we will introduce now, are the building blocks of asynchronous programming in Seastar. Their strength lies in the ease of composing them together into a large, complex, asynchronous program, while keeping the code fairly readable and understandable.

A [future](\ref future) is a result of a computation that may not be available yet.
Examples include:

• a data buffer that we are reading from the network
• the expiration of a timer
• the completion of a disk write
• the result of a computation that requires the values from
  one or more other futures.

The type future variable holds an int that will eventually be available - at this point might already be available, or might not be available yet. The method available() tests if a value is already available, and the method get() gets the value. The type future<> indicates something which will eventually complete, but not return any value.

A future is usually returned by an asynchronous function, a function which returns a future and arranges for this future to be eventually resolved. Because asynchronous functions promise to eventually resolve the future which they returned, asynchronous functions are sometimes called "promises"; But we will avoid this term because it tends to confuse more than it explains.

One simple example of an asynchronous function is Seastar's function sleep():

future<> sleep(std::chrono::duration<Rep, Period> dur);

This function arranges a timer so that the returned future becomes available (without an associated value) when the given time duration elapses.

A continuation is a callback (typically a lambda) to run when a future becomes available. A continuation is attached to a future with the then() method. Here is a simple example:

#include <seastar/core/app-template.hh>
#include <seastar/core/sleep.hh>
#include <iostream>

int main(int argc, char** argv) {
    seastar::app_template app;
    return app.run(argc, argv, [] {
        std::cout << "Sleeping... " << std::flush;
        using namespace std::chrono_literals;
        return seastar::sleep(1s).then([] {
            std::cout << "Done.\n";
        });
    });
}

In this example we see us getting a future from seastar::sleep(1s), and attaching to it a continuation which prints a "Done." message. The future will become available after 1 second has passed, at which point the continuation is executed. Running this program, we indeed see the message "Sleeping..." immediately, and one second later the message "Done." appears and the program exits.

The return value of then() is itself a future which is useful for chaining multiple continuations one after another, as we will explain below. But here we just note that we return this future from app.run's function, so that the program will exit only after both the sleep and its continuation are done.

To avoid repeating the boilerplate "app_engine" part in every code example in this tutorial, let's create a simple main() with which we will compile the following examples. This main just calls function future<> f(), does the appropriate exception handling, and exits when the future returned by f is resolved:

#include <seastar/core/app-template.hh>
#include <seastar/util/log.hh>
#include <iostream>
#include <stdexcept>

extern seastar::future<> f();

int main(int argc, char** argv) {
    seastar::app_template app;
    try {
        return app.run(argc, argv, f);
    } catch(...) {
        std::cerr << "Couldn't start application: "
                  << std::current_exception() << "\n";
        return 1;
    }
}

Compiling together with this main.cc, the above sleep() example code becomes:

#include <seastar/core/sleep.hh>
#include <iostream>

seastar::future<> f() {
    std::cout << "Sleeping... " << std::flush;
    using namespace std::chrono_literals;
    return seastar::sleep(1s).then([] {
        std::cout << "Done.\n";
    });
}

So far, this example was not very interesting—there is no parallelism, and the same thing could have been achieved by the normal blocking POSIX sleep(). Things become much more interesting when we start several sleep() futures in parallel and attach a different continuation to each. Futures and continuations make parallelism easy and natural:

#include <seastar/core/sleep.hh>
#include <iostream>

seastar::future<> f() {
    std::cout << "Sleeping... " << std::flush;
    using namespace std::chrono_literals;
    seastar::sleep(200ms).then([] { std::cout << "200ms " << std::flush; });
    seastar::sleep(100ms).then([] { std::cout << "100ms " << std::flush; });
    return seastar::sleep(1s).then([] { std::cout << "Done.\n"; });
}

Each sleep() and then() call returns immediately: sleep() just starts the requested timer, and then() sets up the function to call when the timer expires. So all three lines happen immediately and f returns. Only then, the event loop starts to wait for the three outstanding futures to become ready, and when each one becomes ready, the continuation attached to it is run. The output of the above program is of course:

$ ./a.out
Sleeping... 100ms 200ms Done.

sleep() returns future<>, meaning it will complete at a future time, but once complete, does not return any value. More interesting futures do specify a value of any type (or multiple values) that will become available later. In the following example, we have a function returning a future, and a continuation to be run once this value becomes available. Note how the continuation gets the future's value as a parameter:

#include <seastar/core/sleep.hh>
#include <iostream>

seastar::future<int> slow() {
    using namespace std::chrono_literals;
    return seastar::sleep(100ms).then([] { return 3; });
}

seastar::future<> f() {
    return slow().then([] (int val) {
        std::cout << "Got " << val << "\n";
    });
}

The function slow() deserves more explanation. As usual, this function returns a future immediately and doesn't wait for the sleep to complete, so the code in f() can chain a continuation to this future's completion. The future returned by slow() is itself a chain of futures: it becomes ready once the sleep's future becomes ready and the value 3 is returned. We'll explain below in more detail how then() returns a future and how this allows chaining futures.

This example begins to show the convenience of the futures programming model, which allows the programmer to neatly encapsulate complex asynchronous operations. slow() might involve a complex asynchronous operation requiring multiple steps, but its user can use it just as easily as a simple sleep(), and Seastar's engine takes care of running the continuations whose futures have become ready at the right time.

Ready futures Copy Chapter

A future value might already be ready when then() is called to chain a continuation to it. This important case is optimized, and the continuation is run immediately instead of being registered to run later in the next iteration of the event loop.

make_ready_future<> can be used to return a future which is already ready. The following example is identical to the previous one, except the promise function fast() returns a future which is already ready, and not one which will be ready in a second as in the previous example. The nice thing is that the consumer of the future does not care, and uses the future in the same way in both cases.

#include <seastar/core/future.hh>
#include <iostream>

seastar::future<int> fast() {
    return seastar::make_ready_future<int>(3);
}

seastar::future<> f() {
    return fast().then([] (int val) {
        std::cout << "Got " << val << "\n";
    });
}

Preemption and Task Quota Copy Chapter

As described above, an existing fiber of execution will yield back to the event loop when it performs a blocking operation such as IO or sleeping, as it has no more work to do until this blocking operation completes. Should a fiber have a lot of CPU bound work to do without any intervening blocking operations, however, it is important that execution is still yielded back to the event loop periodically.

This is implemented via preemption, which can occur only at specific preemption points. At these points, the fiber's remaining task quota is checked, and if it has been exceeded, the fiber yields. The task quota controls how long tasks may run before yielding to the event loop and is set to 500 µs by default.

It is important not to starve the event loop, as this would starve continuations of futures that weren't ready but have since become ready, and also starve the important polling done by the event loop (e.g., checking whether there is new activity on the network card). For example, iterating over a large container while doing CPU-bound work without any suspension points could starve the reactor and cause a reactor stall, which refers to a substantial period of time (e.g., more than 20 milliseconds) during which a task does not yield.

Many seastar constructs such as looping constructs have built-in preemption points. You may also insert your own preemption points by calling seastar::maybe_yield, which performs a preemption check. Coroutines will also perform a preemption check at each co_await. Note that there is not a preemption check between continuations attached to a future with then(), so a recursive future loop without explicit preemption checks may starve the reactor.

Coroutines

Note: coroutines are a C++20 language feature and are supported by all compilers that meet Seastar's current requirements.

The simplest way to write efficient asynchronous code with Seastar is to use coroutines. Coroutines don't share most of the pitfalls of traditional continuations (below), and so are the preferred way to write new code.

A coroutine is a function that returns a seastar::future<T> and uses the co_await or co_return keywords. Coroutines are invisible to their callers and callees; they integrate with traditional Seastar code in either role. If you are not familiar with C++ coroutines, you may want to consult A more general introduction to C++ coroutines; this section focuses on how coroutines integrate with Seastar.

Here's an example of a simple Seastar coroutine:

#include <seastar/core/coroutine.hh>

seastar::future<int> read();
seastar::future<> write(int n);

seastar::future<int> slow_fetch_and_increment() {
    auto n = co_await read();     // #1
    co_await seastar::sleep(1s);  // #2
    auto new_n = n + 1;           // #3
    co_await write(new_n);        // #4
    co_return n;                  // #5
}

In #1, we call the read() function, which returns a future. The co_await keyword instructs Seastar to inspect the returned future. If the future is ready, then the value (an int) is extracted from the future and assigned to n. If the future is not ready, the coroutine arranges for itself to be called when the future becomes ready, and control is returned to Seastar. Once the future becomes ready, the coroutine is awakened and the value is extracted from the future and assigned to n.

In #2, we call seastar::sleep() and wait for the returned future to become ready, which it will in a second. This demonstrates that n is preserved across co_await calls, and the author of the coroutine need not arrange for storage for coroutine local variables.

Line #3 demonstrates the addition operation, with which the reader is assumed to be familiar.

In #4, we call a function that returns a seastar::future<>. In this case, the future carries no value, and so no value is extracted and assigned.

Line #5 demonstrates returning a value. The integer value is used to satisfy the future that our caller got when calling the coroutine.

Lambda coroutines (C++ 23 and later) Copy Chapter

In C++ 23 and later, lambda coroutines can be safely used from all contexts.

If the lambda coroutine is awaited from an outer coroutine, nothing special needs to be done:

future<>
outer_coroutine() {
    co_await [captures...] () {
        co_await ...;
        co_return;
    }();
}

This works even if the lambda is called indirectly, as long as it is captured by value and
awaited in the same statement it is defined in, as C++ will extend the lifetime of the capture
group until the semicolon.

In other cases (lambdas called asynchronously to the caller, or passed to continuations, add
this auto as the first parameter to convert the signature of the generated operator()()
to accept the lambda by value. This copies the capture group to the lambda coroutine frame when
it is called.

future<>
outer_coroutine() {
    asynchronous_function([captures...] (this auto) {
        co_await ...;
        co_return;
    });
    // The lambda coroutine can be called after outer_coroutine() returns
    co_return;
}

Using this auto can add some small overhead to move objects, but is generally safer
when it's not clear the lambda is awaited synchronously.

Lambda coroutines (before C++ 23) Copy Chapter

A lambda function can be a coroutine. Due to an interaction between how C++ lambda coroutines are specified and how
Seastar coroutines work, using lambda coroutines as continuations can result in use-after-free. To avoid such problems,
take one of the following approaches:

1. Use lambda coroutines as arguments to functions that explicitly claim support for them
2. Wrap lambda coroutines with seastar::coroutine::lambda(), and ensure the lambda coroutine is fully awaited within the statement it is defined in.

An example of wrapping a lambda coroutine is:

#include <seastar/core/coroutine.hh>
#include <seastar/coroutine/maybe_yield.hh>

future<> foo() {
    int n = 3;
    int m = co_await seastar::yield().then(seastar::coroutine::lambda([n] () -> future<int> {
        co_await seastar::coroutine::maybe_yield();
        // `n` can be safely used here
        co_return n;
    }));
    assert(n == m);
}

Notes:

1. seastar::future::then() accepts a continuation
2. We wrap the argument to seastar::future::then() with seastar::coroutine::lambda()
3. We ensure evaluation of the lambda completes within the same expression using the outer co_await.

More information can be found in lambda-coroutine-fiasco.md.

Generators in coroutines Copy Chapter

Sometimes, it would be convenient to model a view of input_range with a coroutine which emits the elements one after
another asynchronously. From the consumer of the view's perspective, it can retrieve the elements by co_awaiting
the return value of the coroutine. From the coroutine's perspective, it is able to produce the elements multiple times
using co_yield without "leaving" the coroutine. A function producing a sequence of values can be named "generator".
But unlike the regular coroutine which returns a single seastar::future<T>, a generator should return
seastar::coroutine::experimental::generator<T, Container>. Where T is the type of the elements, while Container
is a template, which is used to store the elements. Because, underneath of Seastar's generator implementation, a
bounded buffer is used for holding the elements not yet retrieved by the consumer, there is a design decision to make --
what kind of container should be used, and what its maximum size should be. To define the bounded buffer, developers
need to:

1. specify the container type via the second template parameter of the generator
2. specify the size of the bounded buffer by passing the size as the first parameter of the generator coroutine.
   The size must have the type seastar::coroutine::experimental::buffer_size_t.

But there is an exception, if the buffer's size is one, we assume that the programmer is likely to use std::optional
for the bounded buffer, so it's not required to pass the maximum size of the buffer as the first parameter in this case.
But if a coroutine uses std::optional as its buffer, and its function signature still lists the size as its first
parameter, it will not break anything, because the underlying implementation simply ignores this parameter.

Following is an example

#include <seastar/core/circular_buffer.hh>
#include <seastar/core/coroutine.hh>
#include <seastar/core/sleep.hh>
#include <seastar/coroutine/generator.hh>

seastar::future<Preprocessed> prepare_ingredients(Ingredients&&);
seastar::future<Dish> cook_a_dish(Preprocessed&&);
seastar::future<> consume_a_dish(Dish&&);

seastar::coroutine::experimental::generator<Dish, seastar::circular_buffer>
make_dishes(coroutine::experimental::buffer_size_t max_dishes_on_table,
            Ingredients&& ingredients) {
    while (ingredients) {
        auto some_ingredients = ingredients.alloc();
        auto preprocessed = co_await prepare_ingredients(std::move(some_ingredients));
        co_yield co_await cook_a_dish(std::move(preprocessed));
    }
}

seastar::future<> have_a_dinner(unsigned max_dishes_on_table) {
    Ingredients ingredients;
    auto dishes = make_dishes(coroutine::experimental::buffer_size_t{max_dishes_on_table},
                              std::move(ingredients));
    while (auto dish = co_await dishes()) {
        co_await consume_a_dish(std::move(dish));
    }
}

In this hypothetical kitchen, a chef and a diner are working in parallel. Instead of preparing
all dishes beforehand, the chef cooks the dishes while the diner is consuming them one after another.
Under most circumstances, neither the chef nor the diner is blocked by the other. The dishes are buffered
using the specified seastar::circular_buffer<Dish>. But if the diner
is so slow that there are max_dishes_on_table dishes left on the table, the chef waits
until the number of dishes is less than this setting. Please note, as explained above, despite that this
parameter is not referenced by the coroutine's body, it is actually passed to the generator's promise
constructor, which in turn creates the buffer, as we are not using std::optional here. On the other hand,
if there are no dishes on the table, the diner waits for the chef to prepare new ones.

Note that generator<T, Container> is still in an early stage of development,
and its public interface is subject to change before it stabilizes.

Exceptions in coroutines Copy Chapter

Coroutines automatically translate exceptions to futures and back.

Calling co_await foo(), when foo() returns an exceptional future, will throw the exception carried by the future.

Similarly throwing within a coroutine will cause the coroutine to return an exceptional future.

Example:

#include <seastar/core/coroutine.hh>

seastar::future<> function_returning_an_exceptional_future();

seastar::future<> exception_handling() {
    try {
        co_await function_returning_an_exceptional_future();
    } catch (...) {
        // exception will be handled here
    }
    throw 3; // will be captured by coroutine and returned as
             // an exceptional future
}

Both throw and std::rethrow_exception() involve managing stack unwinding and exception objects, potentially
impacting performance. Additionally, the C++ standard permits std::rethrow_exception() to create a copy of the
exception object, introducing further overhead. Fortunately, in certain cases, exceptions can also be propagated
directly, without throwing or rethrowing them. There are multiple facilities for this.
coroutine::try_future can be used to propagate exceptions automatically from a called asynchronous function.
It is analogous to Rust's try operator.
If the awaited function resolves with an exception, it is automatically propagated to the coroutine's waiter
without throwing. If manual error handling is needed, one can use coroutine::as_future and coroutine::exception.
coroutine::as_future is analogous to future<>::then_wrapped(). It returns a ready future, allowing the code to
probe for an exception without an expensive throw and catch. The exception can then be propagated directly using
the coroutine::exception wrapper. coroutine::exception only works for coroutines which return future<T>, not
future<>, due to compiler limitations. In particular, the example below won't compile if the return type is
changed to future<>.

Example:

seastar::future<int> exception_propagating() {
    // Will automatically propagate the exception, if there is one.
    auto result = co_await seastar::coroutine::try_future(
        function_returning_an_exceptional_future()
    );

    // Will extract the ready future. Will not result in throwing, even if the future is exceptional.
    auto fut = co_await seastar::coroutine::as_future(
        function_returning_an_exceptional_future()
    );
    if (fut.failed()) {
        // Saved exception pointer can be propagated without rethrowing
        co_return seastar::coroutine::exception(fut.get_exception());
    }

    co_return seastar::coroutine::exception(std::make_exception_ptr(3)); // Custom exceptions can be propagated without throwing
}

Concurrency in coroutines Copy Chapter

The co_await operator allows for simple sequential execution. Multiple coroutines can execute in parallel, but each coroutine has only one outstanding computation at a time.

The seastar::coroutine::all class template allows a coroutine to fork into several concurrently executing sub-coroutines (or Seastar fibers, see below) and join again when they complete. Consider this example:

#include <seastar/core/coroutines.hh>
#include <seastar/coroutine/all.hh>

seastar::future<int> read(int key);

seastar::future<int> parallel_sum(int key1, int key2) {
    int [a, b] = co_await seastar::coroutine::all(
        [&] {
            return read(key1);
        },
        [&] {
            return read(key2);
        }
    );
    co_return a + b;
}

Here, two read() calls are launched concurrently. The coroutine is paused until both reads complete, and the values returned are assigned to a and b. If read(key) is an operation that involves I/O, then the concurrent execution will complete sooner than if we co_awaited each call separately, since I/O can be overlapped.

Note that all waits for all of its sub-computations, even if some throw an exception. If an exception is thrown, it is propagated to the calling coroutine.

The seastar::coroutine::parallel_for_each class template allows a coroutine to fork into several concurrently executing function invocations (or Seastar fibers, see below) over a range of elements and join again when they complete. Consider this example:

#include <seastar/core/coroutines.hh>
#include <seastar/coroutine/parallel_for_each.hh>

seastar::future<bool> all_exist(std::vector<seastar::sstring> filenames) {
    bool res = true;
    co_await seastar::coroutine::parallel_for_each(filenames, [&res] (const seastar::sstring& name) -> seastar::future<> {
        res &= co_await seastar::file_exists(name);
    });
    co_return res;
}

Here, the lambda function passed to parallel_for_each is launched concurrently for each element in the filenames vector. The coroutine is paused until all calls complete.

Breaking up long running computations Copy Chapter

Seastar is generally used for I/O, and coroutines usually launch I/O operations and consume their results, with little computation in between. But occasionally a long running computation is needed, and this risks preventing the reactor from performing I/O and scheduling other tasks.

A coroutine will automatically yield in a co_await expression; but in a computation we do not co_await anything. We can use the seastar::coroutine::maybe_yield class in such cases:

#include <seastar/coroutine/maybe_yield>

seastar::future<int> long_loop(int n) {
    float acc = 0;
    for (int i = 0; i < n; ++i) {
        acc += std::sin(float(i));
        // Give the Seastar reactor opportunity to perform I/O or schedule
        // other tasks.
        co_await seastar::coroutine::maybe_yield();
    }
    co_return acc;
}

Bypassing preemption checks in coroutines Copy Chapter

By default, co_await-ing a future performs a preemption check, and will suspend if the task quota is already depleted. However, in certain cases it might be useful to be able to assume that awaiting a ready future will not yield.
For such cases, it's possible to explicitly bypass the preemption check:

#include <seastar/core/coroutine.hh>

struct resource;
seastar::future<int> compute_always_ready(int i, resource& r);

seastar::future<int> accumulate(int n, resource& important_resource) {
    float acc = 0;
    for (int i = 0; i < n; ++i) {
        // This await will not yield the control, so we're sure that nobody will
        // be able to touch important_resource while we accumulate all the results.
        acc += co_await seastar::coroutine::without_preemption_check(compute_always_ready(i, important_resource));
    }
    co_return acc;
}

Continuations

Capturing state in continuations Copy Chapter

We've already seen that Seastar continuations are lambdas, passed to the then() method of a future. In the examples we've seen so far, lambdas have been nothing more than anonymous functions. But C++11 lambdas have one more trick up their sleeve, which is extremely important for future-based asynchronous programming in Seastar: Lambdas can capture state. Consider the following example:

#include <seastar/core/sleep.hh>
#include <iostream>

seastar::future<int> incr(int i) {
    using namespace std::chrono_literals;
    return seastar::sleep(10ms).then([i] { return i + 1; });
}

seastar::future<> f() {
    return incr(3).then([] (int val) {
        std::cout << "Got " << val << "\n";
    });
}

The future operation incr(i) takes some time to complete (it needs to sleep a bit first), and during that time it needs to save the i value it is working on. In early event-driven programming models, the programmer needed to explicitly define an object to hold this state and manage all such objects. C++ lambdas make this much simpler: the capture syntax "[i]" in the example above means that the value of i, as it existed when incr() was called, is captured in the lambda. A lambda is not just a function; it is an object with both code and data. In essence, the compiler automatically creates the state object, so we neither need to define it nor keep track of it (it is saved with the continuation when the continuation is deferred and deleted automatically after the continuation runs).

One implementation detail worth understanding is that when a continuation has captured state and is run immediately, this capture incurs no runtime overhead. However, when the continuation cannot be run immediately (because the future is not yet ready) and needs to be saved till later, memory needs to be allocated on the heap for this data, and the continuation's captured data needs to be copied there. This has runtime overhead, but it is unavoidable, and is very small compared to the related overhead in the threaded programming model (in a threaded program, this sort of state usually resides on the stack of the blocked thread, but the stack is much larger than our tiny capture state, takes up a lot of memory and causes a lot of cache pollution on context switches between those threads).

In the above example, we captured i by value - i.e., a copy of the value of i was saved into the continuation. C++ has two additional capture options: capturing by reference and capturing by move:

Using capture-by-reference in a continuation is usually a mistake, and can lead to serious bugs. For example, if in the above example we captured a reference to i, instead of copying it,

seastar::future<int> incr(int i) {
    using namespace std::chrono_literals;
    // Oops, the "&" below is wrong:
    return seastar::sleep(10ms).then([&i] { return i + 1; });
}

this would have meant that the continuation would contain the address of i, not its value. But i is a stack variable, and the incr() function returns immediately, so when the continuation eventually gets to run, long after incr() returns, this address will contain unrelated content.

An exception to the capture-by-reference-is-usually-a-mistake rule is the do_with() idiom, which we will introduce later. This idiom ensures that an object lives throughout the life of the continuation, and makes capture-by-reference possible, and very convenient.

Using capture-by-move in continuations is also very useful in Seastar applications. By moving an object into a continuation, we transfer ownership of this object to the continuation, and make it easy for the object to be automatically deleted when the continuation ends. For example, consider a traditional function taking a std::unique_ptr<T>.

int do_something(std::unique_ptr<T> obj) {
     // do some computation based on the contents of obj, let's say the result is 17
     return 17;
     // at this point, obj goes out of scope so the compiler delete()s it.

By using unique_ptr in this way, the caller passes an object to the function, but tells it the object is now its exclusive responsibility - and when the function is done with the object, it automatically deletes it. How do we use unique_ptr in a continuation? The following won't work:

seastar::future<int> slow_do_something(std::unique_ptr<T> obj) {
    using namespace std::chrono_literals;
    // The following line won't compile...
    return seastar::sleep(10ms).then([obj] () mutable { return do_something(std::move(obj)); });
}

The problem is that a unique_ptr cannot be passed into a continuation by value, as this would require copying it, which is forbidden because it violates the guarantee that only one copy of this pointer exists. We can, however, move obj into the continuation:

seastar::future<int> slow_do_something(std::unique_ptr<T> obj) {
    using namespace std::chrono_literals;
    return seastar::sleep(10ms).then([obj = std::move(obj)] () mutable {
        return do_something(std::move(obj));
    });
}

Here the use of std::move() causes obj's move-assignment is used to move the object from the outer function into the continuation. The notion of move (move semantics), introduced in C++11, is similar to a shallow copy followed by invalidating the source copy (so that the two copies do not co-exist, as forbidden by unique_ptr). After moving obj into the continuation, the top-level function can no longer use it (in this case it's of course ok, because we return anyway).

The [obj = ...] init-capture syntax used here was introduced in C++14 and is available in all C++ standards supported by Seastar.

The extra () mutable syntax was needed here because by default when C++ captures a value (in this case, the value of std::move(obj)) into a lambda, it makes this value read-only, so our lambda cannot, in this example, move it again. Adding mutable removes this artificial restriction.

Evaluation order considerations in pre-C++17 code Copy Chapter

C++14 (and below) does not guarantee that lambda captures in continuations will be evaluated after the futures they relate to are evaluated
(See https://en.cppreference.com/w/cpp/language/eval_order).

Consequently, avoid the programming pattern below:

return do_something(obj).then([obj = std::move(obj)] () mutable {
        return do_something_else(std::move(obj));
    });

In the example above, [obj = std::move(obj)] might be evaluated before do_something(obj) is called, potentially leading to use-after-move of obj.

To guarantee the desired evaluation order, the expression above may be broken into separate statements as follows:

auto fut = do_something(obj);
    return fut.then([obj = std::move(obj)] () mutable {
        return do_something_else(std::move(obj));
    });

This was changed in C++17. The expression that creates the object the function then is called on (the future) is evaluated before all the arguments to the function, so this style is not required in C++17 and above.

Chaining continuations Copy Chapter

TODO: We already saw chaining example in slow() above. talk about the return from then, and returning a future and chaining more thens.

Handling exceptions

An exception thrown in a continuation is implicitly captured by the system and stored in the future. A future that stores such an exception is similar to a ready future in that it can cause its continuation to be launched, but it does not contain a value -- only the exception.

Calling .then() on such a future skips over the continuation, and transfers the exception for the input future (the object on which .then() is called) to the output future (.then()'s return value).

This default handling parallels normal exception behavior -- if an exception is thrown in straight-line code, all following lines are skipped:

line1();
line2(); // throws!
line3(); // skipped

is similar to

return line1().then([] {
    return line2(); // throws!
}).then([] {
    return line3(); // skipped
});

Usually, aborting the current chain of operations and returning an exception is what's needed, but sometimes more fine-grained control is required. There are several primitives for handling exceptions:

1. .then_wrapped(): instead of passing the values carried by the future into the continuation, .then_wrapped() passes the input future to the continuation. The future is guaranteed to be in ready state, so the continuation can examine whether it contains a value or an exception, and take appropriate action.
2. .finally(): similar to a Java finally block, a .finally() continuation is executed whether or not its input future carries an exception. The result of the finally continuation is its input future, so .finally() can insert code that is executed unconditionally but otherwise does not alter the flow.

The following example illustrates usage of then_wrapped and finally:

#include <seastar/core/future.hh>
#include <iostream>
#include <exception>

seastar::future<> pass() {
    std::cout << "I passed!!!" << std::endl;
    return seastar::make_ready_future<>();
}

seastar::future<> fail() {
    std::cout << "I failed." << std::endl;
    return seastar::make_exception_future<>(std::exception());
}

seastar::future<> f() {
    return pass().then([] {
        std::cout << "Oh no! I'm gonna fail!" << std::endl;
        return fail(); // throws
    }).then([] () { // skipped
        std::cout << "If I got to this place I will pass!" << std::endl;
        return pass();
    }).then_wrapped([] (seastar::future<> f) {
        if (f.failed()) {
            std::cout << "The input future failed!" << std::endl;
            return f;
        }

        std::cout << "If I got to this place I will pass!" << std::endl;
        return pass();
    }).finally([] () {
        std::cout << "This code will run, regardless of any exceptions" << std::endl;
    });
}

This time the output will be

I passed!!!
Oh no! I'm gonna fail!
I failed.
The input future failed!
This code will run, regardless of any exceptions
ERROR [...] Exiting on unhandled exception: std::exception (std::exception)

TODO: Also mention handle_exception - although perhaps delay that to a later chapter?

Exceptions vs. exceptional futures Copy Chapter

An asynchronous function can fail in one of two ways: It can fail immediately, by throwing an exception, or it can return a future which will eventually fail (resolve to an exception). These two modes of failure appear similar to the uninitiated, but behave differently when attempting to handle exceptions using finally(), handle_exception(), or then_wrapped(). For example, consider the code:

#include <seastar/core/future.hh>
#include <iostream>
#include <exception>

class my_exception : public std::exception {
    virtual const char* what() const noexcept override { return "my exception"; }
};

seastar::future<> fail() {
    return seastar::make_exception_future<>(my_exception());
}

seastar::future<> f() {
    return fail().finally([] {
        std::cout << "cleaning up\n";
    });
}

This code will, as expected, print the "cleaning up" message - the asynchronous function fail() returns a future which resolves to a failure, and the finally() continuation is run despite this failure, as expected.

Now consider that in the above example we had a different definition for fail():

seastar::future<> fail() {
    throw my_exception();
}

Here, fail() does not return a failing future. Rather, it fails to return a future at all! The exception it throws stops the entire function f(), so the finally() continuation is not attached to the future (which was never returned) and never runs. The "cleaning up" message is not printed.

To reduce the chance of such errors, we recommend that asynchronous functions always return a failed future rather than throw an exception. If an asynchronous function calls another function before returning a future, and that second function might throw, it should use try/catch to catch the exception and convert it into a failed future:

void inner() {
    throw my_exception();
}
seastar::future<> fail() {
    try {
        inner();
    } catch(...) {
        return seastar::current_exception_as_future();
    }
    return seastar::make_ready_future<>();
}

Here, fail() catches the exception thrown by inner(), whatever it might be, and returns a failed future with that failure. Written this way, the finally() continuation will be reached, and the "cleaning up" message printed.

The seastar::current_exception_as_future() function is equivalent
to seastar::make_exception_future(std::current_exception()), but
expands to less code.

Despite this recommendation that asynchronous functions avoid throwing, some asynchronous functions do throw exceptions in addition to returning exceptional futures. A common example are functions which allocate memory and throw std::bad_alloc when running out of memory, instead of returning a future. The future<> seastar::semaphore::wait() method is one such function: It returns a future which may be exceptional if the semaphore was broken() or the wait timed out, but may also throw an exception when failing to allocate memory it needs to hold the list of waiters.
Therefore, unless a function --- including asynchronous functions --- is explicitly tagged "noexcept", the application should be prepared to handle exceptions thrown from it. In modern C++, code usually uses RAII to be exception-safe without sprinkling it with try/catch. seastar::defer() is a RAII-based idiom that ensures that some cleanup code is run even if an exception is thrown.

Seastar has a convenient generic function, futurize_invoke(), which can be useful here. futurize_invoke(func, args...) runs a function which may return either a future value or an immediate value, and in both cases convert the result into a future value. futurize_invoke() also converts an immediate exception thrown by the function, if any, into a failed future, just like we did above. So using futurize_invoke() we can make the above example work even if fail() did throw exceptions:

seastar::future<> fail() {
    throw my_exception();
}
seastar::future<> f() {
    return seastar::futurize_invoke(fail).finally([] {
        std::cout << "cleaning up\n";
    });
}

Note that most of this discussion becomes moot if the risk of exception is inside a continuation. Consider the following code:

seastar::future<> f() {
    return seastar::sleep(1s).then([] {
        throw my_exception();
    }).finally([] {
        std::cout << "cleaning up\n";
    });
}

Here, the lambda function of the first continuation does throw an exception instead of returning a failed future. However, we do not have the same problem as before, which only happened because an asynchronous function threw an exception before returning a valid future. Here, f() does return a valid future immediately - the failure will only be known later, after sleep() resolves. The message in finally() will be printed. The methods which attach continuations (such as then() and finally()) run the continuation the same way, so continuation functions may return immediate values or, in this case, throw an immediate exception, and still work properly.

Lifetime management

An asynchronous function starts an operation which may continue long after the function returns: The function itself returns a future<T> almost immediately, but it may take a while until this future is resolved.

When such an asynchronous operation needs to operate on existing objects, or to use temporary objects, we need to worry about the lifetime of these objects: We need to ensure that these objects do not get destroyed before the asynchronous function completes (or it will try to use the freed object and malfunction or crash), and also ensure that the objects are finally destroyed when they are no longer needed (otherwise we will have a memory leak).
Seastar offers a variety of mechanisms for safely and efficiently keeping objects alive for the right duration. In this section we will explore these mechanisms, and when to use each mechanism.

Passing ownership to continuation Copy Chapter

The most straightforward way to ensure that an object is alive when a continuation runs and is destroyed afterward is to pass its ownership to the continuation. When the continuation owns the object, the object is kept until the continuation runs and is destroyed as soon as the continuation is no longer needed (i.e., after it runs or is skipped because an earlier future completed exceptionally).

We already saw above that the way for a continuation to get ownership of an object is through capturing:

seastar::future<> slow_incr(int i) {
    return seastar::sleep(10ms).then([i] { return i + 1; });
}

Here the continuation captures the value of i. In other words, the continuation includes a copy of i. When the continuation runs 10ms later, it will have access to this value, and as soon as the continuation finishes its object is destroyed, together with its captured copy of i. The continuation owns this copy of i.

Capturing by value as we did here - making a copy of the object we need in the continuation - is useful mainly for very small objects such as the integer in the previous example. Other objects are expensive to copy, or sometimes even cannot be copied. For example, the following is not a good idea:

seastar::future<> slow_op(std::vector<int> v) {
    // this makes another copy of v:
    return seastar::sleep(10ms).then([v] { /* do something with v */ });
}

This would be inefficient - as the vector v, potentially very long, will be copied and the copy will be saved in the continuation. In this example, there is no reason to copy v - it was anyway passed to the function by value and will not be used again after capturing it into the continuation, as right after the capture, the function returns and destroys its copy of v.

For such cases, C++14 allows moving the object into the continuation:

seastar::future<> slow_op(std::vector<int> v) {
    // v is not copied again, but instead moved:
    return seastar::sleep(10ms).then([v = std::move(v)] { /* do something with v */ });
}

Now, instead of copying the object v into the continuation, it is moved into the continuation. The C++11-introduced move constructor moves the vector's data into the continuation and clears the original vector. Moving is a quick operation - for a vector it only requires copying a few small fields such as the pointer to the data. As before, once the continuation is dismissed the vector is destroyed - and its data array (which was moved in the move operation) is finally freed.

TODO: talk about temporary_buffer as an example of an object designed to be moved in this way.

In some cases, moving the object is undesirable. For example, some code keeps references to an object or one of its fields and the references become invalid if the object is moved. In some complex objects, even the move constructor is slow. For these cases, C++ provides the useful wrapper std::unique_ptr<T>. A unique_ptr<T> object owns an object of type T allocated on the heap. When a unique_ptr<T> is moved, the object of type T is not touched at all - just the pointer to it is moved. An example of using std::unique_ptr<T> in capture is:

seastar::future<> slow_op(std::unique_ptr<T> p) {
    return seastar::sleep(10ms).then([p = std::move(p)] { /* do something with *p */ });
}

std::unique_ptr<T> is the standard C++ mechanism for passing unique ownership of an object to a function: The object is only owned by one piece of code at a time, and ownership is transferred by moving the unique_ptr object. A unique_ptr cannot be copied: If we try to capture p by value, not by move, we will get a compilation error.

Keeping ownership at the caller Copy Chapter

The technique we described above - giving the continuation ownership of the object it needs to work on - is powerful and safe. But often it becomes hard and verbose to use. When an asynchronous operation involves not just one continuation but a chain of continuations that each needs to work on the same object, we need to pass the ownership of the object between each successive continuation, which can become inconvenient. It is especially inconvenient when we need to pass the same object into two separate asynchronous functions (or continuations) - after we move the object into one, the object needs to be returned so it can be moved again into the second. E.g.,

seastar::future<> slow_op(T o) {
    return seastar::sleep(10ms).then([o = std::move(o)] {
        // first continuation, doing something with o
        ...
        // return o so the next continuation can use it!
        return std::move(o);
    }).then([](T o) {
        // second continuation, doing something with o
        ...
    });
}

This complexity arises because we wanted asynchronous functions and continuations to take the ownership of the objects they operated on. A simpler approach would be to have the caller of the asynchronous function continue to be the owner of the object, and just pass references to the object to the various other asynchronous functions and continuations which need the object. For example:

seastar::future<> slow_op(T& o) {           // <-- pass by reference
    return seastar::sleep(10ms).then([&o] {// <-- capture by reference
        // first continuation, doing something with o
        ...
    }).then([&o]) {                        // <-- another capture by reference
        // second continuation, doing something with o
        ...
    });
}

This approach raises a question: The caller of slow_op is now responsible for keeping the object o alive while the asynchronous code started by slow_op needs this object. But how will this caller know how long this object is actually needed by the asynchronous operation it started?

The most reasonable answer is that an asynchronous function may need access to its parameters until the future it returns is resolved - at which point the asynchronous code completes and no longer needs access to its parameters. We therefore recommend that Seastar code adopt the following convention:

Whenever an asynchronous function takes a parameter by reference, the caller must ensure that the referred object lives until the future returned by the function is resolved.

Note that this is merely a convention suggested by Seastar, and unfortunately nothing in the C++ language enforces it. C++ programmers in non-Seastar programs often pass large objects to functions as a const reference just to avoid a slow copy, and assume that the called function will not save this reference anywhere. But in Seastar code, that is a dangerous practice because even if the asynchronous function did not intend to save the reference anywhere, it may end up doing it implicitly by passing this reference to another function and eventually capturing it in a continuation.

It would be nice if future versions of C++ could help us catch incorrect uses of references. Perhaps we could have a tag for a special kind of reference, an "immediate reference" which a function can use use immediately (i.e, before returning a future), but cannot be captured into a continuation.

With this convention in place, it is easy to write complex asynchronous functions functions like slow_op which pass the object around, by reference, until the asynchronous operation is done. But how does the caller ensure that the object lives until the returned future is resolved? The following is wrong:

seastar::future<> f() {
    T obj; // wrong! will be destroyed too soon!
    return slow_op(obj);
}

It is wrong because the object obj here is local to the call of f, and is destroyed as soon as f returns a future - not when this returned future is resolved! The correct thing for a caller to do would be to create the object obj on the heap (so it does not get destroyed as soon as f returns), and then run slow_op(obj) and when that future resolves (i.e., with .finally()), destroy the object.

Seastar provides a convenient idiom, do_with() for doing this correctly:

seastar::future<> f() {
    return seastar::do_with(T(), [] (auto& obj) {
        // obj is passed by reference to slow_op, and this is fine:
        return slow_op(obj);
    }
}

do_with will do the given function with the given object alive.

do_with saves the given object on the heap, and calls the given lambda with a reference to the new object. Finally it ensures that the new object is destroyed after the returned future is resolved. Usually, do_with is given an rvalue, i.e., an unnamed temporary object or an std::move()ed object, and do_with moves that object into its final place on the heap. do_with returns a future which resolves after everything described above is done (the lambda's future is resolved and the object is destroyed).

For convenience, do_with can also be given multiple objects to hold alive. For example here we create two objects and hold alive them until the future resolves:

seastar::future<> f() {
    return seastar::do_with(T1(), T2(), [] (auto& obj1, auto& obj2) {
        return slow_op(obj1, obj2);
    }
}

While do_with can the lifetime of the objects it holds, if the user accidentally makes copies of these objects, these copies might have the wrong lifetime. Unfortunately, a simple typo like forgetting an "&" can cause such accidental copies. For example, the following code is broken:

seastar::future<> f() {
    return seastar::do_with(T(), [] (T obj) { // WRONG: should be T&, not T
        return slow_op(obj);
    }
}

In this wrong snippet, obj is mistakenly not a reference to the object which do_with allocated, but rather a copy of it - a copy which is destroyed as soon as the lambda function returns, rather than when the future it returns resolved. Such code will most likely crash because the object is used after being freed. Unfortunately the compiler will not warn about such mistakes. Users should get used to always using the type "auto&" with do_with - as in the above correct examples - to reduce the chance of such mistakes.

For the same reason, the following code snippet is also wrong:

seastar::future<> slow_op(T obj); // WRONG: should be T&, not T
seastar::future<> f() {
    return seastar::do_with(T(), [] (auto& obj) {
        return slow_op(obj);
    }
}

Here, although obj was correctly passed to the lambda by reference, we later accidentally passed slow_op() a copy of it (because here slow_op takes the object by value, not by reference), and this copy will be destroyed as soon as slow_op returns, not waiting until the returned future resolves.

When using do_with, always remember it requires adhering to the convention described above: The asynchronous function which we call inside do_with must not use the objects held by do_with after the returned future is resolved. It is a serious use-after-free bug for an asynchronous function to return a future which resolves while still having background operations using the do_with()ed objects.

In general, it is rarely a good idea for an asynchronous function to resolve while leaving behind background operations - even if those operations do not use the do_with()ed objects. Background operations that we do not wait for may cause us to run out of memory (if we don't limit their number) and make it difficult to shut down the application cleanly.

Sharing ownership (reference counting) Copy Chapter

In the beginning of this chapter, we already noted that capturing a copy of an object into a continuation is the simplest way to ensure that the object is alive when the continuation runs and destroyed afterwards. However, complex objects are often expensive (in time and memory) to copy. Some objects cannot be copied at all, or are read-write and the continuation should modify the original object, not a new copy. The solution to all these issues are reference counted, a.k.a. shared objects:

A simple example of a reference-counted object in Seastar is a seastar::file, an object holding an open file object (we will introduce seastar::file in a later section). A file object can be copied, but copying does not involve copying the file descriptor (let alone the file). Instead, both copies point to the same open file, and a reference count is increased by 1. When a file object is destroyed, the file's reference count is decreased by one, and only when the reference count reaches 0 the underlying file is actually closed.

The fact that file objects can be copied very quickly and all copies actually point to the same file, make it very convenient to pass them to asynchronous code; For example,

seastar::future<uint64_t> slow_size(file f) {
    return seastar::sleep(10ms).then([f] {
        return f.size();
    });
}

Note how calling slow_size is as simple as calling slow_size(f), passing a copy of f, without needing to do anything special to ensure that f is only destroyed when no longer needed. That simply happens naturally when nothing refers to f any more.

You may wonder why return f.size() in the above example is safe: Doesn't it start an asynchronous operation on f (the file's size may be stored on disk, so not immediately available), and f may be immediately destroyed when we return and nothing keeps holding a copy of f? If f is really the last reference, that is indeed a bug, but there is another one: the file is never closed. The assumption that makes the code valid is that there is another reference to f that will be used to close it. The close member function holds on to the reference count of that object, so it continues to live even if nothing else keeps holding on to it. Since all futures produced by a file object complete before it is closed, all that is needed for correctness is to remember to always close files.

The reference counting has a run-time cost, but it is usually very small; It is important to remember that Seastar objects are always used by a single CPU only, so the reference-count increment and decrement operations are not the slow atomic operations often used for reference counting, but just regular CPU-local integer operations. Moreover, judicious use of std::move() and the compiler's optimizer can reduce the number of unnecessary back-and-forth increment and decrement of the reference count.

C++11 offers a standard way of creating reference-counted shared objects - using the template std::shared_ptr<T>. A shared_ptr can be used to wrap any type into a reference-counted shared object like seastar::file above. However, the standard std::shared_ptr was designed with multi-threaded applications in mind so it uses slow atomic increment/decrement operations for the reference count which we already noted is unnecessary in Seastar. For this reason Seastar offers its own single-threaded implementation of this template, seastar::shared_ptr<T>. It is similar to std::shared_ptr<T> except no atomic operations are used.

Additionally, Seastar also provides an even lower overhead variant of shared_ptr: seastar::lw_shared_ptr<T>. The full-featured shared_ptr is complicated by the need to support polymorphic types correctly (a shared object created of one class, and accessed through a pointer to a base class). It makes shared_ptr need to add two words to the shared object, and two words to each shared_ptr copy. The simplified lw_shared_ptr - which does not support polymorphic types - adds just one word in the object (the reference count) and each copy is just one word - just like copying a regular pointer. For this reason, the light-weight seastar::lw_shared_ptr<T> should be preferred when possible (T is not a polymorphic type), otherwise seastar::shared_ptr<T>. The slower std::shared_ptr<T> should never be used in sharded Seastar applications.

Saving objects on the stack Copy Chapter

Wouldn't it be convenient if we could save objects on a stack just like we normally do in synchronous code? I.e., something like:

int i = ...;
seastar::sleep(10ms).get();
return i;

Seastar allows writing such code, by using a seastar::thread object which comes with its own stack. A complete example using a seastar::thread might look like this:

seastar::future<> slow_incr(int i) {
    return seastar::async([i] {
        seastar::sleep(10ms).get();
        // We get here after the 10ms of wait, i is still available.
        return i + 1;
    });
}

We present seastar::thread, seastar::async() and seastar::future::get() in the [seastar::thread] section.

Advanced futures

Futures and interruption Copy Chapter

TODO: A future, e.g., sleep(10s) cannot be interrupted. So if we need to, the promise needs to have a mechanism to interrupt it. Mention pipe's close feature, semaphore stop feature, etc.

Futures are single use Copy Chapter

TODO: Talk about if we have a future variable, as soon as we get() or then() it, it becomes invalid - we need to store the value somewhere else. Think if there's an alternative we can suggest

Fibers

Seastar continuations are normally short, but often chained to one another, so that one continuation does a bit of work and then schedules another continuation for later. Such chains can be long, and often even involve loopings - see the following section, "Loops". We call such chains "fibers" of execution.

These fibers are not threads - each is just a string of continuations - but they share some common requirements with traditional threads. For example, we want to avoid one fiber getting starved while a second fiber continuously runs its continuations one after another. As another example, fibers may want to communicate - e.g., one fiber produces data that a second fiber consumes, and we wish to ensure that both fibers get a chance to run, and that if one stops prematurely, the other doesn't hang forever.

TODO: Mention fiber-related sections like loops, semaphores, gates, pipes, etc.

Loops

A majority of time-consuming computations involve using loops. Seastar provides several primitives for expressing them in a way that composes nicely with the future/promise model. A very important aspect of Seastar loop primitives is that each iteration is followed by a preemption point, thus allowing other tasks to run in between iterations.

repeat Copy Chapter

A loop created with repeat executes its body until it receives a stop_iteration object, which informs if the iteration should continue (stop_iteration::no) or stop (stop_iteration::yes). Next iteration will be launched only after the first one has finished. The loop body passed to repeat is expected to have a future<stop_iteration> return type.

seastar::future<int> recompute_number(int number);

seastar::future<> push_until_100(seastar::lw_shared_ptr<std::vector<int>> queue, int element) {
    return seastar::repeat([queue, element] {
        if (queue->size() == 100) {
            return make_ready_future<stop_iteration>(stop_iteration::yes);
        }
        return recompute_number(element).then([queue] (int new_element) {
            queue->push_back(new_element);
            return stop_iteration::no;
        });
    });
}

do_until Copy Chapter

Do until is a close relative of repeat, but it uses an explicitly passed condition to decide whether it should stop iterating. The above example could be expressed with do_until as follows:

seastar::future<int> recompute_number(int number);

seastar::future<> push_until_100(seastar::lw_shared_ptr<std::vector<int>> queue, int element) {
    return seastar::do_until([queue] { return queue->size() == 100; }, [queue, element] {
        return recompute_number(element).then([queue] (int new_element) {
            queue->push_back(new_element);
        });
    });
}

Note that the loop body is expected to return a future<>, which allows composing complex continuations inside the loop.

do_for_each Copy Chapter

A do_for_each is an equivalent of a for loop in Seastar world. It accepts a range (or a pair of iterators) and a function body, which it applies to each argument, in order, one by one. The next iteration will be launched only after the first one has finished, as was the case with repeat. As usual, do_for_each expects its loop body to return a future<>.

seastar::future<> append(seastar::lw_shared_ptr<std::vector<int>> queue1, seastar::lw_shared_ptr<std::vector<int>> queue2) {
    return seastar::do_for_each(queue2, [queue1] (int element) {
        queue1->push_back(element);
    });
}

seastar::future<> append_iota(seastar::lw_shared_ptr<std::vector<int>> queue1, int n) {
    return seastar::do_for_each(boost::make_counting_iterator<size_t>(0), boost::make_counting_iterator<size_t>(n), [queue1] (int element) {
        queue1->push_back(element);
    });
}

do_for_each accepts either an lvalue reference to a container or a pair of iterators. It implies that the responsibility to ensure that the container is alive during the whole loop execution belongs to the caller. If the container needs its lifetime prolonged, it can be easily achieved with do_with:

seastar::future<> do_something(int number);

seastar::future<> do_for_all(std::vector<int> numbers) {
    // Note that the "numbers" vector will be destroyed as soon as this function
    // returns, so we use do_with to guarantee it lives during the whole loop execution:
    return seastar::do_with(std::move(numbers), [] (std::vector<int>& numbers) {
        return seastar::do_for_each(numbers, [] (int number) {
            return do_something(number);
        });
    });
}

parallel_for_each Copy Chapter

Parallel for each is a high concurrency variant of do_for_each. When using parallel_for_each, all iterations are queued simultaneously - which means that there's no guarantee in which order they finish their operations.

seastar::future<> flush_all_files(seastar::lw_shared_ptr<std::vector<seastar::file>> files) {
    return seastar::parallel_for_each(files, [] (seastar::file f) {
        // file::flush() returns a future<>
        return f.flush();
    });
}

parallel_for_each is a powerful tool, as it allows spawning many tasks in parallel. It can be a great performance gain, but there are also caveats. First of all, too high concurrency may be troublesome - the details can be found in chapter Limiting parallelism of loops.

To restrict the concurrency of parallel_for_each by an integer number, use max_concurrent_for_each that is described below.
More details about dealing with parallelism can be found in chapter Limiting parallelism of loops.

Secondly, take note that the order in which iterations will be executed within a parallel_for_each loop is arbitrary - if a strict ordering is needed, consider using do_for_each instead.

TODO: map_reduce, as a shortcut (?) for parallel_for_each which needs to produce some results (e.g., logical_or of boolean results), so we don't need to create a lw_shared_ptr explicitly (or do_with).

TODO: See seastar commit "input_stream: Fix possible infinite recursion in consume()" for an example on why recursion is a possible, but bad, replacement for repeat(). See also my comment on https://groups.google.com/d/msg/seastar-dev/CUkLVBwva3Y/3DKGw-9aAQAJ on why Seastar's iteration primitives should be used over tail call optimization.

max_concurrent_for_each Copy Chapter

Max concurrent for each is a variant of parallel_for_each with restricted parallelism.
It accepts an additional parameter - max_concurrent - with which, up to max_concurrent iterations are queued simultaneously, with no guarantee in which order they finish their operations.

seastar::future<> flush_all_files(seastar::lw_shared_ptr<std::vector<seastar::file>> files, size_t max_concurrent) {
    return seastar::max_concurrent_for_each(files, max_concurrent, [] (seastar::file f) {
        return f.flush();
    });
}

Determining the maximum concurrency limit is out of the scope of this document.
It should typically be derived from the actual capabilities of the system the software is running on, like the number of parallel execution units or I/O channels, so to optimize utilization of resources without overwhelming the system.

when_all: Waiting for multiple futures

Above we've seen parallel_for_each(), which starts a number of asynchronous operations, and then waits for all to complete. Seastar has another idiom, when_all(), for waiting for several already-existing futures to complete.

The first variant of when_all() is variadic, i.e., the futures are given as separate parameters, the exact number of which is known at compile time. The individual futures may have different types. For example,

#include <seastar/core/sleep.hh>

future<> f() {
    using namespace std::chrono_literals;
    future<int> slow_two = sleep(2s).then([] { return 2; });
    return when_all(sleep(1s), std::move(slow_two),
                    make_ready_future<double>(3.5)
           ).discard_result();
}

This starts three futures - one which sleeps for one second (and doesn't return anything), one which sleeps for two seconds and returns the integer 2, and one which returns the double 3.5 immediately - and then waits for them. The when_all() function returns a future which resolves as soon as all three futures resolve, i.e., after two seconds. This future also has a value, which we shall explain below, but in this example, we simply waited for the future to resolve and discarded its value.

Note that when_all() accepts only rvalues, which can be temporaries (like the return value of an asynchronous function or make_ready_future) or a variable holding a future that has been passed to std::move().

The future returned by when_all() resolves to a tuple of futures which are already resolved, and contain the results of the three input futures. Continuing the above example,

future<> f() {
    using namespace std::chrono_literals;
    future<int> slow_two = sleep(2s).then([] { return 2; });
    return when_all(sleep(1s), std::move(slow_two),
                    make_ready_future<double>(3.5)
           ).then([] (auto tup) {
            std::cout << std::get<0>(tup).available() << "\n";
            std::cout << std::get<1>(tup).get() << "\n";
            std::cout << std::get<2>(tup).get() << "\n";
    });
}

The output of this program (which comes after two seconds) is 1, 2, 3.5: the first future in the tuple is available (but has no value), the second has the integer value 2, and the third a double value 3.5 - as expected.

One or more of the waited futures might resolve in an exception, but this does not change how when_all() works: It still waits for all the futures to resolve, each with either a value or an exception, and in the returned tuple some of the futures may contain an exception instead of a value. For example,

future<> f() {
    using namespace std::chrono_literals;
    future<> slow_success = sleep(1s);
    future<> slow_exception = sleep(2s).then([] { throw 1; });
    return when_all(std::move(slow_success), std::move(slow_exception)
           ).then([] (auto tup) {
            std::cout << std::get<0>(tup).available() << "\n";
            std::cout << std::get<1>(tup).failed() << "\n";
            std::get<1>(tup).ignore_ready_future();
    });
}

Both futures are available() (resolved), but the second has failed() (resulted in an exception instead of a value). Note how we called ignore_ready_future() on this failed future, because silently ignoring a failed future is considered a bug, and will result in an "Exceptional future ignored" error message. More typically, an application will log the failed future instead of ignoring it.

The above example demonstrate that when_all() is inconvenient and verbose to use properly. The results are wrapped in a tuple, leading to verbose tuple syntax, and uses ready futures which must all be inspected individually for an exception to avoid error messages.

So Seastar also provides an easier to use when_all_succeed() function. This function too returns a future which resolves when all the given futures have resolved. If all of them succeeded, it passes a tuple of the resulting values to continuation, without wrapping each of them in a future first. Sometimes, it could be tedious to unpack the tuple for consuming the resulting values. In that case, then_unpack() can be used in place of then(). then_unpack() unpacks the returned tuple and passes its elements to the following continuation as its parameters. If, however, one or more of the futures failed, when_all_succeed() resolves to a failed future, containing the exception from one of the failed futures. If more than one of the given future failed, one of those will be passed on (it is unspecified which one is chosen), and the rest will be silently ignored. For example,

using namespace seastar;
future<> f() {
    using namespace std::chrono_literals;
    return when_all_succeed(sleep(1s), make_ready_future<int>(2),
                    make_ready_future<double>(3.5)
            ).then_unpack([] (int i, double d) {
        std::cout << i << " " << d << "\n";
    });
}

Note how the integer and double values held by the futures are conveniently passed, individually to the continuation. Since sleep() does not contain a value, it is waited for, but no third value is passed to the continuation. That also means that if we when_all_succeed() on several future<> (without a value), the result is a future<tuple<>>:

using namespace seastar;
future<> f() {
    using namespace std::chrono_literals;
    return when_all_succeed(sleep(1s), sleep(2s), sleep(3s)).then_unpack([] {
        return make_ready_future<>();
    });
}

This example simply waits for 3 seconds (the maximum of 1, 2 and 3 seconds).

An example of when_all_succeed() with an exception:

using namespace seastar;
future<> f() {
    using namespace std::chrono_literals;
    return when_all_succeed(make_ready_future<int>(2),
                    make_exception_future<double>("oops")
            ).then_unpack([] (int i, double d) {
        std::cout << i << " " << d << "\n";
    }).handle_exception([] (std::exception_ptr e) {
        std::cout << "exception: " << e << "\n";
    });
}

In this example, one of the futures fails, so the result of when_all_succeed is a failed future, so the normal continuation is not run, and the handle_exception() continuation is done.

TODO: also explain when_all and when_all_succeed for vectors.

Semaphores

Seastar's semaphores are the standard computer-science semaphores, adapted for futures. A semaphore is a counter into which you can deposit units or take them away. Taking units from the counter may wait if not enough units are available.

Limiting parallelism with semaphores Copy Chapter

The most common use for a semaphore in Seastar is for limiting parallelism, i.e., limiting the number of instances of some code which can run in parallel. This can be important when each of the parallel invocations uses a limited resource (e.g., memory) so letting an unlimited number of them run in parallel can exhaust this resource.

Consider a case where an external source of events (e.g., an incoming network request) causes an asynchronous function g() to be called. Imagine that we want to limit the number of concurrent g() operations to 100. I.e., If g() is started when 100 other invocations are still ongoing, we want it to delay its real work until one of the other invocations has completed. We can do this with a semaphore:

seastar::future<> g() {
    static thread_local seastar::semaphore limit(100);
    return limit.wait(1).then([] {
        return slow(); // do the real work of g()
    }).finally([] {
        limit.signal(1);
    });
}

In this example, the semaphore starts with the counter at 100. The asynchronous operation slow() is only started when we can reduce the counter by one (wait(1)), and when slow() is done, either successfully or with exception, the counter is increased back by one (signal(1)). This way, when 100 operations have already started their work and have not yet finished, the 101st operation will wait, until one of the ongoing operations finishes and returns a unit to the semaphore. This ensures that at each time we have at most 100 concurrent slow() operations running in the above code.

Note how we used a static thread_local semaphore, so that all calls to g() from the same shard count towards the same limit; As usual, a Seastar application is sharded so this limit is separate per shard (CPU thread). This is usually fine, because sharded applications consider resources to be separate per shard.

Luckily, the above code happens to be exception safe: limit.wait(1) can throw an exception when it runs out of memory (keeping a list of waiters), and in that case the semaphore counter is not decreased but the continuations below are not run so it is not increased either. limit.wait(1) can also return an exceptional future when the semaphore is broken (we'll discuss this later) but in that case the extra signal() call is ignored. Finally, slow() may also throw, or return an exceptional future, but the finally() ensures the semaphore is still increased.

However, as the application code becomes more complex, it becomes harder to ensure that we never forget to call signal() after the operation is done, regardless of which code path or exceptions happen. As an example of what might go wrong, consider the following buggy code snippet, which differs subtly from the above one, and also appears, on first sight, to be correct:

seastar::future<> g() {
    static thread_local seastar::semaphore limit(100);
    return limit.wait(1).then([] {
        return slow().finally([] { limit.signal(1); });
    });
}

But this version is not exception safe: Consider what happens if slow() throws an exception before returning a future (this is different from slow() returning an exceptional future - we discussed this difference in the section about exception handling). In this case, we decreased the counter, but the finally() will never be reached, and the counter will never be increased back. There is a way to fix this code, by replacing the call to slow() with seastar::futurize_invoke(slow). But the point we're trying to make here is not how to fix buggy code, but rather that by using the separate semaphore::wait() and semaphore::signal() functions, you can very easily get things wrong.

For exception safety, in C++ it is generally not recommended to have separate resource acquisition and release functions. Instead, C++ offers safer mechanisms for acquiring a resource (in this case semaphore units) and later releasing it: lambda functions, and RAII ("resource acquisition is initialization"):

The lambda-based solution is a function seastar::with_semaphore() which is a shortcut for the code in the examples above:

seastar::future<> g() {
    static thread_local seastar::semaphore limit(100);
    return seastar::with_semaphore(limit, 1, [] {
        return slow(); // do the real work of g()
    });
}

with_semaphore(), like the earlier code snippets, waits for the given number of units from the semaphore, then runs the given lambda, and when the future returned by the lambda is resolved, with_semaphore() returns back the units to the semaphore. with_semaphore() returns a future which only resolves after all these steps are done.

The function seastar::get_units() is more general. It provides an exception-safe alternative to seastar::semaphore's separate wait() and signal() methods, based on C++'s RAII philosophy: The function returns an opaque units object, which while held, keeps the semaphore's counter decreased - and as soon as this object is destructed, the counter is increased back. With this interface you cannot forget to increase the counter, or increase it twice, or increase without decreasing: The counter will always be decreased once when the units object is created, and if that succeeded, increased when the object is destructed. When the units object is moved into a continuation, no matter how this continuation ends, when the continuation is destructed, the units object is destructed and the units are returned to the semaphore's counter. The above examples, written with get_units(), looks like this:

seastar::future<> g() {
    static thread_local semaphore limit(100);
    return seastar::get_units(limit, 1).then([] (auto units) {
        return slow().finally([units = std::move(units)] {});
    });
}

Note the somewhat convoluted way that get_units() needs to be used: The continuations must be nested because we need the units object to be moved to the last continuation. If slow() returns a future (and does not throw immediately), the finally() continuation captures the units object until everything is done, but does not run any code.

Seastar's programmers should generally avoid using the the semaphore::wait() and semaphore::signal() functions directly, and always prefer either with_semaphore() (when applicable) or get_units().

Limiting resource use Copy Chapter

Because semaphores support waiting for any number of units, not just 1, we can use them for more than simple limiting of the number of parallel invocation. For example, consider we have an asynchronous function using_lots_of_memory(size_t bytes), which uses bytes bytes of memory, and we want to ensure that not more than 1 MB of memory is used by all parallel invocations of this function --- and that additional calls are delayed until previous calls have finished. We can do this with a semaphore:

seastar::future<> using_lots_of_memory(size_t bytes) {
    static thread_local seastar::semaphore limit(1000000); // limit to 1MB
    return seastar::with_semaphore(limit, bytes, [bytes] {
        // do something allocating 'bytes' bytes of memory
    });
}

Watch out that in the above example, a call to using_lots_of_memory(2000000) will return a future that never resolves, because the semaphore will never contain enough units to satisfy the semaphore wait. using_lots_of_memory() should probably check whether bytes is above the limit, and throw an exception in that case. Seastar doesn't do this for you.

Limiting parallelism of loops Copy Chapter

Above, we looked at a function g() which gets called by some external event, and wanted to control its parallelism. In this section, we look at parallelism of loops, which also can be controlled with semaphores.

Consider the following simple loop:

#include <seastar/core/sleep.hh>
seastar::future<> slow() {
    std::cerr << ".";
    return seastar::sleep(std::chrono::seconds(1));
}
seastar::future<> f() {
    return seastar::repeat([] {
        return slow().then([] { return seastar::stop_iteration::no; });
    });
}

This loop runs the slow() function (taking one second to complete) without any parallelism --- the next slow() call starts only when the previous one completed. But what if we do not need to serialize the calls to slow(), and want to allow multiple instances of it to be ongoing concurrently?

Naively, we could achieve more parallelism, by starting the next call to slow() right after the previous call --- ignoring the future returned by the previous call to slow() and not waiting for it to resolve:

seastar::future<> f() {
    return seastar::repeat([] {
        slow();
        return seastar::stop_iteration::no;
    });
}

But in this loop, there is no limit to the amount of parallelism --- millions of sleep() calls might be active in parallel, before the first one ever returned. Eventually, this loop may consume all available memory and crash.

Using a semaphore allows us to run many instances of slow() in parallel, but limit the number of these parallel instances to, in the following example, 100:

seastar::future<> f() {
    return seastar::do_with(seastar::semaphore(100), [] (auto& limit) {
        return seastar::repeat([&limit] {
            return limit.wait(1).then([&limit] {
                seastar::futurize_invoke(slow).finally([&limit] {
                    limit.signal(1);
                });
                return seastar::stop_iteration::no;
            });
        });
    });
}

Note how this code differs from the code we saw above for limiting the number of parallel invocations of a function g():

1. Here we cannot use a single thread_local semaphore. Each call to f() has its loop with parallelism of 100, so needs its own semaphore "limit", kept alive during the loop with do_with().
2. Here we do not wait for slow() to complete before continuing the loop, i.e., we do not return the future chain starting at futurize_invoke(slow). The loop continues to the next iteration when a semaphore unit becomes available, while (in our example) 99 other operations might be ongoing in the background and we do not wait for them.

In the examples in this section, we cannot use the with_semaphore() shortcut. with_semaphore() returns a future which only resolves after the lambda's returned future resolves. But in the above example, the loop needs to know when just the semaphore units are available, to start the next iteration --- and not wait for the previous iteration to complete. We could not achieve that with with_semaphore(). But the more general exception-safe idiom, seastar::get_units(), can be used in this case, and is recommended:

seastar::future<> f() {
    return seastar::do_with(seastar::semaphore(100), [] (auto& limit) {
        return seastar::repeat([&limit] {
    	    return seastar::get_units(limit, 1).then([] (auto units) {
	            slow().finally([units = std::move(units)] {});
	            return seastar::stop_iteration::no;
	        });
        });
    });
}

The above examples are not realistic, because they have a never-ending loop and the future returned by f() will never resolve. In more realistic cases, the loop has an end, and at the end of the loop we need to wait for all the background operations which the loop started. We can do this by wait()ing on the original count of the semaphore: When the full count is finally available, it means that all the operations have completed. For example, the following loop ends after 456 iterations:

seastar::future<> f() {
    return seastar::do_with(seastar::semaphore(100), [] (auto& limit) {
        return seastar::do_for_each(boost::counting_iterator<int>(0),
                boost::counting_iterator<int>(456), [&limit] (int i) {
            return seastar::get_units(limit, 1).then([] (auto units) {
                slow().finally([units = std::move(units)] {});
	        });
        }).finally([&limit] {
            return limit.wait(100);
        });
    });
}

The last finally is what ensures that we wait for the last operations to complete: After the repeat loop ends (whether successfully or prematurely because of an exception in one of the iterations), we do a wait(100) to wait for the semaphore to reach its original value 100, meaning that all operations that we started have completed. Without this finally, the future returned by f() will resolve before all the iterations of the loop actually completed (the last 100 may still be running).

In the idiom we saw in the above example, the same semaphore is used both for limiting the number of background operations, and later to wait for all of them to complete. Sometimes, we want several different loops to use the same semaphore to limit their total parallelism. In that case we must use a separate mechanism for waiting for the completion of the background operations started by the loop. The most convenient way to wait for ongoing operations is using a gate, which we will describe in detail later. A typical example of a loop whose parallelism is limited by an external semaphore:

thread_local seastar::semaphore limit(100);
seastar::future<> f() {
    return seastar::do_with(seastar::gate(), [] (auto& gate) {
        return seastar::do_for_each(boost::counting_iterator<int>(0),
                boost::counting_iterator<int>(456), [&gate] (int i) {
            return seastar::get_units(limit, 1).then([&gate] (auto units) {
                gate.enter();
                seastar::futurize_invoke(slow).finally([&gate, units = std::move(units)] {
                    gate.leave();
                });
	        });
        }).finally([&gate] {
            return gate.close();
        });
    });
}

In this code, we use the external semaphore limit to limit the number of concurrent operations, but additionally have a gate specific to this loop to help us wait for all ongoing operations to complete.

TODO: also allow get_units() or something similar on a gate, and use that instead of the explicit gate.enter/gate.leave.

TODO: say something about semaphore fairness - if someone is waiting for a lot of units and later someone asks for 1 unit, will both wait or will the request for 1 unit be satisfied?

TODO: say something about broken semaphores? (or in later section especially about breaking/closing/shutting down/etc?)

TODO: Have a few paragraphs, or even a section, on additional uses of semaphores. One is for mutual exclusion using semaphore(1) - we need to explain why although why in Seastar we don't have multiple threads touching the same data, if code is composed of different continuations (i.e., a fiber) it can switch to a different fiber in the middle, so if data needs to be protected between two continuations, it needs a mutex. Another example is something akin to wait_all: we start with a semaphore(0), run a known number N of asynchronous functions with finally sem.signal(), and from all this return the future sem.wait(N). PERHAPS even have a separate section on mutual exclusion, where we begin with semaphore(1) but also mention shared_mutex

Pipes

Seastar's pipe<T> is a mechanism to transfer data between two fibers, one producing data, and the other consuming it. It has a fixed-size buffer to ensures a balanced execution of the two fibers, because the producer fiber blocks when it writes to a full pipe, until the consumer fiber gets to run and read from the pipe.

A pipe<T> resembles a Unix pipe, in that it has a read side, a write side, and a fixed-sized buffer between them, and supports either end to be closed independently (and EOF or broken pipe when using the other side). A pipe<T> object holds the reader and write sides of the pipe as two separate objects. These objects can be moved into two different fibers. Importantly, if one of the pipe ends is destroyed (i.e., the continuations capturing it end), the other end of the pipe will stop blocking, so the other fiber will not hang.

The pipe's read and write interfaces are future-based blocking. I.e., the write() and read() methods return a future which is fulfilled when the operation is complete. The pipe is single-reader single-writer, meaning that until the future returned by read() is fulfilled, read() must not be called again (and same for write).
Note: The pipe reader and writer are movable, but not copyable. It is often convenient to wrap each end in a shared pointer, so it can be copied (e.g., used in an std::function which needs to be copyable) or easily captured into multiple continuations.

Shutting down a service with a gate

Consider an application which has some long operation slow(), and many such operations may be started at any time. A number of slow() operations may even even be active in parallel. Now, you want to shut down this service, but want to make sure that before that, all outstanding operations are completed. Moreover, you don't want to allow new slow() operations to start while the shut-down is in progress.

This is the purpose of a seastar::gate. A gate g maintains an internal counter of operations in progress. We call g.enter() when entering an operation (i.e., before running slow()), and call g.leave() when leaving the operation (when a call to slow() completed). The method g.close() closes the gate, which means it forbids any further calls to g.enter() (such attempts will generate an exception); Moreover g.close() returns a future which resolves when all the existing operations have completed. In other words, when g.close() resolves, we know that no more invocations of slow() can be in progress - because the ones that already started have completed, and new ones could not have started.

The construct

seastar::with_gate(g, [] { return slow(); })

can be used as a shortcut to the idiom

g.enter();
slow().finally([&g] { g.leave(); });

Here is a typical example of using a gate:

#include <seastar/core/sleep.hh>
#include <seastar/core/gate.hh>
#include <boost/iterator/counting_iterator.hpp>

seastar::future<> slow(int i) {
    std::cerr << "starting " << i << "\n";
    return seastar::sleep(std::chrono::seconds(10)).then([i] {
        std::cerr << "done " << i << "\n";
    });
}
seastar::future<> f() {
    return seastar::do_with(seastar::gate(), [] (auto& g) {
        return seastar::do_for_each(boost::counting_iterator<int>(1),
                boost::counting_iterator<int>(6),
                [&g] (int i) {
            seastar::with_gate(g, [i] { return slow(i); });
            // wait one second before starting the next iteration
            return seastar::sleep(std::chrono::seconds(1));
		}).then([&g] {
            seastar::sleep(std::chrono::seconds(1)).then([&g] {
                // This will fail, because it will be after the close()
                seastar::with_gate(g, [] { return slow(6); });
            });
            return g.close();
        });
    });
}

In this example, we have a function future<> slow() taking 10 seconds to complete. We run it in a loop 5 times, waiting 1 second between calls, and surround each call with entering and leaving the gate (using with_gate). After the 5th call, while all calls are still ongoing (because each takes 10 seconds to complete), we close the gate and wait for it before exiting the program. We also test that new calls cannot begin after closing the gate, by trying to enter the gate again one second after closing it.

The output of this program looks like this:

starting 1
starting 2
starting 3
starting 4
starting 5
WARNING: exceptional future ignored of type 'seastar::gate_closed_exception': gate closed
done 1
done 2
done 3
done 4
done 5

Here, the invocations of slow() were started at 1 second intervals. After the "starting 5" message, we closed the gate and another attempt to use it resulted in a seastar::gate_closed_exception, which we ignored and hence this message. At this point the application waits for the future returned by g.close(). This will happen once all the slow() invocations have completed: Immediately after printing "done 5", the test program stops.

As explained so far, a gate can prevent new invocations of an operation, and wait for any in-progress operations to complete. However, these in-progress operations may take a very long time to complete. Often, a long operation would like to know that a shut-down has been requested, so it could stop its work prematurely. An operation can check whether its gate was closed by calling the gate's check() method: If the gate is already closed, the check() method throws an exception (the same seastar::gate_closed_exception that enter() would throw at that point). The intent is that the exception will cause the operation calling it to stop at this point.

In the previous example code, we had an un-interruptible operation slow() which slept for 10 seconds. Let's replace it by a loop of 10 one-second sleeps, calling g.check() each second:

seastar::future<> slow(int i, seastar::gate &g) {
    std::cerr << "starting " << i << "\n";
    return seastar::do_for_each(boost::counting_iterator<int>(0),
                                boost::counting_iterator<int>(10),
            [&g] (int) {
        g.check();
        return seastar::sleep(std::chrono::seconds(1));
    }).finally([i] {
        std::cerr << "done " << i << "\n";
    });
}

Now, just one second after gate is closed (after the "starting 5" message is printed), all the slow() operations notice the gate was closed, and stop. As expected, the exception stops the do_for_each() loop, and the finally() continuation is performed so we see the "done" messages for all five operations.

Introducing shared-nothing programming

TODO: Explain in more detail Seastar's shared-nothing approach where the entire memory is divided up-front to cores, malloc/free and pointers only work on one core.

TODO: Introduce our shared_ptr (and lw_shared_ptr) and sstring and say the standard ones use locked instructions which are unnecessary when we assume these objects (like all others) are for a single thread. Our futures and continuations do the same.

More about Seastar's event loop

TODO: Mention the event loop (scheduler). remind that continuations on the same thread do not run in parallel, so do not need locks, atomic variables, etc (different threads shouldn't access the same data - more on that below). continuations obviously must not use blocking operations, or they block the whole thread.

TODO: Talk about polling that we currently do, and how today even sleep() or waiting for incoming connections or whatever, takes 100% of all CPUs.

Introducing Seastar's network stack

TODO: Mention the two modes of operation: Posix and native (i.e., take a L2 (Ethernet) interface (vhost or dpdk) and on top of it we built (in Seastar itself) an L3 interface (TCP/IP)).

For optimal performance, Seastar's network stack is sharded just like Seastar applications are: each shard (thread) takes responsibility for a different subset of the connections. Each incoming connection is directed to one of the threads, and after a connection is established, it continues to be handled on the same thread.

In the examples we saw earlier, main() ran our function f() only once, on the first thread. Unless the server is run with the "-c1" option (one thread only), this will mean that any connection arriving to a different thread will not be handled. So in all the examples below, we will need to run the same service loop on all cores. We can easily do this with the smp::submit_to function:

seastar::future<> service_loop();

seastar::future<> f() {
    return seastar::parallel_for_each(std::views::iota(0u, seastar::smp::count),
            [] (unsigned c) {
        return seastar::smp::submit_to(c, service_loop);
    });
}

Here we ask each of Seastar cores (from 0 to smp::count-1) to run the same function service_loop(). Each of these invocations returns a future, and f() will return when all of them have returned (in the examples below, they will never return - we will discuss shutting down services in later sections).

We begin with a simple example of a TCP network server written in Seastar. This server repeatedly accepts connections on TCP port 1234, and returns an empty response:

#include <seastar/core/seastar.hh>
#include <seastar/core/reactor.hh>
#include <seastar/core/future-util.hh>
#include <seastar/net/api.hh>

seastar::future<> service_loop() {
    return seastar::do_with(seastar::listen(seastar::make_ipv4_address({1234})),
            [] (auto& listener) {
        return seastar::keep_doing([&listener] () {
            return listener.accept().then(
                [] (seastar::accept_result res) {
                    std::cout << "Accepted connection from " << res.remote_address << "\n";
            });
        });
    });
}

This code works as follows:

1. The listen() call creates a server_socket object, listener, which listens on TCP port 1234 (on any network interface).
2. We use do_with() to ensure that the listener socket lives throughout the loop.
3. To handle one connection, we call listener's accept() method. This method returns a future<accept_result>, i.e., is eventually resolved with an incoming TCP connection from a client (accept_result.connection) and the client's IP address and port (accept_result.remote_address).
4. To repeatedly accept new connections, we use the keep_doing() loop idiom. keep_doing() runs its lambda parameter over and over, starting the next iteration as soon as the future returned by the previous iteration completes. The iterations only stop if an exception is encountered. The future returned by keep_doing() itself completes only when the iteration stops (i.e., only on exception).

Output from this server looks like the following example:

$ ./a.out
Accepted connection from 127.0.0.1:47578
Accepted connection from 127.0.0.1:47582
...

If you run the above example server immediately after killing the previous server, it often fails to start again, complaining that:

$ ./a.out
program failed with uncaught exception: bind: Address already in use

This happens because by default, Seastar refuses to reuse the local port if there are any vestiges of old connections using that port. In our silly server, because the server is the side which first closes the connection, each connection lingers for a while in the "TIME_WAIT" state after being closed, and these prevent listen() on the same port from succeeding. Luckily, we can give listen an option to work despite these remaining TIME_WAIT. This option is analogous to socket(7)'s SO_REUSEADDR option:

seastar::listen_options lo;
    lo.reuse_address = true;
    return seastar::do_with(seastar::listen(seastar::make_ipv4_address({1234}), lo),

Most servers will always turn on this reuse_address listen option. Stevens' book "Unix Network Programming" even says that "All TCP servers should specify this socket option to allow the server to be restarted". Therefore in the future Seastar should probably default to this option being on --- even if for historic reasons this is not the default in Linux's socket API.

Let's advance our example server by outputting some canned response to each connection, instead of closing each connection immediately with an empty reply.

#include <seastar/core/seastar.hh>
#include <seastar/core/reactor.hh>
#include <seastar/core/future-util.hh>
#include <seastar/net/api.hh>

const char* canned_response = "Seastar is the future!\n";

seastar::future<> service_loop() {
    seastar::listen_options lo;
    lo.reuse_address = true;
    return seastar::do_with(seastar::listen(seastar::make_ipv4_address({1234}), lo),
            [] (auto& listener) {
        return seastar::keep_doing([&listener] () {
            return listener.accept().then(
                    [] (seastar::accept_result res) {
                auto s = std::move(res.connection);
                auto out = s.output();
                return seastar::do_with(std::move(s), std::move(out),
                        [] (auto& s, auto& out) {
                    return out.write(canned_response).then([&out] {
                        return out.close();
                    });
                });
            });
        });
    });
}

The new part of this code begins by taking the connected_socket's output(), which returns an output_stream<char> object. On this output stream out we can write our response using the write() method. The simple-looking write() operation is in fact a complex asynchronous operation behind the scenes, possibly causing multiple packets to be sent, retransmitted, etc., as needed. write() returns a future saying when it is ok to write() again to this output stream; This does not necessarily guarantee that the remote peer received all the data we sent it, but it guarantees that the output stream has enough buffer space (or in the TCP case, there is enough room in the TCP congestion window) to allow another write to begin.

After write()ing the response to out, the example code calls out.close() and waits for the future it returns. This is necessary, because write() attempts to batch writes so might not have yet written anything to the TCP stack at this point, and only when close() concludes can we be sure that all the data we wrote to the output stream has actually reached the TCP stack --- and only at this point we may finally dispose of the out and s objects.

Indeed, this server returns the expected response:

$ telnet localhost 1234
...
Seastar is the future!
Connection closed by foreign host.

In the above example we only saw writing to the socket. Real servers will also want to read from the socket. The connected_socket's input() method returns an input_stream<char> object which can be used to read from the socket. The simplest way to read from this stream is using the read() method which returns a future temporary_buffer<char>, containing some more bytes read from the socket --- or an empty buffer when the remote end shut down the connection.

temporary_buffer<char> is a convenient and safe way to pass around byte buffers that are only needed temporarily (e.g., while processing a request). As soon as this object goes out of scope (by normal return or exception), the memory it holds is automatically freed. Ownership of the buffer can also be transferred by moving it with std::move(). We'll discuss temporary_buffer in more detail in a later section.

Let's look at a simple example server involving both reads and writes. This is a simple echo server, as described in RFC 862: the server listens for connections from the client, and once a connection is established, any data received is sent back until the client closes the connection.

/* Detailed source-code truncated for AI context efficiency. */

The main function service_loop() loops accepting new connections, and for each connection calls handle_connection() to handle this connection. Our handle_connection() returns a future saying when handling this connection completed, but importantly, we do not wait for this future: Remember that keep_doing will only start the next iteration when the future returned by the previous iteration is resolved. Because we want to allow parallel ongoing connections, we don't want the next accept() to wait until the previously accepted connection was closed. So we call handle_connection() to start the handling of the connection, but return nothing from the continuation, which resolves that future immediately, so keep_doing will continue to the next accept().

This demonstrates how easy it is to run parallel fibers (chains of continuations) in Seastar - When a continuation runs an asynchronous function but ignores the future it returns, the asynchronous operation continues in parallel, but never waited for.

It is often a mistake to silently ignore an exception, so if the future we're ignoring might resolve with an exception, handle this case using, for example, a handle_exception() continuation. In our case, a failed connection is acceptable (for example, the client might close its connection while we're sending it output), so we did not handle the exception.

The handle_connection() function itself is straightforward --- it repeatedly calls read() read on the input stream, to receive a temporary_buffer with some data, and then moves this temporary buffer into a write() call on the output stream. The buffer will eventually be freed, automatically, when the write() is done with it. When read() eventually returns an empty buffer signifying the end of input, we stop repeat's iteration by returning a stop_iteration::yes.

Sharded services

In the previous section we saw that a Seastar application usually needs to run its code on all available CPU cores. We saw that the seastar::smp::submit_to() function allows the main function, which initially runs only on the first core, to start the server's code on all seastar::smp::count cores.

However, usually one needs not just to run code on each core, but also to have an object that contains the state of this code. Additionally, one may like to interact with those different objects, and also have a mechanism to stop the service running on the different cores.

The seastar::sharded<T> template provides a structured way create such a sharded service. It creates a separate object of type T in each core, and provides mechanisms to interact with those copies, to start some code on each, and finally to cleanly stop the service.

To use seastar::sharded, first create a class for the object holding the state of the service on a single core. For example:

#include <seastar/core/future.hh>
#include <iostream>

class my_service {
public:
    std::string _str;
    my_service(const std::string& str) : _str(str) { }
    seastar::future<> run() {
        std::cerr << "running on " << seastar::engine().cpu_id() <<
            ", _str = " << _str << "\n";
        return seastar::make_ready_future<>();
    }
    seastar::future<> stop() {
        return seastar::make_ready_future<>();
    }
};

The only mandatory method in this object is stop(), which will be called in each core when we want to stop the sharded service and want to wait until it stops on all cores.

Now let's see how to use it:

#include <seastar/core/sharded.hh>

seastar::sharded<my_service> s;

seastar::future<> f() {
    return s.start(std::string("hello")).then([] {
        return s.invoke_on_all([] (my_service& local_service) {
            return local_service.run();
        });
    }).then([] {
        return s.stop();
    });
}

The s.start() starts the service by creating a my_service object on each of the cores. The arguments to s.start(), if any (in this example, std::string("hello")), are passed to my_service's constructor.

But s.start() did not start running any code yet (besides the object's constructor). For that, we have the s.invoke_on_all() which runs the given lambda on all the cores - giving each lambda the local my_service object on that core. In this example, we have a run() method on each object, so we run that.

Finally, at the end of the run we want to give the service on all cores a chance to shut down cleanly, so we call s.stop(). This will call the stop() method on each core's object, and wait for all of them to finish. Calling s.stop() before destroying s is mandatory - Seastar will warn you if you forget to do it.

In addition to invoke_on_all() which runs the same code on all shards, another feature a sharded service often needs is for one shard to invoke code another specific shard. This is done by calling the sharded service's invoke_on() method. For example:

seastar::sharded<my_service> s;
...
return s.invoke_on(0, [] (my_service& local_service) {
    std::cerr << "invoked on " << seastar::engine().cpu_id() <<
        ", _str = " << local_service._str << "\n";
});

This runs the lambda function on shard 0, with a reference to the local my_service object on that shard.

Shutting down cleanly

TODO: Handling interrupt, shutting down services, etc.

Move the seastar::gate section here.

Command line options

Standard Seastar command-line options Copy Chapter

All Seastar applications accept a standard set of command-line arguments, such as those we've already seen above: The -c option for controlling the number of threads used, or -m for determining the amount of memory given to the application.

TODO: list and explain more of these options.

Every Seastar application also accepts the -h (or --help) option, which lists and explains all the available options --- the standard Seastar ones, and the user-defined ones as explained below.

Setting options outside the command line Copy Chapter

Options do not have to be passed as arguments. Seastar reads them from three places, in this order of precedence:

1. the command line;
2. the SEASTAR_OPTIONS environment variable;
3. ~/.config/seastar/seastar.conf and ~/.config/seastar/io.conf.

SEASTAR_OPTIONS holds options written exactly as they would be on the command line, and is split the way a shell would split it, so a value containing spaces can be quoted:

$ SEASTAR_OPTIONS='--reactor-backend epoll -c 2' ./my_app

It suits a setting which belongs to an environment rather than to a single invocation, such as running a whole test suite on a particular reactor backend without changing how each test is invoked. Unlike an argument list, it cannot supply positional options.

The two configuration files use the name = value form, one option per line:

reactor-backend = epoll

The files are optional, and are only read when HOME is set. They accept Seastar's own options; SEASTAR_OPTIONS and the command line additionally accept the application's own options, as described below.

The first place which sets an option wins it, so a command line argument overrides SEASTAR_OPTIONS, which overrides the files. An option set in more than one place is not an error and produces no warning: the lower-precedence value is simply ignored. Setting the same option twice in one place is an error, unless that option is one which may be repeated.

An application can replace the file-reading step with app_template::set_configuration_reader(), for instance to read options from somewhere else entirely.

User-defined command-line options Copy Chapter

Seastar parses the command line options (argv[]) when it is passed to app_template::run(), looking for its own standard options. Therefore, it is not recommended that the application tries to parse argv[] on its own because the application might not understand some of the standard Seastar options and not be able to correctly skip them.

Rather, applications which want to have command-line options of their own should tell Seastar's command line parser of these additional application-specific options, and ask Seastar's command line parser to recognize them too. Seastar's command line parser is actually the Boost library's boost::program_options. An application adds its own option by using the add_options() and add_positional_options() methods on the app_template to define options, and later calling configuration() to retrieve the setting of these options. For example,

#include <iostream>
#include <seastar/core/app-template.hh>
#include <seastar/core/reactor.hh>
int main(int argc, char** argv) {
    seastar::app_template app;
    namespace bpo = boost::program_options;
    app.add_options()
        ("flag", "some optional flag")
        ("size,s", bpo::value<int>()->default_value(100), "size")
        ;
    app.add_positional_options({
       { "filename", bpo::value<std::vector<seastar::sstring>>()->default_value({}),
         "sstable files to verify", -1}
    });
    return app.run(argc, argv, [&app] {
        auto& args = app.configuration();
        if (args.count("flag")) {
            std::cout << "Flag is on\n";
        }
        std::cout << "Size is " << args["size"].as<int>() << "\n";
        auto& filenames = args["filename"].as<std::vector<seastar::sstring>>();
        for (auto&& fn : filenames) {
            std::cout << fn << "\n";
        }
        return seastar::make_ready_future<>();
    });
    return 0;
}

In this example, we add via add_options() two application-specific options: --flag is an optional parameter which doesn't take any additional arguments, and --size (or -s) takes an integer value, which defaults (if this option is missing) to 100. Additionally, we ask via add_positional_options() that an unlimited number of arguments that do not begin with a "-" --- the so-called positional arguments --- be collected to a vector of strings under the "filename" option. Some example outputs from this program:

$ ./a.out
Size is 100
$ ./a.out --flag
Flag is on
Size is 100
$ ./a.out --flag -s 3
Flag is on
Size is 3
$ ./a.out --size 3 hello hi
Size is 3
hello
hi
$ ./a.out --filename hello --size 3 hi
Size is 3
hello
hi

boost::program_options has more powerful features, such as required options, option checking and combining, various option types, and more. Please refer to Boost's documentation for more information.

Debugging a Seastar program

Debugging ignored exceptions Copy Chapter

If a future resolves with an exception, and the application neglects to handle that exception or to explicitly ignore it, the application may have missed an important problem. This is likely to be an application bug.

Therefore, Seastar prints a warning message to the log if a future is destroyed when it stores an exception that hasn't been handled.

For example, consider this code:

#include <seastar/core/future.hh>
#include <seastar/core/sleep.hh>
#include <seastar/core/app-template.hh>

class myexception {};

seastar::future<> g() {
    return seastar::make_exception_future<>(myexception());
}

seastar::future<> f() {
    g();
    return seastar::sleep(std::chrono::seconds(1));
}

int main(int argc, char** argv) {
    seastar::app_template app;
    return app.run(argc, argv, f);
}

Here, the main function f() calls g(), but doesn't do anything with the future it returns. But this future resolves with an exception, and this exception is silently ignored. So Seastar prints this warning message about the ignored exception:

WARN  2020-03-31 11:08:09,208 [shard 0] seastar - Exceptional future ignored: myexception, backtrace:   /lib64/libasan.so.5+0x6ce7f
  0x1a64193
  0x1a6265f
  0xf326cc
  0xeaf1a0
  0xeaffe4
  0xead7be
  0xeb5917
  0xee2477
  0xec312e
  0xec8fcf
  0xeec765
  0xee1b29
  0xed9fab
  0xec27c8
  0xec867f
  0xf00acc
  0xef179d
  0xef1824
  0xef18b5
  0xee4827
  0xee470f
  0xf00f81
  0xebac29
  0xeb9095
  0xeb9174
  0xeb925a
  0xeb9964
  0xebef89
  0x10f74c3
  0x10fb439
  0x11005fd
  0xec4f08
  0xec2f43
  0xec3461
  0xeadabe
  /lib64/libc.so.6+0x271a2
  0xead52d

This message says that an exceptional future was ignored, and that the type of the exception was "myexception". The type of the exception is usually not enough to pinpoint where the problem happened, so the warning message also includes the backtrace - the call chain - leading to where the exceptional future was destroyed. The backtrace is given as a list of addresses, where code in other shared libraries is written as a shared library plus offset (when ASLR is enabled, the shared libraries are mapped in a different address each time).

Seastar includes a utility, seastar-addr2line, for translating these addresses into readable backtraces including exact method names, source files and line numbers. This utility needs the unstripped executable. Typically, a stripped executable is used for production, but an unstripped copy is kept separately to be used in debugging - including seastar-addr2line.

To decode the backtrace, we run

seastar-addr2line -e a.out

And then paste the list of addresses in the warning message, and conclude with a control-D (it's also possible, if you want, to put the list of addresses in the seastar-addr2line command line). The result looks like this:

/* Detailed source-code truncated for AI context efficiency. */

Most of the lines at the bottom of this backtrace are not interesting, and just showing the internal details of how Seastar ended up running the main function f(). The only interesting part is the first few lines:

seastar::report_failed_future(std::__exception_ptr::exception_ptr const&) at src/core/future.cc:116
seastar::future_state_base::~future_state_base() at include/seastar/core/future.hh:335
seastar::future_state<>::~future_state() at include/seastar/core/future.hh:414
 (inlined by) seastar::future<>::~future() at include/seastar/core/future.hh:990
f() at test.cc:12

Here we see that the warning message was printed by the seastar::report_failed_future() function which was called when destroying a future (future<>::~future) that had not been handled. The future's destructor was called in line 11 of our test code (26.cc), which is indeed the line where we called g() and ignored its result.
This backtrace gives us an accurate understanding of where our code destroyed an exceptional future without handling it first, which is usually helpful in solving these kinds of bugs. Note that this technique does not tell us where the exception was first created, nor what code passed around the exceptional future before it was destroyed - we just learn where the future was destroyed. To learn where the exception was originally thrown, see the next section:

Finding where an exception was thrown Copy Chapter

Sometimes an application logs an exception, and we want to know where in the code the exception was originally thrown. Unlike languages like Java, C++ does not have a built-in method of attaching a backtrace to every exception. Seastar therefore provides functions for adding the backtrace recorded when an exception was thrown.

For example, in the following code we throw and catch an std::runtime_error normally:

#include <seastar/core/future.hh>
#include <seastar/util/log.hh>
#include <exception>
#include <iostream>

seastar::future<> g() {
    return seastar::make_exception_future<>(std::runtime_error("hello"));
}

seastar::future<> f() {
    return g().handle_exception([](std::exception_ptr e) {
        std::cerr << "Exception: " << e << "\n";
    });
}

The output is

Exception: std::runtime_error (hello)

From this output, we have no way of knowing that the exception was thrown in g(). We can solve this if we use make_exception_future_with_backtrace instead of make_exception_future:

#include <util/backtrace.hh>
seastar::future<> g() {
    return seastar::make_exception_future_with_backtrace<>(std::runtime_error("hello"));
}

Now the output looks like

Exception: seastar::internal::backtraced<std::runtime_error> (hello Backtrace:   0x678bd3
  0x677204
  0x67736b
  0x678cd5
  0x4f923c
  0x4f9c38
  0x4ff4d0
...
)

Which, as above, can be converted to a human-readable backtrace by using the seastar-addr2line script.

In addition to seastar::make_exception_future_with_backtrace(), Seastar also provides a function throw_with_backtrace(), to throw an exception instead of returning an exceptional future. For example:

seastar::throw_with_backtrace<std::runtime_error>("hello");

In the current implementation, both make_exception_future_with_backtrace and throw_with_backtrace require that the original exception type (in the above example, std::runtime_error) is a subclass of the std::exception class. The original exception provides a what() string, and the wrapped exception adds the backtrace to this string, as demonstrated above. Moreover, the wrapped exception type is a subclass of the original exception type, which allows catch(...) code to continue filtering by the exception original type - despite the addition of the backtrace.

Debugging with gdb Copy Chapter

handle SIGUSR1 pass noprint
handle SIGALRM pass noprint

Promise objects

As we already defined above, An asynchronous function, also called a promise, is a function which returns a future and arranges for this future to be eventually resolved. As we already saw, an asynchronous function is usually written in terms of other asynchronous functions, for example we saw the function slow() which waits for the existing asynchronous function sleep() to complete, and then returns 3:

seastar::future<int> slow() {
    using namespace std::chrono_literals;
    return seastar::sleep(100ms).then([] { return 3; });
}

The most basic building block for writing promises is the promise object, an object of type promise<T>. A promise<T> has a future<T> get_future() method that returns a future and a set_value(T) method that resolves it. An asynchronous function can create a promise object, return its future, and arrange for set_value() to be called later, finally resolving the returned future.

In the following example we create a promise that manages the process of printing 10 messages, once every second.
We start by creating an empty promise. We then start a seastar::thread to perform the work. When that work—printing the messages—is complete, we call promise::set_value() to mark the task as complete. We wait for the future generated by our promise just like any other future.

#include <seastar/core/future.hh>
#include <seastar/core/do_with.hh>
#include <seastar/core/sleep.hh>
#include <seastar/core/thread.hh>
#include <iostream>


seastar::future<> f() {
    return seastar::do_with(seastar::promise<>(), [](auto& promise) {
        (void)seastar::async([&promise]() {
                using namespace std::chrono_literals;
                for (int i = 0; i < 10; i++) {
                    std::cout << i << "..." << std::flush;
                    seastar::sleep(1s).wait();
                }
                std::cout << std::endl;

                promise.set_value();
        });

        return promise.get_future();
    });
}

Memory allocation in Seastar

Per-thread memory allocation Copy Chapter

Seastar requires that applications be sharded, i.e., that code running on different threads operate on different objects in memory. We already saw in [Seastar memory] how Seastar takes over a given amount of memory (often, most of the machine's memory) and divides it equally between the different threads. Modern multi-socket machines have non-uniform memory access (NUMA), meaning that some parts of memory are closer to some of the cores, and Seastar takes this knowledge into account when dividing the memory between threads. Currently, the division of memory between threads is static, and equal - the threads are expected to experience roughly equal amount of load and require roughly equal amounts of memory.

To achieve this per-thread allocation, Seastar redefines the C library functions malloc(), free(), and their numerous relatives --- calloc(), realloc(), posix_memalign(), memalign(), malloc_usable_size(), and malloc_trim(). It also redefines the C++ memory allocation functions, operator new, operator delete, and all their variants (including array versions, the C++14 delete taking a size, and the C++17 variants taking required alignment).

It is important to remember that Seastar's different threads can see memory allocated by other threads, but they are nonetheless strongly discouraged from actually doing this. Sharing data objects between threads on modern multi-core machines results in stiff performance penalties from locks, memory barriers, and cache-line bouncing. Rather, Seastar encourages applications to avoid sharing objects between threads when possible (by sharding --- each thread owns a subset of the objects), and when threads do need to interact they do so with explicit message passing, with submit_to(), as we shall see later.

Foreign pointers Copy Chapter

An object allocated on one thread will be owned by this thread, and eventually should be freed by the same thread. Freeing memory on the wrong thread is strongly discouraged, but is currently supported (albeit slowly) to support library code beyond Seastar's control. For example, std::exception_ptr allocates memory; So if we invoke an asynchronous operation on a remote thread and this operation returns an exception, when we free the returned std::exception_ptr this will happen on the "wrong" core. So Seastar allows it, but inefficiently.

In most cases objects should spend their entire life on a single thread and be used only by this thread. But in some cases we want to reassign ownership of an object which started its life on one thread, to a different thread. This can be done using a seastar::foreign_ptr<>. A pointer, or smart pointer, to an object is wrapped in a seastar::foreign_ptr. This wrapper can then be moved into code running in a different thread (e.g., using submit_to()).

The most common use-case is a seastar::foreign_ptr<std::unique_ptr<T>>. The thread receiving this foreign_ptr will get exclusive use of the object, and when it destroys this wrapper, it will go back to the original thread to destroy the object. Note that the object is not only freed on the original shard - it is also destroyed (i.e., its destructor is run) there. This is often important when the object's destructor needs to access other state which belongs to the original shard - e.g., unlink itself from a container.

Although foreign_ptr ensures that the object's destructor automatically runs on the object's home thread, it does not absolve the user from worrying where to run the object's other methods. Some simple methods, e.g., methods which just read from the object's fields, can be run on the receiving thread. However, other methods may need to access other data owned by the object's home shard, or need to prevent concurrent operations. Even if we're sure that object is now used exclusively by the receiving thread, such methods must still be run, explicitly, on the home thread:

// fp is some foreign_ptr<>
    return smp::submit_to(fp.get_owner_shard(), [p=fp.get()]
        { return p->some_method(); });

So seastar::foreign_ptr<> not only has functional benefits (namely, to run the destructor on the home shard), it also has documentational benefits - it warns the programmer to watch out every time the object is used, that this is a foreign pointer, and if we want to do anything non-trivial with the pointed object, we may need to do it on the home shard.

Above, we discussed the case of transferring ownership of an object to a another shard, via seastar::foreign_ptr<std::unique_ptr<T>>. However, sometimes the sender does not want to relinquish ownership of the object. Sometimes, it wants the remote thread to operate on its object and return with the object intact. Sometimes, it wants to send the same object to multiple shards. In such cases, seastar::foreign_ptr<seastar::lw_shared_ptr<T>> is useful. The user needs to watch out, of course, not to operate on the same object from multiple threads concurrently. If this cannot be ensured by program logic alone, some methods of serialization must be used - such as running the operations on the home shard with submit_to() as described above.

Normally, a seastar::foreign_ptr cannot not be copied - only moved. However, when it holds a smart pointer that can be copied (namely, a shared_ptr), one may want to make an additional copy of that pointer and create a second foreign_ptr. Doing this is inefficient and asynchronous (it requires communicating with the original owner of the object to create the copies), so a method future<foreign_ptr> copy() needs to be explicitly used instead of the normal copy constructor.

Seastar::thread

Seastar's programming model, using futures and continuations, is very powerful and efficient. However, as we've already seen in examples above, it is also relatively verbose: Every time that we need to wait before proceeding with a computation, we need to write another continuation. We also need to worry about passing the data between the different continuations (using techniques like those described in the [Lifetime management] section). Simple flow-control constructs such as loops also become more involved using continuations. For example, consider this simple classical synchronous code:

std::cout << "Hi.\n";
    for (int i = 1; i < 4; i++) {
        sleep(1);
        std::cout << i << "\n";
    }

In Seastar, using futures and continuations, we need to write something like this:

std::cout << "Hi.\n";
    return seastar::do_for_each(boost::counting_iterator<int>(1),
        boost::counting_iterator<int>(4), [] (int i) {
        return seastar::sleep(std::chrono::seconds(1)).then([i] {
            std::cout << i << "\n";
        });
    });

But Seastar also allows you to write code that looks more synchronous by using seastar::thread. A seastar::thread provides an execution environment where blocking is tolerated: you can invoke an asynchronous function and wait for it in the same function instead of establishing a callback with future<>::then():

seastar::thread th([] {
        std::cout << "Hi.\n";
        for (int i = 1; i < 4; i++) {
            seastar::sleep(std::chrono::seconds(1)).get();
            std::cout << i << "\n";
        }
    });

A seastar::thread is not a separate operating system thread. It still uses continuations, which are scheduled on Seastar's single thread (per core). It works as follows:

The seastar::thread allocates a 128KB stack, and runs the given function until the it blocks on the call to a future's get() method. Outside a seastar::thread context, get() may only be called on a future which is already available. But inside a thread, calling get() on a future which is not yet available stops running the thread function, and schedules a continuation for this future, which continues to run the thread's function (on the same saved stack) when the future becomes available.

Just like normal Seastar continuations, seastar::threads always run on the same core they were launched on. They are also cooperative: they are never preempted except when seastar::future::get() blocks or on explicit calls to seastar::thread::yield().

It is worth reiterating that a seastar::thread is not a POSIX thread, and it can only block on Seastar futures, not on blocking system calls. The above example used seastar::sleep(), not the sleep() system call. The seastar::thread's function can throw and catch exceptions normally. Remember that get() will throw an exception if the future resolves with an exception.

In addition to seastar::future::get(), we also have seastar::future::wait() to wait without fetching the future's result. This can sometimes be useful when you want to avoid throwing an exception when the future failed (as get() does). For example:

future<char> getchar();
    int try_getchar() noexcept { // run this in seastar::thread context
        future fut = get_char();
        fut.wait();
        if (fut.failed()) {
            return -1;
        } else {
            // Here we already know that get() will return immediately,
            // and will not throw.
            return fut.get();
        }
    }

Starting and ending a seastar::thread Copy Chapter

After we created a seastar::thread object, we need wait until it ends, using its join() method. We also need to keep that object alive until join() completes. A complete example using seastar::thread will therefore look like this:

#include <seastar/core/sleep.hh>
#include <seastar/core/thread.hh>
seastar::future<> f() {
    seastar::thread th([] {
        std::cout << "Hi.\n";
        for (int i = 1; i < 4; i++) {
            seastar::sleep(std::chrono::seconds(1)).get();
            std::cout << i << "\n";
        }
    });
    return do_with(std::move(th), [] (auto& th) {
        return th.join();
    });
}

The seastar::async() function provides a convenient shortcut for creating a seastar::thread and returning a future which resolves when the thread completes:

#include <seastar/core/sleep.hh>
#include <seastar/core/thread.hh>
seastar::future<> f() {
    return seastar::async([] {
        std::cout << "Hi.\n";
        for (int i = 1; i < 4; i++) {
            seastar::sleep(std::chrono::seconds(1)).get();
            std::cout << i << "\n";
        }
    });
}

seastar::async()'s lambda may return a value, and seastar::async() returns it when it completes. For example:

seastar::future<seastar::sstring> read_file(sstring file_name) {
    return seastar::async([file_name] () {  // lambda executed in a thread
        file f = seastar::open_file_dma(file_name).get();  // get() call "blocks"
        auto buf = f.dma_read(0, 512).get();  // "block" again
        return seastar::sstring(buf.get(), buf.size());
    });
};

While seastar::threads and seastar::async() make programming more convenient, they also add overhead beyond that of programming directly with continuations. Most notably, each seastar::thread requires additional memory for its stack. It is therefore not a good idea to use a seastar::thread to handle a highly concurrent operation. For example, if you need to handle 10,000 concurrent requests, do not use a seastar::thread to handle each --- use futures and continuations. But if you are writing code where you know that only a few instances will ever run concurrently, e.g., a background cleanup operation in your application, seastar::thread is a good match. seastar::thread is also great for code which doesn't care about performance --- such as test code.

Isolation of application components

Seastar makes multi-tasking very easy - as easy as running an asynchronous function. It is therefore easy for a server to do many unrelated things in parallel. For example, a server might be in the process of answering 100 users' requests, and at the same time also be making progress on some long background operation.

But in the above example, what percentage of the CPU and disk throughput will the background operation get? How long can one of the user's requests be delayed by the background operation? Without the mechanisms we describe in this section, these questions cannot be reliably answered:

• The background operation may be a very "considerate" single fiber, i.e., run a very short continuation and then schedule the next continuation to run later. At each point the scheduler sees 100 request-handling continuations and just one of the background continuations ready to run. The background task gets around 1% of the CPU time, and users' requests are hardly delayed.
• On the other hand, the background operation may spawn 1,000 fibers in parallel and have 1,000 ready-to-run continuations at each time. The background operation will get about 90% of the runtime, and the continuation handling a user's request may get stuck behind 1,000 of these background continuations, and experience huge latency.

Complex Seastar applications often have different components which run in parallel and have different performance objectives. In the above example we saw two components - user requests and the background operation. The first goal of the mechanisms we describe in this section is to isolate the performance of each component from the others; In other words, the throughput and latency of one component should not depend on decisions that another component makes - e.g., how many continuations it runs in parallel. The second goal is to allow the application to control this isolation, e.g., in the above example allow the application to explicitly control the amount of CPU the background operation receives, so that it completes at a desired pace.

In the above examples we used CPU time as the limited resource that the different components need to share effectively. As we show later, another important shared resource is disk I/O.

Scheduling groups (CPU scheduler) Copy Chapter

Consider the following asynchronous function loop(), which loops until some shared variable stop becomes true. It keeps a counter of the number of iterations until stopping, and returns this counter when finally stopping.

seastar::future<long> loop(int parallelism, bool& stop) {
    return seastar::do_with(0L, [parallelism, &stop] (long& counter) {
        return seastar::parallel_for_each(std::views::iota(0u, parallelism),
            [&stop, &counter]  (unsigned c) {
                return seastar::do_until([&stop] { return stop; }, [&counter] {
                    ++counter;
                    return seastar::make_ready_future<>();
                });
            }).then([&counter] { return counter; });
    });
}

The parallelism parameter determines the parallelism of the silly counting operation: parallelism=1 means we have just one loop incrementing the counter; parallelism=10 means we start 10 loops in parallel all incrementing the same counter.

What happens if we start two loop() calls in parallel and let them run for 10 seconds?

seastar::future<> f() {
    return seastar::do_with(false, [] (bool& stop) {
        seastar::sleep(std::chrono::seconds(10)).then([&stop] {
            stop = true;
        });
        return seastar::when_all_succeed(loop(1, stop), loop(1, stop)).then_unpack(
            [] (long n1, long n2) {
                std::cout << "Counters: " << n1 << ", " << n2 << "\n";
            });
    });
}

It turns out that if the two loop() calls had the same parallelism 1, we get roughly the same amount of work from both of them:

Counters: 3'559'635'758, 3'254'521'376

But if for example we ran a loop(1) in parallel with a loop(10), the result is that the loop(10) gets 10 times more work done:

Counters: 629'482'397, 6'320'167'297

Why does the amount of work that loop(1) can do in ten seconds depend on the parallelism chosen by its competitor, and how can we solve this?

The reason this happens is as follows: When a future resolves and a continuation was linked to it, this continuation becomes ready to run. By default, Seastar's scheduler keeps a single list of ready-to-run continuations (in each shard, of course), and runs the continuations at the same order they became ready to run. In the above example, loop(1) always has one ready-to-run continuation, but loop(10), which runs 10 loops in parallel, always has ten ready-to-run continuations. So for every continuation of loop(1), Seastar's default scheduler will run 10 continuations of loop(10), which is why loop(10) gets 10 times more work done.

To solve this, Seastar allows an application to define separate components known as scheduling groups, which each has a separate list of ready-to-run continuations. Each scheduling group gets to run its own continuations on a desired percentage of the CPU time, but the number of runnable continuations in one scheduling group does not affect the amount of CPU that another scheduling group gets. Let's look at how this is done:

A scheduling group is defined by a value of type scheduling_group. This value is opaque, but internally it is a small integer (similar to a process ID in Linux). We use the seastar::with_scheduling_group() function to run code in the desired scheduling group:

seastar::future<long>
loop_in_sg(int parallelism, bool& stop, seastar::scheduling_group sg) {
    return seastar::with_scheduling_group(sg, [parallelism, &stop] {
        return loop(parallelism, stop);
    });
}

TODO: explain what with_scheduling_group group really does, how the group is "inherited" to the continuations started inside it.

Now let's create two scheduling groups, and run loop(1) in the first scheduling group and loop(10) in the second scheduling group:

seastar::future<> f() {
    return seastar::when_all_succeed(
            seastar::create_scheduling_group("loop1", 100),
            seastar::create_scheduling_group("loop2", 100)).then_unpack(
        [] (seastar::scheduling_group sg1, seastar::scheduling_group sg2) {
        return seastar::do_with(false, [sg1, sg2] (bool& stop) {
            seastar::sleep(std::chrono::seconds(10)).then([&stop] {
                stop = true;
            });
            return seastar::when_all_succeed(loop_in_sg(1, stop, sg1), loop_in_sg(10, stop, sg2)).then_unpack(
                [] (long n1, long n2) {
                    std::cout << "Counters: " << n1 << ", " << n2 << "\n";
                });
        });
    });
}

Here we created two scheduling groups, sg1 and sg2. Each scheduling group has an arbitrary name (which is used for diagnostic purposes only), and a number of shares, a number traditionally between 1 and 1000: If one scheduling group has twice the number of shares than a second scheduling group, it will get twice the amount of CPU time. In this example, we used the same number of shares (100) for both groups, so they should get equal CPU time.

Unlike most objects in Seastar which are separate per shard, Seastar wants the identities and numbering of the scheduling groups to be the same on all shards, because it is important when invoking tasks on remote shards. For this reason, the function to create a scheduling group, seastar::create_scheduling_group(), is an asynchronous function returning a future<scheduling_group>.

Running the above example, with both scheduling group set up with the same number of shares (100), indeed results in both scheduling groups getting the same amount of CPU time:

Counters: 3'353'900'256, 3'350'871'461

Note how now both loops got the same amount of work done - despite one loop having 10 times the parallelism of the second loop.

If we change the definition of the second scheduling group to have 200 shares, twice the number of shares of the first scheduling group, we'll see the second scheduling group getting twice the amount of CPU time:

Counters: 2'273'783'385, 4'549'995'716

Latency Copy Chapter

TODO: Task quota, preempt, loops with built-in preemption check, etc.

Disk I/O scheduler Copy Chapter

TODO

Network scheduler Copy Chapter

TODO: Say that not yet available. Give example of potential problem - e.g., sharing a slow WAN link.

Controllers Copy Chapter

TODO: Talk about how to dynamically change the number of shares, and why.

Multi-tenancy Copy Chapter

TODO

General application structuring tips

Initialization and cleanup Copy Chapter

In C++, RAII is typically used to coordinate initialization and cleanup. In asyncronous programming, RAII
is not available, but for the control plane it is useful to bring up a seastar::thread to make use of it.

In conjunction with seastar::app_template, initialization and cleanup look like so:

#include <seastar/core/app-template.hh>
#include <seastar/core/reactor.hh>
#include <iostream>

int main(int argc, char** argv) {
    seastar::app_template app;
    return app.run(argc, argv, [] {
        return seastar::async() {
            std::cout << "Hello world\n";
            my_class my_object;
            // ... more code

            // my_object will be destroyed here
            return 0;
    });
}

If one needs to invoke an asynchronous function during shutdown, one can make
use of the seastar::defer() function to schedule an arbitrary cleanup function:

#include <seastar/core/app-template.hh>
#include <seastar/core/reactor.hh>
#include <iostream>

int main(int argc, char** argv) {
    seastar::app_template app;
    return app.run(argc, argv, [] {
        return seastar::async() {
            std::cout << "Hello world\n";
            my_class my_object;
            // ... more code

            auto cleanup = defer([] () noexcept {
                // .get() joins the asynchronous function back
                // into the thread.
                call_async_cleanup_function().get();
            });

            // `cleanup` will invoke its function here, if we reached
            // its construction point

            // my_object will be destroyed here
            return 0;
    });
}

See also seastar::deferred_close and seastar::deferred_stop for more ways to
control cleanup.

Doc Websocket (`doc/websocket.md`) Copy Section

WebSocket protocol implementation

Seastar includes an experimental implementation of a WebSocket server. See
RFC 6455 and MDN's guide to
writing WebSocket servers.

Handlers Copy Chapter

A WebSocket server needs a user-defined handler to be functional. The WebSocket specification defines subprotocols, and the Seastar WebSocket server allows one handler to be registered per subprotocol.

Each subprotocol has a unique name and is expected to be sent by the connecting client during handshake,
by sending a Sec-Websocket-Protocol header with the chosen value.

Aside from specifying the chosen subprotocol name for a handler, the developer is expected to provide a function
which handles the incoming stream of data and returns responses into the output stream.

Here's an example of how to register a simple echo protocol:

using namespace seastar;
static experimental::websocket::server ws;
ws.register_handler("echo", [] (input_stream<char>& in, output_stream<char>& out) -> future<> {
    while (true) {
        auto buf = co_await in.read();
        if (buf.empty()) {
            co_return;
        }
        co_await out.write(std::move(buf));
        co_await out.flush();
    }
});

Note: the developers should assume that the input stream provides decoded and unmasked data - so the stream should be treated as if it was backed by a TCP socket. Similarly, responses should be sent to the output stream as is, and the WebSocket server implementation will handle its proper serialization, masking and so on.

Error handling Copy Chapter

Registered WebSocket handlers can throw arbitrary exceptions during their operation. Currently, exceptions that aren't explicitly handled within the handler will cause the established WebSocket connection to be terminated, and a proper error message will be logged.

Secure WebSocket (`wss://`) Copy Chapter

The WebSocket server does not currently support secure WebSocket connections (wss://).

--- METRICS ---

• Files Extracted: 27
• Estimated Token Budget: ~68826 tokens
• Canonical Reference: https://codewiki.google/github.com/scylladb/seastar