seastar::future<> f() {
return seastar::do_with(seastar::promise<>(), [](auto& promise) {
(void)seastar::async([&promise]() {
using namespace std::chrono_literals;
for (int i = 0; i < 10; i++) {
std::cout << i << "..." << std::flush;
seastar::sleep(1s).wait();
}
std::cout << std::endl;
promise.set_value();
});
return promise.get_future();
});
}
```
# Memory allocation in Seastar
## Per-thread memory allocation
Seastar requires that applications be sharded, i.e., that code running on different threads operate on different objects in memory. We already saw in [Seastar memory] how Seastar takes over a given amount of memory (often, most of the machine's memory) and divides it equally between the different threads. Modern multi-socket machines have non-uniform memory access (NUMA), meaning that some parts of memory are closer to some of the cores, and Seastar takes this knowledge into account when dividing the memory between threads. Currently, the division of memory between threads is static, and equal - the threads are expected to experience roughly equal amount of load and require roughly equal amounts of memory.
To achieve this per-thread allocation, Seastar redefines the C library functions `malloc()`, `free()`, and their numerous relatives --- `calloc()`, `realloc()`, `posix_memalign()`, `memalign()`, `malloc_usable_size()`, and `malloc_trim()`. It also redefines the C++ memory allocation functions, `operator new`, `operator delete`, and all their variants (including array versions, the C++14 delete taking a size, and the C++17 variants taking required alignment).
It is important to remember that Seastar's different threads *can* see memory allocated by other threads, but they are nonetheless strongly discouraged from actually doing this. Sharing data objects between threads on modern multi-core machines results in stiff performance penalties from locks, memory barriers, and cache-line bouncing. Rather, Seastar encourages applications to avoid sharing objects between threads when possible (by *sharding* --- each thread owns a subset of the objects), and when threads do need to interact they do so with explicit message passing, with `submit_to()`, as we shall see later.
## Foreign pointers
An object allocated on one thread will be owned by this thread, and eventually should be freed by the same thread. Freeing memory on the *wrong* thread is strongly discouraged, but is currently supported (albeit slowly) to support library code beyond Seastar's control. For example, `std::exception_ptr` allocates memory; So if we invoke an asynchronous operation on a remote thread and this operation returns an exception, when we free the returned `std::exception_ptr` this will happen on the "wrong" core. So Seastar allows it, but inefficiently.
In most cases objects should spend their entire life on a single thread and be used only by this thread. But in some cases we want to reassign ownership of an object which started its life on one thread, to a different thread. This can be done using a `seastar::foreign_ptr<>`. A pointer, or smart pointer, to an object is wrapped in a `seastar::foreign_ptr`. This wrapper can then be moved into code running in a different thread (e.g., using `submit_to()`).
The most common use-case is a `seastar::foreign_ptr>`. The thread receiving this `foreign_ptr` will get exclusive use of the object, and when it destroys this wrapper, it will go back to the original thread to destroy the object. Note that the object is not only freed on the original shard - it is also *destroyed* (i.e., its destructor is run) there. This is often important when the object's destructor needs to access other state which belongs to the original shard - e.g., unlink itself from a container.
Although `foreign_ptr` ensures that the object's *destructor* automatically runs on the object's home thread, it does not absolve the user from worrying where to run the object's other methods. Some simple methods, e.g., methods which just read from the object's fields, can be run on the receiving thread. However, other methods may need to access other data owned by the object's home shard, or need to prevent concurrent operations. Even if we're sure that object is now used exclusively by the receiving thread, such methods must still be run, explicitly, on the home thread:
```
// fp is some foreign_ptr<>
return smp::submit_to(fp.get_owner_shard(), [p=fp.get()]
{ return p->some_method(); });
```
So `seastar::foreign_ptr<>` not only has functional benefits (namely, to run the destructor on the home shard), it also has *documentational* benefits - it warns the programmer to watch out every time the object is used, that this is a *foreign* pointer, and if we want to do anything non-trivial with the pointed object, we may need to do it on the home shard.
Above, we discussed the case of transferring ownership of an object to a another shard, via `seastar::foreign_ptr>`. However, sometimes the sender does not want to relinquish ownership of the object. Sometimes, it wants the remote thread to operate on its object and return with the object intact. Sometimes, it wants to send the same object to multiple shards. In such cases, `seastar::foreign_ptr>` is useful. The user needs to watch out, of course, not to operate on the same object from multiple threads concurrently. If this cannot be ensured by program logic alone, some methods of serialization must be used - such as running the operations on the home shard with `submit_to()` as described above.
Normally, a `seastar::foreign_ptr` cannot not be copied - only moved. However, when it holds a smart pointer that can be copied (namely, a `shared_ptr`), one may want to make an additional copy of that pointer and create a second `foreign_ptr`. Doing this is inefficient and asynchronous (it requires communicating with the original owner of the object to create the copies), so a method `future copy()` needs to be explicitly used instead of the normal copy constructor.
# Seastar::thread
Seastar's programming model, using futures and continuations, is very powerful and efficient. However, as we've already seen in examples above, it is also relatively verbose: Every time that we need to wait before proceeding with a computation, we need to write another continuation. We also need to worry about passing the data between the different continuations (using techniques like those described in the [Lifetime management] section). Simple flow-control constructs such as loops also become more involved using continuations. For example, consider this simple classical synchronous code:
```cpp
std::cout << "Hi.\n";
for (int i = 1; i < 4; i++) {
sleep(1);
std::cout << i << "\n";
}
```
In Seastar, using futures and continuations, we need to write something like this:
```cpp
std::cout << "Hi.\n";
return seastar::do_for_each(boost::counting_iterator(1),
boost::counting_iterator(4), [] (int i) {
return seastar::sleep(std::chrono::seconds(1)).then([i] {
std::cout << i << "\n";
});
});
```
But Seastar also allows you to write code that looks more synchronous by using `seastar::thread`. A `seastar::thread` provides an execution environment where blocking is tolerated: you can invoke an asynchronous function and wait for it in the same function instead of establishing a callback with `future<>::then()`:
```cpp
seastar::thread th([] {
std::cout << "Hi.\n";
for (int i = 1; i < 4; i++) {
seastar::sleep(std::chrono::seconds(1)).get();
std::cout << i << "\n";
}
});
```
A `seastar::thread` is **not** a separate operating system thread. It still uses continuations, which are scheduled on Seastar's single thread (per core). It works as follows:
The `seastar::thread` allocates a 128KB stack, and runs the given function until the it *blocks* on the call to a future's `get()` method. Outside a `seastar::thread` context, `get()` may only be called on a future which is already available. But inside a thread, calling `get()` on a future which is not yet available stops running the thread function, and schedules a continuation for this future, which continues to run the thread's function (on the same saved stack) when the future becomes available.
Just like normal Seastar continuations, `seastar::thread`s always run on the same core they were launched on. They are also cooperative: they are never preempted except when `seastar::future::get()` blocks or on explicit calls to `seastar::thread::yield()`.
It is worth reiterating that a `seastar::thread` is not a POSIX thread, and it can only block on Seastar futures, not on blocking system calls. The above example used `seastar::sleep()`, not the `sleep()` system call. The `seastar::thread`'s function can throw and catch exceptions normally. Remember that `get()` will throw an exception if the future resolves with an exception.
In addition to `seastar::future::get()`, we also have `seastar::future::wait()` to wait *without* fetching the future's result. This can sometimes be useful when you want to avoid throwing an exception when the future failed (as `get()` does). For example:
```cpp
future getchar();
int try_getchar() noexcept { // run this in seastar::thread context
future fut = get_char();
fut.wait();
if (fut.failed()) {
return -1;
} else {
// Here we already know that get() will return immediately,
// and will not throw.
return fut.get();
}
}
```
## Starting and ending a seastar::thread
After we created a `seastar::thread` object, we need wait until it ends, using its `join()` method. We also need to keep that object alive until `join()` completes. A complete example using `seastar::thread` will therefore look like this:
```cpp
#include
#include
seastar::future<> f() {
seastar::thread th([] {
std::cout << "Hi.\n";
for (int i = 1; i < 4; i++) {
seastar::sleep(std::chrono::seconds(1)).get();
std::cout << i << "\n";
}
});
return do_with(std::move(th), [] (auto& th) {
return th.join();
});
}
```
The `seastar::async()` function provides a convenient shortcut for creating a `seastar::thread` and returning a future which resolves when the thread completes:
```cpp
#include
#include
seastar::future<> f() {
return seastar::async([] {
std::cout << "Hi.\n";
for (int i = 1; i < 4; i++) {
seastar::sleep(std::chrono::seconds(1)).get();
std::cout << i << "\n";
}
});
}
```
`seastar::async()`'s lambda may return a value, and `seastar::async()` returns it when it completes. For example:
```cpp
seastar::future read_file(sstring file_name) {
return seastar::async([file_name] () { // lambda executed in a thread
file f = seastar::open_file_dma(file_name).get(); // get() call "blocks"
auto buf = f.dma_read(0, 512).get(); // "block" again
return seastar::sstring(buf.get(), buf.size());
});
};
```
While `seastar::thread`s and `seastar::async()` make programming more convenient, they also add overhead beyond that of programming directly with continuations. Most notably, each `seastar::thread` requires additional memory for its stack. It is therefore not a good idea to use a `seastar::thread` to handle a highly concurrent operation. For example, if you need to handle 10,000 concurrent requests, do not use a `seastar::thread` to handle each --- use futures and continuations. But if you are writing code where you know that only a few instances will ever run concurrently, e.g., a background cleanup operation in your application, `seastar::thread` is a good match. `seastar::thread` is also great for code which doesn't care about performance --- such as test code.
# Isolation of application components
Seastar makes multi-tasking very easy - as easy as running an asynchronous function. It is therefore easy for a server to do many unrelated things in parallel. For example, a server might be in the process of answering 100 users' requests, and at the same time also be making progress on some long background operation.
But in the above example, what percentage of the CPU and disk throughput will the background operation get? How long can one of the user's requests be delayed by the background operation? Without the mechanisms we describe in this section, these questions cannot be reliably answered:
* The background operation may be a very "considerate" single fiber, i.e., run a very short continuation and then schedule the next continuation to run later. At each point the scheduler sees 100 request-handling continuations and just one of the background continuations ready to run. The background task gets around 1% of the CPU time, and users' requests are hardly delayed.
* On the other hand, the background operation may spawn 1,000 fibers in parallel and have 1,000 ready-to-run continuations at each time. The background operation will get about 90% of the runtime, and the continuation handling a user's request may get stuck behind 1,000 of these background continuations, and experience huge latency.
Complex Seastar applications often have different components which run in parallel and have different performance objectives. In the above example we saw two components - user requests and the background operation. The first goal of the mechanisms we describe in this section is to _isolate_ the performance of each component from the others; In other words, the throughput and latency of one component should not depend on decisions that another component makes - e.g., how many continuations it runs in parallel. The second goal is to allow the application to _control_ this isolation, e.g., in the above example allow the application to explicitly control the amount of CPU the background operation receives, so that it completes at a desired pace.
In the above examples we used CPU time as the limited resource that the different components need to share effectively. As we show later, another important shared resource is disk I/O.
## Scheduling groups (CPU scheduler)
Consider the following asynchronous function `loop()`, which loops until some shared variable `stop` becomes true. It keeps a `counter` of the number of iterations until stopping, and returns this counter when finally stopping.
```cpp
seastar::future loop(int parallelism, bool& stop) {
return seastar::do_with(0L, [parallelism, &stop] (long& counter) {
return seastar::parallel_for_each(std::views::iota(0u, parallelism),
[&stop, &counter] (unsigned c) {
return seastar::do_until([&stop] { return stop; }, [&counter] {
++counter;
return seastar::make_ready_future<>();
});
}).then([&counter] { return counter; });
});
}
```
The `parallelism` parameter determines the parallelism of the silly counting operation: `parallelism=1` means we have just one loop incrementing the counter; `parallelism=10` means we start 10 loops in parallel all incrementing the same counter.
What happens if we start two `loop()` calls in parallel and let them run for 10 seconds?
```c++
seastar::future<> f() {
return seastar::do_with(false, [] (bool& stop) {
seastar::sleep(std::chrono::seconds(10)).then([&stop] {
stop = true;
});
return seastar::when_all_succeed(loop(1, stop), loop(1, stop)).then_unpack(
[] (long n1, long n2) {
std::cout << "Counters: " << n1 << ", " << n2 << "\n";
});
});
}
```
It turns out that if the two `loop()` calls had the same parallelism `1`, we get roughly the same amount of work from both of them:
```
Counters: 3'559'635'758, 3'254'521'376
```
But if for example we ran a `loop(1)` in parallel with a `loop(10)`, the result is that the `loop(10)` gets 10 times more work done:
```
Counters: 629'482'397, 6'320'167'297
```
Why does the amount of work that `loop(1)` can do in ten seconds depend on the parallelism chosen by its competitor, and how can we solve this?
The reason this happens is as follows: When a future resolves and a continuation was linked to it, this continuation becomes ready to run. By default, Seastar's scheduler keeps a single list of ready-to-run continuations (in each shard, of course), and runs the continuations at the same order they became ready to run. In the above example, `loop(1)` always has one ready-to-run continuation, but `loop(10)`, which runs 10 loops in parallel, always has ten ready-to-run continuations. So for every continuation of `loop(1)`, Seastar's default scheduler will run 10 continuations of `loop(10)`, which is why loop(10) gets 10 times more work done.
To solve this, Seastar allows an application to define separate components known as **scheduling groups**, which each has a separate list of ready-to-run continuations. Each scheduling group gets to run its own continuations on a desired percentage of the CPU time, but the number of runnable continuations in one scheduling group does not affect the amount of CPU that another scheduling group gets. Let's look at how this is done:
A scheduling group is defined by a value of type `scheduling_group`. This value is opaque, but internally it is a small integer (similar to a process ID in Linux). We use the `seastar::with_scheduling_group()` function to run code in the desired scheduling group:
```cpp
seastar::future
loop_in_sg(int parallelism, bool& stop, seastar::scheduling_group sg) {
return seastar::with_scheduling_group(sg, [parallelism, &stop] {
return loop(parallelism, stop);
});
}
```
TODO: explain what `with_scheduling_group` group really does, how the group is "inherited" to the continuations started inside it.
Now let's create two scheduling groups, and run `loop(1)` in the first scheduling group and `loop(10)` in the second scheduling group:
```cpp
seastar::future<> f() {
return seastar::when_all_succeed(
seastar::create_scheduling_group("loop1", 100),
seastar::create_scheduling_group("loop2", 100)).then_unpack(
[] (seastar::scheduling_group sg1, seastar::scheduling_group sg2) {
return seastar::do_with(false, [sg1, sg2] (bool& stop) {
seastar::sleep(std::chrono::seconds(10)).then([&stop] {
stop = true;
});
return seastar::when_all_succeed(loop_in_sg(1, stop, sg1), loop_in_sg(10, stop, sg2)).then_unpack(
[] (long n1, long n2) {
std::cout << "Counters: " << n1 << ", " << n2 << "\n";
});
});
});
}
```
Here we created two scheduling groups, `sg1` and `sg2`. Each scheduling group has an arbitrary name (which is used for diagnostic purposes only), and a number of *shares*, a number traditionally between 1 and 1000: If one scheduling group has twice the number of shares than a second scheduling group, it will get twice the amount of CPU time. In this example, we used the same number of shares (100) for both groups, so they should get equal CPU time.
Unlike most objects in Seastar which are separate per shard, Seastar wants the identities and numbering of the scheduling groups to be the same on all shards, because it is important when invoking tasks on remote shards. For this reason, the function to create a scheduling group, `seastar::create_scheduling_group()`, is an asynchronous function returning a `future`.
Running the above example, with both scheduling group set up with the same number of shares (100), indeed results in both scheduling groups getting the same amount of CPU time:
```
Counters: 3'353'900'256, 3'350'871'461
```
Note how now both loops got the same amount of work done - despite one loop having 10 times the parallelism of the second loop.
If we change the definition of the second scheduling group to have 200 shares, twice the number of shares of the first scheduling group, we'll see the second scheduling group getting twice the amount of CPU time:
```
Counters: 2'273'783'385, 4'549'995'716
```
## Latency
TODO: Task quota, preempt, loops with built-in preemption check, etc.
## Disk I/O scheduler
TODO
## Network scheduler
TODO: Say that not yet available. Give example of potential problem - e.g., sharing a slow WAN link.
## Controllers
TODO: Talk about how to dynamically change the number of shares, and why.
## Multi-tenancy
TODO
# General application structuring tips
## Initialization and cleanup
In C++, RAII is typically used to coordinate initialization and cleanup. In asyncronous programming, RAII
is not available, but for the control plane it is useful to bring up a seastar::thread to make use of it.
In conjunction with seastar::app_template, initialization and cleanup look like so:
```cpp
#include
#include
#include
int main(int argc, char** argv) {
seastar::app_template app;
return app.run(argc, argv, [] {
return seastar::async() {
std::cout << "Hello world\n";
my_class my_object;
// ... more code
// my_object will be destroyed here
return 0;
});
}
```
If one needs to invoke an asynchronous function during shutdown, one can make
use of the seastar::defer() function to schedule an arbitrary cleanup function:
```cpp
#include
#include
#include
int main(int argc, char** argv) {
seastar::app_template app;
return app.run(argc, argv, [] {
return seastar::async() {
std::cout << "Hello world\n";
my_class my_object;
// ... more code
auto cleanup = defer([] () noexcept {
// .get() joins the asynchronous function back
// into the thread.
call_async_cleanup_function().get();
});
// `cleanup` will invoke its function here, if we reached
// its construction point
// my_object will be destroyed here
return 0;
});
}
```
See also seastar::deferred_close and seastar::deferred_stop for more ways to
control cleanup.
---
### Doc Websocket (`doc/websocket.md`)
# WebSocket protocol implementation
Seastar includes an experimental implementation of a WebSocket server. See
[RFC 6455](https://datatracker.ietf.org/doc/html/rfc6455) and MDN's guide to
[writing WebSocket servers](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers).
## Handlers
A WebSocket server needs a user-defined handler to be functional. The WebSocket specification defines subprotocols, and the Seastar WebSocket server allows one handler to be registered per subprotocol.
Each subprotocol has a unique name and is expected to be sent by the connecting client during handshake,
by sending a `Sec-Websocket-Protocol` header with the chosen value.
Aside from specifying the chosen subprotocol name for a handler, the developer is expected to provide a function
which handles the incoming stream of data and returns responses into the output stream.
Here's an example of how to register a simple echo protocol:
```cpp
using namespace seastar;
static experimental::websocket::server ws;
ws.register_handler("echo", [] (input_stream& in, output_stream& out) -> future<> {
while (true) {
auto buf = co_await in.read();
if (buf.empty()) {
co_return;
}
co_await out.write(std::move(buf));
co_await out.flush();
}
});
```
Note: the developers should assume that the input stream provides decoded and unmasked data - so the stream should be treated as if it was backed by a TCP socket. Similarly, responses should be sent to the output stream as is, and the WebSocket server implementation will handle its proper serialization, masking and so on.
## Error handling
Registered WebSocket handlers can throw arbitrary exceptions during their operation. Currently, exceptions that aren't explicitly handled within the handler will cause the established WebSocket connection to be terminated, and a proper error message will be logged.
## Secure WebSocket (`wss://`)
The WebSocket server does not currently support secure WebSocket connections (`wss://`).
---
--- METRICS ---
- Files Extracted: 27
- Estimated Token Budget: ~68826 tokens
- Canonical Reference: https://codewiki.google/github.com/scylladb/seastar