# Technical Documentation: slawlor/ractor > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/slawlor/ractor) ### Overview (`README.md`) # ractor

*Pronounced rak-ter* A pure-Rust actor framework. Inspired from [Erlang's `gen_server`](https://www.erlang.org/doc/man/gen_server.html), with the speed + performance of Rust! * [github](https://github.com/slawlor/ractor) * [crates.io](https://crates.io/crates/ractor) * [docs.rs](https://docs.rs/ractor) * [docs.rs](https://docs.rs/ractor_cluster) * [](https://github.com/slawlor/ractor/actions/workflows/ci.yaml) * [](https://codecov.io/gh/slawlor/ractor) * `ractor`: * `ractor_cluster`: ## Runtime semantics See the detailed runtime semantics and guarantees in docs/runtime-semantics.md for priority channels, supervision semantics, RPC/cluster behaviors, and recommended best practices: [Semantics](docs/runtime-semantics.md) ## Updates * **Website**: Ractor has a companion website for more detailed getting-started guides along with some best practices and is updated regularly. Api docs will still be available at [docs.rs](https://docs.rs/ractor) however this will be a supplimentary site for `ractor`. Try it out! * **RustConf'24** Ractor was a key part of a presentation at [RustConf'24](https://rustconf.com/schedule/#wednesday). It's used as the basis for Meta's Rust [thrift](https://github.com/facebook/fbthrift) overload protection scheme. The presentation's slides are [available here](https://slawlor.github.io/ractor/assets/rustconf2024_presentation.pdf). ## About `ractor` tries to solve the problem of building and maintaining an Erlang-like actor framework in Rust. It gives a set of generic primitives and helps automate the supervision tree and management of our actors along with the traditional actor message processing logic. It was originally designed to use the `tokio` runtime, however does now support the `async-std` runtime. `ractor` is a modern actor framework written in 100% Rust. Additionally `ractor` has a companion library, `ractor_cluster` which is needed for `ractor` to be deployed in a distributed (cluster-like) scenario. `ractor_cluster` shouldn't be considered production ready, but it is relatively stable and we'd love your feedback! ### Why ractor? There are other actor frameworks written in Rust ([Actix](https://github.com/actix/actix), [riker](https://github.com/riker-rs/riker), or [just actors in Tokio](https://ryhl.io/blog/actors-with-tokio/)) plus a bunch of others like this list compiled on [this Reddit post](https://www.reddit.com/r/rust/comments/n2cmvd/there_are_a_lot_of_actor_framework_projects_on/). Ractor tries to be different by modelling more on a pure Erlang `gen_server`. This means that each actor can also simply be a supervisor to other actors with no additional cost (simply link them together!). Additionally we're aiming to maintain close logic with Erlang's patterns, as they work quite well and are well utilized in the industry. Additionally we wrote `ractor` without building on some kind of "Runtime" or "System" which needs to be spawned. Actors can be run independently, in conjunction with other basic `tokio` runtimes with little additional overhead. We currently have full support for: 1. Single-threaded message processing 2. Actor supervision tree 3. Remote procedure calls to actors in the `rpc` module 4. Timers in the `time` module 5. Named actor registry (`registry` module) from [Erlang's `Registered processes`](https://www.erlang.org/doc/reference_manual/processes.html) 6. Process groups (`ractor::pg` module) from [Erlang's `pg` module](https://www.erlang.org/doc/man/pg.html) On our roadmap is to add more of the Erlang functionality including potentially a distributed actor cluster. ### Performance Actors in `ractor` are generally quite lightweight and there are benchmarks which you are welcome to run on your own host system with: ```bash cargo bench -p ractor ``` Further performance improvements are being tracked in [#262](https://github.com/slawlor/ractor/issues/262) ## Installation Install `ractor` by adding the following to your Cargo.toml dependencies. ```toml [dependencies] ractor = "0.16" ``` The minimum supported Rust version (MSRV) of `ractor` is `1.85`. ## Features `ractor` exposes the following features: 1. `cluster`, which exposes various functionality required for `ractor_cluster` to set up and manage a cluster of actors over a network link. This is work-in-progress and is being tracked in [#16](https://github.com/slawlor/ractor/issues/16). 2. `async-std`, which enables usage of `async-std`'s asynchronous runtime instead of the `tokio` runtime. **However** `tokio` with the `sync` feature remains a dependency because we utilize the messaging synchronization primatives from `tokio` regardless of runtime as they are not specific to the `tokio` runtime. This work is tracked in [#173](https://github.com/slawlor/ractor/pull/173). You can remove default features to "minimize" the tokio dependencies to just the synchronization primatives. 3. `monitors`, Adds support for an erlang-style monitoring api which is an alternative to direct linkage. Akin to [Process Monitors](https://www.erlang.org/doc/system/ref_man_processes.html#monitors) 4. `message_span_propogation`, Propagates the span through the message between actors to keep tracing context. 5. `actor-macros`, which provides opt-in attribute macros for defining actors with one small method per message variant. See the [actor macro guide](docs/actor-macros.md). ## Working with Actors Actors in `ractor` are very lightweight and can be treated as thread-safe. Each actor will only call one of its handler functions at a time, and they will never be executed in parallel. Following the actor model leads to microservices with well-defined state and processing logic. ### Reducing actor boilerplate Enable the `actor-macros` feature to generate the `Actor` implementation from an inherent implementation whose methods describe the messages they handle: ```toml [dependencies] ractor = { version = "0.16", features = ["actor-macros"] } ``` ```rust struct Counter; enum CounterMessage { Add(u64), Read(ractor::RpcReplyPort), } #[ractor::actor(message = CounterMessage, state = u64)] impl Counter { async fn pre_start( &self, _myself: ractor::ActorRef, _arguments: (), ) -> Result { Ok(0) } #[ractor::message(CounterMessage::Add(amount))] fn add(&self, amount: u64, state: &mut u64) { *state += amount; } #[ractor::rpc(CounterMessage::Read(reply))] fn read(&self, state: &u64) -> u64 { *state } } ``` When focused message or RPC handlers are present, their generated dispatch is exhaustive, so adding an unhandled enum variant is a compile error. With neither focused handlers nor a raw `handle`, the actor inherits the trait's no-op message handler; this supports lifecycle- and supervision-only actors. For local actors, `message = enum CounterMessage` can also generate the enum from handler patterns and parameter types, copying handler documentation comments to the generated variants. Focused `#[ractor::supervision(...)]` methods provide the same dispatch style for supervision events, while `#[ractor::rpc(...)]` sends a handler's return value through the omitted reply-port binding. The explicit trait-based API remains available when its flexibility is preferable. See the [guide](docs/actor-macros.md) and the [`actor_macro` example](ractor/examples/actor_macro.rs) for the supported method shapes. An example `ping-pong` actor might be the following ``` /* Detailed source-code truncated for AI context efficiency. */ ``` which will output ```bash $ cargo run ping..pong..ping..pong..ping..pong..ping..pong..ping..pong.. $ ``` ## Messaging actors The means of communication between actors is that they pass messages to each other. A developer can define any message type which is `Send + 'static` and it will be supported by `ractor`. There are 4 concurrent message types, which are listened to in priority. They are 1. Signals: Signals are the highest-priority of all and will interrupt the actor wherever processing currently is (this includes terminating async work). There is only 1 signal today, which is `Signal::Kill`, and it immediately terminates all work. This includes message processing or supervision event processing. 2. Stop: There is also the pre-defined stop signal. You can give a "stop reason" if you want, but it's optional. Stop is a graceful exit, meaning currently executing async work will complete, and on the next message processing iteration Stop will take priority over future supervision events or regular messages. It will **not** terminate currently executing work, regardless of the provided reason. 3. SupervisionEvent: Supervision events are messages from child actors to their supervisors in the event of their startup, death, and/or unhandled panic. Supervision events are how an actor's supervisor(parent) or peer monitors are notified of events of their children/peers and can handle lifetime events for them. If you set `panic = 'abort'` in your `Cargo.toml`, panics **will** start cause program termination and not be caught in the supervision flow. 4. Messages: Regular, user-defined, messages are the last channel of communication to actors. They are the lowest priority of the 4 message types and denote general actor work. The first 3 messages types (signals, stop, supervision) are generally quiet unless it's a lifecycle event for the actor, but this channel is the "work" channel doing what your actor wants to do! ## Ractor in distributed clusters Ractor actors can also be used to build a distributed pool of actors, similar to [Erlang's EPMD](https://www.erlang.org/doc/man/epmd.html) which manages inter-node connections + node naming. In our implementation, we have [`ractor_cluster`](https://crates.io/crates/ractor_cluster) in order to facilitate distributed `ractor` actors. `ractor_cluster` has a single main type in it, namely the `NodeServer` which represents a host of a `node()` process. It additionally has some macros and a procedural macros to facilitate developer efficiency when building distributed actors. The `NodeServer` is responsible for 1. Managing all incoming and outgoing `NodeSession` actors which represent a remote node connected to this host. 2. Managing the `TcpListener` which hosts the server socket to accept incoming session requests. The bulk of the logic for node interconnections however is held in the `NodeSession` which manages 1. The underlying TCP connection managing reading and writing to the stream. 2. The authentication between this node and the connection to the peer 3. Managing actor lifecycle for actors spawned on the remote system. 4. Transmitting all inter-actor messages between nodes. 5. Managing PG group synchronization etc.. The `NodeSession` makes local actors available on a remote system by spawning `RemoteActor`s which are essentially untyped actors that only handle serialized messages, leaving message deserialization up to the originating system. It also keeps track of pending RPC requests, to match request to response upon reply. There are special extension points in `ractor` which are added to specifically support `RemoteActor`s that aren't generally meant to be used outside of the standard ```rust Actor::spawn(Some("name".to_string()), MyActor).await ``` pattern. ### Designing remote-supported actors **Note** not all actors are created equal. Actors need to support having their message types sent over the network link. This is done by overriding specific methods of the `ractor::Message` trait all messages need to support. Due to the lack of specialization support in Rust, if you choose to use `ractor_cluster` you'll need to derive the `ractor::Message` trait for **all** message types in your crate. However to support this, we have a few procedural macros to make this a more painless process #### Deriving the basic Message trait for in-process only actors Many actors are going to be local-only and have no need sending messages over the network link. This is the most basic scenario and in this case the default `ractor::Message` trait implementation is fine. You can derive it quickly with: ```rust use ractor_cluster::RactorMessage; use ractor::RpcReplyPort; #[derive(RactorMessage)] enum MyBasicMessageType { Cast1(String, u64), Call1(u8, i64, RpcReplyPort>), } ``` This will implement the default ```ractor::Message``` trait for you without you having to write it out by hand. #### Deriving the network serializable message trait for remote actors If you want your actor to *support* remoting, then you should use a different derive statement, namely: ```rust use ractor_cluster::RactorClusterMessage; use ractor::RpcReplyPort; #[derive(RactorClusterMessage)] enum MyBasicMessageType { Cast1(String, u64), #[rpc] Call1(u8, i64, RpcReplyPort>), } ``` which adds a significant amount of underlying boilerplate (take a look yourself with `cargo expand`) for the implementation. But the short answer is, each enum variant needs to serialize to a byte array of arguments, a variant name, and if it's an RPC give a port that receives a byte array and de-serialize the reply back. Each of the types inside of either the arguments or reply type need to implement the ```ractor_cluster::BytesConvertable``` trait which just says this value can be written to a byte array and decoded from a byte array. If you're using `prost` for your message type definitions (protobuf), we have a macro to auto-implement this for your types. ```rust ractor_cluster::derive_serialization_for_prost_type! {MyProtobufType} ``` Besides that, just write your actor as you would. The actor itself will live where you define it and will be capable of receiving messages sent over the network link from other clusters! ## Differences between an actor's "state" and `self` Actors can (but don't need to!) have internal state. In order to facilitate this `ractor` gives implementors of the `Actor` trait the ability to define the state type for an actor. The actor's `pre_start` routine is what initializes and sets up this state. You can imagine doing things like 1. Opening a network socket + storing the `TcpListener` in the state 2. Setting up a database connection + authenticating to the DB 3. Initializing basic state variables (counters, stats, whatever) Because of this and the possibility that some of these operations are fallible, `pre_start` captures panic's in the method during the initialization and returns them to the caller of `Actor::spawn`. When designing `ractor`, we made the explicit decision to make a separate state type for an actor, rather than passing around a mutable `self` reference. The reason for this is that if we were to use a `&mut self` reference, creation + instantiation of the `Self` struct would be outside of the actor's specification (i.e. not in `pre_start`) and the safety it gives would be potentially lost, causing potential crashes in the caller when it maybe shouldn't. Lastly is that we would need to change some of the ownership properties that `ractor` is currently based on to pass an owned `self` in each call, returning a `Self` reference which seems clunky in this context. **In the current realization** an actor's `self` is passed as a read-only reference which shouldn't ideally contain state information, but could contain configuration / startup information if you want. However there is also `Arguments` to each `Actor` which allows passing owned values to the state of an actor. In an ideal world, all actor structs would be empty with no stored values. ## Contributors The original author of `ractor` is Sean Lawlor (@slawlor). To learn more about contributing to `ractor` please see [CONTRIBUTING.md](https://github.com/slawlor/ractor/blob/main/CONTRIBUTING.md). ## License This project is licensed under [MIT](https://github.com/slawlor/ractor/blob/main/LICENSE). --- ### Overview (`ractor_cluster/README.md`) # ractor_cluster

*A companion crate to `ractor` for supporting remote actors* [github](https://github.com/slawlor/ractor) [crates.io](https://crates.io/crates/ractor_cluster) [docs.rs](https://docs.rs/ractor_cluster) [](https://github.com/slawlor/ractor/actions/workflows/ci.yaml) [](https://codecov.io/gh/slawlor/ractor) This crate contains extensions to `ractor`, a pure-Rust actor framework. Inspired from [Erlang's `gen_server`](https://www.erlang.org/doc/man/gen_server.html). **Website** Ractor has a companion website for more detailed getting-started guides along with some best practices and is updated regularly. Api docs will still be available at [docs.rs](https://docs.rs/ractor) however this will be a supplimentary site for `ractor`. Try it out! ## About `ractor_cluster` expands upon `ractor` actors to support transmission over a network link and synchronization of actors on networked clusters of actors. ## Installation Install `ractor_cluster` by adding the following to your Cargo.toml dependencies ```toml [dependencies] ractor = { version = "0.13", features = ["cluster"] } ractor_cluster = "0.13" ``` ## Ractor in distribucted clusters Ractor actors can be built in a network-distributed pool of actors, similar to [Erlang's EPMD](https://www.erlang.org/doc/man/epmd.html) which manages inter-node connections + node naming. In our implementation, we have [`ractor_cluster`](https://crates.io/crates/ractor_cluster) in order to facilitate distributed `ractor` actors. `ractor_cluster` has a single main type in it, namely the `NodeServer` which represents a host of a `node()` process. It additionally has some macros and a procedural macros to facilitate developer efficiency when building distributed actors. The `NodeServer` is responsible for: 1. Managing all incoming and outgoing `NodeSession` actors which represent a remote node connected to this host. 2. Managing the `TcpListener` which hosts the server socket to accept incoming session requests (with or without encryption). The bulk of the logic for node interconnections however is held in the `NodeSession` which manages 1. The underlying TCP connection managing reading and writing to the stream. 2. The authentication between this node and the connection to the peer 3. Managing actor lifecycle for actors spawned on the remote system. 4. Transmitting all inter-actor messages between nodes. 5. Managing PG group synchronization etc.. The `NodeSession` makes local actors available on a remote system by spawning `RemoteActor`s which are essentially untyped actors that only handle serialized messages, leaving message deserialization up to the originating system. It also keeps track of pending RPC requests, to match request to response upon reply. There are special extension points in `ractor` which are added to specifically support `RemoteActor`s that aren't generally meant to be used outside of the standard ```rust Actor::spawn(Some("name".to_string()), MyActor).await ``` pattern. ### Quick-start The basics of setting up a networked cluster of actors lives in the `NodeServer` struct. This structure handles the low-level network ownership over a server port along with all of the lifecycle of cluster inter-connections. By spawning this single struct, you're able to accept incoming connections between hosts! Nodes in the network are authenticated to each other with a "magic cookie" following the Erlang specification in [Erlang's distribution protocol](https://www.erlang.org/doc/apps/erts/erl_dist_protocol.html). If you want to connect to another host, you need to 1. Initialize your own `NodeServer` 2. Execute a "client-connection" to the remote `NodeServer` you're trying to connect to like ```rust let host = "1.2.3.4"; let port = "4697"; ractor_cluster::client_connect( &actor, format!("{host}:{port}"), ) ``` Similarly there is a `client_connect_enc` to connect to a `NodeServer` which is utilizing encrypted communication. That's it! If your nodes are sharing a proper magic cookie value, they should authenticate to each other and you'll see remote actors spawned on your local system which you can communciate with through the various `pg` or `pid`-based registries. ### Bring Your Own Transport If you want to run the cluster protocol over a custom transport (for example QUIC, WebSocket, SSH tunnel, or an in-memory pipe), you can implement a tiny trait and inject a connection into the `NodeServer`. - Implement `ractor_cluster::ClusterBidiStream` for your connected stream type. It just needs to split into a read and a write half that implement `tokio::io::AsyncRead`/`AsyncWrite`. - The cluster protocol framing remains identical: a big-endian length prefix followed by a prost-encoded payload. You don't need to touch authentication, control, remote actor, or PG sync logic. - Open a connection in your app, wrap it as `ClusterBidiStream`, and send it to the node server using `NodeServerMessage::ConnectionOpenedExternal` or via the helper `client_connect_external`. Example: ```rust use ractor_cluster::{ClusterBidiStream, BoxRead, BoxWrite}; use tokio::io::DuplexStream; struct MyDuplex(DuplexStream); impl ClusterBidiStream for MyDuplex { fn split(self: Box) -> (BoxRead, BoxWrite) { let (r, w) = tokio::io::split(self.0); (Box::new(r), Box::new(w)) } fn peer_label(&self) -> Option { Some("duplex:peer".into()) } fn local_label(&self) -> Option { Some("duplex:local".into()) } } // elsewhere in async context // let (a, b) = tokio::io::duplex(64 * 1024); // node_server.cast(NodeServerMessage::ConnectionOpenedExternal { stream: Box::new(MyDuplex(a)), is_server: false })?; ``` The existing TCP and TLS paths continue to work as before. External transports are purely additive. ### Designing remote-supported actors **Note** not all actors are created equal. Actors need to support having their message types sent over the network link. This is done by overriding specific methods of the `ractor::Message` trait all messages need to support. Due to the lack of specialization support in Rust, if you choose to use `ractor_cluster` you'll need to derive the `ractor::Message` trait for **all** message types in your crate. However to support this, we have a few procedural macros to make this a more painless process #### Deriving the basic Message trait for in-process only actors Many actors are going to be local-only and have no need sending messages over the network link. This is the most basic scenario and in this case the default `ractor::Message` trait implementation is fine. You can derive it quickly with: ```rust use ractor_cluster::RactorMessage; use ractor::RpcReplyPort; #[derive(RactorMessage)] enum MyBasicMessageType { Cast1(String, u64), Call1(u8, i64, RpcReplyPort>), } ``` The will implement the default ```ractor::Message``` trait for you without you having to write it out by hand. #### Deriving the network serializable message trait for remote actors If you want your actor to *support* remoting, then you should use a different derive statement, namely: ```rust use ractor_cluster::RactorClusterMessage; use ractor::RpcReplyPort; #[derive(RactorClusterMessage)] enum MyBasicMessageType { Cast1(String, u64), #[rpc] Call1(u8, i64, RpcReplyPort>), } ``` which adds a significant amount of underlying boilerplate (take a look yourself with `cargo expand`!) for the implementation. But the short answer is, each enum variant needs to serialize to a byte array of arguments, a variant name, and if it's an RPC give a port that receives a byte array and de-serialize the reply back. Each of the types inside of either the arguments or reply type need to implement the ```ractor_cluster::BytesConvertable``` trait which just says this value can be written to a byte array and decoded from a byte array. If you're using `prost` for your message type definitions (protobuf), we have a macro to auto-implement this for your types. ```rust ractor_cluster::derive_serialization_for_prost_type! {MyProtobufType} ``` Besides that, just write your actor as you would. The actor itself will live where you define it and will be capable of receiving messages sent over the network link from other clusters! ## Contributors The original author of `ractor` is Sean Lawlor (@slawlor). To learn more about contributing to `ractor` please see [CONTRIBUTING.md](https://github.com/slawlor/ractor/blob/main/CONTRIBUTING.md) ## License This project is licensed under [MIT](https://github.com/slawlor/ractor/blob/main/LICENSE). --- ### Overview (`xtask/README.md`) This package is included here to support the automatic reporting of code coverage on Github. To view code coverage locally: Do this once to set it up: ``` rustup component add llvm-tools-preview cargo install grcov ``` Subsequently, run: ``` cargo xtask coverage --dev ``` --- ### Overview (`ractor_cluster_integration_tests/test-ca/README.md`) # Rustls Test CA This directory contains various test certificate authorities, intermediates, end-entity, and client certificates that are used by Rustls integration tests. You can regenerate the data in this directory by running the `rustls/examples/internal/test_ca.rs` tool: ```bash cargo run -p rustls --example test_ca ``` --- ### Actor Macros (`docs/actor-macros.md`) # Actor definition macros The optional `actor-macros` feature removes the repetitive associated types and message dispatch from an actor definition while leaving its behavior visible as ordinary Rust methods. ```toml [dependencies] ractor = { version = "0.16", features = ["actor-macros"] } ``` ## Defining an actor Apply `#[ractor::actor]` to an inherent implementation and mark each message handler with `#[ractor::message]`. The usual form uses an explicitly declared message enum: ```rust use ractor::{ActorProcessingErr, ActorRef, RpcReplyPort}; struct Counter; enum CounterMessage { Add(i64), Read(RpcReplyPort), } #[ractor::actor( message = CounterMessage, state = i64, arguments = i64, )] impl Counter { async fn pre_start( &self, _myself: ActorRef, initial: i64, ) -> Result { Ok(initial) } #[ractor::message(CounterMessage::Add(amount))] fn add(&self, amount: i64, state: &mut i64) { *state += amount; } #[ractor::rpc(CounterMessage::Read(reply))] fn read(&self, state: &i64) -> i64 { *state } } ``` Either `message` or the generated-message form described below is required. `state` and `arguments` default to `()`. The macro generates the `Actor` implementation and, when focused message or RPC handlers are present, an exhaustive `match` over the message enum. Rust therefore reports a new or forgotten message variant at compile time in focused-dispatch mode. Cargo dependency renames are detected automatically, and non-Cargo builds default to `::ractor`. The `crate_path = some::path` option is only necessary when a non-Cargo build renames the dependency. Tuple, struct, and unit variants are supported. Bindings in the attribute pattern become handler parameters with matching names in the same order; `_` can discard an unused field. Keep patterns flat so that the method signature remains an immediate description of the message payload. ## Generating local message enums For local actors, `message = [visibility] enum EnumName` generates the message enum from the handler patterns and parameter types: ```rust struct Counter; #[ractor::actor( message = pub enum CounterMessage, state = i64, arguments = i64, )] impl Counter { async fn pre_start( &self, _myself: ActorRef, initial: i64, ) -> Result { Ok(initial) } /// Add an amount to the counter. #[ractor::message(Add { amount })] fn add(&self, amount: i64, state: &mut i64) { *state += amount; } /// Read the current counter value. #[ractor::rpc(Read)] fn read(&self, state: &i64) -> i64 { *state } } ``` This generates the equivalent of: ```rust pub enum CounterMessage { /// Add an amount to the counter. Add { amount: i64 }, /// Read the current counter value. Read(RpcReplyPort), } ``` Unit, tuple, and struct variants are supported. Every generated field must have a named binding so the macro can find its type; `_` fields are therefore only supported when using an explicit enum. The optional visibility defaults to private and can be any normal Rust visibility, such as `pub(crate)`. Generated enums currently do not support generic actor implementations. Textual documentation comments (`///`, `/** ... */`, and `#[doc = ...]`) on message and RPC handlers are copied to the corresponding generated variants while remaining on the methods. Conditional `cfg` and textual documentation attributes retain their structure. Other method attributes are not copied, and explicit message enums remain entirely caller-owned. Copied links and code examples are documented in the enum variant's context, so write them as message documentation rather than method-specific implementation notes. The unit-form RPC attribute above describes a request with no payload. The generated enum still contains the reply port, so `Read` becomes `Read(RpcReplyPort)`. A tuple or struct RPC pattern can instead name the reply binding explicitly, as described below. Generated-message mode deliberately does not add cluster serialization derives or define a wire contract. Continue using an explicit message enum for remotely serializable actors. In a build with the `cluster` feature, a generated enum used only for local messaging can opt into `Message` with a manual implementation: ```rust #[cfg(feature = "cluster")] impl ractor::Message for CounterMessage {} ``` ## Handler methods A generated handler can be synchronous or asynchronous and has this shape: ```text &self, [ActorRef], [bound message fields...], [&State | &mut State] ``` The actor reference and state parameters are optional. Use `&State` for read-only access or `&mut State` to update state. A handler with no explicit return type (or `-> ()`) is infallible. A handler with an explicit non-unit return type is propagated through `Actor::handle` with `?`, so it must support that operator; `Result` is the usual choice. Lifecycle hooks such as `pre_start`, `post_start`, `handle_supervisor_evt`, and `post_stop` can be written with their normal trait signatures inside the same implementation. If both state and arguments are `()`, an omitted `pre_start` defaults to `Ok(())`; otherwise it is required. For actors that need custom dispatch, define the normal `handle` method instead of any `#[ractor::message]` or `#[ractor::rpc]` methods. Raw `handle` and focused handlers are intentionally mutually exclusive, keeping the dispatch path unambiguous. An actor may also omit both forms. In that case the macro emits no `handle` method and inherits the actor trait's no-op default. This is useful for actors that only run lifecycle hooks or process focused `#[ractor::supervision(...)]` events. The `message` option is still required to select the actor's message type; `message = enum EmptyMessage` produces an empty local enum when there are no focused message or RPC handlers. ## RPC handlers Use `#[ractor::rpc]` when a message contains an `RpcReplyPort` and the handler should return its reply directly. Exactly one named binding from the pattern must be absent from the method parameters. That binding is the reply port; all other bindings follow the same ordering and optional actor-reference/state rules as one-way message handlers: ```rust enum StoreMessage { Get { key: String, reply: RpcReplyPort>, }, AddAndRead(i64, RpcReplyPort, i64), } #[ractor::rpc(StoreMessage::Get { key, reply })] async fn get(&self, key: String, state: &StoreState) -> Option { state.get(&key).cloned() } #[ractor::rpc(StoreMessage::AddAndRead(left, reply, right))] fn add_and_read(&self, left: i64, right: i64, state: &StoreState) -> i64 { left + right + state.offset } ``` The reply port may occur at any tuple or struct field position. A unit pattern such as `#[ractor::rpc(StoreMessage::Status)]` is shorthand for an RPC with no request fields; the explicit enum must still define `Status(RpcReplyPort)`. For generated message enums, the macro adds that field automatically. The `call!` and `call_t!` convenience forms place the reply port after their supplied tuple arguments. For a middle-position port or a struct variant, use `ActorRef::call` with a closure so the message construction remains explicit: ```rust let result = store .call( |reply| StoreMessage::AddAndRead(20, reply, 22), Some(ractor::concurrency::Duration::from_millis(100)), ) .await?; let value = store .call( |reply| StoreMessage::Get { key: "answer".to_owned(), reply, }, Some(ractor::concurrency::Duration::from_millis(100)), ) .await?; ``` By default, the handler's complete return type is the reply type. This is intentional: a domain result such as `Result` is sent to the caller unchanged. Reply types must be concrete; `impl Trait` cannot appear within them because generated message fields and reply ports must be able to name the type. ```rust #[ractor::rpc(StoreMessage::Validate(reply))] fn validate(&self, state: &StoreState) -> Result { state.validate() } ``` To propagate an actor-processing error from the generated `handle` method instead, specify the successful reply type explicitly. The generated dispatch applies `?` to the handler call and only sends the successful value: ```rust #[ractor::rpc(StoreMessage::Read(reply), reply = Value)] async fn read(&self, state: &StoreState) -> Result { state.read().await } ``` Failure to send because the caller dropped its receiver is ignored and does not fail the actor. Use an ordinary `#[ractor::message]` handler when code needs to inspect, retain, forward, or send through the reply port manually. ## Supervision handlers Supervision events can be split into focused methods in the same way as actor messages: ```rust #[ractor::supervision(SupervisionEvent::ActorStarted(child))] fn child_started(&self, child: ActorCell, state: &mut SupervisorState) { state.children.insert(child.get_id()); } #[ractor::supervision(SupervisionEvent::ActorFailed(child, error))] async fn child_failed( &self, myself: ActorRef, child: ActorCell, error: ActorProcessingErr, state: &mut SupervisorState, ) -> Result<(), ActorProcessingErr> { // Apply the supervisor's restart policy. Ok(()) } ``` Supervision handlers may be synchronous or asynchronous, may return a fallible result, and may take an optional leading actor reference and trailing state reference. Events matched by a handler are fully handled by that method. Unmatched events retain Ractor's default behavior, including stopping the supervisor after an unhandled child termination or failure. A raw `handle_supervisor_evt` method remains available for custom dispatch, but cannot be combined with `#[ractor::supervision(...)]` methods in the same actor implementation. ## Thread-local and cluster actors Add `thread_local` to generate a `ThreadLocalActor` implementation: ```rust #[derive(Default)] struct UiActor; #[ractor::actor(thread_local, message = UiMessage, state = UiState)] impl UiActor { // lifecycle hooks and #[ractor::message(...)] handlers } ``` The macro only defines actor behavior. Cluster wire compatibility remains an explicit property of the message type: continue deriving `RactorMessage` for local-only cluster builds or `RactorClusterMessage` for remotely serializable messages. A focused actor `#[ractor::rpc(...)]` handler does not generate or alter the enum's cluster `#[rpc]` metadata, serialization, or wire contract; those remain on the explicit message enum. See the complete [`actor_macro` example](../ractor/examples/actor_macro.rs). --- ### Advanced Benchmarks (`docs/advanced_benchmarks.md`) # Advanced Benchmarks for Ractor Performance Analysis ## Overview This document describes the advanced benchmark suite designed to provide deeper insights into Ractor's performance characteristics beyond the standard benchmarks. ## Benchmark Categories ### 1. Message Size Distribution (`message_size_distribution`) **Purpose:** Understand how message size affects performance and identify allocation overhead. **What it tests:** - Message processing throughput for sizes: 1B, 8B, 16B, 32B, 64B, 128B, 256B, 1KB - Each benchmark sends 10,000 messages and measures end-to-end processing time - Uses `Throughput::Bytes` to show MB/s for comparison **Why it's useful:** - **Identifies optimal message sizes** - Shows where heap allocation overhead becomes significant - **Validates SBO threshold** - If SBO were viable, would show which size threshold (24B, 32B, 48B) gives best results - **Cache effects** - Larger messages may show degradation due to cache misses - **Baseline for optimization** - Establishes cost model for different message types **Expected insights:** - Small messages (≤16B): Should be fastest per message - Medium messages (32-64B): Likely sweet spot for throughput - Large messages (≥256B): May show cache/allocation overhead **How to interpret results:** ```text message_size_distribution/8 time: [150 ns] throughput: [533 MB/s] message_size_distribution/64 time: [165 ns] throughput: [3.88 GB/s] ``` - If 64B is only 10% slower than 8B → allocation overhead is minimal - If 256B is 50% slower than 64B → cache effects or allocation pressure --- ### 2. Supervision Heavy Workload (`supervision_heavy`) **Purpose:** Measure supervision tree overhead under realistic hierarchical actor systems. **What it tests:** - Three configurations: - `depth3_breadth10`: Shallow wide tree (30 total actors) - `depth5_breadth5`: Balanced tree (775 total actors) - `depth10_breadth2`: Deep narrow tree (2046 total actors) - Spawns entire tree with linked supervision - Triggers cascading shutdown from root - Measures full lifecycle including supervision events **Why it's useful:** - **Tests Phase 1.3 optimization** - Validates supervision lock optimization impact - **Real-world workload** - Many actor systems use hierarchical supervision - **Identifies lock contention** - Deep trees stress supervision event handling - **Scalability testing** - Shows how supervision overhead scales with tree size **Expected insights:** - Phase 1.3 should show improvement in wide trees (more parallel supervision events) - Deep trees should benefit from `for_each_child()` optimization (Phase 2.4) - Cascading shutdown time should scale sub-linearly with tree size **How to interpret results:** ```text supervision_heavy/depth3_breadth10 time: [500 µs] supervision_heavy/depth5_breadth5 time: [2.5 ms] supervision_heavy/depth10_breadth2 time: [5 ms] ``` - Compare depth5 vs depth3: Should scale ~5-10x (not 25x despite 25x more actors) - If linear scaling → lock contention problem - If exponential → algorithm issue --- ### 3. Mixed Workload (`mixed_workload_100workers_1000msgs`) **Purpose:** Simulate realistic actor system with spawning, messaging, and supervision combined. **What it tests:** - Spawns coordinator actor with 100 supervised child workers - Sends 1,000 messages to each worker (100,000 total messages) - Workers process messages and maintain state - Coordinator triggers graceful shutdown of all workers - Full lifecycle: spawn → work → supervise → shutdown **Why it's useful:** - **Realistic benchmark** - Combines all optimizations in real-world scenario - **Overall system performance** - Not just isolated hot paths - **Integration testing** - Ensures optimizations don't conflict - **User-facing metric** - Represents actual application performance **Expected insights:** - Should show cumulative benefit of all Phase 1+2 optimizations - Better indicator of real-world improvement than micro-benchmarks - Helps validate that 15-20% improvement claim holds in practice **How to interpret results:** ```text mixed_workload_100workers_1000msgs time: [85 ms] ``` - Compare with baseline before optimizations - Should correlate with `process_messages` benchmark - Larger improvement → optimizations work well together - Smaller improvement → overhead from other operations dominates --- ### 4. Message Send Microbenchmark (`message_send_only_10000`) **Purpose:** Isolate message send operation to measure boxing overhead independent of processing. **What it tests:** - Spawns single no-op actor that doesn't process messages - Sends 10,000 messages as fast as possible - Measures just the send path: boxing + channel send - Actor runs in background but doesn't process (isolates send cost) **Why it's useful:** - **Hot path isolation** - Removes actor processing time from equation - **Boxing overhead** - Directly measures `Message::box_message()` cost - **Type checking cost** - Shows impact of Phase 2.2 optimization - **SBO validation** - Would show if inline storage helps send path **Expected insights:** - Phase 2.2 (type check removal) should improve this benchmark - Phase 2.3 (downcast optimization) won't affect this (only receive side) - Very low variance → consistent boxing performance - High variance → potential contention in channel **How to interpret results:** ```text message_send_only_10000 time: [50 µs] → ~5ns per send ``` - Compare with `process_messages` throughput (~150ns per message) - Send overhead should be <10% of total message processing time - If send is >20% → boxing is bottleneck --- ### 5. Spawn Rate Benchmark (`spawn_rate_1000actors`) **Purpose:** Measure maximum actor creation throughput. **What it tests:** - Spawns 1,000 minimal actors that immediately stop - Waits for all to complete - Measures pure spawn overhead without message processing - Sequential spawning (not parallel) **Why it's useful:** - **Tests Phase 1.2 optimization** - Arc clone reduction during spawn - **Startup performance** - Important for systems that create many actors - **Spawn overhead quantification** - Shows fixed cost per actor - **Scalability baseline** - Reference for how many actors you can create **Expected insights:** - Phase 1.2 should show 1-2% improvement - Should scale linearly with actor count - Helps understand fixed vs variable costs in spawn **How to interpret results:** ```text spawn_rate_1000actors time: [2 ms] → 2µs per actor ``` - Compare with `create_actors` benchmark (includes handle waiting) - Should be slightly faster since actors immediately stop - Throughput: ~500,000 actors/second is good - <100,000 actors/second → spawn overhead too high --- ## Running the Benchmarks ### Run all advanced benchmarks ```bash cargo bench --bench simple_advanced_benchmarks ``` ### Run specific category ```bash cargo bench --bench simple_advanced_benchmarks -- small_messages cargo bench --bench simple_advanced_benchmarks -- large_messages cargo bench --bench simple_advanced_benchmarks -- spawn ``` ### Compare with SBO feature ```bash # Baseline cargo bench --bench simple_advanced_benchmarks -- small_messages > /tmp/baseline.txt # With SBO cargo bench --bench simple_advanced_benchmarks --features small-buffer-optimization -- small_messages > /tmp/sbo.txt # Compare critcmp baseline.txt sbo.txt ``` ### Generate flamegraphs (requires cargo-flamegraph) ```bash cargo flamegraph --bench simple_advanced_benchmarks -- --bench small_messages ``` --- ## Expected Results Analysis ### Phase 1+2 Optimizations (Proven) **Message Size Distribution:** - All sizes: 3-5% improvement - Uniform across sizes (not size-dependent) **Supervision Heavy:** - Wide trees (depth3_breadth10): 5-8% improvement - Deep trees (depth10_breadth2): 2-4% improvement **Mixed Workload:** - Overall: 12-18% improvement (cumulative effect) **Message Send:** - 2-3% improvement from type check removal **Spawn Rate:** - 1-2% improvement from Arc clone reduction ### SBO Feature (Regression Found) **Message Size Distribution:** - ≤32B messages: **20-30% SLOWER** (enum overhead + ManuallyDrop) - >32B messages: **10-15% SLOWER** (larger BoxedMessage struct) **Why SBO Failed:** 1. Enum matching overhead on every box/unbox 2. Larger struct size (80 bytes) → worse cache locality 3. ManuallyDrop wrapper overhead 4. Runtime type ID + size verification 5. Extra indirection in match arms --- ## Diagnostic Workflow ### Problem: Performance regression in production 1. Run **Mixed Workload** to see overall impact 2. Run **Message Size Distribution** to identify if message size is factor 3. Run **Supervision Heavy** if system uses deep actor hierarchies 4. Run **Message Send** to isolate send vs receive overhead ### Problem: Optimization not showing expected gains 1. Run **Spawn Rate** to verify spawn optimization worked 2. Run **Message Send** to verify type check removal worked 3. Run **Mixed Workload** to see if gains are masked by other overhead ### Problem: Validating new optimization 1. Run **all benchmarks** before change → save results 2. Implement optimization 3. Run **all benchmarks** after change → compare 4. Use `critcmp` or Criterion's built-in comparison --- ## Adding New Benchmarks When adding new benchmarks to this suite, ensure: 1. **Clear purpose** - What specific aspect does it measure? 2. **Isolation** - Does it test one thing or multiple? 3. **Reproducibility** - Same results across runs? 4. **Meaningful metrics** - Throughput, latency, or both? 5. **Documentation** - Update this file with interpretation guide --- ## Benchmark Limitations ### What these DON'T test - **Memory allocations** - Use `cargo instruments` or `heaptrack` - **Cache misses** - Use `perf stat -e cache-misses` - **Lock contention** - Use `cargo flamegraph` or `pprof` - **Network overhead** - Cluster features not benchmarked here - **Real workload** - Synthetic benchmarks may not match production ### Complementary Tools ```bash # Allocation profiling cargo instruments -t Allocations --bench simple_advanced_benchmarks # Cache analysis perf stat -e cache-references,cache-misses cargo bench --bench simple_advanced_benchmarks # Lock contention cargo flamegraph --bench simple_advanced_benchmarks -- --profile-time 60 # Memory usage heaptrack cargo bench --bench simple_advanced_benchmarks ``` --- ## Conclusion These advanced benchmarks provide targeted insights into: - ✅ Message size performance characteristics - ✅ Supervision tree overhead - ✅ Real-world mixed workloads - ✅ Isolated hot path operations - ✅ Spawn throughput Combined with the standard benchmarks, they give a complete picture of Ractor's performance profile and help validate optimizations before production deployment. **Key Takeaway:** Always benchmark with both micro-benchmarks AND realistic workloads. Phase 1+2 optimizations showed 15-20% improvement in realistic scenarios, while SBO showed 20-30% regression despite being technically correct. --- ### Runtime Semantics (`docs/runtime-semantics.md`) # Runtime semantics This document describes the runtime semantics, guarantees, and important edge-cases for ractor. It is intended to be a concise, actionable reference for developers building on ractor or ractor_cluster. ## Core guarantees - Single-message processing: An actor processes at most one message handler at a time. Handlers for a single actor are never executed in parallel. - Message types: User message types must be `Send + 'static`. For clustered usage, messages must additionally implement the network serialization traits (`ractor::Message` via derive macros, and `ractor_cluster::BytesConvertable` for payloads). - Pre-start initialization: Actor state is created in `pre_start`. `pre_start` is run at spawn time and may fail; failures are returned to the caller of `Actor::spawn`. - Actor `self` vs state: `self` is read-only and intended for configuration. Mutable actor state belongs to the `State` type returned by `pre_start`. ## Message priority & ordering Message delivery is multiplexed into four prioritized channels. Higher priorities are processed before lower ones: 1. Signals (highest) — e.g., `Signal::Kill`. Signals immediately interrupt processing and will terminate execution (see Signals section). 2. Stop — a graceful stop request. Ongoing asynchronous work is allowed to complete; on the actor's next processing iteration Stop takes priority over supervision events and user messages. 3. SupervisionEvent — lifecycle notifications from child actors (startup, exit, panic). 4. Messages (lowest) — user-defined messages (cast/call). Ordering guarantees: - Per-sender FIFO: Messages sent from a single sender to a single actor are delivered in the order they were sent (FIFO) for the user message channel when using the provided `ActorRef` APIs. - Cross-sender ordering: No ordering guarantee between messages from different senders. - Priority preemption: Higher-priority channel messages can preempt lower-priority messages in the queue. A Stop message will be processed before later-arriving user messages. - Local vs remote: Local ordering guarantees apply for in-process messaging. On the network (ractor_cluster), ordering depends on the transport and node session behavior — ordering can be preserved per-connection but is not guaranteed across reconnections or multi-path routing. ## Signals vs Stop (behavioral differences) - Signal (Kill) - Highest priority and immediate. - Terminates all work including currently executing async handlers; cancellation semantics are immediate and not graceful. - Used for forced termination or emergency shutdowns. - Stop - Graceful termination: currently executing async work is allowed to finish. - On the next message processing iteration the Stop is processed and actor will transition to exiting. - Useful for normal shutdowns where cleanup should run. Important: Stop does not interrupt running handlers. If you require immediate stop, send Kill. ## Supervision & failure handling - Supervision events notify supervising actors of child lifecycle events (started, exited, spawned with error, panicked). - Panic handling: - Panics inside `pre_start` are captured and returned to the spawner. - Panics during message processing are caught and surfaced in supervision events (unless the binary is built with `panic = "abort"`, which will terminate the process). - Supervisors decide restart/stop policies by handling supervision events and using provided restart utilities. ## RPC semantics - RPC calls use `RpcReplyPort` to correlate requests and responses. - RPCs are matched by a reply port id; the origin node holds the awaiting port and the remote node serializes the reply payload. - On network partitions or disconnected nodes, pending RPCs may time out or fail — the user should add timeouts and retry logic for robust behavior. ## Timers & scheduling - Timers are implemented as scheduled events that enqueue messages to the actor mailbox. - Timer delivery follows the same priority rules (timers are user messages unless otherwise specified). - Timers are best-effort accurate — network latencies, scheduling, and runtime load affect delivery time. ## Blocking and long-running work - Actor handlers are asynchronous; long synchronous blocking operations will block the actor and delay its mailbox processing. - Best practice: perform blocking operations in offloaded tasks (spawn on runtime thread pool) and send results back to the actor as messages. ## Concurrency model & runtime integration - Actors are runtime-agnostic (works with tokio and async-std). Internals use async primitives and some tokio sync types even when using async-std (see feature flags). - Actors are cooperative: asynchronous handlers must yield to allow other tasks and timers to run. - Actors may be scheduled across threads by the async runtime, but handler execution remains single-threaded per actor. ## Mailbox semantics - Mailboxes are bounded/unbounded depending on factory/configuration. Check the factory settings used to spawn workers/actors. - Backpressure: when using bounded queues, attempts to send when full may fail or be delayed depending on the chosen factory/queue policy (discard, backpressure, etc.). ## ractor_cluster (remote) specifics - Remote actors are represented by `RemoteActor` wrappers that contain only serialized payloads. Deserialization occurs on the owning node. - `NodeSession` manages transport, authentication, and pending RPC mapping. - Network behavior: - Ordering: per-connection ordering is generally preserved, but there is no distributed global ordering guarantee. - Reconnects: reconnection semantics are best-effort and may result in message loss unless application-level retries/acknowledgement are implemented. - Authentication & encryption: the cluster supports auth and TLS; these features are considered experimental and require production hardening. - Serialization: cluster messages must implement `BytesConvertable` (or be produced via procedural macros for prost types). Versioning of serialized formats should be considered for rolling upgrades. ## Observability & diagnostics - Insert tracing and metrics at: - Mailbox enqueue/dequeue - NodeSession read/write loops - Supervision events and restarts - For debugging actor lifecycles, enable debug logs and use supervision events to trace restarts and failures. ## Safety gotchas & recommendations - Do not rely on synchronous blocking inside handlers — offload blocking work. - Do not assume RPC or remote message delivery is reliable; always add timeouts and retries. - Test supervision logic under stress (panics, rapid spawn/kill cycles). - When using cluster features, add tests for auth, reconnection, and partition behavior. ## Summary of best practices - Keep actors focused and small; offload heavy work. - Use Stop for graceful shutdowns and Kill for forced termination. - Make RPCs resilient: set timeouts and retries. - For production clusters: implement message-level acknowledgements or idempotency when required, and add monitoring/metrics on NodeSession and pending RPC queues. --- --- METRICS --- - Files Extracted: 7 - Estimated Token Budget: ~13991 tokens - Canonical Reference: https://codewiki.google/github.com/slawlor/ractor