## 1. Project Overview & Quickstart (dotnet/orleans) ## File: samples/README.md # Orleans Samples This directory is the canonical home of the official Orleans samples. Samples imported from other Microsoft repositories retain their original source repository in the index and in `gallery.json`. Their source licenses are preserved alongside the imported content. ## Build and validate From the repository root, run: ```powershell pwsh ./samples/Validate-Samples.ps1 ``` The command checks the gallery manifest and builds every project in `Samples.slnx`. External cloud services are required only when running samples which use them, not when compiling. ## Featured samples | Sample | Description | Original source | | --- | --- | --- | | [Adventure](Adventure) | A text adventure game demonstrating grains, external clients, and application modeling. | [dotnet/samples](https://github.com/dotnet/samples) | | [Chirper](Chirper) | A social network sample using persistence, observers, and reentrant grains. | [dotnet/samples](https://github.com/dotnet/samples) | | [GPS Tracker](GPSTracker) | An IoT location tracker integrating Orleans with ASP.NET Core SignalR. | [dotnet/samples](https://github.com/dotnet/samples) | | [Hello World](HelloWorld) | The smallest complete Orleans application for defining and calling a grain. | [dotnet/samples](https://github.com/dotnet/samples) | | [Journaled Todo List](JournaledTodoList) | An Aspire-hosted Web application demonstrating durable journaled grain state. | [dotnet/samples](https://github.com/dotnet/samples) | | [Presence Service](Presence) | A gaming presence service using observers and multiple cooperating grains. | [dotnet/samples](https://github.com/dotnet/samples) | | [Shopping Cart](ShoppingCart) | A Blazor shopping cart using Orleans persistence and Azure Storage. | [dotnet/samples](https://github.com/dotnet/samples) | | [Azure App Service Deployment](Deployment/AzureAppService) | A Windows and Linux Azure App Service deployment with managed identity, private silo networking, health checks, Easy Auth, Bicep, and OIDC. | [dotnet/orleans](https://github.com/dotnet/orleans) | | [Azure Container Apps Deployment](Deployment/AzureContainerApps) | A managed-identity Orleans cluster with clients, dashboard, scaler, and Bicep deployment for Azure Container Apps. | [Azure-Samples/Orleans-Cluster-on-Azure-Container-Apps](https://github.com/Azure-Samples/Orleans-Cluster-on-Azure-Container-Apps) | ## All samples | Sample | Description | Languages | Tags | Original source | | --- | --- | --- | --- | --- | | [Adventure](Adventure) | A text adventure game demonstrating grains, external clients, and application modeling. | C# | games, clients, grains | [dotnet/samples](https://github.com/dotnet/samples) | | [AWS Kinesis and DynamoDB](AWS/KinesisDynamoDB) | An AWS-hosted Orleans application using DynamoDB for clustering, persistence, reminders, and Kinesis checkpoints. | C# | aws, kinesis, dynamodb, streaming | [dotnet/orleans](https://github.com/dotnet/orleans) | | [Bank Account](BankAccount) | A bank transfer simulation demonstrating ACID transactions across stateful grains. | C# | transactions, persistence | [dotnet/samples](https://github.com/dotnet/samples) | | [Basic Clustering](BasicClustering) | A minimal Aspire-hosted Orleans cluster with two silo replicas and Redis membership. | C# | clustering, aspire, redis, getting-started | [dotnet/orleans](https://github.com/dotnet/orleans) | | [Blazor Server](Blazor/BlazorServer) | An interactive Blazor Server application backed by Orleans grains. | C#, Razor | blazor, aspnet-core, web | [dotnet/samples](https://github.com/dotnet/samples) | | [Blazor WebAssembly](Blazor/BlazorWasm) | A hosted Blazor WebAssembly application with an Orleans-backed server. | C#, Razor | blazor, webassembly, web | [dotnet/samples](https://github.com/dotnet/samples) | | [Chat Room](ChatRoom) | A terminal chat application demonstrating Orleans Streams. | C# | streaming, client, terminal | [dotnet/samples](https://github.com/dotnet/samples) | | [Chirper](Chirper) | A social network sample using persistence, observers, and reentrant grains. | C# | observers, persistence, reentrancy | [dotnet/samples](https://github.com/dotnet/samples) | | [Custom grain-call return type](CustomGrainCallReturnType) | An awaitable GrainCall extension demonstrating generated request bases, proxy adaptation, and failure propagation. | C# | serialization, code-generation, grains | [dotnet/orleans](https://github.com/dotnet/orleans) | | [F# Hello World](FSharpHelloWorld) | A minimal Orleans application with grains implemented in F#. | F#, C# | getting-started, fsharp | [dotnet/samples](https://github.com/dotnet/samples) | | [GPS Tracker](GPSTracker) | An IoT location tracker integrating Orleans with ASP.NET Core SignalR. | C#, JavaScript | iot, signalr, web | [dotnet/samples](https://github.com/dotnet/samples) | | [Google Cloud Firestore](GoogleFirestore) | An Orleans application using Firestore for clustering, grain directories, persistence, and reminders. | C# | google-cloud, firestore, clustering, persistence, reminders | [dotnet/orleans](https://github.com/dotnet/orleans) | | [Hello World](HelloWorld) | The smallest complete Orleans application for defining and calling a grain. | C# | getting-started, grains | [dotnet/samples](https://github.com/dotnet/samples) | | [Journaled Todo List](JournaledTodoList) | An Aspire-hosted Web application demonstrating durable journaled grain state. | C#, Razor | journaling, aspire, web | [dotnet/samples](https://github.com/dotnet/samples) | | [Durable Jobs provider migration](DurableJobsMigration) | Drain existing jobs from Azure Blob provider A while provider B accepts new work, with process restart and full-inventory checks. | C# | durable-jobs, journaling, azure-storage, aspire | [dotnet/orleans](https://github.com/dotnet/orleans) | | [Journaling with Azure Blob JSON](JournalingAzureBlobJson) | An Aspire sample using Orleans journaling with JSON events stored in Azure Blob Storage. | C# | journaling, azure-storage, aspire | [dotnet/orleans](https://github.com/dotnet/orleans) | | [Presence Service](Presence) | A gaming presence service using observers and multiple cooperating grains. | C# | gaming, observers, services | [dotnet/samples](https://github.com/dotnet/samples) | | [Shopping Cart](ShoppingCart) | A Blazor shopping cart using Orleans persistence and Azure Storage. | C#, Razor | blazor, persistence, azure-storage | [dotnet/samples](https://github.com/dotnet/samples) | | [Stocks](Stocks) | A stock price application using grain timers, HTTP calls, and temporary caching. | C#, Razor | timers, http, caching | [dotnet/samples](https://github.com/dotnet/samples) | | [Streaming Custom Data Adapter](Streaming/CustomDataAdapter) | An Event Hubs streaming sample consuming data from a non-Orleans publisher. | C# | streaming, event-hubs, azure | [dotnet/samples](https://github.com/dotnet/samples) | | [Simple Streaming](Streaming/Simple) | A compact producer and consumer example using Orleans Streams. | C# | streaming, getting-started | [dotnet/samples](https://github.com/dotnet/samples) | | [Tic Tac Toe](TicTacToe) | A Web-based game demonstrating lobbies and ASP.NET Core integration. | C#, JavaScript | games, aspnet-core, web | [dotnet/samples](https://github.com/dotnet/samples) | | [Transport Layer Security](TransportLayerSecurity) | A Hello World cluster configured with mutual TLS for all network communication. | C# | security, tls, networking | [dotnet/samples](https://github.com/dotnet/samples) | | [Visual Basic Hello World](VBHelloWorld) | A minimal Orleans application with grain contracts and implementations in Visual Basic. | Visual Basic, C# | getting-started, visual-basic | [dotnet/samples](https://github.com/dotnet/samples) | | [Voting](Voting) | A Kubernetes-oriented voting Web application with the Orleans Dashboard. | C#, Razor | kubernetes, dashboard, web | [dotnet/samples](https://github.com/dotnet/samples) | | [Azure App Service Deployment](Deployment/AzureAppService) | A Windows and Linux Azure App Service deployment with managed identity, private silo networking, health checks, Easy Auth, Bicep, and OIDC. | C#, Razor, Bicep, YAML | deployment, azure, app-service, windows, linux | [dotnet/orleans](https://github.com/dotnet/orleans) | | [Azure Container Apps Deployment](Deployment/AzureContainerApps) | A managed-identity Orleans cluster with clients, dashboard, scaler, and Bicep deployment for Azure Container Apps. | C#, Bicep, YAML | deployment, azure, container-apps, managed-identity, networking | [Azure-Samples/Orleans-Cluster-on-Azure-Container-Apps](https://github.com/Azure-Samples/Orleans-Cluster-on-Azure-Container-Apps) | --- ## File: docs/index.md # dotnet/orleans > Routing index for the knowledge base. Copilot reads this file to decide > which docs to open for a given task. Keep entries short — one line per doc. | When working on | Topics | Read | |---|---|---| | Any task in this repo | general | (no docs yet — the sync workflow will populate this table after merge) | ## 2. Official Technical Reference & Guides (dotnet/orleans-docs) ## File: src/docs/deployment/multi-cluster_support/GlobalSingleInstance.md --- title: Global-Single-Instance Grains --- ### Grain Coordination Attributes Developers can indicate when and how clusters should coordinate their grain directories with respect to a particular grain class. The `[GlobalSingleInstance]` attribute means we want the same behavior as when running Orleans in a single global cluster: that is, route all calls to a single activation of the grain. Conversely, the `[OneInstancePerCluster]` attribute indicates that each cluster can have its own independent activation. This is appropriate if communication between clusters is undesired. The attributes are placed on grain implementations. For example: ```csharp using Orleans.MultiCluster; [GlobalSingleInstance] public class MyGlobalGrain : Orleans.Grain, IMyGrain { ... } [OneInstancePerCluster] public class MyLocalGrain : Orleans.Grain, IMyGrain { ... } ``` If a grain class does not specify either one of those attributes, it defaults to `[OneInstancePerCluster]`, or `[GlobalSingleInstance]` if the configuration parameter `UseGlobalSingleInstanceByDefault` is set to true. #### Protocol for Global-Single-Instance Grains When a global-single-instance (GSI) grain is accessed, and no activation is known to exist, a special GSI activation protocol is executed before activating a new instance. Specifically, a request is sent to all other clusters in the current [multi-cluster configuration](MultiClusterConfiguration.md) to check if they already have an activation for this grain. If all responses are negative, a new activation is created in this cluster. Otherwise, the remote activation is used (and a reference to it is cached in the local directory). #### Protocol for One-Instance-Per-Cluster Grains There is no inter-cluster communication for One-Instance-Per-Cluster grains. They simply use the standard Orleans mechanism independently within each cluster. Inside the Orleans framework itself, the following grain classes are marked with the `[OneInstancePerCluster]` attribute: `ManagementGrain`, `GrainBasedMembershipTable`, and `GrainBasedReminderTable`. #### Doubtful Activations If the GSI protocol does not receive conclusive responses from all clusters after 3 retries (or whatever number is specified by the configuration parameter `GlobalSingleInstanceNumberRetries`), it creates a new local "doubtful" activation optimistically, favoring availability over consistency. Doubtful activations may be duplicates (because some remote cluster that did not respond during the GSI protocol activation may nevertheless have an activation of this grain). Therefore, periodically every 30 seconds (or whatever interval is specified by the configuration parameter `GlobalSingleInstanceRetryInterval`) the GSI protocol is run again for all doubtful activations. This ensures that once communication between clusters is restored, duplicate activations can be detected and removed. --- ## File: src/docs/deployment/multi-cluster_support/GossipChannels.md --- title: Multi-Cluster Communication --- ## Multi-Cluster Communication The network must be configured in such a way that any Orleans silo can connect to any other Orleans silo via TCP/IP, regardless of where in the world it is located. Exactly how this is achieved is outside of the scope of Orleans, as it depends on how and where silos are deployed. For example, on Windows Azure, we can use VNETs to connect muliple deployments within a region, and gateways to connect VNETs across different regions. ### Cluster Id Each cluster has its own unique cluster id. The cluster id must be specified in the global configuration. Cluster ids may not be empty, nor may they contain commas. Also, if using Azure Table Storage, cluster ids may not contain the characters forbidden for row keys (/, \, #, ?). We recommend using very short strings for the cluster ids, because cluster ids are transmitted frequently and may be stored in storage by some log-view providers. ### Cluster Gateways Each cluster automatically designates a subset of its active silos to serve as *cluster gateways*. Cluster gateways directly advertise their IP addresses to other clusters, and can thus serve as "points of first contact". By default, at most 10 silos (or whatever number is configured as `MaxMultiClusterGateways`) are designated as cluster gateways. Communication between silos in different clusters does *not* always pass through a gateway. Once a silo has learned and cached the location of a grain activation (no matter in what cluster), it sends messages to that silo directly, even if the silo is not a cluster gateway. ### Gossip Gossip is a mechanism for clusters to share configuration and status information. As the name suggests, gossip is decentralized and bidirectional: each silo communicates directly with other silos, both in the same cluster and in other clusters, to exchange information in both directions. **Content**. Gossip contains some or all of the following information: - The current time-stamped [multi-cluster configuration](MultiClusterConfiguration.md). - A dictionary that contains information about cluster gateways. The key is the silo address, and the value contains (1) a timestamp, (2) the cluster id, and (3) a status, which is either active or inactive. **Fast & Slow Propagation**. When a gateway changes its status, or when an operator injects a new configuration, this gossip information is immediately sent to all silos, clusters, and gossip channels. This happens fast, but is not reliable. Should the message be lost due to any reasons (e.g. races, broken sockets, silo failures), our periodic background gossip ensures that the information eventually spreads, albeit more slowly. All information is eventually propagated everywhere, and is highly resilient to occasional message loss and failures. All gossip data is timestamped, which ensures that newer information replaces older information regardless of the relative timing of messages. For example, newer multi-cluster configurations replace older ones, and newer information about a gateway replaces older information about that gateway. For more details on the representation of gossip data, see the `MultiClusterData` class. It has a `Merge` method that combines gossip data, resolving conflicts using timestamps. ### Gossip Channels When a silo is first started, or when it is restarted after a failure, it needs to have a way to **bootstrap the gossip**. This is the role of the *gossip channel*, which can be configured in the [Silo Configuration](SiloConfiguration.md). On startup, a silo fetches all the information from the gossip channels. After startup, a silo keeps gossiping periodically, every 30 seconds or whatever is configured as `BackgroundGossipInterval`. Each time it synchronizes its gossip information with a partner randomly selected from all cluster gateways and gossip channels. Notes: - Though not strictly required, we recommend to always configure at least two gossip channels, in distinct regions, for better availability. - Latency of communication with gossip channels is not critical. - Multiple different services can use the same gossip channel without interference, as long as the ServiceId Guid (as specified by their respective configuration) is distinct. - There is no strict requirement that all silos use the same gossip channels, as long as the channels are sufficient to let a silo initially connect with the "gossiping community" when it starts up. But if a gossip channel is not part of a silo's configuration, and that silo is a gateway, it does not push its status updates to the channel (fast propagation), so it may take longer before those reach the channel via periodic background gossip (slow propagation). #### Azure-Table-Based Gossip Channel We have already implemented a gossip channel based on Azure Tables. The configuration specifies standard connection strings used for Azure accounts. For example, a configuration could specify two gossip channels with separate Azure storage accounts `usa` and `europe` as follows: ```csharp var silo = new SiloHostBuilder() [...] .Configure(options => { [...] options.GossipChannels.Add("AzureTable", "DefaultEndpointsProtocol=https;AccountName=usa;AccountKey=..."); options.GossipChannels.Add("AzureTable", "DefaultEndpointsProtocol=https;AccountName=europe;AccountKey=...") [...] }) [...] ``` Multiple different services can use the same gossip channel without interference, as long as the ServiceId guid specified by their respective configuration is distinct. #### Other Gossip Channel Implementations We are working on other gossip channel providers, similar to how membership and reminders are packaged for many different storage back-ends. --- ## File: src/docs/deployment/multi-cluster_support/MultiClusterConfiguration.md --- title: Multi-Cluster Configuration --- ## Multi-Cluster Configuration The multi-cluster configuration determines which clusters are currently part of the multi-cluster. It does not change automatically, but is controlled by the operator. Thus, it is quite different from the membership mechanism used within a cluster, which automatically determines the set of silos that are part of the cluster. We use the following terminology for the clusters in a service: - A cluster is *active* if it has at least one active silo, and *inactive* otherwise - A cluster is *joined* if it is part of the current multi-cluster configuration, and *non-joined* otherwise Being active/inactive is independent from being joined/non-joined: all four combinations are possible. All the clusters for a particular service are connected by a [gossip network](GossipChannels.md). The gossip network propagates configuration and status information. ### Injecting a configuration An operator issues configuration changes by injecting them into the multi-cluster network. The configurations can be injected into any cluster, and spread from there to all active clusters. Each new configuration consists of a list of cluster ids that form the multi-cluster. It also has a UTC timestamp that is used to track its propagation through the gossip network. Initially, the multi-cluster configuration is null, which means the multi-cluster list is empty (contains no clusters). Thus, the operator *must* initially inject a multi-cluster configuration. Once injected, this configuration persists in all connected silos (while running) and in all specified gossip channels (if those channels are persistent). We pose some restrictions on the injection of new configurations that an operator must follow: - Each new configuration may add a number of clusters, or remove a number of clusters (but not both at the same time). - An operator should not issue a new configuration while a previous configuration change is still being processed. These restrictions ensure that protocols such as the single-instance-protocol can correctly maintain mutual exclusion of activations even under configuration changes. #### Via Management Grain Multi-cluster configurations can be injected on any node in any cluster, using the Orleans Management Grain. For example, to inject a multi-cluster configuration that consists of the three clusters { us1, eu1, us2 }, we can pass a string enumerable to the management grain: ```csharp var clusterlist = "us1,eu1,us2".Split(','); var mgtGrain = client.GetGrain(0); mgtGrain.InjectMultiClusterConfiguration(clusterlist, "my comment here")); ``` The first argument to `InjectMultiClusterConfiguration` is an enumerable of cluster ids, which is going to define the new multi-cluster configuration. The second argument is an (optional) comment string that can be used to tag configurations with arbitrary information, such as who injected them and why. There is an optional third argument, a boolean called `checkForLaggingSilosFirst`, which defaults to true. It means that the system performs a best-effort check to see if there are any silos anywhere that have not caught up to the current configuration yet, and rejects the change if it finds such a silo. This helps to detect violations of the restriction that only one configuration change should be pending at a time (though it cannot guarantee it under all circumstances). #### Via Default Configuration In situations where the multi-cluster configuration is known in advance and the deployment is fresh every time (e.g. for testing), we may want to supply a default configuration. The global configuration supports an optional attribute `DefaultMultiCluster` which takes a comma-separated list of cluster ids: ```csharp var silo = new SiloHostBuilder() [...] .Configure(options => { [...] options.DefaultMultiCluster = new[] { "us1", "eu1", "us2" }; [...] }) [...] ``` After a silo is started with this setting, it checks to see if the current multi-cluster configuration is null, and if so, injects the given configuration with the current UTC timestamp. WARNING. Persistent multi-cluster gossip channels (e.g. based on AzureTable) retain the last injected configuration unless they are deleted explicitly. In that case, specifying a DefaultMulticluster has no effect when re-deploying a cluster because the configuration stored in the gossip channels is not null.> #### Via Gossip Channel An operator can also inject the configuration directly into the gossip channel. Changes in the channel are picked up and propagated automatically by the periodic background gossip, though possibly very slowly (using the management grain is much faster). A rough estimate on the propagation time is 30 seconds (or whatever gossip interval is specified in the global configuration) times the binary logarithm of the total number of silos in all clusters. But since the gossip pairs are selected randomly, it can be both much quicker or much slower. If using the Azure Table-Based Gossip Channel, operators can inject a new configuration simply by editing the configuration record in the `OrleansGossipTable`, e.g. using some tool for editing data in Azure tables. The configuration record has the following format: |Name | Type | Value | |------------- |-------- |----------| |PartitionKey | String | the ServiceId | |RowKey | String | "CONFIG" | |Clusters | String | comma-separated list of cluster IDs, e.g. "us1,eu1,us2" | |Comment | String | optional comment | |GossipTimestamp | DateTime | UTC timestamp for the configuration | **NOTE**. When editing this record in storage, the GossipTimestamp must also be set to a newer value than it has currently (otherwise the change is ignored). The most convenient and recommended way to do this is to *delete the GossipTimestamp field* - our gossip channel implementation then automatically replaces it with a correct, current Timestamp (it uses the Azure Table Timestamp). ### Cluster Addition/Removal Procedures Adding or removing a cluster from the multi-cluster often needs to be coordinated within some larger context. We recommend to always follow the procedures described below when adding/removing clusters from the multi-cluster. #### Procedure for adding a cluster 1. Start a new Orleans cluster and wait till all silos are up and running. 2. Inject a configuration that contains the new cluster. 3. Start routing user requests to the new cluster. #### Procedure for removing a cluster 1. Stop routing new user requests to the cluster. 2. Inject a configuration that no longer contains the cluster. 3. Stop all silos of the cluster. Once a cluster has been removed in this way, it can be re-added by following the procedure for adding a new cluster. ### Activity on Non-Joined Clusters There can be brief, temporary periods of time where a cluster is both active and non-joined: - A freshly started cluster may start executing code before it is in the multicluster configuration (between steps 1 and 2 of the procedure for adding a cluster) - A cluster that is being decommissioned may still execute code before the silos are shut down (between steps 2 and 3 of the procedure for removing a cluster). During those intermediate situations, the following are possible: - For global-single-instance grains: A grain may have a duplicate activation on a non-joined cluster. - For versioned grains: activations on non-joined clusters do not receive notifications when the grain state changes. --- ## File: src/docs/deployment/multi-cluster_support/Overview.md --- title: Multi-Cluster Support --- # Multi-Cluster Support Orleans v.1.3.0 added support for federating several Orleans clusters into a loosely connected *multi-cluster* that acts like a single service. Multi-clusters facilitate *geo-distribution* of a service, that is, make it easier to run an Orleans application in multiple data-centers around the world. Also, a multi-cluster can be run within a single datacenter to get better failure and performance isolation. All mechanisms are designed with particular attention to (1) minimize communication between clusters, and (2) let each cluster run autonomously even if other clusters fail or become unreachable. ## Configuration and Operation Below we document how to configure and operate a multi-cluster. [**Communication**](GossipChannels.md). Clusters communicate via the same silo-to-silo connections that are used within a cluster. To exchange status and configuration information, Clusters use a gossip mechanism and gossip channel implementations. [**Silo Configuration**](SiloConfiguration.md). Silos need to be configured so they know which cluster they belong to (each cluster is identified by a unique string). Also, each silo needs to be configured with connection strings that allow them to connect to one or more [gossip channels](GossipChannels.md) on startup. [**Multi-Cluster Configuration Injection**](MultiClusterConfiguration.md). At runtime, the service operator can specify and/or change the *multi-cluster configuration*, which contains a list of cluster ids, to specify which clusters are part of the current multi-cluster. This is done by calling the management grain in any one of the clusters. ## Multi-Cluster Grains Below we document how to use multi-cluster functionality at the application level. [**Global-Single-Instance Grains**](GlobalSingleInstance.md). Developers can indicate when and how clusters should coordinate their grain directories with respect to a particular grain class. The ``[GlobalSingleInstance]`` attribute means we want the same behavior as as when running Orleans in a single global cluster: that is, route all calls to a single activation of the grain. Conversely, the ``[OneInstancePerCluster]`` attribute indicates that each cluster can have its own independent activation. This is appropriate if communication between clusters is undesired. **Log-View Grains** _(not in v.1.3.0)_. A special type of grain that uses a new API, similar to event sourcing, for synchronizing or persisting grain state. It can be used to automatically and efficiently synchronize the state of a grain between clusters and with storage. Because its synchronization algorithms are safe to use with reentrant grains, and are optimized to use batching and replication, it can perform better than standard grains when a grain is frequently accessed in multiple clusters, and/or when it is written to storage frequently. Support for log-view grains is not part of the master branch yet. We have a prerelease including samples and a bit of documentation in the [geo-orleans branch](https://github.com/sebastianburckhardt/orleans/tree/geo-samples). It is currently being evaluated in production by an early adopter. --- ## File: src/docs/deployment/multi-cluster_support/SiloConfiguration.md --- title: Multi-Cluster Silo Configuration --- ## Orleans Silo Configuration To get a quick overview, we show all relevant configuration parameters (including optional ones) in XML syntax below: ```html ``` ```csharp var silo = new SiloHostBuilder() [...] .Configure(options => { options.ClusterId = "us3"; options.ServiceId = "myawesomeservice"; }) .Configure(options => { options.HasMultiClusterNetwork = true; options.DefaultMultiCluster = new[] { "us1", "eu1", "us2" }; options.BackgroundGossipInterval = TimeSpan.FromSeconds(30); options.UseGlobalSingleInstanceByDefault = false; options.GlobalSingleInstanceRetryInterval = TimeSpan.FromSeconds(30); options.GlobalSingleInstanceNumberRetries = 3; options.MaxMultiClusterGateways = 10; options.GossipChannels.Add("AzureTable", "DefaultEndpointsProtocol=https;AccountName=usa;AccountKey=..."); options.GossipChannels.Add("AzureTable", "DefaultEndpointsProtocol=https;AccountName=europe;AccountKey=...") [...] }) [...] ``` As usual, all configuration settings can also be read and written programmatically, via the respective members of the `GlobalConfiguration` class. The `Service Id` is an arbitrary ID for identifying this service. It must be the same for all clusters and all silos. The `MultiClusterNetwork` section is optional - if not present, all multi-cluster support is disabled for this silo. The **required parameters** `ClusterId` and `GossipChannel` are explained in the section on [Multi-Cluster Communication](GossipChannels.md). The optional parameters `MaxMultiClusterGateways` and `BackgroundGossipInterval` are explained in the section on [Multi-Cluster Communication](GossipChannels.md). The optional parameter `DefaultMultiCluster` is explained in the section on [Multi-Cluster Configuration](MultiClusterConfiguration.md). The optional parameters `UseGlobalSingleInstanceByDefault`, `GlobalSingleInstanceRetryInterval` and `GlobalSingleInstanceNumberRetries` are explained in the section on [Global-Single-Instance Grains](GlobalSingleInstance.md). ## Orleans Client Configuration No extra configuration is required for Orleans client. The same client may not connect to silos in different clusters (the silo refuses the connection in that situation). --- ## File: src/docs/deployment/azure_web_apps_with_azure_cloud_services.md --- title: Using Azure Web Apps with Azure Cloud Services --- # Using Azure Web Apps with Azure Cloud Services If you would like to connect to an Azure Cloud Services Silo from an [Azure Web App](http://azure.microsoft.com/en-gb/services/app-service/web/) rather than a Web Role hosted within the same cloud service you can. For this to work securely you will need to assign both the Azure Web App and the Worker Role hosting the Silo to an [Azure Virtual Network](http://azure.microsoft.com/en-gb/services/virtual-network/). First we'll setup the Azure Web App, you can follow [this guide](https://azure.microsoft.com/en-us/blog/azure-websites-virtual-network-integration/) which will create the virtual network and assign it to the Azure Web App. Now we can assign the cloud service to the virtual network by modifying the `ServiceConfiguration` file. ``` xml ``` Also make sure the Silo endpoints are configured. ``` xml ``` You can now connect from the Web App to the rest of the cluster. ### Potential Issues If the Web App is having difficulty connecting to the Silo: * Make sure you have at least **two roles**, or two instances of one role in your Azure Cloud Service, or the `InternalEndpoint` firewall rules may not be generated. * Check that both the Web App and the Silo are using the same `ClusterId` and `ServiceId`. * Make sure the network security group is set up to allow internal virtual network connections. If you haven't got one you can create and assign one easily using the following `PowerShell`: ``` c New-AzureNetworkSecurityGroup -Name "Default" -Location "North Europe" Get-AzureNetworkSecurityGroup -Name "Default" | Set-AzureNetworkSecurityGroupToSubnet -VirtualNetworkName "virtual-network-name" -SubnetName "subnet-name" ``` --- ## File: src/docs/deployment/consul_deployment.md --- title: Using Consul as a Membership Provider --- # Using Consul as a Membership Provider ## Introduction to Consul [Consul](https://www.consul.io) is a distributed, highly available and datacenter-aware service discovery platform which includes simple service registration, health checking, failure detection and key/value storage. It is built on the premise that every node in the datacenter is running a Consul agent which is either acting as a server or client which communicate via a scalable gossip protocol. There is a very detailed overview of Consul including comparisons with similar solutions [here](https://www.consul.io/intro/index.html). Consul is written in GO and is [open source](https://github.com/hashicorp/consul); compiled downloads are available for [Mac OS X, FreeBSD, Linux, Solaris and Windows](https://www.consul.io/downloads.html) ## Why Choose Consul? As an [Orleans Membership Provider](~/docs/implementation/cluster_management.md), Consul is a good choice when you need to deliver an **on-premise solution** which does not require your potential customers to have existing infrastructure **and** a co-operative IT provider. Consul is a very lightweight single executable, has no dependencies and as such can easily be built into your own middleware solution. And when Consul is already your solution for discovering, checking and maintaining your microservices, it makes sense to fully integrate with Orleans membership for simplicity and ease of operation. We therefore implemented a membership table in Consul (also known as "Orleans Custom System Store"), which fully integrates with Orleans's [Cluster Management](~/docs/implementation/cluster_management.md). ## Setting up Consul There is very extensive documentation available on [Consul.io](https://www.consul.io) about setting up a stable Consul cluster and it doesn't make sense to repeat that here; however for your convenience we include this guide so you can very quickly get Orleans running with a standalone Consul agent. 1) Create a folder to install Consul into, e.g. C:\Consul 2) Create a subfolder: C:\Consul\Data (Consul will not create this if it doesn't exist) 3) [Download](https://www.consul.io/downloads.html) and unzip Consul.exe into C:\Consul\ 4) Open a command prompt at C:\Consul\ 5) Enter `Consul.exe agent -server -bootstrap -data-dir "C:\Consul\Data" -client=0.0.0.0` `agent` Instructs Consul to run the agent process that hosts the services. Without this the Consul process will attempt to use RPC to configure a running agent. `-server` Defines the agent as a server and not a client (A Consul *client* is an agent that hosts all the services and data, but does not have voting rights to decide, and cannot become, the cluster leader `-bootstrap` The first (and only the first!) node in a cluster must be bootstrapped so that it assumes the cluster leadership. `-data-dir [path]` Specifies the path where all Consul data is stored, including the cluster membership table `-client=0.0.0.0` Informs Consul which IP to open the service on. There are many other parameters, and the option to use a json configuration file. Please consult the Consul documentation for a full listing of the options. 6) Verify that Consul is running and ready to accept membership requests from Orleans by opening the [services](http://localhost:8500/v1/catalog/services) endpoint in your browser. ## Configuration of Orleans ### Server There is currently a known issue with the "Custom" membership provider OrleansConfiguration.xml configuration file that will fail to parse correctly. For this reason you have to provide a placeholder SystemStore in the xml and then configure the provider in code before starting the Silo. **OrleansConfiguration.xml** ```xml ``` **Code** ```csharp public void Start(ClusterConfiguration config) { _siloHost = new SiloHost(System.Net.Dns.GetHostName(), config); _siloHost.Config.Globals.LivenessType = GlobalConfiguration.LivenessProviderType.Custom; _siloHost.Config.Globals.MembershipTableAssembly = "OrleansConsulUtils"; _siloHost.Config.Globals.ReminderServiceType = GlobalConfiguration.ReminderServiceProviderType.Disabled; _siloHost.InitializeOrleansSilo(); var startedok = _siloHost.StartOrleansSilo(); if (!startedok) throw new SystemException(String.Format("Failed to start Orleans silo '{0}' as a {1} node", _siloHost.Name, _siloHost.Type)); Log.Information("Orleans Silo is running.\n"); } ``` Alternatively you could configure the silo entirely in code. ### Client The client configuration is much simpler **ClientConfiguration.xml** ```xml ``` ## Client SDK If you are interested in using Consul for your own service discovery there are [Client SDKs](https://www.consul.io/downloads_tools.html) for most popular languages. ## Implementation Detail The Membership Table Provider makes use of [Consul's Key/Value store](https://www.consul.io/intro/getting-started/kv.html) functionality with CAS. When each Silo starts it registers two KV entries, one which contains the Silo details and one which holds the last time the Silo reported it was alive (the latter refers to diagnostics "I am alive" entries and not to failure detection hearbeats which are sent directly between the silos and are not written into the table). All writes to the table are performed with CAS to provide concurrency control, as necessitated by Orleans's [Cluster Management Protocol](~/docs/implementation/cluster_management.md). Once the Silo is running you can view these entries in your web browser [here](http://localhost:8500/v1/kv/?keys), this will display something like: ```js [ "orleans/MyOrleansDeployment/192.168.1.26:11111@191780753", "orleans/MyOrleansDeployment/192.168.1.26:11111@191780753/iamalive" ] ``` You will notice that the keys are prefixed with *"orleans/"* this is hard coded in the provider and is intended to avoid key space collision with other users of Consul. Each of these keys can be read by appending their key name *(sans quotes of course)* to the [Consul KV root](http://localhost:8500/v1/kv/). Doing so will present you with the following: ```js [ { "LockIndex": 0, "Key": "orleans/MyOrleansDeployment/192.168.1.26:22222@191780753", "Flags": 0, "Value": "[BASE64 UTF8 Encoded String]", "CreateIndex": 10, "ModifyIndex": 12 } ] ``` Decoding the string will give you the actual Orleans Membership data: **http://localhost:8500/v1/KV/orleans/MyOrleansDeployment/[SiloAddress]** ``` { "Hostname": "[YOUR_MACHINE_NAME]", "ProxyPort": 22222, "StartTime": "2016-01-29T16:25:54.9538838Z", "Status": 3, "SuspectingSilos": [] } ``` **http://localhost:8500/v1/KV/orleans/MyOrleansDeployment/[SiloAddress]/IAmAlive** "2016-01-29T16:35:58.9193803Z" When the Clients connect, they read the KVs for all silos in the cluster in one HTTP GET by using the uri `http://192.168.1.26:8500/v1/KV/orleans/MyOrleansDeployment/?recurse`. ## Limitations ### Orleans Extended Membership Protocol (Table Version & ETag) Consul KV currrently does not currently support atomic updates. Therefore, the Orleans Consul Membership Provider only implements the the Orleans Basic Membership Protocol, as described [here](~/docs/implementation/cluster_management.md) and does not support the Extended Membership Protocol. This Extended protocol was introduced as an additional, but not essential, silo connectivity validation and as a foundation to functionality that has not yet been implemented. Providing your infrastructure is correctly configured you will not experience any detrimental effect of the lack of support. ### Multiple Datacenters The Key Value Pairs in Consul are not currently replicated between Consul datacenters. There is a [separate project](https://github.com/hashicorp/consul-replicate) to address this but it has not yet been proven to support Orleans. ### When running on Windows When Consul starts on Windows it logs the following message: ==> WARNING: Windows is not recommended as a Consul server. Do not use in production. This is displayed simply due to lack of focus on testing when running in a Windows environment and not because of any actual known issues. Read the [discussion here](https://groups.google.com/forum/#!topic/consul-tool/DvXYgZtUZyU) before deciding if Consul is the right choice for you. ## Potential Future Enhanecements 1) Prove that the Consul KV replication project is able to support an Orleans cluster in a WAN environment between multiple Consul datacenters. 2) Implement the Reminder Table in Consul. 3) Implement the Extended Membership Protocol. The team behind Consul does plan on implementing atomic operations, once this functionality is available it will be possible to remove the limitations in the provider. --- ## File: src/docs/deployment/docker_deployment.md --- title: Docker Deployment --- # Docker Deployment > **Note**: Even if you are very familiar with Docker and/or Orleans, as any other Orleans documentation, I recommend you to read it to the end in order to avoid problems you may face that we already worked around. > **Note**: This article and its sample are a work in progress. Any feedback, PR or suggestion is very welcome. ## Deploying Orleans solutions to Docker Deploying Orleans to Docker can be tricky given the way Docker orchestrators and clustering stacks was designed. The most complicated thing is to understand the concept of *Overlay Network* from Docker Swarm and Kubernets Networking model. Docker containers and networking model were designed to run mostly stateless and immutable containers. So, spinning up a cluster running node.js or nginx applications, is pretty easy. However, if you try to use something more elaborate, like a real clustered or distributed application (like Orleans-based ones) you will eventually have trouble setting it up. It is possible, but not as simple as web-based applications. Docker clustering consists of putting together multiple hosts to work as a single pool of resources, managed using a *Container Orchestrator*. *Docker Inc.* provide **Swarm** as their option for Container Orchestration while *Google* has **Kubernetes** (aka **K8s**). There are other Orchestrators like **DC/OS**, **Mesos**, etc., but in this document we will talk about Swarm and K8s as they are more widely used. The same grain interfaces and implementation which run anywhere Orleans is already supported, will run on Docker containers as well, so no special considerations are needed in order to be able to run your application in Docker containers. The [Orleans-Docker](https://github.com/dotnet/orleans/tree/master/Samples/Orleans-Docker) sample provides a working example of how to run two console applications. One as Orleans Client and another as Silo, and the details are described below. The concepts discussed here, can be used on both .Net Core and .Net 4.6.1 flavors of Orleans but to ilustrate the cross-platform nature of Docker and .Net Core, we are going to focus on the example considering you are using .Net Core. Platform-specific (Windows/Linux/OSX) details may be provide along this article. ## Pre-requisites This article assume that you have the following prerequisites installed: * [Docker](https://www.docker.com/community-edition#/download) - Docker4X has a easy-to-use installer for the major supported platforms. It contains Docker engine and also Docker Swarm. * [Kubernetes (K8s)](https://kubernetes.io/docs/tutorials/stateless-application/hello-minikube/) - Google's offer for Container Orchestration. It contains a guidance to install *Minikube* (a local deployment of K8s) and *kubectl* along with all its dependencies. * [.Net Core](https://dot.net) - Cross-platform flavor of .Net * [Visual Studio Code (VSCode)](https://code.visualstudio.com/) - You can use whatever IDE you want. VSCode is cross-platform so we are using it to ensure it works on all platforms. Once you installed VSCode, install the [C# extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode.csharp). > **Note**: You are not required to have Kubernetes installed if you are not going to use it. Docker4X installer already includes Swarm so no extra installation is required to use it. > **Note for Windows Users**: On Windows, Docker installer will enable Hyper-V at installation process. As this article and its examples are using .Net Core, the container images used are based on **Windows Server NanoServer**. If you don't plan to use .Net Core and will target .Net 4.6.1 full framework, the image used should be **Windows Server Core** and the 1.4+ version of Orleans (which supports only .net full framework). ## Creating Orleans Solution The following instructions show how to create a regular Orleans solution using the new `dotnet` tooling. > **Note**: Please adapt the commands to whatever is appropriate in your platform. Also, the directory structure is just a suggestion. Please adapt it to your needs. ```bash mkdir Orleans-Docker cd Orleans-Docker dotnet new sln mkdir -p src/OrleansSilo mkdir -p src/OrleansClient mkdir -p src/OrleansGrains mkdir -p src/OrleansGrainInterfaces dotnet new console -o src/OrleansSilo --framework netcoreapp1.1 dotnet new console -o src/OrleansClient --framework netcoreapp1.1 dotnet new classlib -o src/OrleansGrains --framework netstandard1.5 dotnet new classlib -o src/OrleansGrainInterfaces --framework netstandard1.5 dotnet sln add src/OrleansSilo/OrleansSilo.csproj dotnet sln add src/OrleansClient/OrleansClient.csproj dotnet sln add src/OrleansGrains/OrleansGrains.csproj dotnet sln add src/OrleansGrainInterfaces/OrleansGrainInterfaces.csproj dotnet add src/OrleansClient/OrleansClient.csproj reference src/OrleansGrainInterfaces/OrleansGrainInterfaces.csproj dotnet add src/OrleansSilo/OrleansSilo.csproj reference src/OrleansGrainInterfaces/OrleansGrainInterfaces.csproj dotnet add src/OrleansGrains/OrleansGrains.csproj reference src/OrleansGrainInterfaces/OrleansGrainInterfaces.csproj dotnet add src/OrleansSilo/OrleansSilo.csproj reference src/OrleansGrains/OrleansGrains.csproj ``` What we did so far was just boilerplate code to create the solution structure, projects, and add references between projects. Nothing different than a regular Orleans project. By the time this article was written, Orleans 2.0 (which is the only version which support .Net Core and cross-platform) is in Technology Preview so its nugets are hosted in a MyGet feed and not published to Nuget.org official feed. In order to install the preview nugets, we will use `dotnet` cli forcing the source feed and version from MyGet: ```bash dotnet add src/OrleansClient/OrleansClient.csproj package Microsoft.Orleans.Core -s https://dotnet.myget.org/F/orleans-prerelease/api/v3/index.json -v 2.0.0-preview2-201705020000 dotnet add src/OrleansGrainInterfaces/OrleansGrainInterfaces.csproj package Microsoft.Orleans.Core -s https://dotnet.myget.org/F/orleans-prerelease/api/v3/index.json -v 2.0.0-preview2-201705020000 dotnet add src/OrleansGrains/OrleansGrains.csproj package Microsoft.Orleans.Core -s https://dotnet.myget.org/F/orleans-prerelease/api/v3/index.json -v 2.0.0-preview2-201705020000 dotnet add src/OrleansSilo/OrleansSilo.csproj package Microsoft.Orleans.Core -s https://dotnet.myget.org/F/orleans-prerelease/api/v3/index.json -v 2.0.0-preview2-201705020000 dotnet add src/OrleansSilo/OrleansSilo.csproj package Microsoft.Orleans.OrleansRuntime -s https://dotnet.myget.org/F/orleans-prerelease/api/v3/index.json -v 2.0.0-preview2-201705020000 dotnet restore ``` Ok, now you have all the basic dependencies to run a simple Orleans application. Note that so far, nothing changed from your regular Orleans application. Now, lets add some code so we can do something with it. # Implementing your Orleans Application Assuming that you are using **VSCode**, from the solution directory, run `code .`. That will open the directory in **VSCode** and load the solution. This is the solution structure we just created previously. We also added `Program.cs`, `OrleansHostWrapper`, `IGreetingGrain` and `GreetingGrain` files to the interfaces and grain projects respectively and here is the code for those files: `IGreetingGrain.cs`: ```csharp using System; using System.Threading.Tasks; using Orleans; namespace OrleansGrainInterfaces { public interface IGreetingGrain : IGrainWithGuidKey { Task SayHello(string name); } } ``` `GreetingGrain.cs`: ```csharp using System; using System.Threading.Tasks; using OrleansGrainInterfaces; namespace OrleansGrains { public class GreetingGrain : Grain, IGreetingGrain { public Task SayHello(string name) { return Task.FromResult($"Hello from Orleans, {name}"); } } } ``` `OrleansHostWrapper.cs`: ```csharp using System; using System.Net; using Orleans.Runtime; using Orleans.Runtime.Configuration; using Orleans.Runtime.Host; namespace OrleansSilo { public class OrleansHostWrapper { private readonly SiloHost siloHost; public OrleansHostWrapper(ClusterConfiguration config) { siloHost = new SiloHost(Dns.GetHostName(), config); siloHost.LoadOrleansConfig(); } public int Run() { if (siloHost == null) { return 1; } try { siloHost.InitializeOrleansSilo(); if (siloHost.StartOrleansSilo()) { Console.WriteLine($"Successfully started Orleans silo '{siloHost.Name}' as a {siloHost.Type} node."); return 0; } else { throw new OrleansException($"Failed to start Orleans silo '{siloHost.Name}' as a {siloHost.Type} node."); } } catch (Exception exc) { siloHost.ReportStartupError(exc); Console.Error.WriteLine(exc); return 1; } } public int Stop() { if (siloHost != null) { try { siloHost.StopOrleansSilo(); siloHost.Dispose(); Console.WriteLine($"Orleans silo '{siloHost.Name}' shutdown."); } catch (Exception exc) { siloHost.ReportStartupError(exc); Console.Error.WriteLine(exc); return 1; } } return 0; } } } ``` `Program.cs` (Silo): ```csharp using System; using System.Collections.Generic; using System.Linq; using System.Net; using Orleans.Runtime.Configuration; namespace OrleansSilo { public class Program { private static OrleansHostWrapper hostWrapper; static int Main(string[] args) { int exitCode = InitializeOrleans(); Console.WriteLine("Press Enter to terminate..."); Console.ReadLine(); exitCode += ShutdownSilo(); return exitCode; } private static int InitializeOrleans() { var config = new ClusterConfiguration(); config.Globals.DataConnectionString = "[AZURE STORAGE CONNECTION STRING HERE]"; config.Globals.DeploymentId = "Orleans-Docker"; config.Globals.LivenessType = GlobalConfiguration.LivenessProviderType.AzureTable; config.Globals.ReminderServiceType = GlobalConfiguration.ReminderServiceProviderType.AzureTable; config.Defaults.PropagateActivityId = true; config.Defaults.ProxyGatewayEndpoint = new IPEndPoint(IPAddress.Any, 10400); config.Defaults.Port = 10300; var ips = Dns.GetHostAddressesAsync(Dns.GetHostName()).Result; config.Defaults.HostNameOrIPAddress = ips.FirstOrDefault()?.ToString(); hostWrapper = new OrleansHostWrapper(config); return hostWrapper.Run(); } private static int ShutdownSilo() { if (hostWrapper != null) { return hostWrapper.Stop(); } return 0; } } } ``` `Program.cs`(client): ```csharp using System; using System.Net; using System.Threading; using System.Threading.Tasks; using Orleans; using Orleans.Runtime.Configuration; using OrleansGrainInterfaces; namespace OrleansClient { class Program { private static IClusterClient client; private static bool running; static void Main(string[] args) { Task.Run(() => InitializeOrleans()); Console.ReadLine(); running = false; } static async Task InitializeOrleans() { var config = new ClientConfiguration(); config.DeploymentId = "Orleans-Docker"; config.PropagateActivityId = true; var hostEntry = await Dns.GetHostEntryAsync("orleans-silo"); var ip = hostEntry.AddressList[0]; config.Gateways.Add(new IPEndPoint(ip, 10400)); Console.WriteLine("Initializing..."); client = new ClientBuilder().UseConfiguration(config).Build(); await client.Connect(); running = true; Console.WriteLine("Initialized!"); var grain = client.GetGrain(Guid.Empty); while(running) { var response = await grain.SayHello("Gutemberg"); Console.WriteLine($"[{DateTime.UtcNow}] - {response}"); await Task.Delay(1000); } client.Dispose(); } } } ``` We are not going into details about the grain implementation here since it is out of the scope of this article. Please check other documents related to it. Those files are essentially a minimal Orleans application and we will start from it to move forward with the remaining of this article. > **Note**: In this article we are using `OrleansAzureUtils` membership provider but you can use any other already supported by Orleans. # Dockerfile In order to create your container, Docker uses images. For more details on how to create your own, you can check [Docker documentation](https://docs.docker.com/engine/userguide/). In this article we are going to use official [Microsoft images](https://hub.docker.com/r/microsoft/dotnet/). Based on the target and development platforms, you need to pick the appropriate image. In this article, we are using `microsoft/dotnet:1.1.2-sdk` which is a linux-based image. You can use `microsoft/dotnet:1.1.2-sdk-nanoserver` for Windows for example. Pick one that suit your needs. > **Note for Windows users**: As previously mentioned, to be cross-platform, we are using .Net Core and Orleans Technical preview 2.0 in this article. If you want to use Docker on Windows with the fully released Orleans 1.4+, you need to use the images that are based on Windows Server Core since NanoServer and Linux based images, only support .Net Core. `Dockerfile.debug`: ```dockerfile FROM microsoft/dotnet:1.1.2-sdk ENV NUGET_XMLDOC_MODE skip WORKDIR /vsdbg RUN apt-get update \ && apt-get install -y --no-install-recommends \ unzip \ && rm -rf /var/lib/apt/lists/* \ && curl -sSL https://aka.ms/getvsdbgsh | bash /dev/stdin -v latest -l /vsdbg WORKDIR /app ENTRYPOINT ["tail", "-f", "/dev/null"] ``` This dockerfile essentially downloads and installs the VSdbg debugger and starts an empty container, keeping it alive forever so we don't need tear down/up while debugging. Now, for production, the image is smaller since it contains only the .Net Core runtime and not the whole SDK, and the dockerfile is a bit simpler: `Dockerfile`: ```dockerfile FROM microsoft/dotnet:1.1.2-runtime WORKDIR /app ENTRYPOINT ["dotnet", "OrleansSilo.dll"] COPY . /app ``` # docker-compose The `docker-compose.yml` file, essentially defines (within a project) a set of services and its dependencies at service level. Each service contains one or more instances of a given container, which is based on the images you selected on your Dockerfile. More details on the `docker-compose` can be found on [docker-compose documentation](https://docs.docker.com/compose/). For an Orleans deployment, a common use case is to have a `docker-compose.yml` which contains two services. One for Orleans Silo, and other for Orleans Client. The Client would have a dependency on the Silo and that means, it will only start after the Silo service is up. Another case is to add a storage/database service/container, like for example SQL Server, which should start first before the client and the silo, so both services should take a dependency on it. > **Note**: Before you read further (and eventually get crazy with it), please note that _identation_ **matters** in `docker-compose` files. So pay attention to it if you have any problem. Here is how we will describe our services for this article: `docker-compose.override.yml` (Debug): ```yml version: '3.1' services: orleans-client: image: orleans-client:debug build: context: ./src/OrleansClient/bin/PublishOutput/ dockerfile: Dockerfile.Debug volumes: - ./src/OrleansClient/bin/PublishOutput/:/app - ~/.nuget/packages:/root/.nuget/packages:ro depends_on: - orleans-silo orleans-silo: image: orleans-silo:debug build: context: ./src/OrleansSilo/bin/PublishOutput/ dockerfile: Dockerfile.Debug volumes: - ./src/OrleansSilo/bin/PublishOutput/:/app - ~/.nuget/packages:/root/.nuget/packages:ro ``` `docker-compose.yml` (production): ```yml version: '3.1' services: orleans-client: image: orleans-client depends_on: - orleans-silo orleans-silo: image: orleans-silo ``` Note that in production, we don't map the local directory and neither we have the `build:` action. The reason is that in production, the images should be built and pushed to your own Docker Registry. # Put everything together Now we have all the moving parts required to run your Orleans Application, we are going to put it together so we can run our Orleans solution inside Docker (Finally!). > **Note**: The following commands should be performed from the solution directory. First, lets make sure we restore all NuGet packages from our solution. You only need to do it once. You are only required to do it again if you change any package dependency on your project. `# dotnet restore` Now, let's build our solution using `dotnet` CLI as usual and publish it to an output directory: `# dotnet publish -o ./bin/PublishOutput` > **Note**: We are using `publish` here instead of build, to avoid problems with our dynamicaly loaded assemblied in Orleans. We are still looking for a better solution for it. With the application built and published, you need to build your Dockerfile images. This step is only required to be performed once per project and should be only performed again if you change the Dockerfile, docker-compose, or for any reason you cleaned up your local image registry. `# docker-compose build` All the images used in both `Dockerfile` and `docker-compose.yml` are pulled from the registry and cached on your development machine. Your images are built, and you are all set to run. Now lets run it! ```shell # docker-compose up -d Creating network "orleansdocker_default" with the default driver Creating orleansdocker_orleans-silo_1 ... Creating orleansdocker_orleans-silo_1 ... done Creating orleansdocker_orleans-client_1 ... Creating orleansdocker_orleans-client_1 ... done # ``` Now if you run a `docker-compose ps`, you will see 2 containers running for the `orleansdocker` project: ```shell # docker-compose ps Name Command State Ports ------------------------------------------------------------------ orleansdocker_orleans-client_1 tail -f /dev/null Up orleansdocker_orleans-silo_1 tail -f /dev/null Up ``` > **Note for Windows users**: If you are on Windows, and your container is using a Windows image as base, the **Command** column will show you the Powershell relative command to a `tail` on *NIX systems so the container will keep up the same way. Now that you have your containers up, you don't need to stop it every time you want to start your Orleans application. All you need is to integrate your IDE to debug the application inside the container which was previously mapped in your `docker-compose.yml`. ## Scaling Once you have your compose project running, you can easily scale up or down your application using `docker-compose scale` command: ```shell # docker-compose scale orleans-silo=15 Starting orleansdocker_orleans-silo_1 ... done Creating orleansdocker_orleans-silo_2 ... Creating orleansdocker_orleans-silo_3 ... Creating orleansdocker_orleans-silo_4 ... Creating orleansdocker_orleans-silo_5 ... Creating orleansdocker_orleans-silo_6 ... Creating orleansdocker_orleans-silo_7 ... Creating orleansdocker_orleans-silo_8 ... Creating orleansdocker_orleans-silo_9 ... Creating orleansdocker_orleans-silo_10 ... Creating orleansdocker_orleans-silo_11 ... Creating orleansdocker_orleans-silo_12 ... Creating orleansdocker_orleans-silo_13 ... Creating orleansdocker_orleans-silo_14 ... Creating orleansdocker_orleans-silo_15 ... Creating orleansdocker_orleans-silo_6 Creating orleansdocker_orleans-silo_5 Creating orleansdocker_orleans-silo_3 Creating orleansdocker_orleans-silo_2 Creating orleansdocker_orleans-silo_4 Creating orleansdocker_orleans-silo_9 Creating orleansdocker_orleans-silo_7 Creating orleansdocker_orleans-silo_8 Creating orleansdocker_orleans-silo_10 Creating orleansdocker_orleans-silo_11 Creating orleansdocker_orleans-silo_15 Creating orleansdocker_orleans-silo_12 Creating orleansdocker_orleans-silo_14 Creating orleansdocker_orleans-silo_13 ``` Few seconds later, you will see the services scaled to the specific number of instances you requested. ```shell # docker-compose ps Name Command State Ports ------------------------------------------------------------------ orleansdocker_orleans-client_1 tail -f /dev/null Up orleansdocker_orleans-silo_1 tail -f /dev/null Up orleansdocker_orleans-silo_10 tail -f /dev/null Up orleansdocker_orleans-silo_11 tail -f /dev/null Up orleansdocker_orleans-silo_12 tail -f /dev/null Up orleansdocker_orleans-silo_13 tail -f /dev/null Up orleansdocker_orleans-silo_14 tail -f /dev/null Up orleansdocker_orleans-silo_15 tail -f /dev/null Up orleansdocker_orleans-silo_2 tail -f /dev/null Up orleansdocker_orleans-silo_3 tail -f /dev/null Up orleansdocker_orleans-silo_4 tail -f /dev/null Up orleansdocker_orleans-silo_5 tail -f /dev/null Up orleansdocker_orleans-silo_6 tail -f /dev/null Up orleansdocker_orleans-silo_7 tail -f /dev/null Up orleansdocker_orleans-silo_8 tail -f /dev/null Up orleansdocker_orleans-silo_9 tail -f /dev/null Up ``` > **Note**: The `Command` column on those examples are showing the `tail` command just because we are using the debugger container. If we were in production, it would be showing `dotnet OrleansSilo.dll` for example. # Docker Swarm Docker clustering stack is called **Swarm** and you can find more by reading its [documentation here](https://docs.docker.com/engine/swarm/). To run this article in a `Swarm` cluster, you don't have any extra work. When you run `docker-compose up -d` in a `Swarm` node, it will schedule containers based on the configured rules. The same applies to other Swarm-based services like [Docker Datacenter](https://www.docker.com/enterprise-edition), [Azure ACS](https://azure.microsoft.com/en-us/services/container-service/) (in Swarm mode), [AWS ECS Container Service](https://aws.amazon.com/ecs/) and so on. All you need to do is to deploy your `Swarm` cluster before deploy your **dockerized** Orleans application. > **Note**: If you are using a Docker engine with the Swarm mode that already have support to `stack`, `deploy` and `compose` v3, a better approach to deploy your solution would be `docker stack deploy -c docker-compose.yml `. Just keep in mind that it requires v3 compose file support at your Docker engine and the majority of hosted services like Azure and AWS still use v2 and older engines. # Google Kubernetes (K8s) If you plan to use [Kubernetes](https://kubernetes.io/) to host Orleans, there is a community-maintained clustering provider available at [OrleansContrib\Orleans.Clustering.Kubernetes](https://github.com/OrleansContrib/Orleans.Clustering.Kubernetes) and there you can find documentation and samples on how to host Orleans in Kubernetes seamlessly using the provider. # [Bonus topic] Debugging Orleans inside Containers Well, now that you know how to run Orleans in a container from scratch, would be good to leverage one of the most important principles in Docker. Containers are immutable. And they should have (almost) the same image, dependencies, and runtime in development as in production. This ensures the good old statement *"It works on my machine!"* never happens again. To make that possible, you need to have a way to develop *inside* the container and that includes have a debugger attached to your application inside the container. There are multiple ways to achieve that using multiple tools. After evaluating several, by the time I wrote this article, I ended up choosing one that looks more simple and is less intrusive in the application. As mentioned ealier in this article, we are using `VSCode` to develop the sample, so here is how to get the debugger attached to your Orleans Application inside the container. First, change two files inside your `.vscode` directory in your solution: `tasks.json`: ```json { "version": "0.1.0", "command": "dotnet", "isShellCommand": true, "args": [], "tasks": [ { "taskName": "publish", "args": [ "${workspaceRoot}/Orleans-Docker.sln", "-c", "Debug", "-o", "./bin/PublishOutput" ], "isBuildCommand": true, "problemMatcher": "$msCompile" } ] } ``` This file essentially tells `VSCode` that whenever you build the project, it will actually execute the `publish` command as we manually did earlier. `launch.json`: ```json { "version": "0.2.0", "configurations": [ { "name": "Silo", "type": "coreclr", "request": "launch", "cwd": "/app", "program": "/app/OrleansSilo.dll", "sourceFileMap": { "/app": "${workspaceRoot}/src/OrleansSilo" }, "pipeTransport": { "debuggerPath": "/vsdbg/vsdbg", "pipeProgram": "/bin/bash", "pipeCwd": "${workspaceRoot}", "pipeArgs": [ "-c", "docker exec -i orleansdocker_orleans-silo_1 /vsdbg/vsdbg --interpreter=vscode" ] } }, { "name": "Client", "type": "coreclr", "request": "launch", "cwd": "/app", "program": "/app/OrleansClient.dll", "sourceFileMap": { "/app": "${workspaceRoot}/src/OrleansClient" }, "pipeTransport": { "debuggerPath": "/vsdbg/vsdbg", "pipeProgram": "/bin/bash", "pipeCwd": "${workspaceRoot}", "pipeArgs": [ "-c", "docker exec -i orleansdocker_orleans-client_1 /vsdbg/vsdbg --interpreter=vscode" ] } } ] } ``` Now you can just build the solution from `VSCode` (which will publish) and start both the Silo and the Client. It will send a `docker exec` command to the running `docker-compose` service instance/container to start the debugger to the application and thats it. You have the debugger attached to the container and use it as if it was a locally running Orleans application. The difference now is that it is inside the container, and once you are done, you can just publish the container to your registry and pull it on your Docker hosts in production. --- ## File: src/docs/deployment/handling_failures.md --- title: Handling Failures --- # Handling Failures > **Note:** All of the following guidance in this document is provided to serve as examples and food for thought. > You should not think of them as prescriptive solutions to your problems because failure handling is a rather application-specific subject. > These patterns and others are only useful if applied with a good knowledge of the concrete case being worked on. The hardest thing in programming a distributed system is handling failures. The actor model and the way it works makes it much easier to deal with different kinds of failures, but as a developer, you are responsible for dealing with the failure possibilities and handling them in an appropriate way. ## Types of failures When you are coding your grains, all calls are asynchronous and have the potential to go over the network. Each grain call can possibly fail due to one of the following reasons. - The grain was activated on a silo which is unavailable at the moment due to a network partition crash or some other reason. If the silo has not been declared dead yet, your request might time out. - The grain method call can throw an exception signaling that it failed and can not continue its job. - An activation of the grain doesn't exist and cannot be created because the `OnActivateAsync` method throws an exception or is dead-locked. - Network failures don't let you to communicate with the grain before timeout. - Other potential reasons ## Detection of failures Getting a reference to a grain always succeeds and is a local operation. However, method calls can fail, and when they do, you get an exception. You can catch the exception at any level you need and they are propagated even across silos. ## Recovering from failures Part of the recovery job is automatic in Orleans and if a grain is not accessible anymore, Orleans will reactivate it in the next method call. The thing you need to handle and make sure is correct in the context of your application is the state. A grain's state can be partially updated or the operation might be something which should be done across multiple grains and is carried on partially. After you see a grain operation fail you can do one or more of the following. - Simply retry your action, especially if it doesn't involve any state changes which might be half done. This is by far the most typical case. - Try to fix/reset the partially changed state by calling a method which resets the state to the last known correct state or just reads it from storage by calling `ReadStateAsync`. - Reset the state of all related activations as well to ensure a clean state for all of them. - Perform multi-grain state manipulations using a [Process Manager](https://msdn.microsoft.com/en-us/library/jj591569.aspx) or database transaction to make sure it's either done completely or nothing is changed to avoid the state being partially updated. Depending on your application, the retry logic might follow a simple or complex pattern, and you might have to do other stuff like notifying external systems and other things, but generally you either have to retry your action, restart the grain/grains involved, or stop responding until something which is not available becomes available. If you have a grain responsible for database saves and the database is not available, you simply have to fail any request until the database comes back online. If your operation can be retried at the user's will, like failure of saving a comment in a comment grain, you can retry when the user presses the retry button (until a certain number of times in order to not saturate the network). Specific details of how to do it are application specific, but the possible strategies are the same. ## Strategy parameters and choosing a good strategy As described in the section above, the strategy you choose depends on the application and context. Strategies usually have parameters which have to be decided at the application level. For example, if no other processes depend on an action, then you might decide to retry that action a maximum of 5 times per minute until it eventually completes. But you would have to process a user's Login grain request before processing any other requests from that user. If the login action is not working, then you cannot continue. There is a guide [in the Azure documentation](https://docs.microsoft.com/en-us/azure/architecture/patterns/) about good patterns and practices for the cloud which applies to Orleans as well, in most cases. ## A fairly complex example Because in Orleans grains are activated and deactivated automatically and you don't handle their life-cycle, you usually only deal with making sure that grain state is correct and actions are being started and finished correctly in relation to each other. Knowing the dependencies between grains and actions they perform is a big step toward understanding how to handle failure in any complex system. If you need to store relations among grains, you can simply do it and it is a widely followed practice, too. As an example, let's say we have a `GameManager` grain which starts and stops `Game` grains and adds `Player` grains to the games. If my `GameManager`grain fails to do its action regarding starting a game, the related game belonging to it should fail to do its `Start()` as well and the manager can do this for the game by doing orchestration. Managing memory in Orleans is automatic and the system deals with it, you only need to make sure that the game starts and only if manager can do its `Start()` as well. You can achieve this by either calling the related methods in a sequential manner or by doing them in parallel and resetting the state of all involved grains if any of them fail. If one of the games receives a call, it will be reactivated automatically, so if you need the manager to manage the game grains, then all calls to the game which are related to management should go through the `GameManager`. If you need orchestration among grains, Orleans doesn't solve it "automagically" for you and you need to do your orchestration. But the fact that you are not dealing with creating/destroying grains means you don't need to worry about resource management. You don't need to answer any of these questions: - Where should I create my supervision tree? - which grains should I register to be addressable by name? - Is grain X alive so I can send it a message? - ... So, the game start example can be summarized like this: - `GameManager` asks the `Game` grain to start - `Game` grain adds the `Player` grains to itself - `Game` asks `Player` grains to add game to themselves - `Game` sets its state to be started. - `GameManager` adds the game to its list of games. Now, if a player fails to add the game to itself, you don't need to kill all players and the game and start over. You simply reset the state of the other players which added the game to themselves, reset the state of the `Game` and `GameManager` (if required), and redo your work or declare a failure. If you can deal with adding the game to the player later on, you can continue and retry doing that again in a reminder or at some other game call like the `Finish()` method of the game. --- ## File: src/docs/deployment/index.md --- title: Running the Application --- ### Orleans Application A typical Orleans application consists of a cluster of server processes (silos) where grains live, and a set of client processes, usually web servers, that receive external requests, turn them into grain method calls, and return results back. Hence, the first thing one needs to do to run an Orleans application is to start a cluster of silos. For testing purposes, a cluster can consist of a single silo. For a reliable production deployment, we obviously want more than one silos in a cluster for fault tolerance and scale. Once the cluster is running, we can start one or more client processes that connect to the cluster and can send requests to the grains. Clients connect to a special TCP endpoint on silos - gateway. By default, every silo in a cluster has a client gateway enabled. So clients can connect to all silos in parallel for better performance and resilience. ### Configuring and Starting a Silo A silo is configured programmatically via a `ClusterConfiguration` object. It can be instantiated and populated directly, load settings from a file, or created with several available helper methods for different deployment environments. For local testing, the easiest way to go is to use `ClusterConfiguration.LocalhostPrimarySilo()` helper method. The configuration object is then passed to a new instance of `SiloHost` class, that can be initialized and started after that. You can create an empty console application project targeting .NET Framework 4.6.1 or higher for hosting a silo. Add the `Microsoft.Orleans.Server` NuGet meta-package to the project. ``` PM> Install-Package Microsoft.Orleans.Server ``` Here is an example of how a local silo can be started: ```csharp var siloConfig = ClusterConfiguration.LocalhostPrimarySilo(); var silo = new SiloHost("Test Silo", siloConfig); silo.InitializeOrleansSilo(); silo.StartOrleansSilo(); Console.WriteLine("Press Enter to close."); // wait here Console.ReadLine(); // shut the silo down after we are done. silo.ShutdownOrleansSilo(); ``` ### Configuring and Connecting a Client Client for connecting to a cluster of silos and sending requests to grains is configured programmatically via a `ClientConfiguration` object and a `ClientBuilder`. `ClientConfiguration` object can be instantiated and populated directly, load settings from a file, or created with several available helper methods for different deployment environments. For local testing, the easiest way to go is to use `ClientConfiguration.LocalhostSilo()` helper method. The configuration object is then passed to a new instance of `ClientBuilder` class. `ClientBuilder` exposes more methods for configuring additional client features. After that `Build` method of the `ClientBuilder` object is called to get an implementation of `IClusterClient` interface. Finally, we call `Connect()` method on the returned object to connect to the cluster. You can create an empty console application project targeting .NET Framework 4.6.1 or higher for running a client or reuse the console application project you created for hosting a silo. Add the `Microsoft.Orleans.Client` NuGet meta-package to the project. ``` PM> Install-Package Microsoft.Orleans.Client ``` Here is an example of how a client can connect to a local silo: ```csharp var config = ClientConfiguration.LocalhostSilo(); var builder = new ClientBuilder().UseConfiguration(config). var client = builder.Build(); await client.Connect(); ``` ### Production Configurations The configuration examples we used here are for testing silos and clients running on the same machine as `localhost`. In production, silos and clients usually run on different servers and are configured with one of the reliable cluster configuration options. You can find more about that in the [Configuration Guide](~/docs/host/configuration_guide/index.md) and in the description of [Cluster Management](~/docs/implementation/cluster_management.md).