yarp Documentation

1. Project Overview & Quickstart (dotnet/yarp) Copy Chapter

File: README.md Copy Chapter

Welcome to the YARP project

YARP (which stands for "Yet Another Reverse Proxy") is a project to create a reverse proxy server. We found a bunch of internal teams at Microsoft who were either building a reverse proxy for their service or had been asking about APIs and tech for building one, so we decided to get them all together to work on a common solution, this project.

YARP is a reverse proxy toolkit for building fast proxy servers in .NET using the infrastructure from ASP.NET and .NET. The key differentiator for YARP is that it's been designed to be easily customized and tweaked to match the specific needs of each deployment scenario.

We expect YARP to ship as a library and project template that together provide a robust, performant proxy server. Its pipeline and modules are designed so that you can then customize the functionality for your needs. For example, while YARP supports configuration files, we expect that many users will want to manage the configuration programmatically based on their own backend configuration management system, YARP will provide a configuration API to enable that customization in-proc. YARP is designed with customizability as a primary scenario, rather than requiring you to break out to script or having to rebuild from source.

Getting started

• See our Getting Started docs.
• Try our previews.
• Try our latest daily build.
• See our support policy.

Updates

For regular updates, see our releases page. Subscribe to release notifications on this repository to be notified of future updates (Watch -> Custom -> Releases).

If you want to live on the bleeding edge, you can pickup the daily builds.

Build

To build the repo, you should only need to run build.cmd (on Windows) or build.sh (on Linux or macOS). The script will download the .NET SDK and build the solution.

For VS on Windows, install the latest VS 2022 release and then run the startvs.cmd script to launch Visual Studio using the appropriate local copy of the .NET SDK.

To set up local development with Visual Studio, Visual Studio for Mac or Visual Studio Code, you need to put the local copy of the .NET SDK in your PATH environment variable. Our Restore script fetches the latest build of .NET and installs it to a .dotnet directory within this repository.

We provide some scripts to set all this up for you. Just follow these steps:

1. Run the restore.cmd/restore.sh script to fetch the required .NET SDK locally (to the .dotnet directory within this repo)
2. "Dot-source" the activate script to put the local .NET SDK on the PATH
   1. For PowerShell, run: . .\activate.ps1 (note the leading . , it is required!)
   2. For Linux/macOS/WSL, run: . ./activate.sh
   3. For CMD, there is no supported script. You can manually add the .dotnet directory within this repo to your PATH. Ensure where dotnet shows a path within this repository!
3. Launch VS, VS for Mac, or VS Code!

When you're done, you can run the deactivate function to undo the changes to your PATH.

If you're having trouble building the project, or developing in Visual Studio, please file an issue to let us know and we'll help out (and fix our scripts/tools as needed)!

Testing

The command to build and run all tests: build.cmd/sh -test.
To run specific test you may use XunitMethodName property: dotnet build /t:Test /p:XunitMethodName={FullyQualifiedNamespace}.{ClassName}.{MethodName}.
The tests can also be run from Visual Studio if launched using startvs.cmd.

Roadmap

see docs/roadmap.md

Reporting security issues and bugs

Security issues and bugs should be reported privately, via email, to the Microsoft Security Response Center (MSRC) at [email protected]. You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Further information, including the MSRC PGP key, can be found at the Microsoft Security Response Center.

Contributing

This project welcomes contributions and suggestions.
Check out the contributing page for more info.

This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community.
For more information, see the .NET Foundation Code of Conduct.

File: docs/designs/config.md Copy Chapter

Config based proxy apps

[!CAUTION]
These are archived design discussions. Information may be outdated and inaccurate.

RE: https://github.com/dotnet/yarp/issues/9

Config based proxies are common and we'll need to support at least basic proxy scenarios from config. Here are some initial considerations:

• Config sources and systems
• Define routes based on host and/or path
• List multiple back-ends per route for load balancing
• A restart should not be needed to pick up config changes
• You should be able to augment a route's configuration in code. Kestrel does something similar using named endpoints.

Config systems: Copy Chapter

We have three relevant components that already have config systems: Kestrel, UrlRewrite, and ReverseProxy.

• Kestrel
• UrlRewrite
• ReverseProxy

Proposals:

• The Kestrel config and the Proxy/Gateway config should remain adjacent, not merged. Inbound and outbound are distinct concerns. As long as both are available in the same broader config system then that's close enough.
• UrlRewrite should also remain as is. It's not ideal that it's in a separate file and format from the rest of the config, but we'll wait and see if that is a long term blocker.

Route config: Copy Chapter

The proxy has a config mechanism to define routes and map those to back end groups.

"Routes": [
        {
          "RouteId": "backend1/route1",
          "BackendId": "backend1",
          "Rule": "Host('localhost') && Path('/{**catchall}')"
        }
      ]

This maps to a ProxyRoute type.

This basic structure is useful though the "Rule" system seems overly complex. Need to circle back with DavidN on this. We may be able to simplify that down to independent keys for matching Host, Path, Header, etc.. It's not clear that the additional && or || aspects are necessary here. If we used separate properties then it would be implicitly && based. To achieve || you'd define additional routes. This is also an area where augmenting with code defined constraints could be useful to handle the more complex scenarios.

The ProxyRoute.Metadata dictionary may be able to be replaced or supplemented by giving direct access to the config node for that route. Compare to Kestrel's EndpointConfig.ConfigSection property. That would allow for augmenting an endpoint with additional complex custom entries that the app code can reference for additional config actions.

Update: The custom rule system was modified by #24 so that the config now looks like this:

"Routes": [
         {
           "RouteId": "backend1/route1",
           "BackendId": "backend1",
           "Match": {
             "Methods": [ "GET", "POST" ],
             "Host": "localhost",
             "Path": "/{**catchall}"
           }
         }
       ]

Backend configuration Copy Chapter

The proxy code defines the types Backend and BackendEndpoint and allows these to be defined via config and referenced by name from routes.

A BackendEndpoint defines a specific service instance with an id, address, and associated metadata.

A Backend is a collection of one or more BackendEndpoints and a set of policies for choosing which endpoint to rout each request to (load balancing, circuit breakers, health checks, affinities, etc.). This seems a bit monolithic compared to our initial design explorations. We anticipate wanting to break these policies up into distinct steps in a pipeline to make them more replaceable. That said, we'll still need a config model for the default set of components and it may look very much like what's already here.

Question: Why are the backends and the endpoints listed separately in config rather than nested? Object model links endpoints 1:1 with backends, so there doesn't seem to be a reason to list them separately.

Existing:

"Backends": [
        {
          "BackendId": "backend1"
        },
        {
          "BackendId": "backend2"
        }
      ],
      "Endpoints": {
        "backend1": [
          {
            "EndpointId": "backend1/endpoint1",
            "Address": "https://localhost:10000/"
          }
        ],
        "backend2": [
          {
            "EndpointId": "backend2/endpoint1",
            "Address": "https://localhost:10001/"
          }
        ]
      },

Nested:

"Backends": [
        {
          "BackendId": "backend1",
          "Endpoints": [
            {
              "EndpointId": "backend1/endpoint1",
              "Address": "https://localhost:10000/"
            }
          ],
        },
        {
          "BackendId": "backend2"
          "Endpoints": [
            {
              "EndpointId": "backend2/endpoint1",
              "Address": "https://localhost:10001/"
            }
          ],
        }
      ],

Additional feedback: Why is it using arrays instead of objects? These items are not order sensitive, and they already have id properties anyways.

"Backends": {
        "backend1" : {
          "Endpoints": [
            "endpoint1": {
              "Address": "https://localhost:10000/"
            }
          },
        },
        "backend2": {
          "Endpoints": {
            "endpoint1": {
              "Address": "https://localhost:10001/"
            }
          },
        }
      },

Update: The backend and endpoint layout has been modified to the following:

"Backends": {
      "backend1": {
        "Endpoints": {
          "backend1/endpoint1": {
            "Address": "https://localhost:10000/"
          }
        }
      },
      "backend2": {
        "Endpoints": {
          "backend2/endpoint1": {
            "Address": "https://localhost:10001/"
          }
        }
      }
    },

Config reloading Copy Chapter

Config reloading is not yet a blocking requirement but we do expect to need it in the future. This design needs to factor in how reloading might work when it does get added.

NOTE The proxy code has a concept of Signals that is used to convey config change. We need to see how this integrates with change notifications from our config sources and flows through the system.

The Extensions config and options systems have support for change detection and reloading but very few components take advantage of it. Logging is the primary consumer today.

One concern is that some change notification sources like files can trigger multiple times for a single event. The config system does not have built in handling for this, it's up to consumers to 'debounce' and filter out redundant notifications.

Kestrel support for reloading config is tracked by https://github.com/dotnet/aspnetcore/issues/19376.

Reloading proxy config will need to happen atomically and avoid disrupting requests already in flight. We may need to rebuild portions of the app pipeline and swap them out for new requests, drain the old requests, and clean up the old pipelines. We also want to avoid a full reset for small config changes where possible. E.g. if only one route changes then ideally we'd only rebuild that route.

Reloading should be something you can opt into or out of. Right now this is only possible at the config level by opting in or out for a config source, but that affects the whole app.

Updates:

Config reload for proxy routes, backends, and endpoints already works. You edit appsettings.json and it automatically reloads and reconfigures the routes. Note the config change notification usually gets fired twice and logs "Applying proxy configs" each time, but the change diff logic prevents an unnecessary update the second time. We may still want to do some debounce detection to prevent extra config diffs, but that's lower priority.

@halter73 raised the question of how much effort we put in to make the kestrel reload atomic with the routing reload? Conceptually it makes sense to keep the two in sync, but programmatically it's quite difficult as there's no connection between the two systems. They'd be reacting to the same change notification event in serial and requests in flight may see one set of changes without the other. We discussed this in the weekly sync and decided that since kestrel endpoint changes will be a less common scenario we won't initually worry about the atomicity here until we have customer feedback that demonstrates issues.

Also, when a kestrel endpoint is modified or removed should existing connections on that endpoint be eagerly drained and closed, or should they be allowed to run a normal lifecycle? Kestrel does not currently track active connection per endpoint so additional tracking would be needed if we wanted to shut them down.

Updates:

The config reload code has been moved from the sample into the product assemblies.

Augmenting config via code Copy Chapter

Some things are easier to do in code and we want to be able to support that while still pulling more transient data from config. Kestrel has a model where endpoints are named in config and then can be reference by name in code for additional configuration.

{
  "Kestrel": {
    "Endpoints": {
      "NamedEndpoint": { "Url": "http://*:6000" },
      "NamedHttpsEndpoint": {
        "Url": "https://*:6443",
      }
    }
  }
}

options.Endpoint("NamedEndpoint", opt =>
    {

    })
    .Endpoint("NamedHttpsEndpoint", opt =>
    {
        opt.HttpsOptions.SslProtocols = SslProtocols.Tls12;
    });

The proxy code already has named routes, backends, backend endpoints, etc., so we should be able to build a similar code augmentation for those.

Reloadable config complicates this pattern. The code augmentation actions will need to be captured for the lifetime of the app rather than just for startup so they can be re-run later.

File: docs/designs/route-extensibility.md Copy Chapter

Problem Statement Copy Chapter

[!CAUTION]
These are archived design discussions. Information may be outdated and inaccurate.

Today if you want to extend the route or clusters, you can only do it through the metadata property on each, which is a Dictionary<string, string>. If you want to be able to have structured data its not possible without you forcing it into a string and then parsing it when needed. There are scenarios like A/B testing, or authenticating with back end servers (not pass thru) where you want to be able to store a structure of data in config, and have it available at runtime on the route/cluster objects.

If we want there to be pre-built extensions to YARP (#1714), then there needs to be a way for each of the extensions to have its own config data on routes and clusters, and for them to not step on each others toes.

Why is this important to you? Copy Chapter

Taking a canonical example of A/B testing. In this scenario you want to be able to direct traffic to multiple clusters with the traffic patterns determined based on additional criteria. For example, You may want to be able to have a collection of clusters that are used for a route, together with percentages. So the config could look something like:

{
  "ReverseProxy": {
    "Routes": {
      "route1": {
        "ClusterId": "ignore",
        "Match": {
          "Path": "{**catch-all}"
        },
        "Extensions": {
          "A-B": [
            {
              "ClusterId": "c1",
              "Load": 0.4
            },
            {
              "ClusterId": "c2",
              "Load": 0.6
            },
            {
              "ClusterId": "experimental",
              "Load": 0.01
            },
            {
              "ClusterId": "TelemetrySample",
              "Load": 0.01
            }
          ]
        }
      },

The above example is adding an "A-B" extension to the route with its own data. There could be other extensions each of which have their own configuration data.

What would this look like Copy Chapter

Requirements Copy Section

• Multiple extensions can be added to YARP, each of which can have their own config state.
• Be able to have arbitrary data as part of config for routes and/or clusters
  • YARP should not dictate a specific data structure for the extension config
• Be able to access that data as objects from middleware directly off the route and clusters
• Be able to remote that data in the case of a distributed configuration server

Proposal Copy Chapter

• Add an Extensions collection to Route and Cluster. This should follow the same pattern as http features, using an IReadOnlyDictionary<Type, object> and be accessed based on the type of the extension. This would then be accessible in proxy middleware from the route or cluster objects via an Extensions property:

public void Configure(IApplicationBuilder app, IProxyStateLookup lookup)
{
    app.UseRouting();
    app.UseEndpoints(endpoints =>
    {
        // We can customize the proxy pipeline and add/remove/replace steps
        endpoints.MapReverseProxy(proxyPipeline =>
        {
            // Use a custom proxy middleware, defined below
            proxyPipeline.Use((context, next) =>
            {
                var proxyFeature = context.Features.Get();
                var abstate = proxyFeature.Route.Extensions[typeof(ABState)];
                var newClusterName = abstate.SelectSlice(new Random().NextDouble());
                if (lookup?.TryGetCluster(newClusterName, out var cluster))
                {
                    context.ReassignProxyRequest(cluster);
                }
                return next();
            });
            proxyPipeline.UseSessionAffinity();
            proxyPipeline.UseLoadBalancing();
        });
    });
}

Using a dictionary based on object type enables easy access to the object at runtime, and a strongly typed result.

• There needs to be a way for YARP to construct these strongly typed extensions based on the IConfiguration provider. There needs to be a mapping between the key in config, and the type that will be used to store the data. That mapping is handled by enabling factories that can be registered:

services.AddReverseProxy()
    .LoadFromConfig(Configuration.GetSection("ReverseProxy"))
    .AddRouteExtension("A-B", (section, _, _) => new ABState(section));

Along with a similar mechanism for clusters. The extension registration would be something like:

static IReverseProxyBuilder AddRouteExtension(this IReverseProxyBuilder builder, string sectionName, Func factory)

Where the factory is passed:

• The IConfigurationSection for the extension
• The route object that is being extended
• The existing extension instance in the case of configuration updates

When the configuration is parsed, the factory is called based on the configuration key name, and the resultant object is added to the route/cluster objects.

Regular YARP config will do a diff merge to handle config changes, and create new objects if applicable. The same mechanism needs to be used for extensions. If the configuration is updated, and an existing instance of the extension exists, then it will be passed to the factory. The factory can compare the current instance and re-use it, or copy its data across to a new instance based on the changes. Instances must stable, so existing instances shouldn't be modified if it would affect existing in-flight requests. YARP can't really enforce rules on how the objects are changed as we want the types to be user defined.

• When using a custom config provider, the Extensions collection can be populated by the custom provider directly, or the provider can expose an IConfigurationSection implementation and use the factory as described below.

• When YARP is integrated with 3rd party config systems, such as K8s or ServiceFabric, those systems typically have a way of expressing custom properties, some of which will be used by YARP for the definition of routes/ clusters etc. To facilitate the ability for route and cluster extensions to be expressed within those systems, the integration provider should expose an IConfigurationSection implementation that maps between the integrated persistence format and YARP.
  IConfigurationSection is essentially a name/value lookup API, it should map pretty reasonably to the YAML or JSON formats used by the configuration systems, and not be an undue burden to implement on these integrations.

• Integration with 3rd party config systems can involve a remote process that YARP can pull its configuration from. I am proposing another feature, that we formalize this pattern and have the ability to create a central YARP config provider, to which multiple YARP proxies can bind. This enables scalability in terms of being able to push config to multiple instances of YARP at once.

• To support this scenario, we should serialize the IConfiguration data and pass that across to the proxy instances.

File: docs/designs/yarp-tunneling.md Copy Chapter

YARP Tunneling

[!CAUTION]
These are archived design discussions. Information may be outdated and inaccurate.

Introduction Copy Chapter

While many organizations are moving their computing to the cloud, there are occasions where you need to be able to run some services in a local datacenter. The problem then is if you need to be able to communicate with those services from the cloud. Creating a VPN network connection to Azure or other cloud provider is possible, but usually involves a lot of red tape and configuration complexity as the two networks need to be integrated.

If all that the cloud needs to access is resources that are exposed over http, then a simpler solution is to have a gateway that can route traffic to the remote resource. Additionally, outbound connections over http(s) are not usually blocked, so having a on-prem gateway make an outbound connection to the cloud, is the easiest way to establish the route. This is the basis behind the Azure Relay service offering.

That is the principle of the tunnel feature for YARP. You operate two instances of the YARP proxy service, configured as a tunnel. The advantage over Azure Relay is that using a reverse proxy as an on-prem gateway means that both cloud and back end services can be used without needing to update the application other than addresses. This is particularly useful for services that may have been written by a 3rd party, or are no longer under active development, and so making changes to the configuration is complicated and expensive. Relay requires the sender and receiver to be updated to use its connection protocol.

In the on-prem data center, you run an instance of YARP, we'll call this the back-end proxy. This is configured with routes to the resources that should be externally accessible - only routes that are configured via this proxy will be exposed. The back-end proxy is configured to create a tunnel connection to the front-end instance by specifying the connection URL and security details for the connection.

The instance in the cloud, we'll refer to as the front-end, will be configured with a tunnel endpoint URL to be used by the on-prem proxy. The on-prem proxy will create a websocket connection to the tunnel endpoint, this will map the tunnel to a specific cluster. Routes can be directed to use the tunnel connection to the back-end by using the cluster that is used for the tunnel.

Tunnel Protocol Copy Chapter

The tunnel will establish a Websockets connection between the back-end and the front-end. The back-end will establish the connection so that it can more easily break through firewalls. Once the WSS connection is created, it will be treated as a stream over which HTTP/2 traffic will be routed. HTTP/2 is used so that multiple simultaneous requests can be multiplexed over a single connection. The HTTP/2 protocol is only used between the two proxies, the connections either side can be any protocol that the proxy supports. This means we don't put any specific capability requirement on the destination servers.

If the tunnel connection is broken, the back-end will attempt to reconnect to the front-end:

• If the connection fails, then it will continue to reconnect every 30s until the connection is re-established.
• If the connection is refused with a 500 series error, then it will be retried at the next 30s timeout.
• If the connection is refused with a 400 series error then further connections for that tunnel will not be made.

Issue: Do we need an API for the tunnel? As its created from code on the back-end, the app could have additional logic for control over the duration. Does it have API for status, clean shutdown, etc.

Issue: Will additional connections be created for scalability - H2 perf becomes limited after 100 simultaneous requests. How does the front-end know to pair a second back-end connection?

The Front End should keep the WSS connection alive by sending pings every 30s if there is no other traffic. This should be done at the WSS layer.

Moving pieces Copy Chapter

front-end Copy Chapter

The front-end is the proxy that will be called by clients to be able to access resources via the back-end proxy. It will route traffic over a tunnel created using a WSS connection from the back-end proxy. YARP needs a mechanism to know which requests will be routed via the tunnel. This will be achieved by extending the existing cluster concept in YARP - The request to create a tunnel will specify the name of a cluster. Once the tunnel is established, it will be treated as a dynmamically created destination for the named cluster. Routes will not need to be changed, they will point at the cluster, and the tunnels will be used in the same way as destinations.

Tunnel services must be enabled by the proxy server:

builder.Services.AddReverseProxy()
       .LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"));

builder.Services.AddTunnelServices();

The front-end needs to have a tunnel endpoint that the back-end will connect to. The endpoint should be parameterized to include the name of the cluster as part of the URL, and a callback that is used to validate the connection is approved:

• Including the ClusterId in the URL enables the same endpoint mechanism to be used for multiple clusters.
• Using a callback for authentication enables whatever scheme the proxy author(s) wish to use.
  • Trying to encode specific auth schemes will invariably miss a scenario that is needed.
  • The samples that we produce should be based around client certs as it is a good way to manage secure shared secrets.

app.MapReversProxy();
app.MapTunnel("/tunnel/{ClusterId}", async (connectionContext, cluster) => {

    // Use the extensions feature https://github.com/dotnet/yarp/issues/1709 to add auth data for the tunnel
    var tunnelAuth = cluster.Extensions[typeof(TunnelAuth)]; 
    if (!context.Connection.ClientCertificate.Verify()) return false;
    foreach (var c in tunnelAuth.Certs)
    {
        if (c.ThumbPrint == context.Connection.ClientCertificate.Thumbprint) return true;
    }
    return false;
});

The front-end should have configuration for routes that direct to a cluster that is for the tunnel. The cluster must be marked as IsTunnel to enable tunnel capability, and must not include other destinations. The cluster's destinations will be supplied dynamically by back-ends creating tunnel connections.

In the following case it uses the Extensions feature to enable storing thumbprints for client certs that are used to authenticate tunnel connections. The Route will direct all traffic under the path /OnPrem/* to the tunnel.

{
    "ReverseProxy":
    {
        "Routes" : {
            "OnPrem" : {
                "Match" : {
                    "Path" : "/OnPrem/{**any}"
                },
                "ClusterId" : "MyTunnel1"
            }
        },
        "Clusters" : {
            "MyTunnel1" : {
                "IsTunnel" : true,
                "Extensions" : {
                    "TunnelAuth" : {
                        "Certs" : {
                            "name1" : "thumbprint1",
                            "name2" : "thumbprint2"
                        }
                    }
                }
            }
        }
    }
}

To ensure scalability, multiple back-end proxy instances should be able to create tunnel connections for the same cluster. When that happens, the load balancing rules for the cluster should apply, and balance between the active tunnels. Similarly multiple front-ends can be used to reduce the problems with a single point of failure.

back-end Copy Chapter

The back-end instance is the proxy that will reside on the same network as the resources that should be exposed. The back-end will need to be able to connect to those resources, and also be able to create a WebSocket connection to the front-end proxy server(s) via whatever firewalls are between them.

The back-end proxy is configured with routes and destinations that it wishes to expose to the front end. Security is maintained because only URLs matching its routes will be proxyable via it. This prevents attacks at the front-end having arbitrary access to other resources on the back-end network - they need to be explicitly included in the back-end route table.

The outbound connection to the front end needs to be explicitly made for each tunnel that the back-end wishes to create.

builder.Services.AddReverseProxy()
       .LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"));

var url = builder.Configuration["Tunnel:Url"]!; // Eg https://Myfront-end.MyCorp.com/tunnel/MyTunnel1

// Setup additional details for the connection, auth and headers
var tunnelOptions = new TunnelOptions(){
       TunnelClient = new SocketsHttpHandler(),
       ClientCertificates = new X509CertificateCollection { cert }
       AuthCallback = AuthServer;
       };
tunnelOptions.Headers.Add("MyJWTToken", tokenString);

builder.WebHost.UseTunnelTransport(url, tunnelOptions);

The tunnel creation takes an options class that enables the HttpHandler, headers to be set for the tunnel creation to enable authentication flexibility by the app developers. It also should include a callback to enable validation of the server certificate.

The back-end configuration would use the standard routes and cluster definitions.

{
    "ReverseProxy":
    {
        "Routes": {
            "CNCMilling": {
                "Match": {
                    "Path": "/OnPrem/CNCMilling/{**any}"
                },
                "ClusterId": "Milling"
            }
            "3DPrinting" : {
                "Match": {
                    "Path": "/OnPrem/Extrusion/{**any}"
                },
                "ClusterId": "3dPrinting"
            }
        },
        "Clusters": {
            "Milling": {
                "Destinations": {
                    "Bay12" : "https://bay12-efd432/",
                    "Bay15" : "https://bay15-j377d3/"
                }
            }
            "3dPrinting": {
                "Destinations": {
                    "Bay41-controller" : "https://bay41-controller/"
                }
            }
        }
    }
}

In the above case, requests using the paths /OnPrem/CNCMilling/* and /OnPrem/Extrusion/* will be routed by the back-end to their respective services - other paths would result in an error.

Note: Active health checks probably don't make sense to be performed by the front-end against the back-end. Passive health checks will verify the overall condition of the tunnel.

Scalability Copy Chapter

In a large deployment, there needs to be the ability to have multiple front-end and back-end proxies:

• If the front-end receives multiple tunnel connections, then it should treat them as if the cluster has multiple destinations. The cluster can use the load balancing policy to select how it decides to route traffic to the back-end proxies.

Note: The front-end proxy will not be aware of the actual destinations that serve resources - each back-end should have its own cluster definition for the actual destinations, and so can include multiple servers for any route/cluster combination.

• A back-end proxy should be able to create tunnels to multiple front-ends. The tunnels can be to related front-end proxies that are sharing the same load, or to front-ends in different cloud deployments. This enables the Front Ends to be very specific to particular deployments - and have constrained v-Lan configurations in the cloud.

Authentication Copy Chapter

The authentication options for ASP.NET are diverse, and IT departments will likely have their own conditions on what is required to be able to secure a tunnel. So rather than trying to implement the combinatorial matrix of what customers could need, we should use a callback so that the proxy author can decide.

Samples should be created that show best practices using a secure mechanism such as client a certificate.

Issue: Does the back-end need additional mechanisms to validate the connection to the front-end, or is TLS/SNI sufficient?

Security Copy Chapter

The purpose of the tunnel is to simplify service exposure by creating a tunnel through the firewall that enables external requests to be made to destination servers on the back-end network. There are a number of mitigations that reduces the risk of this feature:

• No endpoints are exposed via the firewall - it does not expose any new endpoints that could act as attack vectors. The tunnel is an outbound connection made between the back-end and the front-end.
• Traffic directed via the tunnel will need to have corresponding routes in the Back End configuration. Traffic will only be routed if there is a respective route and cluster configuration. Tunnel traffic can't specify arbitrary URLs that would be directed to a hostname not included in the back-end route table configuration.
• Tunnel connections should only be over HTTPs

Metrics Copy Chapter

? What telemetry and events are needed for this?

Error conditions Copy Chapter

Web Transport Copy Chapter

Web Transport is an interesting future protocol choice for the tunnel.

File: docs/operations/BackportingToPreview.md Copy Chapter

Backporting changes to a preview branch

Backporting changes is very similar to a regular release. Changes are made on the preview branch, the builds are validated and ultimately released.

• Checkout the preview branch

  git checkout release/1.0.0-previewX

• Make and commit any changes

• Push the changes to your own fork and submit a PR against the preview branch

• Once the PR is merged, wait for the internal microsoft-reverse-proxy-official pipeline to produce a build

• Validate the build the same way you would for a regular release docs

• Package Artifacts from this build can be shared to validate the patch. Optionally, the artifacts from the public pipeline can be used

• Continue iterating on the preview branch until satisfied with the validation of the change

• Release the build from the preview branch

• Create a new git tag for the released commit

  While still on the preview branch:

  • git tag v1.0.0-previewX.build.d
  • git push upstream --tags

• Create a new release.

Internal fixes Copy Chapter

Issues with significant security or disclosure concerns need to be fixed privately first. All of this work will happen on the internal Azdo repo and be merged to the public github repo at the time of disclosure.

• Make a separate clone of https://dev.azure.com/dnceng/internal/_git/dotnet-yarp to avoid accidentally pushing to the public repo.
• Create a branch named internal/release/{version being patched} starting from the tagged commit of the prior release.
• Update versioning as needed.
• Create a feature branch, fix the issue, and send a PR using Azdo.
• Release the build from the internal branch.
• Tag the commit and push it to the public repo. Do not push to regular main or release/* branches in the internal mirror.
• Cherry pick the changes to public main as needed.
• Finish the standard release checklist.

File: docs/operations/Branching.md Copy Chapter

Branching Tasks

When we are ready to branch our code, we first need to create the branch:

1. In a local clone, run git checkout main and git pull origin main to make sure you have the latest main
2. Run git checkout -b release/1.1.0-previewX where X is the YARP preview number. When releasing a non-preview version, use release/1.1 instead of release/1.1.0 so that the branch can be used for future patches.
3. If you are releasing a non-preview version:
   • Set the PreReleaseVersionLabel in eng/Versions.props to rtw.
   • Run build.cmd -pack
   • Verify that the packages in artifacts/packages/Debug/Shipping do not have a suffix after the intended version. For example the name should be Yarp.ReverseProxy.1.1.0.nupkg and not Yarp.ReverseProxy.1.1.0-dev.nupkg.
4. Run git push origin release/1.1.0-previewX to push the branch to the server.

Update branding in main:

1. Edit the file eng/Versions.props
2. Set PreReleaseVersionLabel to preview.X (where X is the next preview number)
3. Send a PR and merge it ASAP (auto-merge is your friend).

Update the runtimes and SDKs in global.json in main:

Check that the global.json includes the latest 8.0 runtime versions from the .NET 8.0 download page, and 9.0 from the .NET 9.0 download page.

File: docs/operations/DependencyFlow.md Copy Chapter

Dependency Flow

Dependency Flow Overview Copy Chapter

For full documentation on Arcade, Maestro and darc, see the Arcade documentation

We use the .NET Engineering System (Arcade) to build this repo. Part of the engineering system is a service called "Maestro" which manages dependency flow between repositories. When one repository finishes building, it can automatically publish it's build to a Maestro "Channel". Other repos can subscribe to that channel to receive updated builds. Maestro will automatically open a PR to update dependencies in repositories that are subscribed to changes in dependent repositories.

Maestro can be queried and controlled using the darc command line tool. To use darc you will need to be a member of the dotnet/arcade-contrib GitHub Team. To set up darc:

1. Run .\eng\common\darc-init.ps1 to install the global tool.
2. Once installed, run darc authenticate and follow the instructions.

Running darc with no args will show a list of commands. The darc help [command] command will give you help on a specific command.

Repositories can be configured to publish builds automatically to a certain channel, based on the branch.
To see the current mappings for a repository, you can run darc get-default-channels --source-repo [repo], where [repo] is any substring that matches a full GitHub URL for a repo in the system. The easiest way to use [repo] is to just specify the [owner]/[name] form for a repo. For example:

> darc get-default-channels --source-repo dotnet/aspnetcore
(3796) https://github.com/dotnet/aspnetcore @ release/6.0 -> .NET 6
(5027) https://github.com/dotnet/aspnetcore @ release/8.0 -> .NET 8
(5731) https://github.com/dotnet/aspnetcore @ release/9.0 -> .NET 9
(5732) https://github.com/dotnet/aspnetcore @ main -> .NET 10
(6050) https://github.com/dotnet/aspnetcore @ release/10.0-preview1 -> .NET 10 Preview 1

Subscriptions are managed using the get-subscriptions, add-subscription and update-subscription commands. You can view all subscriptions in the system by running darc get-subscription. You can also filter subscriptions by the source and target using the --source-repo [repo] and --target-repo [repo] arguments. For example, to see everything that dotnet/yarp is subscribed to:

> darc get-subscriptions --target-repo dotnet/yarp
https://github.com/dotnet/arcade (.NET Eng - Latest) ==> 'https://github.com/dotnet/yarp' ('main')
  - Id: 1751e896-c0f1-4247-3909-08d8c8762e9e
  - Update Frequency: EveryWeek
  - Enabled: True
  - Batchable: False
  - PR Failure Notification tags:
  - Source-enabled: False
  - Merge Policies:
    Standard
https://github.com/dotnet/arcade (.NET Eng - Latest) ==> 'https://github.com/dotnet/yarp' ('release/2.2')
  - Id: ebd75d9f-8988-4f50-bd1d-83dfc79fb7ba
  - Update Frequency: EveryWeek
  - Enabled: True
  - Batchable: False
  - PR Failure Notification tags:
  - Source-enabled: False
  - Merge Policies:
    Standard

To add a new subscription, run darc add-subscription with no arguments. An editor window will open with a TODO script like this:

Channel: <required>
Source Repository URL: <required>
Target Repository URL: <required>
Target Branch: <required>
Update Frequency: <'none', 'everyDay', 'everyBuild', 'twiceDaily', 'everyWeek'>
Batchable: False
Merge Policies: []

A number of comments will also be present, describing available values and what they do. Fill these fields in, for example:

Channel: .NET Eng - Latest
Source Repository URL: https://github.com/dotnet/arcade
Target Repository URL: https://github.com/dotnet/yarp
Target Branch: release/42
Update Frequency: EveryWeek
Batchable: False
Merge Policies:
- Name: Standard

Save and exit the editor and the subscription will be created.

Similarly, you can edit an existing subscription by using darc update-subscription --id [ID] (get the [ID] value from get-subscriptions). This will open the same TODO script, but with the current values filled in. Just update them, then save and exit to update.

Prerequisites Copy Chapter

• Properly configured darc global tool, including having run darc authenticate.

When we are ready to branch Copy Chapter

Follow the initial branching steps from Branching.md.

YARP uses darc to consume up-to-date Arcade bits every week.
Set up dependency flow for the new branch:

1. Run darc add-subscription
2. Fill in the template that opens in your editor as follows:
   • Channel = .NET Eng - Latest
   • Source Repository URL = https://github.com/dotnet/arcade
   • Target Repository URL = https://github.com/dotnet/yarp
   • Target Branch = release/X (where X is the YARP release version)
   • Update Frequency = EveryWeek
   • Merge Policies is a multiline value, it should look like this:
     yaml Copy

     Merge Policies:
     - Name: Standard
       Properties: {}

3. Save and close the editor window.

File: docs/operations/README.md Copy Chapter

Operations Playbook

This documentation is primarily for project maintainers, though contributors are welcome to read and learn about our process!

This section has docs on various operations and tasks to be performed in the repo.

• Branching
• Releasing
• Backporting changes onto a preview branch
• Dependency flow

File: docs/operations/Release.md Copy Chapter

Releasing YARP

This document provides a guide on how to release a preview of YARP.

To keep track of the process, open a release checklist issue.

Versioning Copy Chapter

Ensure the eng/Versions.props file has the expected versions and pre-release labels. For a final release set PreReleaseVersionLabel to rtw.

Ensure there's a release branch created. Copy Chapter

See Branching:

• Make the next preview branch.
• Update the branding in main.
• Update the global.json runtime and SDK versions in main.

Identify the Final Build Copy Chapter

First, identify the final build of the dotnet-yarp-official Azure Pipeline (on dnceng/internal). The final build will be the latest successful build in the relevant release/x branch. Use the "Branches" tab on Azure DevOps to help identify it. If the branch hasn't been mirrored yet (see dotnet-mirror-dnceng pipeline) and there are no outstanding changesets in the branch, the build of the corresponding commit from the main branch can be used.

Once you've identified that build, click in to the build details.

Validate the Final Build Copy Chapter

At this point, you can perform any validation that makes sense. At a minimum, we should validate that the sample can run with the candidate packages. You can download the final build using the "Artifacts" which can be accessed under "Related" in the header:

The packages can be accessed from the PackageArtifacts artifact:

Consume .nupkg Copy Section

• Visual Studio: Place it in a local folder and add that folder as a nuget feed in Visual Studio.
• Command Line: dotnet nuget add source <directory> -n local

Walk through the Getting Started instructions and update them in the release branch as needed.

Also validate any major new scenarios this release and their associated docs.

Release the build Copy Chapter

Once validation has been completed, it's time to release.

Go to the dotnet-yarp-release pipeline and select "Run Pipeline".

Under "Resources", select the pipeline run that you've validated artifacts for.

Triple-check the version numbers of the packages in the artifacts against whatever validation was done at this point.

Select "Run". Unless you're a release approver, you're done here!

Approve the release Copy Chapter

The Azure Pipeline will send an email to all the release approvers asking one of them to approve the release.

Click "Review Manual Validation", or navigate to the release pipeline directly in Azure DevOps. You'll see that the stage is "Pending Approval"

Enter a comment such as "release for preview X" approve finalize the release.
After approving, packages will be published automatically. It is possible to cancel the pipeline, but it might be too late. See "Troubleshooting" below.

The packages will be pushed and when the "NuGet.org" stage turns green, the packages are published!

Note: NuGet publishing is quick, but there is a background indexing process that can mean it will take up to several hours for packages to become available

Tag the commit Copy Chapter

Create and push a git tag for the commit associated with the final build (not necessarily the HEAD of the current release branch). See prior tags for the preferred format. Use a lightweight tag, not annotated.

git tag v1.0.0-previewX

Push the tag change to the upstream repo (not your fork)

git push upstream v1.0.0-previewX

Draft release notes Copy Chapter

Create a draft release at https://github.com/dotnet/yarp/releases using the new tag. See prior releases for the recommended content and format.

Publish the release notes Copy Chapter

Publish the draft release notes. These should be referencing the latest docs, packages, etc..

Close the old milestone Copy Chapter

It should be empty now. If it's not, move the outstanding issues to the next one.

Announce on social media Copy Chapter

David Fowler has a lot of twitter followers interested in YARP. Tweet a link to the release notes and let him retweet it.

Set the preview branch to protected Copy Chapter

This is to avoid accidental pushes to/deletions of the preview branch.

Delete the previous preview branch Copy Chapter

There should only be one preview branch on the repo after this point.

Request source code archival Copy Chapter

1. Go to the internal https://dpsopsrequestforms.azurewebsites.net portal
2. Select "Source Code Archival"
3. Proceed through steps 1,2 till the step 3 (all prerequisites are already fulfilled)
4. Fill in the required fields on the steps 3, 4, 5. Please, see the recommended values below.
5. At the last step, check all the info and and submit the request
6. Go to "My Request" tab and wait for a new ticket to appear on the list
7. Copy the ticket link from "Ticket ID" column which will look like https://prod.******
8. Replace the prod word to portal
9. Navigate to the fixed link and check that the ticket is actually created
10. That's all the actions needed to be done immediately. Afterwards, periodically track the ticket progress. It might take many hours.
11. [Offline] Wait for the archival completion report to arrive. Check that the size and number of archived files match the YARP repo.

Recommended fields' values for archival request form Copy Section

Troubleshooting Copy Chapter

Authentication Errors Copy Section

The pipeline is authenticated via a "Service Connection" in Azure DevOps. If there are authentication errors, it's likely the API key is invalid. Follow these steps to update the API key:

1. Go to NuGet.org, log in with an account associated with an @microsoft.com address that has access to the dotnetframework organization.
2. Generate a new API key with "dotnetframework" as the Package Owner and "*" as the Package "glob".
3. Copy that API key and fill it in to the "nuget.org (dotnetframework organization)" Service Connection in Azure DevOps.

In the event you don't have access, contact [email protected] for guidance.

Accidental Overpublish Copy Section

In the event you overpublish (publish a package that wasn't intended to be released), you should "unlist" the package on NuGet. It is not possible to delete packages on NuGet.org, by design, but you can remove them from search results. Users who reference the version you published directly will still be able to download it, but it won't show up in search queries or non-version-specific actions (like installing the latest).

1. Go to NuGet.org, log in with an account associated with an @microsoft.com address that has access to the dotnetframework organization.
2. Go to the package page and click "Manage package" on the "Info" sidebar on the right.
3. Expand "Listing"
4. Select the version that was accidentally published
5. Uncheck the "List in search results" box
6. Click "Save"

Package was rejected Copy Section

NuGet.org has special criteria for all packages starting Microsoft.. If the package is rejected for not meeting one of those criteria, go to the NuGet @ Microsoft page for more information on required criteria and guidance for how to configure the package appropriately.

2. Official Technical Reference & Guides (dotnet/docs) Copy Chapter

.NET Docs

This repository contains the conceptual documentation for .NET. The .NET documentation site is built from multiple repositories:

• .NET docs
• ASP.NET Core docs
• .NET Desktop workloads docs
• .NET MAUI docs
• Entity Framework docs
• Community toolkit

API reference documentation is published from the following repositories:

• .NET API reference
• .NET MAUI API reference
• Android API reference
• Entity Framework 6/Core API reference
• Roslyn API reference
• Community toolkit API reference

These API reference doc repositories are public. Only some accept issues and pull requests. Others are pass-through repositories because API reference is generated directly from the /// in the product source. Some folders in dotnet-api-docs use the product repository as the source of truth.

The C# language specification documentation comes from the following two repositories:

• C# language design
• C# specification - draft

Our team's tasks are tracked in our project boards. You'll see monthly sprint projects, along with long-running projects for major documentation updates. The projects contain documentation issues across the repositories that build .NET docs. Issues are tracked in the relevant repositories. We have a large community using these resources. We make our best effort to respond to issues in a timely fashion. To create a new issue, click the "Open a documentation issue" button at the bottom of any of our published docs, or choose one of the available templates. The control at the bottom of each article automatically routes you to the correct repo and fills in some relevant information based on the article.

:purple_heart: Contribute Copy Chapter

We welcome contributions to help us improve and complete the .NET docs. This is a very large repo, covering a large area. If this is your first visit, see the Contributor guide for information on working with us. Look for issues labeled help wanted :label: for ideas to get started.

Before submitting a PR with 3rd party dependencies, see our policy on 3rd party dependencies.

We work to merge or close PRs in a timely fashion. We regularly review and approve PRs. We encourage contributors to respond to comments in a similar timely fashion. Many times, reviews identify small changes such as spelling or grammar issues, or word choice. Maintainers can accept those suggestions and merge the PR. In other situations, maintainers ask for more significant changes before a PR is ready to merge. We expect contributors to make those changes. Maintainers will help by answering questions or pointing to other resources, if needed.

We consider PRs to be abandoned when they've had no activity (either commits or discussion) in the past 30 days. Team members will close PRs that have been abandoned. Contributors can reopen and continue to work.

:bookmark_tabs: Code of conduct Copy Chapter

This project has adopted the code of conduct defined by the Contributor Covenant
to clarify expected behavior in our community. For more information, see the .NET Foundation: Code of Conduct.

:octocat: GitHub Action workflows Copy Chapter

• : Adds a comment to PRs that were not automated, but rather manually created that target the live branch.
• : Automatically approves and auto-merges PRs originating from the dependabbot[bot].
• : Creates a PR to generate the "What's new" article on the first of every month.
• : The current status for the entire repositories Markdown linter status.
• : Runs various Markdown verifications, beyond the linter, such as ensuring links and redirects are valid.
• : Builds the site for the PR in context, and verifies the build reporting either, success, warnings, or error.
• : Custom .NET build validation, locates code impacted by a PR, and builds.
• : Runs monthly, creating issues on projects that target .NET versions that are out of support.
• : Automatically updates the dependabot configuration weekly, but only if required.
• : Automatically synchronizes issues with Quest (Azure DevOps).
• : Manual bulk import of issues into Quest (Azure DevOps).