swift-nio Documentation

File: README.md Copy Chapter

SwiftNIO

SwiftNIO is a cross-platform asynchronous event-driven network application framework
for rapid development of maintainable high performance protocol servers & clients.

It's like Netty, but written for Swift.

Repository organization Copy Section

The SwiftNIO project is split across multiple repositories:

Within this repository we have a number of products that provide different functionality. This package contains the following products:

• NIO. This is an umbrella module exporting NIOCore, NIOEmbedded and NIOPosix.
• NIOCore. This provides the core abstractions and types for using SwiftNIO (see "Conceptual Overview" for more details). Most NIO extension projects that provide things like new EventLoops and Channels or new protocol implementations should only need to depend on NIOCore.
• NIOPosix. This provides the primary [EventLoopGroup], EventLoop, and Channels for use on POSIX-based systems. This is our high performance core I/O layer. In general, this should only be imported by projects that plan to do some actual I/O, such as high-level protocol implementations or applications.
• NIOEmbedded. This provides EmbeddedChannel and EmbeddedEventLoop, implementations of the NIOCore abstractions that provide fine-grained control over their execution. These are most often used for testing, but can also be used to drive protocol implementations in a way that is decoupled from networking altogether.
• NIOConcurrencyHelpers. This provides a few low-level concurrency primitives that are used by NIO implementations, such as locks and atomics.
• NIOFoundationCompat. This extends a number of NIO types for better interoperation with Foundation data types. If you are working with Foundation data types such as Data, you should import this.
• NIOTLS. This provides a few common abstraction types for working with multiple TLS implementations. Note that this doesn't provide TLS itself: please investigate swift-nio-ssl and swift-nio-transport-services for concrete implementations.
• NIOHTTP1. This provides a low-level HTTP/1.1 protocol implementation.
• NIOWebSocket. This provides a low-level WebSocket protocol implementation.
• NIOTestUtils. This provides a number of helpers for testing projects that use SwiftNIO.
• _NIOFileSystem. This provides async APIs for interacting with the file system.

Protocol Implementations Copy Section

Below you can find a list of a few protocol implementations that are done with SwiftNIO. This is a non-exhaustive list of protocols that are either part of the SwiftNIO project or are accepted into the SSWG's incubation process. All of the libraries listed below do all of their I/O in a non-blocking fashion using SwiftNIO.

Low-level protocol implementations

Low-level protocol implementations are often a collection of ChannelHandlers that implement a protocol but still require the user to have a good understanding of SwiftNIO. Often, low-level protocol implementations will then be wrapped in high-level libraries with a nicer, more user-friendly API.

High-level implementations

High-level implementations are usually libraries that come with an API that doesn't expose SwiftNIO's ChannelPipeline and can therefore be used with very little (or no) SwiftNIO-specific knowledge. The implementations listed below do still do all of their I/O in SwiftNIO and integrate really well with the SwiftNIO ecosystem.

Supported Versions Copy Section

SwiftNIO 2 Copy Section

This is the current version of SwiftNIO and will be supported for the foreseeable future.

Swift Versions Copy Section

We commit to support the most recently released Swift version and the last two minor releases before that unless this is impossible to do in one codebase.
In addition checks are run against the latest beta release (if any) as well as the nightly Swift builds and the intent is that these should pass.

The minimum Swift version supported by SwiftNIO releases are detailed below:

SwiftNIO 1 Copy Section

SwiftNIO 1 is considered end of life - it is strongly recommended that you move to a newer version. The Core NIO team does not actively work on this version. No new features will be added to this version but PRs which fix bugs or security vulnerabilities will be accepted until the end of May 2022.

If you have a SwiftNIO 1 application or library that you would like to migrate to SwiftNIO 2, please check out the migration guide we prepared for you.

The latest released SwiftNIO 1 version supports Swift 4.0, 4.1, 4.2, and 5.0.

Supported Platforms Copy Section

SwiftNIO aims to support all of the platforms where Swift is supported. Currently, it is developed and tested on macOS and Linux, and is known to support the following operating system versions:

• Ubuntu 18.04+
• macOS 10.9+, iOS 7+; (macOS 10.14+, iOS 12+, tvOS 12+ or watchOS 6+ with swift-nio-transport-services)

SwiftNIO has experimental support on OpenBSD for all SwiftNIO libraries except for _NIOFileSystem, which is not yet supported. You can use all other SwiftNIO libraries on OpenBSD by adding them as dependencies in Package.swift.

Compatibility Copy Section

SwiftNIO follows SemVer 2.0.0 with a separate document declaring SwiftNIO's Public API.

What this means for you is that you should depend on SwiftNIO with a version range that covers everything from the minimum SwiftNIO version you require up to the next major version.
In SwiftPM that can be easily done specifying for example from: "2.0.0" meaning that you support SwiftNIO in every version starting from 2.0.0 up to (excluding) 3.0.0.
SemVer and SwiftNIO's Public API guarantees should result in a working program without having to worry about testing every single version for compatibility.

Conceptual Overview Copy Chapter

SwiftNIO is fundamentally a low-level tool for building high-performance networking applications in Swift. It particularly targets those use-cases where using a "thread-per-connection" model of concurrency is inefficient or untenable. This is a common limitation when building servers that use a large number of relatively low-utilization connections, such as HTTP servers.

To achieve its goals, SwiftNIO extensively uses "non-blocking I/O": hence the name! Non-blocking I/O differs from the more common blocking I/O model because the application does not wait for data to be sent to or received from the network: instead, SwiftNIO asks for the kernel to notify it when I/O operations can be performed without waiting.

SwiftNIO does not aim to provide high-level solutions like, for example, web frameworks do. Instead, SwiftNIO is focused on providing the low-level building blocks for these higher-level applications. When it comes to building a web application, most users will not want to use SwiftNIO directly: instead, they'll want to use one of the many great web frameworks available in the Swift ecosystem. Those web frameworks, however, may choose to use SwiftNIO under the covers to provide their networking support.

The following sections will describe the low-level tools that SwiftNIO provides, and provide a quick overview of how to work with them. If you feel comfortable with these concepts, then you can skip right ahead to the other sections of this README.

Basic Architecture Copy Section

The basic building blocks of SwiftNIO are the following 8 types of objects:

• EventLoopGroup, a protocol, provided by NIOCore.
• EventLoop, a protocol, provided by NIOCore.
• Channel, a protocol, provided by NIOCore.
• ChannelHandler, a protocol, provided by NIOCore.
• Bootstrap, several related structures, provided by NIOCore.
• ByteBuffer, a struct, provided by NIOCore.
• EventLoopFuture, a generic class, provided by NIOCore.
• EventLoopPromise, a generic struct, provided by NIOCore.

All SwiftNIO applications are ultimately constructed of these various components.

EventLoops and EventLoopGroups

The basic I/O primitive of SwiftNIO is the event loop. The event loop is an object that waits for events (usually I/O related events, such as "data received") to happen and then fires some kind of callback when they do. In almost all SwiftNIO applications there will be relatively few event loops: usually only one or two per CPU core the application wants to use. Generally speaking, event loops run for the entire lifetime of your application, spinning in an endless loop dispatching events.

Event loops are gathered together into event loop groups. These groups provide a mechanism to distribute work around the event loops. For example, when listening for inbound connections the listening socket will be registered on one event loop. However, we don't want all connections that are accepted on that listening socket to be registered with the same event loop, as that would potentially overload one event loop while leaving the others empty. For that reason, the event loop group provides the ability to spread load across multiple event loops.

In SwiftNIO today there is one EventLoopGroup implementation, and two EventLoop implementations. For production applications there is the MultiThreadedEventLoopGroup, an EventLoopGroup that creates a number of threads (using the POSIX pthreads library) and places one SelectableEventLoop on each one. The SelectableEventLoop is an event loop that uses a selector (either kqueue or epoll depending on the target system) to manage I/O events from file descriptors and to dispatch work. These EventLoops and EventLoopGroups are provided by the NIOPosix module. Additionally, there is the EmbeddedEventLoop, which is a dummy event loop that is used primarily for testing purposes, provided by the NIOEmbedded module.

EventLoops have a number of important properties. Most vitally, they are the way all work gets done in SwiftNIO applications. In order to ensure thread-safety, any work that wants to be done on almost any of the other objects in SwiftNIO must be dispatched via an EventLoop. EventLoop objects own almost all the other objects in a SwiftNIO application, and understanding their execution model is critical for building high-performance SwiftNIO applications.

Channels, Channel Handlers, Channel Pipelines, and Channel Contexts

While EventLoops are critical to the way SwiftNIO works, most users will not interact with them substantially beyond asking them to create EventLoopPromises and to schedule work. The parts of a SwiftNIO application most users will spend the most time interacting with are Channels and ChannelHandlers.

Almost every file descriptor that a user interacts with in a SwiftNIO program is associated with a single Channel. The Channel owns this file descriptor, and is responsible for managing its lifetime. It is also responsible for processing inbound and outbound events on that file descriptor: whenever the event loop has an event that corresponds to a file descriptor, it will notify the Channel that owns that file descriptor.

Channels by themselves, however, are not useful. After all, it is a rare application that doesn't want to do anything with the data it sends or receives on a socket! So the other important part of the Channel is the ChannelPipeline.

A ChannelPipeline is a sequence of objects, called ChannelHandlers, that process events on a Channel. The ChannelHandlers process these events one after another, in order, mutating and transforming events as they go. This can be thought of as a data processing pipeline; hence the name ChannelPipeline.

All ChannelHandlers are either Inbound or Outbound handlers, or both. Inbound handlers process "inbound" events: events like reading data from a socket, reading socket close, or other kinds of events initiated by remote peers. Outbound handlers process "outbound" events, such as writes, connection attempts, and local socket closes.

Each handler processes the events in order. For example, read events are passed from the front of the pipeline to the back, one handler at a time, while write events are passed from the back of the pipeline to the front. Each handler may, at any time, generate either inbound or outbound events that will be sent to the next handler in whichever direction is appropriate. This allows handlers to split up reads, coalesce writes, delay connection attempts, and generally perform arbitrary transformations of events.

In general, ChannelHandlers are designed to be highly re-usable components. This means they tend to be designed to be as small as possible, performing one specific data transformation. This allows handlers to be composed together in novel and flexible ways, which helps with code reuse and encapsulation.

ChannelHandlers are able to keep track of where they are in a ChannelPipeline by using a ChannelHandlerContext. These objects contain references to the previous and next channel handler in the pipeline, ensuring that it is always possible for a ChannelHandler to emit events while it remains in a pipeline.

SwiftNIO ships with many ChannelHandlers built in that provide useful functionality, such as HTTP parsing. In addition, high-performance applications will want to provide as much of their logic as possible in ChannelHandlers, as it helps avoid problems with context switching.

Additionally, SwiftNIO ships with a few Channel implementations. In particular, it ships with ServerSocketChannel, a Channel for sockets that accept inbound connections; SocketChannel, a Channel for TCP connections; and DatagramChannel, a Channel for UDP sockets. All of these are provided by the NIOPosix module. It also provides EmbeddedChannel, a Channel primarily used for testing, provided by the NIOEmbedded module.

A Note on Blocking

One of the important notes about ChannelPipelines is that they are thread-safe. This is very important for writing SwiftNIO applications, as it allows you to write much simpler ChannelHandlers in the knowledge that they will not require synchronization.

However, this is achieved by dispatching all code on the ChannelPipeline on the same thread as the EventLoop. This means that, as a general rule, ChannelHandlers must not call blocking code without dispatching it to a background thread. If a ChannelHandler blocks for any reason, all Channels attached to the parent EventLoop will be unable to progress until the blocking call completes.

This is a common concern while writing SwiftNIO applications. If it is useful to write code in a blocking style, it is highly recommended that you dispatch work to a different thread when you're done with it in your pipeline.

Bootstrap

While it is possible to configure and register Channels with EventLoops directly, it is generally more useful to have a higher-level abstraction to handle this work.

For this reason, SwiftNIO ships a number of Bootstrap objects whose purpose is to streamline the creation of channels. Some Bootstrap objects also provide other functionality, such as support for Happy Eyeballs for making TCP connection attempts.

Currently SwiftNIO ships with three Bootstrap objects in the NIOPosix module: ServerBootstrap, for bootstrapping listening channels; ClientBootstrap, for bootstrapping client TCP channels; and DatagramBootstrap for bootstrapping UDP channels.

ByteBuffer

The majority of the work in a SwiftNIO application involves shuffling buffers of bytes around. At the very least, data is sent and received to and from the network in the form of buffers of bytes. For this reason it's very important to have a high-performance data structure that is optimized for the kind of work SwiftNIO applications perform.

For this reason, SwiftNIO provides ByteBuffer, a fast copy-on-write byte buffer that forms a key building block of most SwiftNIO applications. This type is provided by the NIOCore module.

ByteBuffer provides a number of useful features, and in addition provides a number of hooks to use it in an "unsafe" mode. This turns off bounds checking for improved performance, at the cost of potentially opening your application up to memory correctness problems.

In general, it is highly recommended that you use the ByteBuffer in its safe mode at all times.

For more details on the API of ByteBuffer, please see our API documentation, linked below.

Promises and Futures

One major difference between writing concurrent code and writing synchronous code is that not all actions will complete immediately. For example, when you write data on a channel, it is possible that the event loop will not be able to immediately flush that write out to the network. For this reason, SwiftNIO provides EventLoopPromise<T> and EventLoopFuture<T> to manage operations that complete asynchronously. These types are provided by the NIOCore module.

An EventLoopFuture<T> is essentially a container for the return value of a function that will be populated at some time in the future. Each EventLoopFuture<T> has a corresponding EventLoopPromise<T>, which is the object that the result will be put into. When the promise is succeeded, the future will be fulfilled.

If you had to poll the future to detect when it completed that would be quite inefficient, so EventLoopFuture<T> is designed to have managed callbacks. Essentially, you can chain callbacks off the future that will be executed when a result is available. The EventLoopFuture<T> will even carefully arrange the scheduling to ensure that these callbacks always execute on the event loop that initially created the promise, which helps ensure that you don't need too much synchronization around EventLoopFuture<T> callbacks.

Another important topic for consideration is the difference between how the promise passed to close works as opposed to closeFuture on a Channel. For example, the promise passed into close will succeed after the Channel is closed down but before the ChannelPipeline is completely cleared out. This will allow you to take action on the ChannelPipeline before it is completely cleared out, if needed. If it is desired to wait for the Channel to close down and the ChannelPipeline to be cleared out without any further action, then the better option would be to wait for the closeFuture to succeed.

There are several functions for applying callbacks to EventLoopFuture<T>, depending on how and when you want them to execute. Details of these functions is left to the API documentation.

Design Philosophy Copy Section

SwiftNIO is designed to be a powerful tool for building networked applications and frameworks, but it is not intended to be the perfect solution for all levels of abstraction. SwiftNIO is tightly focused on providing the basic I/O primitives and protocol implementations at low levels of abstraction, leaving more expressive but slower abstractions to the wider community to build. The intention is that SwiftNIO will be a building block for server-side applications, not necessarily the framework those applications will use directly.

Applications that need extremely high performance from their networking stack may choose to use SwiftNIO directly in order to reduce the overhead of their abstractions. These applications should be able to maintain extremely high performance with relatively little maintenance cost. SwiftNIO also focuses on providing useful abstractions for this use-case, such that extremely high performance network servers can be built directly.

The core SwiftNIO repository will contain a few extremely important protocol implementations, such as HTTP, directly in tree. However, we believe that most protocol implementations should be decoupled from the release cycle of the underlying networking stack, as the release cadence is likely to be very different (either much faster or much slower). For this reason, we actively encourage the community to develop and maintain their protocol implementations out-of-tree. Indeed, some first-party SwiftNIO protocol implementations, including our TLS and HTTP/2 bindings, are developed out-of-tree!

Documentation Copy Chapter

• API documentation

Example Usage Copy Chapter

There are currently several example projects that demonstrate how to use SwiftNIO.

• chat client https://github.com/apple/swift-nio/tree/main/Sources/NIOChatClient
• chat server https://github.com/apple/swift-nio/tree/main/Sources/NIOChatServer
• echo client https://github.com/apple/swift-nio/tree/main/Sources/NIOEchoClient
• echo server https://github.com/apple/swift-nio/tree/main/Sources/NIOEchoServer
• UDP echo client https://github.com/apple/swift-nio/tree/main/Sources/NIOUDPEchoClient
• UDP echo server https://github.com/apple/swift-nio/tree/main/Sources/NIOUDPEchoServer
• HTTP client https://github.com/apple/swift-nio/tree/main/Sources/NIOHTTP1Client
• HTTP server https://github.com/apple/swift-nio/tree/main/Sources/NIOHTTP1Server
• WebSocket client https://github.com/apple/swift-nio/tree/main/Sources/NIOWebSocketClient
• WebSocket server https://github.com/apple/swift-nio/tree/main/Sources/NIOWebSocketServer

To build & run them, run following command, replace TARGET_NAME with the folder name under ./Sources

swift run TARGET_NAME

For example, to run NIOHTTP1Server, run following command:

swift run NIOHTTP1Server

Getting Started Copy Chapter

SwiftNIO primarily uses SwiftPM as its build tool, so we recommend using that as well. If you want to depend on SwiftNIO in your own project, it's as simple as adding a dependencies clause to your Package.swift:

dependencies: [
    .package(url: "https://github.com/apple/swift-nio.git", from: "2.0.0")
]

and then adding the appropriate SwiftNIO module(s) to your target dependencies.
The syntax for adding target dependencies differs slightly between Swift
versions. For example, if you want to depend on the NIOCore, NIOPosix and
NIOHTTP1 modules, specify the following dependencies:

Swift 5.4 and newer (`swift-tools-version:5.4`)

dependencies: [.product(name: "NIOCore", package: "swift-nio"),
               .product(name: "NIOPosix", package: "swift-nio"),
               .product(name: "NIOHTTP1", package: "swift-nio")]

Using Xcode Package support Copy Section

If your project is set up as an Xcode project and you're using Xcode 11+, you can add SwiftNIO as a dependency to your
Xcode project by clicking File -> Swift Packages -> Add Package Dependency. In the upcoming dialog, please enter
https://github.com/apple/swift-nio.git and click Next twice. Finally, select the targets you are planning to use (for
example NIOCore, NIOHTTP1, and NIOFoundationCompat) and click finish. Now will be able to import NIOCore (as well as all
the other targets you have selected) in your project.

To work on SwiftNIO itself, or to investigate some of the demonstration applications, you can clone the repository directly and use SwiftPM to help build it. For example, you can run the following commands to compile and run the example echo server:

swift build
swift test
swift run NIOEchoServer

To verify that it is working, you can use another shell to attempt to connect to it:

echo "Hello SwiftNIO" | nc localhost 9999

If all goes well, you'll see the message echoed back to you.

To work on SwiftNIO in Xcode, you can just open the Package.swift
file in Xcode and use Xcode's support for SwiftPM Packages.

Developing SwiftNIO Copy Chapter

Note: This section is only relevant if you would like to develop SwiftNIO yourself. You can ignore the information here if you just want to use SwiftNIO as a SwiftPM package.

For the most part, SwiftNIO development is as straightforward as any other SwiftPM project. With that said, we do have a few processes that are worth understanding before you contribute. For details, please see CONTRIBUTING.md in this repository.

Prerequisites Copy Section

SwiftNIO's main branch is the development branch for the next releases of SwiftNIO 2, it's Swift 5-only.

To be able to compile and run SwiftNIO and the integration tests, you need to
have a few prerequisites installed on your system.

macOS

• Xcode 11.4 or newer, Xcode 12 recommended.

Linux Copy Section

• Swift 5.7 or newer from swift.org/download. We always recommend to use the latest released version.
• netcat (for integration tests only)
• lsof (for integration tests only)
• shasum (for integration tests only)

Ubuntu 18.04

# install swift tarball from https://swift.org/downloads
apt-get install -y git curl libatomic1 libxml2 netcat-openbsd lsof perl

Fedora 28+ Copy Section

dnf install swift-lang /usr/bin/nc /usr/bin/lsof /usr/bin/shasum

File: docs/advanced-performance-analysis.md Copy Chapter

Advanced performance analysis for CPU-bound programs

Most performance problems can be handily debugged with Instruments, perf, or FlameGraphs. At times however, there are changes in performance that are very hard to understand and sometimes very counter intuitive.

Motivating example (feel free to skip) Copy Chapter

A motivating example is this pull request which is supposed to improve the performance of ByteToMessageDecoders in very specific cases. The expectation would be that most performance tests remain the same and everything that uses a ByteToMessageDecoder either remains the same or becomes a little faster. Of course, all the low-level microbenchmarks that don't even use ByteToMessageDecoders should be totally unaffected by this change.

For the most part that was the case, except for two benchmarks in SwiftNIO's performance test suite for which the results were very unexpected:

This means that for whatever reason, the totally unrelated benchmarks circular_buffer_into_byte_buffer_* got slower by 23% and 14% respectively (measured in wall clock seconds) despite using none of the changed code.

Wall clock time Copy Chapter

To start out, we need to verify if the benchmark results are stable regarding wall clock time. If there is too much variation, then it is likely that too much I/O or allocations etc are involved. If a benchmark is unstable because of I/O or allocations, there are better tools available than what is discussed here, for example strace, Instrument's "Allocations" instrument, or Xcode's Memory Graph Debugger. This document solely discusses CPU-bound and mostly single-threaded performance analysis, ie. you should see one CPU core pegged at 100% for the whole duration of the benchmark.

Executed CPU instructions Copy Chapter

If you have two versions of a program, one faster and one slower and you want to find out what is going on, then often it makes sense to look at the number of CPU instructions executed during the benchmark.

On Linux -- assuming you have access to the PMU (Performance Monitoring Unit) -- this is rather straightforward

perf stat -- ./benchmark arguments

The output will look similar to

Performance counter stats for './my-benchmark args':

            449.96 msec task-clock                #    1.001 CPUs utilized
                36      context-switches          #    0.080 K/sec
                 9      cpu-migrations            #    0.020 K/sec
              2989      page-faults               #    0.007 M/sec
        1736698259      cycles                    #    3.860 GHz
          53504877      stalled-cycles-frontend   #    3.08% frontend cycles idle
        5927885177      instructions              #    3.41  insn per cycle
                                                  #    0.01  stalled cycles per insn
        1422566509      branches                  # 3161.555 M/sec
            375326      branch-misses             #    0.03% of all branches

       0.449450377 seconds time elapsed

       0.442788000 seconds user
       0.008050000 seconds sys

First, we can confirm that it is nicely CPU bound (1.001 CPUs utilized).

We can also see that we executed about 5.9B instructions during this benchmark run. The number of instructions executed itself is not very meaningful but it can be very useful when comparing between versions of the same code. For example, let's assume we changed some code and want to confirm in a benchmark that indeed we have made it faster, then it can be very useful to see that we have also reduced the number of instructions.

With perf you could run perf stat -e instructions -- ./benchmark-before-change and perf stat -e instructions -- ./benchmark-after-change and confirm that:

1. The runtime (time elapsed) has gone down and
2. the number of instructions executed has gone down too.

Note that the time it takes to run an instruction varies wildly so it is in theory possible to reduce the number of instructions but make performance actually worse. An example would be if you traded a lot of simple arithmetic for memory accesses. Said that, in most cases where you reduced the number of instructions significantly, you will also see an improvement in the actual runtime. As a control, the number of instructions per cycle can also be useful as it would go down if you traded faster instructions for slower instructions.

Another place where instruction counts come in very handy is when you see this maybe impossible-sounding situation: the run time varies significantly but the number of executed instructions is the same. The motivating example is exactly this situation: The code executed is the same, confirmed by manually checking the generated assembly for the code run as well as checking that the number of instructions is almost equal. Because manual assembly analysis can easily go wrong, it's advisable to compare the number of executed instructions even if you convinced yourself that the executed code is the same. Of course, there are other options like single-stepping in a debugger but that gets tedious quickly. If the executed code and the inputs are the same, then the number of executed instructions should also be the same (ie. < 0.01% variance).

That begs the question: What can we do if we have the runtime differs despite running the exact same instructions? Assuming the code is CPU-bound, the answer is probably that the CPU cannot execute the instructions at the same velocity because it internally has to wait for something (such as memory, more instructions being decoded, another resource, ...). Many of these situations are referred to as CPU stalls.

CPU stalls Copy Chapter

When ending up in situations where a difference in run time is not clearly reflected in the instructions executed, it may be worth looking into CPU stalls. Modern CPUs may be unable to reach their full speed for a variety of reasons. For example it could be that the instruction decoder can't decode the instructions as fast as they could execute (be retired). Or instructions may have to wait to be executed because the data required is slow to load (cache misses etc).

The easiest way to look into such micro-architectural issues are through the CPU's performance counters and with perf they're quite easy to access.

A good starting point for tracing stalls may be

perf stat -e instructions,stalled-cycles-frontend,uops_executed.stall_cycles,resource_stalls.any,cycle_activity.stalls_mem_any -- ./benchmark

Which instructs perf to collect the following performance counters:

• instructions: The number of CPU instructions executed,
• stalled-cycles-frontend: cycles where nothing happened because instructions cannot be decoded fast enough,
• uops_executed.stall_cycles: the number of cycles where no (micro-)operations were dispatched to be executed on a given core,
• resource_stalls.any: Stalls that occurred because the backend had to wait for resources (eg. memory)
• cycle_activity.stalls_mem_any: execution stalls while memory subsystem has an outstanding load.

which will yield output similar to

Performance counter stats for './fast circular_buffer_into_byte_buffer_1kb':

        5927396603      instructions
          64091849      stalled-cycles-frontend
          38429768      uops_executed.stall_cycles
          13278071      resource_stalls.any
          22148955      cycle_activity.stalls_mem_any

       0.452895518 seconds time elapsed

And just like for the instruction counts, the absolute counts of the stalled cycles are not that important, but they can be very important as a comparison. Going back to the motivating example, where we could see:

fast version (perf stat -e counters... -- ./fast):

Performance counter stats for './fast circular_buffer_into_byte_buffer_1kb':

        5927396603      instructions
          34307940      uops_executed.stall_cycles
          13109779      resource_stalls.any
          18148622      cycle_activity.stalls_mem_any

       0.409785892 seconds time elapsed

slow version (perf stat -e counters... -- ./slow):

Performance counter stats for './slow circular_buffer_into_byte_buffer_1kb':

        5927358294      instructions
         308782310      uops_executed.stall_cycles
          14159547      resource_stalls.any
         288072686      cycle_activity.stalls_mem_any

       0.510635133 seconds time elapsed

What we can immediately see is that resource_stalls.any is pretty much the same in both cases, but uops_executed.stall_cycles and cycle_activity.stalls_mem_any are about an order of magnitude higher for the slow case. Remember, this is with the number of instructions being almost the same.

So in this case, we would have identified that the slowdown comes from us getting a lot of "stall cycles" in the CPU. Those are cycles where the CPU was free to run some (micro-)instructions but could not. Unfortunately, from this picture alone, it is unclear why exactly the CPU couldn't actually run some more (micro-)operations. We’ve only learned a little so far, we need to dig deeper.

One strategy is to run

perf list | less

and in there, search for "stall" (using /stall) and "penalty". This will yield some results of available performance counters that allow you to identify which particular stall it may be. It's worth to know that you can specify a bunch of them in one command line (comma separated). Figuring out which ones to choose is really an exercise in brute force with a little educated guessing, there aren't that many such counters so it won't take you too long. In the motivating example, the culprit turned out to be dsb2mite_switches.penalty_cycles

Performance counter stats for './fast circular_buffer_into_byte_buffer_1kb':

           9943811      dsb2mite_switches.penalty_cycles

       0.408032273 seconds time elapsed

vs

Performance counter stats for './slow circular_buffer_into_byte_buffer_1kb':

         229285609      dsb2mite_switches.penalty_cycles

       0.511033898 seconds time elapsed

The names of these stalls/penalties can be quite opaque and understanding what exactly is going on can be rather hard without a pretty good understanding of how a particular CPU micro-architecture works. Nevertheless, being able to pinpoint why a slowdown happens can be very important as it may give information on what to change in the code.

In case you're wondering what dsb2mite_switches.penalty_cycles is, a web search revealed a nicely written blogpost which discusses something very similar. Bottom line is that modern Intel CPUs have a MITE (Micro-operation Translation Engine) which seems to read the instruction stream in 16 byte chunks. So if the alignment of your code is changed (for example by adding/removing an unrelated function to/from your binary) you may incur a penalty if your instructions aren't aligned by 16 bytes (remember, Intel instructions are variable length). This does sound scary (and for some code it is) but the good news is that this should very rarely matter outside of micro-benchmarks. If your whole run time is doing exactly what that micro-benchmark is doing, you should probably care. But most real-world code does much more and more diverse work than some arbitrary micro-benchmarks. So in this particular example we can safely ignore the "regression" which was just unlucky placement of the hot function in the binary. Nevertheless it was important to understand why the regression happened so we could safely decide it does not matter in this instance.

The bad news: PMU access Copy Chapter

Unfortunately, in some environments like Docker for Mac and other solutions relying on virtualisation it can be hard to get access to the PMU (the CPU's performance measuring unit). Check out this guide for tips on getting perf with PMU support to work in your environment.

File: docs/debugging-allocations.md Copy Chapter

How to debug allocation regressions

Usually, your journey will start by detecting that an allocation test regressed. So let's first start by looking at how to run the allocation counter tests.

Running allocation counter tests Copy Chapter

The allocation counter tests are usually run by NIO's integration test framework which lives in IntegrationTests/. In there, you first want to identify the test suite and test case that runs all the allocation counter tests. For the swift-nio repository that is

IntegrationTests/tests_04_performance/test_01_allocation_counts.sh

Next to the test case (test_01_allocation_counts.sh) that runs the allocation counter tests should be a resource directory which contains the individual allocation counter tests. For swift-nio that is

IntegrationTests/tests_04_performance/test_01_resources

From here on, this document expects that you are within this directory, so you might want to start with

cd IntegrationTests/tests_04_performance/test_01_resources/

To run all the allocation tests, simply invoke the runner script in there

./run-nio-alloc-counter-tests.sh

The invocation of this script will take a while because it has to build all of SwiftNIO in release mode first, then it compiles the integration tests and runs them all, multiple times...

Understanding the output of the allocation counter tests Copy Section

The output of this script will look something like

- /Users/johannes/devel/swift-nio/IntegrationTests/tests_04_performance/test_01_resources/test_future_lots_of_callbacks.swift
test_future_lots_of_callbacks.remaining_allocations: 0
test_future_lots_of_callbacks.total_allocations: 75001
test_future_lots_of_callbacks.total_allocated_bytes: 4138056
DEBUG: [["remaining_allocations": 0, "total_allocations": 75001, "total_allocated_bytes": 4138056], ["total_allocations": 75001, "remaining_allocations": 0, "total_allocated_bytes": 4138056], ["total_allocated_bytes": 4138056, "total_allocations": 75001, "remaining_allocations": 0], ["total_allocated_bytes": 4138056, "total_allocations": 75001, "remaining_allocations": 0], ["remaining_allocations": 0, "total_allocations": 75001, "total_allocated_bytes": 4138056], ["total_allocations": 75001, "remaining_allocations": 0, "total_allocated_bytes": 4138056], ["total_allocated_bytes": 4138056, "total_allocations": 75001, "remaining_allocations": 0], ["total_allocations": 75001, "total_allocated_bytes": 4138056, "remaining_allocations": 0], ["remaining_allocations": 0, "total_allocations": 75001, "total_allocated_bytes": 4138056], ["total_allocations": 75001, "total_allocated_bytes": 4138056, "remaining_allocations": 0]]

with this kind of block repeated for each allocation counter test. Let's go through and understand each line. The first line is the name of the specific allocation test. The most relevant part is the file name (test_future_lots_of_callbacks.swift). For this test, we seem to be testing how many allocations futures with many callbacks are doing.

- /Users/johannes/devel/swift-nio/IntegrationTests/tests_04_performance/test_01_resources/test_future_lots_of_callbacks.swift

Next, we see

test_future_lots_of_callbacks.remaining_allocations: 0
test_future_lots_of_callbacks.total_allocations: 75001
test_future_lots_of_callbacks.total_allocated_bytes: 4138056

which are the aggregate values. The first line says remaining_allocations: 0, which means that this allocation test didn't leak, which is good! Then, we see

test_future_lots_of_callbacks.total_allocations: 75001

which means that by the end of the test, we saw 75001 allocations in total. As a rule, we usually run the workload of every allocation test 1000 times. That means you want to divide 75001 by the 1000 runs to get the number of allocations per run. In other words: each iteration of this allocation test allocated 75 times. The extra one allocation is just some noise that we have to ignore: these are usually inserted by operations in the Swift runtime that need to initialize some state. In many (especially multi-threaded test cases) there is some noise, which is why we run the workload repeatedly, which makes it easy to tell the signal from the noise.

Finally, we see

test_future_lots_of_callbacks.total_allocated_bytes: 4138056

Which is the total number of bytes allocated by this test.

Last of all, we see

DEBUG: [["remaining_allocations": 0, "total_allocations": 75001, "total_allocated_bytes": 4138056], ["total_allocations": 75001, "remaining_allocations": 0, "total_allocated_bytes": 4138056], ["total_allocated_bytes": 4138056, "total_allocations": 75001, "remaining_allocations": 0], ["total_allocated_bytes": 4138056, "total_allocations": 75001, "remaining_allocations": 0], ["remaining_allocations": 0, "total_allocations": 75001, "total_allocated_bytes": 4138056], ["total_allocations": 75001, "remaining_allocations": 0, "total_allocated_bytes": 4138056], ["total_allocated_bytes": 4138056, "total_allocations": 75001, "remaining_allocations": 0], ["total_allocations": 75001, "total_allocated_bytes": 4138056, "remaining_allocations": 0], ["remaining_allocations": 0, "total_allocations": 75001, "total_allocated_bytes": 4138056], ["total_allocations": 75001, "total_allocated_bytes": 4138056, "remaining_allocations": 0]]

which is literally just the exact numbers of each of the 10 runs we considered. Note, the lines are prefixed with DEBUG: to make analysis of the output easier, it does not mean we're running in debug mode. If you see a lot of fluctuation between the runs, then the allocation test doesn't allocate deterministically which would be something you want to fix. In the example above, we're totally stable to the last byte of allocated memory, so we have nothing to worry about with this test.

Important notes Copy Chapter

The exact allocation counts are not directly comparable between different operating systems and even different versions of the same operating system. Every new Swift version will also introduce differences (hopefully decreasing over time).

This means that if you get hit by a CI failure, you will have to first reproduce the regression locally by running the allocation counter tests once on main and then again on your failing branch.

Debugging a regression in a particular test Copy Chapter

Because it takes quite a while to run the whole allocation counter test suite, there are some shortcuts you definitely want to know about. First, let's have a look at how we can run just single test.

Running a single test Copy Section

Running a single test is very straightforward, pass the test's .swift file to the runner script. For example:

./run-nio-alloc-counter-tests.sh test_future_lots_of_callbacks.swift

You will notice that it now runs faster but also that it still compiles SwiftNIO for every single time you run the program. So that's still rather slow. If you look closely at the output you will notice that it prints messages such as

Fetching /private/tmp/.nio_alloc_counter_tests_5jMMhk/swift-nio
Fetching /private/tmp/.nio_alloc_counter_tests_5jMMhk/HookedFunctions
Fetching /private/tmp/.nio_alloc_counter_tests_5jMMhk/AtomicCounter

If you change your working directory to common directory above (/private/tmp/.nio_alloc_counter_tests_5jMMhk), then you will find a regular Swift package there and you can just re-run the allocation counter tests using

cd /tmp/.nio_alloc_counter_tests_5jMMhk
swift run -c release

which should run very quickly. But do not forget to pass -c release, otherwise the results will be meaningless.

Debugging with other tools Copy Section

Often, you will want to use external tools such as Instruments or dtrace to analyse why a performance test regressed. Unfortunately, that is a little harder than you might expect because to implement the allocation tests, the allocation counter test framework actually hijacks functions like malloc and free. The counting of allocations happens in the hijacked or 'hooked' functions.
To use an external memory analysis tool, it's best to not do the hooking and fortunately it's very straightforward to switch the hooking off.

To disable the hooking, run the runner script with -- -n followed by the file name of the test you would like to debug. For example

./run-nio-alloc-counter-tests.sh -- -n test_future_lots_of_callbacks.swift

When you run this, you will notice that every allocation counter test now outputs

Fetching /private/tmp/.nio_alloc_counter_tests_0UDNXS/HookedFunctions
Fetching /private/tmp/.nio_alloc_counter_tests_0UDNXS/swift-nio
Fetching /private/tmp/.nio_alloc_counter_tests_0UDNXS/AtomicCounter
[...]
test_future_lots_of_callbacks.total_allocations: 0
test_future_lots_of_callbacks.remaining_allocations: 0
test_future_lots_of_callbacks.total_allocated_bytes: 0
DEBUG: [["total_allocations": 0, "remaining_allocations": 0, "total_allocated_bytes": 0], ["remaining_allocations": 0, "total_allocations": 0, "total_allocated_bytes": 0], ["total_allocations": 0, "total_allocated_bytes": 0, "remaining_allocations": 0], ["total_allocated_bytes": 0, "total_allocations": 0, "remaining_allocations": 0], ["total_allocations": 0, "total_allocated_bytes": 0, "remaining_allocations": 0], ["total_allocated_bytes": 0, "remaining_allocations": 0, "total_allocations": 0], ["total_allocations": 0, "total_allocated_bytes": 0, "remaining_allocations": 0], ["total_allocated_bytes": 0, "remaining_allocations": 0, "total_allocations": 0], ["remaining_allocations": 0, "total_allocated_bytes": 0, "total_allocations": 0], ["total_allocated_bytes": 0, "total_allocations": 0, "remaining_allocations": 0]]

See all the zeroes? Those are all zero because the hooking is disabled. In this mode the output itself is not very useful but you can now switch to using other tools like Intruments or dtrace to debug your allocations.

As explained before, you can change your working directory in the temporary directory (for example cd /private/tmp/.nio_alloc_counter_tests_0UDNXS) and invoke swift run -c release from there to run the allocation counter benchmark. Note that if you disabled hooking (by passing -- -n) it will still print only zeroes.

Debugging with Instruments Copy Section

1. Start Instruments
2. Choose the "Allocations" instrument
3. Tap the executable selector in the toolbar
4. Click "Choose Target..."
5. Tick "Show Hidden Files"
6. Press Cmd+Shift+G
7. Enter the temporary directory path of a non-hooked version of the allocation counter tests (for example /private/tmp/.nio_alloc_counter_tests_0UDNXS)
8. Select .build/release/
9. Find the binary that is named like your allocation counter test (eg. test_lots_of_future_callbacks)
10. Click "Choose"
11. Press the record button

Now you should have a full Instruments allocations trace of the test run. To make sense of what's going on, switch the allocation lifespan from "Created & Persisted" to "Created & Destroyed". After that you should be able to see the type of the allocation and how many times it got allocated. Clicking on the little arrow next to the type name will reveal each individual allocation of this type and the responsible stack trace.

Debugging with `malloc-aggregation.d` or `malloc-aggreation.bt` Copy Section

SwiftNIO ships a dtrace script called dev/malloc-aggregation.d which can give you an aggregation of all allocations by stack trace, and an equivalent bpftrace script called dev/malloc-aggregation.bt. This means that you will see the stack trace of each allocation that happened and how many times this particular stack trace allocated.

Here's an example of how to use the dtrace script:

sudo ~/path/to/swift-nio/dev/malloc-aggregation.d -c .build/release/test_future_lots_of_callbacks

The bpftrace script can be invoked very similarly:

sudo ~/path/to/swift-nio/dev/malloc-aggregation.bt -c .build/release/test_future_lots_of_callbacks

However, for bpftrace it tends to work better if pid-based tracing is used instead. This is because bpftrace has a known limitation where symbolication fails if the process being traced has existed before bpftrace does. This can still be resolved using tools like llvm-symbolizer, but it's trickier.

The output will look something like

[...]
              libsystem_malloc.dylib`malloc
              libswiftCore.dylib`swift_slowAlloc+0x19
              libswiftCore.dylib`swift_allocObject+0x27
              test_1_reqs_1000_conn`SelectableEventLoop.run()+0x231
              test_1_reqs_1000_conn`closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:selectorFactory:initializer:)+0x106
              test_1_reqs_1000_conn`partial apply for closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:selectorFactory:initializer:)+0x2d
              test_1_reqs_1000_conn`thunk for @escaping @callee_guaranteed (@guaranteed NIOThread) -> ()+0xf
              test_1_reqs_1000_conn`partial apply for thunk for @escaping @callee_guaranteed (@guaranteed NIOThread) -> ()+0x11
              test_1_reqs_1000_conn`closure #1 in static ThreadOpsPosix.run(handle:args:detachThread:)+0x1e4
              libsystem_pthread.dylib`_pthread_start+0x94
              libsystem_pthread.dylib`thread_start+0xf
           231000

repeated many times. Each block means that the specific stack trace is responsible for some number of allocations: in the example above, 231000 allocations. Remember that the allocation counter tests run each test 11 times (1 warm up run and 10 measured runs), so this means that in each test run, the above stack trace allocated 21000 times. Also remember that we usually run the workload we want to test 1000 times. That means that in the real world, the above stack trace accounts for 21 out of the total number of allocations (231000 allocations / 11 runs / 1000 executions per run = 21).

When you are looking at allocations in the setup of the test the numbers may be split into one allocation set of 10000 and another of 1000 - for measured code vs warm up run.

The output from malloc-aggreation.d can also be diffed using the stackdiff-dtrace.py script. This can be helpful to track down where additional allocations were made. Right now this strategy doesn't work for bpftrace, but a simple modification to the stackdiff-dtrace script should enable it.

The stdout from malloc-aggregation.d for the two runs to compare should be written to files, and then passsed to stackdiff-dtrace.py:

~/path/to/swift-nio/dev/stackdiff-dtrace.py stack_aggregation.old stack_aggregation.new

Where stack_aggregation.old and stack_aggregation.new are the two outputs from malloc-aggregation.d to compare. The script will output the aggregated stacks which appear:

• only in stack_aggregation.old,
• only in stack_aggregation.new, and
• any differences in numbers for stacks that appear in both.

Similar stack traces will be aggregated into a headline number for matching but where different the full stack traces which made up the collection will be printed indented.

Allocations of less than 1000 in number are ignored - as the tests all run multiples of 1000 runs any values below this are almost certainly from setup in libraries we are using rather than as a consequence of our measured run.

Let's look at some output.

/* Detailed source-code truncated for AI context efficiency. */

These stacks are only slighly different; differing only in the name similar to $s3NIO7Channel_pxq_q0_r1_lyypypAA15EventLoopFutureCyytGIsegnnr_Iegnr_AaB_pypypAEIegnno_Ieggo_TR. They're otherwise the same so we can ignore them and look at more output.

### only in AFTER
[... (previously dismissed stacktrace)]

2000
libsystem_malloc.dylib`malloc
libswiftCore.dylib`swift_slowAlloc
libswiftCore.dylib`swift_allocObject
test_1_reqs_1000_conn`doRequests(group:number:)
test_1_reqs_1000_conn`closure #1 in run(identifier:)
test_1_reqs_1000_conn`partial apply for closure #1 in measureOne #1 (throwAway:_:) in measureAll(_:)
test_1_reqs_1000_conn`partial apply for thunk for @callee_guaranteed () -> (@error @owned Error)
test_1_reqs_1000_conn`thunk for @callee_guaranteed () -> (@error @owned Error)partial apply

    1000
        libsystem_malloc.dylib`malloc
        libswiftCore.dylib`swift_slowAlloc+0x19
        libswiftCore.dylib`swift_allocObject+0x27
        test_1_reqs_1000_conn`doRequests(group:number:)+0x66
[...]
        
    1000
        libsystem_malloc.dylib`malloc
        libswiftCore.dylib`swift_slowAlloc+0x19
        libswiftCore.dylib`swift_allocObject+0x27
        test_1_reqs_1000_conn`doRequests(group:number:)+0x1c4
[...]

### only in BEFORE
[... (previously dismissed stacktrace)]

### different numbers
before: 10000, after: 20000
  AFTER
    10000
        libsystem_malloc.dylib`malloc
        libswiftCore.dylib`swift_slowAlloc+0x19
        libswiftCore.dylib`swift_allocObject+0x27
        test_1_reqs_1000_conn`doRequests(group:number:)+0x66
[...]

Now we see there's another stacktrace in the AFTER section which has no corresponding stacktrace in BEFORE. From the stack we can see it's originating from doRequests(group:number:). In this instance we were working on options applied in this function so it appears we have added allocations. We have also increased the number of allocations which are reported in the different numbers section where stack traces have been paired but with different numbers of allocations.

Debugging with 'heaptrack' Copy Section

In some cases we don't have access to Linux kernel headers, which prevents us from using bpftrace, but there is the heaptrack tool which supports diff analysis of two versions.

Tested on Ubuntu 20.04 (and likely working on other distributions) Install it with:

sudo apt-get install heaptrack

Then using the instructions previously, you need to build the test without hooks such that heaptrack can hook instead, e.g.:

cd IntegrationTests/tests_04_performance/test_01_resources/
./run-nio-alloc-counter-tests.sh -- -n test_1000_autoReadGetAndSet.swift

and then cd to the temp directory from the output (find the tmp directory as above), e.g.:

cd /tmp/.nio_alloc_counter_tests_5jMMhk

Finally, run the binary with heaptrack two times — first we do it for main to get a baseline:

ubuntu@ip-172-31-25-161 /t/.nio_alloc_counter_tests_GRusAy> SWIFTNIO_URING_DISABLED=1 heaptrack .build/x86_64-unknown-linux-gnu/release/test_1000_autoReadGetAndSet
heaptrack output will be written to "/tmp/.nio_alloc_counter_tests_GRusAy/heaptrack.test_1000_autoReadGetAndSet.84341.gz"
starting application, this might take some time...
SWIFTNIO_URING_DISABLED set, disabling liburing.
test_1000_autoReadGetAndSet.total_allocations: 0
test_1000_autoReadGetAndSet.total_allocated_bytes: 0
test_1000_autoReadGetAndSet.remaining_allocations: 0
DEBUG: [["total_allocations": 0, "total_allocated_bytes": 0, "remaining_allocations": 0], ["remaining_allocations": 0, "total_allocated_bytes": 0, "total_allocations": 0], ["total_allocations": 0, "total_allocated_bytes": 0, "remaining_allocations": 0], ["remaining_allocations": 0, "total_allocated_bytes": 0, "total_allocations": 0], ["remaining_allocations": 0, "total_allocated_bytes": 0, "total_allocations": 0], ["remaining_allocations": 0, "total_allocations": 0, "total_allocated_bytes": 0], ["total_allocated_bytes": 0, "total_allocations": 0, "remaining_allocations": 0], ["total_allocated_bytes": 0, "total_allocations": 0, "remaining_allocations": 0], ["remaining_allocations": 0, "total_allocations": 0, "total_allocated_bytes": 0], ["total_allocations": 0, "remaining_allocations": 0, "total_allocated_bytes": 0]]
free(): invalid pointer
Aborted (core dumped)
heaptrack stats:
    allocations:              319347
    leaked allocations:       107
    temporary allocations:    68
Heaptrack finished! Now run the following to investigate the data:

  heaptrack --analyze "/tmp/.nio_alloc_counter_tests_GRusAy/heaptrack.test_1000_autoReadGetAndSet.84341.gz"

Then run it a second time for the feature branch:

ubuntu@ip-172-31-25-161 /t/.nio_alloc_counter_tests_ARusXy> heaptrack .build/x86_64-unknown-linux-gnu/release/test_1000_autoReadGetAndSet
heaptrack output will be written to "/tmp/.nio_alloc_counter_tests_GRusAy/heaptrack.test_1000_autoReadGetAndSet.84372.gz"
starting application, this might take some time...
test_1000_autoReadGetAndSet.remaining_allocations: 0
test_1000_autoReadGetAndSet.total_allocations: 0
test_1000_autoReadGetAndSet.total_allocated_bytes: 0
DEBUG: [["remaining_allocations": 0, "total_allocations": 0, "total_allocated_bytes": 0], ["total_allocated_bytes": 0, "total_allocations": 0, "remaining_allocations": 0], ["total_allocated_bytes": 0, "remaining_allocations": 0, "total_allocations": 0], ["total_allocations": 0, "total_allocated_bytes": 0, "remaining_allocations": 0], ["total_allocations": 0, "remaining_allocations": 0, "total_allocated_bytes": 0], ["remaining_allocations": 0, "total_allocations": 0, "total_allocated_bytes": 0], ["total_allocations": 0, "total_allocated_bytes": 0, "remaining_allocations": 0], ["remaining_allocations": 0, "total_allocated_bytes": 0, "total_allocations": 0], ["remaining_allocations": 0, "total_allocations": 0, "total_allocated_bytes": 0], ["remaining_allocations": 0, "total_allocated_bytes": 0, "total_allocations": 0]]
free(): invalid pointer
Aborted (core dumped)
heaptrack stats:
    allocations:              673989
    leaked allocations:       117
    temporary allocations:    341011
Heaptrack finished! Now run the following to investigate the data:

  heaptrack --analyze "/tmp/.nio_alloc_counter_tests_GRusAy/heaptrack.test_1000_autoReadGetAndSet.84372.gz"
ubuntu@ip-172-31-25-161 /t/.nio_alloc_counter_tests_GRusAy>

Here we could see that we had 673989 allocations in the feature branch version and 319347 in main, so clearly a regression.

Finally, we can analyze the output as a diff from these runs using heaptrack_print:

heaptrack_print -T -d heaptrack.test_1000_autoReadGetAndSet.84341.gz heaptrack.test_1000_autoReadGetAndSet.84372.gz | swift demangle

-T gives us the temporary allocations (as it in this case was not a leak, but a transient alloaction - if you have leaks remove -T).

The output can be quite long, but in this case as we look for transient allocations, scroll down to:

/* Detailed source-code truncated for AI context efficiency. */

And here we could fairly quickly see that the transient extra allocations was due to extra debug printing and querying of environment variables:

NIO.URing.getEnvironmentVar(Swift.String) -> Swift.String?
  at /home/ubuntu/swiftnio/swift-nio/Sources/NIO/LinuxURing.swift:291
  in /tmp/.nio_alloc_counter_tests_GRusAy/.build/x86_64-unknown-linux-gnu/release/test_1000_autoReadGetAndSet
NIO.URing._debugPrint(@autoclosure () -> Swift.String) -> ()

And this code will be removed before final integration of the feature branch, so the diff will go away.

File: docs/io_uring.md Copy Chapter

SwiftNIO io_uring support status [Linux]

Background Copy Chapter

Linux has fairly recently received a new asynchrounus I/O interface — io_uring — that allows high-performance AIO with a minimum amount of system calls. Some background information at LWN and kernel.dk

Current status and supported platforms Copy Chapter

SwiftNIO has basic support for using io_uring as a notification mechanism. io_uring is available on Linux kernels starting from 5.1 and requires support for the POLL operations. SwiftNIO optionally supports multishot polls and poll updates through a configuration option - these are new interfaces that are expected to available from the future release 5.13 (as of this writing). The io_uring implementation has primarily been tested on version 5.12rc.

Performance expectations Copy Chapter

Currently the io_uring implementation does not yet use the underlying interface to its full potential - it is only used for polling and not yet for asynchronous operation of send/recv/read/write. This means performance characteristics currently will be similar to the default epoll() (or slightly worse), but you should test for your specific use case as YMMV. This is expected to change in the future as the io_uring interface is used more extensively.

How to use and requirements Copy Chapter

To enable io_uring, SwiftNIO needs to be built and run on a machine that has the liburing library installed (typically in /usr/local). If built with liburing, it must be available and work on the production machine.

To build SwiftNIO so it uses io_uring instead of epoll, you must specify additional flags when building:
swift build -Xcc -DSWIFTNIO_USE_IO_URING=1 -Xlinker -luring -Xswiftc -DSWIFTNIO_USE_IO_URING

To build it so also use the new poll update / multishot polls (will need kernel 5.13+) you need to do:
swift build -Xcc -DSWIFTNIO_USE_IO_URING=1 -Xlinker -luring -Xswiftc -DSWIFTNIO_USE_IO_URING -Xswiftc -DSWIFTNIO_IO_URING_MULTISHOT

Debug output Copy Chapter

There are currently three debug outputs that can be compiled in (e.g. to verify that liburing is used as expected and for development troubleshooting)
-Xswiftc -DSWIFTNIO_IO_URING_DEBUG_SELECTOR
-Xswiftc -DSWIFTNIO_IO_URING_DEBUG_URING
-Xswiftc -DSWIFTNIO_IO_URING_DEBUG_DUMP_CQE

Roadmap Copy Chapter

The initial support for io_uring is primarily intended to give people interested in it that ability to try it out and provide feedback - in the future the expectation is to add full support for io_uring which is expected to give significant performance improvements on Linux systems supporting it.
The idea is also to add SQPOLL together with fine tune settings such that it can be pinned on a specific CPU.

File: docs/migration-guide-NIO1-to-NIO2.md Copy Chapter

NIO 1 to NIO 2 migration guide

Please take a look at the migration guide included with SwiftNIO 2.29.0, the last version with Swift 5.0 support.

SwiftNIO 2.29.0 includes the _NIO1APIShims module which can help to ease the migration.

File: docs/optimization-tips.md Copy Chapter

Swift NIO Optimization Tips

The following document is a list of tips and advice for writing high performance
code using Swift NIO.

General Optimization Tips Copy Chapter

Although not specific to NIO, the Swift GitHub repository has a
list of tips and tricks for writing high performance Swift code
and is a great place to start.

Wrapping types in `NIOAny` Copy Chapter

Anything passing through a NIO pipeline that is not special-cased by NIOAny
(i.e. is not one of ByteBuffer, FileRegion, IOData or
AddressedEnvelope) needs to have careful attention paid to its
size. If the type isn't one of the aforementioned special cases then it must be
no greater than 24 bytes in size to avoid a heap-allocation each time it is
added to a NIOAny. The size of a type can be checked using
MemoryLayout.size.

If the type being wrapped is a value type then it can be narrowed by placing it
in a copy-on-write box. Alternatively, if the type is an enum with associated
data, then it is possible for the associated data to be boxed by the compiler by
making it an indirect case.

File: docs/public-api.md Copy Chapter

SwiftNIO 2 Public API

The SwiftNIO project (which includes all of github.com/apple/swift-nio*) aims to follow SemVer 2.0.0 which requires projects to declare a public API. This document along with the API documentation declare SwiftNIO's API.

What are acceptable uses of NIO's Public API Copy Chapter

1. Using NIO types, methods, and modules Copy Section

It is acceptable and expected to use any exported type and call any exported method from the NIO* modules unless the type or function name start with an underscore (_). For inits this includes the first parameter's name.

What's the point of this restriction?

If we prefix something with an underscore or put it into one of NIO's internal modules, we can't commit to an API for it and don't expect users to ever use this. Also, there might be different underscored functions/types depending on the platform.

Examples

• ✅ channel.close(promise: nil)
• ❌ channel._channelCore.flush0(), underscored property
• ❌ import CNIOAtomics, module name doesn't start with NIO
• ❌ ByteBuffer(_enableSuperSpecialAllocationMode: true), as the initialiser's first argument is underscored

2. Conforming NIO types to protocols Copy Section

It is acceptable to conform NIO types to protocols you control, i.e. a protocol in your codebase. It is not acceptable to conform any NIO types to protocols in the standard library or outside of your control.

What's the point of this restriction?

NIO in a later version may conform its type to said protocol, or the package that provides the protocol may conform the NIO type to their protocol.

Examples

• ✅ extension EventLoopFuture: MyOwnProtocol { ... }, assuming MyOwnProtocol lives in your codebase
• ❌ extension EventLoopFuture: DebugStringConvertible { ... }, DebugStringConvertible is a standard library protocol

3. Conforming your types to NIO protocols Copy Section

It is acceptable to conform your own types to NIO protocols but it is not acceptable to conform types you do not control to NIO protocols.

What's the point of this restriction?

NIO in a later version might add this conformance or the owner of the type may add the conformance.

Examples

• ✅ extension MyHandler: ChannelHandler { ... }
• ❌ extension Array: EventLoopGroup where Element: EventLoop { ... }, as Array lives in the standard library

4. Extending NIO types Copy Section

It is acceptable to extend NIO types with public methods/properties that either use one of your types as a non-default argument or are prefixed in a way that prevents clashes. Adding private/internal methods/properties is always acceptable.

What's the point of these restrictions?

NIO might later add a member function/property with the same signature as yours.

Examples

• ✅ extension ByteBuffer { public mutating func readMyType(at: Int) -> MyType {...} }, acceptable because it returns a MyType
• ✅ extension ByteBuffer { public mutating func myProjectReadInteger(at: Int) -> Int {...} } , acceptable because prefixed with myProject
• ✅ extension ByteBuffer { internal mutating func readFloat(at: Int) -> Float {...} }, acceptable even though Float is not in your control because the function is internal
• ❌ extension ByteBuffer { public mutating func readBool(at: Int) -> Bool {...} }, because Bool is a standard library type not in your control

Promises the NIO team make Copy Chapter

Before releasing a new major version, i.e. SwiftNIO 3.0.0 we promise the following:

1. No global namespace additions that aren't prefixed Copy Section

In a minor or patch version, all global types/modules/functions that we will add will have NIO*/nio*/CNIO*/cnio*/c_nio* prefixes, this includes C symbols that are visible to others. Be aware that we don't control the exact versions of the system libraries installed on your system so we can't make any guarantees about the symbols exported by the system libraries we depend on.

What does this mean concretely?

• ✅ We might add a new global type called NIOJetPack.
• ✅ We might add a new module called NIOHelpers.
• ✅ We might add a new global functions called nioRun (very unlikely) or c_nio_SSL_write (more likely).
• ❌ We will not add a global new type called Tomato.
• ❌ We will not add a new module called Helpers.
• ❌ We will not add a new global function called SSL_write.
• ❌ We will not add a new system library dependency without a new major version.

Why is this useful to you?

Let's assume your application is composed of three modules:

1. NIO, which in version 2.0.0 does not declare a public struct Tomato
2. Caprese, which declares a public struct Tomato
3. MyApp

Let's also assume that MyApp has both import Caprese and import NIO and it uses Tomato (from Caprese). For example let t = Tomato().

If now in version NIO in version 2.1.0 introduces a public struct Tomato { ... } as well, then now compilation of MyApp's compilation will fail will an ambiguity error.

With the guarantee that in minor versions NIO will only introduce NIO-prefixed types, this can not happen because NIO would call its type NIOTomato rather than Tomato.

C functions are all in a global namespace so even worse issues might hit you if NIO were to just introduce exported C functions with very general names in minor releases.

What's the point of codifying all this? Copy Chapter

We believe that it is important for the ecosystem that all parties can adopt each others' new versions quickly and easily. Therefore we always strongly recommend depending on all the projects in the NIO family which have a major version greater than or equal to 1 up to the next major version. By following the guidelines set out above we should all not run into any trouble even when run with newer versions.

Example:

.package(url: "https://github.com/apple/swift-nio.git", from: "2.0.0"),

Needless to say if you require at least version 2.3.4 you would specify from: "2.3.4". The important part is that you use from: "2.3.4" or .upToNextMajor(from: "2.3.4") and not .exact or .upToNextMinor.

What happens if you ignore these guidelines? Copy Chapter

We are trying to be nice people and we ❤️ our users so we will never break anybody just because they didn't perfectly stick to these guidelines. But just ignoring those guidelines might mean rewriting some of your code, debugging random build, or runtime failures that we didn't hit in the pre-release testing. We do have a source compatibility suite to which you can ask to be added and we try not to break you (within reason). But it is impossible for us to test all of our users' projects and we don't want to lose the ability to move fast without breaking things. Certain failures like clashing protocol conformances can have delicate failure modes.

File: docs/public-async-nio-apis.md Copy Chapter

Async NIO bridges

This is a summary of all the new APIs we introduced to make NIO and the various
network protocols work with new async interfaces. The intention of this document
is to quickly outline the incompatibilities in the original API and then show a
holistic view over all the new APIs.

What APIs are needed? Copy Chapter

The first problem that we had to tackle was bridging NIO's Channel to Swift
Concurrency. To do so we introduced two new foundational types - the
NIOAsyncSequenceProducer and the NIOAsyncWriter. Those allow us to bridge
the read and the write side of the Channel while propagating the back-pressure
across the bridge.

On top of those two types, we built out the NIOAsyncChannel which allows users
to bridge a Channel into Swift Concurrency. To do this it inserts two channel
handlers which bridge the read and write side using the
NIOAsyncSequenceProducer and the NIOAsyncWriter.

Next up we had to look at the bootstraps. Here the import part is that the
Channels must be wrapped at the correct timing otherwise there is the
potential that reads might be dropped. This is not problematic for most of the
bootstraps since they call their various channelInitializers and
childChannelInitializers at the right time. However, there was one tricky
bootstrap - ServerBootstrap. The ServerBootstrap multiplexes the incoming
connections and we have to make sure that the wrapping of the child channels
happens at the correct time. Additionally, the new bootstrap APIs must be
able to relay the type information of the configured channels to the
bind/connect methods.

The next thing we had to tackle was networking protocols that dynamically
re-configure the ChannelPipeline. The two examples that we provide
implementations for are HTTP/1 protocol upgrades and Application Protocol
Negotiation (ALPN) via TLS. Similar to the bootstraps we have to ensure that the
type information is retained so that users can correctly identify which
reconfiguration path has been taken.

Lastly, we had to look at how to handle protocols that multiplex, like HTTP/2.
Multiplexing protocols need to expose a typed async interface to consume new
inbound connections/streams and to open new outbound connections/streams where
applicable.

Proposed APIs Copy Chapter

This section contains all the new APIs that we are adding and gives us a holistic
overview to review them.

`NIOAsyncChannel` Copy Section

/* Detailed source-code truncated for AI context efficiency. */

Bootstraps Copy Section

/* Detailed source-code truncated for AI context efficiency. */

HTTPUpgrade Copy Section

/* Detailed source-code truncated for AI context efficiency. */

Websocket Copy Section

/* Detailed source-code truncated for AI context efficiency. */

ALPN Copy Section

/* Detailed source-code truncated for AI context efficiency. */

HTTP/2.0 Copy Section

/* Detailed source-code truncated for AI context efficiency. */

File: docs/workarounds.md Copy Chapter

Workarounds

The following list of PRs and commits were applied in order to work around bugs
or cases where the Swift compiler was unable to sufficiently optimize the code:

• https://github.com/apple/swift-nio/pull/1374
• https://github.com/apple/swift-nio-ssl/pull/176
• https://github.com/apple/swift-nio/pull/1325
• https://github.com/apple/swift-nio/pull/1299
• https://github.com/apple/swift-nio/pull/1252
• https://github.com/apple/swift-nio/pull/494
• https://github.com/apple/swift-nio/pull/420
• https://github.com/apple/swift-nio/commit/abc963cfe1e1d4856c41421c9d53ea778102e9e8
• https://github.com/apple/swift-nio/pull/1814
• https://github.com/apple/swift-nio/pull/1956
• https://github.com/apple/swift-nio/pull/1961
• https://github.com/apple/swift-nio/pull/2046
• https://github.com/apple/swift-nio/pull/3303
• https://github.com/apple/swift-nio/pull/3452
• https://github.com/apple/swift-nio/pull/3454
• https://github.com/apple/swift-nio/pull/3455

File: IntegrationTests/tests_04_performance/test_01_resources/README.md Copy Chapter

Allocation Counting Test

This briefly describes how the allocation counting test works.

How does it work? Copy Chapter

This is possibly the simplest implementation that counts memory allocations (malloc and friends) and frees (mostly free). It just maintains two atomic variables which count the number of mallocs and the number of frees respectively. We run a simple HTTP1 example -- 1000 requests and responses generated by a simple SwiftNIO based client and server -- and then evaluate the number of mallocs and frees. The difference mallocs - frees should be pretty much 0 and the number of mallocs should remain stable (or decrease) across commits. We can't establish a perfect baseline as the exact number of allocations depends on your operating system, libc and Swift version.

How are the functions hooked? Copy Section

Usually in UNIX it's enough to just define a function, for example

void free(void *ptr) { ... }

in the main binary and all modules will use this free function instead of the real one from the libc. For Linux, this is exactly what we're doing, the bootstrap binary defines such a free function in its main.c. On Darwin (macOS/iOS/...) however that is not the case and you need to use dyld's interpose feature. The odd thing is that dyld's interposing only works if it's in a .dylib and not from a binary's main executable. Therefore we need to build a slightly strange SwiftPM package:

• bootstrap: The main executable's main module (written in C) so we can hook the free function on Linux.
• BootstrapSwift: A SwiftPM module (written in Swift) called in from bootstrap which implements the actual SwiftNIO benchmark (and therefore depends on the NIO module).
• HookedFunctions: A separate SwiftPM package that builds a shared library (.so on Linux, .dylib on Darwin) which contains the replacement_malloc, replacement_free, etc functions which just increment an atomic integers representing the number of operations. On Darwin, we use DYLD_INTERPOSE in this module, interposing libc functions with our replacement_ functions. This needs to be a separate SwiftPM package as otherwise its code would just live inside of the bootstrap executable and the dyld interposing feature wouldn't work.
• AtomicCounter: SwiftPM package (written in C) that implements the atomic counters. It needs to be a separate package as both BoostrapSwift (to read the allocation counter) as well as HookedFree (to increment the allocation counter) depend on it.

What benchmark is run? Copy Chapter

We run a single TCP connection over which 1000 HTTP requests are made by a client written in NIO, responded to by a server also written in NIO. We re-run the benchmark 10 times and return the lowest number of allocations that has been made.

Why do I have to set a baseline? Copy Chapter

By default this test should always succeed as it doesn't actually compare the number of allocations to a certain number. The reason is that this number varies ever so slightly between operating systems and Swift versions. At the time of writing on macOS we got roughly 326k allocations and on Linux 322k allocations for 1000 HTTP requests & responses. To set a baseline simply run

export MAX_ALLOCS_ALLOWED_1000_reqs_1_conn=327000

or similar to set the maximum number of allocations allowed. If the benchmark exceeds these allocations the test will fail.