### Api Requests/Actions # Actions API Request Firecracker microVMs can execute actions that can be triggered via `PUT` requests on the `/actions` resource. Details about the required fields can be found in the [swagger definition](../../src/firecracker/swagger/firecracker.yaml). ## InstanceStart The `InstanceStart` action powers on the microVM and starts the guest OS. It does not have a payload. It can only be successfully called once. ### InstanceStart Example ```bash curl --unix-socket ${socket} -i \ -X PUT "http://localhost/actions" \ -d '{ "action_type": "InstanceStart" }' ``` ## FlushMetrics The `FlushMetrics` action flushes the metrics on user demand. ### FlushMetrics Example ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PUT "http://localhost/actions" \ -d '{ "action_type": "FlushMetrics" }' ``` ## [Intel and AMD only] SendCtrlAltDel This action will send the CTRL+ALT+DEL key sequence to the microVM. By convention, this sequence has been used to trigger a soft reboot and, as such, most Linux distributions perform an orderly shutdown and reset upon receiving this keyboard input. Since Firecracker exits on CPU reset, `SendCtrlAltDel` can be used to trigger a clean shutdown of the microVM. For this action, Firecracker emulates a standard AT keyboard, connected via an i8042 controller. Driver support for both these devices needs to be present in the guest OS. For Linux, that means the guest kernel needs `CONFIG_SERIO_I8042` and `CONFIG_KEYBOARD_ATKBD`. > [!NOTE] > > At boot time, the Linux driver for i8042 spends a few tens of milliseconds > probing the device. This can be disabled by using these kernel command line > parameters: > > ```console > i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd > ``` ### SendCtrlAltDel Example ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PUT "http://localhost/actions" \ -d '{ "action_type": "SendCtrlAltDel" }' ``` --- ### Api Requests/Block Caching # Block device caching strategies Firecracker offers the possiblity of choosing the block device caching strategy. Caching strategy affects the path data written from inside the microVM takes to the host persistent storage. ## How it works When installing a block device through a PUT /drives API call, users can choose the caching strategy by inserting a `cache_type` field in the JSON body of the request. The available cache types are: - `Unsafe` - `Writeback` ### Unsafe mode (default) When configuring the block caching strategy to `Unsafe`, the device will not advertise the VirtIO `flush` feature to the guest driver. ### Writeback mode When configuring the block caching strategy to `Writeback`, the device will advertise the VirtIO `flush` feature to the guest driver. If negotiated when activating the device, the guest driver will be able to send flush requests to the device. When the device executes a flush request, it will perform an `fsync` syscall on the backing block file, committing all data in the host page cache to disk. ## Supported use cases The caching strategy should be used in order to make a trade-off: - `Unsafe` - enhances performance as fewer syscalls and IO operations are performed when running workloads - sacrifices data integrity in situations where the host simply loses the contents of the page cache without committing them to the backing storage (such as a power outage) - recommended for use cases with ephemeral storage, such as serverless environments - `Writeback` - ensures that once a flush request was acknowledged by the host, the data is committed to the backing storage - sacrifices performance, from boot time increases to greater emulation-related latencies when running workloads - recommended for use cases with low power environments, such as embedded environments ## How to configure it Example sequence that configures a block device with a caching strategy: ```bash curl --unix-socket ${socket} -i \ -X PUT "http://localhost/drives/dummy" \ -H "accept: application/json" \ -H "Content-Type: application/json" \ -d "{ \"drive_id\": \"dummy\", \"path_on_host\": \"${drive_path}\", \"is_root_device\": false, \"is_read_only\": false, \"cache_type\": \"Writeback\" }" ``` --- ### Api Requests/Block Io Engine # Block device IO engine For all Firecracker versions prior to v1.0.0, the emulated block device uses a synchronous IO engine for executing the device requests, based on blocking system calls. Firecracker 1.0.0 adds support for an asynchronous block device IO engine. > [!WARNING] > > Support is currently in **developer preview**. See > [this section](#developer-preview-status) for more info. The `Async` engine leverages [`io_uring`](https://kernel.dk/io_uring.pdf) for executing requests in an async manner, therefore getting overall higher throughput by taking better advantage of the block device hardware, which typically supports queue depths greater than 1. The block IO engine is configured via the PUT /drives API call (pre-boot only), with the `io_engine` field taking two possible values: - `Sync` (default) - `Async` (in [developer preview](../RELEASE_POLICY.md)) The `Sync` variant is the default, in order to provide backwards compatibility with older Firecracker versions. > [!NOTE] > > [vhost-user block device](./block-vhost-user.md) is another option for block > IO that requires an external backend process. ## Example configuration ```bash curl --unix-socket ${socket} -i \ -X PUT "http://localhost/drives/rootfs" \ -H "accept: application/json" \ -H "Content-Type: application/json" \ -d "{ \"drive_id\": \"rootfs\", \"path_on_host\": \"${drive_path}\", \"is_root_device\": true, \"is_read_only\": false, \"io_engine\": \"Sync\" }" ``` ## Host requirements Firecracker requires a minimum host kernel version of 5.10.51 for the `Async` IO engine. This requirement is based on the availability of the `io_uring` subsystem, as well as a couple of features and bugfixes that were added in newer kernel versions. If a block device is configured with the `Async` io_engine on a host kernel older than 5.10.51, the API call will return a 400 Bad Request, with a suggestive error message. ## Performance considerations The performance is strictly tied to the host kernel version. The gathered data may not be relevant for kernels other than 5.10. Newer supported host kernels (6.1, 6.18) may exhibit different performance characteristics. ### Device creation When using the `Async` variant, there is added latency on device creation (up to ~110 ms), caused by the extra io_uring system calls performed by Firecracker. This translates to higher latencies on either of these operations: - API call duration for block device config - Boot time for VMs started via JSON config files - Snapshot restore time For use-cases where the lowest latency on the aforementioned operations is desired, it is recommended to use the `Sync` IO engine. ### Block IOPS and efficiency The `Async` engine performance potential is showcased when the block device backing files are placed on a physical disk that supports efficient parallel execution of requests, like an NVME drive. It's also recommended to evenly distribute the backing files across the available drives of a host, to limit contention in high-density scenarios. The performance measurements we've done were made on NVME drives, and we've discovered that: For __read__ workloads which operate on data that is not present in the host page cache, the performance improvement for `Async` is about 1.5x-3x in overall efficiency (IOPS per CPU load) and up to 30x in total IOPS. For __write__ workloads, the `Async` engine brings an improvement of about 20-45% in total IOPS but performs worse than the `Sync` engine in total efficiency (IOPS per CPU load). This means that while Firecracker will achieve better performance, it will be at the cost of consuming more CPU for the kernel workers. In this case, the VMM cpu load is also reduced, which should translate into performance increase in hybrid workloads (block+net+vsock). Whether or not using the `Async` engine is a good idea performance-wise depends on the workloads and the amount of spare CPU available on a host. According to our NVME experiments, io_uring will always bring performance improvements (granted that there are enough available CPU resources). It is recommended that users perform some tests with examples of expected workloads and measure the efficiency as (IOPS/CPU load). ## Developer preview status View the [release policy](../RELEASE_POLICY.md) for information about developer preview terminology. The `Async` io_engine is not yet suitable for production use. It will be made available for production once Firecracker has support for a host kernel that implements mitigation mechanisms for the following threats: ### Threat 1: PID exhaustion The number of io_uring kernel workers assigned to one Firecracker block device is upper-bounded by: ``` (1 + NUMA_COUNT * min(size_of_ring, 4 * NUMBER_OF_CPUS) ``` This formula is derived from the 5.10 linux kernel code, while `size_of_ring` is hardcoded to `128` in Firecracker. Depending on the number of microVMs that can concurrently live on a host and the number of block devices configured for each microVM, the kernel PID limit may be reached, resulting in failure to create any new process. Kernels starting with 5.15 expose a configuration option for customising this upper bound. Once possible, we plan on exposing this in the Firecracker drive configuration interface. ### Threat 2: worker thread resource consumption The io_uring kernel workers are spawned in the root cgroup of the system. They don’t inherit the Firecracker cgroup, cannot be moved out of the root cgroup and their names don't contain any information about the microVM's PID. This makes it impossible to attribute a worker to a specific Firecracker VM and limit the CPU and memory consumption of said workers via cgroups. Starting with kernel 5.12 (currently unsupported), the Firecracker cgroup is inherited by the io_uring workers. ### Path to GA We plan on marking the Async engine as production ready once an LTS linux kernel including mitigations for the aforementioned mitigations is released and support for it is added in Firecracker. Read more about Firecracker's [kernel support policy](../kernel-policy.md). --- ### Api Requests/Block Vhost User # Vhost-user block device > [!WARNING] > > Support is currently in **developer preview**. See > [this section](../RELEASE_POLICY.md#developer-preview-features) for more info. As an alternative to [file-backed block device](block-io-engine.md) `Sync` and `Async` engines, Firecracker supports a vhost-user block device. There is a good introduction of how a vhost-user block device works in general at [FOSDEM23](https://archive.fosdem.org/2023/schedule/event/sds_vhost_user_blk). [Vhost-user](https://qemu-project.gitlab.io/qemu/interop/vhost-user.html) is a userspace protocol that allows to delegate Virtio queue processing to another userspace process on the host, as opposed to performing this task within Firecracker's VMM thread. In the vhost-user architecture, the VMM acts as a vhost-user frontend and it is responsible for: - connecting to the backend via a Unix domain socket (UDS) - feature negotiation with the backend and the guest - handling device configuration requests from the guest - sharing sufficient information about the guest memory and Virtio queues with the backend The vhost-user backend receives the information from the frontend and performs handling of IO requests from the guest. The UDS socket is only used for control plane purposes and does not participate in the data plane. Firecracker only implements a vhost-user frontend. Users are free to choose from [existing open source backends](#backends) or implement their own. ## Topology Each vhost-user device connects to its own UDS socket. There is no way for multiple devices to share a single socket, as there is no way to differentiate messages related to devices at the vhost-user protocol level. Each device can be served by a separate backend or a single backend can serve multiple devices. ## Interactions with the backend There are three points when the vhost-user frontend communicates with the backend: 1. Device initialisation. When a vhost-user device is created, Firecracker connects to the corresponding UDS socket and negotiates Virtio and Vhost features with backend and retrieves device configuration. 1. Device activation. When the guest driver finishes setting up the device, Firecracker shares memory tables and Virtio queue information with the backend. As a part of this, Firecracker shares file descriptors for guest's memory regions, as well as file descriptors for queue notifications. 1. Config update. When receving a [`PATCH` request](./patch-block.md#updating-vhost-user-block-devices-after-boot) on a vhost-user backed drive, Firecracker rerequests the device config from the backend in order to make the new config available to the guest. ## Advantages While vhost-user block is considered an optimisation to Firecracker IO, a naive implementation of the backend is not going to improve performance. The major advantage of using a vhost-user device is that the backend can implement custom processing logic. It can use intelligent algorithms to serve block requests, eg by fetching the block device data over the network or using sophisticated readahead logic. In such cases, the performance improvement will be coming from the fact that the custom logic is implemented in the same process that handles Virtio queues, which reduces the number of required context switches. ## Disadvantages In order for the backend to be able to process virtio requests, guest memory needs to be shared by the frontend to the backend. This means, a shared memory mapping is required to back guest memory. When a vhost-user device is configured, Firecracker uses `memfd_create` instead of creating an anonymous private mapping to achieve that. It was observed that page faults to a shared memory mapping take significantly longer (up to 24% in our testing), because Linux memory subsystem has to use atomic memory operations to update page status, which is an expensive operation under specific conditions. We advise users to profile performance on their workloads when considering to use vhost-user devices. ## Other considerations Compared to virtio block device where Firecracker interacts with a drive file on the host, vhost-user block device is handled by the backend directly. Some workloads may benefit from caching and readahead that the host pagecache offers for the backing file. This benefit is not available in vhost-user block case. Users may need to implement internal caching within the backend if they find it appropriate. ## Backends There are a number of open source implementations of a vhost-user backend available for reference that can help developing a custom backend: 1. [Qemu backend](https://github.com/qemu/qemu/tree/master/contrib/vhost-user-blk) 1. [Cloud Hypervisor backend](https://github.com/cloud-hypervisor/cloud-hypervisor/tree/main/vhost_user_block) 1. [crosvm backend](https://github.com/google/crosvm/blob/main/devices/src/virtio/vhost/user/device/block.rs) 1. [SPDK backend](https://github.com/spdk/spdk/blob/master/lib/vhost/vhost_blk.c) ## Security considerations ### Guest memory sharing By design, a vhost-user frontend must share file descriptors of all guest memory regions to the backend. In order to achive that, guest memory is created as a [memfd](https://man7.org/linux/man-pages/man2/memfd_create.2.html) and mapped as `MAP_SHARED`. #### File descriptor in procfs An open `memfd` is reflected in `procfs` as any other open file descriptor: ```shell $ ls -l /proc/{pid}/fd | grep memfd lrwx------ 1 1234 1234 64 Nov 2 13:39 32 -> /memfd:guest_mem (deleted) ``` Any process on the host that has access to this file in `procfs` will be able to map the file descriptor and observe runtime behaviour of the guest. At the moment, Firecracker does not close the `memfd`, because it must remain open until all the configured vhost-user devices have been activated and their info shared with the backends. This kind of tracking is not implemented in Firecracker, but may be implemented in the future. Meanwhile, users need to make sure that the access to the Firecracker's `procfs` tree is restricted to trusted processes on the host. On the backend side, it is advised that the backend closes the guest memory region file descriptors after mapping them into its own address space. #### Resource limit in jailer The Firecracker [jailer](../jailer.md) allows to configure resource limits for the Firecracker process. Specifically, it allows to set the maximum file size. Since `memfd` that is used to back the guest memory is considered a file, the file size resource limit cannot be less than the biggest guest memory region. This does not require any special action from a user, but needs to be taken into consideration. ### Remote code execution in the backend It is recommended to run Firecracker using the [jailer](../jailer.md). Since the vhost-user backend interacts with the guest via a Virtio queue, there is a potential for the guest to exercise issues in the backend codebase to trigger undesired behaviours. Users should consider running their backend in a jailer or applying other adequate security measures to restrict it. > [!NOTE] > > [Firecracker jailer](../jailer.md) is currently only capable of running > Firecracker as the binary. Vhost-user block device users are expected to use > another jailer to run the backend. It is also recommended to use proactive security measures like running a Virtio-level fuzzer in the guest during testing to make sure that the backend correctly handles all possible classes of inputs (including invalid ones) from the guest. ### Rate limiting / cgroups Virtio block device in Firecracker has a [rate limiting capability](../design.md#io-storage-networking-and-rate-limiting). In the vhost-user case, Firecracker does not participate in handling requests from the guest, so rate limiting becomes backend's responsibility. As an additional indirect measure, users can make use of `cgroups` settings (either via Firecracker jailer or independently) in order to restrict host CPU consumption of the guest, which would transitively limit guest's IO activity. ### Protection against defects in the backend code Due to potential defects in the backend (eg mislocating Virtio queues or writes to a wrong location in the guest memory), the guest execution may be affected. It is advised that customers monitor guest's health periodically. Additionally, in order to avoid orhpaned Firecracker processes if the backend crashes, the backend may need to send a signal, such as `SIGBUS`, to the Firecracker process for it to exit as well. ### Backend timeouts In order to correctly handle the case where the Firecracker process exits before it exchanges all the expected data with the backend, the backend may need to implement a timeout for how long it waits for Firecracker to connect and/or to exchange the data via the vhost-user protocol and exit to avoid resource exhaustion. ## Snapshot support At the moment, [snapshotting](../snapshotting) is not supported for microVMs that have vhost-user devices configured. An attempt to take a snapshot of such a microVM will fail. It is planned to add support for that in the future. ## Example configuration Run a vhost-user backend, eg Qemu backend: ```bash vhost-user-blk --socket-path=${backend_socket} --blk-file=${drive_path} ``` Firecracker API request to add a vhost-user block device: ```bash curl --unix-socket ${fc_socket} -i \ -X PUT "http://localhost/drives/scratch" \ -H "accept: application/json" \ -H "Content-Type: application/json" \ -d "{ \"drive_id\": \"scratch\", \"socket\": \"${backend_socket}\", \"is_root_device\": false }" ``` > [!NOTE] > > Unlike Virtio block device, there is no way to configure a `readonly` > vhost-user drive on the Firecracker side. Instead, this configuration belongs > to the backend. Whenever the backend advertises the `VIRTIO_BLK_F_RO` feature, > Firecracker will accept it, and the device will act as readonly. > [!NOTE] > > Whenever a `PUT` request is sent to the `/drives` endpoint for a vhost-user > device with the `id` that already exists, Firecracker will close the existing > connection to the backend and will open a new one. Users may need to restart > their backend if they do so. --- ### Api Requests/Patch Block # Updating block devices after boot ## Updating Virtio block devices after boot Firecracker offers support to update attached block devices after the microVM has been started. This is provided via PATCH /drives API which notifies Firecracker that the underlying block file has been changed on the host. It should be called when the path to the block device is changed or if the file size has been modified. It is important to note that external changes to the block device file do not automatically trigger a notification in Firecracker so the explicit PATCH API call is mandatory. ### How it works The implementation of the PATCH /drives API does not modify the host backing file. It only updates the emulation layer block device properties, path and length and then triggers a virtio device reconfiguration that is handled by the guest driver which will update the size of the raw block device. With that being said, a sequence which performs resizing/altering of the block underlying host file followed by a PATCH /drives API call is not an atomic operation as the guest can also modify the block file via emulation during the sequence, if the raw block device is mounted or accessible. ### Supported use case This feature was designed to work with a cooperative guest in order to effectively simulate hot plug/unplug functionality for block devices. The following guarantees need to be provided: - guest did not mount the device - guest does not read or write from the raw block device `/dev/vdX` during the update sequence Example sequence that configures a microVM with a placeholder drive and then updates it with the real one: ```bash # Create and set up a block device. touch ${ro_drive_path} curl --unix-socket ${socket} -i \ -X PUT "http://localhost/drives/scratch" \ -H "accept: application/json" \ -H "Content-Type: application/json" \ -d "{ \"drive_id\": \"scratch\", \"path_on_host\": \"${ro_drive_path}\", \"is_root_device\": false, \"is_read_only\": true \"rate_limiter\": { \"bandwidth\": { \"size\": 100000, \"one_time_burst\": 4096, \"refill_time\": 150 }, \"ops\": { \"size\": 10, \"refill_time\": 250 } } }" # Finish configuring and start the microVM. Wait for the guest to boot. # Before mounting the block device in the guest: # Use another backing file of different size to effectively resize the # vm block device. touch ${updated_ro_drive_path} truncate --size ${new_size}M ${updated_ro_drive_path} # Create a filesystem in it. mkfs.ext4 ${updated_ro_drive_path} # PATCH the block device to use the new backing file. curl --unix-socket ${socket} -i \ -X PATCH "http://localhost/drives/scratch" \ -H "accept: application/json" \ -H "Content-Type: application/json" \ -d "{ \"drive_id\": \"scratch\", \"path_on_host\": \"${updated_ro_drive_path}\" }" # It's now safe to mount the block device in the guest and use it # with the updated backing file. ``` ### Data integrity and other issues We do not recommend using this feature outside of its supported use case scope. If the required guarantees are not provided, data integrity and potential other issues may arise depending on the actual use case. There are two major aspects that need be considered here: #### Atomicity of the update sequence If the guest has the opportunity to perform I/O against the block device during the update sequence it can either read data while it is changed or can overwrite data already written by a host process. For example a truncate operation can be undone if the guest issues a write for the last sector of the raw block device, or the guest application can become inconsistent or/and can create inconsistency in the block device itself. #### In flight I/O requests If the atomicity of the operation is guaranteed by using methods to make the microVM quiescence during the update sequence (for example pausing the microVM) the guest itself or block device can still become incosistent from in flight I/O requests in the guest that will be executed after it is resumed. ## Updating vhost-user block devices after boot Unlike with Virtio block device, with vhost-user block devices, Firecracker does not interact with the underlying block file directly (the vhost-user backend does). It means that changes to the file are not automatically seen by Firecracker. There is a mechanism in the [vhost-user protocol](https://qemu-project.gitlab.io/qemu/interop/vhost-user.html) for the backend to notify the frontend about changes in the device config via `VHOST_USER_BACKEND_CONFIG_CHANGE_MSG` message. This requires an extra UDS socket connection between the frontend and backend used for backend-originated messages. This mechanism **is not supported** by Firecracker. Instead, Firecracker makes use of the `PATCH /drives` API request to get notified about such changes. Such an API request only includes the required property (`drive_id`), because optional properties are not relevant to vhost-user. Example of a `PATCH` request for a vhost-user drive: ```bash curl --unix-socket ${socket} -i \ -X PATCH "http://localhost/drives/scratch" \ -H "accept: application/json" \ -H "Content-Type: application/json" \ -d "{ \"drive_id\": \"scratch\" }" ``` A `PATCH` request to a vhost-user drive will make Firecracker retrieve the new device config from the backend and send a config change notification to the guest. --- ### Api Requests/Patch Network Interface # Updating A Network Interface After the microVM is started, the rate limiters assigned to a network interface can be updated via a `PATCH /network-interfaces/{id}` API call. E.g. for a network interface created with: ```console PUT /network-interfaces/iface_1 HTTP/1.1 Host: localhost Content-Type: application/json Accept: application/json { "iface_id": "iface_1", "host_dev_name": "fctap1", "guest_mac": "06:00:c0:a8:34:02", "rx_rate_limiter": { "bandwidth": { "size": 1024, "one_time_burst": 1048576, "refill_time": 1000 } }, "tx_rate_limiter": { "bandwidth": { "size": 1024, "one_time_burst": 1048576, "refill_time": 1000 } } } ``` A `PATCH` request can be sent at any future time, to update the rate limiters: ```console PATCH /network-interfaces/iface_1 HTTP/1.1 Host: localhost Content-Type: application/json Accept: application/json { "iface_id": "iface_1", "rx_rate_limiter": { "bandwidth": { "size": 1048576, "refill_time": 1000 }, "ops": { "size": 2000, "refill_time": 1000 } } } ``` The full specification of the data structures available for this call can be found in our [OpenAPI spec](../../src/firecracker/swagger/firecracker.yaml). > [!NOTE] > > The data provided for the update is merged with the existing data. In the > above example, the RX rate limit is updated, but the TX rate limit remains > unchanged. ## Removing Rate Limiting A rate limit can be disabled by providing a 0-sized token bucket. E.g., following the above example, the TX rate limit can be disabled with: ```console PATCH /network-interfaces/iface_1 HTTP/1.1 Host: localhost Content-Type: application/json Accept: application/json { "iface_id": "iface_1", "tx_rate_limiter": { "bandwidth": { "size": 0, "refill_time": 0 }, "ops": { "size": 0, "refill_time": 0 } } } ``` --- ### Cpu Templates/Boot Protocol # Boot protocol register settings Firecracker makes certain modifications to the guest's registers regardless of whether a CPU template is used to comply with the boot protocol. If a CPU template is used the boot protocol settings are performed _after_ the CPU template is applied. That means that if the CPU template configures CPUID bits used in the boot protocol settings, they will be overwritten. See also: [CPUID normalization](cpuid-normalization.md) ## Boot protocol MSRs (x86_64 only) On x86_64, the following MSRs are set to `0`: - MSR_IA32_SYSENTER_CS - MSR_IA32_SYSENTER_ESP - MSR_IA32_SYSENTER_EIP - MSR_STAR - MSR_CSTAR - MSR_KERNEL_GS_BASE - MSR_SYSCALL_MASK - MSR_LSTAR - MSR_IA32_TSC and MSR_IA32_MISC_ENABLE is set to `1`. ## Boot protocol ARM registers (aarch64 only) On aarch64, the following registers are set: - PSTATE to PSR_MODE_EL1h | PSR_A_BIT | PSR_F_BIT | PSR_I_BIT | PSR_D_BIT - PC to kernel load address (vCPU0 only) - X0 to DTB/FDT address (vCPU0 only) --- ### Cpu Templates/Cpu Template Helper # CPU template helper tool The `cpu-template-helper` tool is a program designed to assist users with creating and managing their custom CPU templates. ## Usage The `cpu-template-helper` tool has two sets of commands: template-related commands and fingerprint-related commands. ### Template-related commands #### Dump command This command dumps guest CPU configuration in the custom CPU template JSON format. ``` cpu-template-helper template dump \ --output \ [--template ] \ [--config ] ``` Users can utilize this as an entry point of a custom CPU template creation to comprehend what CPU configuration are exposed to guests. The guest CPU configuration consists of the following entities: - x86_64 - CPUID - MSRs (Model Specific Registers) - aarch64 - ARM registers It retrieves the above entities exposed to a guest by applying the same preboot process as Firecacker and capturing them in the state just before booting a guest. More details about the preboot process can be found [here](boot-protocol.md) and [here](cpuid-normalization.md). > [!NOTE] > > Some MSRs and ARM registers are not included in the output, since they are not > reasonable to modify with CPU templates. The full list of them can be found in > [Appendix](#appendix). > [!NOTE] > > Since the output depends on underlying hardware and software stack (BIOS, CPU, > kernel, Firecracker), it is required to dump guest CPU configuration on each > combination when creating a custom CPU template targeting them all. #### Strip command This command strips identical entries from multiple guest CPU configuration files generated with the dump command. ``` cpu-template-helper template strip \ --paths [..] \ --suffix ``` One practical use case of the CPU template feature is to provide a consistent CPU feature set to guests running on multiple CPU models. When creating a custom CPU template for this purpose, it is efficient to focus on the differences in guest CPU configurations across those CPU models. Given that a dumped guest CPU configuration typically amounts to approximately 1,000 lines, this command considerably narrows down the scope to consider. #### Verify command This command verifies that the given custom CPU template is applied correctly. ``` cpu-template-helper template verify \ --template \ [--config ] ``` Firecracker modifies the guest CPU configuration after the CPU template is applied. Occasionally, due to hardware and/or software limitations, KVM might not set the given configuration. Since Firecracker does not check them at runtime, it is required to ensure that these situations don't happen with their custom CPU templates before deploying it. When a template is specified both through `--template` and in Firecracker configuration file provided via `--config`, the template specified with `--template` takes precedence. > [!NOTE] > > This command does not ensure that the contents of the template are sensible. > Thus, users need to make sure that the template does not have any inconsistent > entries and does not crash guests. ### Fingerprint-related commands #### Dump command This command not only dumps the guest CPU configuration, but also host information that could affect the validity of custom CPU templates. ``` cpu-template-helper fingerprint dump \ --output \ [--template ] \ [--config ] ``` Keeping the underlying hardware and software stack updated is essential for maintaining security and leveraging new technologies. On the other hand, since the guest CPU configuration can vary depending on the infrastructure, updating it could lead to a situation where a custom CPU template loses its validity. In addition, even if values of the guest CPU configuration don't change, its internal behavior or semantics could still change. For instance, a kernel version update may introduce changes to KVM emulation and a microcode update may alter the behavior of CPU instructions. To ensure awareness of these changes, it is strongly recommended to store the fingerprint file at the time of creating a custom CPU template and to continuously compare it with the current one. #### Compare command This command compares two fingerprint files: one was taken at the time of custom CPU template creation and the other is taken currently. ``` cpu-template-helper fingerprint compare \ --prev \ --curr \ --filters [..] ``` By continously comparing fingerprint files, users can ensure they are aware of any changes that could require revising the custom CPU template. However, it is worth noting that not all of these changes necessarily require a revision, and some changes could be inconsequential to the custom CPU template depending on its use case. To provide users with flexibility in comparing fingerprint files based on situations or use cases, the `--filters` option allows users to select which fields to compare. As examples of when to compare fingerprint files: - When bumping the Firecracker version up - When bumping the kernel version up - When applying a microcode update (or launching a new host (e.g. AWS EC2 metal instance)) ## Sample scenario This section gives steps of creating and managing a custom CPU template in a sample scenario where the template is designed to provide a consistent set of CPU features to a heterogeneous fleet consisting of multiple CPU models. ### Custom CPU template creation 1. Run the `cpu-template-helper template dump` command on each CPU model to retrieve guest CPU configuration. 1. Run the `cpu-template-helper template strip` command to remove identical entries across the dumped guest CPU configuration files. 1. Examine the differences of guest CPU configuration in details, determine which CPU features should be presented to guests and draft a custom CPU template. 1. Run the `cpu-template-helper template verify` command to check the created custom CPU template is applied correctly. 1. Conduct thorough testing of the template as needed to ensure that it does not contain any inconsistent entries and does not lead to guest crashes. ### Custom CPU template management 1. Run the `cpu-template-helper fingerprint dump` command on each CPU model at the same time when creating a custom CPU template. 1. Store the dumped fingerprint files together with the custom CPU template. 1. Run the `cpu-template-helper fingerprint dump` command to ensure the template's validity whenever you expect changes to the underlying hardware and software stack. 1. Run the `cpu-template-helper fingerprint compare` command to identify changes of the underlying environment introduced after creating the template. 1. (if changes are detected) Review the identified changes, make necessary revisions to the CPU template, and replace the fingerprint file with the new one. > [!NOTE] > > It is recommended to review the update process of the underlying stack on your > infrastructure. This can help identify points that may require the above > validation check. ## Appendix ### MSRs excluded from guest CPU configuration dump | Register name | Index | | --------------------------------------- | ----------------------- | | MSR_IA32_TSC | 0x00000010 | | MSR_ARCH_PERFMON_PERFCTRn | 0x000000c1 - 0x000000d2 | | MSR_ARCH_PERFMON_EVENTSELn | 0x00000186 - 0x00000197 | | MSR_ARCH_PERFMON_FIXED_CTRn | 0x00000309 - 0x0000030b | | MSR_CORE_PERF_FIXED_CTR_CTRL | 0x0000038d | | MSR_CORE_PERF_GLOBAL_STATUS | 0x0000038e | | MSR_CORE_PERF_GLOBAL_CTRL | 0x0000038f | | MSR_CORE_PERF_GLOBAL_OVF_CTRL | 0x00000390 | | MSR_K7_EVNTSELn | 0xc0010000 - 0xc0010003 | | MSR_K7_PERFCTR0 | 0xc0010004 - 0xc0010007 | | MSR_F15H_PERF_CTLn + MSR_F15H_PERF_CTRn | 0xc0010200 - 0xc001020c | | MSR_IA32_VMX_BASIC | 0x00000480 | | MSR_IA32_VMX_PINBASED_CTLS | 0x00000481 | | MSR_IA32_VMX_PROCBASED_CTLS | 0x00000482 | | MSR_IA32_VMX_EXIT_CTLS | 0x00000483 | | MSR_IA32_VMX_ENTRY_CTLS | 0x00000484 | | MSR_IA32_VMX_MISC | 0x00000485 | | MSR_IA32_VMX_CR0_FIXEDn | 0x00000486 - 0x00000487 | | MSR_IA32_VMX_CR4_FIXEDn | 0x00000488 - 0x00000489 | | MSR_IA32_VMX_VMCS_ENUM | 0x0000048a | | MSR_IA32_VMX_PROCBASED_CTLS2 | 0x0000048b | | MSR_IA32_VMX_EPT_VPID_CAP | 0x0000048c | | MSR_IA32_VMX_TRUE_PINBASED_CTLS | 0x0000048d | | MSR_IA32_VMX_TRUE_PROCBASED_CTLS | 0x0000048e | | MSR_IA32_VMX_TRUE_EXIT_CTLS | 0x0000048f | | MSR_IA32_VMX_TRUE_ENTRY_CTLS | 0x00000490 | | MSR_IA32_VMX_VMFUNC | 0x00000491 | | MSR_IA32_MCG_STATUS | 0x0000017a | | MSR_IA32_MCG_CTL | 0x0000017b | | MSR_IA32_MCG_EXT_CTL | 0x000004d0 | | HV_X64_MSR_GUEST_OS_ID | 0x40000000 | | HV_X64_MSR_HYPERCALL | 0x40000001 | | HV_X64_MSR_VP_INDEX | 0x40000002 | | HV_X64_MSR_RESET | 0x40000003 | | HV_X64_MSR_VP_RUNTIME | 0x40000010 | | HV_X64_MSR_VP_ASSIST_PAGE | 0x40000073 | | HV_X64_MSR_SCONTROL | 0x40000080 | | HV_X64_MSR_STIMER0_CONFIG | 0x400000b0 | | HV_X64_MSR_CRASH_Pn | 0x40000100 - 0x40000104 | | HV_X64_MSR_CRASH_CTL | 0x40000105 | | HV_X64_MSR_REENLIGHTENMENT_CONTROL | 0x40000106 | | HV_X64_MSR_TSC_EMULATION_CONTROL | 0x40000107 | | HV_X64_MSR_TSC_EMULATION_STATUS | 0x40000108 | | HV_X64_MSR_SYNDBG_CONTROL | 0x400000f1 | | HV_X64_MSR_SYNDBG_STATUS | 0x400000f2 | | HV_X64_MSR_SYNDBG_SEND_BUFFER | 0x400000f3 | | HV_X64_MSR_SYNDBG_RECV_BUFFER | 0x400000f4 | | HV_X64_MSR_SYNDBG_PENDING_BUFFER | 0x400000f5 | | HV_X64_MSR_SYNDBG_OPTIONS | 0x400000ff | | HV_X64_MSR_TSC_INVARIANT_CONTROL | 0x40000118 | ### ARM registers excluded from guest CPU configuration dump | Register name | ID | | --------------------- | ------------------ | | Program Counter | 0x6030000000100040 | | KVM_REG_ARM_TIMER_CNT | 0x603000000013df1a | --- ### Cpu Templates/Cpu Templates # CPU templates Firecracker allows users to customise how the vCPUs are represented to the guest software by changing the following configuration: - CPUID (x86_64 only) - MSRs (Model Specific Registers, x86_64 only) - ARM registers (aarch64 only) - vCPU features (aarch64 only) - KVM capabilities (both x86_64 and aarch64) A combination of the changes to the above entities is called a CPU template. The functionality can be used when a user wants to mask a feature from the guest. A real world use case for this is representing a heterogeneous fleet (a fleet consisting of multiple CPU models) as a homogeneous fleet, so the guests will experience a consistent feature set supported by the host. > [!NOTE] > > Representing one CPU vendor as another CPU vendor is not supported. > [!NOTE] > > CPU templates shall not be used as a security protection against malicious > guests. Disabling a feature in a CPU template does not generally make it > completely unavailable to the guest. For example, disabling a feature related > to an instruction set will indicate to the guest that the feature is not > supported, but the guest may still be able to execute corresponding > instructions if it does not obey the feature bit. Firecracker supports two types of CPU templates: - Static CPU templates - a set of built-in CPU templates for users to choose from - Custom CPU templates - users can create their own CPU templates in json format and pass them to Firecracker > [!NOTE] > > Static CPU templates are deprecated starting from v1.5.0 and will be removed > in accordance with our deprecation policy. Even after the removal, custom CPU > templates are available as an improved iteration of static CPU templates. For > more information about the transition from static CPU templates to custom CPU > templates, please refer to > [this GitHub discussion](https://github.com/firecracker-microvm/firecracker/discussions/4135). > [!NOTE] > > CPU templates for ARM (both static and custom) require the following patch to > be available in the host kernel: > [Support writable CPU ID registers from userspace](https://lore.kernel.org/kvm/20230212215830.2975485-1-jingzhangos@google.com/#t). > Otherwise KVM will fail to write to the ARM registers. ## Static CPU templates At the moment the following set of static CPU templates are supported: | CPU template | CPU vendor | CPU model | | ------------ | ---------- | ------------------------------- | | C3 | Intel | Skylake, Cascade Lake, Ice Lake | | T2 | Intel | Skylake, Cascade Lake, Ice Lake | | T2A | AMD | Milan | | T2CL | Intel | Cascade Lake, Ice Lake | | T2S | Intel | Skylake, Cascade Lake | | V1N1 | ARM | Neoverse V1 | T2 and C3 templates are mapped as close as possible to AWS T2 and C3 instances in terms of CPU features. Note that on a microVM that is lauched with the C3 template and running on processors that do not enumerate FBSDP_NO, PSDP_NO and SBDR_SSDP_NO on IA32_ARCH_CAPABILITIES MSR, the kernel does not apply the mitigation against MMIO stale data vulnerability. The T2S template is designed to allow migrating [snapshots](../snapshotting/versioning.md) between hosts with Intel Skylake and Intel Cascade Lake securely by further restricting CPU features for the guest, however this comes with a performance penalty. Users are encouraged to carry out a performance assessment if they wish to use the T2S template. Note that Firecracker expects the host to always be running the latest version of the microcode. The T2CL template is mapped to be close to Intel Cascade Lake. It is only safe to use it on Intel Cascade Lake and Ice Lake. The only AMD template is T2A. It is considered safe to be used with AMD Milan. Intel T2CL and AMD T2A templates together aim to provide instruction set feature parity between CPUs running them, so they can form a heterogeneous fleet exposing the same instruction sets to the application. The V1N1 template is designed to represent ARM Neoverse V1 as ARM Neoverse N1. ### Configuring static CPU templates Configuration of a static CPU template is performed via the `/machine-config` API endpoint: ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PUT 'http://localhost/machine-config' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "vcpu_count": 2, "mem_size_mib": 1024, "cpu_template": "T2CL" }' ``` ## Custom CPU templates Users can create their own CPU templates by creating a json file containing modifiers for CPUID, MSRs or ARM registers. > [!NOTE] > > Creating custom CPU templates requires expert knowledge of CPU architectures. > Custom CPU templates must be tested thoroughly before use in production. An > inappropriate configuration may lead to guest crashes or making guests > vulnerable to security attacks. For example, if a CPU template signals a > hardware vulnerability mitigation to the guest while the mitigation is in fact > not supported by the hardware, the guest may decide to disable corresponding > software mitigations which will make the guest vulnerable. > [!NOTE] > > Having MSRs or ARM registers in the custom CPU template does not affect access > permissions that guests will have to those registers. The access control is > handled by KVM and is not influenced by CPU templates. > [!NOTE] > > When setting guest configuration, KVM may reject setting some bits quietly. > This is user's responsibility to make sure that their custom CPU template is > applied as expected even if Firecracker does not report an error. In order to assist with creation and usage of CPU templates, there exists a CPU template helper tool. More details can be found [here](cpu-template-helper.md). ### Configuring custom CPU templates Configuration of a custom CPU template is performed via the `/cpu-config` API endpoint. An example of configuring a custom CPU template on x86_64: ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PUT 'http://localhost/cpu-config' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "kvm_capabilities": ["!56"], "cpuid_modifiers": [ { "leaf": "0x1", "subleaf": "0x0", "flags": 0, "modifiers": [ { "register": "eax", "bitmap": "0bxxxx000000000011xx00011011110010" } ] } ], "msr_modifiers": [ { "addr": "0x10a", "bitmap": "0b0000000000000000000000000000000000000000000000000000000000000000" } ] }' ``` This CPU template will do the following: - removes check for KVM capability: KVM_CAP_XCRS. This allows Firecracker to run on old cpus. See [this](https://github.com/firecracker-microvm/firecracker/discussions/3470) discussion. - in leaf `0x1`, subleaf `0x0`, register `eax`: - clear bits `0b00001111111111000011100100001101` - set bits `0b00000000000000110000011011110010` - leave bits `0b11110000000000001100000000000000` intact. - in MSR `0x10`, it will clear all bits. An example of configuring a custom CPU template on ARM: ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PUT 'http://localhost/cpu-config' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "kvm_capabilities": ["171", "172"], "vcpu_features": [{ "index": 0, "bitmap": "0b11xxxxx" }] "reg_modifiers": [ { "addr": "0x603000000013c020", "bitmap": "0bxxxxxxxxxxxx0000xxxxxxxxxxxx0000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" } ] }' ``` This CPU template will do the following: - add checks for KVM capabilities: KVM_CAP_ARM_PTRAUTH_ADDRESS and KVM_CAP_ARM_PTRAUTH_GENERIC. These checks are to ensure that the host have capabilities needed for the vCPU features. - enable additional vCPU features: KVM_ARM_VCPU_PTRAUTH_ADDRESS and KVM_ARM_VCPU_PTRAUTH_GENERIC - modify ARM register `0x603000000013c020`: - clear bits `0b0000000000001111000000000000111100000000000000000000000000000000` - leave bits `0b1111111111110000111111111111000011111111111111111111111111111111` intact. Information about KVM capabilities can be found in the [kernel source](https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/kvm.h). Information about vCPU features on aarch64 can be found in the [kernel source](https://elixir.bootlin.com/linux/latest/source/arch/arm64/include/uapi/asm/kvm.h). Information on how the ARM register addresses are constructed can be found in the [KVM API documentation](https://docs.kernel.org/virt/kvm/api.html#kvm-set-one-reg). ### Custom CPU templates language schema The full description of the custom CPU templates language can be found [here](schema.json). > [!NOTE] > > You can also use `_` to visually separate parts of a bitmap. So instead of > writing: `0b0000xxxx`, it can be `0b0000_xxxx`. #### Expansion of contracted bitmaps If a contracted version of a bitmap is given, for example, `0b101` where a 32-bit bitmap is expected, missing characters are implied to be `x` (`0bxxxxxxxxxxxxxxxxxxxxxxxxxxxxx101`). #### CPUID normalization and boot protocol register settings Some of the configuration set by a custom CPU template may be overwritten by Firecracker. More details can be found [here](cpuid-normalization.md) and [here](boot-protocol.md). #### Information about architecture-specific settings For detailed information when working with custom CPU templates, please refer to hardware specifications from CPU vendors, for example: - [Intel Software Developer Manual](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html) - [AMD Architecture Programmer's Manual](https://www.amd.com/en/support/tech-docs?keyword=programmer%27s+manual) - [ARM Architecture Refernce Manual](https://developer.arm.com/documentation/ddi0487/latest) ## A note about configuration of both static and custom CPU templates If a user configured both a static CPU template (via `/machine-config`) and a custom CPU template (via `/cpu-config`) in the same Firecracker process, only the configuration that was performed the _last_ is applied. This means that if a static CPU template was configured first and a custom CPU template was configured later, only the custom CPU template configuration will be applied when starting a microVM. --- ### Cpu Templates/Cpuid Normalization # CPUID normalization (x86_64 only) On x86_64, Firecracker makes certain modifications to the guest's CPUID regardless of whether a CPU template is used. This is referred to as `CPUID normalization`. If a CPU template is used the normalization is performed _after_ the CPU template is applied. That means that if the CPU template configures CPUID bits used in the normalization process, they will be overwritten. See also: [boot protocol settings](boot-protocol.md) ## x86_64 common CPUID normalization | Description | Leaf | Subleaf | Register | Bits | | ------------------------------------------------------------------------------------ | :--------: | :-----: | :-----------: | :---: | | Pass through vendor ID from host | 0x0 | - | EBX, ECX, EDX | all | | Set CLFLUSH line size | 0x1 | - | EBX | 15:8 | | Set maximum number of addressable IDs for logical processors in the physical package | 0x1 | - | EBX | 23:16 | | Set initial APIC ID | 0x1 | - | EBX | 31:24 | | Disable PDCM (Perfmon and Debug Capability) | 0x1 | - | ECX | 15 | | Enable TSC_DEADLINE | 0x1 | - | ECX | 24 | | Enable HYPERVISOR | 0x1 | - | ECX | 31 | | Set HTT value if the microVM's CPU count is greater than 1 | 0x1 | - | EDX | 28 | | Insert leaf 0xb, subleaf 0x1 if not present | 0xb | 0x1 | all | all | | Fill extended topology enumeration leaf | 0xb | all | all | all | | Pass through L1 cache and TLB information from host | 0x80000005 | - | all | all | | Pass through L2 cache and TLB and L3 cache information from host | 0x80000006 | - | all | all | ## Intel-specific CPUID normalization | Description | Leaf | Subleaf | Register | Bits | | -------------------------------------------------------------- | :--------------------------------: | :-----: | :------: | :---: | | Update deterministic cache parameters | 0x4 | all | EAX | 31:14 | | Disable Intel Turbo Boost technology | 0x6 | - | EAX | 1 | | Disable frequency selection | 0x6 | - | ECX | 3 | | Set FDP_EXCPTN_ONLY bit | 0x7 | 0x0 | EBX | 6 | | Set "Deprecates FPU CS and FPU DS values" bit | 0x7 | 0x0 | EBX | 13 | | Disable WAITPKG (UMONITOR / UMWAIT / TPAUSE) | 0x7 | 0x0 | ECX | 5 | | Disable performance monitoring | 0xa | - | all | all | | Fill v2 extended topology enumeration leaf | 0x1f | all | all | all | | Update brand string to use a default format and real frequency | 0x80000002, 0x80000003, 0x80000004 | - | all | all | ## AMD-specifc CPUID normalization | Description | Leaf | Subleaf | Register | Bits | | --------------------------------------------- | :--------------------------------: | :-----: | :----------------: | :---: | | Set IA32_ARCH_CAPABILITIES MSR as not present | 0x7 | - | EDX | 29 | | Set topology extension bit | 0x80000001 | - | ECX | 22 | | Update brand string with a default AMD value | 0x80000002, 0x80000003, 0x80000004 | - | EAX, EBX, ECX, EDX | all | | Update number of physical threads | 0x80000008 | - | ECX | 7:0 | | Update APIC ID size | 0x80000008 | - | ECX | 15:12 | | Update cache topology information | 0x8000001d | all | all | all | | Update extended APIC ID | 0x8000001e | - | EAX, EBX, ECX | all | --- ### Mmds/Mmds Design # microVM Metadata Service MMDS consists of three major logical components: the backend, the data store, and the minimalist HTTP/TCP/IPv4 stack (named *Dumbo*). They all exist within the Firecracker process, and outside the KVM boundary; the first is a part of the API server, the data store is a global entity for a single microVM, and the last is a part of the device model. ## The MMDS backend Users can add/update the MMDS contents via the backend, which is accessible through the Firecracker API. Setting the initial contents involves a `PUT` request to the `/mmds` API resource, with a JSON body that describes the desired data store structure and contents. Here's a JSON example: ```json { "latest": { "meta-data": { "ami-id": "ami-12345678", "reservation-id": "r-fea54097", "local-hostname": "ip-10-251-50-12.ec2.internal", "public-hostname": "ec2-203-0-113-25.compute-1.amazonaws.com", "network": { "interfaces": { "macs": { "02:29:96:8f:6a:2d": { "device-number": "13345342", "local-hostname": "localhost", "subnet-id": "subnet-be9b61d" } } } } } } } ``` The MMDS contents can be updated either via a subsequent `PUT` (that replaces them entirely), or using `PATCH` requests, which feed the JSON body into the JSON Merge Patch functionality, based on [RFC 7396](https://tools.ietf.org/html/rfc7396). MMDS related API requests come from the host, which is considered a trusted environment, so there are no checks beside the kind of validation done by HTTP server and `serde-json` (the crate used to de/serialize JSON). The size limit for the stored metadata is configurable and defaults to 51200 bytes. When increasing this limit, one must take into consideration that storing and retrieving large amount of data may induce bottlenecks for the HTTP REST API processing, which is based on `micro-http` crate. MMDS contents can be retrieved using the Firecracker API, via a `GET` request to the `/mmds` resource. ## The data store This is a global data structure, currently referenced using a global variable, that represents the strongly-typed version of JSON-based user input describing the MMDS contents. It leverages the recursive [Value](https://docs.serde.rs/serde_json/value/enum.Value.html) type exposed by `serde-json`. It can only be accessed from thread-safe contexts. MMDS data store supports at the moment storing and retrieving JSON values. Data store contents can be retrieved using the Firecracker API server from host and using the embedded MMDS HTTP/TCP/IPv4 network stack from guest. MMDS data store is upper bounded to the value of the `--mmds-size-limit` command line parameter. If left unconfigured, it will default to the value of `--http-api-max-payload-size`, which is 51200 bytes by default. ## Dumbo The *Dumbo* HTTP/TCP/IPv4 network stack handles guest HTTP requests heading towards the configured MMDS IPv4 address. Before going into *Dumbo* specifics, it's worth going through a brief description of the Firecracker network device model. Firecracker only offers Virtio-net paravirtualized devices to guests. Drivers running in the guest OS use ring buffers in a shared memory area to communicate with the device model when sending or receiving frames. The device model associates each guest network device with a TAP device on the host. Frames sent by the guest are written to the TAP fd, and frames read from the TAP fd are handed over to the guest. The *Dumbo* stack can be instantiated once for every network device, and is disabled by default. It can be enabled through the API request body used to configure MMDS by specifying the ID of the network interface inside the `network_interfaces` list. In order for the API call to succeed, the network device must be attached beforehand, otherwise an error is returned. Once enabled, the stack taps into the aforementioned data path. Each frame coming from the guest is examined to determine whether it should be processed by *Dumbo* instead of being written to the TAP fd. Also, every time there is room in the ring buffer to hand over frames to the guest, the device model first checks whether *Dumbo* has anything to send; if not, it resumes getting frames from the TAP fd (when available). We chose to implement our own solution, instead of leveraging existing libraries/implementations, because responding to guest MMDS queries in the context of Firecracker is amenable to a wide swath of simplifications. First of all, we only need to handle `GET` and `PUT` requests, which require a bare-bones HTTP 1.1 server, without support for most headers and more advanced features like chunking. Also, we get to choose what subset of HTTP is used when building responses. Moving lower in the stack, we are dealing with TCP connections over what is essentially a point-to-point link, that seldom loses packets and does not reorder them. This means we can do away with congestion control (we only use flow control), complex reception logic, and support for most TCP options/features. At this point, the layers below (Ethernet and IPv4) don't involve much more than sanity checks of frame/packet contents. *Dumbo* is built using both general purpose components (which we plan to offer as part of one or more libraries), and Firecracker MMDS specific code. The former category consists of various helper modules used to process streams of bytes as protocol data units (Ethernet & ARP frames, IPv4 packets, and TCP segments), a TCP handler which listens for connections while demultiplexing incoming segments, a minimalist TCP connection endpoint implementation, and a greatly simplified HTTP 1.1 server. The Firecracker MMDS specific code is found in the logic which taps into the device model, and the component that parses an HTTP request, builds a response based on MMDS contents, and finally sends back a reply. ### MMDS Network Stack Somewhat confusingly, this is the name of the component which taps the device model. It has a user-configured IPv4 address (see [Firecracker MMDS configuration API](../../src/firecracker/swagger/firecracker.yaml)) and MAC (`06:01:23:45:67:01`) addresses. The latter is also used to respond to ARP requests. For every frame coming from the guest, the following steps take place: 1. Apply a heuristic to determine whether the frame may contain an ARP request for the MMDS IP address, or an IPv4 packet heading towards the same address. There can be no false negatives. Frames that fail both checks are *rejected* (deferred to the device model for regular processing). 1. *Reject* invalid Ethernet frames. *Reject* valid frames if their EtherType is neither ARP, nor IPv4. 1. (**if EtherType == ARP**) *Reject* invalid ARP frames. *Reject* the frame if its target protocol address field is different from the MMDS IP address. Otherwise, record that an ARP request has been received (the stack only remembers the most recent request). 1. (**if EtherType == IPv4**) *Reject* invalid packets. *Reject* packets if their destination address differs from the MMDS IP address. *Drop* (stop processing without deferring to the device model) packets that do not carry TCP segments (by looking at the protocol number field). Send the rest to the inner TCP handler. The current implementation does not support Ethernet 802.1Q tags, and does not handle IP fragmentation. Tagged Ethernet frames are most likely going to be deferred to the device model for processing, because the heuristics do not take the presence of the tag into account. Moreover, their EtherType will not appear to be of interest. Fragmented IP packets do not get reassembled; they are treated as independent packets. Whenever the guest is able to receive a frame, the device model first requests one from the MMDS network stack associated with the current network device. 1. If an ARP request has been previously recorded, send an ARP reply and forget about the request. 1. If the inner TCP handler has any packets to transmit, wrap the next one into a frame and send it. 1. There are no MMDS related frames to send, so tell the device model to read from the TAP fd instead. #### Security Considerations Operators should not rely on the MMDS network stack to filter packets with the MMDS IP as the destination from the guest's outbound traffic. As stated in our [threat model](../design.md#threat-containment), guest traffic should be treated as untrusted, and firewall rules should be put in place at the host-level to prevent guests from accessing restricted IPv4 addresses on the host. Example rules for dropping host IMDS-bound packets may be seen in our [production host setup](../prod-host-setup.md#filtering-guest-egress-network-traffic) documentation. ### TCP handler Handles received packets that appear to carry TCP segments. Its operation is described in the `dumbo` crate documentation. Each connection is associated with an MMDS endpoint. ### MMDS endpoint This component gets the byte stream from an inner TCP connection object, identifies the boundaries of the next HTTP request, and parses it using an HttpRequest object. For each valid `GET` request, the URI is used to identify a key from the metadata store (like in the previous example), and a response is built using the Firecracker implementation of HttpResponse logic, based on the associated value, and sent back to the guest over the same connection. Each endpoint has a fixed size receive buffer, and a variable length response buffer (depending on the size of each response). TCP receive window semantics are used to ensure the guest does not overrun the receive buffer during normal operation (the connection has to drop segments otherwise). There can be at most one response pending at any given time. Here are more details describing what happens when a segment is received by an MMDS endpoint (previously created when a SYN segment arrived at the TCP handler): 1. Invoke the receive functionality of the inner connection object, and append any new data to the receive buffer. 1. If no response is currently pending, attempt to identify the end of the first request in the receive buffer. If no such boundary can be found, and the buffer is full, reset the inner connection (which also causes the endpoint itself to be subsequently removed) because the guest exceeded the maximum allowed request size. 1. If no response is pending, and we can identify a request in the receive buffer, parse it, free up the associated buffer space (also update the connection receive window), and build an HTTP response, which becomes the current pending response. 1. If a FIN segment was received, and there's no pending response, call `close` on the inner connection. If a valid RST is received at any time, mark the endpoint for removal. When the TCP handler asks an MMDS endpoint for any segments to send, the transmission logic of the inner connection is invoked, specifying the pending response (when present) as the payload source. All packets coming from MMDS have the TTL value set to 1 by default. ### Connection Connection objects are minimalist implementation of the TCP protocol. They are used to reassemble the byte stream which carries guest HTTP requests, and to send back segments which contain parts of the response. More details are available in the `dumbo` crate documentation. --- ### Mmds/Mmds User Guide # microVM Metadata Service The Firecracker microVM Metadata Service (MMDS) is a mutable data store which can be used for sharing information between host and guests, in a secure and easy at hand way. ## Configuring and activating the microVM Metadata Service By default, MMDS is not reachable from the guest operating system. At microVM runtime, MMDS is tightly coupled with a network interface, which allows MMDS requests. When configuring the microVM, if MMDS needs to be activated, a network interface has to be configured to allow MMDS requests. This can be achieved in two steps: 1. Attach one (or more) network interfaces through an HTTP `PUT` request to `/network-interfaces/${MMDS_NET_IF}`. The full network configuration API can be found in the [firecracker swagger file](../../src/firecracker/swagger/firecracker.yaml). 1. Configure MMDS through an HTTP `PUT` request to `/mmds/config` resource and include the IDs of the network interfaces that should allow forwarding requests to MMDS in the `network_interfaces` list. The complete MMDS API is described in the [firecracker swagger file](../../src/firecracker/swagger/firecracker.yaml). ### Examples Attaching a network device with ID `MMDS_NET_IF`: ```bash MMDS_NET_IF=eth0 curl --unix-socket /tmp/firecracker.socket -i \ -X PUT 'http://localhost/network-interfaces/${MMDS_NET_IF}' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "iface_id": "${MMDS_NET_IF}", "guest_mac": "AA:FC:00:00:00:01", "host_dev_name": "tap0" }' ``` Configuring MMDS to receive requests through the `MMDS_NET_IF` network interface ID: ```bash MMDS_IPV4_ADDR=169.254.170.2 curl --unix-socket /tmp/firecracker.socket -i \ -X PUT "http://localhost/mmds/config" \ -H "Content-Type: application/json" \ -d '{ "network_interfaces": ["${MMDS_NET_IF}"] }' ``` MMDS can be configured pre-boot only, using the Firecracker API server. Enabling MMDS without at least a network device attached will return an error. The IPv4 address used by guest applications when issuing requests to MMDS can be customized through the same HTTP `PUT` request to `/mmds/config` resource, by specifying the IPv4 address to the `ipv4_address` field. If the IP configuration is not provided before booting up the guest, the MMDS IPv4 address defaults to `169.254.169.254`. ```bash MMDS_IPV4_ADDR=169.254.170.2 curl --unix-socket /tmp/firecracker.socket -i \ -X PUT "http://localhost/mmds/config" \ -H "Content-Type: application/json" \ -d '{ "network_interfaces": ["${MMDS_NET_IF}"], "ipv4_address": "${MMDS_IPV4_ADDR}" }' ``` MMDS is tightly coupled with a network interface which is used to route MMDS packets. To send MMDS intended packets, guest applications must insert a new rule into the routing table of the guest OS. This new rule must forward MMDS intended packets to a network interface which allows MMDS requests. For example: ```bash MMDS_IPV4_ADDR=169.254.170.2 MMDS_NET_IF=eth0 ip route add ${MMDS_IPV4_ADDR} dev ${MMDS_NET_IF} ``` MMDS supports two methods to access the contents of the metadata store from the guest operating system: `V1` and `V2`. More about the particularities of the two mechanisms can be found in the [Retrieving metadata in the guest operating system](#retrieving-metadata-in-the-guest-operating-system) section. The MMDS version used can be specified when configuring MMDS, through the `version` field of the HTTP `PUT` request to `/mmds/config` resource. Accepted values are `V1`(deprecated) and `V2` and the default MMDS version used in case the `version` field is missing is [Version 1](#version-1-deprecated). ```bash MMDS_IPV4_ADDR=169.254.170.2 curl --unix-socket /tmp/firecracker.socket -i \ -X PUT "http://localhost/mmds/config" \ -H "Content-Type: application/json" \ -d '{ "network_interfaces": ["${MMDS_NET_IF}"], "version": "V2", "ipv4_address": "${MMDS_IPV4_ADDR}" }' ``` ## Inserting and updating metadata Inserting and updating metadata is possible through the Firecracker API server. The metadata inserted in MMDS must be any valid JSON. A user can create or update the MMDS data store before the microVM is started or during its operation. To insert metadata into MMDS, an HTTP `PUT` request to the `/mmds` resource has to be issued. This request must have a payload with metadata structured in [JSON](https://tools.ietf.org/html/rfc7159) format. To replace existing metadata, a subsequent HTTP `PUT` request to the `/mmds` resource must be issued, using as a payload the new metadata. A complete description of metadata insertion firecracker API can be found in the [firecracker swagger file](../../src/firecracker/swagger/firecracker.yaml). An example of an API request for inserting metadata is provided below: ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PUT "http://localhost/mmds" \ -H "Content-Type: application/json" \ -d '{ "latest": { "meta-data": { "ami-id": "ami-12345678", "reservation-id": "r-fea54097", "local-hostname": "ip-10-251-50-12.ec2.internal", "public-hostname": "ec2-203-0-113-25.compute-1.amazonaws.com", "network": { "interfaces": { "macs": { "02:29:96:8f:6a:2d": { "device-number": "13345342", "local-hostname": "localhost", "subnet-id": "subnet-be9b61d" } } } } } } }' ``` To partially update existing metadata, an HTTP `PATCH` request to the `/mmds` resource has to be issued, using as a payload the metadata patch, as [JSON Merge Patch](https://tools.ietf.org/html/rfc7396) functionality describes. A complete description of updating metadata Firecracker API can be found in the [firecracker swagger file](../../src/firecracker/swagger/firecracker.yaml). An example API for how to update existing metadata is offered below: ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PATCH "http://localhost/mmds" \ -H "Content-Type: application/json" \ -d '{ "latest": { "meta-data": { "ami-id": "ami-87654321", "reservation-id": "r-79054aef", } } }' ``` ## Retrieving metadata MicroVM metadata can be retrieved both from host and guest operating systems. For the scope of this chapter, let's assume the data store content is the JSON below: ```json { "latest": { "meta-data": { "ami-id": "ami-87654321", "reservation-id": "r-79054aef" } } } ``` ### Retrieving metadata in the host operating system To retrieve existing MMDS metadata from host operating system, an HTTP `GET` request to the `/mmds` resource must be issued. The HTTP response returns the existing metadata, as a JSON formatted text. A complete description of retrieving metadata Firecracker API can be found in the [firecracker swagger file](../../src/firecracker/swagger/firecracker.yaml). Below you can see how to retrieve metadata from the host: ```bash curl -s --unix-socket /tmp/firecracker.socket http://localhost/mmds ``` Output: ```json { "latest": { "meta-data": { "ami-id": "ami-87654321", "reservation-id": "r-79054aef" } } } ``` ### Retrieving metadata in the guest operating system Accessing the contents of the metadata store from the guest operating system can be done using one of the following methods: - `V1`: simple request/response method (deprecated) - `V2`: session-oriented method #### Version 1 (Deprecated) **Version 1 is deprecated and will be removed in the next major version change. Version 2 should be used instead.** To retrieve existing MMDS metadata using MMDS version 1, an HTTP `GET` request must be issued. The requested resource can be referenced by its corresponding [JSON Pointer](https://tools.ietf.org/html/rfc6901), which is also the path of the MMDS request. The HTTP response content will contain the referenced metadata resource. As in version 2, version 1 also supports a session oriented method in order to make the migration easier. See [the next section](#version-2) for the session oriented method. Note that version 1 returns a successful response to a `GET` request even with an invalid token or no token not to break existing workloads. `mmds.rx_invalid_token` and `mmds.rx_no_token` metrics track the number of `GET` requests with invalid tokens and missing tokens respectively, helping users evaluate their readiness for migrating to MMDS version 2. Requests containing any other HTTP methods than `GET` and `PUT` will receive **405 Method Not Allowed** error. ```bash MMDS_IPV4_ADDR=169.254.170.2 RESOURCE_POINTER_OBJ=latest/meta-data curl -s "http://${MMDS_IPV4_ADDR}/${RESOURCE_POINTER_OBJ}" ``` #### Version 2 Similar to [IMDSv2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html), MMDS version 2 (`V2`) is a session oriented method, which makes use of a session token in order to allow fetching metadata contents. The session must start with an HTTP `PUT` request that generates the session token. In order to be successful, the request must respect the following constraints: - must be directed towards `/latest/api/token` path - must contain a `X-metadata-token-ttl-seconds` or `X-aws-ec2-metadata-token-ttl-seconds` header specifying the token lifetime in seconds. The value cannot be lower than 1 or greater than 21600 (6 hours). - must not contain a `X-Forwarded-For` header. ```bash MMDS_IPV4_ADDR=169.254.170.2 TOKEN=`curl -X PUT "http://${MMDS_IPV4_ADDR}/latest/api/token" \ -H "X-metadata-token-ttl-seconds: 21600"` ``` The HTTP response from MMDS is a plaintext containing the session token. During the duration specified by the token's time to live value, all subsequent `GET` requests must specify the session token through the `X-metadata-token` or `X-aws-ec2-metadata-token` header in order to fetch data from MMDS. ```bash MMDS_IPV4_ADDR=169.254.170.2 RESOURCE_POINTER_OBJ=latest/meta-data curl -s "http://${MMDS_IPV4_ADDR}/${RESOURCE_POINTER_OBJ}" \ -H "X-metadata-token: ${TOKEN}" ``` After the token expires, it becomes unusable and a new session token must be issued. ##### Snapshotting considerations The data store is **not** persisted across snapshots, in order to avoid leaking vm-specific information that may need to be reseeded into the data store for a new clone. The MMDS version, network stack configuration and IP address used for accessing the service are persisted across snapshot-restore. If the targeted snapshot version does not support Mmds Version 2, it will not be persisted in the snapshot (the clone will use the default, V1). Similarly, if a snapshotted Vm state contains the Mmds version but the Firecracker version used for restoring does not support persisting the version, the default will be used. ### MMDS formats The response format can be JSON or IMDS. The IMDS documentation can be found [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html). The output format can be selected by specifying the optional `Accept` header. Using `Accept: application/json` will format the output to JSON, while using `Accept: plain/text` or not specifying this optional header at all will format the output to IMDS. Setting `imds_compat` to `true` through PUT request to `/mmds/config` enforces MMDS to always respond in IMDS format regardless of the `Accept` header. This allows code written to work on EC2 IMDS to also work on Firecracker MMDS. Retrieving MMDS resources in IMDS format, other than JSON `string` and `object` types, is not supported. Below is an example on how to retrieve the `latest/meta-data` resource in JSON format: ```bash MMDS_IPV4_ADDR=169.254.170.2 RESOURCE_POINTER_OBJ=latest/meta-data curl -s -H "Accept: application/json" "http://${MMDS_IPV4_ADDR}/${RESOURCE_POINTER_OBJ}" ``` Output: ```json { "ami-id": "ami-87654321", "reservation-id": "r-79054aef" } ``` Retrieving the `latest/meta-data/ami-id` resource in JSON format: ```bash MMDS_IPV4_ADDR=169.254.170.2 RESOURCE_POINTER_STR=latest/meta-data/ami-id curl -s -H "Accept: application/json" "http://${MMDS_IPV4_ADDR}/${RESOURCE_POINTER_STR}" ``` Output: ```json "ami-87654321" ``` Retrieving the `latest` resource in IMDS format: ```bash MMDS_IPV4_ADDR=169.254.170.2 RESOURCE_POINTER=latest curl -s "http://${MMDS_IPV4_ADDR}/${RESOURCE_POINTER}" ``` Output: ```text meta-data/ ``` Retrieving the `latest/meta-data/` resource in IMDS format: ```bash MMDS_IPV4_ADDR=169.254.170.2 RESOURCE_POINTER=latest/meta-data curl -s "http://${MMDS_IPV4_ADDR}/${RESOURCE_POINTER}" ``` Output: ```text ami-id reservation-id ``` Retrieving the `latest/meta-data/ami-id` resource in IMDS format: ```bash MMDS_IPV4_ADDR=169.254.170.2 RESOURCE_POINTER=latest/meta-data/ami-id curl -s "http://${MMDS_IPV4_ADDR}/${RESOURCE_POINTER}" ``` Output: ```text ami-87654321 ``` ## Errors *200* - `Ok` The request was successfully processed and a response was successfully formed. *400* - `Bad Request` The request was malformed. *401* - `Unauthorized` Only when using MMDS `V2`. The HTTP request either lacks the session token, or the token specified is invalid. A token is invalid if it was not generated using an HTTP `PUT` request or if it has expired. *404* - `Not Found` The requested resource can not be found in the MMDS data store. *405* - `Method Not Allowed` The HTTP request uses a not allowed HTTP method and a response with the `Allow` header was formed. When using MMDS `V1`, this is returned for any HTTP method other than `GET`. When MMDS `V2` is configured, the only accepted HTTP methods are `PUT` and `GET`. *501* - `Not Implemented` The requested HTTP functionality is not supported by MMDS or the requested resource is not supported in IMDS format. ## Appendix ### Example use case: credential rotation For this example, the guest expects to find some sort of credentials (say, a secret access key) by issuing a `GET` request to `http://169.254.169.254/latest/meta-data/credentials/secret-key`. Most similar use cases will encompass the following sequence of steps: 1. Some agent running on the host sends a `PUT` request with the initial contents of the MMDS, using the Firecracker API. This most likely takes place before the microVM starts running, but may also happen at a later time. Guest MMDS requests which arrive prior to contents being available receive a *NotFound* response. 1. The contents are saved to MMDS. 1. The guest sends a `GET` request for the secret key, which is intercepted by MMDS. 1. MMDS processes the request and sends back an HTTP response with the ensembled secret key as a JSON string. After a while, the host agent decides to rotate the secret key. It does so by updating the data store with a new value. This can be done via a `PUT` request to the `/mmds` API resource, which replaces everything, or with a `PATCH` request that only touches the desired key. This effectively triggers the first two steps again. The guest reads the new secret key, going one more time through the last three steps. This can happen after a notification from the host agent, or discovered via periodic polling, or some other mechanism. Since access to the data store is thread safe, the guest can only receive either the old version, or the new version of the key, and not some intermediate state caused by the update. --- ### Snapshotting/Handling Page Faults On Snapshot Resume # Handling snapshot memory loading Firecracker allows for a better management of the microVM's memory loading by letting users choose between relying on host OS to handle the page faults when resuming from a snapshot, or having a dedicated userspace process for dealing with page faults, with the help of [Userfaultfd](https://www.kernel.org/doc/html/v4.18/admin-guide/mm/userfaultfd.html). ## Kernel When resuming a microVM from a snapshot, loading the snapshotted guest's memory (which is file-backed) into RAM is usually kernel's responsibility and is handled on a per-page-fault basis. Each time the guest touches a page that is not already in Firecracker's process memory, a page fault occurs, which triggers a context switch and IO operation in order to bring that page into RAM. Depending on the use case, doing this for every page can be time-consuming. ## Userfaultfd Userfaultfd is a mechanism that passes that responsibility of handling page fault events from kernel space to user space. In order to be able to interact with this mechanism, userspace needs to firstly obtain an userfault file descriptor object (UFFD). ### Creating a UFFD object #### Kernel 5.10 For host kernel 5.10 UFFD objects are created by calling into [`userfaultfd` syscall](https://man7.org/linux/man-pages/man2/userfaultfd.2.html). #### Kernel 6.1 For kernel 6.1, UFFD is created through the `/dev/userfaultfd` device. Access to `/dev/userfaultfd` is managed by file system permissions, so the Firecracker process needs to have proper permissions to create the UFFD object. When `/dev/userfaultfd` is present on the host system, jailer makes it available inside the jail and Firecracker process can use it without any further configuration. If a user is not using Firecracker along with the jailer, they should manage manually permissions to `/dev/userfaultfd`. For example, on systems that rely on access control lists (ACLs), this can be achieved by: ```bash sudo setfacl -m u:${USER}:rw /dev/userfaultfd ``` ### Registering memory to be handled via Userfault File Descriptors Next, the memory address range must be registered with the userfault file descriptor so that the userfault object can monitor page faults occurring for those addresses. After this, the user space process can start reading and serving events via the userfault file descriptor. These events will contain the address that triggered the fault. The fault-handling thread can choose to handle these events using these [operations](https://www.kernel.org/doc/html/latest/admin-guide/mm/userfaultfd.html#resolving-userfaults). In the flow described above, there are two userspace processes that interact with each other in order to handle page faults: Firecracker process and the page fault handler. Please note that users are responsible for writing the page fault handler process to monitor userfaultfd events and handle those events. Below is the interaction flow between Firecracker and the page fault handler (designed by the users): - Page fault handler binds and listens on a unix domain socket in order to be able to communicate with the Firecracker process. Please note that when using the Jailer, the page fault handler process, UDS and memory file must reside inside the jail. The UDS must only be accessible to Firecracker and the page fault handler. - PUT snapshot/load API call is issued towards Firecracker's API thread. The request encapsulates in its body the path to the unix domain socket that page fault handler listens to in order to communicate with Firecracker. - Firecracker process creates the userfault object and obtains the userfault file descriptor. - The page fault handler privately mmaps the contents of the guest memory file. - Firecracker anonymously mmaps memory based on the memory description found in the microVM state file and registers the memory regions with the userfault object in order for the userfaultfd to be aware of page fault events on these addresses. Firecracker then connects to the socket previously opened by the page fault process. - Firecracker passes the userfault file descriptor and the guest memory layout (e.g. dimensions of each memory region, and their [page size](../hugepages.md) in KiB) to the page fault handler process through the socket. - After sending the necessary information to the page fault handler, Firecracker continues with the normal cycle to restore from snapshot. It reads from the microVM state file the relevant serialized components and loads them into memory. - Page faults that occur while Firecracker is touching guest memory are handled by the page fault handler process, which listens for events on the userfault file descriptor that Firecracker previously sent. When a page fault event happens, the page fault handler issues `UFFDIO_COPY` to load the previously mmaped file contents into the correspondent memory region. After Firecracker sends the payload (i.e. mem mappings and file descriptor), no other communication happens on the UDS socket (or otherwise) between Firecracker and the page fault handler process. ### Userfaultfd interaction with balloon The balloon device allows the host to reclaim memory from a microVM. For more details on balloon, please refer to [this doc](../ballooning.md). When the balloon device asks for removal of a memory range, Firecracker calls `madvise` with the `MADV_DONTNEED` flag in order to let the kernel know that it can free up memory found in that specific area. On such a system call, the userfaultfd interface sends `UFFD_EVENT_REMOVE`. When implementing the logic for the page fault handler, users must identify events of type `UFFD_EVENT_REMOVE` and handle them by zeroing out those pages. This is because the memory is removed, but the area still remains monitored by userfaultfd. After a cycle of inflation and deflation, page faults might happen again for memory ranges that have been removed by balloon (and subsequently zeroed out by the page fault handler). In such a case, the page fault handler process must zero out the faulted page (instead of bringing it from file), as recommended by [the userfaultfd documentation](https://www.kernel.org/doc/html/latest/admin-guide/mm/userfaultfd.html#non-cooperative-userfaultfd). In case of a compromised balloon driver, the page fault handler can get flooded with `UFFD_EVENT_REMOVE`. We recommend using the jailer's built-in cgroup functionality as defense in depth, in order to limit resource usage of the Firecracker process. ### Caveats If the handler process crashes while Firecracker is resuming the snapshot, Firecracker will hang when a page fault occurs. This is because Firecracker is designed to wait for the requested page to be made available. If the page fault handler process is no longer around when this happens, Firecracker will wait forever. Users are expected to monitor the page fault handler's status or gather metrics of hanged Firecracker process and implement a recycle mechanism if necessary. It is the page fault handler process's responsibility to handle any errors that might occur and also send signals to Firecracker process to inform it of any crashes/exits. The page fault handler can fetch Firecracker's PID through `getsockopt` call with `SO_PEERCRED` option, which fetches credentials of the peer process that is connected to the socket. The returned credentials contain: PID, GID and UID of the peer process (Firecracker in the page fault handler's case). We recommend that the page fault handler includes timeouts for waiting on Firecracker to connect to the UDS or send information over the UDS, in order to account for unexpected cases when Firecracker crashes before being able to connect/send data. ### Example An example of a handler process can be found [here](../../src/firecracker/examples/uffd/on_demand_handler.rs). The process is designed to tackle faults on a certain address by loading into memory the entire region that the address belongs to, but users can choose any other behavior that suits their use case best. --- ### Snapshotting/Network For Clones # Network Connectivity for Clones This document presents a strategy to ensure continued network connectivity for multiple clones created from a single Firecracker microVM snapshot. > [!CAUTION] > > This should be considered as just an example to get you started, and we don't > claim this is a performant or secure setup. ## Setup There are two things which prevent network connectivity from resuming out-of-the-box for clones created from the same snapshot: Firecracker currently saves and attempts to restore network devices using the initially configured TAP names, and each guest will be resumed with the same network configuration, most importantly with the same IP address(es). To work around the former, each clone should be started within a separate network namespace (we can have multiple TAP interfaces with the same name, as long as they reside in distinct network namespaces). The latter can be mitigated by leveraging `iptables` `NAT` support. Let’s have a more detailed look at this approach. We assume each VM has a single network interface attached. If multiple interfaces with full connectivity are required, we simply repeat the relevant parts of this process for each additional interface. A typical setup right before taking a snapshot involves having a VM with a network interface backed by a TAP device (named `vmtap0`, for example) with an IP address (referred to as the TAP IP address, for example `192.168.241.1/29`), and an IP address configured inside the guest for the corresponding virtio device (referred to as the guest IP address, for example `192.168.241.2/29`). ### Network namespaces Attempting to restore multiple clones from the same snapshot faces the problem of every single one of them attempting to use a TAP device with the original name, which is not possible by default. Therefore, we need to start each clone in a separate network namespace. This is already possible using the `--netns` jailer parameter, described in the [documentation](../jailer.md). The specified namespace must already exist, so we have to create it first using ```bash sudo ip netns add fc0 ``` (where `fc0` is the name of the network namespace we plan to use for this specific clone - `clone0`). A new network namespace is initially empty, so we also have to create a new tap interface within using ```bash sudo ip netns exec fc0 ip tuntap add name vmtap0 mode tap ``` The `ip netns exec ` allows us to execute `command` in the context of the specified network namespace (in the previous case, the secondary command creates a new tap interface). Next we configure the new TAP interface to match the expectations of the snapshotted guest by running ```bash sudo ip netns exec fc0 ip addr add 192.168.241.1/29 dev vmtap0 sudo ip netns exec fc0 ip link set vmtap0 up ``` At this point we can start multiple clones, each in its separate namespace, but they won’t have connectivity to the rest of the host, only the respective TAP interfaces. However, interaction over the network is still possible; for example we can connect over ssh to clone0 using ```bash sudo ip netns exec fc0 ssh root@192.168.241.2 ``` ### `veth` interfaces to connect the network namespaces In order to obtain full connectivity we have to begin by connecting the network namespace to the rest of the host, and then solving the *“same guest IP”* problem. The former requires the use of `veth` pairs - *virtual interfaces that are link-local to each other (any packet sent through one end of the pair is immediately received on the other, and the other way around)*. One end resides inside the network namespace, while the other is moved into the parent namespace (the host global namespace in this case), and packets flow in or out according to the network configuration. We have to pick IP addresses for both ends of the veth pair. For clone index `idx`, let’s use `10..<(idx % 30) * 8>.1/24` for the endpoint residing in the host namespace, and the same address ending with `2` for the other end which remains inside the clone's namespace. Thus, for `clone 0` the former is `10.0.0.1` and the latter `10.0.0.2`. The first endpoint must have an unique name on the host, for example chosen as `veth(idx + 1) (so veth1 for clone 0)`. To create and setup the veth pair, we use the following commands (for namespace `fc0`): ```bash # create the veth pair inside the namespace sudo ip link add name veth1 type veth peer name veth0 netns fc0 sudo ip netns exec fc0 ip addr add 10.0.0.2/24 dev veth0 sudo ip netns exec fc0 ip link set dev veth0 up sudo ip addr add 10.0.0.1/24 dev veth1 sudo ip link set dev veth1 up # designate the outer end as default gateway for packets leaving the namespace sudo ip netns exec fc0 ip route add default via 10.0.0.1 ``` ### `iptables` rules for VM egress connectivity The last step involves adding the `iptables` rules which change the source/destination IP address of packets on the fly (thus allowing all clones to have the same internal IP). ```sh # Find the host egress device UPSTREAM=$(ip -j route list default |jq -r '.[0].dev') # anything coming from the VMs, we NAT the address ip netns exec fc0 iptables -t nat -A POSTROUTING -s 192.168.241.1/29 -o veth0 -j MASQUERADE iptables -t nat -A POSTROUTING -s 10.0.0.0/30 -o $UPSTREAM -j MASQUERADE # forward packets by default iptables -P FORWARD ACCEPT ip netns exec fc0 ip route add default via 10.0.0.1 ip netns exec fc0 iptables -P FORWARD ACCEPT ``` You may also want to configure the guest with a default route and a DNS nameserver: ```bash ip route default via 10.0.0.1 echo nameserver 8.8.8.8 >/etc/resolv.conf ``` **Connectivity from the clone should be present at this point.** To make sure the guest also adjusts to the new environment, you can explicitly clear the ARP/neighbour table in the guest: ```bash ip -family inet neigh flush any ip -family inet6 neigh flush any ``` Otherwise, packets originating from the guest might be using old Link Layer Address for up to arp cache timeout seconds. After said timeout period, connectivity will work both ways even without an explicit flush. ### Renaming host device names In some environments where the jailer is not being used, restoring a snapshot may be tricky because the tap device on the host will not be the same as the tap device that the original VM was mapped to when it was snapshotted, for example when the tap device comes from a pool of such devices. In this case you can use the `network_overrides` parameter of the snapshot restore API to specify which guest network device maps to which host tap device. For example, if we have a network interface named `eth0` in the snapshotted microVM, we can override it to point to the host device `vmtap01` during snapshot resume, like this: ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PUT 'http://localhost/snapshot/load' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "snapshot_path": "./snapshot_file", "mem_backend": { "backend_path": "./mem_file", "backend_type": "File" }, "network_overrides": [ { "iface_id": "eth0", "host_dev_name": "vmtap01" } ] }' ``` This may require reconfiguration of the networking inside the VM so that it is still routable externally. [network setup documentation](../network-setup.md#in-the-guest) describes what the typical setup is. If you are not using network namespaces or the jailer, then the guest will have to be made aware (via vsock or other channel) that it needs to reconfigure its network to match the network configured on the tap device. If the new TAP device, say `vmtap3` has been configured to use a guest address of `172.16.3.2` then after snapshot restore you would run something like: ```bash # In the guest # Clear out the previous addr and route ip addr flush dev eth0 ip route flush dev eth0 # Configure the new address ip addr add 172.16.3.2/30 dev eth0 ip route add default via 172.16.3.1/30 dev eth0 ``` # Ingress connectivity The above setup only provides egress connectivity. If in addition we also want to add ingress (in other words, make the guest VM routable outside the network namespace), then we need to choose a "clone address" that will represent this VM uniquely. For our example we can use IPs from `172.16.0.0/12`, for example `172.16.0.1`. Then we can rewrite destination address heading towards the "clone address" to the guest IP. ```bash ip netns exec fc0 iptables -t nat -A PREROUTING -i veth0 \ -d 172.16.0.1 -j DNAT --to 192.168.241.2 ``` And add a route on the host so we can access the guest VM from the host network namespace: ```bash ip route add 172.16.0.1 via 10.0.0.2 ``` To confirm that ingress connectivity works, try ```bash ping 172.16.0.1 # or ssh root@172.16.0.1 ``` # See also For an improved setup with full ingress and egress connectivity to the individual VMs, see [this discussion](https://github.com/firecracker-microvm/firecracker/discussions/4720). --- ### Snapshotting/Random For Clones # Entropy for Clones This document provides a high level perspective on the implications of restoring multiple VM clones from a single snapshot. We start with an overview of the Linux random number generation (RNG) facilities, then go through the potential issues we’ve identified related to cloning state, and finally conclude with a series of recommendations. It’s worth stressing that we aim to prevent stale state being a problem only for the kernel interfaces. Some userspace applications or libraries keep their own equivalent of entropy pools and suffer from the same potential issues after being cloned. There is no generic solution under the current programming model, and all we can do is recommend against their use in pre-snapshot logic. ## Background The Linux kernel exposes three main `RNG` interfaces to userspace: the `/dev/random` and `/dev/urandom` special devices, and the `getrandom` syscall, which are described in the [random(7) man page][1]. Moreover, Firecracker supports the [`virtio-rng`](../entropy.md) device which can provide additional entropy to guest VMs. It draws its random bytes from the [`aws-lc-rs`][8] crate which wraps the [`AWS-LC` cryptographic library][9]. Traditionally, `/dev/random` has been considered a source of “true” randomness, with the downside that reads block when the pool of entropy gets depleted. On the other hand, `/dev/urandom` doesn’t block, which lead people believe that it provides lower quality results. It turns out the distinction in output quality is actually very hard to make. According to [this article][2], for kernel versions prior to 4.8, both devices draw their output from the same pool, with the exception that `/dev/random` will block when the system estimates the entropy count has decreased below a certain threshold. The `/dev/urandom` output is considered secure for virtually all purposes, with the caveat that using it before the system gathers sufficient entropy for initialization may indeed produce low quality random numbers. The `getrandom` syscall helps with this situation; it uses the `/dev/urandom` source by default, but will block until it gets properly initialized (the behavior can be altered via configuration flags). Newer kernels (4.8+) have switched to an implementation where `/dev/random` output comes from a pool called the blocking pool, the output of `/dev/urandom` is given by a CSPRNG (cryptographically secure pseudorandom number generator), and there’s also an input pool which gathers entropy from various sources available on the system, and is used to feed into or seed the other two components. A very detailed description is available [here][3]. The details of this newer implementation are used to make the recommendations present in the document. There are in-kernel interfaces used to obtain random numbers as well, but they are similar to using `/dev/urandom` (or `getrandom` with the default source) from userspace. Whenever a VM clone is created based on a snapshot, execution resumes precisely from the previously saved state. Getting random bytes from either `/dev/random` or `/dev/urandom` does not lead to identical results for different clones created from the same snapshot because multiple parameters (such as timer data, or output from `CPU HWRNG` instructions which are present on Ivy Bridge or newer Intel processors and enabled in a Firecracker guest) are mixed with each result. Extra bits are mixed in both when reading random values, and in conjunction with entropy related events such as interrupts. Moreover, the guest kernel will eventually receive fresh entropy from `virtio-rng`, if attached. There are two questions here: - Is the `CPU HWRNG` output always mixed in when the feature is present (as opposed to only when the `CPU HWRNG` is trusted)? - Is the added noise strong enough to consider the final RNG output sufficiently divergent from all other clones? Both these questions are particularly relevant immediately after resuming a VM from a snapshot. After the VM gets to run for a "sufficient" amount of time it should be able to gather some more entropy by itself and its state should be sufficiently divergent that of any other clones. It seems the `CPU HWRNG` is always added to mix when present. More specifically, [page 32 point 1 (at the top of the page)][3] mentions using the `CPU HWRNG` when present for the entropy pool output function. Page 34 states *in case a CPU random number generator is known to the Linux-RNG, data from that hardware RNG is mixed into the entropy pool in a second step*. With respect to the initialization of the random pools and DRNG behind /dev/urandom. The discussion regarding DRNG state on page 35 mentions *the key part, the counter, and the nonce are XORed with the output of the CPU random number generator if one is present. If it is not present, one high-resolution time stamp obtained with the kernel function random_get_entropy word is XORed with the key part*. The `CPU HWRNG` is also used for the DRNG state transition function (as stated on page 36 point 1), and during the reseed operation (page 37 point 2). The document explicitly mentions when the `CPU HWRNG` has to be trusted (for example, the bullet points at the end of Section 3.3.2.3). It’s not yet clear whether the noise that gets added for each clone post restore is sufficient to consider their RNG states distinct for security purposes. The conservative approach is to presume the stale state has a significant influence on RNG output, so we should reinitialize both sources based on fresh data after each restore. It would seem that simply writing data to `/dev/urandom` is enough to muddle the entropy pools, but the bits only get mixed with the input pool. It’s not certain at this point whether such writes have any immediate impact on the blocking pool, and it’s unlikely they cause the `CSPRNG` to be automatically reseeded. The standard methods of interacting with the kernel RNG sources are documented in the [random(4) man page][4]. It states that any writes to either `/dev/random` or `/dev/urandom` are mixed with the input entropy pool, but do not increase the current entropy estimation. There is also an `ioctl` interface which, given the appropriate privileges, can be used to add data to the input entropy pool while also increasing the count, or completely empty all pools. ### Linux kernels with VMGenID support Linux has support for the [Virtual Machine Generation Identifier](https://learn.microsoft.com/en-us/windows/win32/hyperv_v2/virtual-machine-generation-identifier) since 5.18 for ACPI systems. Since 6.10, Linux added support also for systems that use DeviceTree instead of ACPI. The purpose of VMGenID is to notify the guest about time shift events, such as resuming from a snapshot. The device exposes a 16-byte cryptographically random identifier in guest memory. Firecracker implements VMGenID. When resuming a microVM from a snapshot Firecracker writes a new identifier and injects a notification to the guest. Linux [uses this value](https://elixir.bootlin.com/linux/v5.18.19/source/drivers/virt/vmgenid.c#L77) [as new randomness for its CSPRNG](https://elixir.bootlin.com/linux/v5.18.19/source/drivers/char/random.c#L908). Quoting the random.c implementation of the kernel: ``` /* * Handle a new unique VM ID, which is unique, not secret, so we * don't credit it, but we do immediately force a reseed after so * that it's used by the crng posthaste. */ ``` As a result, values returned by `getrandom()` and `/dev/(u)random` are distinct in all VMs started from the same snapshot, **after** the kernel handles the VMGenID notification. This leaves a race window between resuming vCPUs and Linux CSPRNG getting successfully re-seeded. Firecracker supports VMGenID on ARM systems using the DeviceTree binding that was added for the device in Linux 6.10. However, the latest Linux kernel that Firecracker supports is 6.1. As a result, in order to use VMGenID on ARM systems, users need to use a 6.1 kernel with the DeviceTree binding support backported from 6.10. For our CI we backport the [relevant changes](https://github.com/torvalds/linux/commit/f0cd69b8cca6a5096463644d6dacc9f991bfa521) from 6.10 to 6.1. Consumers of Firecracker that want to use the feature on ARM need to ensure they backport these changes on their guest kernels. Please note that, Firecracker will always enable VMGenID. In kernels where there is no VMGenID driver, the device will not have any effect in the guest. ### User space considerations Init systems (such as `systemd` used by AL2 and other distros) might save a random seed file after boot. For `systemd`, the path is `/var/lib/systemd/random-seed`. Just to be on the safe side, any such file should be deleted before taking a snapshot, to prevent its reuse for any purposes by the guest. There’s also the `/proc/sys/kernel/random/boot_id` special file, which gets initialized with a random string at boot time, and is read-only afterwards. All clones restored from the same snapshot will implicitly read the same value from this file. If that’s not desirable, it’s possible to alter the read result via bind mounting another file on top of `/proc/sys/kernel/random/boot_id`. ## Recommendations - Delete `/var/lib/systemd/random-seed`, or any equivalent files. - If changing the value present in `/proc/sys/kernel/random/boot_id` is important, bind mount another file on top of it. - If microVMs run on machines with IvyBridge or newer Intel processors (which provide RDRAND; in addition, RDSEED is offered starting with Broadwell). Hardware supported reseeding is done on a cadence defined by the Linux Kernel and should be sufficient for most cases. - Use `virtio-rng`. When present, the guest kernel uses the device as an additional source of entropy. - On kernels before 5.18, to be as safe as possible, the direct approach is to do the following (before customer code is resumed in the clone): 1. Open one of the special devices files (either `/dev/random` or `/dev/urandom`). Take note that `RNDCLEARPOOL` no longer [has any effect][7] on the entropy pool. 1. Issue an `RNDADDENTROPY` ioctl call (requires `CAP_SYS_ADMIN`) to mix the provided bytes into the input entropy pool and increase the entropy count. This should also cause the `/dev/urandom` `CSPRNG` to be reseeded. The bytes can be generated locally in the guest, or obtained from the host. 1. Issue a `RNDRESEEDCRNG` ioctl call ([4.14][5], [5.10][6], (requires `CAP_SYS_ADMIN`)) that specifically causes the `CSPRNG` to be reseeded from the input pool. - On kernels starting from 5.18 onwards, the CSPRNG will be automatically reseeded when the guest kernel handles the VMGenID notification. To completely avoid the race condition, users should follow the same steps as with kernels \< 5.18. - [Userspace notifications of loading snapshots](snapshot-support.md#userspace-notifications-of-loading-snapshots) can be used to trigger the direct approach described above. **Annex 1 contains the source code of a C program which implements the previous three steps.** As soon as the guest kernel version switches to 4.19 (or higher), we can rely on the `CONFIG_RANDOM_TRUST_CPU` kernel option (or the random.trust_cpu=on cmdline parameter) to have the entropy pool automatically refilled using the `CPU HWRNG`, so step 3 would no longer be necessary. Another way around step 3 is to attach a `virtio-rng` device. However, we cannot control when the guest kernel will request for random bytes from the device. ## Annex 1: Source code that clears and reinitializes the entropy pool ```cpp #include #include #include #include #include #include #include void exit_usage() { printf("Usage: ./rerand []\n" "The length of the string must be a multiple of 8.\n"); exit(EXIT_FAILURE); } void exit_perror(const char *msg) { perror(msg); exit(EXIT_FAILURE); } int main(int argc, char ** argv) { if (argc > 2) { exit_usage(); } size_t len = 0; struct rand_pool_info *info = NULL; if (argc == 2) { len = strlen(argv[1]); // We want len to be a multiple of 8 such that we have an easier time // parsing argv[1] into an array of u32s. if (len % 8) { exit_usage(); } info = malloc(sizeof(struct rand_pool_info) + len / 8); if (info == NULL) { exit_perror("Could not alloc rand_pool_info struct"); } // This is measured in bits IIRC. info->entropy_count = len * 4; info->buf_size = len / 8; } int fd = open("/dev/urandom", O_RDWR); if (fd < 0) { exit_perror("Unable to open /dev/urandom"); } if (ioctl(fd, RNDCLEARPOOL) < 0) { exit_perror("Error issuing RNDCLEARPOOL operation"); } if (argc == 1) { exit(EXIT_SUCCESS); } // Add the entropy bytes supplied by the user. char num_buf[9] = {}; size_t pos = 0; while (pos < len) { memcpy(num_buf, &argv[1] + pos, 8); info->buf[pos / 8] = strtoul(num_buf, NULL, 16); pos += 8; } if (ioctl(fd, RNDADDENTROPY, info) < 0) { exit_perror("Error issuing RNDADDENTROPY operation"); } } ``` [1]: http://man7.org/linux/man-pages/man7/random.7.html [2]: https://www.2uo.de/myths-about-urandom [3]: https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/Publications/Studies/LinuxRNG/LinuxRNG_EN.pdf [4]: http://man7.org/linux/man-pages/man4/random.4.html [5]: https://elixir.bootlin.com/linux/v4.14.295/source/drivers/char/random.c#L1355 [6]: https://elixir.bootlin.com/linux/v5.10.147/source/drivers/char/random.c#L1360 [7]: https://elixir.bootlin.com/linux/v4.14.295/source/drivers/char/random.c#L1351 [8]: https://docs.rs/aws-lc-rs/latest/aws_lc_rs/index.html [9]: https://github.com/aws/aws-lc --- ### Snapshotting/Snapshot Editor # Snapshot editor The `snapshot-editor` is a program for modification of Firecracker snapshots. ## Prior knowledge Firecracker snapshot consists of 2 files: - `vmstate` file: file with Firecracker internal data such as vcpu states, devices states etc. - `memory` file: file with guest memory. ## Usage ### `edit-memory` command #### `rebase` subcommand > This command is used to merge a `diff` snapshot memory file on top of a base > memory file. > > **Note** You can also use `rebase-snap` (deprecated) tool for this. > > Arguments: > > - `MEMORY_PATH` - path to the `memory` file > - `DIFF_PATH` - path to the `diff` file > > Usage: > > ```bash > snapshot-editor edit-memory rebase \ > --memory-path \ > --diff-path > ``` > > Example: > > ```bash > snapshot-editor edit-memory rebase \ > --memory-path ./memory_file \ > --diff-path ./diff_file > ``` ### `edit-vmstate` command #### `remove-regs` subcommand (aarch64 only) > This command is used to remove specified registers from vcpu states inside > vmstate snapshot file. > > Arguments: > > - `VMSTATE_PATH` - path to the `vmstate` file > - `OUTPUT_PATH` - path to the file where the output will be placed > - `[REGS]` - set of u32 values representing registers ids as they are defined > in KVM. Can be both in decimal and in hex formats. > > Usage: > > ```bash > snapshot-editor edit-vmstate remove-regs \ > --vmstate-path \ > --output-path \ > [REGS]... > ``` > > Example: > > ```bash > ./snapshot-editor edit-vmstate remove-regs \ > --vmstate-path ./vmstate_file \ > --output-path ./new_vmstate_file \ > 0x1 0x2 > ``` ### `info-vmstate` command #### `version` subcommand > This command is used to print version of the provided vmstate file. > > Arguments: > > - `VMSTATE_PATH` - path to the `vmstate` file > > Usage: > > ```bash > snapshot-editor info-vmstate version --vmstate-path > ``` > > Example: > > ```bash > ./snapshot-editor info-vmstate version --vmstate-path ./vmstate_file > ``` #### `vcpu-states` subcommand > This command is used to print the vCPU states inside vmstate snapshot file. > > Arguments: > > - `VMSTATE_PATH` - path to the `vmstate` file > > Usage: > > ```bash > snapshot-editor info-vmstate vcpu-states --vmstate-path > ``` > > Example: > > ```bash > ./snapshot-editor info-vmstate vcpu-states --vmstate-path ./vmstate_file > ``` #### `vm-state` subcommand > This command is used to print the vmstate of snapshot file in readable format > thus, making it easier to compare vmstate of 2 snapshots. > > Arguments: > > - `VMSTATE_PATH` - path to the `vmstate` file > > Usage: > > ```bash > snapshot-editor info-vmstate vm-state --vmstate-path > ``` > > Example: > > ```bash > ./snapshot-editor info-vmstate vm-state --vmstate-path ./vmstate_file > ``` --- ### Snapshotting/Snapshot Support # Firecracker Snapshotting ## Table of Contents - [What is microVM snapshotting?](#about-microvm-snapshotting) - [Snapshotting in Firecracker](#snapshotting-in-firecracker) - [Supported platforms](#supported-platforms) - [Overview](#overview) - [Snapshot files management](#snapshot-files-management) - [Performance](#performance) - [Developer preview status](#developer-preview-status) - [Limitations](#limitations) - [Firecracker Snapshotting characteristics](#firecracker-snapshotting-characteristics) - [Snapshot versioning](#snapshot-versioning) - [Snapshot API](#snapshot-api) - [Pausing the microVM](#pausing-the-microvm) - [Creating snapshots](#creating-snapshots) - [Creating full snapshots](#creating-full-snapshots) - [Creating diff snapshots](#creating-diff-snapshots) - [Resuming the microVM](#resuming-the-microvm) - [Loading snapshots](#loading-snapshots) - [Provisioning host disk space for snapshots](#provisioning-host-disk-space-for-snapshots) - [Ensure continued network connectivity for clones](#ensure-continued-network-connectivity-for-clones) - [Snapshot security and uniqueness](#snapshot-security-and-uniqueness) - [Secure and insecure usage examples](#usage-examples) - [Reusing snapshotted states securely](#reusing-snapshotted-states-securely) - [Userspace notifications of loading snapshots](#userspace-notifications-of-loading-snapshots) - [Vsock device reset](#vsock-device-reset) - [VMGenID device limitation](#vmgenid-device-limitation) - [Where can I resume my snapshots?](#where-can-i-resume-my-snapshots) ## About microVM snapshotting MicroVM snapshotting is a mechanism through which a running microVM and its resources can be serialized and saved to an external medium in the form of a `snapshot`. This snapshot can be later used to restore a microVM with its guest workload at that particular point in time. ## Snapshotting in Firecracker ### Supported platforms The Firecracker snapshot feature is supported on all CPU micro-architectures listed in [README](../../README.md#supported-platforms). ### Overview A Firecracker microVM snapshot can be used for loading it later in a different Firecracker process, and the original guest workload is being simply resumed. The original guest which the snapshot is created from, should see no side effects from this process (other than the latency introduced by the snapshot creation process). Both network and vsock packet loss can be expected on guests that are resumed from snapshots in another Firecracker process. It is also not guaranteed that the state of the network connections survives the process. Furthermore, vsock connections that are open when the snapshot is taken are closed, but existing vsock listen sockets in the guest still remain active and can accept new connections after resume (see [Vsock device reset](#vsock-device-reset)). In order to make restoring possible, Firecracker snapshots save the full state of the following resources: - the guest memory, - the emulated HW state (both KVM and Firecracker emulated HW). The state of the components listed above is generated independently, which brings flexibility to our snapshotting support. This means that taking a snapshot results in multiple files that are composing the full microVM snapshot: - the guest memory file, - the microVM state file, - zero or more disk files (depending on how many the guest had; these are **managed by the users**). The design allows sharing of memory pages and read only disks between multiple microVMs. When loading a snapshot, instead of loading at resume time the full contents from file to memory, Firecracker creates a [MAP_PRIVATE mapping](http://man7.org/linux/man-pages/man2/mmap.2.html) of the memory file, resulting in runtime on-demand loading of memory pages. Any subsequent memory writes go to a copy-on-write anonymous memory mapping. This has the advantage of very fast snapshot loading times, but comes with the cost of having to keep the guest memory file around for the entire lifetime of the resumed microVM. ### Snapshot files management The Firecracker snapshot design offers a very simple interface to interact with snapshots but provides no functionality to package or manage them on the host. The [threat containment model](../design.md#threat-containment) states that the host, host/API communication and snapshot files are trusted by Firecracker. To ensure a secure integration with the snapshot functionality, users need to secure snapshot files by implementing authentication and encryption schemes while managing their lifecycle or moving them across the trust boundary, like for example when provisioning them from a repository to a host over the network. Firecracker is optimized for fast load/resume, and it's designed to do some very basic sanity checks only on the vm state file. It only verifies integrity using a 64-bit CRC value embedded in the vm state file, but this is only a partial measure to protect against accidental corruption, as the disk files and memory file need to be secured as well. It is important to note that CRC computation is validated before trying to load the snapshot. Should it encounter failure, an error will be shown to the user and the Firecracker process will be terminated. ### Performance The Firecracker snapshot create/resume performance depends on the memory size, vCPU count and emulated devices count. The Firecracker CI runs snapshot tests on all [supported platforms](../../README.md#tested-platforms). ### Developer preview status Diff snapshots are still in developer preview while we are diving deep into how the feature can be combined with guest_memfd support in Firecracker. ### Limitations - High snapshot restoration latency when cgroups V1 are in use. We strongly recommend to deploy snapshots on cgroups V2 enabled hosts for the implied kernel versions - [related issue](https://github.com/firecracker-microvm/firecracker/issues/2129). - Guest network connectivity is not guaranteed to be preserved after resume. For recommendations related to guest network connectivity for clones please see [Network connectivity for clones](network-for-clones.md). - Snapshotting on arm64 works for both GICv2 and GICv3 enabled guests. However, restoring between different GIC version is not possible. - If a [CPU template](../cpu_templates/cpu-templates.md) is not used on x86_64, overwrites of `MSR_IA32_TSX_CTRL` MSR value will not be preserved after restoring from a snapshot. - Resuming from a snapshot that was taken during early stages of the guest kernel boot might lead to crashes upon snapshot resume. We suggest that users take snapshot after the guest microVM kernel has booted. Please see [VMGenID device limitation](#vmgenid-device-limitation). ## Firecracker Snapshotting characteristics - Fresh Firecracker microVMs are booted using `anonymous` memory, while microVMs resumed from snapshot load memory on-demand from the snapshot and copy-on-write to anonymous memory. - Resuming from a snapshot is optimized for speed, while taking a snapshot involves some extra CPU cycles for synchronously writing memory pages to the memory snapshot file. Taking a full snapshot of a microVM results in the full contents of guest memory being written to the snapshot, and particularly, in all guest memory being faulted in. - The _memory file_ and _microVM state file_ are generated by Firecracker on snapshot creation. The disk contents are _not_ explicitly flushed to their backing files. - The API calls exposing the snapshotting functionality have clear **Prerequisites** that describe the requirements on when/how they should be used. - The Firecracker microVM's MMDS config is included in the snapshot. However, the data store is not persisted across snapshots. - Configuration information for metrics and logs are not saved to the snapshot. These need to be reconfigured on the restored microVM. - On x86_64, if a vCPU has MSR_IA32_TSC_DEADLINE set to 0 when a snapshot is taken, Firecracker replaces it with the MSR_IA32_TSC value from the same vCPU. This is to guarantee that the vCPU will continue receiving TSC interrupts after restoring from the snapshot even if an interrupt is lost when taking a snapshot. ## Snapshot versioning The microVM state snapshot file uses a data format that has a version in the form of `MAJOR.MINOR.PATCH`. Each Firecracker binary supports a fixed version of the snapshot data format. When creating a snapshot, Firecracker will use the supported data format version. When loading snapshots, Firecracker will check that the snapshot version is compatible with the version it supports. More information about the snapshot data format and details about snapshot data format versions can be found at [versioning](./versioning.md). ## Snapshot API Firecracker exposes the following APIs for manipulating snapshots: `Pause`, `Resume` and `CreateSnapshot` can be called only after booting the microVM, while `LoadSnapshot` is allowed only before boot. ### Pausing the microVM To create a snapshot, first you have to pause the running microVM and its vCPUs with the following API command: ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PATCH 'http://localhost/vm' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "state": "Paused" }' ``` **Prerequisites**: The microVM is booted. Successive calls of this request keep the microVM in the `Paused` state. **Effects**: - _on success_: microVM is guaranteed to be `Paused`. - _on failure_: no side-effects. ### Creating snapshots > [!WARNING] > > Diff snapshot support is in developer preview. See > [this section](#developer-preview-status) for more info. Now that the microVM is paused, you can create a snapshot, which can be either a `full`one or a `diff` one. Full snapshots always create a complete, resume-able snapshot of the current microVM state and memory. Diff snapshots save at least the current microVM state and the memory accessed since the last snapshot (full or diff) in a sparse file (but they might include more pages than strictly needed due to technical limitation in Firecracker's ability to accurately track accesses). Diff snapshots are generally not resume-able, but must be merged with a base snapshot into a full snapshot. The exception here are diff snapshots of booted VMs, which are immediately resumable. In this context, we will refer to the base as the first memory file created by a `/snapshot/create` API call and the layer as a memory file created by a subsequent `/snapshot/create` API call. The order in which the snapshots were created matters and they should be merged in the same order in which they were created. To merge a `diff` snapshot memory file on top of a base, users should copy its content over the base. This can be done using the `rebase-snap` (deprecated) or `snapshot-editor` tools provided with the firecracker release: ```bash snapshot-editor edit-memory rebase \ --memory-path path/to/base \ --diff-path path/to/layer ``` After executing the command above, the base would be a resumable snapshot memory file describing the state of the memory at the moment of creation of the layer. More layers which were created later can be merged on top of this base. This process needs to be repeated for each layer until the one describing the desired memory state is merged on top of the base, which is constantly updated with information from previously merged layers. Please note that users should not merge state files which resulted from `/snapshot/create` API calls and they should use the state file created in the same call as the memory file which was merged last on top of the base. #### Creating full snapshots For creating a full snapshot, you can use the following API command: ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PUT 'http://localhost/snapshot/create' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "snapshot_type": "Full", "snapshot_path": "./snapshot_file", "mem_file_path": "./mem_file" }' ``` Details about the required and optional fields can be found in the [swagger definition](../../src/firecracker/swagger/firecracker.yaml). > [!NOTE] > > If the files indicated by `snapshot_path` and `mem_file_path` don't exist at > the specified paths, then they will be created right before generating the > snapshot. If they exist, the files will be truncated and overwritten. **Prerequisites**: The microVM is `Paused`. **Effects**: - _on success_: - The file indicated by `snapshot_path` (e.g. `/path/to/snapshot_file`) contains the devices' model state and emulation state. The one indicated by `mem_file_path`(e.g. `/path/to/mem_file`) contains a full copy of the guest memory. - The generated snapshot files are immediately available to be used (current process releases ownership). At this point, the block devices backing files should be backed up externally by the user. Please note that block device contents are only guaranteed to be committed/flushed to the host FS, but not necessarily to the underlying persistent storage (could still live in host FS cache). - If dirty page tracking is enabled, the snapshot creation resets then the dirtied page bitmap and marks all pages clean (from a dirty page tracking point of view). - _on failure_: no side-effects. **Notes**: - The separate block device file components of the snapshot have to be handled by the user. #### Creating diff snapshots For creating a diff snapshot, you should use the same API command, but with `snapshot_type` field set to `Diff`. > [!NOTE] > > If not specified, `snapshot_type` is by default `Full`. > > ```bash > curl --unix-socket /tmp/firecracker.socket -i \ > -X PUT 'http://localhost/snapshot/create' \ > -H 'Accept: application/json' \ > -H 'Content-Type: application/json' \ > -d '{ > "snapshot_type": "Diff", > "snapshot_path": "./snapshot_file", > "mem_file_path": "./mem_file" > }' > ``` **Prerequisites**: The microVM is `Paused`. Diff snapshots come in two flavors. If `track_dirty_pages` was set to `true` when configuring the `/machine-config` resource or when restoring from a snapshot via `/snapshot/load`, Firecracker will use KVM's dirty page log runtime functionality to ensure the diff snapshot only contains exactly pages that were written to since boot / snapshot restoration. If `track_dirty_pages` is not enabled, Firecracker uses the [`mincore(2)`][man mincore] syscall to determine which pages to include in the snapshot. As such, this mode of snapshot taking will only work _if swap is disabled_, as mincore does not consider pages written to swap to be "in core". This potentially results in bigger memory files (although they are still sparse), but avoids the runtime overhead of dirty page logging. > [!NOTE] > > Dirty page tracking negates most of the benefits of > [huge pages](../hugepages.md#known-limitations). **Effects**: - _on success_: - The file indicated by `snapshot_path` contains the devices' model state and emulation state, same as when creating a full snapshot. The one indicated by `mem_file_path` contains this time a **diff copy** of the guest memory; the diff consists of the memory pages which have been dirtied since the last snapshot creation or since the creation of the microVM, whichever of these events was the most recent. - All the other effects mentioned in the **Effects** paragraph from **Creating full snapshots** section apply here. - _on failure_: no side-effects. > [!NOTE] > > This is an example of an API command that enables dirty page tracking: > > ```bash > curl --unix-socket /tmp/firecracker.socket -i \ > -X PUT 'http://localhost/machine-config' \ > -H 'Accept: application/json' \ > -H 'Content-Type: application/json' \ > -d '{ > "vcpu_count": 2, > "mem_size_mib": 1024, > "smt": false, > "track_dirty_pages": true > }' > ``` Enabling this support enables KVM dirty page tracking, so it comes at a cost (which consists of CPU cycles spent by KVM accounting for dirtied pages); it should only be used when needed. Creating a snapshot has some minor effects on the currently running microVM: - The vsock device is [reset](#vsock-device-reset), causing the driver to terminate connection on resumption. - On x86_64, a notification for KVM-clock is injected to notify the guest about being paused. ### Resuming the microVM You can resume the microVM by sending the following API command: ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PATCH 'http://localhost/vm' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "state": "Resumed" }' ``` **Prerequisites**: The microVM is `Paused`. Successive calls of this request are ignored (microVM remains in the running state). **Effects**: - _on success_: microVM is guaranteed to be `Resumed`. - _on failure_: no side-effects. ### Loading snapshots If you want to load a snapshot, you can do that only **before** the microVM is configured (the only resources that can be configured prior are the logger and the metrics systems) by sending the following API command: ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PUT 'http://localhost/snapshot/load' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "snapshot_path": "./snapshot_file", "mem_backend": { "backend_path": "./mem_file", "backend_type": "File" }, "track_dirty_pages": true, "resume_vm": false }' ``` The `backend_type` field represents the memory backend type used for loading the snapshot. Accepted values are: - `File` - rely on the kernel to handle page faults when loading the contents of the guest memory file into memory. - `Uffd` - use a dedicated user space process to handle page faults that occur for the guest memory range. Please refer to [this](handling-page-faults-on-snapshot-resume.md) for more details on handling page faults in the user space. The meaning of `backend_path` depends on the `backend_type` chosen: - if using `File`, then `backend_path` should contain the path to the snapshot's memory file to be loaded. - when using `Uffd`, `backend_path` refers to the path of the unix domain socket used for communication between Firecracker and the user space process that handles page faults. When relying on the OS to handle page faults, the command below is also accepted. Note that `mem_file_path` field is currently under the deprecation policy. `mem_file_path` and `mem_backend` are mutually exclusive, therefore specifying them both at the same time will return an error. ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PUT 'http://localhost/snapshot/load' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "snapshot_path": "./snapshot_file", "mem_file_path": "./mem_file", "track_dirty_pages": true, "resume_vm": false }' ``` Details about the required and optional fields can be found in the [swagger definition](../../src/firecracker/swagger/firecracker.yaml). **Prerequisites**: A full memory snapshot and a microVM state file **must** be provided. The disk backing files, network interfaces backing TAPs and/or vsock backing socket that were used for the original microVM's configuration should be set up and accessible to the new Firecracker process (in which the microVM is resumed). These host-resources need to be accessible at the same relative paths to the new Firecracker process as they were to the original one. **Effects:** - _on success_: - The complete microVM state is loaded from snapshot into the current Firecracker process. - It then resets the dirtied page bitmap and marks all pages clean (from a diff snapshot point of view). - The loaded microVM is now in the `Paused` state, so it needs to be resumed for it to run. - The memory file (pointed by `backend_path` when using `File` backend type, or pointed by `mem_file_path`) **must** be considered immutable from Firecracker and host point of view. It backs the guest OS memory for read access through the page cache. External modification to this file corrupts the guest memory and leads to undefined behavior. - The file indicated by `snapshot_path`, that is used to load from, is released and no longer used by this process. - If `track_dirty_pages` is set, subsequent diff snapshots will be based on KVM dirty page tracking. - If `resume_vm` is set, the vm is automatically resumed if load is successful. - _on failure_: A specific error is reported and then the current Firecracker process is ended (as it might be in an invalid state). *Notes*: The `track_dirty_pages` configuration is not saved when creating a snapshot, so you need to explicitly set `track_dirty_pages` again when sending the `LoadSnapshot` command if you want to be able to do dirty page tracking based diff snapshots from a loaded microVM. It is also worth knowing, a microVM that is restored from snapshot will be resumed with the guest OS wall-clock continuing from the moment of the snapshot creation. For this reason, the wall-clock should be updated to the current time, on the guest-side. More details on how you could do this can be found at a [related FAQ](../../FAQ.md#my-guest-wall-clock-is-drifting-how-can-i-fix-it). When using `kvm-clock` as clock source on `x86_64`, it's possible to optionally set the `clock_realtime: true` in the `LoadSnapshot` request to advance the clock on the guest at restore time (host Linux >= 5.16 is required to support this feature). Note that this may cause issues within the guest as the clock will appear to suddenly jump. ## Provisioning host disk space for snapshots Depending on VM memory size, snapshots can consume a lot of disk space. Firecracker integrators **must** ensure that the provisioned disk space is sufficient for normal operation of their service as well as during failure scenarios. If the service exposes the snapshot triggers to customers, integrators **must** enforce proper disk quotas to avoid any DoS threats that would cause the service to fail or function abnormally. ## Ensure continued network connectivity for clones For recommendations related to continued network connectivity for multiple clones created from a single Firecracker microVM snapshot please see [this doc](network-for-clones.md). ## Snapshot security and uniqueness When snapshots are used in a such a manner that a given guest's state is resumed from more than once, guest information assumed to be unique may in fact not be; this information can include identifiers, random numbers and random number seeds, the guest OS entropy pool, as well as cryptographic tokens. Without a strong mechanism that enables users to guarantee that unique things stay unique across snapshot restores, we consider resuming execution from the same state more than once insecure. For more information please see [this doc](random-for-clones.md) ### Usage examples #### Example 1: secure usage ```console Boot microVM A -> ... -> Create snapshot S -> Terminate -> Load S in microVM B -> Resume -> ... ``` Here, microVM A terminates after creating the snapshot without ever resuming work, and a single microVM B resumes execution from snapshot S. In this case, unique identifiers, random numbers, and cryptographic tokens that are meant to be used once are indeed only used once. In this example, we consider microVM B secure. #### Example 2: potentially insecure usage ```console Boot microVM A -> ... -> Create snapshot S -> Resume -> ... -> Load S in microVM B -> Resume -> ... ``` Here, both microVM A and B do work starting from the state stored in snapshot S. Unique identifiers, random numbers, and cryptographic tokens that are meant to be used once may be used twice. It doesn't matter if microVM A is terminated before microVM B resumes execution from snapshot S or not. In this example, we consider both microVMs insecure as soon as microVM A resumes execution. #### Example 3: potentially insecure usage ```console Boot microVM A -> ... -> Create snapshot S -> ... -> Load S in microVM B -> Resume -> ... -> Load S in microVM C -> Resume -> ... [...] ``` Here, both microVM B and C do work starting from the state stored in snapshot S. Unique identifiers, random numbers, and cryptographic tokens that are meant to be used once may be used twice. It doesn't matter at which points in time microVMs B and C resume execution, or if microVM A terminates or not after the snapshot is created. In this example, we consider microVMs B and C insecure, and we also consider microVM A insecure if it resumes execution. ### Reusing snapshotted states securely [Virtual Machine Generation Identifier](https://learn.microsoft.com/en-us/windows/win32/hyperv_v2/virtual-machine-generation-identifier) (VMGenID) is a virtual device that allows VM guests to detect when they have resumed from a snapshot. It works by exposing a cryptographically random 16-bytes identifier to the guest. The VMM ensures that the value of the identifier changes every time the VM a time shift happens in the lifecycle of the VM, e.g. when it resumes from a snapshot. Linux supports VMGenID since version 5.18 for systems with ACPI support. Linux 6.10 added support also for systems that use DeviceTree instead of ACPI. When Linux detects a change in the identifier, it uses its value to reseed its internal PRNG. Firecracker supports VMGenID device both on x86 and Aarch64 platforms. Firecracker will always enable the device. During snapshot resume, Firecracker will update the 16-byte generation ID and inject a notification in the guest before resuming its vCPUs. As a result, guests that run Linux versions >= 5.18 will re-seed their in-kernel PRNG upon snapshot resume. User space applications can rely on the guest kernel for randomness. State other than the guest kernel entropy pool, such as unique identifiers, cached random numbers, cryptographic tokens, etc **will** still be replicated across multiple microVMs resumed from the same snapshot. Users need to implement mechanisms for ensuring de-duplication of such state, where needed. ## Userspace notifications of loading snapshots VMClock device ([specification](https://uapi-group.org/specifications/specs/vmclock/)) is a device that enables efficient application clock synchronization against real wallclock time, for applications running inside virtual machines. VMClock also takes care situations where there is some sort disruption happens to the clock. It handles these through fields in the [`vmlcock_abi`](https://uapi-group.org/specifications/specs/vmclock/#the-vmclock_abi-structure). Currently, it handles two cases: 1. Live migration through the `disruption_marker` field. 1. Restore from snapshots through the `vm_generation_counter`. Whenever a VM starts from a snapshot VMClock will present a new (different that what was previously stored) value in the `vm_generation_counter`. This happens in an atomic way, i.e. `vm_generation_counter` will include the new value as soon as vCPUs are resumed post snapshot loading. User space libraries, e.g. userspace PRNGs can mmap() `vmclock_abi` and monitor changes in `vm_generation_counter` to observe when they need to adapt and/or recreate state. Moreover, VMClock allows processes to call poll() on the VMClock device and get notified about changes through an event loop. For reference, the C code used in our tests is available [here](https://github.com/firecracker-microvm/firecracker/blob/main/tests/host_tools/vmclock.c). > [!IMPORTANT] > > Support for `vm_generation_counter` and `poll()` is implemented in Linux > through the patches > [here](https://lore.kernel.org/all/20260130173704.12575-1-itazur@amazon.com/) > and was merged in Linux kernel v7.0. Users need to make sure that the linked > patches are applied on their kernels. These patches have been backported to > Amazon Linux v5.10 and v6.1 microVM kernels, starting from > [microvm-kernel-5.10.252-285.992.amzn2](https://github.com/amazonlinux/linux/tree/microvm-kernel-5.10.252-285.992.amzn2) > and > [microvm-kernel-6.1.167-27.319.amzn2023](https://github.com/amazonlinux/linux/tree/microvm-kernel-6.1.167-27.319.amzn2023) > respectively. These patched kernels will be automatically downloaded when > running the integration test suite, or they can be manually downloaded using > `tools/devtool download_ci_artifacts`. ## Vsock device reset The vsock device is reset across snapshot/restore to avoid inconsistent state between device and driver leading to breakage ([#2218](https://github.com/firecracker-microvm/firecracker/issues/2218)). This is done by sending a `VIRTIO_VSOCK_EVENT_TRANSPORT_RESET` event to the guest driver during `SnapshotCreate` ([#2562](https://github.com/firecracker-microvm/firecracker/pull/2562)). On `SnapshotResume`, when the VM becomes active again, the vsock driver closes all existing connections. Existing listen sockets still remain active, but their CID is updated to reflect the current `guest_cid`. More details about this event can be found in the official Virtio document [here](https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-4080006). ## VMGenID device limitation During snashot resume, Firecracker updates the 16-byte generation ID of the VMGenID device and injects an interrupt in the guest before resuming vCPUs. If the snapshot was taken at the very early stages of the guest kernel boot process proper interrupt handling might not be in place yet. As a result, the kernel might not be able to handle the injected notification and crash. We suggest to users that they take snapshots only after the guest kernel has completed booting, to avoid this issue. ## Where can I resume my snapshots? Snapshots must be resumed on an software and hardware configuration which is identical to what they were generated on. However, in limited cases, snapshots can be resumed on identical hardware instances where they were taken on, but using newer host kernel versions. While we do not provide any guarantees on this setup (and do not recommend doing this in production), we are currently aware of the compatibility table reported below: | .metal instance type | taken on host kernel | restored on host kernel | | -------------------- | -------------------- | ----------------------- | | {m5n,m6i,m6a} | 5.10 | 6.1 | For example, a snapshot taken on a m6i.metal host (Intel Ice Lake) running a 5.10 host kernel can be restored on a different m6i.metal host running a 6.1 host kernel (but not vice versa), but could not be restored on a m5n.metal host (Intel Cascade Lake). [man mincore]: https://man7.org/linux/man-pages/man2/mincore.2.html --- ### Snapshotting/Versioning # Firecracker snapshot versioning This document describes how Firecracker persists microVM state into Firecracker snapshots. It describes the snapshot format, encoding, compatibility and limitations. ## Introduction Firecracker uses the serde crate [1] along with the bitcode [2] format to serialize its state into Firecracker snapshots. Firecracker snapshots have versions that are independent of Firecracker versions. Each Firecracker version declares support for a specific snapshot data format version. When creating a snapshot, Firecracker will use the supported snapshot format version. When loading a snapshot, Firecracker will check that format of the snapshot file is compatible with the snapshot version Firecracker supports. ## Overview Firecracker persists the microVM state as 2 separate objects: - a **guest memory** file - a **microVM state** file. *The block devices attached to the microVM are not considered part of the state and need to be managed separately.* ### Guest memory The guest memory file contains the microVM memory saved as a dump of all pages. ### MicroVM state In the VM state file, Firecracker stores the internal state of the VMM (device emulation, KVM and vCPUs) with 2 exceptions - serial emulation and vsock backend. While we continuously improve and extend Firecracker's features by adding new capabilities, devices or enhancements, the microVM state file may change both structurally and semantically with each new release. ## MicroVM state file format A Firecracker snapshot has the following format: | Field | Bits | Description | | -------- | ---- | --------------------------------------------------------- | | magic_id | 64 | Firecracker snapshot and architecture (x86_64/aarch64). | | version | M | The snapshot data format version (`MAJOR.MINOR.PATCH`) | | state | N | Bitcode blob containing the microVM state. | | crc | 64 | Optional CRC64 sum of magic_id, version and state fields. | The snapshot format has its own version encoded in the snapshot file itself after the snapshot's `magic_id`. The snapshot format version is independent of the Firecracker version and it is of the form `MAJOR.MINOR.PATCH`. Currently, Firecracker uses the [Serde bitcode encoder](https://github.com/SoftbearStudios/bitcode) for serializing the microVM state. The encoding format that bitcode uses does not allow backwards compatible changes in the state, so essentially every change in the microVM state description will result in bump of the format's `MAJOR` version. If the needs arises, we will look into alternative formats that allow more flexibility with regards to backwards compatibility. If/when this happens, we will define how changes in the snapshot format reflect to changes in its `MAJOR.MINOR.PATCH` version. ## VM state encoding During research and prototyping we considered multiple storage formats. The criteria used for comparing these are: performance, size, rust support, specification, versioning support, community and tooling. Performance, size and Rust support are hard requirements while all others can be the subject of trade offs. More info about this comparison can be found [here](https://github.com/firecracker-microvm/firecracker/blob/9d427b33d989c3225d874210f6c2849465941dc0/docs/snapshotting/design.md#snapshot-format). Key benefits of using *bitcode*: - Minimal snapshot size overhead - Minimal CPU overhead - Simple implementation The current implementation relies on the [Serde bitcode encoder](https://github.com/SoftbearStudios/bitcode). ## Snapshot compatibility ### Host kernel Snapshots can be saved and restored on the same kernel version without any issues. There might be issues when restoring snapshots created on different host kernel version even when using the same Firecracker version. SnapshotCreate and SnapshotLoad operations across different host kernels is considered unstable in Firecracker as the saved KVM state might have different semantics on different kernels. ### Device model The current Firecracker devices are backwards compatible up to the version that introduces them. Ideally this property would be kept over time, but there are situations when a new version of a device exposes new features to the guest that do not exist in an older version. In such cases restoring a snapshot at an older version becomes impossible without breaking the guest workload. The microVM state file links some resources that are external to the snapshot: - tap devices by device name, - block devices by block file path, - vsock backing Unix domain socket by socket name. To successfully restore a microVM one should check that: - tap devices are available, their names match their original names since these are the values saved in the microVM state file, and they are accessible to the Firecracker process where the microVM is being restored, - block devices are set up at their original relative or absolute paths with the proper permissions, as the Firecracker process with the restored microVM will attempt to access them exactly as they were accessed in the original Firecracker process, - the vsock backing Unix domain socket is available, its name matches the original name, and it is accessible to the new Firecracker process. ### CPU model Firecracker microVMs snapshot functionality is available for Intel/AMD/ARM64 CPU models that support the hardware virtualizations extensions, more details are available [here](../../README.md#supported-platforms). Snapshots are not compatible across CPU architectures and even across CPU models of the same architecture. They are only compatible if the CPU features exposed to the guest are an invariant when saving and restoring the snapshot. The trivial scenario is creating and restoring snapshots on hosts that have the same CPU model. Restoring from an Intel snapshot on AMD (or vice-versa) is not supported. It is important to note that guest workloads can still execute instructions that are being [masked](../cpu_templates/cpu-templates.md) by CPUID and restoring and saving of such workloads will lead to undefined result. Firecracker retrieves the state of a discrete list of MSRs from KVM, more specifically, the MSRs corresponding to the guest exposed features. ## Implementation The microVM state file format is implemented in the [snapshot crate](../../src/vmm/src/snapshot/mod.rs) in the Firecracker repository. All Firecracker devices implement the [Persist](../../src/vmm/src/snapshot/persist.rs) trait which exposes an interface that enables creating from and saving to the microVM state. [1]: https://serde.rs [2]: https://github.com/SoftbearStudios/bitcode --- ### Api Change Runbook # Runbook for Firecracker API changes This runbook will cover triaging API changes and ways to implement them appropriately. ## Definitions - *Deprecated* - We will consider a deprecated API element (endpoint and/or parts of an endpoint) to be an element which still provides users with access to its backing functionality and can be used, but will soon be removed completely along with said functionality in an upcoming version. - *Mandatory endpoint* - We will consider an endpoint mandatory if Firecracker cannot operate normally without performing a request to it. - *Optional endpoint* - We will consider an endpoint optional if Firecracker can operate normally without performing a request to it and the functionality behind it is not essential. - *Mandatory header/field* - We will consider a header/field mandatory in an HTTP message if the request will fail without specifying said header/field. - *Optional header/field* - We will consider a header/field optional in an HTTP message if the request will succeeds without specifying said header/field. ## Triaging API changes For the purposes of this document, there are 2 main categories for API changes, namely *breaking* and *non-breaking*. ### What is a breaking change? A breaking change in the API is a change that makes the API incompatible with the previous version (backwards incompatible). In an effort to avoid a breaking change, we may take the route of deprecation and incrementing the minor version in an effort to preserve backwards compatibility, but breaking changes will always ultimately result in incrementing the major version. Here is a non-exhaustive list of such changes: 1. Adding a new mandatory endpoint/HTTP method. 1. Removing an endpoint/method. 1. Adding a mandatory request header/field. 1. Removing a request header/field. 1. Adding a mandatory response field. 1. Removing a response header/field. ### What is NOT a breaking change? A change in the API is not a breaking change if the version resulting from it is compatible with the previous one (backwards compatible). The outcome of a non-breaking change should always include incrementing the minor version but must not lead to incrementing the major version by itself. Here is a non-exhaustive list of such changes: 1. Deprecating an endpoint/method/field. 1. Adding a new optional endpoint/method. 1. Adding an optional request header/field. 1. Adding a response header. 1. Adding additional valid inputs for fields in API requests. 1. Making mandatory headers/fields optional. 1. Making mandatory endpoints optional. 1. Changing the URI of an endpoint. 1. Changing the metrics output format. ## Implementing API changes API changes result in version increases. As Firecracker’s support policy is based on [semantic versioning 2.0.0][1], we will look at API changes from this point of view. > Given a version number MAJOR.MINOR.PATCH, increment the: MAJOR version when > you make incompatible API changes; MINOR version when you add functionality in > a backwards compatible manner; PATCH version when you make backwards > compatible bug fixes. *All deprecated endpoints are supported until at least the next major version release, where they may be _removed_.* ### How to follow the flowchart - with examples We will go through multiple types of API changes and provide ways to ensure we don’t break our backwards compatibility promise to our customers. The list is split into categories of components changed. - *Entire endpoints* - Adding an optional endpoint with new functionality - Increment minor version. - Adding a command line parameter - Increment minor version. - Removing an endpoint - Deprecate endpoint and increment minor version → Remove endpoint when incrementing major version. - Adding a mandatory endpoint - Increment major version. - *Request* - Adding an optional header/field - Increment minor version. - Renaming a header/field - Accept both names and deprecate the old one → Remove old name when incrementing major version. - Removing a header/field - Make said header/field optional → Remove header/field when incrementing major version. - Changing the URI of an endpoint - Redirect the old endpoint to the new one and deprecate the old one → Remove old endpoint when incrementing major version. - Adding a mandatory header/field - Increment major version. - *Response* - Adding a header/field - Create a new, separate endpoint with the changes and deprecate the old one → Remove old endpoint when incrementing major version. - Removing a header/field - Create a new, separate endpoint with the changes and deprecate the old one → Remove old endpoint when incrementing major version. - *Command line parameter* - Renaming a command line parameter - Accept both names and deprecate the old one → Remove old name when incrementing major version. - Changing expected value taken by a command line parameter - Accept both names and deprecate the old one → Remove old name when incrementing major version. In case the outlined solution for your case is not feasible (e.g. because of security concerns), break the glass and increment the major version. ## How to deprecate As outlined in the diagram above, sometimes we have to deprecate endpoints partially or entirely. In this section we will go through different situations where we have to deprecate something and ways of avoiding common pitfalls when doing so. ### Deprecating endpoints Some paths in the flowchart above lead to deprecation. Based on the initial conditions, there are 2 major cases where we need to deprecate an endpoint: - *Changing an existing endpoint* - Often happens because directly changing the endpoint would be a breaking change. - We usually create a clone of the old endpoint we want to deprecate and make the necessary changes to it. - We usually expose both endpoints in the next minor version while marking the old one as deprecated. - The old endpoint retains its previous name. When naming the new endpoint: - for HTTP endpoints we follow a “per-endpoint versioning” scheme; in cases where we can’t find a fitting name for the new endpoint, the simplest way forward is to take the old URI and append `/v2` to it. - for command line endpoints, we can usually find a different name for the new endpoint. - *Deprecating an endpoint without adding a replacement to it* - Often happens when we want to phase out a certain feature or functionality, but doing so immediately would be a breaking change. - We just mark the endpoint as deprecated. ### Keeping Swagger updated Make sure that any changes you make in the code are also reflected in the swagger specification. Some tips: - There is nothing in the swagger file that shows whether an endpoint is mandatory or optional, it’s all code logic. - Mandatory fields in a request or response body are marked with `required: true` in the swagger definition. All other fields are optional. - If you need to redirect an endpoint, you have to clone the old one under the new URI in the swagger specification. ### Marking endpoints as deprecated When marking: - an HTTP endpoint as deprecated: - Add a comment for the parsing function of the endpoint stating that it is deprecated. - Log a `warn!` message stating that the user accessed a deprecated endpoint. - Increment the `deprecatedHttpApi` metric. - Include the `Deprecated` header in the response. - a header field in an HTTP endpoint as deprecated: - Add a comment in the parsing function where we check the presence of the header stating that it is deprecated. - If the header is present, log a `warn!` message stating that the user used a deprecated field. - Increment the `deprecatedHttpApi` metric. - Include the Deprecated header in the response. - a command line parameter as deprecated: - Mention it is deprecated in the help message of the parameter in the argument parser. - Add it in the `warn_deprecated_parameters` function where we log it and increment the `deprecatedCmdLineApi` metric. ### Removing deprecated endpoints on a major release When doing a major release, the API can have breaking changes. This is the _only time_ where we can safely remove deprecated elements of the API. To remove a deprecated element of the API: - Remove the associated functionality from the codebase (usually in `vmm` or `mmds`); - Remove the parsing logic in `api_server`; - Remove any unit and integration tests associated with this element. ## Practical example of an API change In this guide we set out to remove the `vsock_id` field in `PUT`s on `/vsock`. This was implemented in [PR #2763][2] and we will go step by step through the changes in order to understand the process of changing something in the Firecracker API. - We go through the flowchart; we want to remove a field in the body of a HTTP request. So we follow the flowchart like this: - → Change an existing endpoint - → Request - → Remove header or field - → Make it optional - → Deprecate - → Increment minor version. - Now that we know we need to make the field optional and deprecate it, it’s time for the code changes (reference implementation in [this commit][3]). We go to the function in `api_server/src/requests` which is responsible for parsing this request, which is `parse_put_vsock` in this case, and do the following. - We find the associated `vmm_config` struct which `serde_json` uses for deserialization, in this case `VsockDeviceConfig`. - In the struct referenced above, we make the parameter optional by encapsulating it in an `Option` with `#[serde(default)]` and `#[serde(skip_serializing_if = "Option::is_none")]` so that we don’t break existing implementations, but we follow the new, desired usage of the endpoint. - After deserializing the body of the request into the struct, we check for the existence of the field we want to deprecate, in this case by calling `vsock_cfg.vsock_id.is_some()`. - If the field is there, we must mark this request as being deprecated, so we craft a deprecation message (`"PUT /vsock: vsock_id field is deprecated."`) and increment the deprecated HTTP API metric (`METRICS.deprecated_api.deprecated_http_api_calls.inc()`). - We create a new `ParsedRequest` where, if we marked the request as deprecated, we append the deprecation message into its `parsing_info` structure, in this case by calling `parsed_req.parsing_info().append_deprecation_message(msg)`. - Don’t forget to comment your code! Comments should reflect what is deprecated and clearly describe the code paths where you handle the deprecation case. - Add a unit test where you test your new code paths. - Fix all other failing unit tests. - Update the swagger file to reflect the change, in this case by removing the `vsock_id` field from the required parameter list in the `Vsock` definition and adding a description to it stating that it is deprecated since the current version. - Update any relevant documentation. - We update the python integration tests to reflect the change (reference implementation in [this commit][4]). - We refactor the relevant `tests/integration_tests/functional/test_api.py` test to use the artifact model instead of the fixture one. If the test already uses the artifact model, you can skip this step. - We make sure to run the test with the current build, as well as with future Firecracker versions by specifying the unreleased version in the `min_version` parameter of `artifacts.firecrackers()`. We do this in order to ensure that, when we create patch releases on older branches, we test the API with future binaries to enforce backwards compatibility. _Disclaimer_: This test will fail when running with the binary artifact fetched from S3 until you update the binary there with your current build. You should only do this once your PR has all necessary approves and this test is the last thing keeping it from getting merged. - We check that, when the deprecated field is present in the request, the `Deprecation` header is also present in the response by asserting `response.headers['deprecation']`. We do not also check that the header is not present when the field is not present because, in a future version, some other field may be deprecated in the same request and would return the header anyway, resulting in a fail in our test when it shouldn’t. - Fix all other failing integration tests. [1]: https://semver.org/spec/v2.0.0.html [2]: https://github.com/firecracker-microvm/firecracker/pull/2763 [3]: https://github.com/firecracker-microvm/firecracker/commit/83aa098245a42ad93a6b70ccd70ad593cf453a3c [4]: https://github.com/firecracker-microvm/firecracker/commit/472a81dbccd89562578919b76d87c30ee7db17aa --- ### Ballooning # Using the balloon device with Firecracker ## What is the balloon device A memory balloon device is a virtio device that can be used to reclaim and give back guest memory through API commands issued by the host. It does this by allocating memory in the guest, and then sending the addresses of that memory to the host; the host may then remove that memory at will. The device is configured through a number of options, and an integer, which represents the target size of the balloon, in MiB. The options cannot be changed during operation, but the target size can be changed. The behaviour of the balloon is the following: while the actual size of the balloon (i.e. the memory it has allocated) is smaller than the target size, it continually tries to allocate new memory -- if it fails, it prints an error message (`Out of puff! Can't get %d pages`), sleeps for 0.2 seconds, and then tries again. While the actual size of the balloon is larger than the target size, it will free memory until it hits the target size. The device can be configured with the following options: - `deflate_on_oom`: if this is set to `true` and a guest process wants to allocate some memory which would make the guest enter an out-of-memory state, the kernel will take some pages from the balloon and give them to said process instead asking the OOM killer process to kill some processes to free memory. Note that this applies to physical page allocations in the kernel which belong to guest processes. This does not apply to instances when the kernel needs memory for its activities (i.e. constructing caches), when the user requests more memory than the currently available to the balloon for releasing, or when guest processes try to allocate large amounts of memory that are refused by the guest memory manager, which is possible when the guest runs with `vm.overcommit_memory=0` and the allocation does not pass the MM basic checks. Setting `vm.memory_overcommit` to 1 would make the MM approve all allocations, no matter how large, and using the memory mapped for those allocations will always deflate the balloon instead of making the guest enter an OOM state. Note: we do not recommend running with `vm.overcommit_memory=1` because it requires complete control over what allocations are done in the guest and can easily result in unexpected OOM scenarios. - `stats_polling_interval_s`: unsigned integer value which if set to 0 disables the virtio balloon statistics and otherwise represents the interval of time in seconds at which the balloon statistics are updated. The device has two optional features which can be enabled with the following options: - `free_page_reporting`: A mechanism for the guest to continually report ranges of memory which the guest is not using and can be reclaimed. [Read more here](#virtio-balloon-free-page-reporting) - [(Developer Preview)](../docs/RELEASE_POLICY.md#developer-preview-features) `free_page_hinting`: A mechanism to reclaim memory from the guest, this is instead triggered from the host. [Read more here](#virtio-balloon-free-page-hinting) ## Security disclaimer **The balloon device is a paravirtualized virtio device that requires cooperation from a driver in the guest.** In normal conditions, the balloon device will: - not change the target size, which is set directly by the host - consume exactly as many pages as required to achieve the target size - correctly update the value of the actual size of the balloon seen by the host - not use pages that were previously inflated if they were not returned to the guest via a deflate operation (unless the `deflate_on_oom` flag was set and the guest is in an out of memory state) - provide correct statistics when available However, Firecracker does not and cannot introspect into the guest to check the integrity of the balloon driver. As the guest is not trusted, if the driver in the guest becomes compromised, the above statements are **no longer guaranteed**. This means that even though users use the balloon to impose restrictions on memory usage, they can be broken by a compromised driver in the guest. The balloon device operates on a best effort model and users should always ensure the host is prepared to handle a situation in which the Firecracker process uses all of the memory it was given at boot even if the balloon was used to restrict the amount of memory available to the guest. It is also the users' responsibility to monitor the memory consumption of the VM and, in case unexpected increases in memory usage are observed, we recommend the following options: - migrate the VM to a machine with higher memory availability through snapshotting at the cost of disrupting the workload; - kill the Firecracker process that exceeds memory restrictions; - enable swap with a sufficient amount of memory to handle the demand at the cost of memory access speed; Users should also never rely solely on the statistics provided by the balloon when controlling the Firecracker process as they are provided directly by the guest driver and should always be viewed as an indication rather than a guarantee of what the memory state looks like in the guest. Please note that even in the case where the driver is not working properly, the balloon will never leak memory from one Firecracker process to another, nor can a guest within Firecracker access information in memory outside its own guest memory. In other words, memory cannot leak in or out of Firecracker if the driver becomes corrupted. This is guaranteed by the fact that the page frame numbers coming from the driver are checked to be inside the guest memory, then `madvise`d with the `MADV_DONTNEED` flag, which breaks the mappings between host physical memory (where the information is ultimately stored) and Firecracker virtual memory, which is what Firecracker uses to build the guest memory. On subsequent accesses on previously `madvise`d memory addresses, the memory is zeroed. Furthermore, the guest memory is `mmap`ped with the `MAP_PRIVATE` and `MAP_ANONYMOUS` flags, which ensure that even if a Firecracker yields some information through an inflate and that same physical page containing the information is mapped onto another Firecracker process, reads on that address space will see zeroes. ## Prerequisites To support memory ballooning, you must use a kernel that has the memory ballooning driver installed (on Linux 4.14.193, the relevant settings are `CONFIG_MEMORY_BALLOON=y`, `CONFIG_VIRTIO_BALLOON=y`). Other than that, only the requirements mentioned in the `getting-started` document are needed. ## Installing the balloon device In order to use a balloon device, you must install it during virtual machine setup (i.e. before starting the virtual machine). This can be done either through a PUT request on "/balloon" or by inserting the balloon into the JSON configuration file given as a command line argument to the Firecracker process. Here is an example command on how to install the balloon through the API: ```console socket_location=... amount_mib=... deflate_on_oom=... polling_interval=... curl --unix-socket $socket_location -i \ -X PUT 'http://localhost/balloon' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d "{ \"amount_mib\": $amount_mib, \ \"deflate_on_oom\": $deflate_on_oom, \ \"stats_polling_interval_s\": $polling_interval \ }" ``` To use this, set `socket_location` to the location of the firecracker socket (by default, at `/run/firecracker.socket`. Then, set `amount_mib`, `deflate_on_oom` and `stats_polling_interval_s` as desired: `amount_mib` represents the target size of the balloon, and `deflate_on_oom` and `stats_polling_interval_s` represent the options mentioned before. To install the balloon via the JSON config file, insert the following JSON object into your configuration file: ```console "balloon": { "amount_mib": 0, "deflate_on_oom": false, "stats_polling_interval_s": 1 }, ``` After installing the balloon device, users can poll the configuration of the device at any time by sending a GET request on "/balloon". Here is an example of such a request: ```console socket_location=... curl --unix-socket $socket_location -i \ -X GET 'http://localhost/balloon' \ -H 'Accept: application/json' ``` On success, this request returns a JSON object of the same structure as the one used to configure the device (via a PUT request on "/balloon"). ## Operating the traditional balloon device After it has been installed, the balloon device can only be operated via the API through the following command: ```console socket_location=... amount_mib=... polling_interval=... curl --unix-socket $socket_location -i \ -X PATCH 'http://localhost/balloon' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d "{ \"amount_mib\": $amount_mib, \ \"stats_polling_interval_s\": $polling_interval \ }" ``` This will update the target size of the balloon to `amount_mib` and the statistics polling interval to `polling_interval`. > [!NOTE] Balloon inflation instructs the guest to reclaim memory which may > cause performance issues in the guest. The balloon statistics defined > [below](#virtio-balloon-statistics) can be used to decide whether it's > necessary to reclaim memory. ## Virtio balloon statistics The statistics are enabled by setting the `stats_polling_interval_s` field in the balloon configuration to a non-zero value. If enabled, users can receive the latest balloon statistics by issuing a GET request on "/balloon". Here is an example of such a request: ```console socket_location=... curl --unix-socket $socket_location -i \ -X GET 'http://localhost/balloon/statistics' \ -H 'Accept: application/json' ``` The request, if successful, will return a JSON object containing the latest statistics. The JSON object contains information about the target and actual sizes of the balloon as well as virtio traditional memory balloon statistics. The target and actual sizes of the balloon are expressed as follows: - `target_pages`: The target size of the balloon, in 4K pages. - `actual_pages`: The number of 4K pages the device is currently holding. - `target_mib`: The target size of the balloon, in MiB. - `actual_mib`: The number of MiB the device is currently holding. These values are taken directly from the config space of the device and are always up to date, in the sense that they are exactly what the Firecracker process reads when polling the config space. The `actual` fields being accurate are subject to the guest driver working correctly. As defined in the virtio 1.1 specification, the traditional virtio balloon device has support for the following statistics: - `VIRTIO_BALLOON_S_SWAP_IN`: The amount of memory that has been swapped in (in bytes). - `VIRTIO_BALLOON_S_SWAP_OUT`: The amount of memory that has been swapped out to disk (in bytes). - `VIRTIO_BALLOON_S_MAJFLT`: The number of major page faults that have occurred. - `VIRTIO_BALLOON_S_MINFLT`: The number of minor page faults that have occurred. - `VIRTIO_BALLOON_S_MEMFREE`: The amount of memory not being used for any purpose (in bytes). - `VIRTIO_BALLOON_S_MEMTOT`: The total amount of memory available (in bytes). - `VIRTIO_BALLOON_S_AVAIL`: An estimate of how much memory is available (in bytes) for starting new applications, without pushing the system to swap. - `VIRTIO_BALLOON_S_CACHES`: The amount of memory, in bytes, that can be quickly reclaimed without additional I/O. Typically these pages are used for caching files from disk. - `VIRTIO_BALLOON_S_HTLB_PGALLOC`: The number of successful hugetlb page allocations in the guest. - `VIRTIO_BALLOON_S_HTLB_PGFAIL`: The number of failed hugetlb page allocations in the guest. Since linux v6.12, following metrics added(omitted in < v6.12): - `VIRTIO_BALLOON_S_OOM_KILL`: OOM killer invocations, indicating critical memory pressure. - `VIRTIO_BALLOON_S_ALLOC_STALL`: Counter of Allocation enter a slow path to gain more memory page. The reclaim/scan metrics can reveal what is actually happening. - `VIRTIO_BALLOON_S_ASYNC_SCAN`: Amount of memory scanned asynchronously. - `VIRTIO_BALLOON_S_DIRECT_SCAN`: Amount of memory scanned directly. - `VIRTIO_BALLOON_S_ASYNC_RECLAIM`: Amount of memory reclaimed asynchronously. - `VIRTIO_BALLOON_S_DIRECT_RECLAIM`: Amount of memory reclaimed directly. When the pages_high watermark is reached, Linux `kswapd` performs asynchronous page reclaim, which increases ASYNC_SCAN and ASYNC_RECLAIM. When a process allocates more memory than the kernel can provide, the process is stalled while pages are reclaimed directly, which increases DIRECT_SCAN and DIRECT_RECLAIM. > `man sar`: %vmeff Calculated as pgsteal(RECLAIM) / pgscan(SCAN), this is a > metric of the efficiency of page reclaim. If it is near 100% then almost every > page coming off the tail of the inactive list is being reaped. If it gets too > low (e.g. less than 30%) then the virtual memory is having some difficulty. The driver is queried for updated statistics every time the amount of time specified in that field passes. The driver may not provide all the statistics when queried, in which case the old values of the missing statistics are preserved. To change the statistics polling interval, users can sent a PATCH request on "/balloon/statistics". Here is an example of such a request: ```console socket_location=... polling_interval=... curl --unix-socket $socket_location -i \ -X PATCH 'http://localhost/balloon' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d "{ \"stats_polling_interval_s\": $polling_interval }" ``` This will change the statistics polling interval to `polling_interval`. Note that if the balloon was configured without statistics pre-boot, the statistics cannot be enabled later by providing a `polling_interval` non-zero value. Furthermore, if the balloon was configured with statistics pre-boot through a non-zero `stats_polling_interval_s` value, the statistics cannot be disabled through a `polling_interval` value of zero post-boot. ## Virtio balloon free page reporting Free page reporting is a virtio balloon feature which allows the guest OS to report ranges of memory which are not being used. In Firecracker, the balloon device will `madvise` the range with the `MADV_DONTNEED` flag, reducing the RSS of the guest. Reporting can only be enabled pre-boot and will run continually with no option to stop it running. The feature also requires the guest to have the Linux kernel config option `PAGE_REPORTING` enabled. To enable free page reporting when creating the balloon device, the `free_page_reporting` attribute should be set in the JSON object. An example of how to configure the device to enable free page reporting: ```console socket_location=... amount_mib=... deflate_on_oom=... polling_interval=... curl --unix-socket $socket_location -i \ -X PUT 'http://localhost/balloon' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d "{ \"amount_mib\": $amount_mib, \ \"deflate_on_oom\": $deflate_on_oom, \ \"stats_polling_interval_s\": $polling_interval, \ \"free_page_reporting\": true \ }" ``` The Linux driver uses a hook in the free page path to trigger the reporting process, which will begin after a short delay (~2 seconds) and report the ranges. The runtime impact of this feature is heavily workload dependent. The driver gets ranges from the buddy allocator with a minimum page order. This page order dictates the minimum size of ranges reported and can be configured with the `page_reporting_order` module parameter in the guest kernel. The page order comes with trade-offs between performance and memory reclaimed; a good target to maximise memory reclaim is to have the reported ranges match the backing page size. ## Virtio balloon free page hinting Free page hinting is a [developer-preview](../docs/RELEASE_POLICY.md#developer-preview-features) feature, which allows the guest driver to report ranges of memory which are not being used. In Firecracker, the balloon device will `madvise` the range with the `MADV_DONTNEED` flag, reducing the RSS of the guest. Free page hinting differs from reporting as this is instead initiated from the host side, giving more flexibility on when to reclaim memory. To enable free page hinting when creating the balloon device, the `free_page_hinting` attribute should be set in the JSON object. An example of how to configure the device to enable free page hinting: ```console socket_location=... amount_mib=... deflate_on_oom=... polling_interval=... curl --unix-socket $socket_location -i \ -X PUT 'http://localhost/balloon' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d "{ \"amount_mib\": $amount_mib, \ \"deflate_on_oom\": $deflate_on_oom, \ \"stats_polling_interval_s\": $polling_interval, \ \"free_page_hinting\": true \ }" ``` Free page hinting is initiated and managed by Firecracker, the core mechanism to control the run is with the `cmd_id`. When Firecracker sets the `cmd_id` to a new number, the driver will acknowledge this and start reporting ranges, which Firecracker will free. Once the device has reported all the ranges it can find, it will update the `cmd_id` to reflect this. The device will then hold these ranges until Firecracker sends the stop command which allows the guest driver to reclaim the memory. The time required for the guest to complete a hinting run is dependant on a multitude of different factors and is mostly dictated by the guest, however, in testing the average time is ~200 milliseconds for a 1GB VM. This control mechanism in Firecracker is managed through three separate endpoints `/balloon/hinting/start`, `/balloon/hinting/status` and `/balloon/hinting/stop`. For simple operation, call the start endpoint with `acknowledge_on_stop = true`, which will automatically send the stop command once the driver has finished. An example of sending this command: ```console curl --unix-socket $socket_location -i \ -X POST 'http://localhost/balloon/hinting/start' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d "{ \"acknowledge_on_stop\": true \ }" ``` For fine-grained control, using `acknowledge_on_stop = false`, Firecracker will not send the acknowledge message. This can be used to get the guest to hold onto more memory. Using the `/status` endpoint, you can get information about the last `cmd_id` sent by Firecracker and the last update from the guest. An example of the status request and response: ```console curl --unix-socket $socket_location -i \ -X GET 'http://localhost/balloon/hinting/status' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' ``` Response: ```json { "host_cmd": 1, "guest_cmd": 2 } ``` An example of the stop endpoint: ```console curl --unix-socket $socket_location -i \ -X POST 'http://localhost/balloon/hinting/stop' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d "{}" ``` On snapshot restore, the `cmd_id` is **always** set to the stop `cmd_id` to allow the guest to reclaim the memory. If you have a particular use-case which requires this not to be the case, please raise an issue with a description of your scenario. > [!WARNING] > > Free page hinting was primarily designed for live migration, because of this > there is a caveat to the device spec which means the guest is able to reclaim > memory before Firecracker even receives the range to free. This can lead to a > scenario where the device frees memory that has been reclaimed in the guest, > potentially corrupting memory. The chances of this race happening are low, but > not impossible; hence the developer-preview status. > > We are currently working with the kernel community on a feature that will > eliminate this race. Once this has been resolved, we will update the device. > > One way to safely use this feature when using UFFD is: > > 1. Enable `WRITEPROTECT` on the VM memory before starting a hinting run. > 1. Track ranges that are written to. > 1. Skip these ranges when Firecracker reports them for freeing. > > This will prevent ranges which have been reclaimed from being freed. ## Balloon Caveats - Firecracker has no control over the speed of inflation or deflation; this is dictated by the guest kernel driver. - The traditional balloon will continually attempt to reach its target size, which can be a CPU-intensive process. It is therefore recommended to set realistic targets or, after a period of stagnation in the inflation, update the target size to be close to the inflated size. - The `deflate_on_oom` flag is a mechanism to prevent the guest from crashing or terminating processes; it is not meant to be used continually to free memory. Doing this will be a CPU-intensive process, as the traditional balloon driver is designed to deflate and release memory slowly. This is also compounded if the balloon has yet to reach its target size, as it will attempt to inflate while also deflating. --- ### Design # Firecracker Design ## Scope ### What is Firecracker Firecracker is a new virtualization technology that enables customers to deploy lightweight *micro* Virtual Machines or microVMs. Firecracker microVMs combine the security and workload isolation properties of traditional VMs with the speed, agility and resource efficiency enabled by containers. They provide a secure, trusted environment for multi-tenant services, while maintaining minimal overhead. The scope of this document is to describe the features and architecture of the Firecracker virtual machine manager (VMM). ### Features 1. Firecracker can safely run workloads from different customers on the same machine. 1. Customers can create microVMs with any combination of vCPU (up to 32) and memory to match their application requirements. 1. Firecracker microVMs can oversubscribe host CPU and memory. The degree of oversubscription is controlled by customers, who may factor in workload correlation and load in order to ensure smooth host system operation. 1. With a microVM configured with a minimal Linux kernel, single-core CPU, and 128 MiB of RAM, Firecracker supports a steady mutation rate of 5 microVMs per host core per second (e.g., one can create 180 microVMs per second on a host with 36 physical cores). 1. The number of Firecracker microVMs running simultaneously on a host is limited only by the availability of hardware resources. 1. Each microVM exposes a host-facing API via an in-process HTTP server. 1. Each microVM provides guest-facing access to host-configured metadata via the `/mmds` API. ### Specifications Firecracker's technical specifications are available in the [Specifications document](../SPECIFICATION.md). ## Host Integration The following diagram depicts an example host running Firecracker microVMs. Firecracker runs on Linux hosts and with Linux guest OSs (from this point on, referred to as guests). For a complete list of currently supported kernel versions, check out the [kernel support policy](kernel-policy.md). In production environments, Firecracker should be started only via the `jailer` binary. See [Sandboxing](#Sandboxing) for more details. After launching the process, users interact with the Firecracker API to configure the microVM, before issuing the `InstanceStart` command. ### Host Networking Integration Firecracker emulated network devices are backed by TAP devices on the host. To make use of Firecracker, we expect our customers to leverage on-host networking solutions. ### Storage Firecracker emulated block devices are backed by files on the host. To be able to mount block devices in the guest, the backing files need to be pre-formatted with a filesystem that the guest kernel supports. ## Internal Architecture Each Firecracker process encapsulates one and only one microVM. The process runs the following threads: API, VMM and vCPU(s). The API thread is responsible for Firecracker's API server and associated control plane. It's never in the fast path of the virtual machine. The VMM thread exposes the machine model, minimal legacy device model, microVM metadata service (MMDS) and VirtIO device emulated Net, Block and Vsock devices, complete with I/O rate limiting. In addition to them, there are one or more vCPU threads (one per guest CPU core). They are created via KVM and run the `KVM_RUN` main loop. They execute synchronous I/O and memory-mapped I/O operations on devices models. ### Threat Containment From a security perspective, all vCPU threads are considered to be running malicious code as soon as they have been started; these malicious threads need to be contained. Containment is achieved by nesting several trust zones which increment from least trusted or least safe (guest vCPU threads) to most trusted or safest (host). These trusted zones are separated by barriers that enforce aspects of Firecracker security. For example, all outbound network traffic data is copied by the Firecracker I/O thread from the emulated network interface to the backing host TAP device, and I/O rate limiting is applied at this point. These barriers are marked in the diagram below. Firecracker does not perform any network traffic filtering. All egress traffic from a guest is therefore considered untrusted, and should be filtered at the host-level. ## Components and Features ### Machine Model #### Layout Firecracker provides guests with storage and network access via emulated VirtIO Net and VirtIO Block devices. It also exposes a serial console and partial keyboard controller, the latter being used by guests to reset the VM (either soft or hard reset). Within Firecracker, the purpose of the I8042 device is to signal the microVM that the guest has requested a reboot. In addition to the Firecracker provided device models, guests also see the Programmable Interrupt Controllers (PICs), the I/O Advanced Programmable Interrupt Controller (IOAPIC), and the Programmable Interval Timer (PIT) that KVM supports. #### Exposing the CPU to the guest Firecracker allows control of what processor information is exposed to the guest by using [CPU templates](cpu_templates/cpu-templates.md). CPU templates can be set via the Firecracker API. Users can choose from existing static CPU templates and/or creating their own custom CPU templates. #### Clocksources available to guests Firecracker exposes the following clock sources to guests: - x86_64: kvm-clock and tsc. Linux guests >=5.10 will pick tsc by default if stable. - aarch64: arch_sys_counter ### I/O: Storage, Networking and Rate Limiting Firecracker provides VirtIO/block and VirtIO/net emulated devices, along with the application of rate limiters to each volume and network interface to make sure host hardware resources are used fairly by multiple microVMs. These are implemented using a token bucket algorithm based on two buckets. One is associated with the number of operations per second and the other one with the bandwidth. The customer can create and configure rate limiters via the API by specifying token bucket configurations for ingress and egress. Each token bucket is defined via the bucket size, I/O cost, refill rate, maximum burst, and initial value. This enables the customer to define flexible rate limiters that support bursts or specific bandwidth/operations limitations. For vhost-user devices, customers should implement rate limiting on the side of the vhost-user backend that they provide. ### MicroVM Metadata Service Firecracker microVMs expose access to a minimal MicroVM-Metadata Service (MMDS) to the guest through the API endpoint. The metadata stored by the service is fully configured by users. ### Sandboxing #### __Firecracker process__ Firecracker is designed to assure secure isolation using multiple layers. The first layer of isolation is provided by the Linux KVM and the Firecracker virtualization boundary. To assure defense in depth, Firecracker should only run constrained at the process level. This is achieved by the following: seccomp filters for disallowing unwanted system calls, cgroups and namespaces for resource isolation, and dropping privileges by jailing the process. Seccomp filters are automatically installed by Firecracker, while for the latter, we recommend starting Firecracker with the `jailer` binary that's part of each Firecracker release. ##### Seccomp Seccomp filters are used by default to limit the host system calls Firecracker can use. The default filters only allow the bare minimum set of system calls and parameters that Firecracker needs in order to function correctly. The filters are loaded in the Firecracker process, on a per-thread basis, before executing any guest code. For more information, check out the [seccomp documentation](seccomp.md). #### __Jailer process__ The Firecracker process can be started by another `jailer` process. The jailer sets up system resources that require elevated permissions (e.g., cgroup, chroot), drops privileges, and then exec()s into the Firecracker binary, which then runs as an unprivileged process. Past this point, Firecracker can only access resources that a privileged third-party grants access to (e.g., by copying a file into the chroot, or passing a file descriptor). ##### Cgroups and Quotas Each Firecracker microVM can be further encapsulated into a cgroup. By setting the affinity of the Firecracker microVM to a node via the cpuset subsystem, one can prevent the migration of said microVM from one node to another, something that would impair performance and cause unnecessary contention on shared resources. In addition to setting the affinity, each Firecracker microVM can have its own dedicated quota of the CPU time via the cpu subsystem, thus guaranteeing that resources are fairly shared across Firecracker microVMs. ### Monitoring Firecracker emits logs and metric counters, each on a named pipe that is passed via the API. Logs are flushed line by line, whereas metrics are emitted when the instance starts, then every 60 seconds while it's running, and on panic. Firecracker customers are responsible for collecting data in the Firecracker log files. In production builds, Firecracker does not expose the serial console port, since it may contain guest data that the host should not see. ---