firecracker Documentation

Api Requests/Actions Copy Section

Actions API Request

Firecracker microVMs can execute actions that can be triggered via PUT
requests on the /actions resource.

Details about the required fields can be found in the
swagger definition.

InstanceStart Copy Chapter

The InstanceStart action powers on the microVM and starts the guest OS. It
does not have a payload. It can only be successfully called once.

InstanceStart Example Copy Section

curl --unix-socket ${socket} -i \
     -X PUT "http://localhost/actions" \
     -d '{ "action_type": "InstanceStart" }'

FlushMetrics Copy Chapter

The FlushMetrics action flushes the metrics on user demand.

FlushMetrics Example Copy Section

curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT "http://localhost/actions" \
    -d '{ "action_type": "FlushMetrics" }'

[Intel and AMD only] SendCtrlAltDel Copy Chapter

This action will send the CTRL+ALT+DEL key sequence to the microVM. By
convention, this sequence has been used to trigger a soft reboot and, as such,
most Linux distributions perform an orderly shutdown and reset upon receiving
this keyboard input. Since Firecracker exits on CPU reset, SendCtrlAltDel can
be used to trigger a clean shutdown of the microVM.

For this action, Firecracker emulates a standard AT keyboard, connected via an
i8042 controller. Driver support for both these devices needs to be present in
the guest OS. For Linux, that means the guest kernel needs CONFIG_SERIO_I8042
and CONFIG_KEYBOARD_ATKBD.

[!NOTE]

At boot time, the Linux driver for i8042 spends a few tens of milliseconds
probing the device. This can be disabled by using these kernel command line
parameters:

console Copy

i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd

SendCtrlAltDel Example Copy Section

curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT "http://localhost/actions" \
    -d '{ "action_type": "SendCtrlAltDel" }'

Api Requests/Block Caching Copy Section

Block device caching strategies

Firecracker offers the possiblity of choosing the block device caching strategy.
Caching strategy affects the path data written from inside the microVM takes to
the host persistent storage.

How it works Copy Chapter

When installing a block device through a PUT /drives API call, users can choose
the caching strategy by inserting a cache_type field in the JSON body of the
request. The available cache types are:

• Unsafe
• Writeback

Unsafe mode (default) Copy Section

When configuring the block caching strategy to Unsafe, the device will not
advertise the VirtIO flush feature to the guest driver.

Writeback mode Copy Section

When configuring the block caching strategy to Writeback, the device will
advertise the VirtIO flush feature to the guest driver. If negotiated when
activating the device, the guest driver will be able to send flush requests to
the device. When the device executes a flush request, it will perform an fsync
syscall on the backing block file, committing all data in the host page cache to
disk.

Supported use cases Copy Chapter

The caching strategy should be used in order to make a trade-off:

• Unsafe
  • enhances performance as fewer syscalls and IO operations are performed when
    running workloads
  • sacrifices data integrity in situations where the host simply loses the
    contents of the page cache without committing them to the backing storage
    (such as a power outage)
  • recommended for use cases with ephemeral storage, such as serverless
    environments
• Writeback
  • ensures that once a flush request was acknowledged by the host, the data is
    committed to the backing storage
  • sacrifices performance, from boot time increases to greater
    emulation-related latencies when running workloads
  • recommended for use cases with low power environments, such as embedded
    environments

How to configure it Copy Chapter

Example sequence that configures a block device with a caching strategy:

curl --unix-socket ${socket} -i \
     -X PUT "http://localhost/drives/dummy" \
     -H "accept: application/json" \
     -H "Content-Type: application/json" \
     -d "{
             \"drive_id\": \"dummy\",
             \"path_on_host\": \"${drive_path}\",
             \"is_root_device\": false,
             \"is_read_only\": false,
             \"cache_type\": \"Writeback\"
         }"

Api Requests/Block Io Engine Copy Section

Block device IO engine

For all Firecracker versions prior to v1.0.0, the emulated block device uses a
synchronous IO engine for executing the device requests, based on blocking
system calls.

Firecracker 1.0.0 adds support for an asynchronous block device IO engine.

[!WARNING]

Support is currently in developer preview. See
this section for more info.

The Async engine leverages io_uring for
executing requests in an async manner, therefore getting overall higher
throughput by taking better advantage of the block device hardware, which
typically supports queue depths greater than 1.

The block IO engine is configured via the PUT /drives API call (pre-boot only),
with the io_engine field taking two possible values:

• Sync (default)
• Async (in developer preview)

The Sync variant is the default, in order to provide backwards compatibility
with older Firecracker versions.

[!NOTE]

vhost-user block device is another option for block
IO that requires an external backend process.

Example configuration Copy Chapter

curl --unix-socket ${socket} -i \
     -X PUT "http://localhost/drives/rootfs" \
     -H "accept: application/json" \
     -H "Content-Type: application/json" \
     -d "{
             \"drive_id\": \"rootfs\",
             \"path_on_host\": \"${drive_path}\",
             \"is_root_device\": true,
             \"is_read_only\": false,
             \"io_engine\": \"Sync\"
         }"

Host requirements Copy Chapter

Firecracker requires a minimum host kernel version of 5.10.51 for the Async IO
engine.

This requirement is based on the availability of the io_uring subsystem, as
well as a couple of features and bugfixes that were added in newer kernel
versions.

If a block device is configured with the Async io_engine on a host kernel
older than 5.10.51, the API call will return a 400 Bad Request, with a
suggestive error message.

Performance considerations Copy Chapter

The performance is strictly tied to the host kernel version. The gathered data
may not be relevant for kernels other than 5.10. Newer supported host kernels
(6.1, 6.18) may exhibit different performance characteristics.

Device creation Copy Section

When using the Async variant, there is added latency on device creation (up to
~110 ms), caused by the extra io_uring system calls performed by Firecracker.
This translates to higher latencies on either of these operations:

• API call duration for block device config
• Boot time for VMs started via JSON config files
• Snapshot restore time

For use-cases where the lowest latency on the aforementioned operations is
desired, it is recommended to use the Sync IO engine.

Block IOPS and efficiency Copy Section

The Async engine performance potential is showcased when the block device
backing files are placed on a physical disk that supports efficient parallel
execution of requests, like an NVME drive. It's also recommended to evenly
distribute the backing files across the available drives of a host, to limit
contention in high-density scenarios.

The performance measurements we've done were made on NVME drives, and we've
discovered that:

For read workloads which operate on data that is not present in the host
page cache, the performance improvement for Async is about 1.5x-3x in overall
efficiency (IOPS per CPU load) and up to 30x in total IOPS.

For write workloads, the Async engine brings an improvement of about
20-45% in total IOPS but performs worse than the Sync engine in total
efficiency (IOPS per CPU load). This means that while Firecracker will achieve
better performance, it will be at the cost of consuming more CPU for the kernel
workers. In this case, the VMM cpu load is also reduced, which should translate
into performance increase in hybrid workloads (block+net+vsock).

Whether or not using the Async engine is a good idea performance-wise depends
on the workloads and the amount of spare CPU available on a host. According to
our NVME experiments, io_uring will always bring performance improvements
(granted that there are enough available CPU resources).

It is recommended that users perform some tests with examples of expected
workloads and measure the efficiency as (IOPS/CPU load).

Developer preview status Copy Chapter

View the release policy for information about developer
preview terminology.

The Async io_engine is not yet suitable for production use. It will be made
available for production once Firecracker has support for a host kernel that
implements mitigation mechanisms for the following threats:

Threat 1: PID exhaustion Copy Section

The number of io_uring kernel workers assigned to one Firecracker block device
is upper-bounded by:

(1 + NUMA_COUNT * min(size_of_ring, 4 * NUMBER_OF_CPUS)

This formula is derived from the 5.10 linux kernel code, while size_of_ring is
hardcoded to 128 in Firecracker.

Depending on the number of microVMs that can concurrently live on a host and the
number of block devices configured for each microVM, the kernel PID limit may be
reached, resulting in failure to create any new process.

Kernels starting with 5.15 expose a configuration option for customising this
upper bound. Once possible, we plan on exposing this in the Firecracker drive
configuration interface.

Threat 2: worker thread resource consumption Copy Section

The io_uring kernel workers are spawned in the root cgroup of the system. They
don’t inherit the Firecracker cgroup, cannot be moved out of the root cgroup and
their names don't contain any information about the microVM's PID. This makes it
impossible to attribute a worker to a specific Firecracker VM and limit the CPU
and memory consumption of said workers via cgroups.

Starting with kernel 5.12 (currently unsupported), the Firecracker cgroup is
inherited by the io_uring workers.

Path to GA Copy Section

We plan on marking the Async engine as production ready once an LTS linux kernel
including mitigations for the aforementioned mitigations is released and support
for it is added in Firecracker.

Read more about Firecracker's kernel support policy.

Api Requests/Block Vhost User Copy Section

Vhost-user block device

[!WARNING]

Support is currently in developer preview. See
this section for more info.

As an alternative to file-backed block device Sync and
Async engines, Firecracker supports a vhost-user block device.

There is a good introduction of how a vhost-user block device works in general
at
FOSDEM23.

Vhost-user is a
userspace protocol that allows to delegate Virtio queue processing to another
userspace process on the host, as opposed to performing this task within
Firecracker's VMM thread.

In the vhost-user architecture, the VMM acts as a vhost-user frontend and it is
responsible for:

• connecting to the backend via a Unix domain socket (UDS)
• feature negotiation with the backend and the guest
• handling device configuration requests from the guest
• sharing sufficient information about the guest memory and Virtio queues with
  the backend

The vhost-user backend receives the information from the frontend and performs
handling of IO requests from the guest.

The UDS socket is only used for control plane purposes and does not participate
in the data plane.

Firecracker only implements a vhost-user frontend. Users are free to choose from
existing open source backends or implement their own.

Topology Copy Chapter

Each vhost-user device connects to its own UDS socket. There is no way for
multiple devices to share a single socket, as there is no way to differentiate
messages related to devices at the vhost-user protocol level.

Each device can be served by a separate backend or a single backend can serve
multiple devices.

Interactions with the backend Copy Chapter

There are three points when the vhost-user frontend communicates with the
backend:

1. Device initialisation. When a vhost-user device is created, Firecracker
   connects to the corresponding UDS socket and negotiates Virtio and Vhost
   features with backend and retrieves device configuration.
2. Device activation. When the guest driver finishes setting up the device,
   Firecracker shares memory tables and Virtio queue information with the
   backend. As a part of this, Firecracker shares file descriptors for guest's
   memory regions, as well as file descriptors for queue notifications.
3. Config update. When receving a
   PATCH request
   on a vhost-user backed drive, Firecracker rerequests the device config from
   the backend in order to make the new config available to the guest.

Advantages Copy Chapter

While vhost-user block is considered an optimisation to Firecracker IO, a naive
implementation of the backend is not going to improve performance.

The major advantage of using a vhost-user device is that the backend can
implement custom processing logic. It can use intelligent algorithms to serve
block requests, eg by fetching the block device data over the network or using
sophisticated readahead logic. In such cases, the performance improvement will
be coming from the fact that the custom logic is implemented in the same process
that handles Virtio queues, which reduces the number of required context
switches.

Disadvantages Copy Chapter

In order for the backend to be able to process virtio requests, guest memory
needs to be shared by the frontend to the backend. This means, a shared memory
mapping is required to back guest memory. When a vhost-user device is
configured, Firecracker uses memfd_create instead of creating an anonymous
private mapping to achieve that. It was observed that page faults to a shared
memory mapping take significantly longer (up to 24% in our testing), because
Linux memory subsystem has to use atomic memory operations to update page
status, which is an expensive operation under specific conditions. We advise
users to profile performance on their workloads when considering to use
vhost-user devices.

Other considerations Copy Chapter

Compared to virtio block device where Firecracker interacts with a drive file on
the host, vhost-user block device is handled by the backend directly. Some
workloads may benefit from caching and readahead that the host pagecache offers
for the backing file. This benefit is not available in vhost-user block case.
Users may need to implement internal caching within the backend if they find it
appropriate.

Backends Copy Chapter

There are a number of open source implementations of a vhost-user backend
available for reference that can help developing a custom backend:

1. Qemu backend
2. Cloud Hypervisor backend
3. crosvm backend
4. SPDK backend

Security considerations Copy Chapter

Guest memory sharing Copy Section

By design, a vhost-user frontend must share file descriptors of all guest memory
regions to the backend. In order to achive that, guest memory is created as a
memfd and mapped as
MAP_SHARED.

File descriptor in procfs

An open memfd is reflected in procfs as any other open file descriptor:

$ ls -l /proc/{pid}/fd | grep memfd
lrwx------ 1 1234 1234 64 Nov  2 13:39 32 -> /memfd:guest_mem (deleted)

Any process on the host that has access to this file in procfs will be able to
map the file descriptor and observe runtime behaviour of the guest.

At the moment, Firecracker does not close the memfd, because it must remain
open until all the configured vhost-user devices have been activated and their
info shared with the backends. This kind of tracking is not implemented in
Firecracker, but may be implemented in the future. Meanwhile, users need to make
sure that the access to the Firecracker's procfs tree is restricted to trusted
processes on the host.

On the backend side, it is advised that the backend closes the guest memory
region file descriptors after mapping them into its own address space.

Resource limit in jailer

The Firecracker jailer allows to configure resource limits for
the Firecracker process. Specifically, it allows to set the maximum file size.
Since memfd that is used to back the guest memory is considered a file, the
file size resource limit cannot be less than the biggest guest memory region.
This does not require any special action from a user, but needs to be taken into
consideration.

Remote code execution in the backend Copy Section

It is recommended to run Firecracker using the jailer. Since the
vhost-user backend interacts with the guest via a Virtio queue, there is a
potential for the guest to exercise issues in the backend codebase to trigger
undesired behaviours. Users should consider running their backend in a jailer or
applying other adequate security measures to restrict it.

[!NOTE]

Firecracker jailer is currently only capable of running
Firecracker as the binary. Vhost-user block device users are expected to use
another jailer to run the backend.

It is also recommended to use proactive security measures like running a
Virtio-level fuzzer in the guest during testing to make sure that the backend
correctly handles all possible classes of inputs (including invalid ones) from
the guest.

Rate limiting / cgroups Copy Section

Virtio block device in Firecracker has a
rate limiting capability.

In the vhost-user case, Firecracker does not participate in handling requests
from the guest, so rate limiting becomes backend's responsibility.

As an additional indirect measure, users can make use of cgroups settings
(either via Firecracker jailer or independently) in order to restrict host CPU
consumption of the guest, which would transitively limit guest's IO activity.

Protection against defects in the backend code Copy Section

Due to potential defects in the backend (eg mislocating Virtio queues or writes
to a wrong location in the guest memory), the guest execution may be affected.
It is advised that customers monitor guest's health periodically.

Additionally, in order to avoid orhpaned Firecracker processes if the backend
crashes, the backend may need to send a signal, such as SIGBUS, to the
Firecracker process for it to exit as well.

Backend timeouts Copy Section

In order to correctly handle the case where the Firecracker process exits before
it exchanges all the expected data with the backend, the backend may need to
implement a timeout for how long it waits for Firecracker to connect and/or to
exchange the data via the vhost-user protocol and exit to avoid resource
exhaustion.

Snapshot support Copy Chapter

At the moment, snapshotting is not supported for microVMs
that have vhost-user devices configured. An attempt to take a snapshot of such a
microVM will fail. It is planned to add support for that in the future.

Example configuration Copy Chapter

Run a vhost-user backend, eg Qemu backend:

vhost-user-blk --socket-path=${backend_socket} --blk-file=${drive_path}

Firecracker API request to add a vhost-user block device:

curl --unix-socket ${fc_socket} -i \
     -X PUT "http://localhost/drives/scratch" \
     -H "accept: application/json" \
     -H "Content-Type: application/json" \
     -d "{
             \"drive_id\": \"scratch\",
             \"socket\": \"${backend_socket}\",
             \"is_root_device\": false
         }"

[!NOTE]

Unlike Virtio block device, there is no way to configure a readonly
vhost-user drive on the Firecracker side. Instead, this configuration belongs
to the backend. Whenever the backend advertises the VIRTIO_BLK_F_RO feature,
Firecracker will accept it, and the device will act as readonly.

[!NOTE]

Whenever a PUT request is sent to the /drives endpoint for a vhost-user
device with the id that already exists, Firecracker will close the existing
connection to the backend and will open a new one. Users may need to restart
their backend if they do so.

Api Requests/Patch Block Copy Section

Updating block devices after boot

Updating Virtio block devices after boot Copy Chapter

Firecracker offers support to update attached block devices after the microVM
has been started. This is provided via PATCH /drives API which notifies
Firecracker that the underlying block file has been changed on the host. It
should be called when the path to the block device is changed or if the file
size has been modified. It is important to note that external changes to the
block device file do not automatically trigger a notification in Firecracker so
the explicit PATCH API call is mandatory.

How it works Copy Section

The implementation of the PATCH /drives API does not modify the host backing
file. It only updates the emulation layer block device properties, path and
length and then triggers a virtio device reconfiguration that is handled by the
guest driver which will update the size of the raw block device. With that being
said, a sequence which performs resizing/altering of the block underlying host
file followed by a PATCH /drives API call is not an atomic operation as the
guest can also modify the block file via emulation during the sequence, if the
raw block device is mounted or accessible.

Supported use case Copy Section

This feature was designed to work with a cooperative guest in order to
effectively simulate hot plug/unplug functionality for block devices.

The following guarantees need to be provided:

• guest did not mount the device
• guest does not read or write from the raw block device /dev/vdX during the
  update sequence

Example sequence that configures a microVM with a placeholder drive and then
updates it with the real one:

# Create and set up a block device.
touch ${ro_drive_path}

curl --unix-socket ${socket} -i \
     -X PUT "http://localhost/drives/scratch" \
     -H "accept: application/json" \
     -H "Content-Type: application/json" \
     -d "{
             \"drive_id\": \"scratch\",
             \"path_on_host\": \"${ro_drive_path}\",
             \"is_root_device\": false,
             \"is_read_only\": true
             \"rate_limiter\": {
                \"bandwidth\": {
                        \"size\": 100000,
                        \"one_time_burst\": 4096,
                        \"refill_time\": 150
                },
                \"ops\": {
                        \"size\": 10,
                        \"refill_time\": 250
                }
            }
         }"
# Finish configuring and start the microVM. Wait for the guest to boot.

# Before mounting the block device in the guest:
# Use another backing file of different size to effectively resize the
# vm block device.
touch ${updated_ro_drive_path}
truncate --size ${new_size}M ${updated_ro_drive_path}
# Create a filesystem in it.
mkfs.ext4 ${updated_ro_drive_path}

# PATCH the block device to use the new backing file.
curl --unix-socket ${socket} -i \
     -X PATCH "http://localhost/drives/scratch" \
     -H "accept: application/json" \
     -H "Content-Type: application/json" \
     -d "{
             \"drive_id\": \"scratch\",
             \"path_on_host\": \"${updated_ro_drive_path}\"
         }"

# It's now safe to mount the block device in the guest and use it
# with the updated backing file.

Data integrity and other issues Copy Section

We do not recommend using this feature outside of its supported use case scope.
If the required guarantees are not provided, data integrity and potential other
issues may arise depending on the actual use case. There are two major aspects
that need be considered here:

Atomicity of the update sequence

If the guest has the opportunity to perform I/O against the block device during
the update sequence it can either read data while it is changed or can overwrite
data already written by a host process. For example a truncate operation can be
undone if the guest issues a write for the last sector of the raw block device,
or the guest application can become inconsistent or/and can create inconsistency
in the block device itself.

In flight I/O requests

If the atomicity of the operation is guaranteed by using methods to make the
microVM quiescence during the update sequence (for example pausing the microVM)
the guest itself or block device can still become incosistent from in flight I/O
requests in the guest that will be executed after it is resumed.

Updating vhost-user block devices after boot Copy Chapter

Unlike with Virtio block device, with vhost-user block devices, Firecracker does
not interact with the underlying block file directly (the vhost-user backend
does). It means that changes to the file are not automatically seen by
Firecracker. There is a mechanism in the
vhost-user protocol
for the backend to notify the frontend about changes in the device config via
VHOST_USER_BACKEND_CONFIG_CHANGE_MSG message. This requires an extra UDS
socket connection between the frontend and backend used for backend-originated
messages. This mechanism is not supported by Firecracker. Instead,
Firecracker makes use of the PATCH /drives API request to get notified about
such changes. Such an API request only includes the required property
(drive_id), because optional properties are not relevant to vhost-user.

Example of a PATCH request for a vhost-user drive:

curl --unix-socket ${socket} -i \
     -X PATCH "http://localhost/drives/scratch" \
     -H "accept: application/json" \
     -H "Content-Type: application/json" \
     -d "{
             \"drive_id\": \"scratch\"
         }"

A PATCH request to a vhost-user drive will make Firecracker retrieve the new
device config from the backend and send a config change notification to the
guest.

Api Requests/Patch Network Interface Copy Section

Updating A Network Interface

After the microVM is started, the rate limiters assigned to a network interface
can be updated via a PATCH /network-interfaces/{id} API call.

E.g. for a network interface created with:

PUT /network-interfaces/iface_1 HTTP/1.1
Host: localhost
Content-Type: application/json
Accept: application/json

{
    "iface_id": "iface_1",
    "host_dev_name": "fctap1",
    "guest_mac": "06:00:c0:a8:34:02",
    "rx_rate_limiter": {
        "bandwidth": {
            "size": 1024,
            "one_time_burst": 1048576,
            "refill_time": 1000
        }
    },
    "tx_rate_limiter": {
        "bandwidth": {
            "size": 1024,
            "one_time_burst": 1048576,
            "refill_time": 1000
        }
    }
}

A PATCH request can be sent at any future time, to update the rate limiters:

PATCH /network-interfaces/iface_1 HTTP/1.1
Host: localhost
Content-Type: application/json
Accept: application/json

{
    "iface_id": "iface_1",
    "rx_rate_limiter": {
        "bandwidth": {
            "size": 1048576,
            "refill_time": 1000
        },
        "ops": {
            "size": 2000,
            "refill_time": 1000
        }
    }
}

The full specification of the data structures available for this call can be
found in our OpenAPI spec.

[!NOTE]

The data provided for the update is merged with the existing data. In the
above example, the RX rate limit is updated, but the TX rate limit remains
unchanged.

Removing Rate Limiting Copy Chapter

A rate limit can be disabled by providing a 0-sized token bucket. E.g.,
following the above example, the TX rate limit can be disabled with:

PATCH /network-interfaces/iface_1 HTTP/1.1
Host: localhost
Content-Type: application/json
Accept: application/json

{
    "iface_id": "iface_1",
    "tx_rate_limiter": {
        "bandwidth": {
            "size": 0,
            "refill_time": 0
        },
        "ops": {
            "size": 0,
            "refill_time": 0
        }
    }
}

Cpu Templates/Boot Protocol Copy Section

Boot protocol register settings

Firecracker makes certain modifications to the guest's registers regardless of
whether a CPU template is used to comply with the boot protocol. If a CPU
template is used the boot protocol settings are performed after the CPU
template is applied. That means that if the CPU template configures CPUID bits
used in the boot protocol settings, they will be overwritten.

See also: CPUID normalization

Boot protocol MSRs (x86_64 only) Copy Chapter

On x86_64, the following MSRs are set to 0:

• MSR_IA32_SYSENTER_CS
• MSR_IA32_SYSENTER_ESP
• MSR_IA32_SYSENTER_EIP
• MSR_STAR
• MSR_CSTAR
• MSR_KERNEL_GS_BASE
• MSR_SYSCALL_MASK
• MSR_LSTAR
• MSR_IA32_TSC

and MSR_IA32_MISC_ENABLE is set to 1.

Boot protocol ARM registers (aarch64 only) Copy Chapter

On aarch64, the following registers are set:

• PSTATE to PSR_MODE_EL1h | PSR_A_BIT | PSR_F_BIT | PSR_I_BIT | PSR_D_BIT
• PC to kernel load address (vCPU0 only)
• X0 to DTB/FDT address (vCPU0 only)

Cpu Templates/Cpu Template Helper Copy Section

CPU template helper tool

The cpu-template-helper tool is a program designed to assist users with
creating and managing their custom CPU templates.

Usage Copy Chapter

The cpu-template-helper tool has two sets of commands: template-related
commands and fingerprint-related commands.

Template-related commands Copy Section

Dump command

This command dumps guest CPU configuration in the custom CPU template JSON
format.

cpu-template-helper template dump \
    --output <cpu-config> \
    [--template <cpu-template>] \
    [--config <firecracker-config>]

Users can utilize this as an entry point of a custom CPU template creation to
comprehend what CPU configuration are exposed to guests.

The guest CPU configuration consists of the following entities:

• x86_64
  • CPUID
  • MSRs (Model Specific Registers)
• aarch64
  • ARM registers

It retrieves the above entities exposed to a guest by applying the same preboot
process as Firecacker and capturing them in the state just before booting a
guest. More details about the preboot process can be found
here and here.

[!NOTE]

Some MSRs and ARM registers are not included in the output, since they are not
reasonable to modify with CPU templates. The full list of them can be found in
Appendix.

[!NOTE]

Since the output depends on underlying hardware and software stack (BIOS, CPU,
kernel, Firecracker), it is required to dump guest CPU configuration on each
combination when creating a custom CPU template targeting them all.

Strip command

This command strips identical entries from multiple guest CPU configuration
files generated with the dump command.

cpu-template-helper template strip \
    --paths <cpu-config-1> <cpu-config-2> [..<cpu-config-N>] \
    --suffix <suffix>

One practical use case of the CPU template feature is to provide a consistent
CPU feature set to guests running on multiple CPU models. When creating a custom
CPU template for this purpose, it is efficient to focus on the differences in
guest CPU configurations across those CPU models. Given that a dumped guest CPU
configuration typically amounts to approximately 1,000 lines, this command
considerably narrows down the scope to consider.

Verify command

This command verifies that the given custom CPU template is applied correctly.

cpu-template-helper template verify \
    --template <cpu-template> \
    [--config <firecracker-config>]

Firecracker modifies the guest CPU configuration after the CPU template is
applied. Occasionally, due to hardware and/or software limitations, KVM might
not set the given configuration. Since Firecracker does not check them at
runtime, it is required to ensure that these situations don't happen with their
custom CPU templates before deploying it.

When a template is specified both through --template and in Firecracker
configuration file provided via --config, the template specified with
--template takes precedence.

[!NOTE]

This command does not ensure that the contents of the template are sensible.
Thus, users need to make sure that the template does not have any inconsistent
entries and does not crash guests.

Fingerprint-related commands Copy Section

Dump command

This command not only dumps the guest CPU configuration, but also host
information that could affect the validity of custom CPU templates.

cpu-template-helper fingerprint dump \
    --output <output-path> \
    [--template <cpu-template>] \
    [--config <firecracker-config>]

Keeping the underlying hardware and software stack updated is essential for
maintaining security and leveraging new technologies. On the other hand, since
the guest CPU configuration can vary depending on the infrastructure, updating
it could lead to a situation where a custom CPU template loses its validity. In
addition, even if values of the guest CPU configuration don't change, its
internal behavior or semantics could still change. For instance, a kernel
version update may introduce changes to KVM emulation and a microcode update may
alter the behavior of CPU instructions.

To ensure awareness of these changes, it is strongly recommended to store the
fingerprint file at the time of creating a custom CPU template and to
continuously compare it with the current one.

Compare command

This command compares two fingerprint files: one was taken at the time of custom
CPU template creation and the other is taken currently.

cpu-template-helper fingerprint compare \
    --prev  \
    --curr <curr-fingerprint> \
    --filters <field-1> [..<field-N>]

By continously comparing fingerprint files, users can ensure they are aware of
any changes that could require revising the custom CPU template. However, it is
worth noting that not all of these changes necessarily require a revision, and
some changes could be inconsequential to the custom CPU template depending on
its use case. To provide users with flexibility in comparing fingerprint files
based on situations or use cases, the --filters option allows users to select
which fields to compare.

As examples of when to compare fingerprint files:

• When bumping the Firecracker version up
• When bumping the kernel version up
• When applying a microcode update (or launching a new host (e.g. AWS EC2 metal
  instance))

Sample scenario Copy Chapter

This section gives steps of creating and managing a custom CPU template in a
sample scenario where the template is designed to provide a consistent set of
CPU features to a heterogeneous fleet consisting of multiple CPU models.

Custom CPU template creation Copy Section

1. Run the cpu-template-helper template dump command on each CPU model to
   retrieve guest CPU configuration.
2. Run the cpu-template-helper template strip command to remove identical
   entries across the dumped guest CPU configuration files.
3. Examine the differences of guest CPU configuration in details, determine
   which CPU features should be presented to guests and draft a custom CPU
   template.
4. Run the cpu-template-helper template verify command to check the created
   custom CPU template is applied correctly.
5. Conduct thorough testing of the template as needed to ensure that it does not
   contain any inconsistent entries and does not lead to guest crashes.

Custom CPU template management Copy Section

1. Run the cpu-template-helper fingerprint dump command on each CPU model at
   the same time when creating a custom CPU template.
2. Store the dumped fingerprint files together with the custom CPU template.
3. Run the cpu-template-helper fingerprint dump command to ensure the
   template's validity whenever you expect changes to the underlying hardware
   and software stack.
4. Run the cpu-template-helper fingerprint compare command to identify changes
   of the underlying environment introduced after creating the template.
5. (if changes are detected) Review the identified changes, make necessary
   revisions to the CPU template, and replace the fingerprint file with the new
   one.

[!NOTE]

It is recommended to review the update process of the underlying stack on your
infrastructure. This can help identify points that may require the above
validation check.

Appendix Copy Chapter

MSRs excluded from guest CPU configuration dump Copy Section

ARM registers excluded from guest CPU configuration dump Copy Section

Cpu Templates/Cpu Templates Copy Section

CPU templates

Firecracker allows users to customise how the vCPUs are represented to the guest
software by changing the following configuration:

• CPUID (x86_64 only)
• MSRs (Model Specific Registers, x86_64 only)
• ARM registers (aarch64 only)
• vCPU features (aarch64 only)
• KVM capabilities (both x86_64 and aarch64)

A combination of the changes to the above entities is called a CPU template.

The functionality can be used when a user wants to mask a feature from the
guest. A real world use case for this is representing a heterogeneous fleet (a
fleet consisting of multiple CPU models) as a homogeneous fleet, so the guests
will experience a consistent feature set supported by the host.

[!NOTE]

Representing one CPU vendor as another CPU vendor is not supported.

[!NOTE]

CPU templates shall not be used as a security protection against malicious
guests. Disabling a feature in a CPU template does not generally make it
completely unavailable to the guest. For example, disabling a feature related
to an instruction set will indicate to the guest that the feature is not
supported, but the guest may still be able to execute corresponding
instructions if it does not obey the feature bit.

Firecracker supports two types of CPU templates:

• Static CPU templates - a set of built-in CPU templates for users to choose
  from
• Custom CPU templates - users can create their own CPU templates in json format
  and pass them to Firecracker

[!NOTE]

Static CPU templates are deprecated starting from v1.5.0 and will be removed
in accordance with our deprecation policy. Even after the removal, custom CPU
templates are available as an improved iteration of static CPU templates. For
more information about the transition from static CPU templates to custom CPU
templates, please refer to
this GitHub discussion.

[!NOTE]

CPU templates for ARM (both static and custom) require the following patch to
be available in the host kernel:
Support writable CPU ID registers from userspace.
Otherwise KVM will fail to write to the ARM registers.

Static CPU templates Copy Chapter

At the moment the following set of static CPU templates are supported:

T2 and C3 templates are mapped as close as possible to AWS T2 and C3 instances
in terms of CPU features. Note that on a microVM that is lauched with the C3
template and running on processors that do not enumerate FBSDP_NO, PSDP_NO and
SBDR_SSDP_NO on IA32_ARCH_CAPABILITIES MSR, the kernel does not apply the
mitigation against MMIO stale data vulnerability.

The T2S template is designed to allow migrating
snapshots between hosts with Intel Skylake and
Intel Cascade Lake securely by further restricting CPU features for the guest,
however this comes with a performance penalty. Users are encouraged to carry out
a performance assessment if they wish to use the T2S template. Note that
Firecracker expects the host to always be running the latest version of the
microcode.

The T2CL template is mapped to be close to Intel Cascade Lake. It is only safe
to use it on Intel Cascade Lake and Ice Lake.

The only AMD template is T2A. It is considered safe to be used with AMD Milan.

Intel T2CL and AMD T2A templates together aim to provide instruction set feature
parity between CPUs running them, so they can form a heterogeneous fleet
exposing the same instruction sets to the application.

The V1N1 template is designed to represent ARM Neoverse V1 as ARM Neoverse N1.

Configuring static CPU templates Copy Section

Configuration of a static CPU template is performed via the /machine-config
API endpoint:

curl --unix-socket /tmp/firecracker.socket -i  \
  -X PUT 'http://localhost/machine-config' \
  -H 'Accept: application/json'            \
  -H 'Content-Type: application/json'      \
  -d '{
           "vcpu_count": 2,
           "mem_size_mib": 1024,
           "cpu_template": "T2CL"
  }'

Custom CPU templates Copy Chapter

Users can create their own CPU templates by creating a json file containing
modifiers for CPUID, MSRs or ARM registers.

[!NOTE]

Creating custom CPU templates requires expert knowledge of CPU architectures.
Custom CPU templates must be tested thoroughly before use in production. An
inappropriate configuration may lead to guest crashes or making guests
vulnerable to security attacks. For example, if a CPU template signals a
hardware vulnerability mitigation to the guest while the mitigation is in fact
not supported by the hardware, the guest may decide to disable corresponding
software mitigations which will make the guest vulnerable.

[!NOTE]

Having MSRs or ARM registers in the custom CPU template does not affect access
permissions that guests will have to those registers. The access control is
handled by KVM and is not influenced by CPU templates.

[!NOTE]

When setting guest configuration, KVM may reject setting some bits quietly.
This is user's responsibility to make sure that their custom CPU template is
applied as expected even if Firecracker does not report an error.

In order to assist with creation and usage of CPU templates, there exists a CPU
template helper tool. More details can be found here.

Configuring custom CPU templates Copy Section

Configuration of a custom CPU template is performed via the /cpu-config API
endpoint.

An example of configuring a custom CPU template on x86_64:

curl --unix-socket /tmp/firecracker.socket -i  \
  -X PUT 'http://localhost/cpu-config' \
  -H 'Accept: application/json'            \
  -H 'Content-Type: application/json'      \
  -d '{
        "kvm_capabilities": ["!56"],
        "cpuid_modifiers": [
          {
            "leaf": "0x1",
            "subleaf": "0x0",
            "flags": 0,
            "modifiers": [
              {
                "register": "eax",
                "bitmap": "0bxxxx000000000011xx00011011110010"
              }
            ]
          }
        ],
        "msr_modifiers": [
          {
            "addr": "0x10a",
            "bitmap": "0b0000000000000000000000000000000000000000000000000000000000000000"
          }
        ]
      }'

This CPU template will do the following:

• removes check for KVM capability: KVM_CAP_XCRS. This allows Firecracker to run
  on old cpus. See
  this
  discussion.
• in leaf 0x1, subleaf 0x0, register eax:
  • clear bits 0b00001111111111000011100100001101
  • set bits 0b00000000000000110000011011110010
  • leave bits 0b11110000000000001100000000000000 intact.
• in MSR 0x10, it will clear all bits.

An example of configuring a custom CPU template on ARM:

curl --unix-socket /tmp/firecracker.socket -i  \
  -X PUT 'http://localhost/cpu-config' \
  -H 'Accept: application/json'            \
  -H 'Content-Type: application/json'      \
  -d '{
        "kvm_capabilities": ["171", "172"],
        "vcpu_features": [{ "index": 0, "bitmap": "0b11xxxxx" }]
        "reg_modifiers": [
          {
            "addr": "0x603000000013c020",
            "bitmap": "0bxxxxxxxxxxxx0000xxxxxxxxxxxx0000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
          }
        ]
      }'

This CPU template will do the following:

• add checks for KVM capabilities: KVM_CAP_ARM_PTRAUTH_ADDRESS and
  KVM_CAP_ARM_PTRAUTH_GENERIC. These checks are to ensure that the host have
  capabilities needed for the vCPU features.
• enable additional vCPU features: KVM_ARM_VCPU_PTRAUTH_ADDRESS and
  KVM_ARM_VCPU_PTRAUTH_GENERIC
• modify ARM register 0x603000000013c020:
  • clear bits
    0b0000000000001111000000000000111100000000000000000000000000000000
  • leave bits
    0b1111111111110000111111111111000011111111111111111111111111111111 intact.

Information about KVM capabilities can be found in the
kernel source.
Information about vCPU features on aarch64 can be found in the
kernel source.
Information on how the ARM register addresses are constructed can be found in
the
KVM API documentation.

Custom CPU templates language schema Copy Section

The full description of the custom CPU templates language can be found
here.

[!NOTE]

You can also use _ to visually separate parts of a bitmap. So instead of
writing: 0b0000xxxx, it can be 0b0000_xxxx.

Expansion of contracted bitmaps

If a contracted version of a bitmap is given, for example, 0b101 where a
32-bit bitmap is expected, missing characters are implied to be x
(0bxxxxxxxxxxxxxxxxxxxxxxxxxxxxx101).

CPUID normalization and boot protocol register settings

Some of the configuration set by a custom CPU template may be overwritten by
Firecracker. More details can be found here and
here.

Information about architecture-specific settings

For detailed information when working with custom CPU templates, please refer to
hardware specifications from CPU vendors, for example:

• Intel Software Developer Manual
• AMD Architecture Programmer's Manual
• ARM Architecture Refernce Manual

A note about configuration of both static and custom CPU templates Copy Chapter

If a user configured both a static CPU template (via /machine-config) and a
custom CPU template (via /cpu-config) in the same Firecracker process, only
the configuration that was performed the last is applied. This means that if a
static CPU template was configured first and a custom CPU template was
configured later, only the custom CPU template configuration will be applied
when starting a microVM.

Cpu Templates/Cpuid Normalization Copy Section

CPUID normalization (x86_64 only)

On x86_64, Firecracker makes certain modifications to the guest's CPUID
regardless of whether a CPU template is used. This is referred to as
CPUID normalization. If a CPU template is used the normalization is performed
after the CPU template is applied. That means that if the CPU template
configures CPUID bits used in the normalization process, they will be
overwritten.

See also: boot protocol settings

x86_64 common CPUID normalization Copy Chapter

Intel-specific CPUID normalization Copy Chapter

AMD-specifc CPUID normalization Copy Chapter

Mmds/Mmds Design Copy Section

microVM Metadata Service

MMDS consists of three major logical components: the backend, the data store,
and the minimalist HTTP/TCP/IPv4 stack (named Dumbo). They all exist within
the Firecracker process, and outside the KVM boundary; the first is a part of
the API server, the data store is a global entity for a single microVM, and the
last is a part of the device model.

The MMDS backend Copy Chapter

Users can add/update the MMDS contents via the backend, which is accessible
through the Firecracker API. Setting the initial contents involves a PUT
request to the /mmds API resource, with a JSON body that describes the desired
data store structure and contents. Here's a JSON example:

{
"latest": {
            "meta-data": {
                "ami-id": "ami-12345678",
                "reservation-id": "r-fea54097",
                "local-hostname": "ip-10-251-50-12.ec2.internal",
                "public-hostname": "ec2-203-0-113-25.compute-1.amazonaws.com",
                "network": {
                    "interfaces": {
                        "macs": {
                            "02:29:96:8f:6a:2d": {
                                "device-number": "13345342",
                                "local-hostname": "localhost",
                                "subnet-id": "subnet-be9b61d"
                            }
                        }
                    }
                }
            }
        }
}

The MMDS contents can be updated either via a subsequent PUT (that replaces
them entirely), or using PATCH requests, which feed the JSON body into the
JSON Merge Patch functionality, based on
RFC 7396. MMDS related API requests come
from the host, which is considered a trusted environment, so there are no checks
beside the kind of validation done by HTTP server and serde-json (the crate
used to de/serialize JSON). The size limit for the stored metadata is
configurable and defaults to 51200 bytes. When increasing this limit, one must
take into consideration that storing and retrieving large amount of data may
induce bottlenecks for the HTTP REST API processing, which is based on
micro-http crate. MMDS contents can be retrieved using the Firecracker API,
via a GET request to the /mmds resource.

The data store Copy Chapter

This is a global data structure, currently referenced using a global variable,
that represents the strongly-typed version of JSON-based user input describing
the MMDS contents. It leverages the recursive
Value type exposed by
serde-json. It can only be accessed from thread-safe contexts. MMDS data store
supports at the moment storing and retrieving JSON values. Data store contents
can be retrieved using the Firecracker API server from host and using the
embedded MMDS HTTP/TCP/IPv4 network stack from guest. MMDS data store is upper
bounded to the value of the --mmds-size-limit command line parameter. If left
unconfigured, it will default to the value of --http-api-max-payload-size,
which is 51200 bytes by default.

Dumbo Copy Chapter

The Dumbo HTTP/TCP/IPv4 network stack handles guest HTTP requests heading
towards the configured MMDS IPv4 address. Before going into Dumbo specifics,
it's worth going through a brief description of the Firecracker network device
model. Firecracker only offers Virtio-net paravirtualized devices to guests.
Drivers running in the guest OS use ring buffers in a shared memory area to
communicate with the device model when sending or receiving frames. The device
model associates each guest network device with a TAP device on the host. Frames
sent by the guest are written to the TAP fd, and frames read from the TAP fd are
handed over to the guest.

The Dumbo stack can be instantiated once for every network device, and is
disabled by default. It can be enabled through the API request body used to
configure MMDS by specifying the ID of the network interface inside the
network_interfaces list. In order for the API call to succeed, the network
device must be attached beforehand, otherwise an error is returned. Once
enabled, the stack taps into the aforementioned data path. Each frame coming
from the guest is examined to determine whether it should be processed by
Dumbo instead of being written to the TAP fd. Also, every time there is room
in the ring buffer to hand over frames to the guest, the device model first
checks whether Dumbo has anything to send; if not, it resumes getting frames
from the TAP fd (when available).

We chose to implement our own solution, instead of leveraging existing
libraries/implementations, because responding to guest MMDS queries in the
context of Firecracker is amenable to a wide swath of simplifications. First of
all, we only need to handle GET and PUT requests, which require a bare-bones
HTTP 1.1 server, without support for most headers and more advanced features
like chunking. Also, we get to choose what subset of HTTP is used when building
responses. Moving lower in the stack, we are dealing with TCP connections over
what is essentially a point-to-point link, that seldom loses packets and does
not reorder them. This means we can do away with congestion control (we only use
flow control), complex reception logic, and support for most TCP
options/features. At this point, the layers below (Ethernet and IPv4) don't
involve much more than sanity checks of frame/packet contents.

Dumbo is built using both general purpose components (which we plan to offer
as part of one or more libraries), and Firecracker MMDS specific code. The
former category consists of various helper modules used to process streams of
bytes as protocol data units (Ethernet & ARP frames, IPv4 packets, and TCP
segments), a TCP handler which listens for connections while demultiplexing
incoming segments, a minimalist TCP connection endpoint implementation, and a
greatly simplified HTTP 1.1 server. The Firecracker MMDS specific code is found
in the logic which taps into the device model, and the component that parses an
HTTP request, builds a response based on MMDS contents, and finally sends back a
reply.

MMDS Network Stack Copy Section

Somewhat confusingly, this is the name of the component which taps the device
model. It has a user-configured IPv4 address (see
Firecracker MMDS configuration API)
and MAC (06:01:23:45:67:01) addresses. The latter is also used to respond to
ARP requests. For every frame coming from the guest, the following steps take
place:

1. Apply a heuristic to determine whether the frame may contain an ARP request
   for the MMDS IP address, or an IPv4 packet heading towards the same address.
   There can be no false negatives. Frames that fail both checks are rejected
   (deferred to the device model for regular processing).
2. Reject invalid Ethernet frames. Reject valid frames if their EtherType is
   neither ARP, nor IPv4.
3. (if EtherType == ARP) Reject invalid ARP frames. Reject the frame if
   its target protocol address field is different from the MMDS IP address.
   Otherwise, record that an ARP request has been received (the stack only
   remembers the most recent request).
4. (if EtherType == IPv4) Reject invalid packets. Reject packets if
   their destination address differs from the MMDS IP address. Drop (stop
   processing without deferring to the device model) packets that do not carry
   TCP segments (by looking at the protocol number field). Send the rest to the
   inner TCP handler.

The current implementation does not support Ethernet 802.1Q tags, and does not
handle IP fragmentation. Tagged Ethernet frames are most likely going to be
deferred to the device model for processing, because the heuristics do not take
the presence of the tag into account. Moreover, their EtherType will not appear
to be of interest. Fragmented IP packets do not get reassembled; they are
treated as independent packets.

Whenever the guest is able to receive a frame, the device model first requests
one from the MMDS network stack associated with the current network device.

1. If an ARP request has been previously recorded, send an ARP reply and forget
   about the request.
2. If the inner TCP handler has any packets to transmit, wrap the next one into
   a frame and send it.
3. There are no MMDS related frames to send, so tell the device model to read
   from the TAP fd instead.

Security Considerations

Operators should not rely on the MMDS network stack to filter packets with the
MMDS IP as the destination from the guest's outbound traffic. As stated in our
threat model, guest traffic should be treated
as untrusted, and firewall rules should be put in place at the host-level to
prevent guests from accessing restricted IPv4 addresses on the host.

Example rules for dropping host IMDS-bound packets may be seen in our
production host setup
documentation.

TCP handler Copy Section

Handles received packets that appear to carry TCP segments. Its operation is
described in the dumbo crate documentation. Each connection is associated with
an MMDS endpoint.

MMDS endpoint Copy Section

This component gets the byte stream from an inner TCP connection object,
identifies the boundaries of the next HTTP request, and parses it using an
HttpRequest object. For each valid GET request, the URI is used to identify a
key from the metadata store (like in the previous example), and a response is
built using the Firecracker implementation of HttpResponse logic, based on the
associated value, and sent back to the guest over the same connection. Each
endpoint has a fixed size receive buffer, and a variable length response buffer
(depending on the size of each response). TCP receive window semantics are used
to ensure the guest does not overrun the receive buffer during normal operation
(the connection has to drop segments otherwise). There can be at most one
response pending at any given time.

Here are more details describing what happens when a segment is received by an
MMDS endpoint (previously created when a SYN segment arrived at the TCP
handler):

1. Invoke the receive functionality of the inner connection object, and append
   any new data to the receive buffer.
2. If no response is currently pending, attempt to identify the end of the first
   request in the receive buffer. If no such boundary can be found, and the
   buffer is full, reset the inner connection (which also causes the endpoint
   itself to be subsequently removed) because the guest exceeded the maximum
   allowed request size.
3. If no response is pending, and we can identify a request in the receive
   buffer, parse it, free up the associated buffer space (also update the
   connection receive window), and build an HTTP response, which becomes the
   current pending response.
4. If a FIN segment was received, and there's no pending response, call close
   on the inner connection. If a valid RST is received at any time, mark the
   endpoint for removal.

When the TCP handler asks an MMDS endpoint for any segments to send, the
transmission logic of the inner connection is invoked, specifying the pending
response (when present) as the payload source. All packets coming from MMDS have
the TTL value set to 1 by default.

Connection Copy Section

Connection objects are minimalist implementation of the TCP protocol. They are
used to reassemble the byte stream which carries guest HTTP requests, and to
send back segments which contain parts of the response. More details are
available in the dumbo crate documentation.

Mmds/Mmds User Guide Copy Section

microVM Metadata Service

The Firecracker microVM Metadata Service (MMDS) is a mutable data store which
can be used for sharing information between host and guests, in a secure and
easy at hand way.

Configuring and activating the microVM Metadata Service Copy Chapter

By default, MMDS is not reachable from the guest operating system. At microVM
runtime, MMDS is tightly coupled with a network interface, which allows MMDS
requests. When configuring the microVM, if MMDS needs to be activated, a network
interface has to be configured to allow MMDS requests. This can be achieved in
two steps:

1. Attach one (or more) network interfaces through an HTTP PUT request to
   /network-interfaces/${MMDS_NET_IF}. The full network configuration API can
   be found in the
   firecracker swagger file.
2. Configure MMDS through an HTTP PUT request to /mmds/config resource and
   include the IDs of the network interfaces that should allow forwarding
   requests to MMDS in the network_interfaces list. The complete MMDS API is
   described in the
   firecracker swagger file.

Examples Copy Section

Attaching a network device with ID MMDS_NET_IF:

MMDS_NET_IF=eth0
curl --unix-socket /tmp/firecracker.socket -i                 \
  -X PUT 'http://localhost/network-interfaces/${MMDS_NET_IF}' \
  -H 'Accept: application/json'                               \
  -H 'Content-Type: application/json'                         \
  -d '{
      "iface_id": "${MMDS_NET_IF}",
      "guest_mac": "AA:FC:00:00:00:01",
      "host_dev_name": "tap0"
    }'

Configuring MMDS to receive requests through the MMDS_NET_IF network interface
ID:

MMDS_IPV4_ADDR=169.254.170.2
curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT "http://localhost/mmds/config"     \
    -H "Content-Type: application/json"       \
    -d '{
             "network_interfaces": ["${MMDS_NET_IF}"]
    }'

MMDS can be configured pre-boot only, using the Firecracker API server. Enabling
MMDS without at least a network device attached will return an error.

The IPv4 address used by guest applications when issuing requests to MMDS can be
customized through the same HTTP PUT request to /mmds/config resource, by
specifying the IPv4 address to the ipv4_address field. If the IP configuration
is not provided before booting up the guest, the MMDS IPv4 address defaults to
169.254.169.254.

MMDS_IPV4_ADDR=169.254.170.2
curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT "http://localhost/mmds/config"     \
    -H "Content-Type: application/json"       \
    -d '{
             "network_interfaces": ["${MMDS_NET_IF}"],
             "ipv4_address": "${MMDS_IPV4_ADDR}"
    }'

MMDS is tightly coupled with a network interface which is used to route MMDS
packets. To send MMDS intended packets, guest applications must insert a new
rule into the routing table of the guest OS. This new rule must forward MMDS
intended packets to a network interface which allows MMDS requests. For example:

MMDS_IPV4_ADDR=169.254.170.2
MMDS_NET_IF=eth0
ip route add ${MMDS_IPV4_ADDR} dev ${MMDS_NET_IF}

MMDS supports two methods to access the contents of the metadata store from the
guest operating system: V1 and V2. More about the particularities of the two
mechanisms can be found in the
Retrieving metadata in the guest operating system
section. The MMDS version used can be specified when configuring MMDS, through
the version field of the HTTP PUT request to /mmds/config resource.
Accepted values are V1(deprecated) and V2 and the default MMDS version used
in case the version field is missing is Version 1.

MMDS_IPV4_ADDR=169.254.170.2
curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT "http://localhost/mmds/config"     \
    -H "Content-Type: application/json"       \
    -d '{
             "network_interfaces": ["${MMDS_NET_IF}"],
             "version": "V2",
             "ipv4_address": "${MMDS_IPV4_ADDR}"
    }'

Inserting and updating metadata Copy Chapter

Inserting and updating metadata is possible through the Firecracker API server.
The metadata inserted in MMDS must be any valid JSON. A user can create or
update the MMDS data store before the microVM is started or during its
operation. To insert metadata into MMDS, an HTTP PUT request to the /mmds
resource has to be issued. This request must have a payload with metadata
structured in JSON format. To replace
existing metadata, a subsequent HTTP PUT request to the /mmds resource must
be issued, using as a payload the new metadata. A complete description of
metadata insertion firecracker API can be found in the
firecracker swagger file.

An example of an API request for inserting metadata is provided below:

curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT "http://localhost/mmds"            \
    -H "Content-Type: application/json"       \
    -d '{
            "latest": {
                  "meta-data": {
                       "ami-id": "ami-12345678",
                       "reservation-id": "r-fea54097",
                       "local-hostname": "ip-10-251-50-12.ec2.internal",
                       "public-hostname": "ec2-203-0-113-25.compute-1.amazonaws.com",
                       "network": {
                            "interfaces": {
                                 "macs": {
                                      "02:29:96:8f:6a:2d": {
                                           "device-number": "13345342",
                                           "local-hostname": "localhost",
                                           "subnet-id": "subnet-be9b61d"
                                      }
                                 }
                            }
                       }
                  }
            }
    }'

To partially update existing metadata, an HTTP PATCH request to the /mmds
resource has to be issued, using as a payload the metadata patch, as
JSON Merge Patch functionality describes.
A complete description of updating metadata Firecracker API can be found in the
firecracker swagger file.

An example API for how to update existing metadata is offered below:

curl --unix-socket /tmp/firecracker.socket -i \
    -X PATCH "http://localhost/mmds"          \
    -H "Content-Type: application/json"       \
    -d '{
            "latest": {
                  "meta-data": {
                       "ami-id": "ami-87654321",
                       "reservation-id": "r-79054aef",
                  }
            }
    }'

Retrieving metadata Copy Chapter

MicroVM metadata can be retrieved both from host and guest operating systems.
For the scope of this chapter, let's assume the data store content is the JSON
below:

{
    "latest": {
          "meta-data": {
               "ami-id": "ami-87654321",
               "reservation-id": "r-79054aef"
          }
    }
}

Retrieving metadata in the host operating system Copy Section

To retrieve existing MMDS metadata from host operating system, an HTTP GET
request to the /mmds resource must be issued. The HTTP response returns the
existing metadata, as a JSON formatted text. A complete description of
retrieving metadata Firecracker API can be found in the
firecracker swagger file.

Below you can see how to retrieve metadata from the host:

curl -s --unix-socket /tmp/firecracker.socket http://localhost/mmds

Output:

{
    "latest": {
          "meta-data": {
               "ami-id": "ami-87654321",
               "reservation-id": "r-79054aef"
          }
    }
}

Retrieving metadata in the guest operating system Copy Section

Accessing the contents of the metadata store from the guest operating system can
be done using one of the following methods:

• V1: simple request/response method (deprecated)
• V2: session-oriented method

Version 1 (Deprecated)

Version 1 is deprecated and will be removed in the next major version change.
Version 2 should be used instead.

To retrieve existing MMDS metadata using MMDS version 1, an HTTP GET request
must be issued. The requested resource can be referenced by its corresponding
JSON Pointer, which is also the path of
the MMDS request. The HTTP response content will contain the referenced metadata
resource.

As in version 2, version 1 also supports a session oriented method in order to
make the migration easier. See the next section for the session
oriented method. Note that version 1 returns a successful response to a GET
request even with an invalid token or no token not to break existing workloads.
mmds.rx_invalid_token and mmds.rx_no_token metrics track the number of GET
requests with invalid tokens and missing tokens respectively, helping users
evaluate their readiness for migrating to MMDS version 2.

Requests containing any other HTTP methods than GET and PUT will receive
405 Method Not Allowed error.

MMDS_IPV4_ADDR=169.254.170.2
RESOURCE_POINTER_OBJ=latest/meta-data
curl -s "http://${MMDS_IPV4_ADDR}/${RESOURCE_POINTER_OBJ}"

Version 2

Similar to
IMDSv2,
MMDS version 2 (V2) is a session oriented method, which makes use of a session
token in order to allow fetching metadata contents.

The session must start with an HTTP PUT request that generates the session
token. In order to be successful, the request must respect the following
constraints:

• must be directed towards /latest/api/token path
• must contain a X-metadata-token-ttl-seconds or
  X-aws-ec2-metadata-token-ttl-seconds header specifying the token lifetime in
  seconds. The value cannot be lower than 1 or greater than 21600 (6 hours).
• must not contain a X-Forwarded-For header.

MMDS_IPV4_ADDR=169.254.170.2
TOKEN=`curl -X PUT "http://${MMDS_IPV4_ADDR}/latest/api/token" \
      -H "X-metadata-token-ttl-seconds: 21600"`

The HTTP response from MMDS is a plaintext containing the session token.

During the duration specified by the token's time to live value, all subsequent
GET requests must specify the session token through the X-metadata-token or
X-aws-ec2-metadata-token header in order to fetch data from MMDS.

MMDS_IPV4_ADDR=169.254.170.2
RESOURCE_POINTER_OBJ=latest/meta-data
curl -s "http://${MMDS_IPV4_ADDR}/${RESOURCE_POINTER_OBJ}" \
    -H "X-metadata-token: ${TOKEN}"

After the token expires, it becomes unusable and a new session token must be
issued.

Snapshotting considerations

The data store is not persisted across snapshots, in order to avoid leaking
vm-specific information that may need to be reseeded into the data store for a
new clone.

The MMDS version, network stack configuration and IP address used for accessing
the service are persisted across snapshot-restore.

If the targeted snapshot version does not support Mmds Version 2, it will not be
persisted in the snapshot (the clone will use the default, V1). Similarly, if a
snapshotted Vm state contains the Mmds version but the Firecracker version used
for restoring does not support persisting the version, the default will be used.

MMDS formats Copy Section

The response format can be JSON or IMDS. The IMDS documentation can be found
here.
The output format can be selected by specifying the optional Accept header.
Using Accept: application/json will format the output to JSON, while using
Accept: plain/text or not specifying this optional header at all will format
the output to IMDS. Setting imds_compat to true through PUT request to
/mmds/config enforces MMDS to always respond in IMDS format regardless of the
Accept header. This allows code written to work on EC2 IMDS to also work on
Firecracker MMDS.

Retrieving MMDS resources in IMDS format, other than JSON string and object
types, is not supported.

Below is an example on how to retrieve the latest/meta-data resource in JSON
format:

MMDS_IPV4_ADDR=169.254.170.2
RESOURCE_POINTER_OBJ=latest/meta-data
curl -s -H "Accept: application/json" "http://${MMDS_IPV4_ADDR}/${RESOURCE_POINTER_OBJ}"

Output:

{
    "ami-id": "ami-87654321",
    "reservation-id": "r-79054aef"
}

Retrieving the latest/meta-data/ami-id resource in JSON format:

MMDS_IPV4_ADDR=169.254.170.2
RESOURCE_POINTER_STR=latest/meta-data/ami-id
curl -s -H "Accept: application/json" "http://${MMDS_IPV4_ADDR}/${RESOURCE_POINTER_STR}"

Output:

"ami-87654321"

Retrieving the latest resource in IMDS format:

MMDS_IPV4_ADDR=169.254.170.2
RESOURCE_POINTER=latest
curl -s "http://${MMDS_IPV4_ADDR}/${RESOURCE_POINTER}"

Output:

meta-data/

Retrieving the latest/meta-data/ resource in IMDS format:

MMDS_IPV4_ADDR=169.254.170.2
RESOURCE_POINTER=latest/meta-data
curl -s "http://${MMDS_IPV4_ADDR}/${RESOURCE_POINTER}"

Output:

ami-id
reservation-id

Retrieving the latest/meta-data/ami-id resource in IMDS format:

MMDS_IPV4_ADDR=169.254.170.2
RESOURCE_POINTER=latest/meta-data/ami-id
curl -s "http://${MMDS_IPV4_ADDR}/${RESOURCE_POINTER}"

Output:

ami-87654321

Errors Copy Chapter

200 - Ok

The request was successfully processed and a response was successfully formed.

400 - Bad Request

The request was malformed.

401 - Unauthorized

Only when using MMDS V2. The HTTP request either lacks the session token, or
the token specified is invalid. A token is invalid if it was not generated using
an HTTP PUT request or if it has expired.

404 - Not Found

The requested resource can not be found in the MMDS data store.

405 - Method Not Allowed

The HTTP request uses a not allowed HTTP method and a response with the Allow
header was formed. When using MMDS V1, this is returned for any HTTP method
other than GET. When MMDS V2 is configured, the only accepted HTTP methods
are PUT and GET.

501 - Not Implemented

The requested HTTP functionality is not supported by MMDS or the requested
resource is not supported in IMDS format.

Appendix Copy Chapter

Example use case: credential rotation Copy Section

For this example, the guest expects to find some sort of credentials (say, a
secret access key) by issuing a GET request to
http://169.254.169.254/latest/meta-data/credentials/secret-key. Most similar
use cases will encompass the following sequence of steps:

1. Some agent running on the host sends a PUT request with the initial
   contents of the MMDS, using the Firecracker API. This most likely takes place
   before the microVM starts running, but may also happen at a later time. Guest
   MMDS requests which arrive prior to contents being available receive a
   NotFound response.
2. The contents are saved to MMDS.
3. The guest sends a GET request for the secret key, which is intercepted by
   MMDS.
4. MMDS processes the request and sends back an HTTP response with the ensembled
   secret key as a JSON string.

After a while, the host agent decides to rotate the secret key. It does so by
updating the data store with a new value. This can be done via a PUT request
to the /mmds API resource, which replaces everything, or with a PATCH
request that only touches the desired key. This effectively triggers the first
two steps again.

The guest reads the new secret key, going one more time through the last three
steps. This can happen after a notification from the host agent, or discovered
via periodic polling, or some other mechanism. Since access to the data store is
thread safe, the guest can only receive either the old version, or the new
version of the key, and not some intermediate state caused by the update.

Snapshotting/Handling Page Faults On Snapshot Resume Copy Section

Handling snapshot memory loading

Firecracker allows for a better management of the microVM's memory loading by
letting users choose between relying on host OS to handle the page faults when
resuming from a snapshot, or having a dedicated userspace process for dealing
with page faults, with the help of
Userfaultfd.

Kernel Copy Chapter

When resuming a microVM from a snapshot, loading the snapshotted guest's memory
(which is file-backed) into RAM is usually kernel's responsibility and is
handled on a per-page-fault basis. Each time the guest touches a page that is
not already in Firecracker's process memory, a page fault occurs, which triggers
a context switch and IO operation in order to bring that page into RAM.
Depending on the use case, doing this for every page can be time-consuming.

Userfaultfd Copy Chapter

Userfaultfd is a mechanism that passes that responsibility of handling page
fault events from kernel space to user space. In order to be able to interact
with this mechanism, userspace needs to firstly obtain an userfault file
descriptor object (UFFD).

Creating a UFFD object Copy Section

Kernel 5.10

For host kernel 5.10 UFFD objects are created by calling into
userfaultfd syscall.

Kernel 6.1

For kernel 6.1, UFFD is created through the /dev/userfaultfd device. Access to
/dev/userfaultfd is managed by file system permissions, so the Firecracker
process needs to have proper permissions to create the UFFD object. When
/dev/userfaultfd is present on the host system, jailer makes it available
inside the jail and Firecracker process can use it without any further
configuration.

If a user is not using Firecracker along with the jailer, they should manage
manually permissions to /dev/userfaultfd. For example, on systems that rely on
access control lists (ACLs), this can be achieved by:

sudo setfacl -m u:${USER}:rw /dev/userfaultfd

Registering memory to be handled via Userfault File Descriptors Copy Section

Next, the memory address range must be registered with the userfault file
descriptor so that the userfault object can monitor page faults occurring for
those addresses. After this, the user space process can start reading and
serving events via the userfault file descriptor. These events will contain the
address that triggered the fault. The fault-handling thread can choose to handle
these events using these
operations.

In the flow described above, there are two userspace processes that interact
with each other in order to handle page faults: Firecracker process and the page
fault handler. Please note that users are responsible for writing the page fault
handler process to monitor userfaultfd events and handle those events.

Below is the interaction flow between Firecracker and the page fault handler
(designed by the users):

• Page fault handler binds and listens on a unix domain socket in order to be
  able to communicate with the Firecracker process.

Please note that when using the Jailer, the page fault handler process, UDS and
memory file must reside inside the jail. The UDS must only be accessible to
Firecracker and the page fault handler.

• PUT snapshot/load API call is issued towards Firecracker's API thread. The
  request encapsulates in its body the path to the unix domain socket that page
  fault handler listens to in order to communicate with Firecracker.

• Firecracker process creates the userfault object and obtains the userfault
  file descriptor.

• The page fault handler privately mmaps the contents of the guest memory file.

• Firecracker anonymously mmaps memory based on the memory description found in
  the microVM state file and registers the memory regions with the userfault
  object in order for the userfaultfd to be aware of page fault events on these
  addresses. Firecracker then connects to the socket previously opened by the
  page fault process.

• Firecracker passes the userfault file descriptor and the guest memory layout
  (e.g. dimensions of each memory region, and their page size
  in KiB) to the page fault handler process through the socket.

• After sending the necessary information to the page fault handler, Firecracker
  continues with the normal cycle to restore from snapshot. It reads from the
  microVM state file the relevant serialized components and loads them into
  memory.

• Page faults that occur while Firecracker is touching guest memory are handled
  by the page fault handler process, which listens for events on the userfault
  file descriptor that Firecracker previously sent. When a page fault event
  happens, the page fault handler issues UFFDIO_COPY to load the previously
  mmaped file contents into the correspondent memory region.

After Firecracker sends the payload (i.e. mem mappings and file descriptor), no
other communication happens on the UDS socket (or otherwise) between Firecracker
and the page fault handler process.

Userfaultfd interaction with balloon Copy Section

The balloon device allows the host to reclaim memory from a microVM. For more
details on balloon, please refer to this doc.

When the balloon device asks for removal of a memory range, Firecracker calls
madvise with the MADV_DONTNEED flag in order to let the kernel know that it
can free up memory found in that specific area. On such a system call, the
userfaultfd interface sends UFFD_EVENT_REMOVE.

When implementing the logic for the page fault handler, users must identify
events of type UFFD_EVENT_REMOVE and handle them by zeroing out those pages.
This is because the memory is removed, but the area still remains monitored by
userfaultfd. After a cycle of inflation and deflation, page faults might happen
again for memory ranges that have been removed by balloon (and subsequently
zeroed out by the page fault handler). In such a case, the page fault handler
process must zero out the faulted page (instead of bringing it from file), as
recommended by
the userfaultfd documentation.

In case of a compromised balloon driver, the page fault handler can get flooded
with UFFD_EVENT_REMOVE. We recommend using the jailer's built-in cgroup
functionality as defense in depth, in order to limit resource usage of the
Firecracker process.

Caveats Copy Section

If the handler process crashes while Firecracker is resuming the snapshot,
Firecracker will hang when a page fault occurs. This is because Firecracker is
designed to wait for the requested page to be made available. If the page fault
handler process is no longer around when this happens, Firecracker will wait
forever. Users are expected to monitor the page fault handler's status or gather
metrics of hanged Firecracker process and implement a recycle mechanism if
necessary.

It is the page fault handler process's responsibility to handle any errors that
might occur and also send signals to Firecracker process to inform it of any
crashes/exits. The page fault handler can fetch Firecracker's PID through
getsockopt call with SO_PEERCRED option, which fetches credentials of the
peer process that is connected to the socket. The returned credentials contain:
PID, GID and UID of the peer process (Firecracker in the page fault handler's
case).

We recommend that the page fault handler includes timeouts for waiting on
Firecracker to connect to the UDS or send information over the UDS, in order to
account for unexpected cases when Firecracker crashes before being able to
connect/send data.

Example Copy Section

An example of a handler process can be found
here. The process is
designed to tackle faults on a certain address by loading into memory the entire
region that the address belongs to, but users can choose any other behavior that
suits their use case best.

Snapshotting/Network For Clones Copy Section

Network Connectivity for Clones

This document presents a strategy to ensure continued network connectivity for
multiple clones created from a single Firecracker microVM snapshot.

[!CAUTION]

This should be considered as just an example to get you started, and we don't
claim this is a performant or secure setup.

Setup Copy Chapter

There are two things which prevent network connectivity from resuming
out-of-the-box for clones created from the same snapshot: Firecracker currently
saves and attempts to restore network devices using the initially configured TAP
names, and each guest will be resumed with the same network configuration, most
importantly with the same IP address(es). To work around the former, each clone
should be started within a separate network namespace (we can have multiple TAP
interfaces with the same name, as long as they reside in distinct network
namespaces). The latter can be mitigated by leveraging iptables NAT support.

Let’s have a more detailed look at this approach. We assume each VM has a single
network interface attached. If multiple interfaces with full connectivity are
required, we simply repeat the relevant parts of this process for each
additional interface. A typical setup right before taking a snapshot involves
having a VM with a network interface backed by a TAP device (named vmtap0, for
example) with an IP address (referred to as the TAP IP address, for example
192.168.241.1/29), and an IP address configured inside the guest for the
corresponding virtio device (referred to as the guest IP address, for example
192.168.241.2/29).

Network namespaces Copy Section

Attempting to restore multiple clones from the same snapshot faces the problem
of every single one of them attempting to use a TAP device with the original
name, which is not possible by default. Therefore, we need to start each clone
in a separate network namespace. This is already possible using the --netns
jailer parameter, described in the documentation. The specified
namespace must already exist, so we have to create it first using

sudo ip netns add fc0

(where fc0 is the name of the network namespace we plan to use for this
specific clone - clone0). A new network namespace is initially empty, so we
also have to create a new tap interface within using

sudo ip netns exec fc0 ip tuntap add name vmtap0 mode tap

The ip netns exec <ns_name> <command> allows us to execute command in the
context of the specified network namespace (in the previous case, the secondary
command creates a new tap interface). Next we configure the new TAP interface to
match the expectations of the snapshotted guest by running

sudo ip netns exec fc0 ip addr add 192.168.241.1/29 dev vmtap0
sudo ip netns exec fc0 ip link set vmtap0 up

At this point we can start multiple clones, each in its separate namespace, but
they won’t have connectivity to the rest of the host, only the respective TAP
interfaces. However, interaction over the network is still possible; for example
we can connect over ssh to clone0 using

sudo ip netns exec fc0 ssh [email protected]

`veth` interfaces to connect the network namespaces Copy Section

In order to obtain full connectivity we have to begin by connecting the network
namespace to the rest of the host, and then solving the “same guest IP”
problem. The former requires the use of veth pairs - virtual interfaces that
are link-local to each other (any packet sent through one end of the pair is
immediately received on the other, and the other way around). One end resides
inside the network namespace, while the other is moved into the parent namespace
(the host global namespace in this case), and packets flow in or out according
to the network configuration. We have to pick IP addresses for both ends of the
veth pair. For clone index idx, let’s use
10..<(idx % 30) * 8>.1/24 for the endpoint residing in the host
namespace, and the same address ending with 2 for the other end which remains
inside the clone's namespace. Thus, for clone 0 the former is 10.0.0.1 and
the latter 10.0.0.2.

The first endpoint must have an unique name on the host, for example chosen as
veth(idx + 1) (so veth1 for clone 0). To create and setup the veth pair, we
use the following commands (for namespace fc0):

# create the veth pair inside the namespace
sudo ip link add name veth1 type veth peer name veth0 netns fc0

sudo ip netns exec fc0 ip addr add 10.0.0.2/24 dev veth0
sudo ip netns exec fc0 ip link set dev veth0 up

sudo ip addr add 10.0.0.1/24 dev veth1
sudo ip link set dev veth1 up

# designate the outer end as default gateway for packets leaving the namespace
sudo ip netns exec fc0 ip route add default via 10.0.0.1

`iptables` rules for VM egress connectivity Copy Section

The last step involves adding the iptables rules which change the
source/destination IP address of packets on the fly (thus allowing all clones to
have the same internal IP).

# Find the host egress device
UPSTREAM=$(ip -j route list default |jq -r '.[0].dev')
# anything coming from the VMs, we NAT the address
ip netns exec fc0 iptables -t nat -A POSTROUTING -s 192.168.241.1/29 -o veth0 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 10.0.0.0/30 -o $UPSTREAM -j MASQUERADE
# forward packets by default
iptables -P FORWARD ACCEPT
ip netns exec fc0 ip route add default via 10.0.0.1
ip netns exec fc0 iptables -P FORWARD ACCEPT

You may also want to configure the guest with a default route and a DNS
nameserver:

ip route default via 10.0.0.1
echo nameserver 8.8.8.8 >/etc/resolv.conf

Connectivity from the clone should be present at this point.

To make sure the guest also adjusts to the new environment, you can explicitly
clear the ARP/neighbour table in the guest:

ip -family inet neigh flush any
ip -family inet6 neigh flush any

Otherwise, packets originating from the guest might be using old Link Layer
Address for up to arp cache timeout seconds. After said timeout period,
connectivity will work both ways even without an explicit flush.

Renaming host device names Copy Section

In some environments where the jailer is not being used, restoring a snapshot
may be tricky because the tap device on the host will not be the same as the tap
device that the original VM was mapped to when it was snapshotted, for example
when the tap device comes from a pool of such devices.

In this case you can use the network_overrides parameter of the snapshot
restore API to specify which guest network device maps to which host tap device.

For example, if we have a network interface named eth0 in the snapshotted
microVM, we can override it to point to the host device vmtap01 during
snapshot resume, like this:

curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT 'http://localhost/snapshot/load' \
    -H  'Accept: application/json' \
    -H  'Content-Type: application/json' \
    -d '{
            "snapshot_path": "./snapshot_file",
            "mem_backend": {
                "backend_path": "./mem_file",
                "backend_type": "File"
            },
            "network_overrides": [
                 {
                     "iface_id": "eth0",
                     "host_dev_name": "vmtap01"
                 }
            ]
    }'

This may require reconfiguration of the networking inside the VM so that it is
still routable externally.
network setup documentation describes what
the typical setup is. If you are not using network namespaces or the jailer,
then the guest will have to be made aware (via vsock or other channel) that it
needs to reconfigure its network to match the network configured on the tap
device.

If the new TAP device, say vmtap3 has been configured to use a guest address
of 172.16.3.2 then after snapshot restore you would run something like:

# In the guest

# Clear out the previous addr and route
ip addr flush dev eth0
ip route flush dev eth0

# Configure the new address
ip addr add 172.16.3.2/30 dev eth0
ip route add default via 172.16.3.1/30 dev eth0

Ingress connectivity

The above setup only provides egress connectivity. If in addition we also want
to add ingress (in other words, make the guest VM routable outside the network
namespace), then we need to choose a "clone address" that will represent this VM
uniquely. For our example we can use IPs from 172.16.0.0/12, for example
172.16.0.1.

Then we can rewrite destination address heading towards the "clone address" to
the guest IP.

ip netns exec fc0 iptables -t nat -A PREROUTING -i veth0 \
    -d 172.16.0.1 -j DNAT --to 192.168.241.2

And add a route on the host so we can access the guest VM from the host network
namespace:

ip route add 172.16.0.1 via 10.0.0.2

To confirm that ingress connectivity works, try

ping 172.16.0.1
# or
ssh [email protected]

See also

For an improved setup with full ingress and egress connectivity to the
individual VMs, see
this discussion.

Snapshotting/Random For Clones Copy Section

Entropy for Clones

This document provides a high level perspective on the implications of restoring
multiple VM clones from a single snapshot. We start with an overview of the
Linux random number generation (RNG) facilities, then go through the potential
issues we’ve identified related to cloning state, and finally conclude with a
series of recommendations. It’s worth stressing that we aim to prevent stale
state being a problem only for the kernel interfaces. Some userspace
applications or libraries keep their own equivalent of entropy pools and suffer
from the same potential issues after being cloned. There is no generic solution
under the current programming model, and all we can do is recommend against
their use in pre-snapshot logic.

Background Copy Chapter

The Linux kernel exposes three main RNG interfaces to userspace: the
/dev/random and /dev/urandom special devices, and the getrandom syscall,
which are described in the random(7) man page. Moreover, Firecracker
supports the virtio-rng device which can provide additional
entropy to guest VMs. It draws its random bytes from the aws-lc-rs crate
which wraps the AWS-LC cryptographic library.

Traditionally, /dev/random has been considered a source of “true” randomness,
with the downside that reads block when the pool of entropy gets depleted. On
the other hand, /dev/urandom doesn’t block, which lead people believe that it
provides lower quality results.

It turns out the distinction in output quality is actually very hard to make.
According to this article, for kernel versions prior to 4.8, both devices
draw their output from the same pool, with the exception that /dev/random will
block when the system estimates the entropy count has decreased below a certain
threshold. The /dev/urandom output is considered secure for virtually all
purposes, with the caveat that using it before the system gathers sufficient
entropy for initialization may indeed produce low quality random numbers. The
getrandom syscall helps with this situation; it uses the /dev/urandom source
by default, but will block until it gets properly initialized (the behavior can
be altered via configuration flags).

Newer kernels (4.8+) have switched to an implementation where /dev/random
output comes from a pool called the blocking pool, the output of /dev/urandom
is given by a CSPRNG (cryptographically secure pseudorandom number generator),
and there’s also an input pool which gathers entropy from various sources
available on the system, and is used to feed into or seed the other two
components. A very detailed description is available here.

The details of this newer implementation are used to make the recommendations
present in the document. There are in-kernel interfaces used to obtain random
numbers as well, but they are similar to using /dev/urandom (or getrandom
with the default source) from userspace.

Whenever a VM clone is created based on a snapshot, execution resumes precisely
from the previously saved state. Getting random bytes from either /dev/random
or /dev/urandom does not lead to identical results for different clones
created from the same snapshot because multiple parameters (such as timer data,
or output from CPU HWRNG instructions which are present on Ivy Bridge or newer
Intel processors and enabled in a Firecracker guest) are mixed with each result.
Extra bits are mixed in both when reading random values, and in conjunction with
entropy related events such as interrupts. Moreover, the guest kernel will
eventually receive fresh entropy from virtio-rng, if attached. There are two
questions here:

• Is the CPU HWRNG output always mixed in when the feature is present (as
  opposed to only when the CPU HWRNG is trusted)?
• Is the added noise strong enough to consider the final RNG output sufficiently
  divergent from all other clones?

Both these questions are particularly relevant immediately after resuming a VM
from a snapshot. After the VM gets to run for a "sufficient" amount of time it
should be able to gather some more entropy by itself and its state should be
sufficiently divergent that of any other clones.

It seems the CPU HWRNG is always added to mix when present. More specifically,
page 32 point 1 (at the top of the page) mentions using the CPU HWRNG
when present for the entropy pool output function. Page 34 states in case a CPU
random number generator is known to the Linux-RNG, data from that hardware RNG
is mixed into the entropy pool in a second step. With respect to the
initialization of the random pools and DRNG behind /dev/urandom. The discussion
regarding DRNG state on page 35 mentions the key part, the counter, and the
nonce are XORed with the output of the CPU random number generator if one is
present. If it is not present, one high-resolution time stamp obtained with the
kernel function random_get_entropy word is XORed with the key part. The
CPU HWRNG is also used for the DRNG state transition function (as stated on
page 36 point 1), and during the reseed operation (page 37 point 2). The
document explicitly mentions when the CPU HWRNG has to be trusted (for
example, the bullet points at the end of Section 3.3.2.3).

It’s not yet clear whether the noise that gets added for each clone post restore
is sufficient to consider their RNG states distinct for security purposes. The
conservative approach is to presume the stale state has a significant influence
on RNG output, so we should reinitialize both sources based on fresh data after
each restore. It would seem that simply writing data to /dev/urandom is enough
to muddle the entropy pools, but the bits only get mixed with the input pool.
It’s not certain at this point whether such writes have any immediate impact on
the blocking pool, and it’s unlikely they cause the CSPRNG to be automatically
reseeded.

The standard methods of interacting with the kernel RNG sources are documented
in the random(4) man page. It states that any writes to either
/dev/random or /dev/urandom are mixed with the input entropy pool, but do
not increase the current entropy estimation. There is also an ioctl interface
which, given the appropriate privileges, can be used to add data to the input
entropy pool while also increasing the count, or completely empty all pools.

Linux kernels with VMGenID support Copy Section

Linux has support for the
Virtual Machine Generation Identifier
since 5.18 for ACPI systems. Since 6.10, Linux added support also for systems
that use DeviceTree instead of ACPI. The purpose of VMGenID is to notify the
guest about time shift events, such as resuming from a snapshot. The device
exposes a 16-byte cryptographically random identifier in guest memory.
Firecracker implements VMGenID. When resuming a microVM from a snapshot
Firecracker writes a new identifier and injects a notification to the guest.
Linux
uses this value
as new randomness for its CSPRNG.
Quoting the random.c implementation of the kernel:

/*
 * Handle a new unique VM ID, which is unique, not secret, so we
 * don't credit it, but we do immediately force a reseed after so
 * that it's used by the crng posthaste.
 */

As a result, values returned by getrandom() and /dev/(u)random are distinct
in all VMs started from the same snapshot, after the kernel handles the
VMGenID notification. This leaves a race window between resuming vCPUs and Linux
CSPRNG getting successfully re-seeded.

Firecracker supports VMGenID on ARM systems using the DeviceTree binding that
was added for the device in Linux 6.10. However, the latest Linux kernel that
Firecracker supports is 6.1. As a result, in order to use VMGenID on ARM
systems, users need to use a 6.1 kernel with the DeviceTree binding support
backported from 6.10. For our CI we backport the
relevant changes
from 6.10 to 6.1. Consumers of Firecracker that want to use the feature on ARM
need to ensure they backport these changes on their guest kernels.

Please note that, Firecracker will always enable VMGenID. In kernels where there
is no VMGenID driver, the device will not have any effect in the guest.

User space considerations Copy Section

Init systems (such as systemd used by AL2 and other distros) might save a
random seed file after boot. For systemd, the path is
/var/lib/systemd/random-seed. Just to be on the safe side, any such file
should be deleted before taking a snapshot, to prevent its reuse for any
purposes by the guest. There’s also the /proc/sys/kernel/random/boot_id
special file, which gets initialized with a random string at boot time, and is
read-only afterwards. All clones restored from the same snapshot will implicitly
read the same value from this file. If that’s not desirable, it’s possible to
alter the read result via bind mounting another file on top of
/proc/sys/kernel/random/boot_id.

Recommendations Copy Chapter

• Delete /var/lib/systemd/random-seed, or any equivalent files.
• If changing the value present in /proc/sys/kernel/random/boot_id is
  important, bind mount another file on top of it.
• If microVMs run on machines with IvyBridge or newer Intel processors (which
  provide RDRAND; in addition, RDSEED is offered starting with Broadwell).
  Hardware supported reseeding is done on a cadence defined by the Linux Kernel
  and should be sufficient for most cases.
• Use virtio-rng. When present, the guest kernel uses the device as an
  additional source of entropy.
• On kernels before 5.18, to be as safe as possible, the direct approach is to
  do the following (before customer code is resumed in the clone):
  1. Open one of the special devices files (either /dev/random or
     /dev/urandom). Take note that RNDCLEARPOOL no longer
     has any effect on the entropy pool.
  2. Issue an RNDADDENTROPY ioctl call (requires CAP_SYS_ADMIN) to mix the
     provided bytes into the input entropy pool and increase the entropy count.
     This should also cause the /dev/urandom CSPRNG to be reseeded. The
     bytes can be generated locally in the guest, or obtained from the host.
  3. Issue a RNDRESEEDCRNG ioctl call (4.14, 5.10, (requires
     CAP_SYS_ADMIN)) that specifically causes the CSPRNG to be reseeded from
     the input pool.
• On kernels starting from 5.18 onwards, the CSPRNG will be automatically
  reseeded when the guest kernel handles the VMGenID notification. To completely
  avoid the race condition, users should follow the same steps as with kernels
  < 5.18.
• Userspace notifications of loading snapshots
  can be used to trigger the direct approach described above.

Annex 1 contains the source code of a C program which implements the previous
three steps. As soon as the guest kernel version switches to 4.19 (or higher),
we can rely on the CONFIG_RANDOM_TRUST_CPU kernel option (or the
random.trust_cpu=on cmdline parameter) to have the entropy pool automatically
refilled using the CPU HWRNG, so step 3 would no longer be necessary. Another
way around step 3 is to attach a virtio-rng device. However, we cannot control
when the guest kernel will request for random bytes from the device.

Annex 1: Source code that clears and reinitializes the entropy pool Copy Chapter

#include <errno.h>
#include <fcntl.h>
#include <linux/random.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>

void exit_usage() {
    printf("Usage: ./rerand [<hexadecimal_string>]\n"
           "The length of the string must be a multiple of 8.\n");
    exit(EXIT_FAILURE);
}

void exit_perror(const char *msg) {
    perror(msg);
    exit(EXIT_FAILURE);
}

int main(int argc, char ** argv) {
    if (argc > 2) {
        exit_usage();
    }

    size_t len = 0;
    struct rand_pool_info *info = NULL;

    if (argc == 2) {
        len = strlen(argv[1]);
        // We want len to be a multiple of 8 such that we have an easier time
        // parsing argv[1] into an array of u32s.
        if (len % 8) {
            exit_usage();
        }

        info = malloc(sizeof(struct rand_pool_info) + len / 8);
        if (info == NULL) {
            exit_perror("Could not alloc rand_pool_info struct");
        }
        // This is measured in bits IIRC.
        info->entropy_count = len * 4;
        info->buf_size = len / 8;
    }

    int fd = open("/dev/urandom", O_RDWR);
    if (fd < 0) {
        exit_perror("Unable to open /dev/urandom");
    }

    if (ioctl(fd, RNDCLEARPOOL) < 0) {
        exit_perror("Error issuing RNDCLEARPOOL operation");
    }

    if (argc == 1) {
        exit(EXIT_SUCCESS);
    }

    // Add the entropy bytes supplied by the user.
    char num_buf[9] = {};
    size_t pos = 0;

    while (pos < len) {
        memcpy(num_buf, &argv[1] + pos, 8);
        info->buf[pos / 8] = strtoul(num_buf, NULL, 16);
        pos += 8;
    }

    if (ioctl(fd, RNDADDENTROPY, info) < 0) {
        exit_perror("Error issuing RNDADDENTROPY operation");
    }
}

Snapshotting/Snapshot Editor Copy Section

Snapshot editor

The snapshot-editor is a program for modification of Firecracker snapshots.

Prior knowledge Copy Chapter

Firecracker snapshot consists of 2 files:

• vmstate file: file with Firecracker internal data such as vcpu states,
  devices states etc.
• memory file: file with guest memory.

Usage Copy Chapter

`edit-memory` command Copy Section

`rebase` subcommand

This command is used to merge a diff snapshot memory file on top of a base
memory file.

Note You can also use rebase-snap (deprecated) tool for this.

Arguments:

• MEMORY_PATH - path to the memory file
• DIFF_PATH - path to the diff file

Usage:

bash Copy

snapshot-editor edit-memory rebase \
     --memory-path <MEMORY_PATH> \
     --diff-path <DIFF_PATH>

Example:

bash Copy

snapshot-editor edit-memory rebase \
     --memory-path ./memory_file \
     --diff-path ./diff_file

`edit-vmstate` command Copy Section

`remove-regs` subcommand (aarch64 only)

This command is used to remove specified registers from vcpu states inside
vmstate snapshot file.

Arguments:

• VMSTATE_PATH - path to the vmstate file
• OUTPUT_PATH - path to the file where the output will be placed
• [REGS] - set of u32 values representing registers ids as they are defined
  in KVM. Can be both in decimal and in hex formats.

Usage:

bash Copy

snapshot-editor edit-vmstate remove-regs \
    --vmstate-path <VMSTATE_PATH> \
    --output-path <OUTPUT_PATH> \
    [REGS]...

Example:

bash Copy

./snapshot-editor edit-vmstate remove-regs \
    --vmstate-path ./vmstate_file \
    --output-path ./new_vmstate_file \
    0x1 0x2

`info-vmstate` command Copy Section

`version` subcommand

This command is used to print version of the provided vmstate file.

Arguments:

• VMSTATE_PATH - path to the vmstate file

Usage:

bash Copy

snapshot-editor info-vmstate version --vmstate-path <VMSTATE_PATH>

Example:

bash Copy

./snapshot-editor info-vmstate version --vmstate-path ./vmstate_file

`vcpu-states` subcommand

This command is used to print the vCPU states inside vmstate snapshot file.

Arguments:

• VMSTATE_PATH - path to the vmstate file

Usage:

bash Copy

snapshot-editor info-vmstate vcpu-states --vmstate-path <VMSTATE_PATH>

Example:

bash Copy

./snapshot-editor info-vmstate vcpu-states --vmstate-path ./vmstate_file

`vm-state` subcommand

This command is used to print the vmstate of snapshot file in readable format
thus, making it easier to compare vmstate of 2 snapshots.

Arguments:

• VMSTATE_PATH - path to the vmstate file

Usage:

bash Copy

snapshot-editor info-vmstate vm-state --vmstate-path <VMSTATE_PATH>

Example:

bash Copy

./snapshot-editor info-vmstate vm-state --vmstate-path ./vmstate_file

Snapshotting/Snapshot Support Copy Section

Firecracker Snapshotting

Table of Contents Copy Chapter

• What is microVM snapshotting?
• Snapshotting in Firecracker
  • Supported platforms
  • Overview
  • Snapshot files management
  • Performance
  • Developer preview status
  • Limitations
• Firecracker Snapshotting characteristics
• Snapshot versioning
• Snapshot API
  • Pausing the microVM
  • Creating snapshots
    • Creating full snapshots
    • Creating diff snapshots
  • Resuming the microVM
  • Loading snapshots
• Provisioning host disk space for snapshots
• Ensure continued network connectivity for clones
• Snapshot security and uniqueness
  • Secure and insecure usage examples
  • Reusing snapshotted states securely
  • Userspace notifications of loading snapshots
• Vsock device reset
• VMGenID device limitation
• Where can I resume my snapshots?

About microVM snapshotting Copy Chapter

MicroVM snapshotting is a mechanism through which a running microVM and its
resources can be serialized and saved to an external medium in the form of a
snapshot. This snapshot can be later used to restore a microVM with its guest
workload at that particular point in time.

Snapshotting in Firecracker Copy Chapter

Supported platforms Copy Section

The Firecracker snapshot feature is supported on all CPU micro-architectures
listed in README.

Overview Copy Section

A Firecracker microVM snapshot can be used for loading it later in a different
Firecracker process, and the original guest workload is being simply resumed.

The original guest which the snapshot is created from, should see no side
effects from this process (other than the latency introduced by the snapshot
creation process).

Both network and vsock packet loss can be expected on guests that are resumed
from snapshots in another Firecracker process. It is also not guaranteed that
the state of the network connections survives the process. Furthermore, vsock
connections that are open when the snapshot is taken are closed, but existing
vsock listen sockets in the guest still remain active and can accept new
connections after resume (see Vsock device reset).

In order to make restoring possible, Firecracker snapshots save the full state
of the following resources:

• the guest memory,
• the emulated HW state (both KVM and Firecracker emulated HW).

The state of the components listed above is generated independently, which
brings flexibility to our snapshotting support. This means that taking a
snapshot results in multiple files that are composing the full microVM snapshot:

• the guest memory file,
• the microVM state file,
• zero or more disk files (depending on how many the guest had; these are
  managed by the users).

The design allows sharing of memory pages and read only disks between multiple
microVMs. When loading a snapshot, instead of loading at resume time the full
contents from file to memory, Firecracker creates a
MAP_PRIVATE mapping of the
memory file, resulting in runtime on-demand loading of memory pages. Any
subsequent memory writes go to a copy-on-write anonymous memory mapping. This
has the advantage of very fast snapshot loading times, but comes with the cost
of having to keep the guest memory file around for the entire lifetime of the
resumed microVM.

Snapshot files management Copy Section

The Firecracker snapshot design offers a very simple interface to interact with
snapshots but provides no functionality to package or manage them on the host.

The threat containment model states that the
host, host/API communication and snapshot files are trusted by Firecracker.

To ensure a secure integration with the snapshot functionality, users need to
secure snapshot files by implementing authentication and encryption schemes
while managing their lifecycle or moving them across the trust boundary, like
for example when provisioning them from a repository to a host over the network.

Firecracker is optimized for fast load/resume, and it's designed to do some very
basic sanity checks only on the vm state file. It only verifies integrity using
a 64-bit CRC value embedded in the vm state file, but this is only a partial
measure to protect against accidental corruption, as the disk files and memory
file need to be secured as well. It is important to note that CRC computation is
validated before trying to load the snapshot. Should it encounter failure, an
error will be shown to the user and the Firecracker process will be terminated.

Performance Copy Section

The Firecracker snapshot create/resume performance depends on the memory size,
vCPU count and emulated devices count. The Firecracker CI runs snapshot tests on
all supported platforms.

Developer preview status Copy Section

Diff snapshots are still in developer preview while we are diving deep into how
the feature can be combined with guest_memfd support in Firecracker.

Limitations Copy Section

• High snapshot restoration latency when cgroups V1 are in use. We strongly
  recommend to deploy snapshots on cgroups V2 enabled hosts for the implied
  kernel versions -
  related issue.
• Guest network connectivity is not guaranteed to be preserved after resume. For
  recommendations related to guest network connectivity for clones please see
  Network connectivity for clones.
• Snapshotting on arm64 works for both GICv2 and GICv3 enabled guests. However,
  restoring between different GIC version is not possible.
• If a CPU template is not used on x86_64,
  overwrites of MSR_IA32_TSX_CTRL MSR value will not be preserved after
  restoring from a snapshot.
• Resuming from a snapshot that was taken during early stages of the guest
  kernel boot might lead to crashes upon snapshot resume. We suggest that users
  take snapshot after the guest microVM kernel has booted. Please see
  VMGenID device limitation.

Firecracker Snapshotting characteristics Copy Chapter

• Fresh Firecracker microVMs are booted using anonymous memory, while microVMs
  resumed from snapshot load memory on-demand from the snapshot and
  copy-on-write to anonymous memory.
• Resuming from a snapshot is optimized for speed, while taking a snapshot
  involves some extra CPU cycles for synchronously writing memory pages to the
  memory snapshot file. Taking a full snapshot of a microVM results in the full
  contents of guest memory being written to the snapshot, and particularly, in
  all guest memory being faulted in.
• The memory file and microVM state file are generated by Firecracker on
  snapshot creation. The disk contents are not explicitly flushed to their
  backing files.
• The API calls exposing the snapshotting functionality have clear
  Prerequisites that describe the requirements on when/how they should be
  used.
• The Firecracker microVM's MMDS config is included in the snapshot. However,
  the data store is not persisted across snapshots.
• Configuration information for metrics and logs are not saved to the snapshot.
  These need to be reconfigured on the restored microVM.
• On x86_64, if a vCPU has MSR_IA32_TSC_DEADLINE set to 0 when a snapshot is
  taken, Firecracker replaces it with the MSR_IA32_TSC value from the same vCPU.
  This is to guarantee that the vCPU will continue receiving TSC interrupts
  after restoring from the snapshot even if an interrupt is lost when taking a
  snapshot.

Snapshot versioning Copy Chapter

The microVM state snapshot file uses a data format that has a version in the
form of MAJOR.MINOR.PATCH. Each Firecracker binary supports a fixed version of
the snapshot data format. When creating a snapshot, Firecracker will use the
supported data format version. When loading snapshots, Firecracker will check
that the snapshot version is compatible with the version it supports. More
information about the snapshot data format and details about snapshot data
format versions can be found at versioning.

Snapshot API Copy Chapter

Firecracker exposes the following APIs for manipulating snapshots: Pause,
Resume and CreateSnapshot can be called only after booting the microVM,
while LoadSnapshot is allowed only before boot.

Pausing the microVM Copy Section

To create a snapshot, first you have to pause the running microVM and its vCPUs
with the following API command:

curl --unix-socket /tmp/firecracker.socket -i \
    -X PATCH 'http://localhost/vm' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
            "state": "Paused"
    }'

Prerequisites: The microVM is booted. Successive calls of this request keep
the microVM in the Paused state. Effects:

• on success: microVM is guaranteed to be Paused.
• on failure: no side-effects.

Creating snapshots Copy Section

[!WARNING]

Diff snapshot support is in developer preview. See
this section for more info.

Now that the microVM is paused, you can create a snapshot, which can be either a
fullone or a diff one. Full snapshots always create a complete, resume-able
snapshot of the current microVM state and memory. Diff snapshots save at least
the current microVM state and the memory accessed since the last snapshot (full
or diff) in a sparse file (but they might include more pages than strictly
needed due to technical limitation in Firecracker's ability to accurately track
accesses). Diff snapshots are generally not resume-able, but must be merged with
a base snapshot into a full snapshot. The exception here are diff snapshots of
booted VMs, which are immediately resumable. In this context, we will refer to
the base as the first memory file created by a /snapshot/create API call and
the layer as a memory file created by a subsequent /snapshot/create API call.
The order in which the snapshots were created matters and they should be merged
in the same order in which they were created. To merge a diff snapshot memory
file on top of a base, users should copy its content over the base. This can be
done using the rebase-snap (deprecated) or snapshot-editor tools provided
with the firecracker release:

snapshot-editor edit-memory rebase \
     --memory-path path/to/base \
     --diff-path path/to/layer

After executing the command above, the base would be a resumable snapshot memory
file describing the state of the memory at the moment of creation of the layer.
More layers which were created later can be merged on top of this base.

This process needs to be repeated for each layer until the one describing the
desired memory state is merged on top of the base, which is constantly updated
with information from previously merged layers. Please note that users should
not merge state files which resulted from /snapshot/create API calls and they
should use the state file created in the same call as the memory file which was
merged last on top of the base.

Creating full snapshots

For creating a full snapshot, you can use the following API command:

curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT 'http://localhost/snapshot/create' \
    -H  'Accept: application/json' \
    -H  'Content-Type: application/json' \
    -d '{
            "snapshot_type": "Full",
            "snapshot_path": "./snapshot_file",
            "mem_file_path": "./mem_file"
    }'

Details about the required and optional fields can be found in the
swagger definition.

[!NOTE]

If the files indicated by snapshot_path and mem_file_path don't exist at
the specified paths, then they will be created right before generating the
snapshot. If they exist, the files will be truncated and overwritten.

Prerequisites: The microVM is Paused.

Effects:

• on success:

  • The file indicated by snapshot_path (e.g. /path/to/snapshot_file)
    contains the devices' model state and emulation state. The one indicated by
    mem_file_path(e.g. /path/to/mem_file) contains a full copy of the guest
    memory.
  • The generated snapshot files are immediately available to be used (current
    process releases ownership). At this point, the block devices backing files
    should be backed up externally by the user. Please note that block device
    contents are only guaranteed to be committed/flushed to the host FS, but not
    necessarily to the underlying persistent storage (could still live in host
    FS cache).
  • If dirty page tracking is enabled, the snapshot creation resets then the
    dirtied page bitmap and marks all pages clean (from a dirty page tracking
    point of view).

• on failure: no side-effects.

Notes:

• The separate block device file components of the snapshot have to be handled
  by the user.

Creating diff snapshots

For creating a diff snapshot, you should use the same API command, but with
snapshot_type field set to Diff.

[!NOTE]

If not specified, snapshot_type is by default Full.

bash Copy

curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT 'http://localhost/snapshot/create' \
    -H  'Accept: application/json' \
    -H  'Content-Type: application/json' \
    -d '{
            "snapshot_type": "Diff",
            "snapshot_path": "./snapshot_file",
            "mem_file_path": "./mem_file"
    }'

Prerequisites: The microVM is Paused.

Diff snapshots come in two flavors. If track_dirty_pages was set to true
when configuring the /machine-config resource or when restoring from a
snapshot via /snapshot/load, Firecracker will use KVM's dirty page log runtime
functionality to ensure the diff snapshot only contains exactly pages that were
written to since boot / snapshot restoration. If track_dirty_pages is not
enabled, Firecracker uses the mincore(2) syscall to determine
which pages to include in the snapshot. As such, this mode of snapshot taking
will only work if swap is disabled, as mincore does not consider pages written
to swap to be "in core". This potentially results in bigger memory files
(although they are still sparse), but avoids the runtime overhead of dirty page
logging.

[!NOTE]

Dirty page tracking negates most of the benefits of
huge pages.

Effects:

• on success:
  • The file indicated by snapshot_path contains the devices' model state and
    emulation state, same as when creating a full snapshot. The one indicated by
    mem_file_path contains this time a diff copy of the guest memory; the
    diff consists of the memory pages which have been dirtied since the last
    snapshot creation or since the creation of the microVM, whichever of these
    events was the most recent.
  • All the other effects mentioned in the Effects paragraph from Creating
    full snapshots section apply here.
• on failure: no side-effects.

[!NOTE]

This is an example of an API command that enables dirty page tracking:

bash Copy

curl --unix-socket /tmp/firecracker.socket -i  \
    -X PUT 'http://localhost/machine-config' \
    -H 'Accept: application/json'            \
    -H 'Content-Type: application/json'      \
    -d '{
            "vcpu_count": 2,
            "mem_size_mib": 1024,
            "smt": false,
            "track_dirty_pages": true
    }'

Enabling this support enables KVM dirty page tracking, so it comes at a cost
(which consists of CPU cycles spent by KVM accounting for dirtied pages); it
should only be used when needed.

Creating a snapshot has some minor effects on the currently running microVM:

• The vsock device is reset, causing the driver to
  terminate connection on resumption.
• On x86_64, a notification for KVM-clock is injected to notify the guest about
  being paused.

Resuming the microVM Copy Section

You can resume the microVM by sending the following API command:

curl --unix-socket /tmp/firecracker.socket -i \
    -X PATCH 'http://localhost/vm' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
            "state": "Resumed"
    }'

Prerequisites: The microVM is Paused. Successive calls of this request are
ignored (microVM remains in the running state). Effects:

• on success: microVM is guaranteed to be Resumed.
• on failure: no side-effects.

Loading snapshots Copy Section

If you want to load a snapshot, you can do that only before the microVM is
configured (the only resources that can be configured prior are the logger and
the metrics systems) by sending the following API command:

curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT 'http://localhost/snapshot/load' \
    -H  'Accept: application/json' \
    -H  'Content-Type: application/json' \
    -d '{
            "snapshot_path": "./snapshot_file",
            "mem_backend": {
                "backend_path": "./mem_file",
                "backend_type": "File"
            },
            "track_dirty_pages": true,
            "resume_vm": false
    }'

The backend_type field represents the memory backend type used for loading the
snapshot. Accepted values are:

• File - rely on the kernel to handle page faults when loading the contents of
  the guest memory file into memory.
• Uffd - use a dedicated user space process to handle page faults that occur
  for the guest memory range. Please refer to
  this for more details on
  handling page faults in the user space.

The meaning of backend_path depends on the backend_type chosen:

• if using File, then backend_path should contain the path to the snapshot's
  memory file to be loaded.
• when using Uffd, backend_path refers to the path of the unix domain socket
  used for communication between Firecracker and the user space process that
  handles page faults.

When relying on the OS to handle page faults, the command below is also
accepted. Note that mem_file_path field is currently under the deprecation
policy. mem_file_path and mem_backend are mutually exclusive, therefore
specifying them both at the same time will return an error.

curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT 'http://localhost/snapshot/load' \
    -H  'Accept: application/json' \
    -H  'Content-Type: application/json' \
    -d '{
            "snapshot_path": "./snapshot_file",
            "mem_file_path": "./mem_file",
            "track_dirty_pages": true,
            "resume_vm": false
    }'

Details about the required and optional fields can be found in the
swagger definition.

Prerequisites: A full memory snapshot and a microVM state file must be
provided. The disk backing files, network interfaces backing TAPs and/or vsock
backing socket that were used for the original microVM's configuration should be
set up and accessible to the new Firecracker process (in which the microVM is
resumed). These host-resources need to be accessible at the same relative paths
to the new Firecracker process as they were to the original one.

Effects:

• on success:
  • The complete microVM state is loaded from snapshot into the current
    Firecracker process.
  • It then resets the dirtied page bitmap and marks all pages clean (from a
    diff snapshot point of view).
  • The loaded microVM is now in the Paused state, so it needs to be resumed
    for it to run.
  • The memory file (pointed by backend_path when using File backend type,
    or pointed by mem_file_path) must be considered immutable from
    Firecracker and host point of view. It backs the guest OS memory for read
    access through the page cache. External modification to this file corrupts
    the guest memory and leads to undefined behavior.
  • The file indicated by snapshot_path, that is used to load from, is
    released and no longer used by this process.
  • If track_dirty_pages is set, subsequent diff snapshots will be based on
    KVM dirty page tracking.
  • If resume_vm is set, the vm is automatically resumed if load is
    successful.
• on failure: A specific error is reported and then the current Firecracker
  process is ended (as it might be in an invalid state).

Notes: The track_dirty_pages configuration is not saved when creating a
snapshot, so you need to explicitly set track_dirty_pages again when sending
the LoadSnapshot command if you want to be able to do dirty page tracking
based diff snapshots from a loaded microVM.

It is also worth knowing, a microVM that is restored from snapshot will be
resumed with the guest OS wall-clock continuing from the moment of the snapshot
creation. For this reason, the wall-clock should be updated to the current time,
on the guest-side. More details on how you could do this can be found at a
related FAQ.
When using kvm-clock as clock source on x86_64, it's possible to optionally
set the clock_realtime: true in the LoadSnapshot request to advance the
clock on the guest at restore time (host Linux >= 5.16 is required to support
this feature). Note that this may cause issues within the guest as the clock
will appear to suddenly jump.

Provisioning host disk space for snapshots Copy Chapter

Depending on VM memory size, snapshots can consume a lot of disk space.
Firecracker integrators must ensure that the provisioned disk space is
sufficient for normal operation of their service as well as during failure
scenarios. If the service exposes the snapshot triggers to customers,
integrators must enforce proper disk quotas to avoid any DoS threats that
would cause the service to fail or function abnormally.

Ensure continued network connectivity for clones Copy Chapter

For recommendations related to continued network connectivity for multiple
clones created from a single Firecracker microVM snapshot please see
this doc.

Snapshot security and uniqueness Copy Chapter

When snapshots are used in a such a manner that a given guest's state is resumed
from more than once, guest information assumed to be unique may in fact not be;
this information can include identifiers, random numbers and random number
seeds, the guest OS entropy pool, as well as cryptographic tokens. Without a
strong mechanism that enables users to guarantee that unique things stay unique
across snapshot restores, we consider resuming execution from the same state
more than once insecure.

For more information please see this doc

Usage examples Copy Section

Example 1: secure usage

Boot microVM A -> ... -> Create snapshot S -> Terminate
                                           -> Load S in microVM B -> Resume -> ...

Here, microVM A terminates after creating the snapshot without ever resuming
work, and a single microVM B resumes execution from snapshot S. In this case,
unique identifiers, random numbers, and cryptographic tokens that are meant to
be used once are indeed only used once. In this example, we consider microVM B
secure.

Example 2: potentially insecure usage

Boot microVM A -> ... -> Create snapshot S -> Resume -> ...
                                           -> Load S in microVM B -> Resume -> ...

Here, both microVM A and B do work starting from the state stored in snapshot S.
Unique identifiers, random numbers, and cryptographic tokens that are meant to
be used once may be used twice. It doesn't matter if microVM A is terminated
before microVM B resumes execution from snapshot S or not. In this example, we
consider both microVMs insecure as soon as microVM A resumes execution.

Example 3: potentially insecure usage

Boot microVM A -> ... -> Create snapshot S -> ...
                                           -> Load S in microVM B -> Resume -> ...
                                           -> Load S in microVM C -> Resume -> ...
                                           [...]

Here, both microVM B and C do work starting from the state stored in snapshot S.
Unique identifiers, random numbers, and cryptographic tokens that are meant to
be used once may be used twice. It doesn't matter at which points in time
microVMs B and C resume execution, or if microVM A terminates or not after the
snapshot is created. In this example, we consider microVMs B and C insecure, and
we also consider microVM A insecure if it resumes execution.

Reusing snapshotted states securely Copy Section

Virtual Machine Generation Identifier
(VMGenID) is a virtual device that allows VM guests to detect when they have
resumed from a snapshot. It works by exposing a cryptographically random
16-bytes identifier to the guest. The VMM ensures that the value of the
identifier changes every time the VM a time shift happens in the lifecycle of
the VM, e.g. when it resumes from a snapshot.

Linux supports VMGenID since version 5.18 for systems with ACPI support. Linux
6.10 added support also for systems that use DeviceTree instead of ACPI. When
Linux detects a change in the identifier, it uses its value to reseed its
internal PRNG.

Firecracker supports VMGenID device both on x86 and Aarch64 platforms.
Firecracker will always enable the device. During snapshot resume, Firecracker
will update the 16-byte generation ID and inject a notification in the guest
before resuming its vCPUs.

As a result, guests that run Linux versions >= 5.18 will re-seed their in-kernel
PRNG upon snapshot resume. User space applications can rely on the guest kernel
for randomness. State other than the guest kernel entropy pool, such as unique
identifiers, cached random numbers, cryptographic tokens, etc will still be
replicated across multiple microVMs resumed from the same snapshot. Users need
to implement mechanisms for ensuring de-duplication of such state, where needed.

Userspace notifications of loading snapshots Copy Chapter

VMClock device
(specification) is a
device that enables efficient application clock synchronization against real
wallclock time, for applications running inside virtual machines. VMClock also
takes care situations where there is some sort disruption happens to the clock.
It handles these through fields in the
vmlcock_abi.
Currently, it handles two cases:

1. Live migration through the disruption_marker field.
2. Restore from snapshots through the vm_generation_counter.

Whenever a VM starts from a snapshot VMClock will present a new (different that
what was previously stored) value in the vm_generation_counter. This happens
in an atomic way, i.e. vm_generation_counter will include the new value as
soon as vCPUs are resumed post snapshot loading.

User space libraries, e.g. userspace PRNGs can mmap() vmclock_abi and monitor
changes in vm_generation_counter to observe when they need to adapt and/or
recreate state.

Moreover, VMClock allows processes to call poll() on the VMClock device and get
notified about changes through an event loop.

For reference, the C code used in our tests is available
here.

[!IMPORTANT]

Support for vm_generation_counter and poll() is implemented in Linux
through the patches
here
and was merged in Linux kernel v7.0. Users need to make sure that the linked
patches are applied on their kernels. These patches have been backported to
Amazon Linux v5.10 and v6.1 microVM kernels, starting from
microvm-kernel-5.10.252-285.992.amzn2
and
microvm-kernel-6.1.167-27.319.amzn2023
respectively. These patched kernels will be automatically downloaded when
running the integration test suite, or they can be manually downloaded using
tools/devtool download_ci_artifacts.

Vsock device reset Copy Chapter

The vsock device is reset across snapshot/restore to avoid inconsistent state
between device and driver leading to breakage
(#2218). This
is done by sending a VIRTIO_VSOCK_EVENT_TRANSPORT_RESET event to the guest
driver during SnapshotCreate
(#2562). On
SnapshotResume, when the VM becomes active again, the vsock driver closes all
existing connections. Existing listen sockets still remain active, but their CID
is updated to reflect the current guest_cid. More details about this event can
be found in the official Virtio document
here.

VMGenID device limitation Copy Chapter

During snashot resume, Firecracker updates the 16-byte generation ID of the
VMGenID device and injects an interrupt in the guest before resuming vCPUs. If
the snapshot was taken at the very early stages of the guest kernel boot process
proper interrupt handling might not be in place yet. As a result, the kernel
might not be able to handle the injected notification and crash. We suggest to
users that they take snapshots only after the guest kernel has completed
booting, to avoid this issue.

Where can I resume my snapshots? Copy Chapter

Snapshots must be resumed on an software and hardware configuration which is
identical to what they were generated on. However, in limited cases, snapshots
can be resumed on identical hardware instances where they were taken on, but
using newer host kernel versions. While we do not provide any guarantees on this
setup (and do not recommend doing this in production), we are currently aware of
the compatibility table reported below:

For example, a snapshot taken on a m6i.metal host (Intel Ice Lake) running a
5.10 host kernel can be restored on a different m6i.metal host running a 6.1
host kernel (but not vice versa), but could not be restored on a m5n.metal host
(Intel Cascade Lake).

Snapshotting/Versioning Copy Section

Firecracker snapshot versioning

This document describes how Firecracker persists microVM state into Firecracker
snapshots. It describes the snapshot format, encoding, compatibility and
limitations.

Introduction Copy Chapter

Firecracker uses the serde crate 1 along with the bitcode 2 format to
serialize its state into Firecracker snapshots. Firecracker snapshots have
versions that are independent of Firecracker versions. Each Firecracker version
declares support for a specific snapshot data format version. When creating a
snapshot, Firecracker will use the supported snapshot format version. When
loading a snapshot, Firecracker will check that format of the snapshot file is
compatible with the snapshot version Firecracker supports.

Overview Copy Chapter

Firecracker persists the microVM state as 2 separate objects:

• a guest memory file
• a microVM state file.

The block devices attached to the microVM are not considered part of the state
and need to be managed separately.

Guest memory Copy Section

The guest memory file contains the microVM memory saved as a dump of all pages.

MicroVM state Copy Section

In the VM state file, Firecracker stores the internal state of the VMM (device
emulation, KVM and vCPUs) with 2 exceptions - serial emulation and vsock
backend.

While we continuously improve and extend Firecracker's features by adding new
capabilities, devices or enhancements, the microVM state file may change both
structurally and semantically with each new release.

MicroVM state file format Copy Chapter

A Firecracker snapshot has the following format:

The snapshot format has its own version encoded in the snapshot file itself
after the snapshot's magic_id. The snapshot format version is independent of
the Firecracker version and it is of the form MAJOR.MINOR.PATCH.

Currently, Firecracker uses the
Serde bitcode encoder for
serializing the microVM state. The encoding format that bitcode uses does not
allow backwards compatible changes in the state, so essentially every change in
the microVM state description will result in bump of the format's MAJOR
version. If the needs arises, we will look into alternative formats that allow
more flexibility with regards to backwards compatibility. If/when this happens,
we will define how changes in the snapshot format reflect to changes in its
MAJOR.MINOR.PATCH version.

VM state encoding Copy Chapter

During research and prototyping we considered multiple storage formats. The
criteria used for comparing these are: performance, size, rust support,
specification, versioning support, community and tooling. Performance, size and
Rust support are hard requirements while all others can be the subject of trade
offs. More info about this comparison can be found
here.

Key benefits of using bitcode:

• Minimal snapshot size overhead
• Minimal CPU overhead
• Simple implementation

The current implementation relies on the
Serde bitcode encoder.

Snapshot compatibility Copy Chapter

Host kernel Copy Section

Snapshots can be saved and restored on the same kernel version without any
issues. There might be issues when restoring snapshots created on different host
kernel version even when using the same Firecracker version.

SnapshotCreate and SnapshotLoad operations across different host kernels is
considered unstable in Firecracker as the saved KVM state might have different
semantics on different kernels.

Device model Copy Section

The current Firecracker devices are backwards compatible up to the version that
introduces them. Ideally this property would be kept over time, but there are
situations when a new version of a device exposes new features to the guest that
do not exist in an older version. In such cases restoring a snapshot at an older
version becomes impossible without breaking the guest workload.

The microVM state file links some resources that are external to the snapshot:

• tap devices by device name,
• block devices by block file path,
• vsock backing Unix domain socket by socket name.

To successfully restore a microVM one should check that:

• tap devices are available, their names match their original names since these
  are the values saved in the microVM state file, and they are accessible to the
  Firecracker process where the microVM is being restored,
• block devices are set up at their original relative or absolute paths with the
  proper permissions, as the Firecracker process with the restored microVM will
  attempt to access them exactly as they were accessed in the original
  Firecracker process,
• the vsock backing Unix domain socket is available, its name matches the
  original name, and it is accessible to the new Firecracker process.

CPU model Copy Section

Firecracker microVMs snapshot functionality is available for Intel/AMD/ARM64 CPU
models that support the hardware virtualizations extensions, more details are
available here. Snapshots are not
compatible across CPU architectures and even across CPU models of the same
architecture. They are only compatible if the CPU features exposed to the guest
are an invariant when saving and restoring the snapshot. The trivial scenario is
creating and restoring snapshots on hosts that have the same CPU model.

Restoring from an Intel snapshot on AMD (or vice-versa) is not supported.

It is important to note that guest workloads can still execute instructions that
are being masked by CPUID and restoring and
saving of such workloads will lead to undefined result. Firecracker retrieves
the state of a discrete list of MSRs from KVM, more specifically, the MSRs
corresponding to the guest exposed features.

Implementation Copy Chapter

The microVM state file format is implemented in the
snapshot crate in the Firecracker
repository. All Firecracker devices implement the
Persist trait which exposes an
interface that enables creating from and saving to the microVM state.

Api Change Runbook Copy Section

Runbook for Firecracker API changes

This runbook will cover triaging API changes and ways to implement them
appropriately.

Definitions Copy Chapter

• Deprecated - We will consider a deprecated API element (endpoint and/or
  parts of an endpoint) to be an element which still provides users with access
  to its backing functionality and can be used, but will soon be removed
  completely along with said functionality in an upcoming version.
• Mandatory endpoint - We will consider an endpoint mandatory if Firecracker
  cannot operate normally without performing a request to it.
• Optional endpoint - We will consider an endpoint optional if Firecracker can
  operate normally without performing a request to it and the functionality
  behind it is not essential.
• Mandatory header/field - We will consider a header/field mandatory in an
  HTTP message if the request will fail without specifying said header/field.
• Optional header/field - We will consider a header/field optional in an HTTP
  message if the request will succeeds without specifying said header/field.

Triaging API changes Copy Chapter

For the purposes of this document, there are 2 main categories for API changes,
namely breaking and non-breaking.

What is a breaking change? Copy Section

A breaking change in the API is a change that makes the API incompatible with
the previous version (backwards incompatible). In an effort to avoid a breaking
change, we may take the route of deprecation and incrementing the minor version
in an effort to preserve backwards compatibility, but breaking changes will
always ultimately result in incrementing the major version. Here is a
non-exhaustive list of such changes:

1. Adding a new mandatory endpoint/HTTP method.
2. Removing an endpoint/method.
3. Adding a mandatory request header/field.
4. Removing a request header/field.
5. Adding a mandatory response field.
6. Removing a response header/field.

What is NOT a breaking change? Copy Section

A change in the API is not a breaking change if the version resulting from it is
compatible with the previous one (backwards compatible). The outcome of a
non-breaking change should always include incrementing the minor version but
must not lead to incrementing the major version by itself. Here is a
non-exhaustive list of such changes:

1. Deprecating an endpoint/method/field.
2. Adding a new optional endpoint/method.
3. Adding an optional request header/field.
4. Adding a response header.
5. Adding additional valid inputs for fields in API requests.
6. Making mandatory headers/fields optional.
7. Making mandatory endpoints optional.
8. Changing the URI of an endpoint.
9. Changing the metrics output format.

Implementing API changes Copy Chapter

API changes result in version increases. As Firecracker’s support policy is
based on semantic versioning 2.0.0, we will look at API changes from this
point of view.

Given a version number MAJOR.MINOR.PATCH, increment the: MAJOR version when
you make incompatible API changes; MINOR version when you add functionality in
a backwards compatible manner; PATCH version when you make backwards
compatible bug fixes.

All deprecated endpoints are supported until at least the next major version
release, where they may be removed.

How to follow the flowchart - with examples Copy Section

We will go through multiple types of API changes and provide ways to ensure we
don’t break our backwards compatibility promise to our customers. The list is
split into categories of components changed.

• Entire endpoints
  • Adding an optional endpoint with new functionality - Increment minor
    version.
  • Adding a command line parameter - Increment minor version.
  • Removing an endpoint - Deprecate endpoint and increment minor version →
    Remove endpoint when incrementing major version.
  • Adding a mandatory endpoint - Increment major version.
• Request
  • Adding an optional header/field - Increment minor version.
  • Renaming a header/field - Accept both names and deprecate the old one →
    Remove old name when incrementing major version.
  • Removing a header/field - Make said header/field optional → Remove
    header/field when incrementing major version.
  • Changing the URI of an endpoint - Redirect the old endpoint to the new one
    and deprecate the old one → Remove old endpoint when incrementing major
    version.
  • Adding a mandatory header/field - Increment major version.
• Response
  • Adding a header/field - Create a new, separate endpoint with the changes and
    deprecate the old one → Remove old endpoint when incrementing major version.
  • Removing a header/field - Create a new, separate endpoint with the changes
    and deprecate the old one → Remove old endpoint when incrementing major
    version.
• Command line parameter
  • Renaming a command line parameter - Accept both names and deprecate the old
    one → Remove old name when incrementing major version.
  • Changing expected value taken by a command line parameter - Accept both
    names and deprecate the old one → Remove old name when incrementing major
    version.

In case the outlined solution for your case is not feasible (e.g. because of
security concerns), break the glass and increment the major version.

How to deprecate Copy Chapter

As outlined in the diagram above, sometimes we have to deprecate endpoints
partially or entirely. In this section we will go through different situations
where we have to deprecate something and ways of avoiding common pitfalls when
doing so.

Deprecating endpoints Copy Section

Some paths in the flowchart above lead to deprecation. Based on the initial
conditions, there are 2 major cases where we need to deprecate an endpoint:

• Changing an existing endpoint
  • Often happens because directly changing the endpoint would be a breaking
    change.
  • We usually create a clone of the old endpoint we want to deprecate and make
    the necessary changes to it.
  • We usually expose both endpoints in the next minor version while marking the
    old one as deprecated.
  • The old endpoint retains its previous name. When naming the new endpoint:
    • for HTTP endpoints we follow a “per-endpoint versioning” scheme; in cases
      where we can’t find a fitting name for the new endpoint, the simplest way
      forward is to take the old URI and append /v2 to it.
    • for command line endpoints, we can usually find a different name for the
      new endpoint.
• Deprecating an endpoint without adding a replacement to it
  • Often happens when we want to phase out a certain feature or functionality,
    but doing so immediately would be a breaking change.
  • We just mark the endpoint as deprecated.

Keeping Swagger updated Copy Section

Make sure that any changes you make in the code are also reflected in the
swagger specification.

Some tips:

• There is nothing in the swagger file that shows whether an endpoint is
  mandatory or optional, it’s all code logic.
• Mandatory fields in a request or response body are marked with
  required: true in the swagger definition. All other fields are optional.
• If you need to redirect an endpoint, you have to clone the old one under the
  new URI in the swagger specification.

Marking endpoints as deprecated Copy Section

When marking:

• an HTTP endpoint as deprecated:
  • Add a comment for the parsing function of the endpoint stating that it is
    deprecated.
  • Log a warn! message stating that the user accessed a deprecated endpoint.
  • Increment the deprecatedHttpApi metric.
  • Include the Deprecated header in the response.
• a header field in an HTTP endpoint as deprecated:
  • Add a comment in the parsing function where we check the presence of the
    header stating that it is deprecated.
  • If the header is present, log a warn! message stating that the user used a
    deprecated field.
  • Increment the deprecatedHttpApi metric.
  • Include the Deprecated header in the response.
• a command line parameter as deprecated:
  • Mention it is deprecated in the help message of the parameter in the
    argument parser.
  • Add it in the warn_deprecated_parameters function where we log it and
    increment the deprecatedCmdLineApi metric.

Removing deprecated endpoints on a major release Copy Section

When doing a major release, the API can have breaking changes. This is the only
time where we can safely remove deprecated elements of the API. To remove a
deprecated element of the API:

• Remove the associated functionality from the codebase (usually in vmm or
  mmds);
• Remove the parsing logic in api_server;
• Remove any unit and integration tests associated with this element.

Practical example of an API change Copy Chapter

In this guide we set out to remove the vsock_id field in PUTs on /vsock.
This was implemented in PR #2763 and we will go step by step through the
changes in order to understand the process of changing something in the
Firecracker API.

• We go through the flowchart; we want to remove a field in the body of a HTTP
  request. So we follow the flowchart like this:
  • → Change an existing endpoint
  • → Request
  • → Remove header or field
  • → Make it optional
  • → Deprecate
  • → Increment minor version.
• Now that we know we need to make the field optional and deprecate it, it’s
  time for the code changes (reference implementation in this commit). We
  go to the function in api_server/src/requests which is responsible for
  parsing this request, which is parse_put_vsock in this case, and do the
  following.
  • We find the associated vmm_config struct which serde_json uses for
    deserialization, in this case VsockDeviceConfig.
  • In the struct referenced above, we make the parameter optional by
    encapsulating it in an Option with #[serde(default)] and
    #[serde(skip_serializing_if = "Option::is_none")] so that we don’t break
    existing implementations, but we follow the new, desired usage of the
    endpoint.
  • After deserializing the body of the request into the struct, we check for
    the existence of the field we want to deprecate, in this case by calling
    vsock_cfg.vsock_id.is_some().
  • If the field is there, we must mark this request as being deprecated, so we
    craft a deprecation message ("PUT /vsock: vsock_id field is deprecated.")
    and increment the deprecated HTTP API metric
    (METRICS.deprecated_api.deprecated_http_api_calls.inc()).
  • We create a new ParsedRequest where, if we marked the request as
    deprecated, we append the deprecation message into its parsing_info
    structure, in this case by calling
    parsed_req.parsing_info().append_deprecation_message(msg).
  • Don’t forget to comment your code! Comments should reflect what is
    deprecated and clearly describe the code paths where you handle the
    deprecation case.
  • Add a unit test where you test your new code paths.
  • Fix all other failing unit tests.
  • Update the swagger file to reflect the change, in this case by removing the
    vsock_id field from the required parameter list in the Vsock definition
    and adding a description to it stating that it is deprecated since the
    current version.
  • Update any relevant documentation.
• We update the python integration tests to reflect the change (reference
  implementation in this commit).
  • We refactor the relevant tests/integration_tests/functional/test_api.py
    test to use the artifact model instead of the fixture one. If the test
    already uses the artifact model, you can skip this step.
  • We make sure to run the test with the current build, as well as with future
    Firecracker versions by specifying the unreleased version in the
    min_version parameter of artifacts.firecrackers(). We do this in order
    to ensure that, when we create patch releases on older branches, we test the
    API with future binaries to enforce backwards compatibility. Disclaimer:
    This test will fail when running with the binary artifact fetched from S3
    until you update the binary there with your current build. You should only
    do this once your PR has all necessary approves and this test is the last
    thing keeping it from getting merged.
  • We check that, when the deprecated field is present in the request, the
    Deprecation header is also present in the response by asserting
    response.headers['deprecation']. We do not also check that the header is
    not present when the field is not present because, in a future version, some
    other field may be deprecated in the same request and would return the
    header anyway, resulting in a fail in our test when it shouldn’t.
  • Fix all other failing integration tests.

Ballooning Copy Section

Using the balloon device with Firecracker

What is the balloon device Copy Chapter

A memory balloon device is a virtio device that can be used to reclaim and give
back guest memory through API commands issued by the host. It does this by
allocating memory in the guest, and then sending the addresses of that memory to
the host; the host may then remove that memory at will. The device is configured
through a number of options, and an integer, which represents the target size of
the balloon, in MiB. The options cannot be changed during operation, but the
target size can be changed.

The behaviour of the balloon is the following: while the actual size of the
balloon (i.e. the memory it has allocated) is smaller than the target size, it
continually tries to allocate new memory -- if it fails, it prints an error
message (Out of puff! Can't get %d pages), sleeps for 0.2 seconds, and then
tries again. While the actual size of the balloon is larger than the target
size, it will free memory until it hits the target size.

The device can be configured with the following options:

• deflate_on_oom: if this is set to true and a guest process wants to
  allocate some memory which would make the guest enter an out-of-memory state,
  the kernel will take some pages from the balloon and give them to said process
  instead asking the OOM killer process to kill some processes to free memory.
  Note that this applies to physical page allocations in the kernel which belong
  to guest processes. This does not apply to instances when the kernel needs
  memory for its activities (i.e. constructing caches), when the user requests
  more memory than the currently available to the balloon for releasing, or when
  guest processes try to allocate large amounts of memory that are refused by
  the guest memory manager, which is possible when the guest runs with
  vm.overcommit_memory=0 and the allocation does not pass the MM basic checks.
  Setting vm.memory_overcommit to 1 would make the MM approve all allocations,
  no matter how large, and using the memory mapped for those allocations will
  always deflate the balloon instead of making the guest enter an OOM state.
  Note: we do not recommend running with vm.overcommit_memory=1 because it
  requires complete control over what allocations are done in the guest and can
  easily result in unexpected OOM scenarios.
• stats_polling_interval_s: unsigned integer value which if set to 0 disables
  the virtio balloon statistics and otherwise represents the interval of time in
  seconds at which the balloon statistics are updated.

The device has two optional features which can be enabled with the following
options:

• free_page_reporting: A mechanism for the guest to continually report ranges
  of memory which the guest is not using and can be reclaimed.
  Read more here
• (Developer Preview)
  free_page_hinting: A mechanism to reclaim memory from the guest, this is
  instead triggered from the host.
  Read more here

Security disclaimer Copy Chapter

The balloon device is a paravirtualized virtio device that requires
cooperation from a driver in the guest.

In normal conditions, the balloon device will:

• not change the target size, which is set directly by the host
• consume exactly as many pages as required to achieve the target size
• correctly update the value of the actual size of the balloon seen by the host
• not use pages that were previously inflated if they were not returned to the
  guest via a deflate operation (unless the deflate_on_oom flag was set and
  the guest is in an out of memory state)
• provide correct statistics when available

However, Firecracker does not and cannot introspect into the guest to check the
integrity of the balloon driver. As the guest is not trusted, if the driver in
the guest becomes compromised, the above statements are no longer
guaranteed.

This means that even though users use the balloon to impose restrictions on
memory usage, they can be broken by a compromised driver in the guest. The
balloon device operates on a best effort model and users should always ensure
the host is prepared to handle a situation in which the Firecracker process uses
all of the memory it was given at boot even if the balloon was used to restrict
the amount of memory available to the guest. It is also the users'
responsibility to monitor the memory consumption of the VM and, in case
unexpected increases in memory usage are observed, we recommend the following
options:

• migrate the VM to a machine with higher memory availability through
  snapshotting at the cost of disrupting the workload;
• kill the Firecracker process that exceeds memory restrictions;
• enable swap with a sufficient amount of memory to handle the demand at the
  cost of memory access speed;

Users should also never rely solely on the statistics provided by the balloon
when controlling the Firecracker process as they are provided directly by the
guest driver and should always be viewed as an indication rather than a
guarantee of what the memory state looks like in the guest.

Please note that even in the case where the driver is not working properly, the
balloon will never leak memory from one Firecracker process to another, nor can
a guest within Firecracker access information in memory outside its own guest
memory. In other words, memory cannot leak in or out of Firecracker if the
driver becomes corrupted. This is guaranteed by the fact that the page frame
numbers coming from the driver are checked to be inside the guest memory, then
madvised with the MADV_DONTNEED flag, which breaks the mappings between host
physical memory (where the information is ultimately stored) and Firecracker
virtual memory, which is what Firecracker uses to build the guest memory. On
subsequent accesses on previously madvised memory addresses, the memory is
zeroed. Furthermore, the guest memory is mmapped with the MAP_PRIVATE and
MAP_ANONYMOUS flags, which ensure that even if a Firecracker yields some
information through an inflate and that same physical page containing the
information is mapped onto another Firecracker process, reads on that address
space will see zeroes.

Prerequisites Copy Chapter

To support memory ballooning, you must use a kernel that has the memory
ballooning driver installed (on Linux 4.14.193, the relevant settings are
CONFIG_MEMORY_BALLOON=y, CONFIG_VIRTIO_BALLOON=y). Other than that, only the
requirements mentioned in the getting-started document are needed.

Installing the balloon device Copy Chapter

In order to use a balloon device, you must install it during virtual machine
setup (i.e. before starting the virtual machine). This can be done either
through a PUT request on "/balloon" or by inserting the balloon into the JSON
configuration file given as a command line argument to the Firecracker process.

Here is an example command on how to install the balloon through the API:

socket_location=...
amount_mib=...
deflate_on_oom=...
polling_interval=...

curl --unix-socket $socket_location -i \
    -X PUT 'http://localhost/balloon' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d "{
        \"amount_mib\": $amount_mib, \
        \"deflate_on_oom\": $deflate_on_oom, \
        \"stats_polling_interval_s\": $polling_interval \
    }"

To use this, set socket_location to the location of the firecracker socket (by
default, at /run/firecracker.socket. Then, set amount_mib, deflate_on_oom
and stats_polling_interval_s as desired: amount_mib represents the target
size of the balloon, and deflate_on_oom and stats_polling_interval_s
represent the options mentioned before.

To install the balloon via the JSON config file, insert the following JSON
object into your configuration file:

"balloon": {
    "amount_mib": 0,
    "deflate_on_oom": false,
    "stats_polling_interval_s": 1
},

After installing the balloon device, users can poll the configuration of the
device at any time by sending a GET request on "/balloon". Here is an example of
such a request:

socket_location=...

curl --unix-socket $socket_location -i \
    -X GET 'http://localhost/balloon' \
    -H 'Accept: application/json'

On success, this request returns a JSON object of the same structure as the one
used to configure the device (via a PUT request on "/balloon").

Operating the traditional balloon device Copy Chapter

After it has been installed, the balloon device can only be operated via the API
through the following command:

socket_location=...
amount_mib=...
polling_interval=...

curl --unix-socket $socket_location -i \
    -X PATCH 'http://localhost/balloon' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d "{
        \"amount_mib\": $amount_mib, \
        \"stats_polling_interval_s\": $polling_interval \
    }"

This will update the target size of the balloon to amount_mib and the
statistics polling interval to polling_interval.

[!NOTE] Balloon inflation instructs the guest to reclaim memory which may
cause performance issues in the guest. The balloon statistics defined
below can be used to decide whether it's
necessary to reclaim memory.

Virtio balloon statistics Copy Chapter

The statistics are enabled by setting the stats_polling_interval_s field in
the balloon configuration to a non-zero value. If enabled, users can receive the
latest balloon statistics by issuing a GET request on "/balloon". Here is an
example of such a request:

socket_location=...

curl --unix-socket $socket_location -i \
    -X GET 'http://localhost/balloon/statistics' \
    -H 'Accept: application/json'

The request, if successful, will return a JSON object containing the latest
statistics. The JSON object contains information about the target and actual
sizes of the balloon as well as virtio traditional memory balloon statistics.

The target and actual sizes of the balloon are expressed as follows:

• target_pages: The target size of the balloon, in 4K pages.
• actual_pages: The number of 4K pages the device is currently holding.
• target_mib: The target size of the balloon, in MiB.
• actual_mib: The number of MiB the device is currently holding.

These values are taken directly from the config space of the device and are
always up to date, in the sense that they are exactly what the Firecracker
process reads when polling the config space. The actual fields being accurate
are subject to the guest driver working correctly.

As defined in the virtio 1.1 specification, the traditional virtio balloon
device has support for the following statistics:

• VIRTIO_BALLOON_S_SWAP_IN: The amount of memory that has been swapped in (in
  bytes).
• VIRTIO_BALLOON_S_SWAP_OUT: The amount of memory that has been swapped out to
  disk (in bytes).
• VIRTIO_BALLOON_S_MAJFLT: The number of major page faults that have occurred.
• VIRTIO_BALLOON_S_MINFLT: The number of minor page faults that have occurred.
• VIRTIO_BALLOON_S_MEMFREE: The amount of memory not being used for any
  purpose (in bytes).
• VIRTIO_BALLOON_S_MEMTOT: The total amount of memory available (in bytes).
• VIRTIO_BALLOON_S_AVAIL: An estimate of how much memory is available (in
  bytes) for starting new applications, without pushing the system to swap.
• VIRTIO_BALLOON_S_CACHES: The amount of memory, in bytes, that can be quickly
  reclaimed without additional I/O. Typically these pages are used for caching
  files from disk.
• VIRTIO_BALLOON_S_HTLB_PGALLOC: The number of successful hugetlb page
  allocations in the guest.
• VIRTIO_BALLOON_S_HTLB_PGFAIL: The number of failed hugetlb page allocations
  in the guest.

Since linux v6.12, following metrics added(omitted in < v6.12):

• VIRTIO_BALLOON_S_OOM_KILL: OOM killer invocations, indicating critical
  memory pressure.
• VIRTIO_BALLOON_S_ALLOC_STALL: Counter of Allocation enter a slow path to
  gain more memory page. The reclaim/scan metrics can reveal what is actually
  happening.
• VIRTIO_BALLOON_S_ASYNC_SCAN: Amount of memory scanned asynchronously.
• VIRTIO_BALLOON_S_DIRECT_SCAN: Amount of memory scanned directly.
• VIRTIO_BALLOON_S_ASYNC_RECLAIM: Amount of memory reclaimed asynchronously.
• VIRTIO_BALLOON_S_DIRECT_RECLAIM: Amount of memory reclaimed directly.

When the pages_high watermark is reached, Linux kswapd performs asynchronous
page reclaim, which increases ASYNC_SCAN and ASYNC_RECLAIM.

When a process allocates more memory than the kernel can provide, the process is
stalled while pages are reclaimed directly, which increases DIRECT_SCAN and
DIRECT_RECLAIM.

man sar: %vmeff Calculated as pgsteal(RECLAIM) / pgscan(SCAN), this is a
metric of the efficiency of page reclaim. If it is near 100% then almost every
page coming off the tail of the inactive list is being reaped. If it gets too
low (e.g. less than 30%) then the virtual memory is having some difficulty.

The driver is queried for updated statistics every time the amount of time
specified in that field passes. The driver may not provide all the statistics
when queried, in which case the old values of the missing statistics are
preserved.

To change the statistics polling interval, users can sent a PATCH request on
"/balloon/statistics". Here is an example of such a request:

socket_location=...
polling_interval=...

curl --unix-socket $socket_location -i \
    -X PATCH 'http://localhost/balloon' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d "{ \"stats_polling_interval_s\": $polling_interval }"

This will change the statistics polling interval to polling_interval. Note
that if the balloon was configured without statistics pre-boot, the statistics
cannot be enabled later by providing a polling_interval non-zero value.
Furthermore, if the balloon was configured with statistics pre-boot through a
non-zero stats_polling_interval_s value, the statistics cannot be disabled
through a polling_interval value of zero post-boot.

Virtio balloon free page reporting Copy Chapter

Free page reporting is a virtio balloon feature which allows the guest OS to
report ranges of memory which are not being used. In Firecracker, the balloon
device will madvise the range with the MADV_DONTNEED flag, reducing the RSS
of the guest. Reporting can only be enabled pre-boot and will run continually
with no option to stop it running. The feature also requires the guest to have
the Linux kernel config option PAGE_REPORTING enabled.

To enable free page reporting when creating the balloon device, the
free_page_reporting attribute should be set in the JSON object.

An example of how to configure the device to enable free page reporting:

socket_location=...
amount_mib=...
deflate_on_oom=...
polling_interval=...

curl --unix-socket $socket_location -i \
    -X PUT 'http://localhost/balloon' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d "{
        \"amount_mib\": $amount_mib, \
        \"deflate_on_oom\": $deflate_on_oom, \
        \"stats_polling_interval_s\": $polling_interval, \
        \"free_page_reporting\": true \
    }"

The Linux driver uses a hook in the free page path to trigger the reporting
process, which will begin after a short delay (~2 seconds) and report the
ranges. The runtime impact of this feature is heavily workload dependent. The
driver gets ranges from the buddy allocator with a minimum page order. This page
order dictates the minimum size of ranges reported and can be configured with
the page_reporting_order module parameter in the guest kernel. The page order
comes with trade-offs between performance and memory reclaimed; a good target to
maximise memory reclaim is to have the reported ranges match the backing page
size.

Virtio balloon free page hinting Copy Chapter

Free page hinting is a
developer-preview
feature, which allows the guest driver to report ranges of memory which are not
being used. In Firecracker, the balloon device will madvise the range with the
MADV_DONTNEED flag, reducing the RSS of the guest. Free page hinting differs
from reporting as this is instead initiated from the host side, giving more
flexibility on when to reclaim memory.

To enable free page hinting when creating the balloon device, the
free_page_hinting attribute should be set in the JSON object.

An example of how to configure the device to enable free page hinting:

socket_location=...
amount_mib=...
deflate_on_oom=...
polling_interval=...

curl --unix-socket $socket_location -i \
    -X PUT 'http://localhost/balloon' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d "{
        \"amount_mib\": $amount_mib, \
        \"deflate_on_oom\": $deflate_on_oom, \
        \"stats_polling_interval_s\": $polling_interval, \
        \"free_page_hinting\": true \
    }"

Free page hinting is initiated and managed by Firecracker, the core mechanism to
control the run is with the cmd_id. When Firecracker sets the cmd_id to a
new number, the driver will acknowledge this and start reporting ranges, which
Firecracker will free. Once the device has reported all the ranges it can find,
it will update the cmd_id to reflect this. The device will then hold these
ranges until Firecracker sends the stop command which allows the guest driver to
reclaim the memory. The time required for the guest to complete a hinting run is
dependant on a multitude of different factors and is mostly dictated by the
guest, however, in testing the average time is ~200 milliseconds for a 1GB VM.

This control mechanism in Firecracker is managed through three separate
endpoints /balloon/hinting/start, /balloon/hinting/status and
/balloon/hinting/stop. For simple operation, call the start endpoint with
acknowledge_on_stop = true, which will automatically send the stop command
once the driver has finished.

An example of sending this command:

curl --unix-socket $socket_location -i \
    -X POST 'http://localhost/balloon/hinting/start' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d "{
        \"acknowledge_on_stop\": true \
    }"

For fine-grained control, using acknowledge_on_stop = false, Firecracker will
not send the acknowledge message. This can be used to get the guest to hold onto
more memory. Using the /status endpoint, you can get information about the
last cmd_id sent by Firecracker and the last update from the guest.

An example of the status request and response:

curl --unix-socket $socket_location -i \
    -X GET 'http://localhost/balloon/hinting/status' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json'

Response:

{
  "host_cmd": 1,
  "guest_cmd": 2
}

An example of the stop endpoint:

curl --unix-socket $socket_location -i \
    -X POST 'http://localhost/balloon/hinting/stop' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d "{}"

On snapshot restore, the cmd_id is always set to the stop cmd_id to
allow the guest to reclaim the memory. If you have a particular use-case which
requires this not to be the case, please raise an issue with a description of
your scenario.

[!WARNING]

Free page hinting was primarily designed for live migration, because of this
there is a caveat to the device spec which means the guest is able to reclaim
memory before Firecracker even receives the range to free. This can lead to a
scenario where the device frees memory that has been reclaimed in the guest,
potentially corrupting memory. The chances of this race happening are low, but
not impossible; hence the developer-preview status.

We are currently working with the kernel community on a feature that will
eliminate this race. Once this has been resolved, we will update the device.

One way to safely use this feature when using UFFD is:

1. Enable WRITEPROTECT on the VM memory before starting a hinting run.
2. Track ranges that are written to.
3. Skip these ranges when Firecracker reports them for freeing.

This will prevent ranges which have been reclaimed from being freed.

Balloon Caveats Copy Chapter

• Firecracker has no control over the speed of inflation or deflation; this is
  dictated by the guest kernel driver.

• The traditional balloon will continually attempt to reach its target size,
  which can be a CPU-intensive process. It is therefore recommended to set
  realistic targets or, after a period of stagnation in the inflation, update
  the target size to be close to the inflated size.

• The deflate_on_oom flag is a mechanism to prevent the guest from crashing or
  terminating processes; it is not meant to be used continually to free memory.
  Doing this will be a CPU-intensive process, as the traditional balloon driver
  is designed to deflate and release memory slowly. This is also compounded if
  the balloon has yet to reach its target size, as it will attempt to inflate
  while also deflating.

Design Copy Section

Firecracker Design

Scope Copy Chapter

What is Firecracker Copy Section

Firecracker is a new virtualization technology that enables customers to deploy
lightweight micro Virtual Machines or microVMs. Firecracker microVMs combine
the security and workload isolation properties of traditional VMs with the
speed, agility and resource efficiency enabled by containers. They provide a
secure, trusted environment for multi-tenant services, while maintaining minimal
overhead.

The scope of this document is to describe the features and architecture of the
Firecracker virtual machine manager (VMM).

Features Copy Section

1. Firecracker can safely run workloads from different customers on the same
   machine.
2. Customers can create microVMs with any combination of vCPU (up to 32) and
   memory to match their application requirements.
3. Firecracker microVMs can oversubscribe host CPU and memory. The degree of
   oversubscription is controlled by customers, who may factor in workload
   correlation and load in order to ensure smooth host system operation.
4. With a microVM configured with a minimal Linux kernel, single-core CPU, and
   128 MiB of RAM, Firecracker supports a steady mutation rate of 5 microVMs per
   host core per second (e.g., one can create 180 microVMs per second on a host
   with 36 physical cores).
5. The number of Firecracker microVMs running simultaneously on a host is
   limited only by the availability of hardware resources.
6. Each microVM exposes a host-facing API via an in-process HTTP server.
7. Each microVM provides guest-facing access to host-configured metadata via the
   /mmds API.

Specifications Copy Section

Firecracker's technical specifications are available in the
Specifications document.

Host Integration Copy Chapter

The following diagram depicts an example host running Firecracker microVMs.

Firecracker runs on Linux hosts and with Linux guest OSs (from this point on,
referred to as guests). For a complete list of currently supported kernel
versions, check out the kernel support policy.

In production environments, Firecracker should be started only via the jailer
binary. See Sandboxing for more details.

After launching the process, users interact with the Firecracker API to
configure the microVM, before issuing the InstanceStart command.

Host Networking Integration Copy Section

Firecracker emulated network devices are backed by TAP devices on the host. To
make use of Firecracker, we expect our customers to leverage on-host networking
solutions.

Storage Copy Section

Firecracker emulated block devices are backed by files on the host. To be able
to mount block devices in the guest, the backing files need to be pre-formatted
with a filesystem that the guest kernel supports.

Internal Architecture Copy Chapter

Each Firecracker process encapsulates one and only one microVM. The process runs
the following threads: API, VMM and vCPU(s). The API thread is responsible for
Firecracker's API server and associated control plane. It's never in the fast
path of the virtual machine. The VMM thread exposes the machine model, minimal
legacy device model, microVM metadata service (MMDS) and VirtIO device emulated
Net, Block and Vsock devices, complete with I/O rate limiting. In addition to
them, there are one or more vCPU threads (one per guest CPU core). They are
created via KVM and run the KVM_RUN main loop. They execute synchronous I/O
and memory-mapped I/O operations on devices models.

Threat Containment Copy Section

From a security perspective, all vCPU threads are considered to be running
malicious code as soon as they have been started; these malicious threads need
to be contained. Containment is achieved by nesting several trust zones which
increment from least trusted or least safe (guest vCPU threads) to most trusted
or safest (host). These trusted zones are separated by barriers that enforce
aspects of Firecracker security. For example, all outbound network traffic data
is copied by the Firecracker I/O thread from the emulated network interface to
the backing host TAP device, and I/O rate limiting is applied at this point.
These barriers are marked in the diagram below.

Firecracker does not perform any network traffic filtering. All egress traffic
from a guest is therefore considered untrusted, and should be filtered at the
host-level.

Components and Features Copy Chapter

Machine Model Copy Section

Layout

Firecracker provides guests with storage and network access via emulated VirtIO
Net and VirtIO Block devices. It also exposes a serial console and partial
keyboard controller, the latter being used by guests to reset the VM (either
soft or hard reset). Within Firecracker, the purpose of the I8042 device is to
signal the microVM that the guest has requested a reboot.

In addition to the Firecracker provided device models, guests also see the
Programmable Interrupt Controllers (PICs), the I/O Advanced Programmable
Interrupt Controller (IOAPIC), and the Programmable Interval Timer (PIT) that
KVM supports.

Exposing the CPU to the guest

Firecracker allows control of what processor information is exposed to the guest
by using CPU templates. CPU templates can be
set via the Firecracker API. Users can choose from existing static CPU templates
and/or creating their own custom CPU templates.

Clocksources available to guests

Firecracker exposes the following clock sources to guests:

• x86_64: kvm-clock and tsc. Linux guests >=5.10 will pick tsc by default if
  stable.
• aarch64: arch_sys_counter

I/O: Storage, Networking and Rate Limiting Copy Section

Firecracker provides VirtIO/block and VirtIO/net emulated devices, along with
the application of rate limiters to each volume and network interface to make
sure host hardware resources are used fairly by multiple microVMs. These are
implemented using a token bucket algorithm based on two buckets. One is
associated with the number of operations per second and the other one with the
bandwidth. The customer can create and configure rate limiters via the API by
specifying token bucket configurations for ingress and egress. Each token bucket
is defined via the bucket size, I/O cost, refill rate, maximum burst, and
initial value. This enables the customer to define flexible rate limiters that
support bursts or specific bandwidth/operations limitations. For vhost-user
devices, customers should implement rate limiting on the side of the vhost-user
backend that they provide.

MicroVM Metadata Service Copy Section

Firecracker microVMs expose access to a minimal MicroVM-Metadata Service (MMDS)
to the guest through the API endpoint. The metadata stored by the service is
fully configured by users.

Sandboxing Copy Section

__Firecracker process__

Firecracker is designed to assure secure isolation using multiple layers. The
first layer of isolation is provided by the Linux KVM and the Firecracker
virtualization boundary. To assure defense in depth, Firecracker should only run
constrained at the process level. This is achieved by the following: seccomp
filters for disallowing unwanted system calls, cgroups and namespaces for
resource isolation, and dropping privileges by jailing the process. Seccomp
filters are automatically installed by Firecracker, while for the latter, we
recommend starting Firecracker with the jailer binary that's part of each
Firecracker release.

Seccomp

Seccomp filters are used by default to limit the host system calls Firecracker
can use. The default filters only allow the bare minimum set of system calls and
parameters that Firecracker needs in order to function correctly.

The filters are loaded in the Firecracker process, on a per-thread basis, before
executing any guest code.

For more information, check out the seccomp documentation.

__Jailer process__

The Firecracker process can be started by another jailer process. The jailer
sets up system resources that require elevated permissions (e.g., cgroup,
chroot), drops privileges, and then exec()s into the Firecracker binary, which
then runs as an unprivileged process. Past this point, Firecracker can only
access resources that a privileged third-party grants access to (e.g., by
copying a file into the chroot, or passing a file descriptor).

Cgroups and Quotas

Each Firecracker microVM can be further encapsulated into a cgroup. By setting
the affinity of the Firecracker microVM to a node via the cpuset subsystem, one
can prevent the migration of said microVM from one node to another, something
that would impair performance and cause unnecessary contention on shared
resources. In addition to setting the affinity, each Firecracker microVM can
have its own dedicated quota of the CPU time via the cpu subsystem, thus
guaranteeing that resources are fairly shared across Firecracker microVMs.

Monitoring Copy Section

Firecracker emits logs and metric counters, each on a named pipe that is passed
via the API. Logs are flushed line by line, whereas metrics are emitted when the
instance starts, then every 60 seconds while it's running, and on panic.
Firecracker customers are responsible for collecting data in the Firecracker log
files. In production builds, Firecracker does not expose the serial console
port, since it may contain guest data that the host should not see.