openvmm Documentation

Technical Documentation: microsoft/openvmm

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

OpenVMM

OpenVMM is a modular, cross-platform Virtual Machine Monitor (VMM), written in
Rust.

Although it can function as a traditional VMM, OpenVMM's development is
currently focused on its role in the OpenHCL paravisor.

This repo is the home for both projects.

For more information, read our guide
or our introductory blog post.

Getting Started

For info on how to run, build, and use OpenVMM, check out the The OpenVMM Guide.

The guide is published out of this repo via Markdown files.
Please keep them up-to-date.

The maintainers of this project use Discord to collaborate with external contributors and users.
Please join if you have any questions about contributing to or using OpenVMM/OpenHCL.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit Contributor License Agreements.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct.
For more information see the Code of Conduct FAQ or
contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must follow
Microsoft's Trademark & Brand Guidelines.
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.

Overview (`guest_test_uefi/README.md`)

`guest_test_uefi`

See the guide for more info on how to build/run the code in this crate.

Overview (`opentmk/README.md`)

OpenTMK

See the guide for more info on how to build/run the code in this crate.

Overview (`support/README.md`)

This directory is for library and infrastructure crates that are VM agnostic.

Or, in other words:

They contain no reference to VMs or VM concepts at all
They contain no dependencies to crates in the repo other than crates in this
directory
They are general purpose enough that they could be open source and used by
other projects

One way to think of this folder is OpenVMM's private crates.io mirror for crates
that are not yet ready to be open sourced, but could be.

Please name crates based on what they provide, and not general catch all names
like "util, base, common" based on what they contain. See the section in the
OpenVMM guide on naming crates.

Overview (`vm/README.md`)

This directory is for library and infrastructure crates that provide
VM-related functionality, without being tied to any particular "VMM frontend"
(e.g: OpenVMM, OpenHCL, etc...)

Crates in this directory must not take any direct dependencies on any code
outside of vm/, except for crates pulled from crates.io, or crates under
../support/.

Overview (`Guide/mdbook-openvmm-shim/README.md`)

mdbook-openvmm-shim

This is a small standalone Rust binary which implements a mdbook preprocessor
"shim", used to give OpenVMM additional control over how mdbook finds and
interacts with external preprocessors.

Notably: it allows us to work around the following outstanding mdbook issue:
Preprocessor cannot add folders/files directly to book directory
#1222. Instead, this shim
will dynamically invoke each preprocessor's install subcommand on-demand,
avoiding the need to check-in the somewhat large third-party css/js/html blobs
that these plugins depend on.

In addition, this shim has the benefit of keeping the mdbook end-user
experience as vanilla as possible, without requiring a project-specific wrapper
script (e.g: cargo xtask mdbook).

Why is this its own workspace?

Since book.toml uses cargo run to invoke this script, having this crate live
in the main OpenVMM Rust workspace would result in delays of ~0.5s on each
command invocation (due to cargo run needing to scan the whole workspace on
each invocation). Having it in its own workspace results in a cargo run
overhead of roughly 0.01 seconds (on average), which is far more tolerable.

In the future, when cargo-script finally lands (see
https://github.com/rust-lang/cargo/issues/12207), this shim should simply be a
standalone file in the Guide/ directory.

Overview (`petri/guest-bootstrap/README.md`)

This directory contains files needed to bootstrap the guest with the pipette
agent.

meta-data and user-data: cloud-init files for Linux guests
imc.hiv: an IMC hive for Windows guests

To update imc.hiv, on a Windows machine run cargo run -p make_imc_hive PATH/TO/imc.hiv

Overview (`petri/logview/README.md`)

Petri Logview Web Application

A React-based web application for viewing and analyzing Petri logs, built with TypeScript, Vite, and modern React ecosystem tools.

Prerequisites

Node.js (version >=22.22.0, as required by react-router 8)
npm (comes with Node.js)
Browser: Safari 16.4+, Chrome 111+, or Firefox 128+ (required by Tailwind CSS 4)

Tailwind CSS 4 runs through the @tailwindcss/vite plugin in vite.config.ts.
tailwind.css imports the theme and utilities without Preflight and retains
the existing gray palette, preserving browser defaults and the site's custom
styles. Source scanning is limited to TypeScript files under src/.

Initial Setup

Follow these steps to set up the project from scratch:

1. Install Dependencies

Navigate to the project directory and install all required packages:

powershell

# Navigate to the logview directory
cd .\petri\logview

# Install all dependencies exactly as locked
npm ci

npm ci installs the exact versions recorded in package-lock.json, which is
what CI runs. Use npm install only when you intend to add or update a
dependency, and commit the resulting package-lock.json changes.

2. Verify Installation

After installation, you should have the following key dependencies:

Runtime Dependencies:

react & react-dom - React framework
react-router - Client-side routing
@tanstack/react-query - Data fetching and caching
@tanstack/react-table - Table component library
@tanstack/react-virtual - Virtual scrolling

Development Dependencies:

typescript - TypeScript compiler
vite - Build tool and dev server
@vitejs/plugin-react - Vite React plugin
eslint - Code linting
@types/react & @types/react-dom - TypeScript definitions

3. Development Commands

powershell

# Start development server (runs on http://localhost:3000)
npm run dev

# Build for production
npm run build

# Preview production build
npm run preview

# Run linting
npx eslint .

Project Structure

text

logview/
├── src/
│   ├── main.tsx          # Application entry point
│   ├── style/            # All CSS and global styles (CSS, PostCSS, or other style assets)
│   │   └── ...           # e.g. globals.css, themes, component styles
│   └── table_defs/       # Column definitions for tables used across the app
│       └── ...           # e.g. logsColumns.ts, eventsColumns.ts, devicesColumns.ts
├── index.html            # HTML template
├── package.json          # Dependencies and scripts
├── tsconfig.json         # TypeScript configuration
├── tsconfig.node.json    # TypeScript config for build tools
├── vite.config.ts        # Vite configuration
├── eslint.config.ts      # ESLint configuration
└── README.md             # This file

Troubleshooting

Common Issues

Module not found errors: Ensure all dependencies are installed with npm ci
TypeScript errors: Make sure both tsconfig.json and tsconfig.node.json are present
Port already in use: The dev server uses port 3000 by default. You can change this in vite.config.ts

Important: Build Before Pushing / Merging (Temporary Feature)

Please run a production build before pushing or opening a merge request to
ensure the front-end assets are up to date.

powershell

npm run build

Where the output goes:
- The build artifacts will be written to the assets/ folder at the project
  root (e.g. logview/assets/).
- You can run npm run preview to verify the production build locally.

Note: This is a temporary requirement and will be automated in the future.

Recommended: Prettier Setup for Consistent Formatting

To ensure consistent code formatting, it is recommended to install the
Prettier extension in your code editor (such as VS
Code).

For VS Code users, add the following to your settings to automatically format files on save:

json

"editor.defaultFormatter": "esbenp.prettier-vscode"

Housekeeping Stuff

1. Updating branch quick-filters

The branches we want to track test and runs for are bound to frequently change
as new releases come through. For convenience these filters are defined as a
list in the branch_quick_filters.tsx file. run_filters will update the
filters that show up on the Runs page. test_filters will update the filters
that show up for the Test and TestDetails pages. all is a reserved branch
filter name and will display all the branches.

Overview (`vmm_tests/vmm_test_igvm_agent/README.md`)

Use test IGVM Agent for local single-test runs

Windows-only helper to start test_igvm_agent_rpc_server.exe when it is not already running
(flowey still owns CI). For local nextest runs we assume one test process at a time; the helper
starts the server at test begin, streams stderr into the test output, and tears it down via a
guard when the test ends. Concurrency is not supported or guaranteed.

Local runs must explicitly opt-in to the helper by exporting
VMM_TEST_IGVM_AGENT_LOCAL_AUTOSTART=1 (or true/yes). CI never sets this variable so tests
fail loudly if Flowey forgets to launch the server. The crate exposes this variable as the
LOCAL_AUTOSTART_ENV constant for reuse in callers.

Overview (`vm/devices/support/README.md`)

These support crates are tailored to specific devices, and are a bit too
specialized to be a part of the top level support/ folder.

Overview (`vm/loader/manifests/README.md`)

This folder contains manifest recipes for building IGVM files. Create the
resource file and run igvmfilegen manifest directly.

SNP Linux-direct profile

Flowey builds snp-linux-direct-pipette.json for the linux_direct_x64[snp]
VMM test configuration. It injects the statically linked Linux pipette into the shared test
initrd from openvmm-deps, then runs it as PID 1 with rdinit=/pipette. The
test also uses the SNP guest kernel from openvmm-deps and the in-tree
snp_bootshim. The manual snp-linux-direct.json bring-up profile instead
starts the initrd shell. These profiles have these assumptions:

x64 and one VTL0 SEV-SNP guest that boots Linux directly
at least 160 MiB of RAM (40,960 4-KiB pages), which the image measures
DT-described CPUs, RAM, NUMA nodes, UARTs, and optional PCIe bridges, with
the fixed enlightened x86 chipset
no shared GPA boundary, normal interrupt injection, and secure AVIC disabled
base SNP policy 0x30000; enable_debug adds the debug bit to produce the
current debug-capable policy 0xb0000
an initrd and the kernel command line
console=ttyS0 earlyprintk=serial earlycon panic=-1
SNP C-bit position 51; the value must be bit 32 or higher because the
startup page tables identity-map the lower 4 GiB

The normal-injection output is a shared artifact: the same binary is intended
to boot on KVM and MSHV. Its SnpVpContext uses the SNP initial-VMSA GPA
0xffff_ffff_f000. KVM synthesizes its measured VMSA at that GPA, while MSHV
maps and imports the file-provided VMSA there. Both backends use the policy
encoded in the file, but only MSHV submits its SNP ID block.

The snp-linux-direct-restricted.json profile encodes restricted interrupt
injection in its IGVM VMSA. It is intended only for MSHV bring-up.

Every SNP Linux-direct image requests the host's device tree through an
unmeasured IGVM parameter area. After launch, the measured bootshim reads the
CPUs, RAM, PCIe host bridges, and COM ports from the tree and generates DSDT,
FADT, MADT, SRAT, XSDT and RSDP. It adds MCFG and a PCIe SSDT only when
bridges are present. Images contain zero-filled output storage, not prebuilt
ACPI tables. One image therefore boots with any supported CPU count, RAM
size, NUMA layout, and PCIe configuration.

The runtime describes COM1/COM2 when any serial backend is configured, and
COM3/COM4 only when their individual backend is configured. The shim emits
only the UARTs in that description. OpenHCL still selects a described COM3
as its console. Both consumers use the existing IGVM PIO binding, in which
the reg property of pio-bus/serial@ starts with a big-endian 64-bit
base. The shim finds each COM port by that base and uses the fixed COM I/O
port and IRQ.

The measured handoff supplies the end of the measured RAM and bounded
workspace locations. Older experimental images and shims are incompatible.
Rebuild both; there is no static-table fallback.

The image contains a small measured bootshim. Only pages containing the kernel,
initrd, boot metadata, SNP special pages, bootshim, or bootshim parameters are
included as IGVM PageData, along with zero-filled permanent ACPI and RSDP
storage. The DT uses private, already-imported parameter pages. The shim
reads the DT without allocating before it accepts any omitted pages. Then it
accepts the remaining private RAM with PVALIDATE and enters Linux. This avoids
loading and measuring every configured RAM page, but still accepts all RAM
before Linux starts.

The shim trusts the RAM that the host device tree describes. Below the
measured RAM end, it accepts only the gaps that the measured parameter page
lists, so it never accepts a measured page again. It accepts all host RAM
above the measured RAM end and adds that RAM to the Linux E820 map.

The IGVM contains only the BSP VMSA, regardless of processor count. Backends
are responsible for any AP launch state they require. Current KVM constructs
and measures the initial VMSAs itself rather than accepting the IGVM VMSA page.
Those KVM-created VMSAs are not part of the IGVM launch measurement, so the
file's SNP ID block is not valid for KVM. KVM attestation against that ID block
remains unsupported until KVM accepts userspace-provided VMSAs.

To build it manually, create a resources file containing absolute paths:

json

{
    "resources": {
        "linux_kernel": "/absolute/path/to/vmlinux-or-bzImage",
        "linux_initrd": "/absolute/path/to/initrd",
        "snp_bootshim": "/absolute/path/to/snp_bootshim"
    }
}

Build the bootshim first:

bash

MINIMAL_RT_BUILD=1 cargo build \
  --profile boot-dev \
  --target x86_64-unknown-none \
  -p snp_bootshim

Build the host-native generator:

bash

cargo build -p igvmfilegen

Then generate the image:

bash

repo=/absolute/path/to/openvmm
cargo run -p igvmfilegen -- manifest \
  --manifest "$repo/vm/loader/manifests/snp-linux-direct.json" \
  --resources /absolute/path/to/snp-linux-direct-resources.json \
  --output /absolute/path/to/snp-linux-direct.bin

The standard outputs are:

snp-linux-direct.bin
snp-linux-direct.bin.map
snp-linux-direct-snp.json

Launching on MSHV

The guest needs at least memory_page_count * 4096 bytes of RAM. Choose the
CPU count, larger RAM sizes, and NUMA layout at launch:

bash

openvmm --hypervisor mshv --isolation snp \
  --igvm firmware=path/to/snp-linux-direct.bin,personality=linux-direct \
  --hv --no-vmbus \
  --memory 2GB --processors 4 \
  --com1 console

Keep COM1 for the profile's console=ttyS0 kernel command line. If any serial
backend is configured, the host describes COM1 and COM2. It describes COM3
and COM4 only when their individual backends are configured. The shim emits
only described UARTs and supports shared IRQ4 for COM1/COM3 and IRQ3 for
COM2/COM4. It does not use console metadata to infer UART presence.

Host-described PCIe on MSHV

Use the same image. The runtime supplies the resolved PCIe layout; no
ECAM-base or MMIO-base overrides are needed. For example:

bash

openvmm --hypervisor mshv --isolation snp --hv --no-vmbus \
  --igvm firmware=path/to/snp-linux-direct.bin,personality=linux-direct \
  --memory 160MB --processors 2 --vps-per-socket 2 \
  --com1 file=path/to/serial.log \
  --pcie-root-complex rc0,segment=0,start_bus=0,end_bus=31,low_mmio=64M,high_mmio=1G,node=0 \
  --pcie-ecam-below-4gb \
  --pcie-root-port rc0:disk0 --pcie-root-port rc0:net0 \
  --virtio-blk file:path/to/disk.raw,pcie_port=disk0 \
  --virtio-net pcie_port=net0:consomme

The below-4-GiB option selects a placement class, not a fixed address.
Some direct-boot Linux kernels reject MCFG entries above 4 GiB unless SMBIOS
reports a sufficiently recent BIOS date. Keep this option for those kernels;
the converter supports 64-bit addresses but cannot bypass that guest policy.

This implementation supports at most 255 CPUs, 64 RAM ranges that end at or
below 512 GiB, the four fixed COM ports, and eight generic ECAM bridges in
distinct segments. The tables put every bridge on node 0, with native x86
MSI/MSI-X and identity low and high MMIO windows; either window may be
omitted. The bootshim stops the guest when the device tree exceeds these
limits or puts two bridges in one segment. The x86 device tree does not
describe an IOMMU or CXL, so the guest does not see them.

The image reserves 64 KiB for the device tree, 64 KiB for generated ACPI,
a temporary 1-MiB heap, and a 32-KiB bootshim stack. The heap is omitted from
page data and accepted with the RAM gaps before use. ACPI output, RSDP, DT,
the shim, measured handoff pages, and SNP special pages are imported, not
accepted again. No final table points into the DT, heap, or shim.
The bootshim stops the guest if it cannot parse a PCIe bridge, or if a PCIe
ECAM or MMIO window overlaps RAM or reaches the C-bit. The ACPI arena is
ACPI-reclaimable in E820; the pinned RSDP remains reserved. The shim copies
all tables to permanent storage before it publishes the RSDP and Linux
zero-page pointer. It never enters Linux with partial ACPI.

The shim ignores all other device-tree nodes and properties, including the
/chosen and VMBus nodes. OpenVMM adds the /openhcl node only for
personality=openhcl.

Host-selected topology is unmeasured. Attestation policy for those values is
deferred for bring-up; the shim does not bind them into the launch identity.
When the shim stops the guest, it sends a GHCB termination request with
reason-code set 0xF and a reason code from Failure in
vm/loader/snp_bootshim/src/main.rs.

Overview (`vm/devices/tpm/test_data/README.md`)

vTPMState Blob

The blob is used for testing TPM with pre-provisioned states. Refer to the following tests for usage examples:

tpm_device::tests::test_fix_corrupted_vmgs
tpm_lib::tests::test_with_pre_provisioned_state
tpm_lib::tests::test_initialize_guest_secret_key

Steps to Create the Blob

Initialize TPM (an MsTpm20RefPlatform instance) with callbacks that define the data store where TPM will write its NV states.
Configure the TPM with the desired commands to establish the required state.
Once the TPM state is prepared, export the data store to generate the blob.

Guide Src Index (`Guide/src/index.md`)

Introduction

OpenVMM is a modular, cross-platform Virtual Machine Monitor (VMM), written in
Rust.

OpenVMM can run as a traditional VMM and also serves as the VMM within the
OpenHCL paravisor.

The project is open-source, MIT Licensed, and developed publicly at
microsoft/openvmm on GitHub.

Cross-Platform

OpenVMM supports a variety of host operating systems, architectures, and
virtualization backends:

Host OS	Architecture	Virtualization API
Linux (paravisor)	x64 / Aarch64	MSHV (using VSM / TDX / SEV-SNP)
Windows	x64 / Aarch64	WHP (Windows Hypervisor Platform)
Linux	x64 / Aarch64	KVM
	x64 / Aarch64	MSHV (Microsoft Hypervisor)
macOS	Aarch64	Hypervisor.framework

Running in the OpenHCL paravisor

OpenVMM is the VMM that runs in the OpenHCL paravisor.

Unlike in traditional virtualization, where a VMM runs in a privileged host/root
partition and provides virtualization services to a unprivileged guest
partition, the "paravisor" model enables a VMM to provide virtualization
services from within the guest partition itself.

It can be considered a form of "virtual firmware", running at a higher privilege
level than the primary guest OS.

Paravisors are quite exciting, as they enable a wide variety of useful and novel
virtualization scenarios! For example: at Microsoft, OpenHCL plays a key role in
enabling several important Azure scenarios:

Enabling existing workloads to seamlessly leverage Azure Boost (Azure's
next-generation hardware accelerator), without requiring any modifications to
the guest VM image.
Enabling existing guest operating systems to run inside Confidential VMs.
Powering Trusted Launch VMs - VMs that support Secure Boot, and include a
vTPM.

Standalone VMM

OpenVMM can also run as a general-purpose VMM on a Windows, Linux, or macOS
host. This configuration shares most of its code with OpenVMM running in a
paravisor and provides a convenient environment for running and testing virtual
machines directly on a host.

Relationship to other Rust-based VMMs

OpenVMM's core security principles are aligned with those of the Rust-based
Cloud Hypervisor, Firecracker, and crosvm projects, which is why we also chose
to write OpenVMM in Rust. However, OpenVMM's unique goal of running efficiently
in a paravisor environment made it difficult to leverage existing projects.
OpenVMM requires fine-grained control over thread and task scheduling in order
to avoid introducing jitter and other performance issues into guest VMs. It is
difficult to achieve these requirements with traditional, thread-based
designs.

Instead, OpenVMM uses Rust's async support throughout its codebase, decoupling
the policy details of where code runs (which OS threads) from the mechanism of
what runs (device-specific emulators). In a paravisor or resource-constrained
environment, OpenVMM can run with one thread per guest CPU and ensure that
device work is cooperatively scheduled along with the guest OS. In more
traditional virtualization host, OpenVMM can run with one thread per device to
use host CPUs to fully parallelize guest CPU and IO processing.

This approach has a significant impact on the design and implementation of the
codebase, and bringing this model to an existing VMM would be a major
undertaking. We came to the conclusion that a new project was the best way to
achieve this goal.

We are indebted to the Rust VMM community for their trailblazing work. Now that
the OpenVMM project is open source, we hope to find ways to collaborate on
shared code while maintaining the benefits of the OpenVMM architecture.

Guest Compatibility

Similar to other general-purpose VMMs (such as Hyper-V, QEMU, VirtualBox),
OpenVMM is able to host a wide variety of both modern and legacy guest operating
systems on-top of its flexible virtual hardware platform.

Modern operating systems can boot via UEFI, and interface with a wide
selection of paravirtualized devices for services like networking, storage, and
graphics.
Legacy x86 operating systems can boot via BIOS, and are presented with a
PC-compatible emulated device platform which includes legacy hardware such as
IDE hard-disk/optical drives, floppy disk drives, and VGA graphics cards.

OpenVMM is regularly tested to ensure compatibility with popular operating
systems (such as Windows, Linux, and FreeBSD), and strives to maintain
reasonable compatibility with other, more niche/legacy operating systems as
well.

To learn more about different facets of the OpenVMM project, check out the
following links:


Getting Started: OpenVMM	Running OpenVMM as traditional host VMM
Getting Started: OpenHCL	Running OpenVMM as a paravisor (OpenHCL)
Developer Guide: Getting Started	Building OpenVMM / OpenHCL locally
[Github] microsoft/openvmm	Viewing / Downloading OpenVMM source code
[Github] OpenVMM issue tracker	Reporting OpenVMM issues

Guide Src Reference Devices Firmware Overview (`Guide/src/reference/devices/firmware/overview.md`)

Firmware and Boot Modes

OpenVMM supports several ways to boot a guest VM, each with different
firmware requirements and guest OS compatibility:

Boot mode	Architecture	Firmware	Use case
UEFI	x86_64, AArch64	mu_msvm	Windows, modern Linux, full UEFI environment
PCAT BIOS	x86_64	Hyper-V PCAT BIOS	Legacy OS, Gen1-style boot
Linux Direct	x86_64, AArch64	None (VMM is the bootloader)	Fast Linux boot, development, testing
IGVM	x86_64, AArch64	Packaged in IGVM file	OpenHCL paravisor, confidential VMs

The boot mode is selected by which --kernel, --uefi, --pcat, or
--igvm flag is passed on the command line (or the equivalent ttrpc
configuration).

admonish note

Not all boot modes are available on all architectures — see the table
above for supported combinations.

Guide Src Reference Emulated NVMe Overview (`Guide/src/reference/emulated/NVMe/overview.md`)

NVMe emulator

Among the devices that OpenVMM emulates, an NVMe controller is one. The OpenVMM NVMe emulator comes in two flavors:

An NVMe emulator that can be used to serve IO workloads (but pragmatically is only used by OpenVMM for test scenarios today)
An NVMe emulator used to test OpenHCL (nvme_test), which allows test authors to inject faults and inspect the state of NVMe devices used by the guest.

This guide provides a brief overview of the architecture shared by the NVMe emulators. For how NVMe fits into the broader storage pipeline — including how namespaces map to DiskIo backends, online disk resize via AEN, and the layered disk model — see the storage pipeline page.

Guide Src Reference Emulated Pcie Overview (`Guide/src/reference/emulated/pcie/overview.md`)

PCIe Emulation

OpenVMM emulates a PCIe topology that presents root complexes,
root ports, and optional switches to the guest. Endpoint devices
(NVMe, virtio, etc.) attach to root ports via a generic bus
interface.

Topology

text

Root Complex (GenericPcieRootComplex)
├── Root Port 0  (PcieDownstreamPort)  → endpoint device
├── Root Port 1  (PcieDownstreamPort)  → endpoint device or switch
└── Root Port N  (PcieDownstreamPort)  → ...

The root complex owns the ECAM MMIO region. When the guest reads
or writes a config space address, the root complex decodes the
bus/device/function from the ECAM offset and routes the access
to the correct port. Each port has a Type 1 (bridge)
configuration space with PCIe Express and MSI capabilities.

Ports may optionally be hotplug-capable. Devices behind
non-hotplug ports are attached at VM construction time.
Hotplug-capable ports start empty and devices can be added or
removed at runtime.

PCIe Hotplug

Native PCIe hotplug follows the same interrupt-driven model as
real hardware (PCIe Base Spec §6.7). No ACPI GPE, SCI, or custom
protocol is needed — the guest's pciehp driver handles
everything via config space registers and MSI.

How it works

VMM sets port state: When a device is hot-added, the VMM
atomically updates the port's Slot Status
(presence_detect_state, presence_detect_changed,
data_link_layer_state_changed) and Link Status
(data_link_layer_link_active).
MSI fires: If the guest has enabled
hot_plug_interrupt_enable in the port's Slot Control
register, the VMM fires the port's MSI.
Guest handles the event: The guest's pciehp driver
receives the MSI, reads Slot Status to see what changed,
programs the bridge's bus numbers, scans the secondary bus
for new devices, and clears the RW1C status bits.
Device removal follows the same flow in reverse —
presence and link active are cleared, changed bits are set,
and MSI fires.

Runtime API

Hot-add and hot-remove are triggered via
VmRpc::AddPcieDevice and VmRpc::RemovePcieDevice messages.
These resolve a device resource, create the device with MMIO
registration, attach it to the named port, and fire the hotplug
notification.

See the
PetriVmRuntime
trait for the test API (add_pcie_device / remove_pcie_device)
and the pcie_hotplug test in
vmm_tests/vmm_tests/tests/tests/multiarch/pcie.rs for a
working example.

ACPI _OSC

The SSDT includes an _OSC method on each PCIe root complex
that grants native PCIe control to the OS (ACPI spec §6.2.11,
PCI Firmware Spec §4.5.1). This tells the OS it can use native
hotplug, PME, AER, and other PCIe features rather than
ACPI-based fallbacks. Linux assumes native control regardless,
but Windows requires _OSC to enable native hotplug.

MSI Interrupt Routing (aarch64)

On aarch64, PCIe MSI/MSI-X interrupts are routed through either
a GICv3 ITS or a GICv2m MSI frame, depending on the platform:

GICv3 ITS (default on KVM with GICv3): The VMM creates a
KVM in-kernel ITS device. Each MSI is delivered with a 32-bit
ITS device ID of the form (segment << 16) | BDF. The BDF is
resolved lazily at interrupt-delivery time by the device's
MsiTarget, which combines the parent port's AssignedBusRange
(updated by the guest as it programs secondary/subordinate bus
numbers) with the device's devfn. Multi-function devices and
root/switch ports that need a specific RID pass it explicitly.
A single ItsSignalMsi/ItsIrqFd wrapper per segment then
prepends the segment to produce the final ITS device ID — no
per-device wrappers and no push-based RID synchronization are
needed. ACPI boots emit an IORT with an ITS Group node and
per-root-complex ID mappings; the device tree includes an its
child node under the GIC with msi-controller.
GICv2m: MSI writes map to a fixed pool of 64 SPIs via
a v2m doorbell register. The MADT includes a GICv2m MSI
frame entry.

The MSI controller can be overridden with the --gic-msi
CLI option (auto, its, or v2m).

Implementation notes

admonish note title="no command completed support"

Hotplug ports advertise `no_command_completed_support` in Slot
Capabilities. Our emulation applies Slot Control changes
instantly, so the guest does not need to wait for command
completion. This avoids an interrupt storm that would occur if
`command_completed` were set on every Slot Control write — see
the comment in `with_hotplug_support()` in `pci_express.rs`
for details.

admonish note title="spurious pme interrupts"

The `pcieport` driver's PME service shares the same MSI vector
as `pciehp`. When a hotplug MSI fires, the PME handler also
runs and may log "Spurious native interrupt!" since there is no
PME event. These warnings are cosmetic and do not affect
functionality. A future improvement could use MSI-X (multiple
vectors) to give each service its own interrupt.

Guide Src SUMMARY (`Guide/src/SUMMARY.md`)

Summary

Introduction

Getting Started

Developer Guide

Reference

OpenVMM Rust Crate API Docs

Guide Src Rustdoc (`Guide/src/rustdoc.md`)

OpenVMM Rust Crate API Docs

Rust crate API documentation generated by cargo doc is available at the
following URLs:

Linux x86 - https://openvmm.dev/rustdoc/linux/openvmm
Windows x86 - https://openvmm.dev/rustdoc/windows/openvmm

Guide Src Dev Guide Contrib (`Guide/src/dev_guide/contrib.md`)

Contributing

This chapter discusses various things developers should be aware of prior to
submitting changes to the OpenVMM project.

This includes both code-level coding conventions, as well
as instructions on properly submitting changes to the OpenVMM
GitHub repo, and expectations around code reviews.

Guide Src Dev Guide Dev Tools (`Guide/src/dev_guide/dev_tools.md`)

Dev Tools / Utilities

While most tasks in the OpenVMM repo can be accomplished directly via standard
Rust tooling (i.e: cargo run, cargo build), there are many dev tasks that
don't neatly fall under the cargo umbrella. e.g:

running code formatters / linters
orchestrating multi-stage, multi-component OpenHCL builds
running different kinds of test suites
building/downloading test images for VMM testing
setting up git hooks
etc...

The following chapter discusses some of the various dev-facing tools / utilities
you may encounter and/or find useful when working on OpenVMM.

Rust-based Tooling

As with many projects, OpenVMM initially took the simple approach of spinning up
ad-hoc Bash/Python scripts, and hand-written YAML workflow automation.

This worked for a while... but as the project continued to grow, our once small
and focused set of scripts evolved into a mass of interconnected dependencies,
magic strings, and global variables!

To pay down mounting tech debt, and to foster a culture where all devs are
empowered to contribute and maintain OpenVMM's project tooling, we have adopted
a policy of migrating as much core tooling away from loosely-typed languages
(like Bash, Python, and hand-written Workflow YAML), and towards new
strongly-typed Rust-based tooling.

Guide Src Dev Guide Getting Started (`Guide/src/dev_guide/getting_started.md`)

Getting Started (for Developers)

This chapter discusses all the basic steps required to begin building code in
the OpenVMM project.

By the end of this chapter, you will have:

Cloned the OpenVMM git repo
Installed all required pre-build dependencies (e.g: Rust)
Built local copies of both OpenVMM and OpenHCL
(optionally) Set up a suggested VSCode-based development environment

_Aside:_ What is HvLite? Underhill?

As you explore the OpenVMM repo, you may find references to things called
HvLite and Underhill.

Simply put:

OpenVMM is synonymous with HvLite
OpenHCL is synonymous with Underhill

HvLite and Underhill were former Microsoft-internal codenames for OpenVMM and
OpenHCL.

Migrating all existing code and documentation away from these codewords is not
an overnight process, and it's quite likely these terms will linger in various
code comments, variable names, library names, etc... the forseeable future.

Guide Src Dev Guide Snapshot Format (`Guide/src/dev_guide/snapshot_format.md`)

Snapshot Format

This page documents the on-disk format used by OpenVMM snapshots, intended
for developers working on the save/restore subsystem.

Directory layout

A snapshot is stored as a directory containing three files:

text

snapshot-dir/
├── manifest.bin   # Protobuf-encoded SnapshotManifest
├── state.bin      # Protobuf-encoded device saved state
└── memory.bin     # Hard link to the guest memory backing file

Manifest format

The manifest is a protobuf message defined as
SnapshotManifest
in openvmm/openvmm_helpers/src/snapshot.rs, encoded using the mesh
crate's protobuf encoding.

Device state (`state.bin`)

The device state contains every device's saved state, collected via the
SaveRestore trait and encoded as a mesh protobuf message. The
Save State compatibility rules (mesh tag stability,
default values, forward/backward compatibility) apply.

Memory (`memory.bin`)

memory.bin is a hard link to the file-backed guest RAM file. During a save,
write_snapshot() creates this hard link using std::fs::hard_link.

admonish note

The hard-link approach means the memory backing file and snapshot directory
must reside on the same filesystem. If they are on different filesystems,
`write_snapshot` returns an error with a suggestion to place the backing
file inside the snapshot directory.

Same-file detection

If the user passes --memory file=<snapshot_dir>/memory.bin, the source and
target of the hard link are the same file. The code detects this by
canonicalizing both paths and comparing them. When they match, the hard-link
step is skipped.

Code references

Manifest type and I/O: openvmm/openvmm_helpers/src/snapshot.rs
Restore entry point: prepare_snapshot_restore() in
openvmm/openvmm_entry/src/lib.rs
File-backed memory: SharedMemoryFd type alias in
openvmm/openvmm_defs/src/worker.rs

Device state architecture

Each VM component that participates in save/restore is registered as a
"state unit" with a unique string name via StateUnits::add("name").
During save, every state unit receives a StateRequest::Save. Units that
have state return Ok(Some(blob)); units with no persistent state (e.g.
the input distributor) return Ok(None) and are omitted from state.bin.

The resulting state.bin contains a Vec<SavedStateUnit>, where each
entry pairs a unit name with its opaque protobuf-encoded state blob.

Restore matching rules

During restore, StateUnits::restore() matches saved-state entries to
currently registered units by name:

Scenario	Result
Names match exactly	State is dispatched to the unit
Saved entry has no matching unit	Error — `unknown unit name`
Unit exists with no saved entry	Unit is skipped (keeps default state)
Duplicate name in saved state	Error — `duplicate unit name`

This means removing a device between save and restore will fail, but
adding a new device is allowed (it initialises to its power-on defaults).

Unit naming conventions

Chipset devices — registered via arc_mutex_device("name") in
vmotherboard, e.g. "pit", "rtc", "uefi", "ide".
VMBus devices — named "{interface_name}:{instance_id}", e.g.
"StorageVsp:ba6163d9-...". The instance GUID makes each offer
unique.
Infrastructure units — "vmtime", "input", "vmbus".

Devices that do not support save/restore

Not all devices implement save/restore. Devices signal this in one of
two ways:

SaveError::NotSupported — the save() method returns this error.
If any state unit does this, the entire save operation fails.
supports_save_restore() -> false (virtio) or
supports_save_restore() -> None (VMBus) — transport-level check
that causes the transport's save() to return
SaveError::NotSupported.

Key unsupported categories:

PCIe — GenericPcieRootComplex, GenericPcieSwitch return
SaveError::NotSupported.
NVMe — NvmeController returns SaveError::NotSupported.
Pass-through PCI — AssignedPciDevice, RelayedVpciDevice.
VGA / GDMA — marked todo!() (will panic on save).
Virtio devices — the VirtioDevice trait defaults
supports_save_restore() to false. Only virtio-blk,
virtio-net, virtio-pmem, and virtio-rng override it to true.
Devices with host-side session state (virtio-9p, virtiofs,
virtio-console) intentionally leave it false.
Some VMBus devices — GuestCrashDevice, GuestEmulationDevice,
VmbusSerialHost, Vmbfs return None from
supports_save_restore().

Extending the format

When adding new fields to SnapshotManifest, use the next available mesh
tag number. The protobuf encoding is forward-compatible: older readers will
ignore unknown fields. However, removing or reordering existing fields is a
breaking change. See Save State for the full set of
compatibility rules.

admonish warning

Changing the mesh tag numbers of existing fields will break compatibility
with previously saved snapshots.

Guide Src Dev Guide Tests (`Guide/src/dev_guide/tests.md`)

Testing

This chapter discusses the various kinds of tests you will encounter as an
OpenVMM developer.

Guide Src Reference Architecture (`Guide/src/reference/architecture.md`)

Architecture

This chapter discusses the architecture of various key parts of the OpenVMM
codebase.

Timekeeping covers the clock domains,
timers, and lifecycle behavior shared by OpenVMM and OpenHCL.

Guide Src Reference Openhcl (`Guide/src/reference/openhcl.md`)

OpenHCL Features

This section discuss various features that are exclusive to OpenHCL.

Guide Src Reference Openvmm (`Guide/src/reference/openvmm.md`)

OpenVMM Features

This section discuss various features that are exclusive to OpenVMM.

Guide Src User Guide Openhcl (`Guide/src/user_guide/openhcl.md`)

OpenHCL

OpenHCL is an execution environment which runs OpenVMM as a paravisor.

It can be considered a form of "virtual firmware", running at a higher privilege
level than the primary guest OS.

Enabling existing workloads to seamlessly leverage Azure Boost (Azure's
next-generation hardware accelerator), without requiring any modifications to
the guest VM image.
Enabling existing guest operating systems to run inside Confidential VMs.
Powering Trusted Launch VMs - VMs that support Secure Boot, and include a
vTPM.

To learn more about OpenHCL's architecture, please refer to
OpenHCL Architecture.

admonish note

_Note:_ As you explore the OpenVMM repo, you may find references to the term
**Underhill**.

Underhill was the former codename for OpenHCL, so whenever you see the term
"Underhill", you can treat it as synonymous to "OpenHCL".

We are actively migrating existing code and docs away from using the term
"Underhill".

Guide Src User Guide Openvmm Img (`Guide/src/user_guide/openvmm-img.md`)

openvmm-img

openvmm-img creates, inspects, validates, and converts disk images using
OpenVMM's native storage implementations. It is part of the OpenVMM software
suite, but can be used independently of the virtual machine monitor.

The tool currently supports VHDX image management and conversion between raw
and VHDX images. Run it from the repository with Cargo:

bash

cargo run -p openvmm-img -- --help

Creating Images

Create a dynamic VHDX image by naming it with a .vhdx extension or by
specifying the format explicitly:

bash

cargo run -p openvmm-img -- create path/to/disk.vhdx --size 64G
cargo run -p openvmm-img -- create path/to/disk --format vhdx --size 64G

Format-specific options are passed as comma-separated key-value pairs. Use
allocation=fixed to provision all VHDX payload blocks during creation:

bash

cargo run -p openvmm-img -- create path/to/disk.vhdx --size 64G \
    --format-options allocation=fixed

Use --parent to create a differencing image. The parent implies dynamic
allocation, so it cannot be combined with allocation=fixed:

bash

cargo run -p openvmm-img -- create path/to/child.vhdx \
    --size 64G --parent path/to/parent.vhdx

A differencing child inherits its parent's logical sector size unless one is
specified explicitly. An explicit logical sector size must match the parent.
The child and parent may have different virtual disk sizes.

Run format-options without a format to list the supported formats, or pass a
format to list its accepted options:

bash

cargo run -p openvmm-img -- format-options
cargo run -p openvmm-img -- format-options vhdx

Inspecting and Validating Images

Use info for image metadata, map for allocated virtual ranges, and check
to validate VHDX metadata and its differencing parent chain:

bash

cargo run -p openvmm-img -- info path/to/disk.vhdx
cargo run -p openvmm-img -- map path/to/disk.vhdx
cargo run -p openvmm-img -- check path/to/disk.vhdx

Pass --json to info or map for machine-readable output. check exits
with status 2 for an inconsistent image and status 1 for other failures.

Use replay to repair an image with a dirty VHDX write-ahead log. Pass
--dry-run to report whether replay is required without changing the image.

bash

cargo run -p openvmm-img -- replay path/to/disk.vhdx --dry-run
cargo run -p openvmm-img -- replay path/to/disk.vhdx

Converting Images

Convert between raw and VHDX images by specifying the output format:

bash

cargo run -p openvmm-img -- convert path/to/disk.raw \
    --output path/to/disk.vhdx --output-format vhdx
cargo run -p openvmm-img -- convert path/to/disk.vhdx \
    --output path/to/disk.raw --output-format raw

VHDX output options use the same key-value syntax. For example, this creates a
fixed VHDX with 4 MiB payload blocks:

bash

cargo run -p openvmm-img -- convert path/to/disk.raw \
    --output path/to/disk.vhdx --output-format vhdx \
    --format-options allocation=fixed,block_size=4M

The input format is inferred from .raw, .img, or .vhdx. Use
--input-format when the input has another extension. Differencing VHDX inputs
are not supported by convert.

All-zero copy chunks are skipped so dynamic VHDX and raw outputs remain sparse
when the host filesystem supports sparse files.

Guide Src User Guide Openvmm (`Guide/src/user_guide/openvmm.md`)

OpenVMM

OpenVMM can be configured to run as a conventional hosted, or
"type-2" Virtual
Machine Monitor (VMM).

At the moment, OpenVMM can be built and run on the following host platforms:

Host OS	Architecture	Virtualization API
Windows	x64 / Aarch64	WHP (Windows Hypervisor Platform)
Linux	x64 / Aarch64	KVM
	x64 / Aarch64	MSHV (Microsoft Hypervisor)
macOS	Aarch64	Hypervisor.framework

When compiled, OpenVMM consists of a single standalone openvmm / openvmm.exe
executable.[^dlls]

For practical launch commands, see Running OpenVMM.

admonish note

As you explore the OpenVMM repo, you may find references to the term **HvLite**.

HvLite was the former codename for OpenVMM, so whenever you see the term
"HvLite", you can treat it as synonymous to "OpenVMM".

We are actively migrating existing code and docs away from using the term
"HvLite".

Notable Features

This non-exhaustive list provides a broad overview of some notable features,
devices, and scenarios OpenVMM currently supports.

Boot modes
- UEFI - via microsoft/mu_msvm firmware
- BIOS - via the Hyper-V PCAT BIOS firmware
- Linux Direct Boot
Devices
- Paravirtualized
  - Virtio
    - virtio-fs
    - virtio-9p
    - virtio-net
    - virtio-pmem
  - VMBus
    - storvsp
    - netvsp
    - serial
    - framebuffer
    - keyboard / mouse
    - vpci
- Direct Assigned (experimental, WHP only)
- Emulated
  - vTPM
  - NVMe
  - Serial UARTs (both 16550, and PL011)
  - Legacy x86
    - i440BX + PIIX4 chipset (PS/2 kbd/mouse, RTC, PIT, etc)
    - IDE HDD/Optical, Floppy
    - PCI
    - VGA graphics (experimental)
Device backends
- Graphics / Mouse / Keyboard (VNC)
- Serial (term, socket, tcp)
- Storage (raw img, VHD/VHDx, Linux blockdev, HTTP)
- Networking (various)
Management interfaces (evolving)
- CLI
- Interactive console
- gRPC
- ttrpc

For more information on any / all of these features, see their corresponding
pages under the Reference section of the OpenVMM Guide.

admonish note title="management interface compatibility"

OpenVMM's management interfaces continue to evolve and may change between
releases. Consult the documentation for your OpenVMM version, especially when
upgrading.

[^dlls]: though, depending on the platform and compiled-in feature-set, some
additional DLLs and/or system libraries may need to be installed.

Guide Src Dev Guide Contrib Code (`Guide/src/dev_guide/contrib/code.md`)

Coding conventions

One of our major goals with OpenVMM is to provide a high quality coding
experience for contributors, starting first-and-foremost by having a consistent
set of coding conventions in the project.

Do your part and keep OpenVMM clean!

`rustfmt`

Checked Automatically: Yes (via cargo xtask fmt --pass rustfmt)

OpenVMM source must be formatted using
rustfmt, which automatically and
mechanically applies standard formatting to all the code. This eliminates time
spent discussing or reviewing stylistic issues in pull requests.

The CI will run rustfmt --check to enforce consistent formatting, and will
fail if it notices any discrepancies.

Unfortunately, rustfmt isn't infinitely customizable, and there are several
rules that are must be manually enforced:

All lines must end with LF, not CRLF.
Top-level use imports should be non-nested.

admonish note

Some of these manually-enforced conventions were introduced late into
OpenVMM's development, and there may be chunks of the codebase that do not
adhere to these conventions.

If you're working in a file and notice that it isn't following a certain
convention, please take a moment to fix it!

Assuming you've followed the suggested dev env setup
and set up rust-analyzer to format-on-save, you should rarely have to think
about formatting in .rs files.

Custom Lints

Checked Automatically: Yes (via cargo xtask fmt --pass lints)

Custom lints are a set of miscellaneous code lints that are specific to OpenVMM, which
are enforced using a custom in-house tool:

enforce the presence of the standard Microsoft copyright header
enforce in-repo crate names don't use '-' in their name (use '_' instead)
enforce Cargo.toml files don't include autogenerated "see more keys" comments
enforce Cargo.toml files don't contain author or version fields
enforce files end with a single trailing newline
deny usage of #[repr(packed)] (you want #[repr(C, packed)])
justify usage of cfg(target_arch = ...) (use guest_arch instead!)
justify usage of expect(unsafe_code) with an UNSAFETY comment
and more!

Some of these lints are self explanatory, whereas others are described in more
detail elsewhere on this page.

Unused `Cargo.toml` Dependencies

Checked Automatically: Yes (via cargo xtask fmt --pass lints)

We have an in-repo fork of
cargo-machete that ensures
Cargo.toml files only include dependencies that are actually being used.

Avoiding unused dependencies makes it easier to reason about what a crate is
doing just by looking at its dependencies, and also helps cut-down on
incremental compile times.

Formatting (`Cargo.toml`)

Checked Automatically: No

When defining dependencies in Cargo.toml files, please organize dependencies
into the following groups:

crate-specific "subcrates"
crates under vm/
crates under vm/vmcore/
crates under support/
external dependencies

The rationale here is that crates should be grouped according to how related to
how widely applicable they are. i.e: crates from crates.io and support are
widely applicable outside of OpenVMM, whereas crates under vmcore/ only make
sense within the context of OpenVMM, and crate-specific subcrates are - by
definition - only applicable to the crate they are being imported from.

Additionally, we make use of the
workspace dependencies
feature to ensure that all our dependencies stay in sync. This requires defining
dependencies in your crate's Cargo.toml file and in the project's root Cargo.toml.

So, for example:

toml

[package]
name = "openvmm"

[dependencies]
# crate-specific subcrates
openvmm_core.workspace = true
...

# /vmcore
vmcore.workspace = true
...

# /vm/devices
firmware_uefi_custom_vars.workspace = true
storvsp.workspace = true
...

# /support
guid.workspace = true
inspect.workspace = true
inspect_proto.workspace = true
...

# external dependencies
anyhow.workspace = true
cfg-if.workspace = true
clap.workspace = true
...

Linting (via `clippy`)

Checked Automatically: Yes

OpenVMM uses cargo clippy to
supplement rustc's built-in lints.

Assuming you've followed the guide and set up rust-analyzer to
use clippy, you should see clippy lints
appear inline when working on Rust code.

The CI runs cargo clippy on every crate in the repo prior to building the
project, and will fast-fail if it catches any warnings / errors.

Suppressing Lints

In general, lints should be fixed by modifying the code to satisfy the lint.
However, there are cases where a lint may need to be expect'd inline.

In these cases, you must provide a inline comment providing reasonable
justification for the suppressed lint.

e.g:

rust,ignore

// x86_64-unknown-linux-musl targets have a different type defn for
// `libc::cmsghdr`, hence why these lints are being suppressed.
#[expect(clippy::needless_update, clippy::useless_conversion)]
libc::cmsghdr {
    cmsg_level: libc::SOL_SOCKET,
    cmsg_type: libc::SCM_RIGHTS,
    cmsg_len: (size_of::<libc::cmsghdr>() + size_of_val(fds))
        .try_into()
        .unwrap(),
    ..std::mem::zeroed()
}

OpenVMM's `clippy` Configuration

We stick fairly close to the default set of rustc / clippy lints, though there
are some default lints that we've decided to disabled project wide, and other
non-default lints which we've explicitly opted into.

See the [workspace.lints] sections of OpenVMM's root
Cargo.toml
for a list of globally enabled/disabled lints, along with justification as to
why certain lints have been enabled/disabled.

Unsafe Code Policy

admonish danger

When possible, try to avoid introducing new `unsafe` code!

Before rolling your own `unsafe` code, check to see if a safe abstraction
already exists, either in-tree, on crates.io\*, or in the standard library.

\*subject to an unsafe-code audit

Rather than synthesizing our own unsafe code conventions, we follow the
guidelines outlined in the following two resources:

In a nutshell:

unsafe functions are required to include /// # Safety documentation
describing any preconditions the caller must uphold when calling the function.
unsafe {} blocks are required to include a // SAFETY: comment describing
how the preconditions for calling the unsafe function(s) within the block
are being satisfied.
expect(unsafe_code) annotations are required to include an // UNSAFETY:
comment justifying why the code in question needs to use unsafe. This
annotation must be placed at the module or crate level.

These requirements are enforced by CI, and will cause the build to fail if
required documentation is missing.

Editing a file containing unsafe code will trigger CI to label your PR and warn
reviewers that the PR touches unsafe code. This is to ensure that all unsafe code
is audited for correctness by area experts.

Uses of `cfg(target_arch = ...)` must be justified

Checked Automatically: Yes (via cargo xtask fmt --pass lints)

Unless you're working on something that's genuinely tied to the host's CPU
architecture, you should use cfg(guest_arch = ...) instead of cfg(target_arch = ...).

OpenVMM is a multi-architecture VMM framework, capable of running running x64
guests on a x64 host, as well as Aarch64 guests on Aarch64 hosts (with the
potential of adding additional platforms in the future).

At the moment, OpenVMM requires that the host architecture and guest
architecture match. That said, it's possible that at some point in the future,
OpenVMM may also support mismatched guest/host architectures, via an emulated
CPU virtualization backend (akin to QEMU).

Having an emulated CPU backend would enable OpenVMM to support such useful
scenarios as:

running Arch64 guests on x86 machines
running x86 guests on ARM
running something exotic (e.g: RISC-V) on x86/ARM machines
- ...assuming we had the bandwidth to implement + maintain something like that
running OpenVMM on systems without hardware-accelerated CPU virtualization enabled

With these scenarios in mind, it would be short-sighted to rely entirely on
cfg(target_arch = ...) to gate guest-facing arch-specific functionality, as it
would inexorably tie the guest's arch to the host's arch, making any future
initiatives to pry the two apart significantly more difficult!

As such, the OpenVMM repo includes infrastructure to specify a custom,
OpenVMM-specific cfg(guest_arch = ...) cfg parameter.

By default, cfg(guest_arch = ...) will act the same way as cfg(target_arch = ...), but it can be swapped to a different architecture by setting the
OPENVMM_GUEST_TARGET env var at compile-time.

There are very few reasons to use cfg(target_arch = ...) within the OpenVMM
repo, and to enforce this rule, we have an in-house xtask fmt --pass lints
check that lints each use of cfg(target_arch = ...) to include a
"justification" for why it's being used.

e.g: cfg(target_arch = ...) would be applicable when feature-gating a CPU
intrinsic (such as CPUID, or a SIMD instruction), or when implementing a
*-sys crate where the underlying C API/ABI varies between architectures.

...otherwise, use cfg(guest_arch = ...)!

Avoid `Default` when using `zerocopy::FromZeros`

Checked Automatically: No

The rule:

A type can derive(Default) XOR derive(FromZeros).
A type that is FromZeros can also impl Default, but it must be a
conscious, explicit choice, with justification (read: inline comment) as to
why that particular default value was chosen.
N.B. derive(IntoBytes) or derive(FromBytes) imply derive(FromZeros). That is:
this convention applies to those types as well.

The why:

The all-zero type is often not a semantically valid Default value for a type
There are plenty of types that don't have a Default value, but do have a
valid all-zero repr

Additional context

As per the Rust docs for Default::default:

admonish quote

`fn default() -> Self`

Returns the “default value” for a type.

Default values are often some kind of initial value, identity value, or anything
else that may make sense as a default.

Notably, default should not be used as some shorthand to "zero initialize"
values! For most non-trivial structs, the all-zero representation is not a
semantically valid Default!

This is true in many contexts... but one that's particularly relevant in OpenVMM
is that of FFI via C-style APIs and ABIs.

In C, it's very common for types to undergo multi-stage instantiation, where
they are initially allocated as all-zero, and then get "filled in" by some
secondary init code. Notably: it's quite rare for that initial "all-zero" struct
to be a valid instance of the type!

Example: C FFI

For example, a common pattern in C libraries might look something like:

struct Handle {
    uint16_t opaque_handle;
}

struct Handle handle = {0};
init_handle(&handle);

update_handle(handle, options);
do_thing(handle);

In this example: it would be an error to invoke update_handle or do_thing
with an all-zero handle, as the type hadn't finished being fully initialized.

If we wanted to use this library from Rust, a "naive" approach would be to do
something like:

rust,ignore

#[repr(C)]
#[derive(Default)]
struct Handle {
    opaque_handle: u16,
}

let mut handle = Handle::default(); // BAD!
unsafe { init_handle(&handle) };

unsafe { update_handle(handle, options) };
unsafe { do_thing(handle) };

While this technically works... using Default here is kinda bogus!

After all - Handle::default() doesn't actually call init_handle, which means
the value returned by default() doesn't match the "promise" of the trait!
Namely: the returned value is not semantically valid yet!

In many other Rust codebase, this "overloading" of Default to represent both
semantically valid value and all-zero values (in FFI) is par for the course,
as once structs get more complicated, having a derive that is able to fully
init a "uninitialized" struct in-memory is quite handy...

In OpenVMM, we don't do this. Instead, we use a separate trait to init all-zero
structs.

In OpenVMM, we use FromZeros and FromZeros::new_zeroed() to work with types
that have valid all-zero representations, without implying that those types
also have valid all-zero default values!

So, for the example above:

rust,ignore

#[repr(C)]
#[derive(zerocopy::FromZeros)]
struct Handle {
    opaque_handle: u16
}

let mut handle = Handle::new_zeroed(); // GOOD!
unsafe { init_handle(&handle) };

Now, it's impossible for code elsewhere to obtain a Handle via
Handle::default, and mistakenly forget to invoke init_handle on it.

...but if it so happens that we do want a Default impl for Handle, we can
do so by manually implementing derive(Default) ourselves:

rust,ignore

// Default + FromZeros: `default` returns fully initialized handle
impl Default for Handle {
    fn default() -> Handle {
        let mut handle = Handle::new_zeroed();
        unsafe { init_handle(&handle) };
        handle
    }
}

Avoid Requiring `Debug` on Traits

Checked Automatically: No

TL;DR: Don't do this:

rust,ignore

trait MyTrait: std::fmt::Debug

Implementations of the standard library's Debug trait can be surprisingly large,
and the final binary size of OpenHCL and related binaries is a major concern
for us. Unused implementations of this trait are usually removed during the
optimization process (like all dead code), making this a non-issue. However when
traits, and more specifically trait objects, are involved, the compiler has a
much more difficult time proving that implementations are unused. This can
result in large amounts of functionally dead code ending up in the final
binaries.

If you need to implement Debug for a struct containing such a trait object, you
will need to do so manually, so that you can skip over that field.

Moreover, it's usually not good form to leave tracing statements that log a
struct's Debug representation in production. Prefer tracing just the fields
you're interested in, and/or connecting objects to the Inspect graph.

Crate Naming

Checked Automatically: Yes (via cargo xtask fmt --pass lints)

Crates must be named with underscores, not dashes and underscores used in
folder names.

Bad: my-cool-crate
Good: my_cool_crate

Rust does not allow dashes in imports, with any dashes getting replaced with
underscores when used in the code. Avoiding dashes altogether makes it easier to
grep for crate names, and makes things more consistent across the repo.

This convention is enforced by CI

Do not name crates with the words "base, util, common" or other terms that are
overly general.

For example, consider a crate that provides a common data structure used by
multiple devices:

Bad: devices_common
Good: range_map

Libraries that contain the following eventually become a mishmash of unrelated
functionality that is located there for convenience. This
blog post
goes more in-depth as to why.

Instead, name things based on what they logically provide, like functionality or
data types.

Release Gates Workflow

Triggered Manually: Yes (via GitHub labels)

In addition to the standard PR gates that run in debug mode, OpenVMM also provides
a "Release Gates" workflow that runs the same checks but compiled in release mode.
This workflow takes significantly longer to run but can catch issues that only
manifest in optimized builds.

When to Use Release Gates

The release gates workflow should be used when:

Making changes to performance-critical code paths
Modifying compiler flags or build configuration
Implementing low-level optimizations
Debugging issues that only appear in release builds
Before merging large refactoring changes

How to Trigger Release Gates

To run the release gates on your PR:

Ensure your PR is ready for review (not in draft mode)
Add the release-ci-required label to your PR
The workflow will automatically trigger and run all checks in release mode

The workflow will only run when the specific label is present, so you have full
control over when to use this more resource-intensive testing.

Label Management

Only repository maintainers can add labels to PRs. If you need release gates
run on your PR, ask a maintainer to add the release-ci-required label for you.

Guide Src Dev Guide Contrib Code Review (`Guide/src/dev_guide/contrib/code_review.md`)

Code Review Guidance

The Code Review Process

Although the OpenVMM repository requires a maintainer to sign off before a pull
request can be merged, this does not mean non-maintainer reviews are not
valuable. It is extremely important for the health of the project for people to
review areas of code they are familiar with. For example, for a change that
touches multiple components, somebody familiar with one component could leave a
review that says, "The changes to component A look good, but I'm not familiar
with the rest of the code." These types of reviews are still extremely useful
and help move changes forward. Maintainers will also feel more confident merging
changes once subject-area experts have weighed in.

Merge Permissions

Approval and merge access are separate. A maintainer must approve a pull request
before it is merged. Contributors who need permission to merge pull requests
should be added to the microsoft/openvmm-write GitHub team, which grants
general write access to the repository. Contact an OpenVMM maintainer to request
membership.

Membership in the team allows a contributor to perform the merge after the
required reviews and status checks have completed. It does not replace the
maintainer approval requirement.

Submitter's Responsibility

If your patch or change isn’t gaining traction, it is your responsibility as
the submitter to follow up and ensure the right people review your changes. This
means sending polite reminders to potential reviewers. Do not rely on GitHub
notifications for this because people may have them filtered or disabled. The
OpenVMM repository has a
CODEOWNERS
file, which will automatically add teams to the review for some components. If
you are unsure who the correct audience for the review is, please ask for
guidance from maintainers or other contributors.

Guide Src Dev Guide Contrib Crypto Backends (`Guide/src/dev_guide/contrib/crypto_backends.md`)

Crypto Backends

The crypto crate (support/crypto) abstracts over several backend
implementations of the cryptographic primitives OpenVMM and OpenHCL need.
Exactly one backend must be selected per binary, chosen via Cargo features:

Feature	Backend
`openssl`	OpenSSL (typical for Linux / OpenHCL).
`symcrypt`	SymCrypt.
`rust`	Pure-Rust implementations from RustCrypto.
`native`	Platform default — OpenSSL on Linux, BCrypt/CNG on Windows, Security.framework on macOS.

Selection happens in support/crypto/build.rs,
which emits a cfg (openssl, symcrypt, rust, or native) based on
the enabled features.

The "wrong number of backends" error

Because Cargo unifies features across a workspace, building two binaries
in the same cargo invocation that ask for different crypto backends
will result in the crypto crate being compiled with all of those
features enabled at once. Conversely, building a library crate that
transitively depends on crypto without itself selecting a backend
(e.g. running its unit tests) results in crypto being compiled with
zero backends enabled. In either case there is no sensible single
backend to pick.

To keep workspace-wide cargo check and cargo test usable, the build
script does not panic in this situation — it picks a placeholder backend
so the crate still compiles. To still guarantee that a shipping binary
has linked exactly one backend, each binary opts into a link-time check
by invoking the crypto::ensure_single_backend!() macro (typically gated
on #[cfg(not(test))]). The macro emits a #[used] reference to a symbol
that is only defined by crypto when exactly one backend is selected.
The result:

cargo check --workspace — succeeds. No linking happens.
cargo test -p — succeeds, even when that crate
transitively depends on crypto without selecting a backend.
cargo build -p for a binary that invokes the macro — fails
at link time if zero or multiple backends end up enabled, with:
text
```
rust-lld: error: undefined symbol:
  __openvmm_crypto_ensure_single_backend__enable_exactly_one__see_support_crypto
```

The symbol name is the diagnostic: the offending binary has the wrong
number of crypto backend features enabled and needs to pick exactly one.

Fixing it

Identify which binary you are building and which crypto features it
(transitively) enables. cargo tree -e features -p -i crypto
is usually the fastest way.
Either narrow your cargo build invocation (e.g. -p
instead of --workspace), or adjust the offending crate's
Cargo.toml so it stops enabling a backend it shouldn't.
Remember that adding features = ["native"] (or any other backend)
to a library crate's dependency on crypto will force that
backend on every binary that links the library. Backends should
normally be selected by binary crates only.

Guide Src Dev Guide Contrib Guide (`Guide/src/dev_guide/contrib/guide.md`)

Updating this Guide

We gladly welcome PRs that improve the quality of the OpenVMM guide!

The OpenVMM Guide is written in Markdown, and rendered to HTML using
mdbook. You can find the source-code of
this Guide in the main OpenVMM GitHub repo, in the
Guide/ folder.

Guide conventions

Line Length: Wrap lines at 80 characters.

Editing the Guide

Small Changes

For small changes, you can simply click the "Suggest an Edit" button in the
top-right corner of any page to automatically open up a GitHub Edit page.

Medium Changes

For medium changes, we suggest cloning the repo locally, and previewing changes
to Markdown in your editor (Visual Studio Code has good support for this).

Large Changes

For large changes, we suggest cloning the repo locally, and building a fully
rendered copy of the Guide using mdbook.

This is very useful when making changes that leverage mdbook preprocessors, such
as using mermaid diagrams, or previewing admonishments.

admonish info

For example, the `mdbook-admonish` preprocessor is what powers this nice looking
"Info" box!

Building the Guide locally is quite straightforward:

Install mdbook and the additional preprocessors we use locally:

bash

cargo install mdbook
cargo install mdbook-admonish
cargo install mdbook-mermaid

Navigate into the Guide/ directory, and run mdbook:

bash

cd Guide/
# must be run inside the `Guide/` directory!
mdbook serve

Navigate to the localhost URL in your web browser (typically
http://127.0.0.1:3000/)

Troubleshooting

Running `mdbook serve` outside the `Guide/` directory

Error:

text

2024-10-29 16:26:22 [INFO] (mdbook::book): Book building has started
error: manifest path `./mdbook-openvmm-shim/Cargo.toml` does not exist

Solution:

Ensure you have changed your working-directory to the Guide/ folder (e.g: via
cd Guide/), and then run mdbook serve.

Rust is not installed

Error:

text

2024-10-29 16:35:49 [INFO] (mdbook::book): Book building has started
2024-10-29 16:35:49 [WARN] (mdbook::preprocess::cmd): The command wasn't found, is the "admonish" preprocessor installed?
2024-10-29 16:35:49 [WARN] (mdbook::preprocess::cmd):   Command: cargo run --quiet --manifest-path ./mdbook-openvmm-shim/Cargo.toml mdbook-admonish

Solution:

The OpenVMM Guide hooks into a custom Rust utility called mdbook-openvmm-shim,
which must be compiled in order for mdbook to successfully build the OpenVMM
guide.

Please ensure you have installed Rust.

Guide Src Dev Guide Contrib Openvmm Packaging (`Guide/src/dev_guide/contrib/openvmm_packaging.md`)

Packaging OpenVMM for Linux

This page describes the source inputs and build configuration OpenVMM
provides for Linux distribution packages.

Release inputs

Releases are tagged openvmm-v<VERSION>. GitHub automatically
generates the source archive for every release tag. The .tar.gz URL
is:

text

https://github.com/microsoft/openvmm/archive/refs/tags/openvmm-v<VERSION>.tar.gz

OpenVMM also uploads one release asset:
openvmm-<VERSION>-vendor.tar.gz.

<VERSION> is the canonical [workspace.package] version in the root
Cargo.toml.

GitHub derives the source archive root from the repository and tag
names. For the release tag above, expect
openvmm-openvmm-v<VERSION>/. Confirm the root against the downloaded
archive before finalizing distribution setup macros.

The source archive contains no .git directory. The workspace version
is already in Cargo.toml, so a binary built from the extracted archive
reports the plain workspace version.

The source archive covers the entire tracked tree, including components
a Linux package does not build. Build -p openvmm and package only that
binary.

The vendor archive contains exactly these top-level paths:

text

cargo_config
vendor/

cargo_config is the exact stdout of
cargo vendor --locked --versioned-dirs vendor. The vendor/ tree
comes from the same invocation, so it includes both crates.io and locked
git dependencies needed by the workspace.

cargo vendor operates on the workspace, so the vendor tree includes
dependencies not compiled by the OpenVMM Linux binary.

admonish note

GitHub guarantees stable extracted source contents when a tag continues
to name the same commit, but its compression settings may change.
Distribution lookaside caches should retain the exact downloaded source
archive and vendor archive used by the package.

Verifying the release

Confirm that the release tag points to the expected commit:

bash

git ls-remote https://github.com/microsoft/openvmm.git \
    refs/tags/openvmm-v<VERSION>

Download the source archive and vendor archive once when updating the
distribution package, record their digests in the distribution's source
metadata, and retain those exact bytes in the distribution's lookaside
cache.

OpenVMM does not upload a checksum file, an OpenPGP signature, or a
provenance attestation for either archive.

Build requirements

The distribution build requires:

the Rust toolchain required by the workspace;
a C compiler and linker;
glibc development headers;
Linux UAPI headers;
OpenSSL development headers;
pkg-config;
a Protocol Buffers compiler providing protoc.

These map to the following distribution packages, which is what
OpenVMM's release validation job installs:

Requirement	Debian / Ubuntu	Fedora
C compiler and linker	`build-essential`	`gcc`, `binutils`
Linux UAPI headers	`linux-libc-dev`	`kernel-headers`
OpenSSL development headers	`libssl-dev`	`openssl-devel`
`pkg-config`	`pkg-config`	`pkgconf-pkg-config`
`protoc`	`protobuf-compiler`	`protobuf-compiler`

Building with the optional tpm feature additionally requires CMake, provided
by the cmake package on both Debian / Ubuntu and Fedora.

Do not use cargo xflowey restore-packages when building a distribution
package. That command restores prebuilt native dependencies intended for
repository development.

Offline build configuration

Extract the GitHub source archive, then extract the vendor archive into
that source tree:

bash

tar -xzf openvmm-v<VERSION>.tar.gz
cd openvmm-openvmm-v<VERSION>
tar -xzf ../openvmm-<VERSION>-vendor.tar.gz

Append the shipped cargo_config bytes to .cargo/config.toml. Do not
overwrite the existing file:

bash

printf '\n' >> .cargo/config.toml
cat cargo_config >> .cargo/config.toml

Then build the host x86_64-unknown-linux-gnu target with vendored
dependencies:

bash

export PROTOC="$(command -v protoc)"
export OPENSSL_NO_VENDOR=1
cargo build --release --locked --offline -p openvmm \
    --target x86_64-unknown-linux-gnu

admonish warning

The shipped `.cargo/config.toml` must keep its existing settings. If
your packaging flow has already added a `[source]` table, merge the
vendored-source configuration manually instead of overwriting the file.

The vendor tree naturally preserves license files shipped by upstream
dependencies. OpenVMM does not add an aggregate license inventory to the
vendor archive. Downstream tooling may generate feature-aware manifests
from the extracted vendor tree when distribution policy requires them.

Distribution integration

Mapping GitHub's generated source archive onto a distribution's
conventions is the packager's job, but the points below are the ones
OpenVMM's layout affects directly.

RPM

GitHub's generated source archive is expected to unpack into
openvmm-openvmm-v%{version} rather than %{name}-%{version}. Confirm
the root directory, then pass it explicitly:

spec

%autosetup -n openvmm-openvmm-v%{version}

Fedora's Rust packaging
guidelines
additionally require, for a package like OpenVMM that ships an
executable with statically linked Rust dependencies:

%cargo_prep in %prep, after the source and vendor archives are
unpacked and the vendored source configuration is appended.
%cargo_license_summary (and %cargo_license) in %build after the
build, because the package includes a binary target. Any feature flags
passed to the build must also be passed to these macros, or the
recorded licenses will not match what was compiled.
%cargo_vendor_manifest in %build when policy requires a generated
manifest for the vendored sources. It writes cargo-vendor.txt, which
can then be shipped as a %license file.
%cargo_generate_buildrequires in %generate_buildrequires except
when building only from vendored dependencies.

Debian

Configure debian/watch for the GitHub-generated source archive and
fetch the matching vendor archive as an additional source. See the
Debian watch documentation for
the GitHub-specific patterns.

Record the vendored crate licenses in debian/copyright for the same
reason Fedora may generate a vendor manifest from the extracted tree.

admonish note

OpenVMM does not publish an OpenPGP signature or `SHA256SUMS` alongside
either archive, so `uscan` signature verification cannot be enabled.
Confirm the release tag and retain the exact downloaded archives used
for the package.

Package identity

The OpenVMM binary reports the upstream product version committed in the
source tree. Record a distribution-specific package revision in the
distribution package metadata rather than replacing the binary version.

OpenVMM deliberately offers no environment variable or build flag that
overrides the reported version. The version is chosen by a reviewed
commit to the root Cargo.toml, which keeps a binary's self-reported
identity tied to the source it was built from.

Runtime dependencies

Confirm the exact dependencies for the packaged executable with
readelf or the distribution's automatic dependency generator. The
expected shared libraries include glibc, OpenSSL (libssl and
libcrypto), and libgcc_s. SQLite is compiled into the binary and
does not add a shared runtime dependency.

Guide Src Dev Guide Contrib Openvmm Source Release (`Guide/src/dev_guide/contrib/openvmm_source_release.md`)

Cutting an OpenVMM source release

This page explains how maintainers create an OpenVMM source release on GitHub.

For the packager's view of the shipped inputs, see
Packaging OpenVMM for Linux.

What a release is

An OpenVMM source release is a GitHub release tagged
openvmm-v<VERSION>.

GitHub automatically provides the source archive for that tag. OpenVMM
uploads exactly one additional asset:
openvmm-<VERSION>-vendor.tar.gz.

That vendor archive contains the vendored Cargo dependency tree and the
exact cargo vendor source-replacement snippet needed for offline
builds. No custom source archive is assembled or uploaded.

A release publishes source. Prebuilt binaries, and any commitment to
service a published version, are outside this process.

<VERSION> is the version field under [workspace.package] in the
OpenVMM repository's root Cargo.toml. Nothing is stamped into the
source archive, and the tree is never rewritten, so the version a
release publishes is the version that was already reviewed and merged.

Selecting the version

Open an ordinary pull request that sets that field to the version being
released, and merge it. That review is the decision to release that
version; the release workflow only ever publishes what the tree already
says.

If the committed version has never been released and is already the
intended value, the pull request may instead state explicitly that it
selects that existing version. A no-op edit to Cargo.toml is not
required.

admonish note

The version stays at the released value after publication. Commits made
afterwards report `<VERSION>+g<COMMIT>` and are identifiable by the
appended commit, so there is no second commit to "reopen" the
version.

Running the workflow

Dispatch the OpenVMM Source Release workflow against the commit to
release. It has no automatic triggers; it only runs when someone asks
for it.

The workflow:

reads [workspace.package] version and the commit from the checkout;
assembles openvmm-<VERSION>-vendor.tar.gz once with
cargo vendor --locked --versioned-dirs;
checks out the same revision, appends the generated cargo_config
to .cargo/config.toml, and builds openvmm with
--locked --offline;
creates or verifies openvmm-v<VERSION> at that commit, after
confirming no release already exists for it;
creates a draft release for that tag, relies on GitHub's
automatic source archive, and uploads only the vendor archive.

The workflow does not upload SHA256SUMS or a provenance attestation.

Publishing

Before publishing, confirm the draft still targets the commit the
workflow pinned. The workflow logs that commit when it creates
openvmm-v<VERSION>, and
git ls-remote origin refs/tags/openvmm-v<VERSION> reports where the
tag points now. They must match.

That check is the one thing publication cannot do for you. A draft
release tracks a tag name, not a commit, and GitHub enforces
immutability only after publication. Creating the tag up front means
publication cannot invent a tag at some other commit, but nothing in the
workflow prevents someone from moving an existing tag while the draft
sits in review.

Then review the draft, write its notes, and click Publish release.

admonish warning

The tag is publicly visible while the release is still a draft. Do not
move or delete it during review. Publishing is the irreversible step: a
published tag is not moved or deleted. Correcting a release means
merging a pull request that selects a new patch version and running the
workflow again.

The workflow fails rather than overwriting an existing release for the
same tag, since that release may already have been reviewed or
published. It checks for that release before creating the tag, so a
refused run does not leave a tag behind. If a run must be retried before
publication, it reuses the tag only when the tag still names the exact
pinned commit. A maintainer may delete the draft and keep the matching
tag. Deleting the tag is reserved for exceptional pre-publication
cleanup.

Limitations

The workflow validates one Linux openvmm build with the uploaded
vendor archive. Distribution-specific packaging steps still belong to
the downstream package.

admonish note

OpenVMM does not publish an OpenPGP signature, `SHA256SUMS`, or a
provenance attestation for either the GitHub-generated source archive or
the uploaded vendor archive. Consumers can confirm that the release tag
names the expected commit and retain the exact downloaded bytes they
package.

Guide Src Dev Guide Contrib Pr (`Guide/src/dev_guide/contrib/pr.md`)

Submitting Changes

Follow the CONTRIBUTING guide.

Guide Src Dev Guide Contrib Release (`Guide/src/dev_guide/contrib/release.md`)

Release Management

The OpenVMM repository periodically creates releases from main, typically
about once a month. Releases are named for the year and month in which they
were forked, using YYMM. For example, a release forked in September 2026 is
release 2609.

Each monthly release is represented by a branch named release/<YYMM>, such
as release/2609. The branch remains available as the release baseline and
receives backported fixes if that release needs stabilization or servicing.

The repository produces both the OpenVMM VMM and the OpenHCL paravisor.
OpenVMM and OpenHCL retain their own product versions; the YYMM release
identifies their repository baseline. OpenHCL currently ships about twice a
year and selects one of the repository releases as the baseline for each
product release.

Before publishing an OpenVMM source release from main or a release branch,
ensure that the workspace version is the intended SemVer and has not previously
been published. If the current version has already been published, update it in
a normal pull request. A new feature release from main typically increments
the minor version, such as 1.1.0 to 1.2.0. A servicing release from a
release branch increments the patch version, such as 1.1.0 to 1.1.1. An
already committed, unpublished version can be selected without changing
Cargo.toml. The same workspace version may exist in multiple branches during
development, but it can be published as an openvmm-v<VERSION> release only
once. Continuing development or creating a release branch does not by itself
require a version bump. See
Cutting an OpenVMM source release for the
publication process.

We expect a high quality bar for all code that goes into the OpenVMM main
branch, and we ask developers to hold release branches to the highest quality
standards. The OpenVMM maintainers will gradually slow the rate of churn into
these branches as we get closer to a shipping date.

admonish note title="see also"

[Security Releases](security_releases.md) describes private reporting and
coordinated disclosure for fixes that cannot be developed publicly.

admonish note

Some existing release branches use the older
`release/<MAJOR>.<MINOR>.<YYMM>` format, such as `release/1.8.2607`.

This process should not impact your typical workflow; all new work should go
into the main branch. But, to ease the cherry-picks, we may ask that you hold
off from making breaking or large refactoring changes at points in this
process.

Marking, Approval Process, Code Flow

The OpenVMM maintainers will publish various dates for the upcoming releases.
Currently, these dates are driven by a Microsoft-internal process and can, and
do, often change. Microsoft does not mean to convey any new product launches by
choices of these dates.

Releases naturally fall into several phases:

Phase	Meaning
Active Development	Regular development phase where new features and fixes are added.
Stabilization	Phase focused on stabilizing the release by fixing bugs.
Ask Mode	Only critical fixes are allowed; changes are scrutinized. No new features. This is the last phase before a release is closed.
Servicing	Only essential fixes are made to support the release (a.k.a. maintenance mode).
Out of service	A previous release which is no longer receiving updates.

By default, a monthly release branch receives short-term servicing until the
next monthly release supersedes it. Branches selected for an OpenHCL product
release receive extended servicing.

For OpenHCL, the plan of record is to service the current product release and
the two preceding product releases. Because OpenHCL currently releases about
twice a year, this is approximately 18 months of servicing. The release-count
policy is authoritative if the release cadence changes. When a new OpenHCL
release ships, the release that is now three product releases behind moves out
of service.

This servicing policy applies prospectively to releases created under the new
model. Existing release branches retain the lifecycle states listed below.

Backport acceptance bar

Release branches accept security fixes and critical bug fixes only. A critical
bug fix addresses one of the following:

A crash, data corruption, or data loss.
A severe reliability or availability regression.
A critical compatibility or servicing failure.
An issue that prevents a supported scenario from functioning.

New features, routine performance improvements, cleanup, and refactoring are
not accepted. A narrowly scoped prerequisite for an approved fix may be
considered with the same level of scrutiny. Changes must land in main first
and are selectively backported after assessing their risk to the serviced
release.

Release branch process

In the label and tooling examples below, <RELEASE> is the suffix of the
release branch. For release/2609, <RELEASE> is 2609.

When creating a monthly release branch:

Create the release_<RELEASE>, backport_<RELEASE>, and
backported_<RELEASE> labels in GitHub.
Add the release_<RELEASE> base-branch rule to .github/labeler.yml so pull
requests targeting the branch are labeled consistently.
Add the branch to both run_filters and test_filters in
petri/logview/src/branch_quick_filters.tsx so it appears in the Runs,
Tests, and TestDetails quick filters.

We track the state of candidates for a given release by tagging the PRs with the following labels:

backport_<RELEASE>: This PR (to main) is a candidate to be included in the release.
- N.B.: A maintainer will remove this tag if the fix is not accepted into the release.
backported_<RELEASE>: This PR (to main) has been cherry-picked to the release branch.

The repo_support/relabel_backported.py script can be used to automatically transition PRs from backport_<RELEASE> to backported_<RELEASE> once they have been cherry-picked to the release branch.

Seeking Approval for Backport

To seek approval to include a change in a release branch, follow these steps:

Tag your PR to main with the backport_<RELEASE> label.
Wait for the PR to be merged to main.
Cherry-pick the change to the appropriate release branch in your fork and
stage a PR to that same branch in the main repository.

Please reach out to the maintainers before staging that PR if you have any
doubts.

Backport PR Best Practices

When creating a backport PR to a release branch:

Clean cherry-picks are strongly preferred. A clean cherry-pick minimizes
reviewer effort and reduces the risk of introducing regressions.
If the backport is not a clean cherry-pick (e.g., requires conflict
resolution or additional modifications), clearly indicate this in the PR
description. This signals to the reviewer that extra care is needed during
the review process.

Existing Release Branches

Release	Phase	Notes
release/2411	Out of service
release/2505	Out of service	Supports runtime servicing from release/2411.
release/1.7.2511	Servicing	Supports runtime servicing from release/2411 and release/2505.
release/1.8.2607	Ask Mode	Supports runtime servicing from release/2411, release/2505, and release/1.7.2511.
tbd, in main	Active Development	Supports runtime servicing from release/2411, release/2505, and release/1.7.2511.

Taking a Dependency on a Release

We welcome feedback, especially if you would like to depend on a reliable
release process. Please reach out!

Guide Src Dev Guide Contrib Save State (`Guide/src/dev_guide/contrib/save-state.md`)

Save State

OpenHCL supports the mechanism of saving & restoring state. This primitive can
be used for various VM operations, but a key use case is for updating OpenHCL at
runtime (a.k.a. "Servicing"). This save state can be stored in memory or on
durable media, read back at a later time.

Save State First Principles

Here are the principles you must maintain when adding new save & restore code:

Save & Restore is Forward & Backwards Compatible: A newer version of
OpenHCL must understand save state from a prior version, and an older version
must not crash when reading save state from a newer version.
Do not break save state after that state is in use: Save state must be
compatible from any commit to any other commit, once the product has
shipped and started using that save state.[^1]
All Save State is Protocol Buffers: All save state is encoded as
ProtoBuf, using mesh.

Best practices

Put save state in it's own module: This makes PR reviews easier, to catch
any mistakes updating save state.
Create a unique package per crate: A logical grouping of saved state
should have the same package
Avoid unsupported types, when possible: These types don't support the
default values needed for safely extending save state:
- Arrays: if you need to add an array, consider a vec or Option<[T; N]>
  instead.
- Enum: if you need to add an enum, add it as Option<MyEnum> instead.
Be particularly careful when updating saved state: See below.

Updating (Extending) Saved State

Since saved state is just Protocol Buffers, use the guide to updating Protocol
Buffers messages as a
starting point, with the following caveats:

OpenVMM uses sint32 to represent the i32 native type. Therefore, changing
i32 to u32 is a breaking change, for example.
The Protocol Buffers docs mention what happens for newly added fields, but it
bears adding some nuance here:
1. Plain arrays and enums are not supported. Reading newer save state
  with an array or enum will fail on an older build. Instead, wrap
  these in an Option.
2. Old -> New Save State: Save state from a prior revision will not contain
  some newly added fields. Those fields will get the default
  values. This is
  how that breaks down for the rust types:
  - Option<T> => None
  - Structs => each field gets that field's default value
  - Vecs => empty vec
  - Numbers => 0
  - Strings => ""
3. New -> Old Save State: Unknown fields are ignored.
Ensure that default values for new saved state fields make semantic sense.
Here is the heuristic: if you need a conditional to check if the restored
value is the default because it was not included in save state, you should
use an Option. If you wouldn't otherwise need a conditional, you should not
use an Option.
Define your saved state so that the natural default of bool types is
false.

Defining Saved State

Saved state is defined as a struct that has #[derive(Protobuf)] and
#[mesh(package = "package_name")] attributes. Here is an example, taken from
the nvme_driver:

rust,ignore

pub mod save_restore {
    use super::*;

    /// Save/restore state for IoQueue.
    #[derive(Protobuf, Clone, Debug)]
    #[mesh(package = "nvme_driver")]
    pub struct IoQueueSavedState {
        #[mesh(1)]
        /// Which CPU handles requests.
        pub cpu: u32,
        #[mesh(2)]
        /// Interrupt vector (MSI-X)
        pub iv: u32,
        #[mesh(3)]
        pub queue_data: QueuePairSavedState,
    }
}

[^1]: Saved state is in use when it reaches a release branch that is in tell
mode. See release management for details.

Guide Src Dev Guide Contrib Security Releases (`Guide/src/dev_guide/contrib/security_releases.md`)

Security Releases

This page describes how OpenVMM coordinates fixes that cannot be developed in
public before disclosure.

Report vulnerabilities privately

Report potential vulnerabilities through the
Microsoft Security Response Center. Do not open a
public issue, pull request, or discussion for a suspected vulnerability.

MSRC coordinates severity assessment, CVE handling, disclosure timing, and
communication with affected parties. Not every security fix requires an
embargo. When public development would not expose users to additional risk, the
maintainers may use the normal public contribution process.

Embargoed fix development

When an embargo is warranted, a small need-to-know incident team develops and
reviews the fix in a restricted workspace containing the public OpenVMM source
and the embargoed changes. The workspace must not introduce proprietary source
or dependencies into the eventual public change.

The team prepares the public patch, tests, release notes, advisory, and CVE
material before the coordinated disclosure time. Embargoes should be no longer
than necessary to produce and validate fixes and give affected downstreams a
reasonable opportunity to prepare.

Coordinating with downstreams

MSRC may authorize advance sharing with qualified downstream security teams
when they need time to validate or integrate a fix. Advance sharing is
case-by-case, not a general early-access program.

An advance package should identify the exact public base commit and affected
versions, include a reviewable patch series and its hashes, and provide build,
test, disclosure, and handling instructions. Recipients must keep the material
need-to-know, use approved secure channels, and must not redistribute the
patches or patched artifacts before disclosure unless MSRC explicitly
authorizes it.

An operator may be authorized to deploy early to a hosted service when users
cannot retrieve the patched software or infer the vulnerability from the
deployment. This also requires explicit incident-specific approval.

admonish warning

Receiving an embargoed patch does not grant permission to publish it, ship a
patched product, or disclose the vulnerability before the coordinated time.
The instructions supplied with the incident are authoritative.

Coordinated disclosure

At the agreed disclosure time, the incident team coordinates:

Publishing the fix to each affected public branch.
Publishing a new patch-version OpenVMM source release.
Publishing the security advisory and CVE information.
Notifying coordinated downstreams that the embargo has lifted.

The source release, advisory, and fix should become public in the same release
window. Detailed exploit material may be delayed when MSRC determines that
doing so reduces risk without preventing users from understanding and applying
the fix.

Support for older OpenVMM releases is decided per incident. A fix normally
targets supported releases; an affected unsupported release does not
automatically become supported.

After disclosure

After the public fix is available, development and review continue through the
normal public process. Embargo recipients should replace private patches with
the public fix when practical and securely delete pre-disclosure material when
their incident instructions require it.

Guide Src Dev Guide Contrib Style Guide (`Guide/src/dev_guide/contrib/style_guide.md`)

Style Guide

This page defines the writing conventions for the OpenVMM developer guide.
Follow these standards when creating or editing any page.

Voice and tone

Write as if explaining to a smart colleague who is new to the project.
Use second person ("you") for instructions: "To build OpenHCL, run..." not
"One can build OpenHCL by running..." State facts. Avoid hedging ("I believe",
"some reason", "maybe we can"). If something is uncertain, say "As of
[date], verify that..."

Page structure

Open every page with a one-sentence summary of what it covers. Use heading
levels consistently: # Page Title, ## Major Section, ### Subsection.
Keep pages under 400 lines. If a page grows beyond that, consider splitting
it into sub-pages.

Callouts

This guide uses
mdbook-admonish for
callouts. Use fenced code block syntax with the admonish keyword:

markdown

```admonish note
Supplementary information the reader should be aware of.
```

```admonish tip
Helpful suggestions or shortcuts.
```

```admonish warning
Potential pitfalls or easy mistakes.
```

```admonish danger
Actions that could cause data loss or security issues.
```

You can add a custom title with the title attribute:

markdown

```admonish warning title="Breaking change"
This API was removed in v1.8.
```

For cross-links to related pages, use a See also callout:

markdown

```admonish note title="See also"
[Debugging OpenHCL](../../reference/openhcl/debugging.md) for serial logs,
crash dumps, and diagnostic tools.
```

Code blocks

Shell labeling

Always specify the language. Never leave a code fence unlabeled.

Label	When
`bash`	Linux / WSL / macOS shell commands
`powershell`	PowerShell commands and scripts
`rust`	Rust source code
`toml`	TOML configuration
`json`	JSON data
`text`	Plain text output, logs, or paths

Placeholders

Two conventions are used for placeholder values. Use whichever is clearest
in context:

path/to/... style — for file paths where the structure is obvious.
This is the predominant style in the existing guide.
bash
```
cargo run -- --uefi --vmbus-scsi id=scsi0 --disk memdiff:file:path/to/disk.vhdx,on=scsi0
```
<SCREAMING_SNAKE_CASE> style — for values that need extra emphasis or
where the expected format isn't obvious (build numbers, VM names, keys).
powershell
```
Set-VMComPort -VMName <VM_NAME> -Number 3 -Path \\.\pipe\<PIPE_NAME>
```

Bad (hardcoded path that won't work for the reader):

bash

cargo run -- --uefi --vmbus-scsi id=scsi0 --disk memdiff:file:/home/alice/disks/myvm.vhdx,on=scsi0

File paths in code examples

Use forward slashes (Unix-style) for file paths in bash code blocks,
even when the path refers to a Windows filesystem location accessed via WSL:

bash

# Good — forward slashes in bash
cargo run -- --vmbus-scsi id=scsi0 --disk memdiff:file:path/to/disk.vhdx,on=scsi0

# Good — wslpath output uses backslashes, but that's a Windows path
cargo run -- --vmbus-scsi id=scsi0 --disk "memdiff:file:$(wslpath -w /mnt/c/vhds/disk.vhdx),on=scsi0"

Use backslashes for paths in powershell code blocks:

powershell

# Good — backslashes in PowerShell
cargo run -- --vmbus-scsi id=scsi0 --disk memdiff:file:C:\vhds\disk.vhdx,on=scsi0

Length

Keep code blocks under 30 lines. If longer, split with explanatory text
between blocks. Diagrams (ASCII art in text fences) are exempt from
this limit — keep them as a single block so the visual structure isn't
broken. Comments inside code blocks should explain why, not what.

Line wrapping

Wrap prose lines at approximately 80 characters. This keeps diffs
readable and makes review easier. Lines inside tables and code blocks
are exempt.

Flowey

Flowey is an in-house, Rust library for writing maintainable, cross-platform automation. It enables developers to define CI/CD pipelines and local workflows as type-safe Rust code that can generate backend-specific YAML (Azure DevOps, GitHub Actions) or execute directly on a local machine. Rather than writing automation logic in YAML with implicit dependencies, flowey treats automation as first-class Rust code with explicit, typed dependencies tracked through a directed acyclic graph (DAG).

Why Flowey?

Traditional CI/CD pipelines using YAML-based configuration (e.g., Azure DevOps Pipelines, GitHub Actions workflows) have several fundamental limitations that become increasingly problematic as projects grow in complexity:

The Problems with Traditional YAML Pipelines

Non-Local Reasoning and Global State

YAML pipelines heavily rely on global state and implicit dependencies (environment variables, file system state, installed tools)
Understanding what a step does often requires mentally tracking state mutations across the entire pipeline
Debugging requires reasoning about the entire pipeline context rather than isolated units of work
Changes in one part of the pipeline can have unexpected effects in distant, seemingly unrelated parts

Maintainability Challenges

YAML lacks type safety, making it easy to introduce subtle bugs (typos in variable names, incorrect data types, etc.)
No compile-time validation means errors only surface at runtime
Refactoring is risky and error-prone without automated tools to catch breaking changes
Code duplication is common because YAML lacks good abstraction mechanisms
Testing pipeline logic requires actually running the pipeline, making iteration slow and expensive

Platform Lock-In

Pipelines are tightly coupled to their specific CI backend (ADO, GitHub Actions, etc.)
Multi-platform support means maintaining multiple, divergent YAML files

Local Development Gaps

Developers can't easily test pipeline changes before pushing to CI
Reproducing CI failures locally is difficult or impossible
The feedback loop is slow: push → wait for CI → debug → repeat

Flowey's Solution

Flowey addresses these issues by treating automation as first-class Rust code:

Type Safety: Rust's type system catches errors at compile-time rather than runtime
Local Reasoning: Dependencies are explicit through typed variables, not implicit through global state
Portability: Write once, generate YAML for any backend (ADO, GitHub Actions, or run locally)
Reusability: Nodes are composable building blocks that can be shared across pipelines

Flowey's Directory Structure

Flowey is architected as a standalone tool with a layered crate structure that separates project-agnostic core functionality from project-specific implementations:

flowey_core: Provides the core types and traits shared between user-facing and internal Flowey code, such as the essential abstractions for nodes and pipelines.
flowey: Thin wrapper around flowey_core that exposes the public API for defining nodes and pipelines.
flowey_cli: Command-line interface for running flowey - handles YAML generation, local execution, and pipeline orchestration.
schema_ado_yaml: Rust types for Azure DevOps YAML schemas used during pipeline generation.
flowey_lib_common: Ecosystem-wide reusable nodes (installing Rust, running Cargo, downloading tools, etc.) that could be useful across projects outside of OpenVMM.
flowey_lib_hvlite: OpenVMM-specific nodes and workflows that build on the common library primitives.
flowey_hvlite: The OpenVMM pipeline definitions that compose nodes from the libraries above into complete CI/CD workflows.

Guide Src Dev Guide Dev Tools Guest Test Uefi (`Guide/src/dev_guide/dev_tools/guest_test_uefi.md`)

guest_test_uefi

guest_test_uefi is a minimal no_std + alloc UEFI application for
OpenVMM-specific bare-metal guest tests.

The application runs without a general-purpose guest OS, so tests can exercise
firmware variables, watchdog behavior, MMIO, and port I/O with little
intervening software.

Runtime and action selection

Firmware launches the EFI executable from a boot disk. On startup, the
application reads the PetriBootAction UEFI variable from the global variable
namespace.

The current actions are:

Value	Behavior
Missing, unreadable, or unknown	Run the normal UEFI test suite
`hibernate`	Request platform hibernation and do not return

Petri seeds this variable through the VM's UEFI NVRAM state when a test needs a
non-default action. The default makes a manually booted image run its ordinary
test suite without additional configuration.

Current tests

The default suite runs tests in a fixed order:

Allocate and print a heap-backed string, validating global allocation.
Read the Secure Boot db and dbDefault variables and compare their data.
Arm the UEFI watchdog for five seconds and wait long enough for it to reset
the VM.

The watchdog test runs last because success destroys the current VM execution.
A write-protection test for dbDefault exists but is disabled until the
firmware behavior it depends on is available.

The application draws changing splash values between tests and writes progress
to the UEFI console. Capture both console and VM lifecycle events when
diagnosing a failure.

Building and running

guest_test_uefi must be built for *-unknown-uefi targets. These are not
installed by default, so you'll need to install the correct target via rustup.
For example:

bash

rustup target add x86_64-unknown-uefi

Since this code runs in the guest, package the built .efi binary into a disk
image that UEFI can read.

To streamline the process of obtaining such a disk image, cargo xtask includes
a helper to generate properly formatted .img files containing a given .efi
image. e.g:

bash

# build the UEFI test application
cargo build -p guest_test_uefi --target x86_64-unknown-uefi
# create the disk image
cargo xtask guest-test uefi \
  --bootx64 target/x86_64-unknown-uefi/debug/guest_test_uefi.efi
# test in OpenVMM
cargo run -- --uefi --gfx --hv --processors 1 \
  --vmbus-scsi id=scsi0 \
  --disk memdiff:target/x86_64-unknown-uefi/debug/guest_test_uefi.img,on=scsi0

The output is a generic UEFI boot image and can also run under Hyper-V, QEMU,
or another VMM with compatible UEFI firmware.

admonish note

When launching a prebuilt `openvmm` executable instead of `cargo run`, provide
the UEFI firmware path explicitly. The Cargo environment normally supplies it
from restored packages.

Selecting hibernation in Petri

Petri tests should use the guest-action seeding helper rather than manually
editing the VMGS. The helper writes PetriBootAction=hibernate into the custom
UEFI NVRAM delta before boot and then observes the VM's hibernation behavior.

Manual launches run the default suite unless you prepare equivalent NVRAM
state.

Converting the image

To convert the raw .img into other formats, qemu-img is very helpful:

bash

# Adjust for the target architecture and build profile.
IMAGE_DIR=target/x86_64-unknown-uefi/debug
# VMware
qemu-img convert -f raw -O vmdk \
  ${IMAGE_DIR}/guest_test_uefi.img ${IMAGE_DIR}/guest_test_uefi.vmdk
# Hyper-V
qemu-img convert -f raw -O vhdx \
  ${IMAGE_DIR}/guest_test_uefi.img ${IMAGE_DIR}/guest_test_uefi.vhdx

Troubleshooting

If firmware does not find the application, verify the target architecture
and the --bootx64 or corresponding image-builder input.
If db or dbDefault is missing, use firmware containing the expected
Secure Boot variable defaults.
If the watchdog test returns and panics, the firmware did not honor the
watchdog reset request.
If Petri performs the wrong action, inspect the custom UEFI NVRAM delta and
confirm the variable name, namespace, and UTF-8 value.
If no output appears, attach a graphical or serial console supported by the
selected firmware and inspect early firmware boot logs.

Guide Src Dev Guide Dev Tools Hypestv (`Guide/src/dev_guide/dev_tools/hypestv.md`)

hypestv

hypestv is an interactive command-line interface for Hyper-V VMs, designed for
making OpenHCL developers' lives easier.

Similar to ohcldiag-dev, it can interact with the OpenHCL paravisor
running inside a Hyper-V VM. But unlike ohcldiag-dev, it sports an interactive
terminal interface (with history and tab completion), and it is specifically
designed to interact with Hyper-VMs.

Platform and build

Hypestv runs on Windows and controls VMs registered with the local Hyper-V
service. It is not a remote management client and has no non-Windows support.

Build it from a Windows checkout or a configured WSL cross environment:

powershell

cargo build -p hypestv

Start detached or select an initial VM by name:

powershell

.\target\debug\hypestv.exe
.\target\debug\hypestv.exe <VM_NAME>

The interface is deliberately interactive, it is not intended for use in automation.

In many ways, it is similar to the OpenVMM interactive console. In time, it may
end up sharing code and capabilities with it and with ohcldiag-dev, but it
will always be a Hyper-V specific tool.

Currently, it can:

Change VM state (starting/stopping/resetting)
Enable serial port output to standard output, or input/output to another
terminal window
Enable paravisor log output to standard output or another terminal window
Inspect paravisor state
Modify the paravisor (update the paravisor command line, reload the paravisor)

In the future, it might be able to:

Enable Hyper-V log output
Capture serial port output to a file
Inspect host state
Persistence workspaces (save/restore configured serial ports and logs)

Example session

hypestv launches into a detached mode, unless you specify a VM name on the
command line. To select a VM to work on, the VM named tdxvm in this example,
use the select command. If successful, you will now see the name and VM state
in the prompt:

text

> select tdxvm
tdxvm [off]>

After this, all commands will implicitly operate on tdxvm. Use select again
to work on another VM.

To enable serial port output, use the serial command. This can be used at any
time, even while the VM is not running. E.g., to open a separate window for
interactive use of COM1 and enable logging serial port output for COM2:

text

tdxvm [off]> serial 1 term
tdxvm [off]> serial 2 log

You can also enable paravisor log output at any time:

text

tdxvm [off]> paravisor kmsg log

Start a VM with start. This is an asynchronous command: you can continue to
type other commands at the prompt while the VM starts. You should see an output
message when the VM finishes starting, as well as output about any configured
serial ports connecting.

Note that, due to limitations of the rustyline crate, the displayed VM state
on the prompt may not be accurate until you type another command or press Enter.

text

tdxvm [off]> start
com1 connected
com2 connected
VM started
tdxvm [off]>
tdxvm [running]>

At this point, the VM is running, including the paravisor (if one is
configured). As in the OpenVMM interactive console, you can inspect paravisor
state with the inspect or x command, but under the paravisor/pv command:

text

tdxvm [running]> pv x
{
    build_info: _,
    control_state: "started",
    mesh: _,
    proc: _,
    trace: _,
    uhdiag: _,
    vm: _,
}

You can terminate the VM with kill. This will disconnect any connected serial
ports as well, but they will reconnect next time the VM starts. Killing a VM
does not detach/deselect it; subsequent commands will continue to operate on the
VM.

text

tdxvm [running]> kill
com1 disconnected
com2 disconnected
VM killed
tdxvm [stopping]>
tdxvm [off]>

Asynchronous event model

Power operations and endpoint connections can complete after the prompt is
redrawn. Hypestv prints completion, disconnection, and failure events as they
arrive. The prompt's state is therefore a recent observation rather than a
transactional lock on Hyper-V state.

Press Enter to refresh the prompt after an asynchronous transition. Avoid
starting a conflicting operation while the VM is still starting, stopping, or
reloading.

Troubleshooting

If select fails, use list and match the registered VM name exactly.
If a WMI kill fails while Hyper-V believes the VM is transitioning, retry
with kill --force only when abrupt power loss is acceptable.
If guest shutdown has no effect, verify that the guest shutdown service and
integration path are running.
If pv commands cannot connect, confirm that the VM contains OpenHCL and that
the paravisor reached its diagnostic service.
For a failed reload, enable pv kmsg log and the relevant serial ports before
trying again so early output is retained.
If displayed state appears stale, wait for the asynchronous completion event
and press Enter.

Guide Src Dev Guide Dev Tools Igvmfilegen (`Guide/src/dev_guide/dev_tools/igvmfilegen.md`)

igvmfilegen

igvmfilegen constructs, measures, inspects, and signs metadata for Isolated
Guest Virtual Machine (IGVM) files.

Role in an OpenHCL build

An OpenHCL IGVM combines several independently built inputs, including the boot
loader, Linux kernel, initrd, openvmm_hcl, optional sidecar, and guest boot
resources. A manifest describes how to place those resources and which VBS,
SEV-SNP, or Intel TDX platform definitions the file supports.

igvmfilegen turns that description into the final binary layout and computes
the platform launch measurements and endorsement material associated with it.

Use cargo xflowey build-igvm <RECIPE> for normal OpenHCL builds. Flowey
selects a supported manifest and produces the resource file from concrete build
artifacts. Invoke igvmfilegen directly when developing manifests, inspecting
an image, or completing a signing workflow.

Build

Build the host utility with:

bash

cargo build -p igvmfilegen

Display the current top-level interface with:

bash

cargo run -p igvmfilegen -- --help

Build from a manifest

The manifest command consumes two JSON files:

The manifest describes images, platform policies, memory layout, command
lines, and the relationship between components.
The resources file maps symbolic manifest resources to concrete files.

Build an image with:

bash

cargo run -p igvmfilegen -- manifest \
  --manifest path/to/manifest.json \
  --resources path/to/resources.json \
  --output path/to/openhcl.bin

--debug-validation enables additional build-time checks.
--disable-secure-avic overrides secure AVIC for supported debug SNP
configurations.

admonish danger

`--confidential-debug` changes the measured OpenHCL command line so a
confidential guest permits diagnostics and trusts host-provided debug options.
Use it only for an intentionally debuggable development image. It weakens the
security assumptions expected from a production confidential image.

For each measurable platform, the command also writes sibling identity and
endorsement files next to the output:

text

<BASE>-snp.json     <BASE>-snp.cbor
<BASE>-tdx.json     <BASE>-tdx.cbor
<BASE>-vbs.json     <BASE>-vbs.cbor
<BASE>-snp.idblock

Only files corresponding to platforms present in the manifest are emitted.
The .idblock file is the SNP signing payload used by the production signing
mode described below.

Inspect an IGVM

Dump headers and directives in a human-readable form:

bash

cargo run -p igvmfilegen -- dump \
  --filepath path/to/openhcl.bin

The input may be a raw IGVM or a supported vmfirmwareigvm resource DLL. For a
DLL, the embedded IGVM is selected automatically.

Inspect CoRIM entries and optionally extract their payloads:

bash

cargo run -p igvmfilegen -- dump-corim \
  --filepath path/to/openhcl.bin \
  --platform snp \
  --output path/to/corim-output

--header-type document|signature narrows the result. Without filters, the
command reports every supported CoRIM entry in the file.

Add an SNP ID block

An SNP ID block binds an identity key and guest SVN to the image launch
measurement. The IGVM must already contain a compatible SNP platform and guest
policy, and a file that already has an ID block is rejected.

For development, generate an ephemeral P-384 key and specify an SVN:

bash

cargo run -p igvmfilegen -- add-snp-id-block \
  --input path/to/input.bin \
  --output path/to/output.bin \
  --guest-svn <GUEST_SVN>

Alternatively, --manifest path/to/manifest.json sources the guest SVN and
image identity from the manifest's SNP configuration.

Production signing keeps the private key outside this process. Sign the exact
.idblock payload emitted by manifest, then attach the payload, DER ECDSA
signature, and signer public key or certificate:

bash

cargo run -p igvmfilegen -- add-snp-id-block \
  --input path/to/input.bin \
  --output path/to/output.bin \
  --id-block path/to/image-snp.idblock \
  --id-signature path/to/signature.der \
  --id-public-key path/to/public-key.pem

The output path may equal the input path, but a separate output is easier to
recover and compare during development.

Patch a CoRIM signature

manifest writes the CoRIM document into the IGVM. After an external signer
produces a signed bundle or detached signature, attach it for one platform:

bash

cargo run -p igvmfilegen -- patch-corim-signature \
  --input path/to/input.bin \
  --output path/to/output.bin \
  --platform snp \
  --corim-bundle path/to/signed-corim.cose

Use --corim-signature instead when the input is already a detached COSE_Sign1
signature.

The tool verifies a well-formed PS384 envelope and verifies its signature over
the IGVM-embedded document using the key carried in x5chain or x5bag.

admonish warning title="signature trust"

This verification checks signature integrity and algorithm compatibility. It
does not establish certificate-chain trust, check revocation, or enforce
certificate policy. The caller must obtain the signature from a trusted signing
pipeline and validate that pipeline's identity separately.

Troubleshooting

A missing resource names the symbolic manifest entry that could not be
resolved; check the resources JSON and the referenced build output.
A platform-policy error usually indicates incompatible manifest directives,
guest policy, architecture, or isolation type.
A measurement mismatch should be investigated as a layout or input change,
not bypassed.
ID-block errors can indicate a signing payload from a different image, an
unsupported key, or an IGVM that already contains a block.
CoRIM errors distinguish document mismatch, unsupported algorithms, malformed
COSE, and failed signature math. None of these establish signer trust.

Guide Src Dev Guide Dev Tools Vmgstool (`Guide/src/dev_guide/dev_tools/vmgstool.md`)

VmgsTool

vmgstool creates, inspects, and modifies version 3 VM Guest State (VMGS)
files for provisioning and debugging.

OpenHCL uses VMGS v3 on both Hyper-V and OpenVMM. OpenVMM also uses VMGS v3 for
VMs without OpenHCL. Hyper-V VMs without an HCL use a different state format.

VMGS persists firmware variables and security state on behalf of a VM. The
store is packaged as a VHD and contains numbered logical files. These are data
slots rather than host filesystem files; for example, BIOS NVRAM and vTPM
state occupy separate well-known file IDs.

Confidential VMs can encrypt selected VMGS contents before deployment so the
host handles encrypted state rather than plaintext guest secrets.

Data model

A VMGS store contains redundant headers, a file table, and allocated data
blocks. Commands identify logical data through either a numeric file ID or a
known symbolic name. See vmgs_format::FileId for the authoritative list
of IDs and names.

Use dump-file-table to discover which IDs are allocated before modifying a
store:

powershell

vmgstool.exe dump-file-table --file-path path\to\guest.vmgs

admonish note

`dump-file-table` can inspect an encrypted store without its key. However, its
per-file encryption information can be inaccurate when the store was last
modified by a pre-1.8 version of OpenHCL, VmgsTool, or OpenVMM.

Alternatively: Pre-Built Binaries

If you would prefer to use VmgsTool without building it from scratch, you can
download pre-built copies of the binary from
OpenVMM CI.

Simply select a successful pipeline run (should have a Green checkbox), and
scroll down to select an appropriate *-vmgstool artifact for your particular
architecture and operating system.

Running

admonish tip

Note: The examples in this section use the Windows executable `vmgstool.exe`,
which can be replaced with the Linux executable `vmgstool`.

Developers who have already setup their development environment may also use
the appropriate `cargo run` command. For more details on building,

see the [build](#building) section below.

VmgsTool commands continue to evolve, so use vmgstool.exe --help for the
current interface. Every command and nested operation also has help:

powershell

vmgstool.exe uefi-nvram dump --help

Except where explicitly documented, stdout and stderr are for humans and are
not stable automation formats.

Read and Write Raw Data

To read raw data from a VMGS file, use the dump command. For example, to
export the decrypted binary contents of the BIOS_NVRAM (--fileid 1) to a file:

powershell

vmgstool.exe dump --filepath path\to\guest.vmgs `
    --keypath path\to\key.bin --datapath path\to\nvram.bin --fileid 1

To write raw data to a VMGS file, use the write command. For example, to write
those NVRAM variables to a different, unencrypted VMGS file:

powershell

vmgstool.exe write --filepath path\to\guest.vmgs `
    --datapath path\to\nvram.bin --fileid 1

By default, write refuses to replace a nonempty slot. Add
--allow-overwrite only after confirming the destination ID.

If dump has no --data-path, it writes an ASCII hexadecimal representation
to stdout. --raw-stdout selects raw bytes and cannot be combined with an
output path.

Read and Parse UEFI NVRAM Variables

Furthermore, VmgsTool contains parsers to help debug UEFI NVRAM variables in
VMGS file ID 1 (BIOS_NVRAM). To dump the variables from an encrypted VMGS and
truncate binary data without a parser:

powershell

vmgstool.exe uefi-nvram dump --filepath path\to\guest.vmgs `
    --keypath path\to\key.bin --truncate

Read DLL File to Write IGVMfile to VMGS

VmgsTool can extract an IGVM from a resource DLL and write it to VMGS file ID 8
(GUEST_FIRMWARE). Select one of NONCONFIDENTIAL, SNP, TDX,
SNP_NO_HCL, TDX_NO_HCL, or OPENHCL:

powershell

vmgstool.exe copy-igvmfile --filepath path\to\guest.vmgs `
    --datapath path\to\vmfirmwareigvm.dll --resource-code SNP

Delete Boot Variables to Recover a VM that Fails to Boot

A VM may fail to boot if the disk configuration changes and
UEFI's DefaultBootAlwaysAttempt setting is disabled.
Deleting the existing (invalid) boot entries using VmgsTool
will trigger a default boot (which attempts to boot all available partitions and devices).

To print the boot entries in an encrypted VMGS file:

powershell

vmgstool.exe uefi-nvram remove-boot-entries `
    --filepath path\to\guest.vmgs --keypath path\to\key.bin --dry-run

To actually remove the boot entries from the VMGS file, remove --dry-run.
This will remove all Boot#### variables and the BootOrder variable.

To remove a specific boot entry or another UEFI NVRAM variable, use
remove-entry. For example, to remove Boot0000:

powershell

vmgstool.exe uefi-nvram remove-entry --filepath path\to\guest.vmgs `
    --keypath path\to\key.bin --name Boot0000 `
    --vendor 8be4df61-93ca-11d2-aa0d-00e098032b8c

Always run remove-boot-entries --dry-run first. Removing all boot variables
causes firmware to fall back to default boot enumeration and can change which
disk starts.

Troubleshooting

Expected at least N more bytes, but only found M

If you get an error similar to the one below, it is likely that you are trying
to read an encrypted VMGS file and haven't provided the correct decryption key.

text

ERROR: remove_boot_entries error
Caused by:
    0: error loading data from Nvram storage
    1: unexpected EOF. expected at least 2330702412 more bytes, but only found 30067

Building

Prior to building VmgsTool, please ensure you have built either OpenVMM or
OpenHCL at least once, to ensure you have all necessary build dependencies
installed.

VmgsTool can be built with cargo build -p vmgstool for Windows and Linux.
To interact with encrypted VMGS files, you will need to compile with the
encryption feature: cargo build --features encryption -p vmgstool

Guide Src Dev Guide Dev Tools Xflowey (`Guide/src/dev_guide/dev_tools/xflowey.md`)

cargo xflowey

cargo xflowey runs OpenVMM's local Flowey pipelines for builds, dependency
restoration, test execution, and generated CI workflows.

Pipeline overview

Important local pipelines include:

Pipeline	Purpose
`restore-packages`	Download and unpack external build dependencies
`build-igvm`	Build a supported or customized OpenHCL IGVM image
`vmm-tests-run`	Discover artifacts, build them, and run selected VMM tests
`regen`	Regenerate checked-in CI definitions from Flowey pipelines

List the current surface with:

bash

cargo xflowey --help

How a pipeline runs

Flowey separates planning from execution. A pipeline declares jobs and their
node dependencies, and requests typed variables or artifacts from those nodes.
The runtime resolves the resulting graph, schedules ready steps, and records
enough state to avoid repeating valid work.

Local runs commonly write to:

flowey-out for requested artifacts and user-facing output.
flowey-persist for cached node state and downloaded inputs.
target for Cargo build products and test-specific scratch roots.

Restoring packages

Restore external dependencies needed by common builds:

bash

cargo xflowey restore-packages

This can include firmware, kernels, sysroots, protocol compilers, and other
versioned packages. The exact set is selected from the build configuration and
host platform.

Some pipelines support --install-missing-deps, allowing Flowey to install
reported host prerequisites. Without it, the failure should explain the manual
installation needed.

Building OpenHCL

Build a standard x64 image:

bash

cargo xflowey build-igvm x64

Select release output or another supported recipe:

bash

cargo xflowey build-igvm x64-cvm --release

The pipeline builds and stages openvmm_hcl, the boot loader, sidecar, kernel,
initrd, manifest resources, and final IGVM. Output is placed below
flowey-out/artifacts/build-igvm by recipe and build profile.

admonish warning

The local `build-igvm` CLI is a developer interface and does not promise stable
automation syntax. Checked-in automation should depend on Flowey nodes and
typed artifacts rather than shelling out to this local command.

Running VMM tests

Run one test or group with a nextest filter expression:

bash

cargo xflowey vmm-tests-run --filter "test(my_test)"

The pipeline first enumerates matching tests and their static artifact
requirements. It then selects builds and downloads, initializes a test-content
directory, performs required image preparation, and invokes nextest with the
resolved environment.

Use --build-only to produce the selected artifacts without executing tests.
Cross-compilation is selected through --target and related
options described in the VMM-test guide.

Regenerating CI

Flowey pipeline definitions are the source of truth for generated CI YAML.
After changing those definitions, regenerate them with:

bash

cargo xflowey regen

Do not hand-edit generated files under ci-flowey or generated workflow paths.
The verify-flowey xtask pass checks whether regeneration would change them.

The same graph definitions used for local pipelines are adapted to CI backends.
CI adds pools, triggers, permissions, and published artifacts around the shared
nodes rather than maintaining an unrelated shell implementation.

`xflowey` vs `xtask`

Use the ownership boundary:

cargo xtask: implements novel, standalone tools/utilities
cargo xflowey: orchestrates invoking a sequence of tools/utilities, without
doing any non-trivial data processing itself

Troubleshooting

Read the first failed node's context rather than only the final pipeline
summary; later requested artifacts often fail because of that dependency.
Use the pipeline's --help to find supported verbosity and dependency
installation options.
A stale or corrupt cached download can be isolated by cleaning the specific
state reported in flowey-persist; avoid deleting every package by default.
When cross-compiling from WSL, source the Windows cross environment before
invoking a Windows target.
If generated CI verification fails, run cargo xflowey regen and review the
generated diff rather than editing YAML manually.

Guide Src Dev Guide Dev Tools Xtask (`Guide/src/dev_guide/dev_tools/xtask.md`)

cargo xtask

cargo xtask runs standalone, repository-specific development utilities
implemented by the xtask binary.

Invocation

Run commands from the repository root so task-relative paths resolve
consistently:

bash

cargo xtask --help

Command overview

Command	Purpose
`clean`	Remove generated build-system, package, and test artifacts
`fmt`	Run repository formatting and structural validation passes
`fuzz`	Discover, build, execute, minimize, and inspect fuzz targets
`guest-test`	Create or download guest test images
`install-git-hooks`	Install repository-managed git hooks
`verify-size`	Compare ELF section sizes between two binaries

Run cargo xtask <COMMAND> --help for the current command-specific interface.

Cleaning generated state

The default clean removes directories that cargo clean does not own:

flowey-out
flowey-persist
.packages
vmm_test_results

Preview the removal first:

bash

cargo xtask clean --dry-run

Add --cargo to run cargo clean and remove target as well.

admonish warning

The default clean removes restored packages and test results in addition to
generated Flowey state. Use `--dry-run` when you only intend to reclaim Cargo
output or need to retain diagnostic artifacts.

Repository formatting

cargo xtask fmt is the canonical repository formatter and validator. It runs
four pass families:

Pass	Coverage
`rustfmt`	Rust formatting and generated formatting expectations
`lints`	Repository source, metadata, documentation, and policy lints
`verify-fuzzers`	Fuzz-target manifest and registration consistency
`verify-flowey`	Flowey pipeline and generated-definition consistency

Run every pass and apply supported fixes:

bash

cargo xtask fmt --fix

Run only selected passes against changed files:

bash

cargo xtask fmt --only-diffed \
	--pass rustfmt --pass lints

Without --fix, independent passes run in parallel unless --no-parallel is
set. A failed pass prints the corresponding fixing command where applicable.

Fuzzing commands

The fuzz command wraps cargo-fuzz with workspace target discovery and
OpenVMM-specific build configuration.

Start by listing registered targets:

bash

cargo xtask fuzz list

Run a target or reproduce one artifact:

bash

cargo xtask fuzz run fuzz_ide
cargo xtask fuzz run fuzz_ide path/to/crash-artifact

The fuzzing section documents prerequisites, corpora, minimization, and
coverage workflows in depth.

Guest test images

guest-test uefi places a UEFI executable into a bootable raw image. For
example:

bash

cargo xtask guest-test uefi \
	--bootx64 path/to/guest_test_uefi.efi

guest-test download-image obtains the repository's supported guest test
image. Use each subcommand's help for architecture and output options.

Git hooks

Install the repository-managed hooks with:

bash

cargo xtask install-git-hooks

The installed hook locates the built automation binary through
target/xtask-path, avoiding an unnecessary Cargo invocation when possible.
Hook behavior still comes from the current checkout, so rerun installation if
the hook setup changes.

Fix a hook-reported issue with the same cargo xtask command shown in its
output. Do not bypass a failing hook without understanding which required check
is being skipped.

Binary size comparison

Compare section sizes of two ELF binaries:

bash

cargo xtask verify-size \
	--original path/to/original \
	--new path/to/new

The command reports changed sections, net size difference, and total absolute
section difference. It fails when the total difference exceeds its current
50-KiB threshold. This is a coarse guard for unexpected layout changes, not a
replacement for scenario-specific image size limits.

`xtask` versus `xflowey`

Use xtask for a tool that performs project-specific data processing or a
focused repository operation. Use xflowey to orchestrate a dependency graph
of builds, downloads, and existing tools across platforms.

Examples:

Formatting Rust and validating fuzzer manifests belong in xtask.
Building OpenHCL from many components and resolving VMM-test artifacts belong
in xflowey.

Troubleshooting

Run from the repository root unless using --custom-root.
If an automation change is not reflected, allow Cargo to rebuild the light
profile instead of invoking a stale path directly.
On Windows, locked files can prevent cleanup; stop VMs, debuggers, and helper
processes before retrying.
Use --no-parallel when interleaved formatter output obscures the failing
pass.
A missing cargo-fuzz or target toolchain is a fuzzing prerequisite error,
not an xtask registration failure.

The command follows the general
cargo-xtask convention.

Guide Src Dev Guide Getting Started Build Ohcl Kernel (`Guide/src/dev_guide/getting_started/build_ohcl_kernel.md`)

Building a custom OpenHCL Linux Kernel

admonish note

This step is NOT required in order to build OpenHCL!

Unless you have a specific reason, it is _strongly_ recommended to stick to
the pre-built Kernel image which is automatically downloaded as part of the
OpenHCL build process.

Cloning the kernel repository

If you need to rebuild the kernel, the sources are available in the OHCL-Linux-Kernel repo :

the main branch is product/hcl-main/6.6,
the dev branch is project/hcl-dev/6.6.

Unless you need the entire repo history, cloning just one branch and depth = 1, saves
significant time and disk space:

git clone https://github.com/microsoft/OHCL-Linux-Kernel.git -b product/hcl-main/6.6 --depth=1

Cloning under Windows is likely to fail due to some files using names that Windows reserves, e.g. aux.c, and
NTFS being non case-sensitive by default as there are few files in the Linux kernel repo whose names differ
in their case only. To clone successfully under Windows, need a fix in ntdll (merged in Ni?),
and a case-sensitive NTFS partition. Best to start with the default WSL2 if there is no existing working setup.

Building the kernel locally

The following instructions for building the kernel locally target the Ubuntu distributions.
In order to build on rpm-based systems the only changes are likely to be the way the package
manager is invoked and installing the kernel-devel package instead of what is needed for
Ubuntu.

The paths below are relative to the cloned kernel repository root.

Do once for every machine that hasn't run this step successfully:

./Microsoft/install-deps.sh

If you plan to use your custom kernel to work with confidential VMs, you need to enable a few more options
in the kernel. Do once after cloning the kernel repository or every time you remove local changes from your
kernel configuration file:

./Microsoft/merge-cvm-config.sh

Every time the kernel needs to be rebuilt:

./Microsoft/build-hcl-kernel.sh

The output directory is ./out, it contains the kernel binary, the kernel modules,
and the debug symbols, and cargo xflowey build-igvm can be pointed to it to use
as a source of the kernel binary and the modules.

In the case you are iterating on a change, install ccache
for decreasing the kernel build time significantly, might be close to an order of
magnitude. To use the compiler cache, prepend CC="ccache gcc" to the build command.
To see if the cache integrates itself into the toolchain:

host:~$ which gcc

/usr/lib64/ccache/gcc

Guide Src Dev Guide Getting Started Build Openhcl (`Guide/src/dev_guide/getting_started/build_openhcl.md`)

Building OpenHCL

Prerequisites:

Getting started on Linux / WSL2.

Reminder: OpenHCL cannot currently be built on Windows hosts!

An OpenHCL IGVM firmware image is composed of several distinct binaries and
artifacts. For example: the openvmm_hcl usermode binary, the OpenHCL boot
shim, the OpenHCL Linux kernel and initrd, etc....

Some of these components are built directly out of the OpenVMM repo, whereas
others must be downloaded as pre-built artifacts from other associated repos.
Various tools and scripts will then transform, package, and re-package these
artifacts into a final OpenHCL IGVM firmware binary.

Fortunately, we don't expect you do to all those steps manually!

All the complexity of installing the correct system dependencies, building the
right binaries, downloading the right artifacts, etc... is neatly encapsulated
behind a single cargo xflowey build-igvm command, which orchestrates the
entire end-to-end OpenHCL build process.

Using the build-igvm flow is as simple as running:

bash

cargo xflowey build-igvm [RECIPE]

The first build will take some time as all the dependencies are
installed/downloaded/built.

Note: At this time, OpenHCL can only be built on Linux (or WSL2)!

A "recipe" corresponds to one of the pre-defined IGVM SKUs that are actively
supported and tested in OpenVMM's build infrastructure.

A single recipe encodes all the details of what goes into an individual IGVM
file, such as what build flags openvmm_hcl should be built with, what goes
into a VTL2 initrd, what igvmfilegen manifest is being used, etc...

e.g: x64, for a "standard" x64 IGVM
e.g: aarch64, for a "standard" aarch64 IGVM
e.g: x64-cvm, for a x64 CVM IGVM
e.g: x64-test-linux-direct, for x64 IGVM booting a test linux direct image
for a full list of available recipes, please run cargo xflowey build-igvm --help

New recipes can be added by modifying the build-igvm source code.

Build output is then binplaced to: flowey-out/artifacts/build-igvm/{release-mode}/{recipe}/openhcl-{recipe}.bin

So, for example:

bash

cargo xflowey build-igvm x64-cvm
# output: flowey-out/artifacts/build-igvm/debug/x64-cvm/openhcl-x64-cvm.bin

cargo xflowey build-igvm x64 --release
# output: flowey-out/artifacts/build-igvm/release/x64/openhcl-x64.bin

admonish warning

`cargo xflowey build-igvm` is designed to be used as part of the
developer inner-loop, and does _NOT_ have a stable CLI suitable for CI or any
other form of production automation!

In-tree pipelines and automation should interface with the underlying `flowey`
infrastructure that powers `cargo xflowey build-igvm`, _without_ relying on
the details of its CLI.

Building ohcldiag-dev

ohcldiag-dev is typically built as a Windows binary.

This can be done directly from Windows, or using
cross-compilation from WSL2.

The command to build ohcldiag-dev is simply:

# you may need to run `rustup target add x86_64-pc-windows-msvc` first
cargo build -p ohcldiag-dev --target x86_64-pc-windows-msvc

Note: Thanks to x86 emulation built into Windows, ohcldiag-dev.exe that is
built for x64 Windows will work on Aarch64 Windows as well.

Troubleshooting

This section documents some common errors you may encounter while building
OpenHCL.

If you are still running into issues, consider filing an issue on the OpenVMM
GitHub Issue tracker.

Help! The build failed due to a missing dependency

If you don't mind having xflowey install some dependencies globally
on your machine (i.e: via apt install, or rustup toolchain add),
you can pass --install-missing-deps to your invocation of
build-igvm:

bash

cargo xflowey build-igvm x64 --install-missing-deps

This will automatically install all required dependencies, including
the .NET SDK, Rust toolchains, Node.js, and any necessary system
packages.

Alternatively - build-igvm should emit useful human-readable
error messages when it encounters a dependency that isn't installed,
with a suggestion on how to install it.

If it doesn't - please file an Issue!

Help! Everything is rebuilding even though I only made a small change

Cargo's target triple handling can be a bit buggy. Try running with:

bash

CARGO_BUILD_TARGET=x86_64-unknown-linux-gnu cargo build-igvm [RECIPE]

or adding the below to your .bashrc:

bash

export CARGO_BUILD_TARGET=x86_64-unknown-linux-gnu

Build Customization

Aside from building IGVM files corresponding the the built-in IGVM recipes,
build-igvm also offers a plethora of customization options for developers who
wish to build specialized custom IGVM files for local testing.

Some examples of potentially useful customization include:

--override-manifest: Override the recipe's igvmfilegen manifest file
via, in order to tweak different kernel command line options, different VTL0
boot configuration, or different VTL2 memory sizes.

--custom-openvmm-hcl: Specify a pre-built openvmm_hcl binary. This is
useful in case you have already built it with some custom settings, e.g.:

bash

cargo build --target x86_64-unknown-linux-musl -p openvmm_hcl --features myfeature
cargo xflowey build-igvm x64 --custom-openvmm-hcl target/x86_64-unknown-linux-musl/debug/openvmm_hcl

Specify a custom VTL2 kernel vmlinux / Image, instead of using the
packaged main kernel.
bash
```
cargo xflowey build-igvm x64 --custom-kernel path/to/my/prebuilt/vmlinux
```
The packaged dev kernel variants are disabled by default. A custom kernel
remains available for local kernel development without enabling those
variants.

For a full list of available customizations, refer to build-igvm --help.

Advanced

Depending on what you're doing, you may need to build the individual components
that go into an OpenHCL IGVM build.

Our flowey-based pipelines handle the complexities of properly invoking and
orchestrating the various individual build tools / scripts used to construct
IGVM files, but a sufficiently motivated user can go through these steps
manually.

Please consult the source code for cargo xflowey build-igvm for a breakdown of
all build steps and available customization options.

Note that the canonical "source of truth" for how to build end-to-end OpenHCL
IGVM files are these build scripts themselves, and the specific flow is subject
to change over time!

Guide Src Dev Guide Getting Started Build Openvmm (`Guide/src/dev_guide/getting_started/build_openvmm.md`)

Building OpenVMM

Prerequisites:

One of:
- Getting started on Windows
- Getting started on Linux / WSL2.

admonish tip

It is strongly suggested that you use WSL2, and [cross compile](./cross_compile.md)
for Windows when necessary.

Build Dependencies

OpenVMM currently requires a handful of external dependencies to be present in
order to properly build / run. e.g: a copy of protoc to compile Protobuf
files, a copy of the mu_msvm UEFI firmware, some test linux kernels, etc...

Running the following command will fetch and unpack these various artifacts into
the correct locations within the repo:

cargo xflowey restore-packages

If you intend to cross-compile, refer to the command's --help for additional
options related to downloading packages for other architectures.

Building

OpenVMM uses the standard Rust build system, cargo.

To build OpenVMM, simply run:

cargo build

Note that certain features may require compiling with additional --feature
flags.

Troubleshooting

This section documents some common errors you may encounter while building
OpenVMM.

If you are still running into issues, consider filing an issue on the OpenVMM
GitHub Issue tracker.

failed to invoke protoc

Error:

text

error: failed to run custom build command for `inspect_proto v0.0.0 (/home/daprilik/src/openvmm/support/inspect_proto)`

Caused by:
  process didn't exit successfully: `/home/daprilik/src/openvmm/target/debug/build/inspect_proto-e959f9d63c672ccc/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at support/inspect_proto/build.rs:23:10:
  called `Result::unwrap()` on an `Err` value: Custom { kind: NotFound, error: "failed to invoke protoc (hint: https://docs.rs/prost-build/#sourcing-protoc): (path: \"/home/daprilik/src/openvmm/.packages/Google.Protobuf.Tools/tools/protoc\"): No such file or directory (os error 2)" }
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

Note: the specific package that throws this error may vary, and may not always be inspect_proto

Solution:

You attempted to build OpenVMM without first restoring necessary packages.

Please run cargo xflowey restore-packages, and try again.

use of unstable library feature

Error:

text

error[E0658]: use of unstable library feature 'absolute_path'
  --> flowey/flowey/src/lib.rs:37:17
   |
37 |                 std::path::absolute(self)
   |                 ^^^^^^^^^^^^^^^^^^^
   |
   = note: see issue #92750 <https://github.com/rust-lang/rust/issues/92750> for more information

For more information about this error, try `rustc --explain E0658`.
error: could not compile `flowey` (lib) due to previous error

Solution:

Install Rust using the official instructions for Linux or Windows.

Guide Src Dev Guide Getting Started Cross Compile (`Guide/src/dev_guide/getting_started/cross_compile.md`)

\[WSL2] Cross Compiling from WSL2 to Windows

Setting up cross compilation is very useful, as it allows using the same repo
cloned in WSL2 to both develop OpenHCL, as well as launch it via OpenVMM via the
WHP backend.

Required Dependencies

Note that this requires some additional dependencies, described below.

Windows deps

Visual Studio build tools must be installed, along with the Windows SDK.
This is the same as what's required to build OpenVMM on windows.

WSL deps

The msvc target x86_64-pc-windows-msvc must be installed for the toolchain
being used in WSL. This can be added by doing the following:

bash

rustup target add x86_64-pc-windows-msvc

Additional build tools must be installed as well. If your distro has LLVM 14
available (Ubuntu 22.04 or newer):

bash

sudo apt install clang-tools lld llvm-dev

Otherwise, follow the steps at https://apt.llvm.org/ to install a specific
version, by adding the correct apt repos. Note that you must install
clang-tools as default clang uses gcc style arguments, where
clang-cl uses msvc style arguments. You can use their helper script as
well:

bash

wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
sudo ./llvm.sh
sudo apt install clang-tools

Setting up the terminal environment

Source the build_support/setup_windows_cross.sh script from your terminal
instance. For example, the following script will do this along with setting a
default cargo build target:

bash

#!/bin/bash

# Setup environment and windows cross tooling.

export CARGO_BUILD_TARGET=x86_64-unknown-linux-gnu
cd path/to/openvmm || exit
. build_support/setup_windows_cross.sh
exec "$SHELL"

For developers using shells other than bash, you may need to run the
setup_windows_cross.sh script in bash then launch your shell in order to get
the correct environment variables.

Editing with vscode

You can have rust-analyzer target Windows, which will allow you to use the same
repo for OpenHCL, Linux, and Windows changes, but the vscode remote server
must be launched from the terminal window that sourced the setup script. You can
do this by closing all vscode windows then opening your workspace with
code in your terminal.

Add the following to your workspace settings for a vscode workspace
dedicated to Windows:

json

"settings": {
    "rust-analyzer.cargo.target": "x86_64-pc-windows-msvc"
}

Running Windows OpenVMM from within WSL

You can build and run the windows version of OpenVMM by overriding the target
field of cargo commands, via --target x86_64-pc-windows-msvc. For example, the
following command will run OpenVMM with WHP:

bash

cargo run --target x86_64-pc-windows-msvc

You can optionally set cargo aliases for this so that you don't have to type out
the full target every time. Add the following to your ~/.cargo/config.toml:

toml

[alias]
winbuild = "build --target x86_64-pc-windows-msvc"
wincheck = "check --target x86_64-pc-windows-msvc"
winclippy = "clippy --target x86_64-pc-windows-msvc"
windoc = "doc --target x86_64-pc-windows-msvc"
winrun = "run --target x86_64-pc-windows-msvc"
wintest = "test --target x86_64-pc-windows-msvc"

You can then run the windows version of OpenVMM by running:

bash

cargo winrun

OpenVMM configures some environment variables that specify the default Linux kernel,
initrd, and UEFI firmware. To make those variables available in Windows, run the following:

bash

export WSLENV=$WSLENV:X86_64_OPENVMM_LINUX_DIRECT_KERNEL:X86_64_OPENVMM_LINUX_DIRECT_INITRD:AARCH64_OPENVMM_LINUX_DIRECT_KERNEL:AARCH64_OPENVMM_LINUX_DIRECT_INITRD:X86_64_OPENVMM_UEFI_FIRMWARE:AARCH64_OPENVMM_UEFI_FIRMWARE:RUST_BACKTRACE

Speeding up Windows OpenVMM launch

Due to filesystem limitations on WSL, launching OpenVMM directly will be somewhat
slow. Instead, you can copy the built binaries to a location on in the Windows
filesystem and then launch them via WSL.

Quite a few folks working on the OpenVMM project have hacked together personal
helper scripts to automate this process. Here is one example: (update the
variables at the top of the file as necessary.)

text

/* Detailed source-code truncated for AI context efficiency. */

Running Windows binaries from WSL2

When you cross-compile to x86_64-pc-windows-msvc and run the resulting .exe
from WSL, it runs as a native Windows process. This means it sees Windows
paths (C:\...), not Linux paths (/mnt/c/...). You must translate paths at the
command line.

Use wslpath -w to convert a WSL path to a Windows path:

bash

# Convert a WSL path to a Windows path
wslpath -w /mnt/c/vhds/server25.vhdx
# Output: C:\vhds\server25.vhdx

Here is a full working example that launches OpenVMM with a VHDX disk:

bash

cargo run --target x86_64-pc-windows-msvc -- \
  --vmbus-scsi id=scsi0 \
  --disk "memdiff:file:$(wslpath -w /mnt/c/vhds/server25.vhdx),on=scsi0" \
    --uefi "firmware=$(wslpath -w .packages/hyperv.uefi.mscoreuefi.x64.RELEASE/MsvmX64/RELEASE_VS2022/FV/MSVM.fd)" \
  --gfx -p 6 -m 8GB

admonish tip

If you have the [WSLENV export](#running-windows-openvmm-from-within-wsl)
active, you can omit the `firmware` option. The `X86_64_OPENVMM_UEFI_FIRMWARE`
env var (set in `.cargo/config.toml`) is automatically translated to a
Windows path:

```bash
export WSLENV=$WSLENV:X86_64_OPENVMM_UEFI_FIRMWARE/p
cargo run --target x86_64-pc-windows-msvc -- \
  --vmbus-scsi id=scsi0 \
  --disk "memdiff:file:$(wslpath -w /mnt/c/vhds/server25.vhdx),on=scsi0" \
  --uefi --gfx -p 6 -m 8GB
```

admonish warning

A common mistake is passing `/mnt/c/...` paths directly. The Windows binary
does not understand Linux paths, so you'll get "Access is denied" or "file not
found" errors.

admonish tip

Keep VHDX files on the Windows filesystem (e.g., `C:\vhds\`), not on the WSL
ext4 volume. The WSL volume is slow for large I/O and has limited size. The
same performance note from
[Speeding up Windows OpenVMM launch](#speeding-up-windows-openvmm-launch)
applies to disk images — large files should live on the Windows side.

Guide Src Dev Guide Getting Started Devcontainer (`Guide/src/dev_guide/getting_started/devcontainer.md`)

Getting started via Dev Container

This page provides instructions for setting up a development environment using
the repo provided Dev Container configuration for local development or
development using GitHub Codespaces.

The repo provides an Ubuntu devcontainer.json that installs Rust and the
supported targets for the project.

Developing using a GitHub Codespace

Either create a GitHub Codespace via your fork by clicking on the Code box, or
visit the link here and select your fork
and branch.

If you plan on using rust-analyzer or doing any sort of dev work, it's
recommended to use an 8 core SKU or beefier.

More documentation can be found at the official GitHub
docs.

Developing using a local dev container

This will use Docker + the dev container vscode extension to launch the repo
provided devcontainer.json on your local machine.

Follow the install instructions outlined
here.

From there, use the dev container extension in vscode to create a new dev
container for the repository.

It's recommended to clone the repo inside the dev container using the Dev Containers: Clone Repository Inside Container Volume... command, as the
filesystem otherwise will be very slow over the bind mount, which will make your
builds & rust-analyzer very slow.

More documentation can be found at the official vscode
docs.

Customizing your dev container

Both GitHub codespaces and local dev containers support dotfile repos which can
be used to run personalized install scripts like installing your favorite tools
and shells, and copying over your dotfiles and configuration.

For codespaces, see the documentation
here.
For dev containers, see the documentation here.

You can use the same dotfiles repo for both, but note that codespaces has a few
more limitations outlined in their documentation.

Guide Src Dev Guide Getting Started Linux (`Guide/src/dev_guide/getting_started/linux.md`)

Getting started on Linux / WSL2

This page provides instructions for installing the necessary dependencies to
build OpenVMM or OpenHCL on Linux / WSL2.

\[WSL2] Installing WSL2

To install Windows Subsystem for Linux, run the following command in an
elevated Powershell window:

powershell

PS> wsl --install

This should install WSL2 using the default Ubuntu linux distribution.
You can check that the installation completed successfully by running the
following command in a Powershell window.

powershell

PS> wsl -l -v
  NAME            STATE           VERSION
* Ubuntu          Running         2

Once that command has completed, you will need to open WSL to complete the
installation and set your password. You can open WSL by typing wsl or bash
into Command Prompt or Powershell, or by opening the "Ubuntu" Windows Terminal
profile that should have been created.

admonish info

If you intend to cross-compile OpenVMM for Windows, please ensure you are
running a recent version of Windows 11. Windows 10 is no longer supported as a
development platform, due to needed WHP APIs.

For the best OpenHCL development experience, we recommend **Windows 11 26H1**
(Insider Canary channel, build 28000+) if available for your device — this
enables COM3 serial output for kernel logs and matches the CI runners. See
[Debugging OpenHCL](../../reference/openhcl/debugging.md#recommended-host-os-for-openhcl-development)
for details.

All subsequent commands on this page must be run within WSL2.

Installing Rust

To build OpenVMM or OpenHCL, you first need to install Rust.

The OpenVMM project actively tracks the latest stable release of Rust, though it
may take a week or two after a new stable is released until OpenVMM switches
over to it.

Please follow the official instructions to do so.

\[Linux] Additional Dependencies

On Linux, there are various other dependencies you will need depending on what
you're working on. On Debian-based distros such as Ubuntu, running the following
command within WSL will install these dependencies.

In the future, it is likely that this step will be folded into the
cargo xflowey restore-packages command.

bash

$ sudo apt install \
  binutils              \
  build-essential       \
  cmake                 \
  gcc-aarch64-linux-gnu \
  libssl-dev            \
  pkg-config

Cloning the OpenVMM source

If using WSL2: Do NOT clone the repo into Windows then try to access said
clone from Linux! It will result in serious performance issues, and may break
certain functionality (e.g: cross-compiling Windows binaries).

bash

cd ~/src/
git clone https://github.com/microsoft/openvmm.git

Automatic Dependency Installation

When building OpenHCL, you can pass --install-missing-deps to have
the build system automatically install any missing dependencies
(e.g., .NET SDK, cross-compilation toolchains, system packages):

bash

cargo xflowey build-igvm x64 --install-missing-deps

admonish warning

This flag may install packages and toolchains globally on your system
(e.g., via `apt install` or `rustup toolchain add`).

This is the easiest way to get a working build environment without
manually tracking down each dependency.

Next Steps

You are now ready to build OpenVMM or
OpenHCL!

Guide Src Dev Guide Getting Started Running From Wsl (`Guide/src/dev_guide/getting_started/running_from_wsl.md`)

Running OpenVMM from WSL2

WSL2 runs inside a Hyper-V VM, so when you run OpenVMM from WSL you have two
options with very different behavior: cross-compile to Windows, or build natively
for Linux.

Option A: Cross-compile to Windows (recommended)

Cross-compiling produces a Windows .exe that runs natively on the host, outside
the WSL2 VM. This is the recommended approach for most development.

bash

cargo run --target x86_64-pc-windows-msvc

Uses WHP (Windows Hypervisor Platform) — runs directly on the host
hypervisor with no nesting overhead.
Supports OpenHCL / VTL2 — the full Hyper-V paravisor stack works because
WHP provides VTL support.
Best performance — your guest VM runs at native speed on the host.

Setup:

bash

rustup target add x86_64-pc-windows-msvc

You also need Visual Studio Build Tools installed on the Windows side. See the
Cross Compiling page for full setup instructions.

Because the resulting binary is a Windows process, you must translate WSL paths
to Windows paths with wslpath -w. See
Running Windows binaries from WSL2
for details.

Option B: Build natively for Linux

Building with the default Linux target produces a Linux binary that runs inside
the WSL2 VM.

bash

cargo run

Uses KVM as the hypervisor backend. (MSHV — the Microsoft Hypervisor for
Linux — is tried first but requires a specialized kernel module that is not
publicly available. In WSL2, KVM is what you'll get.)
Does not support OpenHCL. OpenHCL requires Hyper-V VTL support, which KVM
does not provide.
Performance suffers from nested virtualization — your guest VM runs inside
WSL2's Hyper-V VM, adding overhead.
Fine for testing OpenVMM without VTL2, for example booting a simple guest VM
with --uefi.

admonish note

KVM requires access to `/dev/kvm`. Add your user to the `kvm` group:

~~~bash
sudo usermod -aG kvm $USER
~~~

Then log out and back in. Verify with `groups | grep kvm`.

Backend auto-detection

OpenVMM auto-selects the hypervisor backend based on the platform. On Linux it
tries MSHV first (requires a specialized kernel module not publicly available),
then falls back to KVM. On Windows it uses WHP. You can override with
--hypervisor but rarely need to.

Summary

Approach	Binary	Backend	OpenHCL?	Performance
Cross-compile (`--target x86_64-pc-windows-msvc`)	Windows	WHP	✅ Yes	Best — native, no nesting
Native Linux build	Linux	KVM/MSHV	❌ No	Nested — may be slow

admonish tip

Use cross-compilation for most development. It's what the project's CI uses,
it supports OpenHCL, and it avoids nested virtualization overhead.

Guide Src Dev Guide Getting Started Suggested Dev Env (`Guide/src/dev_guide/getting_started/suggested_dev_env.md`)

Suggested Dev Environment

Prerequisites:

One of:
One of:
- Building OpenVMM
- Building OpenHCL

This page is for those interested in actively iterating on OpenVMM or OpenHCL.

Setting up VSCode

These instructions assume you're using VSCode.

If you're using a different development environment, we nonetheless suggest
reading through this section, so you can enable similar settings in whatever
editor / IDE you happen to be using.

admonish tip

Just want the recommended editor settings? Put this in `openvmm/.vscode/settings.json`:

```json
{
    "rust-analyzer.linkedProjects": [
        "Cargo.toml",
    ],
    "rust-analyzer.cargo.targetDir": true,
    "rust-analyzer.imports.granularity.group": "item",
    "rust-analyzer.imports.group.enable": false,
    "[rust]": {
        "editor.formatOnSave": true
    },
}
```

\[WSL2] Connecting to WSL using VSCode

When using Visual Studio Code with WSL, be sure to use the
WSL extension
instead of accessing your files using the \\wsl.localhost share (the repo
should be cloned in the WSL filesystem, as mentioned in the WSL getting started
guide). This will ensure that all VSCode extensions and features to work properly.

Once the
extension is installed, click the blue arrows in the bottom left corner and
select "Connect to WSL". Then open the folder you cloned the repository into.
More information is available
here.

Configuring `rust-analyzer`

rust-analyzer
provides IDE-like functionality when writing Rust code (e.g: autocomplete, jump
to definition, refactoring, etc...). It is a massive productivity multiplier
when working with Rust code, and it would be a very bad idea to work in the
OpenVMM repo without having it set up correctly.

Check out the rust-analyzer manual
for a comprehensive overview of rust-analyzer's features.

Once installed, we suggest you specify the following additional configuration
options in the OpenVMM workspace's .vscode/settings.json file:

json

{
    "rust-analyzer.linkedProjects": [
        "Cargo.toml",
    ]
}

(Strongly Suggested) Avoiding cache invalidation

To avoid unnecessary re-builds or lock-contention in the build directory between
rust-analyzer and manual builds, set the following configuration option to give
rust-analyzer a separate target directory:

json

{
    "rust-analyzer.cargo.targetDir": true,
}

(Strongly Suggested) Disable nested imports

When auto-importing deps, rust-analyzer defaults to nesting imports, which isn't
the OpenVMM convention.

This can be changed to one-dep-per-line by specifying the following settings:

json

{
    "rust-analyzer.imports.granularity.group": "item",
    "rust-analyzer.imports.group.enable": false,
}

Enabling `clippy`

CI will fail if the code is not clippy-clean.
Clippy is a linter that helps catch
common mistakes and improves the quality of our Rust code.

By default, rust-analyzer will use cargo check to lint code, but it can be
configured to use cargo clippy instead:

json

{
  "rust-analyzer.check.command": "clippy",
}

Enabling Format on Save

CI will fail if code is not formatted with rustfmt.

You can enable the "format on save" option in VSCode to automatically run
rustfmt whenever you save a file:

json

{
    "[rust]": {
        "editor.formatOnSave": true
    },
}

Enhanced "Enter"

rust-analyzer can override the "Enter" key to make it smarter:

"Enter" inside triple-slash comments automatically inserts ///
"Enter" in the middle or after a trailing space in // inserts //
"Enter" inside //! doc comments automatically inserts //!
"Enter" after { indents contents and closing } of single-line block

This action needs to be assigned to shortcut explicitly, which can be done by
adding the following line to keybindings.json:

json

// must be put into keybindings.json, NOT .vscode/settings.json!
{
    "key": "Enter",
    "command": "rust-analyzer.onEnter",
    "when": "editorTextFocus && !suggestWidgetVisible && editorLangId == rust"
}

GitHub Pull Request Integration

As the repo is hosted on GitHub, you might find convenient to use the
GitHub Pull Request
VSCode extension. That allows working through the PR feedback and
issues without leaving the comfort of VSCode.

(Possibly Useful) Enabling 'lints' formatting on-save

Aside from using rustfmt, the OpenVMM project also relies on a handful of
extra formatting custom lints. e.g: enforcing the presence of copyright headers,
enforcing single-trailing newlines, etc...

CI will fail if files are not formatted with cargo xtask fmt --pass lints.

In general, there are 3 ways to fix our custom lints:

Manually fixing issues in response to automated feedback
Invoking cargo xtask fmt --pass lints --fix to fix the whole project
Invoking cargo xtask fmt --pass lints --fix --only-diffed to fix only changes with a git diff

If you would prefer having lints enforced whenever you save a file in
VSCode, you can install the
RunOnSave
extension, and add the following configuration to .vscode/settings.json:

json

{
    "emeraldwalk.runonsave": {
        "commands": [
            {
                "match": ".*",
                "cmd": "cd ${workspaceFolder}"
            },
            {
                "match": ".*",
                "isAsync": true,
                "cmd": "$(cat ./target/xtask-path) --run-on-save fmt --pass lints --fix --only-diffed"
            }
        ]
    },
}

Setting up pre-commit and pre-push hooks

It's never fun having CI reject your changes due to some minor formatting issue,
especially when it's super quick to run those formatting checks locally. Running
cargo xtask fmt before pushing up your code is quick and easy, and will save
you the annoyance of wrestling with formatting check-in gates!

Of course, it's very easy to forget to run cargo xtask fmt after making code
changes, but thankfully, you can set up some git hooks
that will do this for you automatically!

You can run cargo xtask install-git-hooks --help for more details on what
hooks are available and their various configuration options, but for most users,
we suggest the following config:

bash

cargo xtask install-git-hooks --pre-push --with-fmt=yes

And you'll be all set!

admonish success

If you're worried about time, the `pre-push` hook should only take ~5
seconds to run locally. That's far better than waiting ~20+ minutes only
for CI to fail on your pull request.

\[WSL2] Cross Compiling from WSL2 to Windows

You may also want to set up cross compiling in WSL2
to you can build for both Windows and Linux in one dev environemnt.

Guide Src Dev Guide Getting Started Windows (`Guide/src/dev_guide/getting_started/windows.md`)

Getting started on Windows

This page provides instructions for installing the necessary dependencies to
build OpenVMM on Windows.

admonish tip

We _strongly_ suggest using [WSL2](./linux.md) for OpenVMM development, rather
than developing on Windows directly.

Developing in WSL2 offers a smoother development experience, while still
allowing you to build and run OpenVMM on Windows through the use of
[cross compilation](./cross_compile.md).

Additionally, it allows you to have a single clone of the OpenVMM repo
suitable for both OpenVMM and OpenHCL development.

You must be running a recent version of Windows 11. Windows 10 is no longer
supported as a development platform, due to needed WHP APIs.

admonish note title="recommended: windows 11 26h1"

For the best OpenHCL development experience, we recommend running a
**Windows 11 26H1** Insider flight (build 28000+) if it is available for your
device. This enables COM3 serial output for OpenHCL kernel logs and matches
the OS used on the project's CI runners.

See [What to know about Windows 11 version 26H1](https://techcommunity.microsoft.com/blog/windows-itpro-blog/what-to-know-about-windows-11-version-26h1/4491941)
and the [Windows Insider Flight Hub](https://learn.microsoft.com/en-us/windows-insider/flight-hub/)
for availability. If 26H1 is not available for your device, Windows 11
24H2/25H2 works — see [Debugging OpenHCL](../../reference/openhcl/debugging.md#recommended-host-os-for-openhcl-development)
for details on the tradeoffs.

NOTE: OpenHCL does NOT build on Windows.

If you are interested in building OpenHCL, please follow the getting started
guide for Linux / WSL2.

Installing Rust

To build OpenVMM, you first need to install Rust.

The OpenVMM project actively tracks the latest stable release of Rust, though it
may take a week or two after a new stable is released until OpenVMM switches
over to it.

Please follow the official instructions to do so.

Visual Studio C++ Build Tools and Windows SDK

If you don't already have it, you will need to install
Visual Studio C++ Build tools
or Visual Studio with the components
"Desktop Development for C++" and the Windows SDK. Windows 11 SDK version 26100 is the latest
version as of this update, but you should install the newest version if there is a newer
version.

The C++ build tools can be installed via Visual Studio Installer -> Modify -> Individual Components
-> MSVC v143 - VS 2022 C++ x64/x86 build tools (latest).

The Windows SDK can be installed via Visual Studio Installer -> Modify -> Individual Components
-> Windows 11 SDK (10.0.26100.0).

The C++ build tools and Windows SDK can be installed at the same time.

Or, you can install the tool via the powershell command below.

powershell

PS> winget install Microsoft.VisualStudio.2022.Community --override "--quiet --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.Windows11SDK.26100"

Aarch64 support

To build ARM64, you need an additional dependency.
This can be installed via Visual Studio Installer -> Modify -> Individual Components
-> MSVC v143 - VS 2022 C++ ARM64/ARM64EC build tools (latest).

Or, you can install the tool via the powershell command below.

powershell

PS> winget install Microsoft.VisualStudio.2022.Community --override "--quiet --add Microsoft.VisualStudio.Component.VC.Tools.ARM64"

Cloning the OpenVMM source

If you haven't already installed git, you can download it
here.

powershell

PS> git clone https://github.com/microsoft/openvmm.git

Next Steps

You are now ready to build OpenVMM!

Guide Src Dev Guide Tests Fuzzing (`Guide/src/dev_guide/tests/fuzzing.md`)

Fuzzing in OpenVMM

Fuzzing infrastructure in OpenVMM is based on the excellent
cargo-fuzz project, which makes it super easy to get
up-and-running with fuzzing in Rust projects.

Under-the-hood, cargo-fuzz hooks into LLVM's
libFuzzer to do the actual fuzzing.

admonish important

Fuzzing only works on **Linux**. libfuzzer-sys doesn't support Windows.
On **aarch64**, set `RUSTFLAGS="-Ctarget-feature=+lse,+neon"` before any
cargo-fuzz command, or builds will fail with atomics errors.

OpenVMM fuzzers target several categories of code:

Chipset devices (battery, CMOS/RTC, IDE) — PIO, MMIO, and PCI config
interfaces exposed to guests
VMBus devices (storvsp) — VMBus channel protocol and SCSI command
processing
Driver stacks (NVMe) — driver-side fuzzing against fuzzed device
responses
Unsafe abstractions (scsi_buffers, guestmem, sparse_mmap) — safe API
surface over unsafe internals
Protocol parsers (UEFI NVRAM, mesh ttrpc, UCS-2) — parsing and
validation of structured data

Guide Src Dev Guide Tests Incubator (`Guide/src/dev_guide/tests/incubator.md`)

Incubator

Incubator runs cross-compiled test executables inside a QEMU-emulated Linux
environment when the host cannot provide the required architecture or hardware
model.

When to use Incubator

Use Incubator for tests that need hardware behavior outside the normal OpenVMM
host test environment.

QEMU TCG is significantly slower than native execution,
so Incubator is reserved for tests that need its emulated platform.

Execution model

Incubator introduces an outer VM around the normal Petri test process:

text

x64 Linux host
  `- Incubator process
       `- QEMU TCG AArch64 Linux VM (L1)
            |- Pipette agent
            `- VMM test executable
                 `- OpenVMM using KVM
                      `- test VM (L2)

The VMM test executable is cross-compiled for
*-unknown-linux-musl. It runs inside the QEMU VM and starts OpenVMM
there. OpenVMM then uses the emulated KVM interface to create the test VM.

Incubator is therefore not an OpenVMM backend. It is a Cargo target runner that
places the existing test executable in a machine capable of running it.

Direct invocation

The binary also has a direct CLI for debugging the runner itself:

bash

cargo run -p incubator -- \
  --profile petri/incubator/profiles/aarch64-tcg-pcie.toml \
  --share path/to/shared-root \
  --map-command-path \
  path/to/shared-root/test-binary

Direct use requires a suitable kernel, initrd, Pipette binary, QEMU, and shared
test artifacts. Running through cargo xflowey vmm-tests-run is preferred
because Flowey resolves and connects those inputs automatically.

Output and failures

Incubator writes a per-process serial log named
incubator-serial..log under the test output directory. Guest command
stdout and stderr flow through Pipette to nextest.

When a run fails, check the serial log first. It distinguishes a guest boot or
Pipette startup failure from a failure in the nested VMM test itself. QEMU
stderr is also captured and reported when the process exits.

Guide Src Dev Guide Tests Make Imc Hive (`Guide/src/dev_guide/tests/make_imc_hive.md`)

IMC Hive Generator

make_imc_hive regenerates the Windows registry hive used to configure Petri
test guests during image preparation.

Purpose

The checked-in IMC hive is applied to an offline Windows installation by
prep_steps. It describes settings that must exist before the prepared test
guest first boots, most importantly the Pipette service.

This is a generator for a repository artifact, not part of every VMM test run.
Normal tests consume the already-generated hive.

Platform requirement

The implementation uses Windows offline-registry APIs and only runs on
Windows. The workspace builds a stub on other platforms, but that stub exits
with an unsupported-platform error.

Regenerating the hive

Run the generator from Windows and pass the output path as its single
positional argument:

powershell

cargo run -p make_imc_hive -- petri\guest-bootstrap\imc.hiv

The tool removes an existing output file before saving the new hive.

Regenerate the hive when changing Pipette's service path, startup account,
dependencies, isolation policy, or initial crash-dump configuration. Changes
that can be made after Pipette connects usually belong in a Petri test instead.

How the hive is consumed

prep_steps embeds or resolves the generated hive and makes it available to
the temporary utility guest. Windows applies the hive to the offline SYSTEM
configuration on the target test disk. The prepared disk is then cached for
later VMM tests.

Troubleshooting

An offline-registry error generally means the destination cannot be created or
the platform APIs rejected a key operation. A later guest failure can indicate
that the hive was valid structurally but contains an incorrect service path or
startup dependency. Inspect the prepared guest's SYSTEM hive and Windows
service logs in that case.

Guide Src Dev Guide Tests Opentmk (`Guide/src/dev_guide/tests/opentmk.md`)

OpenTMK

opentmk is a configurable UEFI Test Microkernel payload for low-level Hyper-V
and OpenHCL test scenarios.

Distinction from `simple_tmk`

The repository contains two separate microkernel systems:

System	Image	Selection model	Executor
`tmk/simple_tmk`	Bare-metal ELF	Static `tmk_tests` descriptors	Host `tmk_vmm` process
`opentmk`	UEFI executable in a VHD	Embedded JSON configuration	Payload dispatches its own test

Use OpenTMK when the test needs its UEFI-based runtime, Hyper-V test contexts,
multiple VTLs, or a configuration patched into the boot image. Use simple_tmk
for small tests that should run through the multi-backend tmk_vmm executor.

Build and package

The supported local entry point builds the UEFI executable and packages it in
a bootable VHD:

bash

cargo xflowey build-opentmk

Embedded configuration

OpenTMK contains a fixed configuration region that host tooling can locate and
patch without relinking the UEFI executable. Supply a JSON selection file to
the build pipeline:

bash

cargo xflowey build-opentmk x86-64 \
  --config path/to/test-config.json

The configuration identifies the backend and test to dispatch and carries
test-specific parameters. The exact schema follows opentmk_protocol; use
configurations from the test being developed as the starting point.

If --config is omitted, the VHD contains the unconfigured OpenTMK executable.
That artifact is useful as an intermediate build output but cannot select a
test until a harness patches it.

Guest lifecycle

When firmware launches the image, OpenTMK:

Initializes UEFI services and the OpenTMK runtime.
Reads and validates the embedded configuration region.
Selects the registered backend.
Dispatches the configured test with its parameters.
Reports progress through its logging and assertion infrastructure.

Running the VHD

The Flowey output is a guest disk. Attach it to a UEFI VM using the environment
required by the selected test. OpenTMK itself has no host command line.

A generic OpenVMM launch has this shape:

bash

cargo run -p openvmm -- --uefi \
  --vmbus-scsi id=scsi0 \
  --disk memdiff:path/to/opentmk.vhd,on=scsi0

Most OpenTMK tests require additional Hyper-V or OpenHCL configuration. Copy
the VM setup from the owning test rather than assuming the generic launch is
sufficient.

Failures and diagnostics

An unknown backend or test indicates a mismatch between the JSON configuration
and the payload's registered dispatch table. A missing or malformed config
usually fails before test code starts.

Capture the guest serial output from the beginning of boot. It distinguishes
UEFI/runtime initialization failures from assertions inside the selected test.
Because the payload runs without a general-purpose OS, there is no Pipette or
shell to inspect after a fatal failure.

Guide Src Dev Guide Tests Opentmk Executor (`Guide/src/dev_guide/tests/opentmk_executor.md`)

OpenTMK Executor

opentmk_executor is a UEFI payload that receives generated programs over a
serial protocol and executes registered low-level operations.

Intended use

The executor provides a small, deterministic environment for an external test
generator such as syzkaller. Instead of rebuilding the guest for each case, the
host boots the executor once, sends encoded programs, and receives an
acknowledgement or error packet for each case.

It is distinct from the configuration-driven opentmk test payload. The
executor has no embedded test selection.

Build

Install the architecture's Rust UEFI target, then build the binary. For x64:

bash

rustup target add x86_64-unknown-uefi
cargo build -p opentmk_executor --target x86_64-unknown-uefi

The output is a UEFI executable. Package it in a bootable image using the same
UEFI image tooling used by other guest test applications, or integrate it into
the external harness that owns the VM and serial channel.

Serial channel

COM1 carries the executor protocol. After OpenTMK runtime initialization, the
guest performs a three-way handshake with fixed SYN, SYN-ACK, and ACK values.
Normal packets are framed with header and footer magic values and carry a
serialized OpenTMKPacket.

Do not use COM1 for human-readable logging while driving the protocol. The
runtime uses another serial path for logs so arbitrary text does not corrupt
packet framing.

Packet sequence

A host session follows this order:

Complete the serial handshake.
Send a Configuration packet selecting a grammar deserializer and providing
its function mapping.
Wait for an Ack packet.
Send one or more FuzzTest packets containing encoded programs.
Read an Ack with the program result code after each test.
Treat an Error packet as a terminal protocol or execution failure.

Memory safety boundary

Decoded programs refer to generated memory through the SafeMemoryMap
interface. Read operations place their result into that map rather than
treating generated addresses as native pointers. Function implementations
validate arity and variable types before issuing privileged operations.

The environment still deliberately performs low-level I/O and hypercalls. Run
it only in a disposable test VM controlled by the fuzzing harness.

Failure behavior

Bad handshake magic, packet framing, serialization, mappings, or packet order
produces an OpenTMKErrorPacket when possible and ends the executor loop.
Individual generated operations can return a nonzero acknowledgement code
without necessarily invalidating the session.

If the host appears to hang:

Verify that the UEFI payload reached OpenTMK runtime initialization.
Confirm both sides use the same packet and grammar definitions.
Check that COM1 is a clean binary channel with no terminal translation.
Ensure the host waits for configuration acknowledgement before sending a
test case.

admonish note title="see also"

[OpenTMK](opentmk.md) describes the configuration-driven UEFI test payload.

Guide Src Dev Guide Tests Perf (`Guide/src/dev_guide/tests/perf.md`)

Performance Tests (burette)

burette is a standalone binary that runs performance benchmarks
against OpenVMM using the petri test framework. It measures boot
time, memory overhead, and concurrent VM scaling behavior, producing
JSON reports that can be compared across builds.

Use Burette for developer-controlled experiments, report comparison, and
self-contained remote runs.

Prerequisites

Linux host with /dev/kvm access (or Windows with Hyper-V)
Built openvmm and burette built for the host target, and pipette
built for <arch>-unknown-linux-musl.
Test kernel/initrd artifacts for the appropriate architecture installed
to their magic paths.
iperf3 package installed (for networking tests).

For example, to download and build everything on an x64 Ubuntu system:

bash

cargo xflowey restore-packages
sudo apt update
sudo apt install iperf3
cargo build --release -p burette -p openvmm
cargo build --release --target x86_64-unknown-linux-musl -p pipette

Architecture

Each benchmark controls an OpenVMM process through Petri:

text

burette
  |- resolve host and guest artifacts
  |- launch OpenVMM
  |    `- boot Linux test guest
  |- connect to Pipette in the guest
  |- run or observe the workload
  |- collect host and guest measurements
  `- stop the VM and write JSON results

Some tests rebuild the VM for every sample, while throughput tests can retain a
warm VM and run repeated guest workloads. Read the benchmark description when
comparing first-run and steady-state results.

Running Tests

admonish note

In all of the below commands, replace `burette` with the path to your built
burette executable, or `cargo run --release -p burette` to build and run
burette within a local clone.

Boot time

Measures launch-to-pipette-connect time using Linux direct boot:

bash

burette run --test boot-time -o report.json

Available profiles control the VM configuration:

Profile	Description
`standard`	Full device set, serial agent, shared memory
`quiet-serial`	Like standard but suppresses kernel console
`minimal`	Pipette-as-init, minimal devices, shared memory
`minimal-private`	Same as minimal but with private memory (fastest)

bash

# Use a specific profile
burette run --test boot-time --profile minimal -o report.json

# Custom iteration count and guest RAM
burette run --test boot-time --iterations 20 --mem-mb 1024

Memory overhead

Boots a single VM and measures host-side memory consumption for the
openvmm process tree:

bash

burette run --test memory -o memory.json

On Windows, the available process accounting differs and VMM overhead follows
the private-memory measurement.

Network throughput

Measures TCP throughput (Gbps) and UDP packet rate (pps) using iperf3
with a linux_direct VM, an erofs tool image, and Consomme networking:

bash

burette run --test network -o network.json

# Test with virtio-net instead of VMBus
burette run --test network --nic virtio-net -o network.json

The report records TCP throughput and UDP packet-rate metrics. Keep the NIC,
network backend, host CPU placement, and MTU constant between compared runs.

Compare shared vs. private memory overhead:

bash

burette run --test memory --profile minimal -o shared.json
burette run --test memory --profile minimal-private -o private.json
burette compare shared.json private.json

Scale boot

Launches N VMs concurrently to measure boot time under contention
and per-VM memory overhead. Default sweep: N = 1, 2, 4, 8, 16, 32,
64.

bash

# Full geometric sweep (auto-stops at 90% host memory)
burette run --test scale-boot --mem-mb 256 -o scale.json

# Single data point
burette run --test scale-boot --vms 16 --mem-mb 256

# Custom sweep
burette run --test scale-boot --vms 1,2,4,8 --max-vms 32

Per-N metrics include scale_{N}_mean_boot_ms,
scale_{N}_p99_boot_ms, scale_{N}_last_ready_ms,
scale_{N}_per_vm_memory_mib, and others.

Disk I/O

Measures block I/O throughput (MiB/s) and IOPS using fio in a linux_direct
VM with an erofs tool image and a data disk. Supports virtio-blk and storvsc
(synthetic SCSI) backends:

bash

# Virtio-blk with RAM-backed disk (measures virtio overhead)
burette run --test disk-io -o disk.json

# Storvsc backend
burette run --test disk-io --disk-backend storvsc -o disk.json

# File-backed disk for realistic host I/O latency
burette run --test disk-io --data-disk /tmp/test.raw --data-disk-size-gib 8

Reported metrics per backend:

fio_{backend}_seq_read_bw / fio_{backend}_seq_write_bw — sequential bandwidth (MiB/s)
fio_{backend}_rand_read_bw / fio_{backend}_rand_write_bw — random bandwidth (MiB/s)
fio_{backend}_rand_read_iops / fio_{backend}_rand_write_iops — random IOPS

By default a RAM-backed disk is used to isolate virtio/storvsc overhead
without host filesystem noise. Pass --data-disk with a path on fast
storage (e.g., NVMe) for end-to-end latency measurements.

virtio-fs

The virtio-fs benchmark boots a minimal Linux guest, mounts a host-exported
filesystem, and runs fio against a test file through the virtio-fs data path:

bash

burette run --test virtio-fs -o virtio-fs.json

Use --virtiofs-file-size-mib to change the generated test-file size. Ensure
the host filesystem has enough free space and keep its storage medium constant
between runs.

Test lifecycle and cleanup

Burette creates a Petri log source, resolves only the artifacts required by the
selected test, and then runs that benchmark's harness. VM teardown is part of
the measured test's cleanup path. A failed workload can therefore leave useful
OpenVMM and guest logs even when no JSON sample is recorded.

Use --log-dir to retain logs outside the default
vmm_test_results/burette location. On Linux, --perf-dir can retain host
performance traces for supported tests.

Comparing Reports

bash

burette compare baseline.json candidate.json

Prints a table of deltas and percentage changes for each metric.
Optionally write the comparison to JSON:

bash

burette compare baseline.json candidate.json -o diff.json

The comparison displays absolute and percentage deltas. A positive percentage
is not universally better: higher throughput is desirable, while higher boot
latency or memory overhead is a regression. Interpret the metric's unit and
direction before drawing a conclusion.

Remote Deployment

Package all binaries and artifacts into a self-contained tarball:

bash

burette package -o burette_bundle.tar.gz

On the remote machine:

bash

tar xzf burette_bundle.tar.gz
cd burette_bundle
VMM_TESTS_CONTENT_DIR=$PWD ./burette run -o report.json

The bundle includes burette, openvmm, pipette, the test kernel,
and initrd — no Rust toolchain or repo checkout needed.

The remote host still needs a supported virtualization interface and any host
facilities required by the selected network, storage, or filesystem backend.

Running All Tests

Omit --test to run every test:

bash

burette run -o full_report.json

JSON Report Format

Reports are JSON files with git revision info, timestamps, and
per-metric statistics:

json

{
  "git_revision": "abc123",
  "git_commit_date": "2026-03-18T00:00:00Z",
  "date": "2026-03-18T01:00:00Z",
  "results": [
    {
      "name": "boot_time_ms",
      "unit": "ms",
      "iterations": 10,
      "mean": 126.3,
      "std_dev": 1.5,
      "min": 124.4,
      "max": 128.1
    }
  ]
}

The report records source revision and time alongside per-metric summary
statistics. Preserve the complete report rather than extracting only the mean;
sample count, standard deviation, minimum, and maximum help distinguish a real
change from host noise.

Measurement practice

For useful before/after comparisons:

Use the same host, power policy, kernel, firmware, OpenVMM features, and
guest artifacts.
Stop unrelated CPU, disk, and network workloads.
Use release builds for representative performance.
Run enough iterations to expose variance.
Keep raw reports and logs for both revisions.
Repeat surprising results before attributing them to a code change.

The scale test deliberately applies host-memory limits. A truncated sweep is
not directly comparable with a sweep that reached larger VM counts.

Troubleshooting

A missing-artifact error names the Petri artifact that could not be resolved;
build or restore that exact target rather than substituting a host binary.
A launch-to-Pipette timeout usually means the guest failed to boot or the
selected transport did not connect. Read the Petri and guest serial logs.
Network failures can come from the helper process, host backend, guest tool
image, or firewall. Verify a single warm-up run before collecting samples.
File-backed disk and virtio-fs results are sensitive to host caching and free
space. Use a dedicated fast filesystem when measuring end-to-end I/O.
A report with large variance is not made reliable by compare; remove the
source of noise or increase the iteration count.

Guide Src Dev Guide Tests Pipette (`Guide/src/dev_guide/tests/pipette.md`)

Pipette Guest Agent

pipette is the in-guest agent through which Petri tests execute commands and
perform operating-system operations.

Role in a VMM test

Petri divides guest control between a host client and the Pipette agent:

text

VMM test process
  `- Pipette client
       `- VSocket or TCP connection
            `- Pipette agent in the guest
                 |- execute process
                 |- transfer files
                 |- collect output
                 `- shut down the guest

The test obtains a PipetteClient after the VM boots. Commands created through
that client are serialized to the guest, executed there, and represented by a
remote child process whose status and standard streams are returned to the
host.

Pipette is test infrastructure, not an OpenVMM management protocol. Its API and
wire format are shared by Petri components and can change with the repository.

Supported guests

Pipette runs in Linux and Windows guests on the architectures built by the VMM
test pipeline. The binary is compiled for the guest, not necessarily for the
developer's host.

The default transport is VSocket. TCP is used when the guest configuration has
no usable VSocket path, including the current Windows no-vmbus image.

Windows builds also accept --service. The generated test-image registry hive
uses this mode so Pipette starts automatically as LocalSystem.

Linux PID 1 mode

Some Linux direct-boot configurations use Pipette as rdinit. When its process
ID is 1, Pipette performs the minimal init responsibilities needed by the test
guest before starting the request agent. These include setting up essential
mounts and reaping child processes.

This mode deliberately does not try to become a general init system. The guest
image should contain only the services required by its test workload.

Connection lifecycle

At startup, Pipette initializes its selected transport and establishes the
protocol connection. It then serves requests until the host disconnects. A
disconnect is not fatal: the process creates a new agent connection and waits
for the host again.

The exact request set is defined by the pipette_protocol crate. Tests should
normally use pipette_client rather than manually constructing protocol messages.

Building Pipette

The supported path is to let cargo xflowey vmm-tests-run discover and build
the guest artifact required by the selected tests:

bash

cargo xflowey vmm-tests-run --filter "test(my_test)" --build-only

For a focused build, choose a target matching the guest image. For example:

bash

cargo build -p pipette --target x86_64-unknown-linux-musl

bash

cargo build -p pipette --target x86_64-pc-windows-msvc

admonish warning

A host-native `cargo build -p pipette` is only useful when the host target also
matches the intended guest. Petri resolves Pipette by guest OS and
architecture, so placing a binary under the wrong target directory does not
satisfy the artifact requirement.

Image integration

Linux test images either include Pipette in an initrd or expose it through a
shared test-content disk. Windows images are prepared with a registry hive that
defines the Pipette service and points it at the injected executable.

The prep_steps tool prepares reusable Windows images. The make_imc_hive
tool regenerates the source hive when the service definition changes.

Troubleshooting

If a test times out waiting for Pipette:

Check the guest serial log for Pipette startup or panic messages.
Confirm that the binary matches the guest architecture and operating system.
Confirm that the configured transport matches the VM's available VSocket or
TCP path.
For Linux direct boot, verify that the kernel command line starts the
intended init and that essential mounts succeeded.
For Windows, verify that the service image path matches the drive containing
pipette.exe.

Incubator additionally waits for a PIPETTE READY serial marker before opening
its forwarded TCP connection. A missing marker generally means the outer guest
failed before Pipette initialized.

Guide Src Dev Guide Tests Prep Steps (`Guide/src/dev_guide/tests/prep_steps.md`)

Windows Test Image Preparation

prep_steps prepares reusable Windows disks for Petri VMM tests by modifying
them from a temporary OpenVMM guest.

Why preparation is separate

Windows guest images require changes that are awkward or unsafe to perform by
editing the VHD directly. prep_steps boots a known Windows utility VM,
attaches a copy of the target image as a data disk, and performs the changes
through Windows itself.

Keeping this work outside individual tests has two benefits:

Every test starts from the same prepared image.
Expensive service and driver installation can be cached and reused.

The tool remains intentionally small. Changes that can be made after the test
guest boots should be performed through Pipette in the test instead.

Normal use

You normally do not invoke prep_steps yourself. Artifact discovery in
cargo xflowey vmm-tests-run determines whether a selected test needs a
prepared Windows image, builds the helper, and runs the required preparation
mode before nextest starts.

To force regeneration while developing preparation logic, disable reuse:

bash

cargo xflowey vmm-tests-run \
  --no-reuse-prepped-vhds \
  --filter "test(my_windows_test)"

The resulting images and preparation logs live below the VMM test content
directory selected by --dir.

Standard image flow

The standard path performs the following work:

Resolve the utility boot image, source test VHD, firmware, OpenVMM, Pipette,
and output directory.
Copy the source disk to its prepared output path.
Randomize disk and partition identifiers so the utility VM can attach the
source and result disks without Windows treating them as duplicates.
Boot the utility Windows VM with the result disk attached.
Connect to the utility VM's Pipette agent.
Apply the IMC registry hive to the offline Windows installation on the
attached disk.
Shut down the utility VM and retain the prepared disk.

The hive configures Pipette as an automatically started Windows service and
enables the VMBus behavior expected by isolated test guests.

Outputs and caching

Prepared images are derived artifacts. Reusing one is correct only while its
source image and preparation inputs remain compatible. Use
--no-reuse-prepped-vhds after changing:

prep_steps behavior.
The Pipette service hive.
Drivers installed by the no-vmbus mode.
The source Windows image assumptions.

Do not edit a shared source VHD in place. Preparation works on a result copy so
tests remain repeatable.

Troubleshooting

Preparation failures usually occur before the actual nextest run. Check the
preparation log for the stage that failed:

Artifact errors indicate a missing source VHD, firmware, Pipette, or driver
build.
Disk errors can indicate a corrupt image or failure while rewriting disk
identifiers.
A Pipette timeout indicates that the utility boot VM did not start its agent.
Windows servicing failures in no-vmbus usually identify a missing or
incompatible virtio-win file.

When cross-running Windows tests from WSL, Flowey reports if the output
directory must be placed on a Windows filesystem.

Guide Src Dev Guide Tests Simple Tmk (`Guide/src/dev_guide/tests/simple_tmk.md`)

simple_tmk

simple_tmk is the bare-metal guest payload containing the repository's basic
Test Microkernel processor and interrupt tests.

Runtime model

The payload is a no_std, minimal_rt executable. It has no bootloader,
firmware interface, process model, or command line. tmk_vmm loads its ELF
directly, initializes the processor, and selects one test by index.

The shared tmk_core runtime provides:

A test entry point and panic handler.
Guest-to-host logging and completion commands.
Scoped interrupt-handler registration.
Architecture-specific helpers such as x86 MSR access.

The payload is useful for code that must touch processor state directly. Use a
normal VMM test when the behavior requires firmware, devices, or a guest OS.

Building

Build the x86_64 payload with the same command emitted by Petri's artifact
resolver:

bash

RUSTC_BOOTSTRAP=1 cargo build -p simple_tmk \
  --config openhcl/minimal_rt/x86_64-config.toml

Build AArch64 with:

bash

RUSTC_BOOTSTRAP=1 cargo build -p simple_tmk \
  --config openhcl/minimal_rt/aarch64-config.toml

The output is an ELF consumed by tmk_vmm. Running it as a host process is not
supported; non-minimal_rt builds contain only a rejecting stub.

Flowey builds the correct payload automatically when selected Petri tests
require a SIMPLE_TMK_* artifact.

Defining a test

A test is a function annotated with #[tmk_test]:

rust,ignore

#[tmk_test]
fn example(_: TestContext<'_>) {
    log!("example started");
    assert_eq!(2 + 2, 4);
}

The macro emits a static descriptor into the ELF tmk_tests section. The
descriptor gives tmk_vmm the test name, entry point, and flags needed for
discovery and execution.

Tests that intentionally fault can be marked as expected failures:

rust,ignore

#[tmk_test(expected_failure, linux_only)]
fn expected_abort(_: TestContext<'_>) {
    // Trigger the architecture condition under test.
}

Scoped interrupt handlers

On x86_64, TestContext exposes a Scope. A test can create a nested scope,
install an ISR for a vector, enable interrupts, and let the scope restore its
handler state before returning. This keeps one test's interrupt setup from
leaking into another test implementation.

Running and listing

After building both binaries:

bash

target/debug/tmk_vmm --tmk \
  target/x86_64-unknown-none/debug/simple_tmk --list

Run one test by passing its discovered name:

bash

target/debug/tmk_vmm --tmk \
  target/x86_64-unknown-none/debug/simple_tmk ud2

Output appears in the tmk_vmm tracing stream. A panic becomes a structured
protocol message when the runtime remains operational; an earlier processor
fault is reported from the host VP loop.

Adding coverage safely

When adding a test:

Keep the initial machine assumptions explicit.
Install handlers before enabling or triggering interrupts.
Bound polling loops so a missing hardware event becomes a failure.
Restore mutable processor state when later operations in the same test need
a known configuration.
Run through every backend whose behavior the test claims to cover.

Tests share the image but receive fresh VM state from tmk_vmm, so global
state does not need to survive between test cases.

admonish note title="see also"

[TMK VMM](tmk_vmm.md) documents the host executor, hypervisor backends, and
result protocol.

Guide Src Dev Guide Tests Test Igvm Agent Rpc Server (`Guide/src/dev_guide/tests/test_igvm_agent_rpc_server.md`)

IGVM Agent Test Server

test_igvm_agent_rpc_server is a Windows-only test double for the host IGVM
agent RPC interface used by guest attestation flows.

Purpose

Attestation tests need deterministic host responses for conditions that are
difficult to reproduce with a real service. This executable hosts the expected
Windows RPC facade and can install predefined response plans before accepting
requests.

It is test infrastructure, not an IGVM agent suitable for deployment. The
implementation and scenario names follow the in-tree tests rather than a
stable external API.

Normal orchestration

For tests that declare this artifact, Flowey:

Builds the server for Windows MSVC.
Copies it into the VMM-test content directory.
Starts it before nextest and redirects output to
test_igvm_agent_rpc_server.log.
Verifies that it did not exit immediately.
Runs the selected attestation tests.
Terminates the background server during cleanup.

This lifecycle is part of cargo xflowey vmm-tests-run; most developers do not
need to start the process themselves.

Direct use

For focused Windows debugging, build and launch it directly:

powershell

cargo build -p test_igvm_agent_rpc_server `
  --target x86_64-pc-windows-msvc

powershell

.\target\x86_64-pc-windows-msvc\debug\test_igvm_agent_rpc_server.exe `
  --test-config AkCertRequestFailureAndRetry

The server continues running while it services RPC requests. Stop it after the
test so it does not affect a later scenario.

Some local test paths support an explicit autostart environment variable. Use
the instructions emitted by the test or its local-autostart helper rather than
running multiple server instances.

Troubleshooting

An immediate unsupported-platform failure means the executable was launched
outside Windows.
A startup failure can indicate that another instance owns the RPC endpoint.
Check test_igvm_agent_rpc_server.log before guest logs when no host request
reaches the planned scenario.
Confirm that the selected test configuration matches the scenario expected
by the Petri test.
Stop stale instances before rerunning a different plan.

Guide Src Dev Guide Tests Tmk Vmm (`Guide/src/dev_guide/tests/tmk_vmm.md`)

TMK VMM

tmk_vmm is a small host-side VMM that discovers and executes tests from a
Test Microkernel ELF image.

Why use a TMK

A full OpenVMM VM includes firmware, an operating system, a broad device model,
and guest-control infrastructure. That is useful for integration coverage but
makes a poor fixture for a focused processor or hypervisor-interface test.

tmk_vmm creates only the VM state needed by the Test Microkernel:

A single virtual processor.
4 MiB of guest RAM.
Architecture-specific initial register state.
A fixed MMIO command address for guest-to-host reporting.

This keeps failures close to the hypervisor behavior under test.
tmk_vmm does not support general-purpose VMs.

Host and guest split

text

tmk_vmm host process
  |- parse the TMK ELF
  |- enumerate test descriptors
  |- create a minimal partition and VP
  |- handle guest MMIO commands
  `- report test results

simple_tmk guest image
  |- initialize the minimal runtime
  |- invoke one selected test
  `- send log, panic, or completion commands

The host reloads the TMK and creates fresh execution state for each selected
test.

Building the executor

Build a native executor with Cargo:

bash

cargo build -p tmk_vmm

The available hypervisor backend is determined by the build platform and guest
architecture. Flowey builds additional native and static Linux variants for
VMM-test artifacts.

Running a TMK

List tests without creating a VM:

bash

target/debug/tmk_vmm --tmk path/to/simple_tmk --list

Run every test using the automatically selected hypervisor:

bash

target/debug/tmk_vmm --tmk path/to/simple_tmk

Pass test names as positional arguments to run a subset:

bash

target/debug/tmk_vmm --tmk path/to/simple_tmk boot apic_timer

ELF test discovery

The input is a 64-bit ELF containing a tmk_tests section. Each descriptor
stores:

A pointer and length for the UTF-8 test name.
The guest entry point.
An expected_failure flag.
A linux_only flag.

tmk_vmm --list parses and relocates this metadata without entering the guest.
When running a test, the loader maps the ELF, prepares page tables and initial
registers, and passes a StartInput structure to the TMK.

Result protocol

The host reserves MMIO address 0xffff0000. The guest writes a pointer to a
tmk_protocol::Command at that address. Commands can:

Log a UTF-8 string held in guest memory.
Report a panic with message, file, and line.
Complete explicitly with a success value.

If the VP halts or faults before completion, tmk_vmm reports the halt reason
and available register state. A normal failure makes the process exit nonzero.
For a descriptor marked expected_failure, a failure or fault is success and a
normal pass is an error.

Troubleshooting

no hypervisor available means no compiled native backend passed its
availability check. Verify host permissions or pass --hv explicitly.
An empty --list result usually means the image was not built with the TMK
linker configuration or contains no registered tests.
A protocol error indicates that guest pointers or command contents were not
valid for the mapped memory.
A Faulted result includes processor state; use it before adding logging to
code that may fail before the command channel works.

admonish note title="see also"

[simple_tmk](simple_tmk.md) describes the guest payload and test authoring
model.

Guide Src Dev Guide Tests Tpm Guest Tests (`Guide/src/dev_guide/tests/tpm_guest_tests.md`)

TPM Guest Test Utility

tpm_guest_tests is an in-guest CLI for probing TPM 2.0 and OpenHCL
attestation behavior from Petri tests.

Execution environment

The binary runs inside a Linux or Windows guest with a visible TPM, normally an
OpenVMM or OpenHCL vTPM. It does not operate on a VMGS file and is not a
host-side TPM provisioning utility.

On Unix, it opens /dev/tpmrm0 first and falls back to /dev/tpm0. On Windows,
it sends commands through TPM Base Services. The tool adapts those transports
to the shared tpm_lib command implementation.

Petri usually copies the architecture-appropriate executable into the guest
and runs it through Pipette. You can also invoke it from an interactive guest
shell.

Building and deployment

Let VMM-test artifact discovery build the correct Linux or Windows guest
binary:

bash

cargo xflowey vmm-tests-run \
  --build-only \
  --filter "test(tpm)"

A focused Linux build can be produced with:

bash

cargo build -p tpm_guest_tests \
  --target x86_64-unknown-linux-musl

The binary must still be copied into a guest with the corresponding
architecture and OS before it is useful.

Exit status and diagnostics

Successful commands exit with status zero. CLI errors, TPM transport failures,
malformed responses, oversized data, and comparison failures print an error
chain and exit nonzero. NV reads include a bounded hexadecimal and ASCII dump
to aid diagnosis without flooding the test log.

If opening the TPM fails, confirm that the VM configuration exposed a vTPM and
that the guest user can access its device or TBS context. An AK certificate may
appear asynchronously during attestation initialization; use comparison retry
only when the test expects that transition.

Guide Src Dev Guide Tests Unit (`Guide/src/dev_guide/tests/unit.md`)

Unit Tests

admonish tip

Note: We recommend using [cargo-nextest](https://nexte.st/) to run unit / VMM
tests. It is a significant improvement over the built-in `cargo test` runner,
and is the test runner we use in all our CI pipelines.

You can install it locally by running: `cargo install cargo-nextest --locked`

See the [cargo-nextest](https://nexte.st/) documentation for more info.

Unit tests test individual functions or components without pulling in lots of
ambient infrastructure. In Rust, these are usually written in the same file as
the product code--this ensures that the test has access to any internal methods
or state it requires, and it makes it easier to ensure that tests and code are
updated at the same time.

A typical module with unit tests might look something like this:

rust

fn add_5(n: u32) -> u32 {
    n + 5
}

#[cfg(test)]
mod tests {
    #[test]
    fn test_add_5() {
        assert_eq!(add_5(3), 8);
    }
}

In the OpenVMM repo, all the unit tests are run on every pull request, on an
arbitrary build machine. As a result of this approach, it's important that unit
tests:

run quickly
do not affect the state of the machine that runs them
do not take a dependency on machine configuration
- e.g: no root/administrator access or virtualization requirement

We may loosen these guidelines over time if it becomes necessary. You can also
mark tests with #[ignore] if they do not meet these guidelines but are useful
for manual testing.

See the unit testing section
in "Rust by example" for more details.

Doc tests

Rust has another type of unit tests known as doc tests. These are unit tests
that are written in the API documentation comments of public functions. They
will be run automatically along with the unit tests, so the same guidelines
apply.

When do you choose a doc test over a unit test?

Doc tests can only access public functionality, and they are intended to
document the usage of a function or method, not to exhaustively check every
case. So write doc tests primarily as examples for other developers, and rely on
unit tests for your main coverage.

An example might look like this:

rust

/// Adds 5 to `n`.
///
/// ```
/// assert_eq!(mycrate::add_5(3), 8);
/// ```
pub fn add_5(n: u32) -> u32 {
    n + 5
}

See the documentation testing
section in
Rust by example for more info.

Guide Src Dev Guide Tests Vmm (`Guide/src/dev_guide/tests/vmm.md`)

VMM Tests

The OpenVMM repo contains a set of "heavyweight" VMM tests that fully boot a
virtual machine and run validation against it. Unlike Unit tests, these are all
centralized in a single top-level vmm_tests directory.

The OpenVMM PR and CI pipelines will run the full test suite on all supported
platforms; you'd typically run only the tests relevant to the changes you're
working on.

Scheduled Pipelines

The OpenVMM Burn In workflow repeatedly runs Windows x64 VMM tests using
artifacts from a successful CI run on main to measure failure frequency.

The OpenVMM Patina Nightly workflow runs the release check-in gates using
the configured mu_msvm RELEASE ClangPDB Patina firmware for standalone
OpenVMM UEFI tests and newly built OpenHCL images. You can also start it
manually using GitHub Actions' Run workflow.

Writing VMM Tests

To streamline the process of booting and interacting with VMs during VMM tests, the
OpenVMM project uses an in-house test framework/library called petri.

The library does not yet have a stable API, so at this time, the best way to
learn how to write new VMM tests is by reading through the existing corpus of
tests (start with vmm_tests/vmm_tests/tests/tests/multiarch.rs),
as well as reading through petri's rustdoc-generated API docs.

The tests are currently generated using a macro (#[vmm_test]) that allows
the same test body to be run in a variety of scenarios, with different guest
operating systems, firmwares, and VMMs (including Hyper-V, which is useful
for testing certain OpenHCL features that aren't supported when using
OpenVMM as the host VMM).

"heavy" tests

The global nextest.toml
configures how tests run in our test environments. The default and ci profiles
control things like timeouts, and how many resources we allocate to a given test. The number
of required threads is a fuzzy requirement relative to the number of VPs consumed by the VM under
test, the amount of memory your test needs, the host test framework (petri itself), and so on.

We have some pre-defined overrides that perform filterset matching on test name. These overrides
are curated to balance individual trial (test case) performance against overall concurrency on
engineers' local machines and in CI. Put these special words in your test to opt in to that override:

heavy - if your test is heavier than the typical vmm_test. E.g., your test explicitly requests 16 virtual processors.
very_heavy if your test is heavier than a heavy test. E.g., your test explicitly requests 32 virtual processors.

"unstable" tests

If a test is not yet reliable enough to gate PRs, mark it unstable. Every
unstable marker requires a reason string documenting why the test is
unstable (ideally referencing a tracking issue).

To mark a single variant unstable, wrap it in unstable(reason = "...", <config>):

rust,ignore

#[vmm_test(
    // unstable variant:
    unstable(reason = "flaky on aarch64 in CI (microsoft/openvmm#1234)", hyperv_openhcl_uefi_aarch64(vhd(windows_11_enterprise_aarch64))),
    // other reliable variants:
    hyperv_openhcl_uefi_aarch64(vhd(ubuntu_2404_server_aarch64))
    // ...
)]
async fn my_test<T: PetriVmmBackend>(config: PetriVmBuilder<T>) -> anyhow::Result<()> {
    // ...
}

To mark all variants of the test unstable, apply unstable(reason = "...") as a
test-wide attribute alongside configs(...):

rust,ignore

#[vmm_test_with(unstable(reason = "flaky in CI (microsoft/openvmm#1234)"), configs(
    hyperv_openhcl_uefi_aarch64(vhd(windows_11_enterprise_aarch64)),
    hyperv_openhcl_uefi_aarch64(vhd(ubuntu_2404_server_aarch64))
    // ...
))]
async fn my_test<T: PetriVmmBackend>(config: PetriVmBuilder<T>) -> anyhow::Result<()> {
    // ...
}

Unstable tests run in the same CI job as stable tests. In CI the
PETRI_IGNORE_UNSTABLE_FAILURES environment variable is set, so when an unstable
test fails the CI run passes with a warning. Outside of CI that variable is not
set by default, so an unstable test failure is reported as a failure like any
other test.

To promote an unstable test to stable, remove the unstable(...) wrapper. This
is a single-place change — no CI or configuration updates are required.

To suppress failures from unstable tests when running locally (matching the CI
behavior), set the following environment variable: PETRI_IGNORE_UNSTABLE_FAILURES=1

"ignore" tests

If a test (or a specific variant) should not run by default at all — for example
it depends on functionality that isn't ready yet — mark it ignore, analogous to
libtest's #[ignore]. Like unstable, every ignore marker requires a reason
string (ideally referencing a tracking issue). Ignored tests are skipped unless
explicitly requested.

To ignore a single variant, wrap it in ignore(reason = "...", <config>):

rust,ignore

#[vmm_test(
    ignore(reason = "OpenVMM VBS boot on Ubuntu is unreliable (microsoft/openvmm#2139)", openvmm_openhcl_uefi_x64[vbs](vhd(ubuntu_2504_server_x64))),
    hyperv_openhcl_uefi_aarch64(vhd(ubuntu_2404_server_aarch64))
    // ...
)]
async fn my_test<T: PetriVmmBackend>(config: PetriVmBuilder<T>) -> anyhow::Result<()> {
    // ...
}

To ignore all variants of the test, apply ignore(reason = "...") as a test-wide
attribute alongside configs(...):

rust,ignore

#[vmm_test_with(ignore(reason = "needs functionality not yet implemented (microsoft/openvmm#1234)"), configs(
    hyperv_openhcl_uefi_aarch64(vhd(windows_11_enterprise_aarch64)),
    hyperv_openhcl_uefi_aarch64(vhd(ubuntu_2404_server_aarch64))
    // ...
))]
async fn my_test<T: PetriVmmBackend>(config: PetriVmBuilder<T>) -> anyhow::Result<()> {
    // ...
}

Nested Tests

Petri supports writing nested tests by defining a function to create the L1 VM host using Petri and annotating the L2 test definition as follows:

rust,ignore

#[vmm_test(
    nested(
        (my_vm_host, l1_config),
        (NEXTEST_ARCHIVE_WITH_NESTED_TEST, "test_binary", "test::module::path", l2_config),
    ),
)]
async fn my_test<T: PetriVmmBackend>(config: PetriVmBuilder<T>) -> anyhow::Result<()> {
    // ...
}

async fn my_vm_host<T: PetriVmmBackend>(config: PetriVmBuilder<T>) -> anyhow::Result<()> {
    let (vm, agent) = config
        .with_nested_virt()
        // other l1 configuration
        .run().await?;

    // use the pipette agent to configure the l1 if necessary

    // run the nested test, optionally with custom environment variables.
    vm.run_nested_test(&agent, Default::default()).await?;

    // power off and clean up
    agent.power_off().await?;
    vm.wait_for_clean_teardown().await?;
    Ok(())
}

Running VMM Tests (Flowey)

The easiest way to run VMM tests locally is cargo xflowey vmm-tests-run. It
automatically discovers required artifacts, builds dependencies, and runs your
tests in a single command.

To run a specific test (or set of tests), use --filter with a
nextest filter expression:

bash

cargo xflowey vmm-tests-run --filter "test(my_test_name)" --dir /tmp/vmm-tests-run

Targeting a Platform

By default, vmm-tests-run builds for the current host. Use --target to
build for a different platform. The supported targets are:

Target	Description
`windows-x64`	Windows x86_64 (Hyper-V / WHP)
`windows-aarch64`	Windows ARM64 (Hyper-V / WHP)
`linux-x64`	Linux x86_64

Cross-compiling for Windows from WSL2 is fully supported, you can build
and run Windows VMM tests directly from your WSL2 shell. This requires the
cross-compilation environment to be set up first.

Then target Windows as usual:

bash

cargo xflowey vmm-tests-run --target windows-x64

Most tests work with the default output directory (on the WSL filesystem),
because their disk images are either streamed on demand or opened as plain
files (fixed VHD1, VMGS, ISO), which work fine over the \\wsl$ path. You only
need to override --dir to a Windows filesystem path (a DrvFs mount like
/mnt/d/...) when the selected tests use disk images that require a Windows
filesystem — namely Hyper-V tests (their VHDs are attached to a real
Hyper-V VM) or tests using VHDX / dynamic VHD1 images (opened via the
Windows virtual-disk mount API):

bash

cargo xflowey vmm-tests-run --target windows-x64 --dir /mnt/d/vmm_tests

vmm-tests-run detects this automatically and will tell you if --dir is
required for your selection.

For full cross-compilation setup instructions, see
Cross Compiling for Windows.

When running Hyper-V tests, your user account must be a member of the
Hyper-V Administrators group.

To see all available options: cargo xflowey vmm-tests-run --help.

Lazy Fetch (Default)

By default, VHD/ISO disk images are streamed on demand via HTTP and cached
locally in a SQLite database. This avoids multi-GB upfront downloads and
significantly speeds up the dev inner loop once the cache is warm.

Lazy fetch is automatically disabled for Hyper-V tests, which require local
files. To force all images to be downloaded upfront, pass --no-lazy-fetch:

bash

cargo xflowey vmm-tests-run --filter "test(my_test)" --dir /tmp/vmm-tests-run --no-lazy-fetch

Running VMM Tests (Manual)

admonish tip

Note: We recommend using [cargo-nextest](https://nexte.st/) to run unit / VMM
tests. It is a significant improvement over the built-in `cargo test` runner,
and is the test runner we use in all our CI pipelines.

You can install it locally by running: `cargo install cargo-nextest --locked`

See the [cargo-nextest](https://nexte.st/) documentation for more info.

You can directly invoke cargo test or cargo nextest to run the vmm
tests manually.

Unlike Unit Tests, VMM tests may rely on additional external artifacts in order
to run. e.g: Virtual Disk Images, pre-built OpenHCL binaries, UEFI / PCAT
firmware blobs, etc.

As such, the first step in running a VMM test is to ensure you have acquired all
external test artifacts it may depend upon.

The VMM test infrastructure does not automatically fetch / rebuild
necessary artifacts unless you are using flowey.
However, the test infrastructure is designed to report clear
and actionable error messages whenever a required test artifact cannot be found,
which provide detailed instructions on how to build / acquire the missing
artifact. Some dependencies can only be built on Linux (OpenHCL and Linux
pipette, for example). If you are building on Linux and want to run Windows
guest tests, pipette will need to be
cross compiled for Windows.

admonish warning

`cargo nextest run` won't rebuild any of your changes. Make sure you `cargo build`
or `cargo xflowey build-igvm [RECIPE]` first!

VMM tests are run using standard Rust test infrastructure, and are invoked via
cargo test / cargo nextest.

bash

cargo nextest run -p vmm_tests [TEST_FILTERS]

For example, to run a simple VMM test that simply boots using UEFI:

bash

cargo nextest run -p vmm_tests multiarch::openvmm_uefi_x64_frontpage

And, for further example, to rebuild everything* and run all* the tests
(see below for details on these steps):

*This will not work for Hyper-V tests. TMK tests need additional build steps.

bash

# Install (most) of the dependencies; cargo nextest run may tell you
# about other deps.
rustup target add x86_64-unknown-none
rustup target add x86_64-unknown-uefi
rustup target add x86_64-pc-windows-msvc
sudo apt install clang-tools lld

cargo install cargo-nextest --locked

cargo xtask guest-test download-image
cargo xtask guest-test uefi --bootx64

# Rebuild all, and run all tests
cargo build --target x86_64-pc-windows-msvc -p pipette
cargo build --target x86_64-unknown-linux-musl -p pipette

cargo build --target x86_64-pc-windows-msvc -p openvmm

cargo xflowey build-igvm x64-test-linux-direct
cargo xflowey build-igvm x64-cvm
cargo xflowey build-igvm x64

cargo nextest run --target x86_64-pc-windows-msvc -p vmm_tests --filter-expr 'all() & !test(hyperv) & !test(tmk)'

\[Linux] Cross-compiling `pipette.exe`

These commands might use the test agent (pipette) that is put inside the VM,
and if the host machine OS and the guest machine OS are different, a setup
is required for cross-building. The recommended approach is to use WSL2 and
cross-compile using the freely available Microsoft Visual Studio Build Tools
or Microsoft Visual Studio Community Edition as described in
[WSL2] Cross Compiling from WSL2 to Windows

If that is not possible, here is another option that relies on MinGW-w64
and doesn't require installing Windows:

bash

# Do 1 once, do 2 as needed.
#
# 1. Setup the toolchain
rustup target add x86_64-pc-windows-gnu
sudo apt-get install mingw-w64-x86-64-dev
mingw-genlib -a x86_64 ./support/pal/api-ms-win-security-base-private-l1-1-1.def
sudo mv libapi-ms-win-security-base-private-l1-1-1.a /usr/x86_64-w64-mingw32/lib

# 2. Build Pipette (builds target/x86_64-pc-windows-gnu/debug/pipette.exe first)
cargo build --target x86_64-pc-windows-gnu -p pipette

bash

# Run a test
cargo nextest run -p vmm_tests multiarch::openvmm_uefi_x64_windows_datacenter_core_2022_x64_boot

Printing logs for VMM Tests

In order to see the OpenVMM logs while running a VMM test, do the following:

Add the --no-capture flag to your cargo nextest command.
Set OPENVMM_LOG=trace, replacing trace with the log level you want to view.

Guide Src Dev Guide Tests Vmm Perf (`Guide/src/dev_guide/tests/vmm_perf.md`)

VMM.Perf Runner

vmm_perf uses Microsoft's VirtualClient workload runner to execute
packaged performance profiles against an OpenVMM build and collect their
metrics and diagnostic files.

When to use VMM.Perf

Use VMM.Perf for the standardized fio, iperf3, and boot-time workloads
assembled by the performance pipeline. It is suitable for repeatable local or
CI runs where the runtime archive and profile definitions must remain pinned.

Run through Flowey

The preferred command is:

bash

cargo xflowey vmm-perf

Flowey builds OpenVMM and vmm_perf, restores UEFI firmware, downloads the
pinned runtime archive, chooses a native target, and supplies all low-level
paths.

Run one profile:

bash

cargo xflowey vmm-perf --profile fio

Profile selections use WORKLOAD[:MODIFIER...]. Bare fio and iperf3
select their VMBus profiles for compatibility, while boot-time has no
default modifier. Select VIRTIO explicitly, or repeat --profile to run both
device variants:

bash

cargo xflowey vmm-perf \
  --profile fio:vmbus \
  --profile fio:virtio

Modifiers are colon-delimited ASCII alphanumeric tokens. Multiple ordered
modifiers select a profile with the same suffix order, such as
fio:ssh:virtio selecting FIO-SSH-VIRTIO, when that profile exists in the
pinned runtime.

The pipeline selects a GNU Linux x64 target on Linux and Windows x64 on
Windows. A static Linux runner can be selected with --target linux-x64-musl
when required by the host environment.

VM sizing

The default configuration requests 16 virtual processors and 64 GiB of guest
memory. The runner validates requested CPU and memory against host capacity and
fails before starting a workload when the host is too small.

Select smaller or additional shapes through Flowey:

bash

cargo xflowey vmm-perf \
  --profile fio \
  --vmm-perf-vmsizes 'CpuCount=2,MemoryMB=4096' \
  --vmm-perf-vmsizes 'CpuCount=8,MemoryMB=16384'

Each shape becomes a separate named configuration. Running multiple profiles
multiplies the number of VirtualClient executions by the number of shapes.

Direct runner interface

Direct invocation is useful when a pipeline has already produced all inputs:

bash

vmm_perf \
  --openvmm path/to/openvmm \
  --firmware path/to/MSVM.fd \
  --runtime-archive path/to/vmm-perf-runtime.tar.gz \
  --output-dir path/to/results \
  --profile fio

The required inputs are:

An OpenVMM executable for the current host.
Matching UEFI firmware.
A supported VirtualClient runtime archive.
A destination for retained diagnostics.

--temp-dir changes the scratch root. --vm-sizes-json accepts an array of
named or unnamed configuration objects, while --parameters-json applies
additional scalar VirtualClient parameters to every selected configuration.

For example:

bash

vmm_perf <REQUIRED_PATH_OPTIONS> \
  --vm-sizes-json \
  '[{"name":"small","parameters":{"CpuCount":2,"MemoryMB":4096}}]'

Use raw parameter overrides only when you understand the selected VirtualClient
profile. They can change the workload as well as the VM shape.

Execution lifecycle

For each profile and configuration, the runner:

Validates input files and host capacity.
Extracts or reuses a runtime cache adjacent to the archive.
Verifies that the platform-specific profile exists in that runtime.
Creates isolated temporary, work, and output directories.
Constructs a VirtualClient command with OpenVMM, firmware, shape, and
metadata parameters.
Executes one VirtualClient iteration.
Restores file ownership where required and gathers diagnostics.

The runner continues through remaining profile/configuration pairs after an
individual failure, then returns one combined failure summary.

Output layout

Flowey defaults to target/vmm_perf, with scratch files under t and retained
results under results. Each profile/configuration output can contain:

results/ for workload metrics.csv files.
openvmm-logs/ for collected VMM logs.
virtual-client/logs/ for console.log, metrics.csv, vc.metrics,
vc.traces, and split metric files.
virtual-client/runtime/ for publishable files emitted by the runtime.

A successful VirtualClient process that does not produce vc.metrics is
treated as a failed run.

Troubleshooting

A capacity error should be fixed by selecting a smaller VM shape, not by
bypassing the check.
A missing profile usually means the runtime archive does not match the
runner's OS, architecture, or expected package version.
An archive error can indicate an unsupported extension or multiple runtime
roots inside the archive.
Read virtual-client/logs/console.log for workload startup failures and the
collected OpenVMM logs for VM failures.
On Linux, ownership restoration errors can follow a workload that created
root-owned output; retain the first workload error as the primary signal.

Guide Src Reference Architecture Devices (`Guide/src/reference/architecture/devices.md`)

Device architecture

This section covers the internal architecture of device emulators and
their backends — the shared machinery that both OpenVMM and OpenHCL
use to connect guest-visible storage, networking, and other devices to
their backing implementations.

OpenHCL Architecture

Prerequisites:

Overview

OpenHCL is a paravisor execution environment that runs within the guest partition of a virtual machine. It provides virtualization services to the guest OS from within the guest partition itself, rather than from the host as is traditionally done.

The following diagram offers a brief, high-level overview of the OpenHCL Architecture.

Virtual Trust Levels (VTLs)

OpenHCL relies on Virtual Trust Levels (VTLs) to establish a security boundary between itself and the guest OS.

VTL2: OpenHCL runs here[^sk]. It has higher privileges and is isolated from VTL0.
VTL0 (and sometimes VTL1): The Guest OS (e.g., Windows, Linux) runs here. It cannot access VTL2 memory or resources.

This isolation is enforced by the system configured by the underlying virtual machine monitor (Hyper-V) and can be backed by:

Hardware TEEs, like Intel TDX and AMD SEV-SNP.
Software-based constructs, like Hyper-V VSM.

Scenarios

OpenHCL enables several key scenarios by providing a trusted execution environment within the VM:

Azure Boost

OpenHCL acts as a compatibility layer for Azure Boost. It translates legacy synthetic device interfaces (like VMBus networking and storage) used by the guest OS into the hardware-accelerated interfaces (proprietary Microsoft Azure Network Adapter (MANA) and NVMe) provided by the Azure Boost infrastructure. This allows unmodified guests to take advantage of next-generation hardware.

The following diagram shows a high level overview of how synthetic networking is supported in OpenHCL over Microsoft Azure Network Adapter (MANA)

The following diagram shows a high level overview of how accelerated networking is supported in OpenHCL over MANA

Confidential Computing

In Confidential VMs (CVMs), the host is not trusted. OpenHCL runs inside the encrypted VM context (VTL2) and provides necessary services (like device emulation and TPM) that the untrusted host cannot securely provide. Security-sensitive devices, such as the virtual TPM, can be further isolated by running them in separate worker processes within VTL2 for defense-in-depth protection.

Trusted Launch

OpenHCL hosts a virtual TPM (vTPM) and enforces Secure Boot policies, ensuring the integrity of the guest boot process.

Architecture Components

OpenHCL is built on top of a specialized Linux kernel and consists of several userspace processes that work together to provide these services.

For more details on the internal components and selected OpenHCL device paths, see:

[^sk]: Why not VTL1? Windows already uses VTL1 in order to host the Secure Kernel.

Guide Src Reference Architecture Openvmm (`Guide/src/reference/architecture/openvmm.md`)

OpenVMM Architecture

This section describes the architecture of OpenVMM when it runs as a hosted
VMM.

Memory Layout describes the guest physical
address space.
Memory Backing explains how guest RAM is
allocated and shared.
NUMA Topology covers guest NUMA configuration and host
memory placement.
Device Tree describes the device trees that
OpenVMM builds for IGVM files and ARM64 Linux direct boot.
Timekeeping explains clocks, timers, and their
behavior across VM lifecycle changes.
mesh describes OpenVMM's inter-process communication
framework.

Guide Src Reference Architecture Timekeeping (`Guide/src/reference/architecture/timekeeping.md`)

Timekeeping

This page explains the clocks and timers used by OpenVMM and OpenHCL, how
they relate to guest-visible time, and what happens when execution stops.

There is no single "VM clock". A guest can read a processor counter, read a
hypervisor reference counter, ask firmware for the date, and receive a
timesync message. These interfaces have different origins, owners, and
pause behavior. A timer also has state beyond the clock it reads: its
deadline, periodic phase, and any interrupt or message awaiting delivery.

Clock domains

Domain	Representation	Purpose
Host monotonic time	`pal_async::timer::Instant`, nanoseconds	Scheduling host tasks and implementing elapsed-time clocks. Not a date.
Host system time	`SystemTime` or `jiff::Timestamp`	Calendar time, subject to host clock adjustments.
Software VM time	`vmcore::vmtime::VmTime`, 100 ns ticks	Pausable timeline used by software device emulation.
Partition reference time	`ReferenceTimeSource`, 100 ns ticks	Hyper-V reference-counter reads, synthetic timer deadlines, and correlation with UTC. Its source depends on the backend.
Architectural counters	x86 TSC; Arm physical/virtual counters	Guest instruction-visible counters, with architecture-specific frequencies and virtualization offsets.
Device wall-clock time	`LocalClockTime`, milliseconds since the Unix epoch	Reprogrammable calendar time for RTC and firmware services.
Guest OS time	Maintained by the guest	The OS derives monotonic and wall-clock time from available counters and synchronization services.

Equal units do not imply equal clocks. In particular, VmTime and
partition reference time both use 100 ns ticks, but generally have
different origins. TSC ticks are not nanoseconds or CPU instruction cycles;
you need the counter's frequency to convert them to a duration.

Here, "host" means the OS running the relevant userspace component. For
hosted OpenVMM that is the host OS; for OpenHCL userspace it is the Linux
environment in VTL2, not the root partition.

text

Host monotonic clock
  +-- host task deadlines (PolledTimer)
  +-- software VmTime, with a running/stopped mapping
        +-- emulated device counters and timer deadlines
        +-- Hyper-V reference time in some emulated configurations

Architectural counter / hypervisor clock
  +-- guest TSC or Arm counter
  +-- partition reference time in native and counter-backed configurations

System time / platform-provided UTC
  +-- LocalClock --> RTC or UEFI calendar time
  +-- UTC + partition-reference sample --> timesync message --> guest OS

The branches describe possible sources, not universal wiring. Clock
offsets and scales cannot be inferred from the counters' numeric values.

Software VM time

VmTimeKeeper owns a software timeline. Hosted OpenVMM creates it
stopped at zero. While running, its value is the VM time at the last start
plus elapsed host monotonic time. Stopping captures a fixed value;
starting again establishes a new host-time anchor without adding the
stopped interval. Reset sets it to zero, and restore installs a saved VM
time rather than an old host timestamp.
This measures elapsed time, not scheduled VP runtime: it advances while
guest CPUs are idle or descheduled unless the keeper is stopped.

The underlying pal_async clock uses CLOCK_MONOTONIC on
Unix and QueryUnbiasedInterruptTimePrecise on Windows. A VM pause,
host-system sleep, and guest CPU idle are different events: VM pause is
handled by the keeper, while host sleep behavior follows the underlying
OS clock. Do not use these timestamps as UTC.

Devices obtain a VmTimeAccess from a VmTimeSource. Each accessor can
read time and register a deadline; the backing task translates VM
deadlines into host timer deadlines while running. This lets a virtual
device wait without advancing its guest-visible timeline during a pause.
An overdue deadline still needs the consumer to run and process it:
deadline expiration is not the same as guest interrupt delivery.

VmTimeSourceBuilder provides synchronized cross-process access over
mesh, but requires a shared OS monotonic clock. It cannot synchronize
different machines or an OpenVMM host with OpenHCL. Stop reconciles the
secondary keepers to a common value without moving accessors backward.

The keeper saves the timeline value; devices save their own timer state.
Monotonicity does not extend across reset or restore of an older snapshot.

Partition reference time and processor counters

ReferenceTimeSource is a read interface, not a clock
controller. A read returns a reference count and may also return a UTC
timestamp captured at the same instant, allowing the two values to be
correlated. The reference count is not itself UTC. A missing UTC value means
that the source cannot cheaply provide the correlated timestamp.

For Hyper-V-compatible guests, the reference counter is a partition-wide
elapsed-time interface. On x86, guests can read HV_X64_MSR_TIME_REF_COUNT.
On Arm, the corresponding synthetic register is HvRegisterTimeRefCount,
accessed through a hypercall when the backend supports that interface.
The reference-TSC page provides an alternative that avoids an intercept:

text

reference_time = high64(virtual_tsc * scale) + offset

The guest must follow the sequence-number protocol. A zero
sequence requires fallback to the reference counter: advertising the
facility does not mean that the page is usable at every instant.

The shared hv1_emulator accepts an explicit reference-time source. It
publishes a usable reference-TSC page only when configured for its
TSC-backed, zero-offset model; otherwise it leaves the sequence invalid.
A VmTime-backed reference counter therefore does not automatically
provide a direct TSC fast path.

Architectural counters have their own state. Changing reference time
does not inherently change x86 RDTSC, an Arm counter offset, or a timer
deadline expressed in counter ticks. Similarly, stopping a VP thread or
leaving a hypervisor run call does not by itself freeze those counters.

Timer implementations

Choose the clock by the guest-visible contract, not by whichever timer
API is easiest to call.

Timer or counter	Implementation and clock
PIT	`chipset::pit` advances counting elements from elapsed `VmTime` and schedules wakeups for interrupt generation.
ACPI PM timer	Software reads scale `VmTime` to 3.579545 MHz. With PM timer assist, the hypervisor handles reads using its own time source instead.
CMOS RTC	Calendar registers use `LocalClock`; alarm, periodic, and update interrupt scheduling uses `VmTime`.
UEFI watchdog	`watchdog_core` receives a `VmTimeAccess`, so its deadline belongs to the software VM timeline.
Software local APIC	`virt_support_apic` counts down using `VmTime`; backend-native APIC emulation instead has backend-owned timer state.
Hyper-V synthetic timers	Deadlines use partition reference time. Native implementations belong to the hypervisor; `hv1_emulator::synic` evaluates them in software.
Arm architectural timers	Compare architectural counter values with programmed deadlines; backend-specific code handles virtualization and interrupt delivery.
Host work timers	`PolledTimer` waits on host monotonic time. It has no implicit association with VM pause.

Synthetic timers can be one-shot or periodic. In the shared emulator, a
one-shot count is an absolute reference-time deadline; a periodic count
is an interval. Expiration can request a direct interrupt or post a
SynIC message. Masking, a busy message slot, and VP scheduling can delay
delivery after the deadline.

This distinction matters during save/restore. Restoring a clock value
alone does not restore a periodic timer's phase, retract an already
pending interrupt, or preserve an undelivered message. Conversely,
blindly reloading a timer can discard events that were already pending.
Backend state interfaces and device saved state must be considered
together.

Calendar time and synchronization

RTC and UEFI

LocalClock is an instance-local, reprogrammable calendar
clock. Setting it changes that instance, not the host's system clock.
Unlike VmTime, its intended behavior is to account for elapsed
wall-clock time while the VM is paused.

Hosted RTC devices use SystemTimeClock, which returns host
SystemTime plus an offset. Guest writes change that offset. Host
calendar-clock adjustments, including backward jumps, remain visible.
The default UEFI time source is also a SystemTimeClock; independently
created clock instances do not share guest-written offsets.

RTC dates can advance during a pause while VmTime-based interrupt
machinery is stopped. Periodic interrupt rates do not define wall time.

On Arm, the UEFI time service exposes the supplied local clock through
GetTime/SetTime. It retains timezone and daylight fields for readback
but does not apply them as timezone conversions. Its saved state stores
those fields; persistence of the backing clock is the platform's
responsibility. Similarly, CMOS register saved state is not a serialized
LocalClock implementation.

Hyper-V timesync

The hyperv_ic timesync device sends a UTC sample paired with partition
reference time. The pair allows the guest to account for elapsed
reference time between sampling and consuming the message. This is
separate from setting the RTC or changing the partition clock.

It uses correlated UTC from ReferenceTimeSource when available;
otherwise it samples local system time immediately after reference time.
It waits for ring-buffer space before sampling to reduce delivery delay.

The current implementation negotiates timesync version 4, sends an
initial synchronization message, and then sends samples on a five-second
host-timer cadence while the channel runs. On snapshot restore it
renegotiates and sends a fresh synchronization message. Whether and how
the guest OS adjusts its time is a guest policy decision.

After resume, elapsed time can exclude the pause while RTC reads and
timesync report the current date. Resynchronizing calendar time does not
mean a frozen elapsed-time clock was restored incorrectly.

Stop, resume, and saved state

The optional PartitionTimeControl interface controls
backend time separately from VmTimeKeeper. Supporting backends create
partitions frozen. The lifecycle contract requires stopped VPs for both
operations, makes repeated freeze/thaw calls harmless, and treats an
unexpected transition failure as fatal inside the backend.

For hosted OpenVMM, the relevant full-stop ordering is:

text

Stop:   stop all VPs --> freeze supported backend time
                    --> stop dependent device work --> stop VmTime
Start:  start VmTime --> start dependent device work
                    --> thaw supported backend time --> allow VPs to run

The state-unit dependency graph establishes this ordering; it is not an
atomic freeze of every clock. Backend time and software device time can
have different stop/start anchors.

Operation	Timekeeping consequence
Full VM stop	Stops software VM time and requests backend freeze where supported. Does not freeze host clocks or calendar time.
Temporary VP stop or debugger halt	Does not itself stop `VmTime` or freeze backend time. Timers can become due while VPs cannot execute.
Full VM start	Thaws backend time even if a debugger halt or outstanding temporary-stop guard still prevents VP execution.
Reset	Resets software VM time to zero. Supporting backends remain frozen while partition and VP state are reset.
Snapshot restore	Restores stopped software time and the state supported by each backend/device. A time-state setter must not implicitly thaw a supporting backend.
VTL scrub	May reset and freeze a backing VTL clock. The partition unit thaws it before restarting VPs; this is not a freeze of every VTL.

The x86 virt model separates reference time/page, TSC, APIC, and TSC deadline.
Deadline state requires guest/backend support and restores after APIC/TSC:
zero disarms; absent state is not restored. Native MSHV/WHP and KVM support
it, unlike software APICs. Reference time is included only when Hyper-V
enlightenments are enabled; lifecycle freezing cannot be hidden in its setter.
Lifecycle freeze is separate; can_freeze_time gates exact state comparison.
A reference-clock-only implementation is not sufficient to claim that
capability.

Arm state coverage differs: the generic virt partition-state schema is
empty and its VP schema does not currently include architectural
counter/timer state. A native freeze API is not, by itself, evidence of
complete portable snapshot support.

Hosted backend configurations

These are current implementations, not uniform virt guarantees.
Guest-visible Hyper-V facilities require the relevant enlightenments.

Backend/configuration	Reference-time source	Backend freeze on full stop
MSHV, x86 or Arm	Hyper-V partition `ReferenceTime` property	Yes, through `TimeFreeze`.
WHP, offloaded Hyper-V	Native WHP reference time	Yes, separately for each backing partition.
WHP, emulated Hyper-V on x86	Software `VmTime`	Yes for native WHP time; `VmTime` stops separately.
KVM, x86	KVM clock, converted from ns to 100 ns	No; stopping software VM time does not freeze KVM time.
KVM, Arm	No Hyper-V reference-time source exposed	No; architectural timers remain KVM-owned.
HVF, Arm	Software `VmTime`	No architectural-counter freeze; software time still stops.

MSHV and WHP reference-time sources return no correlated UTC timestamp.
KVM x86 returns one when KVM_GET_CLOCK supplies KVM_CLOCK_REALTIME.
HVF and WHP's software source return only the VM-time count.

MSHV and WHP

virt_mshv uses native Hyper-V time and interrupt-controller facilities.
It reads reference time through a partition property rather than a VP
register, avoiding a query that would need to wait for a running VP.
Creation freezes the partition before guest initialization; runtime
freeze/thaw updates TimeFreeze. Guest reference-TSC-page exposure is
capability-dependent and is currently disabled for MSHV SNP partitions.
That restriction is separate from freezing the partition.

For non-isolated VM reset, virt_mshv uses MSHV_SCRUB_PARTITION when the
kernel implements it, then explicitly reasserts TimeFreeze. An ENOTTY
response selects the existing state-by-state reset path; other scrub
errors fail reset. Per-VP user-mode state is reset on both paths, while a
successful scrub replaces bulk hypervisor-state writes.

virt_whp offloads Hyper-V facilities when offload is enabled, the
required WHP features are available, and a user-mode APIC is not selected.
Otherwise, its x86 software path uses hv1_emulator with a
VmTime-backed reference counter and software synthetic timers. The
user-mode APIC also uses VmTime. Arm does not support the user-mode-APIC
or disabled-offload options; its virtual timer PPI is configured in WHP's
GIC parameters.

WHP uses WHvSuspendPartitionTime and explicit resume,
not the implicit resume performed by entering a VP. With VTL2 emulation,
VTL0 and VTL2 have distinct WHP backing partitions and frozen-state
bookkeeping. A full stop freezes both. Scrubbing VTL2 freezes only its
backing clock and, on x86, preserves its reference count around native
reset. This simulation excludes the time spent in that reset; it is not
an exact model of servicing downtime.

The WHP x86 partition-state accessors still have TODOs for locally
emulated Hyper-V state: they access native WHP state even in that
configuration. Do not infer full emulated-enlightenment snapshot
coverage from the presence of the freeze interface.

KVM

On x86, KVM owns the virtual TSC, LAPIC timers, and enabled Hyper-V
synthetic timers. OpenVMM reads the KVM clock for its reference-time
source. Saving reference time rounds the nanosecond clock upward to
100 ns so conversion does not move it backward on restore. Setting
reference time uses KVM_SET_CLOCK with flags zero; it does not request
KVM's optional UTC-based addition of elapsed downtime.

That setter rebases the KVM clock; it does not freeze it, rewind raw
guest TSC, or undo a LAPIC timer expiration. KVM can record timer work
while userspace has stopped running VPs. The backend currently exposes
no PartitionTimeControl, and its x86 can_freeze_time is false.
Rebasing reference time alone would not implement full counter and
timer freeze semantics.

KVM's synthetic-timer snapshot path saves config/count MSRs, but cannot
preserve timer adjustment or undelivered expiration-message state.

On Arm, KVM manages architectural counters/timers and the in-kernel
interrupt controller. OpenVMM configures the virtual timer PPI from the
topology; the physical timer PPI required by KVM is not advertised to
the guest. There is no Hyper-V reference-time source or SynIC support
in this backend, and no software pause compensation for the counters.

HVF

On macOS/Arm, HVF's synthetic reference counter and software SynIC timers
use VmTime. Architectural counters use a different basis:
CNTPCT_EL0 reflects the host counter and CNTVCT_EL0 reflects
mach_absolute_time() minus HVF's virtual-timer offset. The backend does
not adjust that offset to remove VM pause duration.

For a VP waiting after WFI, the backend converts the remaining
architectural virtual-timer interval into a VmTime wakeup. It then
re-enters HVF, whose timer-activation exit raises the emulated GIC PPI.
Using VmTime to arrange this wakeup does not make the architectural
counter itself a VmTime clock.

OpenHCL: time inside a partition

OpenHCL runs in VTL2 inside a partition whose lifecycle is controlled
outside OpenHCL. Stopping lower-VTL execution or servicing OpenHCL is not
equivalent to freezing that surrounding partition. Its VmPartition
wrapper therefore leaves the backend time-control hooks as no-ops.

OpenHCL does have its own VmTimeKeeper for software device emulation.
It starts at zero, participates in state-unit stop/start, and has saved
state. Servicing also records partition reference time at stop and uses
it to report blackout duration on restart. That measurement is separate
from restoring the software VM-time value; it is not an instruction to
rewind the surrounding partition clock or add downtime to VmTime.

For non-hardware-isolated configurations, virt_mshv_vtl obtains native
reference time through HCL's VTL2 TimeRefCount register access. In x86
SNP and TDX configurations, it emulates lower-VTL Hyper-V facilities with
a reference source derived from RDTSC and a frequency-based scale.
This source is not VmTime. The reference-TSC page uses the same
zero-offset model. The frequency comes from the hypervisor; TDX also
validates it against hardware-advertised frequency information.

The emulated SynIC compares deadlines against that reference source.
To wait using software VM time, the implementation converts a relative
interval, next_reference_time - current_reference_time, into a
VmTime deadline instead of equating the clocks' origins. SNP also
reconciles kernel-handled STIMER0 writes with the userspace SynIC;
kernel expiry wakes VTL2, while the SynIC scan performs delivery. TDX
uses an L2 virtual-TSC execution deadline when supported, with a
VmTime-based fallback. The shared software APIC timer still uses
VmTime; hardware acceleration and interrupt delivery are distinct
from the reference-clock choice. The x86 CVM TSC model must not be
assumed to describe Arm hardware-isolated configurations.

OpenHCL's calendar clock is different again. UnderhillLocalClock
queries root-provided time over Guest Emulation Transport, caching
samples for up to one second, and adds a guest-programmable offset.
It persists the offset and saves it for servicing; without a saved
offset it uses the host-provided timezone offset. Thus RTC calendar
values need not represent UTC even though the internal type uses an
epoch-based numeric representation.

admonish warning

Clock availability does not establish trust in UTC. In a confidential VM,
root-provided calendar time is an external input, not authenticated time.
A processor-derived reference counter is not a trusted calendar clock
either. These mechanisms do not by themselves provide rollback-resistant
timestamps or guarantees about host scheduling and timer delivery.

Implementation map

Start with these repository paths:

vm/vmcore/src/{vmtime,reference_time}.rs: timelines and accessors.
vmm_core/src/{partition_unit,vmtime_unit}.rs: lifecycle operations;
openvmm/openvmm_core/src/worker/dispatch.rs: hosted dependency wiring.
vm/hv1/hv1_emulator/src/{hv,synic}.rs: reference pages and timers.
vmm_core/virt_{mshv,whp,kvm,hvf}/src/: backend state and VP loops.
vm/devices/chipset/src/ (pit, pm, cmos_rtc.rs) and
support/local_clock/src/: device timers and calendar clocks.
vm/devices/hyperv_ic/src/timesync.rs: UTC/reference-time samples.
openhcl/virt_mshv_vtl/src/: native and confidential-VM time sources;
openhcl/underhill_core/src/emuplat/local_clock.rs: calendar time.

See also snapshot behavior and
the snapshot format.

Guide Src Reference Backends Consomme (`Guide/src/reference/backends/consomme.md`)

Consomme

Consomme is a user-mode NAT that provides guest VM networking through
ordinary host sockets — no driver installation and no system-wide
network configuration. Because all traffic flows through standard
socket APIs, host networking policies (firewalls, VPN routing, proxy
settings) apply naturally, just as they would for any other application.

The guest sees a normal network: it gets an IP address via DHCP, has a
default gateway, and can resolve DNS names. Behind the scenes, Consomme
intercepts the raw Ethernet frames from the virtual NIC, parses them,
and translates them into socket calls on the host.

mermaid

flowchart TB
    subgraph VM ["Guest VM"]
        App["Application"] --> GuestStack["TCP/IP Stack"] --> VNIC["Virtual NIC"]
    end

    subgraph UserMode ["Host User Mode"]
        VNIC -- "Ethernet frames" --> Parse["Parse & Dispatch"]
        Parse --> TCPProxy["TCP proxy"]
        Parse --> UDPProxy["UDP proxy"]
        Parse --> ICMPFwd["ICMP forwarder"]
        Parse --> DNSFwd["DNS forwarder"]
        Parse --> DHCPSrv["DHCP / ARP / NDP"]
        TCPProxy & UDPProxy & ICMPFwd --> Sockets["Host sockets"]
        DNSFwd -- "DNS API call" --> HostDNS["Platform DNS resolver"]
        DHCPSrv -. "replies directly to guest" .-> VNIC
    end

    subgraph Kernel ["Host Kernel"]
        Sockets --> HostStack["Host TCP/IP stack + firewall + VPN"]
        HostDNS --> HostStack
        HostStack --> NIC["Physical NIC"]
    end

    NIC --> Internet((Internet))

Default network topology

Role	IPv4 Address
Guest	`10.0.0.2`
Gateway	`10.0.0.1`
Subnet mask	`255.255.255.0`

The subnet is configurable via the --net consomme:<cidr> CLI option.
The gateway is always .1 and the guest is always .2.

Port forwarding

By default, Consomme only provides outbound connectivity from the guest.
To accept inbound connections, use the hostfwd= option to forward
host ports into the guest:

bash

--net consomme:hostfwd=tcp::2222-:22

This binds host port 2222 and forwards incoming TCP connections to
guest port 22. The general syntax is:

text

hostfwd=<proto>:[<hostaddr>]:<hostport>-[<guestaddr>]:<guestport>

Field	Description
`proto`	`tcp` or `udp`
`hostaddr`	Host IP to bind (default: all IPv4 interfaces, `0.0.0.0`). Use `[addr]` brackets for IPv6 addresses
`hostport`	Host port to listen on. Use `0` for any available port
`guestaddr`	Currently ignored — traffic always forwards to the guest IP
`guestport`	Guest port to forward to

Multiple forwards can be specified with commas:

bash

--net consomme:hostfwd=tcp::2222-:22,hostfwd=tcp::3389-:3389

Port forwards can also be combined with a CIDR:

bash

--net consomme:10.0.0.0/24,hostfwd=tcp::2222-:22

Examples:

bash

# Forward host port 8080 to guest port 80
--net consomme:hostfwd=tcp::8080-:80

# Bind only to localhost
--net consomme:hostfwd=tcp:127.0.0.1:8080-:80

# IPv6 localhost
--net consomme:hostfwd=tcp:[::1]:8080-:80

# UDP forwarding
--net consomme:hostfwd=udp::5353-:53

IPv6 is enabled when the host has a routable IPv6 address. Consomme
advertises a prefix via SLAAC and the guest auto-configures its own
address. IPv6 DNS servers are advertised via RDNSS (in Router
Advertisements) and DHCPv6 Information-Request.

Protocol handling

TCP

Consomme maintains a full TCP state machine on the guest-facing side
and opens a real host socket per connection. Data flows through ring
buffers between the two sides. This is a split-TCP design: there are
two independent TCP connections (guest↔Consomme and host↔remote server)
stitched together.

Key implications of the split-TCP design:

Congestion control is disconnected from the real network. The
guest sees a virtual link with zero packet loss and sub-microsecond
RTT. Its congestion window opens wide almost immediately. The host
kernel's TCP stack handles all real congestion and retransmission.
Another TCP stack to debug. When a connection stalls, there are
three TCP implementations in play: the guest's, Consomme's, and the
host kernel's. Diagnosing which one is misbehaving requires visibility
into Consomme's internal state.

Consomme supports TCP Segmentation Offload (TSO) from the guest NIC,
window scaling, and MSS negotiation.

The per-connection ring buffers autotune. Each connection starts with a
small buffer (16 KiB by default) and grows on demand — doubling under
sustained throughput pressure up to a ceiling (4 MiB by default) — so
idle and short-lived connections stay cheap while bulk transfers can
ramp up to a large window. Embedders can override the bounds via the
tcp_rx_buffer and tcp_tx_buffer fields on ConsommeParams (each a
TcpBufferBounds { initial, max }); the values are clamped to
[16 KiB, 4 MiB] and rounded up to a power of two. Setting
initial == max pins a fixed size and disables growth.

Connections waiting for the peer to advance a TCP close handshake use a
60-second inactivity timeout by default, including connections closed before
their initial handshake completes. CloseWait is excluded because the local
backend can still produce data; its timeout starts when the backend closes and
the connection enters LastAck. ACKs and newly accepted data restart the
timeout. Embedders can override it with the tcp_close_timeout field on
ConsommeParams. The same duration is used for TimeWait, where it
represents the 2*MSL wait required by RFC 9293. A separate fixed 60-second
timeout bounds initial handshakes that receive no close request.

Inbound connections require explicit port forwarding — OpenVMM can
programmatically bind host ports and forward them into the guest.

UDP

Each unique guest source socket (guest IP + source port) gets a
host-side UDP socket bound to an ephemeral port. Datagrams are
forwarded in both directions. Idle bindings are cleaned up after a
configurable timeout (5 minutes by default, per RFC 4787).

UDP packets to the gateway on well-known ports are intercepted:

Port 67 — handled by the built-in DHCP server
Port 53 — handled by the DNS forwarder

UDP segmentation offload and receive coalescing are not currently
supported. Each datagram requires a separate socket call, which may
matter for QUIC/HTTP/3 performance.

DNS

The guest is told (via DHCP) to use the gateway as its DNS server.
Queries to the gateway on port 53 are intercepted and forwarded to the
host's DNS resolver. On Windows, this uses the DnsQueryRaw API, which
forwards the raw DNS packet without parsing — the host resolver handles
all the logic and returns a raw response. On Unix-like hosts, Consomme
uses the system resolver configuration (for example /etc/resolv.conf)
and forwards queries via host UDP/TCP sockets. This means guest DNS
resolution matches the host's behavior, including corporate VPNs and
split-tunnel DNS configurations.

DNS over TCP is also supported for large responses.

ICMP

Echo requests (ping) are forwarded through a host-side ICMP socket.
The guest's TTL is preserved, so ping works and traceroute partially
works (the echo reply comes back, but intermediate TTL Exceeded hops
are not relayed — see limitations).

ARP and NDP

ARP — Responds to requests for the gateway MAC address. All other
ARP traffic is dropped.
NDP — Responds to Router Solicitations (advertising the IPv6
prefix and DNS servers) and Neighbor Solicitations for the gateway's
link-local address. DAD is silently ignored.

DHCP

Built-in DHCPv4 server assigns the guest its IP address, gateway, and
DNS servers. Lease duration is 24 hours. DHCPv6 Information-Request is
also handled (stateless — provides DNS servers to SLAAC-configured
guests).

Checksum and segmentation offload

Consomme supports the following offloads from the guest NIC:

IPv4 header checksum offload
TCP checksum offload
UDP checksum offload
TCP Segmentation Offload (TSO)

Limitations

No multicast or broadcast

Beyond ARP, DHCP, and the NDP multicast messages required for IPv6,
Consomme only processes unicast traffic. Protocols that rely on
multicast discovery do not work:

mDNS / Bonjour — printer, AirPlay, Chromecast discovery
SSDP / UPnP — device discovery, NAT-PMP port mapping
LLMNR — Windows local name resolution fallback
NetBIOS — legacy Windows browsing

No ICMP error relay

ICMP error messages (Destination Unreachable, TTL Exceeded, etc.) are
not forwarded to the guest:

TCP — unreachable hosts produce a TCP RST instead of an ICMP
error. The connection fails, but the guest can't distinguish "host
unreachable" from "connection refused."
UDP — Port Unreachable errors are silently swallowed.
Traceroute — TTL Exceeded replies from intermediate routers are
not relayed, so the hop-by-hop path is not visible.

No IP fragment reassembly

Fragmented IP packets from the guest are silently dropped. This only
affects UDP datagrams larger than ~1472 bytes. TCP is never affected
because the guest kernel segments at the TCP layer. On the host side,
fragmentation is handled transparently by the kernel socket layer.

No jumbo frames

MTU is fixed at 1500 bytes.

No ICMPv6 Echo

IPv6 ping does not work. IPv6 TCP and UDP work normally.

No non-TCP/UDP/ICMP protocols

GRE, ESP, SCTP, and other IP protocols are silently dropped. This
means VPNs that terminate inside the guest and use IPsec (ESP) or
PPTP (GRE) will not work. UDP-based VPNs like WireGuard work fine.
IKEv2 with NAT-T (UDP-encapsulated ESP on port 4500) also works.

VPNs running on the host work transparently — Consomme's traffic flows
through the host network stack like any other application.

IPv6 depends on host

IPv6 is only enabled if the host has a routable IPv6 address at VM
start time.

Static DNS records only support the `A` record type

Only A records are currently supported.
AAAA queries for names with a static A record return NOERROR with no
answers, rather than being forwarded to the upstream resolver.

Static DNS records don't support DNSSEC

All static DNS replies are returned with the AD flag unset.

Crate structure

Crate	Path	Purpose
`consomme`	`vm/devices/net/net_consomme/consomme/`	Core user-mode TCP/IP stack
`net_consomme`	`vm/devices/net/net_consomme/`	`net_backend::Endpoint` integration

Guide Src Reference Backends Networking (`Guide/src/reference/backends/networking.md`)

Networking backends

The networking backend system connects guest-facing NICs (frontends)
to host-side packet I/O (backends) through a shared trait interface
defined in the net_backend crate. This page explains how the
pieces fit together, how packets flow, and how to navigate the code.

Architecture overview

text

┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│ virtio_net   │  │   netvsp     │  │  gdma/bnic   │
│ (frontend)   │  │ (frontend)   │  │ (frontend)   │
└───────┬──────┘  └───────┬──────┘  └───────┬──────┘
        │                 │                 │
        │  &mut dyn BufferAccess            │
        │  (owned by frontend)              │
        │                 │                 │
        ▼                 ▼                 ▼
┌──────────────────────────────────────────────────┐
│  dyn Queue  (per-queue)                          │
│  poll_ready · rx_avail · rx_poll                 │
│  tx_avail · tx_poll                              │
└──────────────────────────────────────────────────┘
        ▲                 ▲                 ▲
        │                 │                 │
┌───────┴──────┐  ┌───────┴──────┐  ┌───────┴──────┐
│ TapQueue     │  │ConsommeQueue │  │  ManaQueue   │
│ DioQueue     │  │LoopbackQueue │  │ (hardware)   │
│   ...        │  │  NullQueue   │  │              │
└──────────────┘  └──────────────┘  └──────────────┘

There are three layers:

Frontend — the guest-visible NIC device (virtio_net,
netvsp, or gdma). Owns the BufferAccess implementation
(no Arc or Mutex needed — each queue is driven from a single
async task), translates between the guest-specific descriptor
format and the generic Queue interface, and drives the poll
loop.
Queue — a single TX/RX data path created by the backend.
Frontends interact with it entirely through the
Queue
trait. A device may have multiple queues for RSS.
Endpoint — a backend factory. One per NIC. The frontend calls
Endpoint::get_queues
when the guest activates the NIC and
Endpoint::stop
on teardown.

See the
net_backend rustdoc
for the full trait signatures and type definitions.

Packet flow

Transmit (guest → host)

The guest posts a TX descriptor (e.g. a virtio descriptor chain
or a VMBus RNDIS message).
The frontend reads the descriptor from guest memory, extracts any
offload metadata (checksum, TSO, USO), and builds a TxSegment array.
Each segment carries a guest physical address and a length — no
data is copied at this point.
The frontend calls queue.tx_avail(&mut pool, &segments). The
backend reads data directly from guest memory via
pool.guest_memory() and transmits it (e.g. writes to a TAP fd,
posts to hardware, or feeds it to a user-space TCP stack).
If the backend completes synchronously (tx_avail returns
sync = true), the frontend can immediately mark the descriptor
done. Otherwise, it polls tx_poll later for async completions.

Receive (host → guest)

The frontend pre-populates the backend with receive buffers by
calling queue.rx_avail(&mut pool, &buffer_ids).
When queue.poll_ready(cx, &mut pool) signals readiness, the
backend has received a packet. It writes the packet data into
guest memory through pool.write_packet(rx_id, metadata, data).
The frontend calls queue.rx_poll(&mut pool, &mut ids) to
collect the IDs of completed buffers, then delivers them to the
guest (e.g. by completing virtio descriptors or sending VMBus
completion packets).
The guest eventually returns the buffer, and the frontend recycles
it via rx_avail.

Guest memory access

The Queue interface works with guest physical addresses rather
than host buffers, giving each backend flexibility in how it
accesses packet data. The patterns fall into three categories:

GPA pass-through (hardware DMA). net_mana converts guest
physical addresses into IO virtual addresses (GuestMemory::iova)
and posts them as scatter-gather entries directly to GDMA hardware.
The NIC DMAs packet data to/from guest memory without any host-side
copy. This is the fastest path, but requires IOMMU mappings and
contiguous-enough buffers; when those conditions aren't met, MANA
falls back to bounce buffers.

Host-mediated copy. Software backends like net_consomme and
net_dio read TX data from guest memory with
GuestMemory::read_at, process or forward it, and write RX data
back with BufferAccess::write_packet. The data passes through
host memory, but the Queue interface avoids any extra copies
between the frontend and backend layers — the backend reads/writes
guest RAM directly.

Lifecycle

The frontend creates a
BufferAccess
implementation and one QueueConfig per queue.
It calls endpoint.get_queues(configs, rss, &mut queues).
It enters the poll loop: poll_ready → rx_avail / rx_poll /
tx_avail / tx_poll.
On shutdown, it drops the queues and calls endpoint.stop().

Backends

Backend	Crate	Transport	Platform
TAP	`net_tap`	Linux TAP device	Linux
DirectIO	`net_dio`	Windows vmswitch	Windows
Consomme	`net_consomme`	User-space TCP/IP stack	Any
MANA	`net_mana`	Azure hardware NIC (MANA/GDMA)	Linux
Loopback	`net_backend`	Reflects TX → RX	Any
Null	`net_backend`	Drops everything	Any

Frontends

Frontend	Crate	Guest interface
virtio-net	`virtio_net`	Virtio network device
netvsp	`netvsp`	VMBus synthetic NIC
GDMA/BNIC	`gdma`	MANA Basic NIC (emulated GDMA)

Wrappers

Wrappers implement Endpoint by delegating to an inner endpoint,
adding cross-cutting behavior:

PacketCapture (net_packet_capture) — intercepts rx_poll
and tx_avail to write PCAP-format packet traces. The capture
path reads packet data from guest memory via BufferAccess and
writes enhanced packet blocks to a ring buffer. Capture can be
toggled at runtime; when disabled, the wrapper adds only an atomic
load per call.
Disconnectable (net_backend) — supports hot-plug and
hot-unplug by swapping the inner endpoint at runtime.

RSS and multi-queue

When a frontend supports Receive Side Scaling (RSS), it passes
multiple QueueConfig entries and an RssConfig (hash key +
indirection table) to get_queues. The backend creates one Queue
per entry and uses the RSS configuration to steer incoming packets
to the appropriate queue. Each queue is driven independently by its
own async task.

Currently netvsp and net_mana support multi-queue; virtio_net
is limited to a single queue pair.

Guide Src Reference Backends Storage (`Guide/src/reference/backends/storage.md`)

Storage backends

Storage backends implement the
DiskIo
trait, the shared abstraction that all storage frontends use to read
and write data. A frontend holds a
Disk
handle and doesn't know what kind of backend is behind it — the same
frontend code works with a local file, a Linux block device, a remote
blob, or a layered composition of multiple backends.

Backend catalog

Backend	Crate	Wraps	Platform	Key characteristic
FileDisk	`disk_file`	Host file	Cross-platform	Simplest backend. Blocking I/O via `unblock()`.
Vhd1Disk	`disk_vhd1`	VHD1 fixed file	Cross-platform	Parses VHD footer for geometry.
VhdmpDisk	`disk_vhdmp`	Windows vhdmp driver	Windows	Dynamic and differencing VHD/VHDX.
VhdxDisk	`vhdx`	VHDX file	Cross-platform	Pure-Rust VHDX parser. Dynamic, fixed, and differencing.
BlobDisk	`disk_blob`	HTTP / Azure Blob	Cross-platform	Read-only. HTTP range requests.
BlockDeviceDisk	`disk_blockdevice`	Linux block device or file	Linux	io_uring, resize via uevent, PR passthrough. Default for raw files on Linux in both OpenHCL and OpenVMM.
NvmeDisk	`disk_nvme`	Physical NVMe (VFIO)	Linux/Windows	User-mode NVMe driver. Resize via AEN.
StripedDisk	`disk_striped`	Multiple Disks	Cross-platform	Stripes data across underlying disks.

Decorators

Decorators wrap another
Disk
and transform I/O in transit. Features compose by stacking decorators
without modifying the backends underneath.

Decorator	Crate	Transform
CryptDisk	`disk_crypt`	XTS-AES-256 encryption. Encrypts on write, decrypts on read.
DelayDisk	`disk_delay`	Adds configurable latency to each I/O operation.
DiskWithReservations	`disk_prwrap`	In-memory SCSI persistent reservation emulation.

Layered disks

A LayeredDisk
composes multiple layers into a single DiskIo implementation. Each
layer tracks which sectors it has; reads fall through from top to
bottom until a layer has the requested data. This powers the
memdiff: and mem: CLI options.

Two layer implementations exist today:

RamDiskLayer (disklayer_ram) — ephemeral, in-memory.
SqliteDiskLayer (disklayer_sqlite) — persistent, file-backed (dev/test only).

The storage pipeline page covers
the full architecture: how frontends, backends, decorators, and the
layered disk model connect, plus cross-cutting concerns like online
disk resize and virtual optical media.

Guide Src Reference Backends Vhdx (`Guide/src/reference/backends/vhdx.md`)

VHDX parser

The vhdx crate (vm/devices/storage/vhdx/) is a pure-Rust
implementation of the
VHDX format specification.
It supports dynamic, fixed, and differencing VHDX virtual hard disk
files on all platforms — no Windows APIs or kernel drivers required.

Features

Create and open VHDX files (read-only or writable)
Dynamic block allocation with four-priority free space management
Write-ahead log (WAL) for crash-consistent metadata updates
Sector bitmap tracking for partially-present (differencing) blocks
Block trim/unmap with multiple modes (file space, free space, zero,
transparent, soft-anchor removal)
Concurrent flush coalescing
Parent locator parsing for differencing disk chains

Architecture

A VHDX file stores a virtual disk as a collection of fixed-size data
blocks (default 2 MiB) tracked by a Block Allocation Table (BAT).
The crate's write path uses a three-stage pipeline for crash
consistency:

text

┌───────────┐   commit   ┌──────────┐   apply   ┌────────────┐
│   Cache   │ ──────────►│ Log Task │ ─────────►│ Apply Task │
│ (dirty    │  dirty     │ (WAL     │  logged   │ (final     │
│  pages)   │  pages     │  writer) │  pages    │  offsets)  │
└───────────┘            └──────────┘           └────────────┘

The cache accumulates dirty 4 KiB metadata pages (BAT entries,
sector bitmap bits). When the dirty count reaches a threshold or
flush() is called, pages are committed to the log task.
The log task writes WAL entries to the circular log region in
the VHDX file. On crash, replay_log() restores metadata from
the WAL.
The apply task writes logged pages to their final file offsets.

Backpressure is managed by a permit semaphore that limits in-flight
pages. A flush sequencer coalesces concurrent flush requests so at
most one file flush is in progress at a time.

Lifecycle

rust,ignore

// Create a new empty VHDX file.
create::create(&file, &mut params).await?;

// Open for writing.
let vhdx = VhdxFile::open(file)
    .block_alignment(2 * 1024 * 1024)
    .writable(&spawner)
    .await?;

// Resolve a read — returns file-level ranges.
let mut ranges = Vec::new();
let guard = vhdx.resolve_read(offset, len, &mut ranges).await?;
// ... perform file I/O at the returned offsets ...
drop(guard);

// Resolve a write — returns file-level ranges + I/O guard.
let mut ranges = Vec::new();
let guard = vhdx.resolve_write(offset, len, &mut ranges).await?;
// ... write data at the returned offsets ...
guard.complete().await?;

// Flush to stable storage.
vhdx.flush().await?;

// Clean close (clears log GUID).
vhdx.close().await?;

I/O model

The crate separates metadata I/O from payload I/O.

Metadata I/O (headers, BAT pages, sector bitmaps, WAL entries) is
handled internally through the AsyncFile trait — the caller provides
an AsyncFile implementation at open time and never thinks about
metadata again.

Payload I/O (guest data reads and writes) is the caller's
responsibility. resolve_read() and resolve_write() translate
virtual disk offsets into file-level byte ranges (ReadRange /
WriteRange). The caller performs its own data I/O at those offsets
using whatever mechanism it prefers (io_uring, standard file I/O,
etc.), then finalizes metadata via the returned I/O guard. This
separation lets the caller use a different, potentially more
performant I/O path for bulk data without the crate imposing any
particular strategy.

The vhdx crate provides the low-level VHDX format implementation
and I/O resolution API. For OpenVMM integration, the disklayer_vhdx
crate supplies a LayerIo-compatible backend used in the layered
disk storage pipeline.
For differencing disks, the vhdx crate parses parent locator
metadata, while disklayer_vhdx::chain::open_vhdx_chain walks and
opens parent chains automatically.

Parent lookup

The Rust chain opener and openvmm-img check interpret parent metadata through
ParentLocator::vhdx_parent(). This validates the locator type and requires a
parseable parent linkage GUID. Both verify that the selected parent's
data-write GUID matches the child's recorded linkage.

Both callers use VhdxParent::candidate_paths() to enumerate paths. On Windows,
the order is relative path, volume GUID path, then absolute Win32 path, as
specified by MS-VHDX section 2.6.2.6.3. On Unix, only the relative locator is
used, with backslashes translated to native separators. Windows-only locators
are ignored even if their strings happen to name existing Unix files.

VhdxParent stores locator strings without host-specific path validation or
normalization. These are VHDX locator strings, not native filesystem paths;
conversion to native paths happens during candidate enumeration. Path syntax
is not validated by these helpers.

Candidate enumeration does not open files. The callers select an existing
candidate before verifying linkage; they do not retry later candidates on a
linkage mismatch or repair stale locators. OpenVMM's normal Windows .vhdx
path uses native VHDMP instead of this Rust chain opener.

Storage backends — catalog of all storage backends
Storage pipeline — how
frontends, backends, and layers connect

Guide Src Reference Dev Feats Gdbstub (`Guide/src/reference/dev_feats/gdbstub.md`)

Hardware Debugging (gdbstub)

Think EXDI from Hyper-V, except instead of using the EXDI interface, we use the
GDB Remote Serial Protocol
(via the gdbstub Rust library).

Hardware debugging has several benefits over using an in-guest / in-kernel debugger:

Debugging early-boot scenarios (before UEFI / Windows / Linux debuggers are set up)
Debugging low-level ISRs
Non-intrusive debugging = easier to repro certain bugs
Debugging SNP/TDX/VBS Confidential VMs

Enabling the Debugger

OpenVMM

Pass the --gdb flag at startup to enable the
debug worker. e.g., --gdb 9001

To pause the VM until the debugger has been attached, pass --paused at startup.

OpenHCL

Pass the OPENHCL_GDBSTUB=1 OPENHCL_GDBSTUB_PORT=<gdbstub port> parameters to enable gdbstub. e.g., Set-VmFirmwareParameters -Name UhVM -CommandLine OPENHCL_GDBSTUB=1 OPENHCL_GDBSTUB_PORT=5900.
To expose a TCP port, run ohcldiag-dev.exe <name> vsock-tcp-relay --allow-remote --reconnect <gdbstub port> <tcp port>.

To pause VTL0 boot until desired, pass OPENHCL_VTL0_STARTS_PAUSED=1 as a parameter. Then once the debugger is attached, you can start VTL0 with ohcldiag-dev.exe <name> resume.

Connecting via GDB

The quickest way to get connected to a OpenVMM VM is via gdb directly.

Note that GDB does not support debugging PDBs, so if you're trying to debug
Windows, you'll be limited to plain disassembly. See the Connecting via WinDbg section below if this is your use-case.

On the flipside, if you're trying to debug ELF images with DWARF debug info
(e.g., a vmlinux binary), then you'll likely want to use gdb directly, as it
will support source-level debugging with symbols, whereas WinDbg will not.

You can install gdb via your distro's package manager. e.g., on Ubuntu:

bash

sudo apt install gdb

Once gdb is installed, run it, and enter the following gdb command (swapping
9001 for whatever port you specified at the CLI)

text

target remote :9001

If all goes well, you should get output similar to this:

text

(gdb) target remote :9001
Remote debugging using :9001
warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0xfffff8015c054c1f in ?? ()
(gdb)

At this point, you can try some basic GDB commands to make sure things are working.

e.g., start / interrupt the VM's execution using cont and ctrl-c

text

(gdb) cont
Continuing.
^C                                          # <-- hit ctrl-c in the terminal
Thread 1 received signal SIGINT, Interrupt.
0xfffff8015c054c1f in ?? ()
(gdb)

e.g., inspecting register state

text

(gdb) info registers
rax            0x0                 0
rbx            0x0                 0
rcx            0x40086             262278
rdx            0x0                 0
rsi            0xffff960d4eea5010  -116491073990640
rdi            0x0                 0
rbp            0x0                 0x0
rsp            0xfffff8015b3f5ec8  0xfffff8015b3f5ec8
r8             0x0                 0
r9             0xffffffff          4294967295
r10            0xfffff8015bfff1f0  -8790254554640
r11            0x0                 0
r12            0xffffffff          4294967295
...etc...

e.g., setting data breakpoints

text

(gdb) awatch *0xfffff804683190e0
Hardware access (read/write) watchpoint 1: *0xfffff804683190e0

e.g., single stepping

text

0xfffff8047a309686 in ?? ()
(gdb) si
0xfffff8047a309689 in ?? ()

You may find this blog post
useful, as it includes a table of common gdb commands along with their WinDbg
counterparts.

Connecting via WinDbg

WinDbg doesn't understand the GDB Remote Serial Protocol directly, but
thankfully, some smart folks over on the WinDbg team have developed a GDB Remote
Serial Protocol <-> WinDbg translation layer!

For more information, see
Setting Up QEMU Kernel-Mode Debugging using EXDI

Getting this working with OpenVMM or OpenHCL is as easy as following the guide,
except you'll need to enable our debugger instead of
running QEMU.

It's easiest to connect through the GUI. The steps are relatively simple: Open Windbgx -> File -> Attach to kernel -> EXDI. On the form, fill out:

Target Type: QEMU
Target Architecture: X64
Target OS: Windows
Image Screening heuristic size: 0xFFFE - NT
Gdb server and port: <server>: e.g., 127.0.0.1:1337 (use whatever port you set above)

Known WinDbg Bugs

Hardware breakpoints are issued with ba. The Access Size parameter is incorrectly multiplied by 8 when sent to the stub. Consequently, it must be set to 1.
Unlike GDB, WinDbg doesn't implicitly set software breakpoints via our offered write_addrs implementation.

Supported Features

At the time of writing (8/16/24) the debugger supports the following operations:

read/write guest memory
read guest registers *
start/interrupt execution
watchpoints
hardware breakpoints
single stepping

TODO Features

If you're looking for work, and want to improve the debugging experience for
everyone, consider implementing one or more of the following features:

* reading all guest registers, including fpu, xmm, and various key msrs
software breakpoints:
- Intercept guest breakpoint exceptions into VTL2
writing guest registers
exposing the OpenVMM interactive console via the
MonitorCmd
interface
- Custom commands sent using monitor (gdb) / .exdicmd (WinDbg)
- e.g., being able to invoke x device/to/inspect directly from the debugger
any other features supported by the gdbstub library

Guide Src Reference Dev Feats Kdnet (`Guide/src/reference/dev_feats/kdnet.md`)

Kernel Debugging (KDNET)

Kernel Debugging is available for Windows guests via KDNET over VMBus.

Enabling and Starting the Debugger

Set up KDNET on the guest and start the debugger as described on
Set up KDNET network kernel debugging manually | Microsoft Learn.
Setting busparams is not necessary.

admonish warning

If you're using a `memdiff:` disk (the default in most examples), guest state
is not persisted between runs. You'll need to configure KDNET in the guest
each time you launch — or configure it once on the base disk image before
wrapping it with `memdiff:`.

With OpenVMM and WHP as Host

Launch the VM with UEFI and networking enabled. KDNET requires --net consomme
for the network transport:

bash

# Without OpenHCL
cargo run -- --uefi --hv --net consomme \
  --vmbus-scsi id=scsi0 \
  --disk memdiff:file:path/to/windows.vhdx,on=scsi0

# With OpenHCL
cargo run -- --uefi --hv --vtl2 --net consomme \
  --igvm firmware=path/to/openhcl.igvm,personality=openhcl \
  --vmbus-scsi id=scsi0 \
  --disk memdiff:file:path/to/windows.vhdx,on=scsi0

Known Issues

Networking backend: KDNET currently requires --net consomme. The
consomme backend creates a new network adapter in the guest on each
OpenVMM restart — this is harmless and can be ignored. Using
--net vmnic:<switch id> connects but hangs immediately due to an
undiagnosed vmbusproxy issue.
Debugger reconnection: If you quit OpenVMM without shutting down the
guest first, the same WinDbg instance cannot reconnect on next boot.
Close and relaunch WinDbg to reconnect.
Synic warnings with OpenHCL: When launching an OpenHCL VM with KDNET,
you may see a stream of failed to signal synic errors from
virt_whp::synic for several seconds. These do not affect VM functionality
and can be ignored.

Debugging workflow

Once KDNET is connected:

In WinDbg, go to File → Kernel Debug → Net and enter the key and port from the guest setup.
Start the VM. WinDbg should connect during Windows boot.
To break into the debugger: Debug → Break (or Ctrl+Break).

admonish note

KDNET debugs the VTL0 (guest) Windows kernel. For debugging the VTL2
(OpenHCL) side, see [Debugging OpenHCL](../openhcl/debugging.md) which covers
serial logs, crash dumps, and `ohcldiag-dev`.

Guide Src Reference Devices Pci Ids (`Guide/src/reference/devices/pci_ids.md`)

PCI IDs

This page records the PCI identifiers that OpenVMM reports for the PCI and PCIe
devices it emulates.

admonish note

These IDs describe how a device advertises itself to the guest in PCI config
space. They are distinct from VMBus/VPCI instance GUIDs, which identify device
*instances* at runtime and are typically randomly generated per VM.

Microsoft vendor ID (`0x1414`)

Most first-party OpenVMM devices report Microsoft's PCI vendor ID, 0x1414.

Device IDs under this vendor ID are allocated and centrally managed by
Microsoft. You cannot pick an arbitrary device ID when adding a new emulated
device: a device ID has to be formally assigned so that it does not collide
with any other Microsoft product that shares this vendor ID. The values below
are the assignments OpenVMM currently uses.

These identifiers — the vendor ID, each device ID, and the default subsystem
ID — are all defined in one place, the microsoft module of the pci_core
crate (vm/devices/pci/pci_core/src/microsoft.rs), and referenced from each
device.

Device ID	Device	Crate
`0x00BA`	MANA/GDMA network adapter	`gdma`
`0x5353`	VGA adapter	`vga`
`0xC030`	PCIe root port	`pcie`
`0xC031`	PCIe upstream switch port	`pcie`
`0xC032`	PCIe downstream switch port	`pcie`
`0xC03E`	NVMe controller	`nvme`, `nvme_test`

These devices all report a subsystem vendor ID and subsystem ID of 0.

Devices that report standardized IDs

Some devices emulate a real, standardized controller and therefore report the
hardware vendor/device ID that stock guest drivers bind to, rather than a
Microsoft-allocated ID:

virtio devices use the virtio-over-PCI scheme: vendor ID 0x1AF4 and
device ID 0x1040 plus the virtio device type. They report Microsoft's
0x1414 as the subsystem vendor ID to identify OpenVMM as the host
environment, with a subsystem ID of 0x2000. Source:
vm/devices/virtio/virtio/src/transport/pci.rs.
AMD IOMMU reports AMD's vendor ID 0x1022 and device ID 0x1451
(a family 17h / Zen IOMMU) so that the guest's AMD IOMMU driver binds to it.
It reports Microsoft's 0x1414 as its subsystem vendor ID and 0x2000 as
its subsystem ID to identify OpenVMM as the host environment. Source:
vm/devices/iommu/amd_iommu/src/lib.rs.

Guide Src Reference Emulated Ipmi Kcs (`Guide/src/reference/emulated/ipmi_kcs.md`)

IPMI KCS

OpenVMM provides a minimal virtual IPMI baseboard management controller (BMC)
using the Keyboard Controller Style (KCS) system interface. The device allows
UEFI guests to write and query a bounded System Event Log (SEL).

The implementation is provided by the ipmi_kcs crate. Wire-level command and
record definitions are in the ipmi_protocol crate.

Configuration

The device is created when the platform configuration enables IPMI. OpenHCL
accepts this setting only for UEFI guests; PCAT and Linux-direct boot modes are
not supported.

The guest-visible register interface depends on the architecture:

Architecture	Interface	Registers
x86-64	Port I/O	Data at `0xCA2`; status/command at `0xCA3`
AArch64	MMIO	Data at `0xEFFE7000`; status/command at `0xEFFE7004`

Only one-byte register accesses are supported.

Supported commands

The virtual BMC implements the IPMI application Get Device ID command and
these SEL storage commands:

Get SEL Info
Reserve SEL
Get SEL Entry
Add SEL Entry
Clear SEL
Get SEL Time
Set SEL Time

The SEL stores at most 128 records. Reservations are exposed for guest software
compatibility but are not enforced because the device has one serialized KCS
requestor and no independent SEL mutators.

Completed SEL records are retained by the device and forwarded to the host on
a best-effort basis. Forwarding is limited to 256 records per trusted
wall-clock second; reaching that limit does not remove records from the SEL.

Servicing

Saved state includes the KCS transaction, SEL records, record allocation state,
reservation identifier, and guest-selected SEL time offset. Diagnostic
forwarding counters and rate-limiter state are reset after restore.

See the
ipmi_kcs rustdoc for the
device API.

Guide Src Reference Openhcl Debugging (`Guide/src/reference/openhcl/debugging.md`)

Debugging OpenHCL

OpenHCL provides several debugging tools for investigating issues at the
user-mode and kernel level. See ohcldiag-dev for the
diagnostic client and Tracing for serial and event log
tracing.

On-demand memory dumps

Use ohcldiag-dev dump to capture a live user-mode memory dump of the OpenHCL
process at any time. The dump is an ELF core file that you can analyze with
lldb, gdb, or rust-lldb. See the
ohcldiag-dev page for the full command reference.

User-mode crash dumps

When an OpenHCL user-mode process crashes, a crash dump is automatically
generated via the underhill-crash infrastructure and sent to the host over
VMBus. On Windows hosts, these dumps are collected by Windows Error Reporting
(WER). Use lldb or gdb to analyze the resulting ELF core dump.

Kernel crash dumps

Kernel-mode crash dumps (kdump) are not currently supported in OpenHCL. The
OpenHCL kernel does not have CONFIG_KDUMP or CONFIG_KEXEC compiled in. If
the kernel panics, no dump is generated. The only diagnostic output is COM3
serial (if enabled), which captures the panic message in real time. If the
diagnostic service was running before the panic, ohcldiag-dev may have ring
buffer messages up to that point, but it cannot capture the panic itself since
the service is terminated by the panic.

For debugging kernel-level issues, the best approach is to enable serial output
via COM3 (see below) — it captures output from the very first instruction of
kernel boot.

Getting OpenHCL kernel logs (COM3 vs ohcldiag-dev)

Two methods exist for capturing OpenHCL kernel (kmsg) output:

COM3 serial uses direct UART I/O — it streams output from the very first
instruction of OpenHCL boot in real time.

ohcldiag-dev connects over vsock to the diagnostic service, which reads
/dev/kmsg. Because /dev/kmsg preserves the kernel ring buffer, early boot
messages are replayed when you connect — you get them even if you connect
late. However, ohcldiag-dev only works if the diagnostic service successfully
starts.

Boot phase	COM3 serial	ohcldiag-dev
Very early kernel (entry → memory setup)	✅ live	✅ replayed from ring buffer
Device initialization (VMBus, etc.)	✅ live	✅ replayed from ring buffer
Kernel panic before userspace	✅ live	❌ service never starts
Boot hang (kernel stuck)	✅ live	❌ service never starts
After diagnostic service starts	✅ live	✅ live

For most development, ohcldiag-dev is sufficient — boot succeeds and you get
logs. COM3 is essential for debugging early boot failures, kernel panics, and
init crashes.

Enabling COM3 on Hyper-V

COM3 support requires a host OS build that includes the EnableAdditionalComPorts
code path. This was added in Windows 11 26H1 (build 28000+, Insider Canary channel).
It is not available on Windows 11 24H2, 25H2, or Windows Server 2025.

To enable COM3 on a supported build:

powershell

# Enable additional COM ports (requires reboot or VMMS restart)
reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Virtualization" /v EnableAdditionalComPorts /t REG_DWORD /d 1 /f

# Attach COM3 to a named pipe for a VM
Set-VMComPort -VMName $VmName -Number 3 -Path "\\.\pipe\openhcl-com3"

# Read the serial output
hvc serial -c -p 3 -r $VmName

admonish note

The `flowey` test runner (`install_vmm_tests_external_deps`) sets this registry key
automatically when running VMM tests. If you run `cargo xflowey` to execute
tests, you'll be prompted to allow the registry change.

Recommended host OS for OpenHCL development

We recommend running a Windows 11 26H1 Insider flight (Canary channel,
build 28000+) on your development machine, if it is available for your device.
This gives you COM3 support via the registry key above, plus access to the
latest Hyper-V features. This matches the OS used on the project's self-hosted
CI runners.

Note that 26H1 Insider builds may not be available for all hardware — see
What to know about Windows 11 version 26H1
for details and the
Windows Insider Flight Hub
for availability.

If you're on Windows 11 24H2/25H2 (builds 26100/26200), COM3 is not available
via the registry key. Use ohcldiag-dev for kernel logs instead.

ARM64 limitation

On Hyper-V, additional serial ports (COM3+) are not supported on ARM64. The
Hyper-V serial device for ARM64 does not support ports beyond COM1 and COM2. On
ARM64 hosts, use ohcldiag-dev for OpenHCL kernel logs.

This limitation is Hyper-V-specific — when running OpenVMM directly (without
Hyper-V), ARM64 serial output works via PL011 UART.

Guide Src Reference Openhcl Diag (`Guide/src/reference/openhcl/diag.md`)

Diagnostics

This chapter discusses several of the diagnostic tools available when working
with OpenHCL.

Guide Src Reference Openvmm Graphical Console (`Guide/src/reference/openvmm/graphical_console.md`)

Graphical Console

OpenVMM supports a graphical console exposed via VNC. To enable it, pass --gfx
on the command line--this will start a VNC server on localhost port 5900. The
port value can be changed with the --vnc-port option. The bind address
can be changed with --vnc-listen <ADDRESS> (default: 127.0.0.1). Use
--vnc-listen 0.0.0.0 for all IPv4 interfaces, or --vnc-listen :: for
dual-stack IPv4+IPv6. The maximum number of concurrent clients can be set with
--vnc-max-clients <COUNT> (default: 16). With --vnc-evict-oldest, the
oldest client is disconnected to make room for new connections instead of
rejecting them.

OpenVMM's VNC server also includes "pseudo" client-clipboard support, whereby the
"Ctrl-Alt-P" key sequence will be intercepted by the server to type out the
contents of the VNC clipboard.

The VNC server supports RFB protocol versions 3.3, 3.7, and 3.8, with no
authentication (security type "None"). It negotiates the following optional
features based on client capabilities:

Zlib compression (encoding type 6) -- when the client advertises support,
tile data is zlib-compressed to reduce bandwidth.
Cursor pseudo-encoding -- a local arrow cursor is rendered client-side when
supported, eliminating server-side cursor compositing.
DesktopSize pseudo-encoding -- resolution changes are relayed to the client.
Clients that do not advertise this encoding will be disconnected on a
resolution change.
QEMU extended key events -- when available, the server uses scancode-based
key input instead of xkeysym translation.
Client reconnection -- a new VNC client connecting will cleanly disconnect
the previous session and take over.

Once OpenVMM starts, you can connect to the VNC server using any supported VNC
client. The following clients have been tested working with OpenVMM:

TightVNC
TigerVNC
RealVNC
noVNC (browser-based)
MobaXterm (Windows terminal with built-in VNC)

Once you have downloaded and installed it you can connect to localhost with
the appropriate port to see your VM.

Guide Src Reference Openvmm Logging (`Guide/src/reference/openvmm/logging.md`)

OpenVMM Logging

Configuring OpenVMM logging messages to emit

To configure logging, set the OPENVMM_LOG environment variable. The default
level is info. For example:

OPENVMM_LOG=debug — enable debug events from all modules
OPENVMM_LOG=info,mesh=trace — enable trace events from the mesh crate
and info events from everything else

This is backed by the
EnvFilter
type; see the associated documentation for more details.

Span events

By default, OpenVMM does not log span enter/exit events. To enable them, set
OPENVMM_LOG_SPANS=1.

Rate limiting

Trace events that can be triggered repeatedly by guest interactions are
rate-limited by default. To disable rate limiting (useful for debugging), set
OPENVMM_DISABLE_TRACING_RATELIMITS=1.

Configuring OpenHCL Trace Logging

OpenHCL also supports EnvFilter-style trace logging, configured via the
-c OPENVMM_LOG= command line argument. The -c flag passes arguments to
OpenHCL initialization. The filter syntax is the same as for OpenVMM.

OpenHCL tracing can also be configured and dumped at runtime with
ohcldiag-dev. See: OpenHCL Diagnostics

To retrieve OpenHCL log output at runtime, an output console or file must
attach to the OpenHCL logging COM port. By default, OpenHCL outputs to COM3.

To open a new terminal window with global OpenHCL debug level tracing enabled:

shell

openvmm -c "OPENVMM_LOG=debug" --com3 "term,name=VTL2 OpenHCL" [...]

Configure log levels of only a given module name:

shell

openvmm -c "OPENVMM_LOG=mesh=trace" --com3 "term,name=VTL2 OpenHCL" [...]

Multiple modules can be specified by separating them with a comma:

shell

openvmm -c "OPENVMM_LOG=mesh=trace,nvme_driver=trace" \
    --com3 "term,name=VTL2 OpenHCL" [...]

admonish tip

For more configuration examples of serial ports and the OpenVMM CLI, see the
[Running OpenHCL Guide](../../../user_guide/openhcl/run/openvmm.md) and CLI
`--help` output.

Capturing the ETW traces on the host

On Windows, OpenVMM also logs to ETW, via the Microsoft.HvLite provider.

To capture the trace, start a session:

powershell

logman start trace <SESSION_NAME> -ow -o trace.etl `
    -p "{22bc55fe-2116-5adc-12fb-3fadfd7e360c}" 0xffffffffffffffff 0xff `
    -nb 16 16 -bs 16 -mode 0x2 -ets

admonish note

For OpenHCL traces, use `{AA5DE534-D149-487A-9053-05972BA20A7C}` as the
provider GUID.

To flush:

powershell

logman update <SESSION_NAME> -ets -fd

To stop:

powershell

logman stop <SESSION_NAME> -ets

To decode as CSV:

powershell

tracerpt trace.etl -y -of csv -o trace.csv -summary trace-summary.txt

Guide Src Reference Openvmm Management (`Guide/src/reference/openvmm/management.md`)

Configuration and Management

OpenVMM exposes three configuration and management interfaces. The CLI and
interactive console support direct VM configuration and lifecycle operations,
while gRPC and ttrpc provide programmatic control.

admonish note title="interface compatibility"

These interfaces continue to evolve and may change between releases. Consult
the documentation for your OpenVMM version, especially when upgrading.

CLI: Used to configure and launch a single VM
- This allows configuring static VM resource assignments, such as the
  number of processors, RAM size, UEFI, graphic console, etc.. as well as what
  devices are exposed to the Guest, such as a virtual NIC, Storage, vTPM,
  etc..
Interactive console: Used to interact with a VM at runtime
- This interface allows users to perform core VM operations such as stop,
  restart, save, restore, pause, resume, etc.. as well as things like storage
  hot-add, VTL2 servicing, running Inspect queries, etc..
gRPC / ttrpc: Programmatic APIs for configuring and interacting with VMs.
API compatibility and implementation coverage continue to evolve.

Management Capabilities

The following table summarizes the status of selected management features.

Feature	Status
Suspend / Resume a VM	OpenVMM's existing save/restore infrastructure theoretically

text

supports this, but the end-to-end flow has not been wired up to
            any management interface at the moment. |

| Snapshots | OpenVMM has core infrastructure for performing save/restore operations,
but there are gaps in device support (notably: no support for storage snapshots). |
| Managing multiple running VMs | OpenVMM currently runs a single VM per-process.
It is not yet clear whether OpenVMM will support managing
multiple VMs via a single OpenVMM process, or if OpenVMM will
rely on external management tools (e.g: libvirt) interfacing
with its existing APIs in order to launch and manage multiple VMs. |

For additional feature requests and status information, search the
OpenVMM GitHub issues.

Guide Src User Guide Openhcl Next Steps (`Guide/src/user_guide/openhcl/next_steps.md`)

Next Steps

To learn more about how OpenHCL works, and how to interact with it:


Reference: OpenHCL Features	Configure and interact with OpenHCL
Dev Guide: Building OpenHCL	Build OpenHCL locally
Reference: OpenHCL Architecture	Understand how OpenHCL works "under the hood"

Guide Src User Guide Openhcl Run (`Guide/src/user_guide/openhcl/run.md`)

Running OpenHCL

This chapter provides a high-level overview of different ways to launch and
interact with OpenHCL.

OpenHCL Platforms

High-level Overview

In order to run OpenHCL, an existing host VMM must first load the OpenHCL
environment into a VM, much akin to existing virtual firmware layers, like UEFI,
or BIOS[^vtls].

OpenHCL is distributed as an IGVM file (Independent Guest Virtual Machine),
which encapsulates all the directives and data required to launch a particular
virtual machine configuration on any given virtualization stack.

At this time, the only VMMs which are able to load and host OpenHCL IGVM files
are Hyper-V, and OpenVMM.

Obtaining a copy of OpenHCL

To get started, ensure you have a copy of an OpenHCL IGVM firmware image, via
one of the following options:

Building OpenHCL Locally

Follow the instructions on: Building OpenHCL.

Note: At this time, OpenHCL can only be built on Linux / WSL2.

Pre-Built Binaries

If you would prefer to try OpenHCL without building it from scratch, you can
download pre-built copies of OpenHCL IGVM files from
OpenVMM CI.

Simply select a successful pipeline run (should have a Green checkbox), and
scroll down to select an appropriate *-openhcl-igvm artifact for your
particular architecture and operating system.

[^vtls]: Though, unlike UEFI / BIOS, OpenHCL is loaded into a distinct, higher
privilege execution context within the VM, called
VTL2.

Guide Src User Guide Openhcl Troubleshooting (`Guide/src/user_guide/openhcl/troubleshooting.md`)

OpenHCL Troubleshooting

This page includes a miscellaneous collection of troubleshooting tips for common
issues you may encounter when running OpenHCL.

If you are still running into issues, consider filing an issue on the OpenVMM
GitHub Issue tracker.

\[Hyper-V] VTL2/VTL0 failed to start

VTL2/VTL0 fails to boot is when either VTL2 or VTL0 has crashed. When the crash happens, they will emit an event to the Hyper-V worker channel.

First, check Hyper-V worker events at Applications and Services Logs -> Microsoft -> Windows -> Hyper-V-Worker-Admin

Alternatively, some queries you can use to get Hyper-V-Worker logs:

Display the {n} most recent events - wevtutil qe Microsoft-Windows-Hyper-V-Worker-Admin /c:{n} /rd:true /f:text
Export events to file - wevtutil epl Microsoft-Windows-Hyper-V-Worker-Admin C:\vtl2_0_crash.evtx

Checking OpenHCL logging output

OpenHCL logging output can be useful for debugging issues with startup or runtime behavior.

See OpenHCL Tracing for more details about how to enable OpenHCL logging.

DeviceTree errors or warnings in the VTL2 kernel log

Retrieve the DeviceTree blob from OpenHCL:

powershell

uhdiag-dev.exe linux-uhvm00 file --file-path "/sys/firmware/fdt" > uh.dtb

Install the DeviceTree compiler and convert the blob to the textual representation:

sudo apt-get install dtc
dtc -I dtb -o uh.dts uh.dtb

Check on the errors and warnings and should any have been produced, fix them in the
DeviceTree generation code. If that doesn't resolve the issues, inspect the DT parsing
code in the Linux kernel.

Guide Src User Guide Openvmm Alpine (`Guide/src/user_guide/openvmm/alpine.md`)

Alpine Linux, via Direct Boot

This guide boots Alpine Linux from a cloud disk image using Linux direct boot
(no UEFI or BIOS firmware required). It demonstrates how to set up PCIe, attach
virtio-blk disks, and use virtio-net for networking.

Quick start

Run the setup script from the openvmm repo:

bash

./scripts/setup-alpine.sh [output-dir]

The default output directory is ./alpine-direct-boot. The script prints the
boot command when it finishes. It also writes a README file to the output
directory with the same command for future reference.

admonish warning

The default password is intended for local development only. Change it
or configure SSH keys before using a networking backend that exposes
the VM to other hosts.

What the script does

Downloads the Alpine v3.21 "nocloud" UEFI tiny cloud image (qcow2).
Converts it to a raw disk image (disk.raw) with qemu-img.
Extracts the kernel and initramfs (vmlinuz-virt, initramfs-virt)
from the ext4 root partition inside the image. This step requires sudo
for the loopback mount.
Extracts an ELF kernel (vmlinux-virt) from the bzImage. OpenVMM's
direct boot loader requires an uncompressed ELF, not a compressed bzImage.
The script uses Python to find and decompress the gzip-embedded ELF.
Creates a cloud-init data disk (cidata.img), a small FAT image
labeled cidata containing user-data and meta-data files. Alpine's
tiny-cloud service reads these on first boot. The user-data uses
runcmd to set the root password and add a getty on hvc0 (Alpine does
not spawn one by default).

Required host tools

curl, qemu-img, python3, mkfs.vfat, mcopy (from mtools), and
sudo (for the loopback mount).

Key points about the boot command

--hv: Enables Hyper-V enlightenments for better performance.
--pcie-root-complex and --pcie-root-port: Required for PCI device
visibility. The default direct boot DSDT does not include a PCI bus, so
without these flags, virtio devices will not be detected by the kernel.
--virtio-blk ...,pcie_port=disk: Attaches the raw disk image as a
virtio-blk device on a PCIe root port.
--virtio-blk ...,ro,pcie_port=cidata: Attaches the cloud-init data
image read-only on a second PCIe root port.
--virtio-net pcie_port=net:consomme: Adds a virtio-net NIC using
the consomme user-mode NAT backend, on a third PCIe root port.
--com1 none: Disables the default COM1 serial port (which would
otherwise claim the console).
--virtio-console console --virtio-console-pcie-port console: Adds a
virtio-console device (/dev/hvc0 in the guest) on a fourth PCIe root
port. This replaces COM1 as the interactive console.
root=/dev/vda2: The root filesystem is on partition 2 of the virtio
disk.
modules=virtio_pci,virtio_blk,ext4: Tells the Alpine initramfs to
load these modules early, before attempting to mount the root filesystem.
console=hvc0: Directs the kernel to use the virtio-console
(/dev/hvc0) for console output instead of the legacy serial port.

admonish tip

Use `ctrl-q` then `q` to quit OpenVMM (not `ctrl-c`).

Guide Src User Guide Openvmm Next Steps (`Guide/src/user_guide/openvmm/next_steps.md`)

Next Steps

To learn more about how OpenVMM works, and how to interact with it:


Reference: OpenVMM Features	Configure and interact with OpenVMM
Dev Guide: Building OpenVMM	Build OpenVMM locally
Reference: OpenVMM Architecture	Understand how OpenVMM works "under the hood"

Guide Src User Guide Openvmm Run (`Guide/src/user_guide/openvmm/run.md`)

Running OpenVMM

This page offers a high-level overview of different ways to launch and interact
with OpenVMM.

These examples provide a starting point for launching and configuring OpenVMM.

Obtaining a copy of OpenVMM

To get started, ensure you have a copy of the OpenVMM executable and its runtime
dependencies, via one of the following options:

Building OpenVMM Locally

Follow the instructions on: Building OpenVMM.

Pre-Built Binaries

If you would prefer to try OpenVMM without building it from scratch, you can
download pre-built copies of the binary from
OpenVMM CI.

Simply select a successful pipeline run (should have a Green checkbox), and
scroll down to select an appropriate *-openvmm artifact for your particular
architecture and operating system. You must be signed into GitHub in order
to download artifacts.

Examples

admonish tip

These examples all use `cargo run --`, with the assumption that you are a
developer building your own copy of OpenVMM locally!

To run these examples using a pre-compiled copy of OpenVMM, swap `cargo run
--` with `/path/to/openvmm`.

admonish tip title="uefi firmware required when running outside cargo"

When running via `cargo run`, environment variables in `.cargo/config.toml`
automatically point OpenVMM to the `mu_msvm` UEFI firmware (`MSVM.fd`)
downloaded by `cargo xflowey restore-packages`.

When running the `openvmm` binary directly, these environment variables are
**not set**, and you will get:

> fatal error: must provide uefi firmware when booting with uefi

To fix this, **explicitly pass the firmware** using the `firmware` option:

```shell
openvmm --uefi firmware=path/to/MSVM.fd \
  --vmbus-scsi id=scsi0 \
  --disk memdiff:path/to/disk.vhdx,on=scsi0
```

If you ran `cargo xflowey restore-packages`, the firmware is at:

```text
.packages/hyperv.uefi.mscoreuefi.x64.RELEASE/MsvmX64/RELEASE_VS2022/FV/MSVM.fd        # x64
.packages/hyperv.uefi.mscoreuefi.AARCH64.RELEASE/MsvmAARCH64/RELEASE_CLANGPDB/FV/MSVM.fd # aarch64
```

If you used `cargo xflowey vmm-tests-run --build-only --dir <out>`, the firmware
is copied into that output directory under the same relative path.

Alternatively, set the environment variable so you don't need the flag each time:

```shell
# x64
export X86_64_OPENVMM_UEFI_FIRMWARE=path/to/MSVM.fd

# aarch64
export AARCH64_OPENVMM_UEFI_FIRMWARE=path/to/MSVM.fd
```

If you run into any issues, please refer to Troubleshooting.

_Preface:_ Quitting OpenVMM

By default, OpenVMM will connect the guests's COM1 serial port to the current
terminal session, forwarding all keystrokes directly to the VM.

As such, a simple ctrl-c does not suffice to quit OpenVMM!

Instead, you can type crtl-q to enter OpenVMM's interactive console, and enter q to quit.

Sample Linux Kernel, via direct-boot

This example will launch Linux via direct boot (i.e: without going through UEFI
or BIOS), and appends single to the kernel command line.

The Linux guest's console will be hooked up to COM1, and is relayed to the host
terminal by default.

To launch Linux with an interactive console into the shell within initrd, simply
run:

shell

cargo run

This works by setting the default [env] vars in .cargo/config.toml to
configure OpenVMM to use a set of pre-compiled test kernel + initrd images,
which are downloaded as part of the cargo xflowey restore-packages command.
Note that this behavior only happens when run via cargo run (as cargo is the
tool which ensures the required env-vars are set).

The source for the sample kernel + initrd can be found on the
microsoft/openvmm-deps repo.

The kernel and initrd can be controlled via options:

--kernel : The kernel image. Must be an uncompressed kernel (vmlinux, not bzImage).
--initrd : The initial ramdisk image.
-c <STRING> or --cmdline <STRING>: Extra kernel command line options, such as root=/dev/sda.

Windows, via UEFI

This example will launch a modern copy of Windows via UEFI, using the mu_msvm
firmware package.

A copy of the mu_msvm UEFI firmware is automatically downloaded via cargo xflowey restore-packages.

shell

cargo run -- --uefi \
  --vmbus-scsi id=scsi0 \
  --disk memdiff:path/to/windows.vhdx,on=scsi0 \
  --gfx

For more info on --gfx, and how to actually interact with the VM using a
mouse/keyboard/video, see the Graphical Console
docs.

The file windows.vhdx can be any format of VHD(X).

VHDX files (dynamic, fixed, and differencing) are supported on non-Windows
platforms via the pure-Rust vhdx
parser. On Windows, .vhdx files use the native kernel-mode VHD path
instead. Fixed VHD1 images work on all platforms. Dynamic and differencing VHD1
files are not supported — convert them to VHDX first:

bash

qemu-img convert -f vpc -O vhdx dynamic.vhd converted.vhdx

Also, note the use of memdiff, which creates a memory-backed "differencing
disk" shim between the VMM and the backing disk image, which ensures that any
writes the VM makes to the VHD are not persisted between runs. This is very
useful when iterating on OpenVMM code, since booting the VM becomes repeatable
and you don't have to worry about shutting down properly. Use file instead for
normal persistent storage.

OpenHCL, via Linux Direct Boot

This example will boot OpenHCL in Linux direct mode, running a minimal shell
inside VTL2. This is the same configuration used by the openhcl_linux_direct_x64
integration tests.

First, build the test artifacts from Linux or WSL using vmm-tests-run --build-only.
The IGVM must be built on Linux:

shell

cargo xflowey vmm-tests-run --build-only --dir <out> --target windows-x64

admonish tip

If you only need the IGVM binary (and already have `openvmm.exe`), you can
use `cargo xflowey build-igvm` instead — it's faster than building the full
test suite.

This places openvmm.exe and openhcl-x64-test-linux-direct.bin in the
<out> directory. Then, on Windows, from the <out> directory:

powershell

.\openvmm.exe `
    --hv `
    --vtl2 `
    --igvm "firmware=openhcl-x64-test-linux-direct.bin,personality=openhcl" `
    -c "panic=-1 reboot=triple UNDERHILL_SERIAL_WAIT_FOR_RTS=1 UNDERHILL_CMDLINE_APPEND=rdinit=/bin/sh" `
    -m 2GB `
    --vmbus-com1-serial "term,name=VTL0 Linux" `
    --com3 "term,name=VTL2 OpenHCL" `
    --vmbus-vtl2-vsock-path $env:temp\ohcldiag-dev

admonish warning

The `--vmbus-com1-serial` flag is **required** when using `rdinit=/bin/sh`.
The shell running as PID 1 needs a controlling terminal (tty) — without one
it exits immediately, causing a kernel panic and infinite reboot loop.

The `--com3` flag is optional but recommended — it gives you VTL2 (OpenHCL)
kernel console output for debugging.

For more details on running OpenHCL on OpenVMM, including
VMBus relay and device
assignment, see Running OpenHCL: OpenVMM.

Alpine Linux, via Direct Boot

See the dedicated Alpine Linux guide for a full walkthrough of
booting Alpine from a cloud disk image using direct boot with PCIe and
virtio-blk.

DOS, via PCAT BIOS

While DOS in particular is not a scenario that the OpenVMM has heavily invested
in, the fact DOS is able to boot in OpenVMM serves as a testament to OpenVMM's
solid support of legacy x86 devices and infrastructure.

The following command will boot a copy of DOS from a virtual floppy disk, using
the Hyper-V PCAT BIOS.

Booting via PCAT is not just for DOS though! Many older operating systems,
including older copies of Windows / Linux, require booting via BIOS.

bash

cargo run -- --pcat --gfx --floppy memdiff:/path/to/msdos.vfd --pcat-boot-order=floppy,optical,hdd

Guide Src User Guide Openvmm Snapshots (`Guide/src/user_guide/openvmm/snapshots.md`)

Snapshots

OpenVMM supports saving and restoring VM snapshots, allowing you to capture
the complete state of a running VM and resume it later.

Overview

A snapshot captures three pieces of state:

Guest RAM — the full contents of guest memory
Device state — the saved state of all emulated devices
Manifest — metadata describing the snapshot (architecture, memory size,
VP count, page size, etc.)

These are stored as three files in a snapshot directory:

File	Contents
`manifest.bin`	Protobuf-encoded snapshot metadata
`state.bin`	Serialized device state
`memory.bin`	Memory backing file

Prerequisites

Snapshots require file-backed guest memory. Pass file= in the
--memory option when launching the VM so that guest RAM is written to a
file on disk rather than held in anonymous memory.

admonish warning

The memory backing file and the snapshot directory must be on the **same
filesystem**. OpenVMM creates a hard link from the backing file to
`memory.bin` inside the snapshot directory, which does not work across
filesystem boundaries.

Saving a snapshot

Start a VM with file-backed memory:

bash

cargo run -- \
  --uefi \
  --vmbus-scsi id=scsi0 \
  --disk memdiff:file:path/to/disk.vhdx,on=scsi0 \
  --memory size=4096M,file=path/to/memory.bin

Once the VM is running, open the interactive console and issue a save command,
specifying the output directory:

text

save-snapshot path/to/snapshot-dir

OpenVMM writes manifest.bin, state.bin, and a hard link to memory.bin
into the specified directory.

admonish warning

After saving, the VM remains **paused** and resume is blocked. Resuming
would mutate guest RAM through `memory.bin`, corrupting the snapshot.
Use `shutdown` to exit OpenVMM after saving.

Restoring a snapshot

To restore, pass the snapshot directory with --restore-snapshot:

bash

cargo run -- \
  --uefi \
  --vmbus-scsi id=scsi0 \
  --disk memdiff:file:path/to/disk.vhdx,on=scsi0 \
  --memory size=4096M \
  --processors 4 \
  --restore-snapshot path/to/snapshot-dir

--restore-snapshot automatically opens memory.bin from the snapshot
directory, so file=... should not be specified in --memory (the two
options are mutually exclusive).

admonish note

The `--memory` and `--processors` values must match the values recorded in
the snapshot manifest. If they do not match, OpenVMM will report a
validation error and refuse to start.

Device configuration on restore

The snapshot only stores device state, not device configuration. All
device flags (e.g. --disk, --nic, --serial, --virtio-blk, etc.)
must be specified on the restore command line exactly as they were when
the snapshot was saved — they are not read from the snapshot.

The snapshot manifest validates that --memory, --processors,
architecture, and page size match the values recorded at save time. However,
it does not record the list of CLI device flags. Instead, device
configuration compatibility is enforced at the state-unit level: each
emulated device saves its state under a unique name (e.g. "pit",
"vmbus", "ide"), and restore matches saved-state entries to the
currently instantiated devices by name.

The rules are:

Scenario	Result
Device set matches exactly	Restore succeeds
Snapshot contains a device not in current config	Restore fails — unknown unit name
Current config has a device not in snapshot	Restore succeeds — device starts in its default/initial state

In practice this means:

You must pass the same device flags on restore as you did on save.
Removing a device that was present at save time will cause restore to
fail.
Adding a new device that was not present at save time is technically
allowed — the new device will start in its power-on default state.
This is not tested and the device may not be functional, since the
guest OS will not have enumerated or initialised it during boot.
The supported path is to use the same device flags on save and restore.

admonish warning

There is no single error message that tells you "your device configuration
changed". Instead you will see errors like `restore failed: unknown unit
name` when saved-state entries cannot be matched. If you see this, compare
your restore command line with the one used at save time.

Device save/restore support

Not all devices support save/restore. If a VM includes a device that does
not support saving, the save-snapshot command will fail with
SaveError::NotSupported.

The following table summarises support for the device types relevant to
OpenVMM snapshots:

Device	Bus	Save/Restore
PIT, PIC, I/O APIC, DMA	Chipset (ISA)	Yes
CMOS RTC, Power Management	Chipset (ISA)	Yes
i8042 (PS/2 keyboard/mouse)	Chipset (ISA)	Yes
Serial 16550	Chipset (ISA) / PCI	Yes
UEFI firmware	Chipset (MMIO)	Yes
Framebuffer	Chipset (MMIO)	Yes
TPM	Chipset (MMIO)	Yes
IDE controller	PCI	Yes
PIIX4 bridges, bus, PM, RTC	PCI	Yes
Generic PCI bus	PCI	Yes
StorVsp (SCSI)	VMBus	Yes
NetVsp (NIC)	VMBus	Yes
Shutdown / Timesync / KVP ICs	VMBus	Yes
VMBus Keyboard / Mouse / Video	VMBus	Yes
Guest Emulation Log	VMBus	Yes
virtio-blk	Virtio (PCI/MMIO)	Yes
virtio-net	Virtio (PCI/MMIO)	Yes
virtio-pmem	Virtio (PCI/MMIO)	Yes
virtio-rng	Virtio (PCI/MMIO)	Yes
NVMe	PCI	No
VGA	PCI	No (`todo!()`)
GDMA (MANA network)	PCI	No (`todo!()`)
PCIe root complex / switch	PCIe	No
Assigned PCI (pass-through)	PCI	No
Relayed vPCI	PCI	No
PCAT BIOS firmware	Chipset (ISA)	No (see limitations)
virtio-9p, virtiofs	Virtio (PCI/MMIO)	No
virtio-console	Virtio (PCI/MMIO)	No
Guest Crash Device	VMBus	No
Guest Emulation Device (GED)	VMBus	No
VMBus serial (host)	VMBus	No
Vmbfs	VMBus	No

admonish tip

If you are unsure whether your VM configuration supports snapshots, try
issuing `save-snapshot` to a scratch directory. The save will fail
immediately with a clear error if any active device does not support it.

Limitations

Snapshots are not portable across architectures (e.g., you cannot
restore an x86_64 snapshot on aarch64)
After restoring, memory.bin in the snapshot directory becomes the live
guest RAM backing file and will be modified as the VM runs. To restore
from the same snapshot multiple times, copy the snapshot directory before
each restore.
VMs using VPCI or PCIe devices do not currently support save/restore
OpenHCL-based VMs do not currently support this snapshot mechanism
VMs using PCAT firmware do not support save/restore
--memory and --processors must be specified on restore and match the
snapshot manifest values. A future version may read these from the snapshot
automatically.

Guide Src User Guide Openvmm Troubleshooting (`Guide/src/user_guide/openvmm/troubleshooting.md`)

OpenVMM Troubleshooting

This page includes a miscellaneous collection of troubleshooting tips for common
issues you may encounter when running OpenVMM.

If you are still running into issues, consider filing an issue on the OpenVMM
GitHub Issue tracker.

failed to open `/dev/kvm/`

Error:

text

fatal error: failed to launch vm worker

Caused by:
    0: failed to launch worker
    1: failed to create the prototype partition
    2: kvm error
    3: failed to open /dev/kvm
    4: Permission denied (os error 13)

Solution:

When launching from a Linux/WSL host, your user account will need permission to
interact with /dev/kvm.

For example, you could add yourself to the group that owns that file:

bash

sudo usermod -a -G <group> <username>

For this change to take effect, you may need to restart. If using WSL2, you can
simply restart WSL2 (run wsl --shutdown from Powershell and reopen the WSL
window).

Alternatively, for a quick-and-dirty solution that will only persist for the
duration of the current user session:

bash

sudo chown <username> /dev/kvm

Guide Src User Guide Openvmm Vfio (`Guide/src/user_guide/openvmm/vfio.md`)

VFIO Device Assignment

This page explains how to assign a physical PCI device to an OpenVMM guest using Linux VFIO.

VFIO device assignment lets a guest VM directly access a physical PCI device (such as an NVMe controller or GPU). The guest sees the real device in its PCI bus and can interact with its config space.

admonish warning

VFIO device assignment is experimental. PCI config space, BAR MMIO passthrough, MSI-X interrupts (via irqfd), and DMA are functional. Devices such as NVMe controllers work end-to-end.

Overview

OpenVMM running on a Linux host can assign physical PCI devices to guest VMs using VFIO. The device is bound to the vfio-pci kernel driver, then OpenVMM opens it via VFIO and presents it to the guest as a PCIe endpoint.

The guest sees the device's real config space with some filtering applied:
each assigned device appears as a single-function device (the multi-function
bit is cleared), and certain PCIe extended capabilities are hidden because they
don't make sense in a virtual topology. Currently filtered extended
capabilities are SR-IOV, ARI, and Resizable BAR.

text

Linux Host
└── OpenVMM
    └── Guest VM
        └── sees physical PCI device on its PCI bus

Prerequisites

A Linux host with IOMMU support enabled (Intel VT-d or AMD-Vi)
A PCI device available for passthrough
The vfio-pci kernel module loaded
The vfio_iommu_type1 kernel module loaded

Step 1: Identify the device

Find the PCI device you want to assign:

bash

lspci -D

Look for the device's PCI address in domain:bus:device.function format, for example 0000:01:00.0.

Step 2: Enable unsafe interrupts

Some IOMMU implementations do not support interrupt remapping. If VFIO fails to set up the IOMMU with an "interrupt remapping" error, allow it to proceed without:

bash

echo 1 | sudo tee /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts

admonish note

This flag is required in environments where the IOMMU does not support interrupt remapping (e.g., some nested virtualization setups). The "unsafe" label means a device could theoretically forge interrupt messages. In practice, this is acceptable when the host platform already constrains device behavior.

To make this persistent across reboots, add a modprobe config:

bash

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" | sudo tee /etc/modprobe.d/vfio.conf

Step 3: Bind the device to vfio-pci

If the device is currently bound to another driver (e.g., nvme), unbind it first:

bash

echo "0000:01:00.0" | sudo tee /sys/bus/pci/devices/0000:01:00.0/driver/unbind

Then bind to vfio-pci:

bash

echo "vfio-pci" | sudo tee /sys/bus/pci/devices/0000:01:00.0/driver_override
echo "0000:01:00.0" | sudo tee /sys/bus/pci/drivers/vfio-pci/bind

Verify the binding:

bash

ls -la /sys/bus/pci/devices/0000:01:00.0/driver
# Should show: ... -> ../../../../bus/pci/drivers/vfio-pci

admonish warning

If the device is an NVMe controller backing a mounted filesystem, unbinding it will cause data loss. Make sure you are not using the device before unbinding.

Step 4: Verify VFIO group

Check that a VFIO group device was created:

bash

ls /dev/vfio/

You should see a numbered group (e.g., /dev/vfio/0). An IOMMU group is required — NoIommu mode is not supported.

Step 5: Launch OpenVMM with the VFIO device

Use the --vfio flag to assign the device to a PCIe root port. You also need to create a PCIe root complex and root port for the device to attach to:

bash

sudo openvmm \
  --pcie-root-complex rc0 \
  --pcie-root-port rc0:rp0 \
  --vfio host=0000:01:00.0,port=rp0 \
  --kernel /path/to/vmlinux \
  --initrd /path/to/initrd \
  --cmdline "console=ttyS0" \
  --com1 console \
  --memory 256M \
  --processors 2

The --vfio value is a comma-separated list of key=value pairs:

host= (required) — the PCI BDF of the VFIO device on the host (e.g., 0000:01:00.0)
port=<name> (required) — the name of the PCIe root port to attach the device to (must match a --pcie-root-port name)
iommu= (optional) — reference to an --iommu context; see Using iommufd (cdev path) below
bar0=host through bar5=host (optional) — pin the specified BAR to its
physical host address (GPA = HPA); see Peer-to-peer DMA
below
bar0=0x<addr> through bar5=0x<addr> (optional) — pin the specified BAR
to an explicit host physical address; use this for BARs synthesized by a
VFIO variant driver whose address is not reported through sysfs

admonish tip

You can assign multiple devices by adding more root ports and `--vfio` flags:

    --pcie-root-port rc0:rp0 \
    --pcie-root-port rc0:rp1 \
    --vfio host=0000:01:00.0,port=rp0 \
    --vfio host=334c:00:00.0,port=rp1

Using iommufd (cdev path)

By default, --vfio uses the legacy VFIO group/container interface with the
Type1v2 IOMMU driver. On hosts with Linux kernel 6.6 or newer, OpenVMM can
instead use the modern VFIO cdev (per-device fd) + iommufd interface. Enable
it by declaring an --iommu context and referencing it from each --vfio
device with the iommu= key:

bash

sudo openvmm \
  --pcie-root-complex rc0 \
  --pcie-root-port rc0:rp0 \
  --iommu id=iommu0 \
  --vfio host=0000:01:00.0,port=rp0,iommu=iommu0 \
  ...

The --iommu syntax is id=<name>. All --vfio devices that reference the
same id share a single iommufd IOAS (one set of IOMMU page tables and one
DMA mapper registration). The IOAS is allocated on demand the first time a
device referencing the id is opened.

Devices opened via the cdev path read their device node from
/sys/bus/pci/devices//vfio-dev/vfioN and open
/dev/vfio/devices/vfioN instead of /dev/vfio/<group>.

Assigning a device behind a guest IOMMU (aarch64)

By default an assigned device sees guest physical addresses directly: the
guest programs DMA with GPAs and the host IOMMU maps them to host physical
addresses. To instead expose an IOMMU to the guest — so the guest builds
its own IOVA→GPA translations — put the device's root complex behind an
accelerated SMMUv3 with --smmu rc=<name>,accel:

bash

sudo openvmm \
  --pcie-root-complex rc0 \
  --pcie-root-port rc0:rp0 \
  --smmu rc=rc0,accel \
  --iommu id=iommu0 \
  --vfio host=0000:01:00.0,port=rp0,iommu=iommu0 \
  ...

The guest's stage-1 page tables are installed into the host IOMMU as a
nested domain, so the physical SMMU performs the translation. This needs a
host SMMUv3 with iommufd nesting support, every device behind the same SMMU
must reference the same --iommu id, and the guest must boot with ACPI
(--device-tree is rejected).

Omitting accel leaves the SMMU translating in software, which cannot
program the host IOMMU — assignment then fails rather than silently
bypassing the guest's translations. See
Arm SMMUv3 for the device
reference.

Step 6: Verify in the guest

If the guest boots with PCI support, the assigned device should be visible:

bash

lspci

The device will appear with its real vendor and device ID from the physical hardware.

Peer-to-peer DMA

Normally, peer-to-peer (P2P) DMA between two passthrough devices works
via ATS (Address Translation Services): each device translates DMA
addresses through the IOMMU, so guest BAR placement doesn't matter.

Some platforms — notably NVIDIA GB200 and GB300 — do not support ATS in
their root complex. On these machines, P2P DMA between devices (e.g., GPU
and NIC) works by disabling ACS on the physical PCIe switch so that TLPs
route directly between devices without going through the IOMMU. Since
there is no translation layer, the devices use raw physical addresses for
P2P DMA. This means the guest BAR addresses must be identity-mapped to
the host BAR addresses (GPA = HPA), or P2P DMA will target the wrong
location.

To enable this, pin the relevant BARs with bar<N>=host on each --vfio
device and set preserve_bars on the root complex so the PCI resource
allocator keeps pinned BARs at their physical addresses:

bash

sudo openvmm \
  --pcie-root-complex \
    rc0,preserve_bars,low_mmio_base=0xc0000000,high_mmio_base=0x100000000 \
  --pcie-root-port rc0:rp0 \
  --pcie-root-port rc0:rp1 \
  --vfio host=0000:01:00.0,port=rp0,bar0=host \
  --vfio host=0000:02:00.0,port=rp1,bar0=host \
  ...

For a BAR synthesized by a VFIO variant driver, the host physical address may
not appear in the device's sysfs resource table. In that case, provide the
address directly with bar<N>=0x<addr>. For example, the nvgrace-gpu driver
exposes CPU-coherent GPU memory as a synthetic 64-bit BAR4 whose physical base
comes from the nvidia,gpu-mem-base-pa ACPI _DSD property:

bash

--vfio host=0008:06:00.0,port=rp0,bar0=host,bar2=host,bar4=0x110000000000

The low_mmio_base= and high_mmio_base= options pin the MMIO apertures
to fixed addresses so the allocator can place both pinned and dynamic BARs
correctly.

Optional: use hugetlb-backed guest RAM

For large VMs, VFIO DMA setup can be much faster when guest RAM is backed by
Linux hugetlb pages. Without hugetlb backing, the kernel may have to pin each
4 KB page individually during VFIO_IOMMU_MAP_DMA. With 2 MB or 1 GB hugetlb
pages, the same mapping work is performed over far fewer pages.

Hugetlb pages must be reserved on the host before starting OpenVMM. For
example, reserve enough 2 MB pages for a 64 GB VM:

bash

echo 32768 | sudo tee /proc/sys/vm/nr_hugepages

For 1 GB pages, reserve pages through the size-specific sysfs pool:

bash

echo 64 | sudo tee \
  /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

Then request hugepage-backed RAM with the --memory option:

bash

sudo openvmm \
  --pcie-root-complex rc0 \
  --pcie-root-port rc0:rp0 \
  --vfio host=0000:01:00.0,port=rp0 \
  --kernel /path/to/vmlinux \
  --initrd /path/to/initrd \
  --cmdline "console=ttyS0" \
  --com1 console \
  --memory size=64GB,hugepages=on,hugepage_size=2MB \
  --processors 16

Use hugepage_size=1GB to request 1 GB pages. If hugepage_size is omitted,
OpenVMM uses 2 MB pages.

admonish note

Hugepage-backed RAM is Linux-only. It cannot be combined with file-backed
memory, private memory (`shared=off`), or legacy x86 RAM splitting such as
PCAT firmware.

Troubleshooting

"No such file or directory" for `/dev/vfio/*`

Make sure you completed Step 2 (allow unsafe interrupts) and that the device is bound to vfio-pci. An IOMMU must be available — NoIommu mode is not supported.

"No interrupt remapping" / ENODEV on IOMMU setup

Run Step 2 to enable allow_unsafe_interrupts. This is needed when the platform's IOMMU does not support interrupt remapping.

"failed to open VFIO device" / permission denied

Run OpenVMM with sudo, or add your user to the vfio group and set appropriate permissions on /dev/vfio/ devices.

Device not visible in guest `lspci`

Verify the device is bound to vfio-pci (Step 3)
Verify the VFIO group exists in /dev/vfio/ (Step 4)
Verify the --vfio port name matches a --pcie-root-port name
Check OpenVMM log output for errors during VFIO device initialization

Hugepage allocation fails

If OpenVMM fails while allocating guest RAM with hugepages=on, verify that
the host has enough free pages in the requested hugetlb pool:

bash

grep -i Huge /proc/meminfo

For 1 GB pages, also check the size-specific pool:

bash

grep . /sys/kernel/mm/hugepages/hugepages-1048576kB/*

Current Limitations

No save/restore — VMs with VFIO devices cannot be saved or migrated.
No hot-plug — VFIO devices must be specified at VM creation time and cannot be added or removed while the VM is running.
Linux only — the --vfio flag is only available when OpenVMM is built and run on Linux. On Windows, use --device with WHP for device assignment.

Guide Src User Guide Openvmm Vm Configurations (`Guide/src/user_guide/openvmm/vm_configurations.md`)

VM Configurations: Gen1 vs Gen2 Equivalents

If you're familiar with Hyper-V's Gen1 and Gen2 VM concepts, this page maps
those to the equivalent OpenVMM CLI flags.

Background

Hyper-V defines two VM "generations" that differ in firmware, device model, and
boot mechanism:

	Gen1	Gen2
Firmware	BIOS (PCAT)	UEFI
Boot disk	IDE	SCSI (VMBus storvsp)
Guest OS	Legacy and modern (older Windows, DOS, Linux with BIOS support)	Modern (Windows 10+, most Linux)
Secure Boot	Not available	Available

OpenVMM doesn't use the "Gen1/Gen2" terminology — you select the components
directly via CLI flags.

Gen2-equivalent (UEFI boot) — the common case

Most development and testing uses UEFI boot. This is the default for modern
Windows and Linux guests.

bash

cargo run -- \
  --uefi \
  --vmbus-scsi id=scsi0 \
  --disk memdiff:file:path/to/disk.vhdx,on=scsi0 \
  -p 4 -m 4GB \
  --gfx

Key flags:

--uefi — boot using mu_msvm UEFI firmware (implicitly enables Hyper-V
enlightenments and VMBus, so --hv is not needed separately)
--vmbus-scsi — creates a named SCSI controller
--disk ...,on=scsi0 — attaches a disk to that controller

Gen1-equivalent (PCAT BIOS boot)

Use PCAT for operating systems that support BIOS boot. This includes legacy
systems (DOS, older Windows) as well as modern OSes that still support BIOS
boot (most Linux distributions, Windows 10+).

bash

cargo run -- \
  --pcat \
  --ide memdiff:file:path/to/disk.vhd \
  --gfx

Key flags:

--pcat — boot using the Microsoft Hyper-V PCAT BIOS
--ide — expose a disk via emulated IDE controller (the traditional Gen1
storage path, no --hv required)

See the PCAT BIOS reference for more
details on PCAT boot, including floppy and optical boot order.

With OpenHCL (VTL2)

To run with OpenHCL, add --hv --vtl2 and
--igvm firmware=path/to/openhcl.igvm,personality=openhcl. You don't need to
separately specify --uefi or --pcat — the IGVM file contains the OpenHCL
paravisor, and most IGVM builds bundle the
mu_msvm UEFI firmware
for VTL0 guest boot. The build recipe controls whether UEFI is included
(see IGVM architecture).

Note: --vtl2 requires --hv to be passed explicitly on the command line,
even though other flags like --uefi imply it internally.

bash

cargo run -- \
  --hv --vtl2 \
  --igvm firmware=path/to/openhcl.igvm,personality=openhcl \
  --vmbus-scsi id=scsi0 \
  --disk memdiff:file:path/to/disk.vhdx,on=scsi0 \
  -p 4 -m 4GB

admonish note

OpenVMM does not currently support OpenHCL with PCAT (Gen1-style) boot.
OpenHCL + PCAT is supported on Hyper-V, where the host provides the PCAT
firmware. If you need Gen1-style boot with OpenHCL, use
[Hyper-V](../openhcl/run/hyperv.md) rather than openvmm standalone.

See Running OpenHCL with OpenVMM
for full setup instructions.

Quick reference

Scenario	Flags	Notes
Modern Windows/Linux guest	`--uefi --vmbus-scsi id=scsi0 --disk memdiff:file:disk.vhdx,on=scsi0`	Most common
With graphical console	add `--gfx`	VNC-based, see Graphical Console
With networking	add `--nic`	Consomme user-mode NAT
With OpenHCL	`--hv --vtl2 --igvm firmware=path/to/openhcl.igvm,personality=openhcl --vmbus-scsi id=scsi0 --disk memdiff:file:disk.vhdx,on=scsi0`	IGVM carries the paravisor; no `--uefi`/`--pcat` needed
Legacy OS (DOS, old Windows)	`--pcat --ide memdiff:file:disk.vhd --gfx`	IDE storage, BIOS boot
Linux direct boot (no firmware)	`--kernel vmlinux --initrd initrd`	Skips UEFI/PCAT entirely

Guide Src User Guide Openvmm Vm Memory Dumps (`Guide/src/user_guide/openvmm/vm_memory_dumps.md`)

VM Memory Dumps

OpenVMM can dump a running VM's processor state and memory to a .vmrs
file compatible with WinDbg. This is useful for offline debugging of
guest crashes without needing a live debugger attached.

Overview

The dump-state command captures:

VP registers — general-purpose, control, segment, table, debug,
and FP/XSAVE registers for every virtual processor
Guest RAM — the full contents of guest physical memory, streamed
in 1 MiB blocks

The output is a .vmrs file that can be opened in WinDbg via the
VmSavedStateDumpProvider.dll from the Windows SDK.

Usage

While the VM is running, open the interactive console and run:

text

dump-state path/to/dump.vmrs

The shorter alias dump also works:

text

dump path/to/dump.vmrs

OpenVMM pauses the VM, collects state, streams guest memory to disk,
and then resumes the VM. If the VM was already paused, it remains
paused after the dump completes.

The file is written atomically — a temporary file is created first
and renamed into place on success, so readers never see a
partially-written dump.

Dumping automatically on a guest crash

To capture a dump automatically whenever the guest triple-faults, pass
--crash-dump-path on the command line:

text

openvmm --crash-dump-path path/to/crash.vmrs ...

When the guest crashes, OpenVMM writes the .vmrs dump to that path
before applying the configured --guest-crash-action (halt, reset,
or exit). If the dump fails, the error is logged and the crash action
still runs. Each crash overwrites the file at the given path.

Opening in WinDbg

Install the Windows SDK
(the "Debugging Tools for Windows" component includes
VmSavedStateDumpProvider.dll).
Open the .vmrs file in WinDbg:
text
```
windbg -z path/to/dump.vmrs
```

WinDbg will load the saved processor state and allow you to inspect
registers, stack traces, and memory contents as if attached to a live
VM.

admonish note

Unlike snapshots, dump files do **not** capture device state and cannot
be used to restore a VM. They are strictly for offline debugging.

Guide Src Dev Guide Dev Tools Flowey Artifacts (`Guide/src/dev_guide/dev_tools/flowey/artifacts.md`)

Artifacts

Artifacts enable typed data transfer between jobs with automatic dependency management, abstracting away CI system complexities like name collisions and manual job ordering.

Typed vs Untyped Artifacts

Typed artifacts (recommended) provide type-safe artifact handling by defining
a custom type that implements the Artifact trait:

rust,ignore

#[derive(Serialize, Deserialize)]
struct MyArtifact {
    #[serde(rename = "output.bin")]
    binary: PathBuf,
    #[serde(rename = "metadata.json")]
    metadata: PathBuf,
}

impl Artifact for MyArtifact {}

let (pub_artifact, use_artifact) = pipeline.new_typed_artifact("my-files");

Untyped artifacts provide simple directory-based artifacts for simpler cases:

rust,ignore

let (pub_artifact, use_artifact) = pipeline.new_artifact("my-files");

For detailed examples of defining and using artifacts, see the Artifact trait documentation.

Both pipeline.new_typed_artifact("name") and pipeline.new_artifact("name") return a tuple of handles: (pub_artifact, use_artifact). When defining a job you convert them with the job context:

rust,ignore

// In a producing job:
let artifact_out = ctx.publish_artifact(pub_artifact);
// artifact_out : WriteVar<MyArtifact>   (typed)
// or WriteVar<PathBuf> for untyped

// In a consuming job:
let artifact_in = ctx.use_artifact(use_artifact);
// artifact_in : ReadVar<MyArtifact>     (typed)
// or ReadVar<PathBuf> for untyped

After conversion, you treat the returned WriteVar / ReadVar like any other flowey variable (claim them in steps, write/read values).
Key concepts:

The Artifact trait works by serializing your type to JSON in a format that reflects a directory structure
Use #[serde(rename = "file.exe")] to specify exact file names
Typed artifacts ensure compile-time type safety when passing data between jobs
Untyped artifacts are simpler but don't provide type guarantees
Tuple handles must be lifted with ctx.publish_artifact(...) / ctx.use_artifact(...) to become flowey variables

How Flowey Manages Artifacts Under the Hood

During the pipeline resolution phase (build-time), flowey:

Identifies artifact producers and consumers by analyzing which jobs write to vs read from each artifact's WriteVar/ReadVar
Constructs the job dependency graph ensuring producers run before consumers
Generates backend-specific upload/download steps in the appropriate places:
- For ADO: Uses PublishPipelineArtifact and DownloadPipelineArtifact tasks
- For GitHub Actions: Uses actions/upload-artifact and actions/download-artifact
- For local execution: Uses filesystem copying

At runtime, the artifact ReadVar and WriteVar work just like any other flowey variable:

Producing jobs write artifact files to the path from WriteVar
Flowey automatically uploads those files as an artifact
Consuming jobs read the path from ReadVar where flowey has downloaded the artifact

Guide Src Dev Guide Dev Tools Flowey Flowey Fundamentals (`Guide/src/dev_guide/dev_tools/flowey/flowey_fundamentals.md`)

Flowey Fundamentals

Before diving into how flowey works, let's establish the key building blocks that form the foundation of flowey's automation model. These concepts are flowey's Rust-based abstractions for common CI/CD workflow primitives.

The Automation Workflow Model

In traditional CI/CD systems, workflows are defined using YAML with implicit dependencies and global state. Flowey takes a fundamentally different approach: automation workflows are modeled as a directed acyclic graph (DAG) of typed, composable Rust components. Each component has explicit inputs and outputs, and dependencies are tracked through the type system.

Core Building Blocks

Flowey's model consists of a hierarchy of components:

Pipelines are the top-level construct that defines a complete automation workflow. A pipeline specifies what work needs to be done and how it should be organized. Pipelines can target different execution backends (local machine, Azure DevOps, GitHub Actions) and generate appropriate configuration for each.

Jobs represent units of work that run on a specific platform (Windows, Linux, macOS) and architecture (x86_64, Aarch64). Jobs can run in parallel when they don't depend on each other, or sequentially when one job's output is needed by another. Each job is isolated and runs in its own environment.

Nodes are reusable units of automation logic that perform specific tasks (e.g., "install Rust toolchain", "run cargo build", "publish test results"). Nodes are invoked by jobs and emit one or more steps to accomplish their purpose. Nodes can depend on other nodes, forming a composable ecosystem of automation building blocks.

Steps are the individual units of work that execute at runtime. A step might run a shell command, execute Rust code, or interact with the CI backend. Steps are emitted by nodes during the build-time phase and executed in dependency order during runtime.

Connecting the Pieces

These building blocks are connected through three key mechanisms:

Variables (ReadVar/WriteVar) enable data flow between steps. A WriteVar<T> represents a promise to produce a value of type T at runtime, while a ReadVar<T> represents a dependency on that value. Variables enforce write-once semantics (each value has exactly one producer) and create explicit dependencies in the DAG. For example, a "build" step might write a binary path to a WriteVar, and a "test" step would read from the corresponding ReadVar. This echoes Rust's "shared XOR mutable" ownership rule: a value has either one writer or multiple readers, never both concurrently.

Artifacts enable data transfer between jobs. Since jobs may run on different machines or at different times, artifacts package up files (like compiled binaries, test results, or build outputs) for transfer. Flowey automatically handles uploading artifacts at the end of producing jobs and downloading them at the start of consuming jobs, abstracting away backend-specific artifact APIs.

Side Effects represent dependencies without data. Sometimes step B needs to run after step A, but A doesn't produce any data that B consumes (e.g., "install dependencies" must happen before "run tests", even though the test step doesn't directly use the installation output). Side effects are represented as ReadVar<SideEffect> and establish ordering constraints in the DAG without transferring actual values.

Putting It Together

Here's an example of how these pieces relate:

txt

Pipeline
  ├─ Job 1 (Linux x86_64)
  │   ├─ Node A (install Rust) 
  │   │   └─ Step: Run rustup install
  │   │       └─ Produces: WriteVar<SideEffect> (installation complete)
  │   └─ Node B (build project)
  │       └─ Step: Run cargo build
  │           └─ Consumes: ReadVar<SideEffect> (installation complete)
  │           └─ Produces: WriteVar<PathBuf> (binary path) → Artifact
  │
  └─ Job 2 (Windows x86_64)
      └─ Node C (run tests)
          └─ Step: Run binary with test inputs
              └─ Consumes: ReadVar<PathBuf> (binary path) ← Artifact
              └─ Produces: WriteVar<PathBuf> (test results)

In this example:

The Pipeline defines two jobs that run on different platforms
Job 1 installs Rust and builds the project, with step dependencies expressed through variables
Job 2 runs tests using the binary from Job 1, with the binary transferred via an artifact
Variables create dependencies within a job (build depends on install)
Artifacts create dependencies between jobs (Job 2 depends on Job 1's output)
Side Effects represent the "Rust is installed" state without carrying data

Two-Phase Execution Model

Flowey operates in two distinct phases:

Build-Time (Resolution Phase): When you run cargo xflowey regen, flowey:
- Reads .flowey.toml to determine which pipelines to regenerate
- Builds the flowey binary (e.g., flowey-hvlite) via cargo build
- Runs the flowey binary with pipeline --out <file> <cmd> for each pipeline definition
- During this invocation, flowey constructs a directed acyclic graph (DAG) by:
  - Instantiating all nodes (reusable units of automation logic) defined in the pipeline
  - Processing their requests
  - Resolving dependencies between nodes via variables and artifacts
  - Determining the execution order
  - Performing flowey-specific validations (dependency resolution, type checking, etc.)
- Generates YAML files for CI systems (ADO, GitHub Actions) at the paths specified in .flowey.toml
Runtime (Execution Phase): The generated YAML is executed by the CI system (or locally via cargo xflowey ). Steps (units of work) run in the order determined at build-time:
- Variables are read and written with actual values
- Commands are executed
- Artifacts (data packages passed between jobs) are published/consumed
- Side effects (dependencies) are resolved

The .flowey.toml file at the repo root defines which pipelines to generate and where. For example:

toml

[[pipeline.flowey_hvlite.github]]
file = ".github/workflows/openvmm-pr.yaml"
cmd = ["ci", "checkin-gates", "--config=pr"]

When you run cargo xflowey regen:

It reads .flowey.toml
Builds the flowey-hvlite binary
Runs flowey-hvlite pipeline github --out .github/workflows/openvmm-pr.yaml ci checkin-gates --config=pr
This generates/updates the YAML file with the resolved pipeline

Key Distinction:

cargo build -p flowey-hvlite - Only compiles the flowey code to verify it builds successfully. Does not construct the DAG or generate YAML files.
cargo xflowey regen - Compiles the code and runs the full build-time resolution to construct the DAG, validate the pipeline, and regenerate all YAML files defined in .flowey.toml.

Always run cargo xflowey regen after modifying pipeline definitions to ensure the generated YAML files reflect your changes.

Backend Abstraction

Flowey supports multiple execution backends:

Local: Runs directly on your development machine
ADO (Azure DevOps): Generates ADO Pipeline YAML
GitHub Actions: Generates GitHub Actions workflow YAML

admonish warning

Nodes should be written to work across ALL backends whenever possible. Relying on `ctx.backend()` to query the backend or manually emitting backend-specific steps (via `emit_ado_step` or `emit_gh_step`) should be avoided unless absolutely necessary. Most automation logic should be backend-agnostic, using `emit_rust_step` for cross-platform Rust code that works everywhere. Writing cross-platform flowey code enables locally testing pipelines which can be invaluable when iterating over CI changes.

If a node only supports certain backends, it should immediately fast‑fail with a clear error ("<Node> not supported on ``") instead of silently proceeding. That failure signals it's time either to add the missing backend support or introduce a multi‑platform abstraction/meta‑node that delegates to platform‑specific nodes.

Guide Src Dev Guide Dev Tools Flowey Nix (`Guide/src/dev_guide/dev_tools/flowey/nix.md`)

Nix

In order to enable reproducible builds, experimental flowey features are being built to utilize the Nix package manager for dependency management and environment isolation.
The root of the nix configuration lives in shell.nix. If you're unsure where the Nix definition is for a dependency, you should be able to track it down from there.

Updating Nix Packages

Nix dependencies require a hash of their contents to ensure integrity and reproducibility. When updating a dependency, you'll need to update the release that's being pulled and its corresponding hash.

For instance, let's say we have a new release of the OpenHCL Kernel and we want to update it in our Nix configuration:

Go to the corresponding .nix file (in this case, openhcl_kernel.nix)
Clear the hash to an empty string
Update the version
Run nix-shell --pure and use the printed error to get the new hash

Warning: Because Nix caches dependencies based on the hash, if you don't clear the hash to an empty string before updating the version, nix-shell --pure will run without error even though the dependency hasn't actually been updated.

Here's an example of what the error will look like when done correctly:

bash

error: hash mismatch in fixed-output derivation '/nix/store/cc7hhyslx1dnw01nmjx11zqim2l50awp-source.drv':
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-wUDWFazJM80oztKqpuRwj8Wvto2Uo/OuVGvhpszIw+A=
error: Cannot build '/nix/store/spg5vbm6mzmsxpg5v2ibg97qrz8khc70-openhcl-kernel-x64-6.12.52.4.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /nix/store/kv8s7pld70yvzxzd77swz9hb3pygkrhl-openhcl-kernel-x64-6.12.52.4

Given this error, you would update the corresponding hash to sha256-An1N76i1MPb+rrQ1nBpoiuxnNeD0E+VuwqXdkPzaZn0= in the openhcl_kernel.nix file.

Updating `mu_msvm` UEFI (`nix/uefi_mu_msvm.nix`)

When bumping mu_msvm, update both the Nix fetch and the Flowey version
constant in the same PR.

Update nix/uefi_mu_msvm.nix:
- Set version to the new release.
- Ensure the fetchzip URL uses that version.
- Set both architecture hashes to empty strings.
Update flowey/flowey_lib_hvlite/src/_jobs/cfg_versions.rs:
- Bump MU_MSVM to the same version.
From the repo root, run sudo nix-shell (or nix-shell if sudo is not
required in your environment).
Copy the first got: hash from the mismatch error into the matching entry
in nix/uefi_mu_msvm.nix.
Run sudo nix-shell again to fetch the other architecture and copy its
got: hash.
Re-run sudo nix-shell a final time to verify both hashes are correct.

Expected mismatch output:

bash

error: hash mismatch in fixed-output derivation '/nix/store/...-source.drv':
      specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
        got:    sha256-<new hash from downloaded artifact>

Use the got: value in the .nix file.

OpenHCL dev kernel packages

The Nix shell does not evaluate or export OpenHCL dev kernel packages by
default. The main and dev kernels often contain the same promoted changes, so
building both variants adds unnecessary download and build work.

To restore the dev kernel packages when the branches diverge:

Set enableDevKernel to true in shell.nix.
Update the dev version in nix/openhcl_kernel.nix.
Clear and replace the dev hashes using the procedure above.

This restores the NIX_KERNEL_*_DEV paths and
CARGO_BUILD_ARGS_*_DEVKERN arguments in the Nix shell.

Updating the Rust Version

The Nix shell derives its Rust toolchain version from rust-version in the root Cargo.toml and resolves it against a pinned rust-overlay commit in shell.nix. When rust-version is bumped in Cargo.toml, the pinned rust-overlay may not yet include the new version, causing an error like:

bash

error: No rust version matching 1.XX.* found in rust-overlay

To fix this, update the rust-overlay pin in shell.nix to a commit that includes the new Rust version:

Go to the rust-overlay stable branch commits and copy the latest commit SHA
In shell.nix, find the rust_overlay fetchTarball block and replace the commit SHA in the url with the new one (the long hex string in the URL path, e.g., https://github.com/oxalica/rust-overlay/archive/<commit-sha>.tar.gz)
Clear the sha256 field to an empty string — this is a content hash of the tarball (not the commit SHA), and Nix will compute the correct value for you
Run nix-shell --pure and use the printed error to get the new sha256
Update the sha256 with the correct hash from the error

Guide Src Dev Guide Dev Tools Flowey Nodes (`Guide/src/dev_guide/dev_tools/flowey/nodes.md`)

Nodes

At a conceptual level, a Flowey node is analogous to a strongly typed function: you "invoke" it by submitting one or more Request values (its parameters), and it responds by emitting steps that perform work and produce outputs (values written to WriteVars, published artifacts, or side-effect dependencies).

The Node/Request Pattern

Every node has an associated Request type that defines what operations the node can perform. Requests are defined using the flowey_request! macro and registered with new_flow_node! or new_simple_flow_node! macros.

For complete examples, see the FlowNode trait documentation.

FlowNode vs SimpleFlowNode

Flowey provides two node implementation patterns with a fundamental difference in their Request structure and complexity:

SimpleFlowNode - for straightforward, function-like operations:

Uses a single struct Request type
Processes one request at a time independently
Behaves like a "plain old function" that resolves its single request type
Each invocation is isolated - no shared state or coordination between requests
Simpler implementation with less boilerplate
Ideal for straightforward operations like running a command or transforming data

Example use case: A node that runs cargo build - each request is independent and just needs to know what to build.

FlowNode - for complex nodes requiring coordination and non-local configuration:

Often uses an enum Request with multiple variants
Receives all requests as a Vec<Request> and processes them together
Can aggregate, optimize, and consolidate multiple requests into fewer steps
Enables non-local configuration - critical for simplifying complex pipelines

The Non-Local Configuration Pattern

The key advantage of FlowNode is its ability to accept configuration from different parts of the node graph without forcing intermediate nodes to be aware of that configuration. This is the "non-local" aspect.

For nodes that accept config values — versions, feature flags, local paths — use FlowNodeWithConfig with a typed config struct. Config is declared with the flowey_config! macro, set via ctx.config(...) by any caller, and automatically merged across all callers before being delivered to emit() separately from action requests.

Consider an "install Rust toolchain" node:

rust,ignore

flowey_config! {
    pub struct Config {
        pub version: Option<String>,
        pub auto_install: Option<bool>,
    }
}

flowey_request! {
    pub enum Request {
        GetToolchain(WriteVar<PathBuf>),
    }
}

new_flow_node_with_config!(struct Node);

impl FlowNodeWithConfig for Node {
    type Request = Request;
    type Config = Config;

    fn imports(ctx: &mut ImportCtx<'_>) { /* ... */ }

    fn emit(
        config: Config,
        requests: Vec<Self::Request>,
        ctx: &mut NodeCtx<'_>,
    ) -> anyhow::Result<()> {
        let version = config.version
            .expect("version must be set by cfg_versions");
        // ... use version to install, then fulfill GetToolchain requests
        Ok(())
    }
}

Callers set config and submit requests independently:

rust,ignore

// A top-level job configuration node sets the version once:
ctx.config(install_rust::Config {
    version: Some("1.75".into()),
    ..Default::default()
});

// Any node that needs the Rust toolchain just requests it:
let toolchain = ctx.reqv(|v| install_rust::Request::GetToolchain(v));

txt

Root Node → config(version: "1.75")
  → Node A
    → Node B
      → Node C → req(GetToolchain)

Flowey merges all config partials automatically. If two callers set the same field, the values must agree or the build fails. The intermediate nodes (A, B, C) never need to know about or pass through the version.

This pattern:

Eliminates plumbing complexity in large pipelines
Allows global configuration to be set once at the top level
Keeps unrelated nodes decoupled from configuration they don't need
Validates consistency — conflicting config values are caught at build time
Separates concerns — config (what version?) is distinct from requests (give me the toolchain path)

Additional Benefits of FlowNode:

Optimize and consolidate multiple similar requests into fewer steps (e.g., installing a tool once for many consumers)
Resolve conflicts or enforce consistency across requests

For detailed comparisons and examples, see the FlowNode and SimpleFlowNode documentation.

Node Registration

Nodes are automatically registered using macros that handle most of the boilerplate:

new_flow_node!(struct Node) - registers a FlowNode
new_flow_node_with_config!(struct Node) - registers a FlowNodeWithConfig (a FlowNode that also accepts typed config)
new_simple_flow_node!(struct Node) - registers a SimpleFlowNode
flowey_request! - defines the Request type and implements IntoRequest
flowey_config! - defines a Config struct with automatic merge support and implements IntoConfig

The imports() Method

The imports() method declares which other nodes this node might depend on. This enables flowey to:

Validate that all dependencies are available
Build the complete dependency graph
Catch missing dependencies at build-time

admonish warning

Flowey does not catch unused imports today as part of its build-time validation step.

Why declare imports? Flowey needs to know the full set of potentially-used nodes at compilation time to properly resolve the dependency graph.

For more on node imports, see the FlowNode::imports documentation.

The emit() Method

The emit() method is where a node's actual logic lives. For FlowNode, it receives all requests together and must:

Aggregate and validate requests (ensuring consistency where needed)
Emit steps to perform the work
Wire up dependencies between steps via variables

For SimpleFlowNode, the equivalent process_request() method processes one request at a time.

For complete implementation examples, see the FlowNode::emit documentation.

Node Design Philosophy

Flowey nodes are designed around several key principles:

1. Composability

Nodes should be reusable building blocks that can be combined to build complex
workflows. Each node should have a single, well-defined responsibility.

❌ Bad: A node that "builds and tests the project"
✅ Good: Separate nodes for "build project" and "run tests"

2. Explicit Dependencies

Dependencies between steps should be explicit through variables, not implicit
through side effects.

❌ Bad: Assuming a tool is already installed
✅ Good: Taking a ReadVar<SideEffect> that proves installation happened

3. Backend Abstraction

Nodes should work across all backends when possible. Backend-specific behavior
should be isolated and documented.

4. Separation of Concerns

Keep node definition (request types, dependencies) separate from step
implementation (runtime logic):

Node definition: What the node does, what it depends on
Step implementation: How it does it

Common Patterns

Node Config vs Request

When designing a FlowNode, separate config (values that must be consistent across all callers) from requests (actions the node performs):

Config (flowey_config!): version strings, feature flags, local override paths, auto-install toggles. Use FlowNodeWithConfig and ctx.config(...). Config fields are Option<T> or BTreeMap<K, V> — flowey merges them automatically and errors on conflicts.
Requests (flowey_request!): actions that produce outputs, e.g. GetBinary(WriteVar). Multiple callers can each submit requests, and the node fulfills all of them.

admonish tip

If a value needs `same_across_all_reqs` validation, it should probably be a config field instead. The `flowey_config!` macro provides this validation automatically during merge.

Request Aggregation and Validation

When a FlowNode receives multiple requests, it may need to aggregate or validate them. Common techniques:

Iterate through all requests and separate them by variant
Collect output variables that can have multiple instances
Validate that required values were provided

For values that must be consistent across all callers (versions, flags), prefer using flowey_config! with FlowNodeWithConfig instead of manual validation — see The Non-Local Configuration Pattern above.

Conditional Execution Based on Backend/Platform

Nodes can query the current backend and platform to emit platform-specific or backend-specific steps. This allows nodes to adapt their behavior based on the execution environment.

Key concepts:

Use ctx.backend() to check if running locally, on ADO, or on GitHub Actions
Use ctx.platform() to check the operating system (Windows, Linux, macOS)
Use ctx.arch() to check the architecture (x86_64, Aarch64)
Emit different steps or use different tool configurations based on these values

When to use:

Installing platform-specific tools or dependencies
Using different commands on Windows vs Unix systems
Optimizing for local development vs CI environments

For more on backend and platform APIs, see the NodeCtx documentation.

Guide Src Dev Guide Dev Tools Flowey Pipelines (`Guide/src/dev_guide/dev_tools/flowey/pipelines.md`)

Pipelines

Pipelines define complete automation workflows consisting of jobs that run nodes. See the IntoPipeline trait documentation for detailed examples.

Pipeline Jobs

PipelineJob instances are configured using a builder pattern:

rust,ignore

let job = pipeline
    .new_job(platform, arch, "my-job")
    .with_timeout_in_minutes(60)
    .with_condition(some_param)
    .ado_set_pool("my-pool")
    .gh_set_pool(GhRunner::UbuntuLatest)
    .dep_on(|ctx| {
        // Define what nodes this job depends on
        some_node::Request { /* ... */ }
    })
    .finish();

Pipeline Parameters

Parameters allow runtime configuration of pipelines. In Azure DevOps, parameters appear as editable fields in the Run pipeline UI (name, description, default).

rust,ignore

// Define a boolean parameter
let verbose = pipeline.new_parameter_bool(
    "verbose",
    "Run with verbose output",
    ParameterKind::Stable,
    Some(false) // default value
);

// Use the parameter in a job
let job = pipeline.new_job(...)
    .dep_on(|ctx| {
        let verbose = ctx.use_parameter(verbose);
        // verbose is now a ReadVar<bool>
    })
    .finish();

Stable vs Unstable Parameters

Every parameter in flowey must be declared as either Stable or Unstable using ParameterKind. This classification determines the parameter's visibility and API stability:

Stable Parameters (ParameterKind::Stable)

Stable parameters represent a public, stable API for the pipeline:

External Visibility: The parameter name is exposed as-is in the generated CI YAML, making it callable by external pipelines and users.
API Contract: Once a parameter is marked stable, its name and behavior should be maintained for backward compatibility. Removing or renaming a stable parameter is a breaking change.
Use Cases:
- Parameters that control major pipeline behavior (e.g., enable_tests, build_configuration)
- Parameters intended for use by other teams or external automation
- Parameters documented as part of the pipeline's public interface

Unstable Parameters (ParameterKind::Unstable)

Unstable parameters are for internal use and experimentation:

Internal Only: The parameter name is prefixed with __unstable_ in the generated YAML (e.g., __unstable_debug_mode), signaling that it's not part of the stable API.
No Stability Guarantee: Unstable parameters can be renamed, removed, or have their behavior changed without notice. External consumers should not depend on them.
Use Cases:
- Experimental features or debugging flags
- Internal pipeline configuration that may change frequently
- Parameters for development/testing that shouldn't be used in production

Guide Src Dev Guide Dev Tools Flowey Steps (`Guide/src/dev_guide/dev_tools/flowey/steps.md`)

Steps

Steps are units of work that will be executed at runtime. Different
step types exist for different purposes.

Types of Steps

Rust Steps

Rust steps execute Rust code at runtime and are the most common step type in flowey.

emit_rust_step: The primary method for emitting steps that run Rust code. Steps can claim variables, read inputs, perform work, and write outputs. Returns an optional ReadVar<SideEffect> that other steps can use as a dependency.

emit_minor_rust_step: Similar to emit_rust_step but for steps that cannot fail (no Result return) and don't need visibility in CI logs. Used for simple transformations and glue logic. Using minor steps also improve performance, since there is a slight cost to starting and ending a 'step' in GitHub and ADO. During the build stage, minor steps that are adjacent to each other will get merged into one giant CI step.

emit_rust_stepv: Convenience method that combines creating a new variable and emitting a step in one call. The step's return value is automatically written to the new variable.

For detailed examples of Rust steps, see the NodeCtx emit methods documentation.

ADO Steps

emit_ado_step: Emits a step that generates Azure DevOps Pipeline YAML. Takes a closure that returns a YAML string snippet which is interpolated into the generated pipeline.

For ADO step examples, see the NodeCtx::emit_ado_step documentation.

GitHub Steps

emit_gh_step: Creates a GitHub Actions step using the fluent GhStepBuilder API. Supports specifying the action, parameters, outputs, dependencies, and permissions. Returns a builder that must be finalized with .finish(ctx).

For GitHub step examples, see the GhStepBuilder documentation.

Side Effect Steps

emit_side_effect_step: Creates a dependency relationship without executing code. Useful for aggregating multiple side effect dependencies into a single side effect. More efficient than emitting an empty Rust step.

For side effect step examples, see the NodeCtx::emit_side_effect_step documentation.

Isolated Working Directories and Path Immutability

admonish warning title="critical constraint"

**Each step gets its own fresh local working directory.** This avoids the "single global working directory dumping ground" common in bash + YAML systems.

However, while flowey variables enforce sharing XOR mutability at the type-system level, **developers must manually enforce this at the filesystem level**:

**Steps must NEVER modify the contents of paths referenced by `ReadVar<PathBuf>`.**

When you write a path to WriteVar, you're creating an immutable contract. Other steps reading that path must treat it as read-only. If you need to modify files from a ReadVar, copy them to your step's working directory.

Runtime Services

Runtime services provide the API available during step execution (inside the
closures passed to emit_rust_step, etc.).

RustRuntimeServices

RustRuntimeServices is the primary runtime service available in Rust steps. It provides:

Variable Operations

Reading and writing flowey variables
Secret handling (automatic secret propagation for safety)
Support for reading values of any type that implements ReadVarValue

Environment Queries

Backend identification (Local, ADO, or GitHub)
Platform detection (Windows, Linux, macOS)
Architecture information (x86_64, Aarch64)

AdoStepServices

AdoStepServices provides integration with Azure DevOps-specific features when emitting ADO YAML steps:

ADO Variable Bridge:

Convert ADO runtime variables (like BUILD.SOURCEBRANCH) into flowey vars
Convert flowey vars back into ADO variables for use in YAML
Handle secret variables appropriately

Repository Resources:

Resolve repository IDs declared as pipeline resources
Access repository information in ADO-specific steps

GhStepBuilder

GhStepBuilder is a fluent builder for constructing GitHub Actions steps with:

Step Configuration:

Specifying the action to use (e.g., actions/checkout@v4)
Adding input parameters via .with()
Capturing step outputs into flowey variables
Setting conditional execution based on variables

Dependency Management:

Declaring side-effect dependencies via .run_after()
Ensuring steps run in the correct order

Permissions:

Declaring required GITHUB_TOKEN permissions
Automatic permission aggregation at the job level

Secret Variables and CI Backend Integration

Flowey provides built-in support for handling sensitive data like API keys, tokens, and credentials through secret variables. Secret variables are treated specially to prevent accidental exposure in logs and CI outputs.

How Secret Handling Works

When a variable is marked as secret, flowey ensures:

The value is not logged or printed in step output
CI backends (ADO, GitHub Actions) are instructed to mask the value in their logs
Secret status is automatically propagated to prevent leaks

Automatic Secret Propagation

To prevent accidental leaks, flowey uses conservative automatic secret propagation:

admonish warning

If a step reads a secret value, **all subsequent writes from that step are automatically marked as secret** by default. This prevents accidentally leaking secrets through derived values.

For example:

rust,ignore

ctx.emit_rust_step("process token", |ctx| {
    let secret_token = secret_token.claim(ctx);
    let output_var = output_var.claim(ctx);
    |rt| {
        let token = rt.read(secret_token);  // Reading a secret
        
        // This write is AUTOMATICALLY marked as secret
        // (even though we're just writing "done")
        rt.write(output_var, &"done".to_string());
        
        Ok(())
    }
});

If you need to write non-secret data after reading a secret, use write_not_secret():

rust,ignore

rt.write_not_secret(output_var, &"done".to_string());

Best Practices for Secrets

Never use ReadVar::from_static() for secrets - static values are encoded in plain text in the generated YAML
Always use write_secret() when writing sensitive data like tokens, passwords, or keys
Minimize secret lifetime - read secrets as late as possible and don't pass them through more variables than necessary

Guide Src Dev Guide Dev Tools Flowey Variables (`Guide/src/dev_guide/dev_tools/flowey/variables.md`)

Variables

Variables are flowey's mechanism for creating typed data dependencies between steps. When a node emits steps, it uses ReadVar<T> and WriteVar<T> to declare what data each step consumes and produces. This creates explicit edges in the dependency graph: if step B reads from a variable that step A writes to, flowey ensures step A executes before step B.

Claiming Variables

Before a step can use a ReadVar or WriteVar, it must claim it. Claiming serves several purposes:

Registers that this step depends on (or produces) this variable
Converts ReadVar<T, VarNotClaimed> to ReadVar<T, VarClaimed>
Allows flowey to track variable usage for graph construction

Variables can only be claimed inside step closures using the claim() method.

Nested closure pattern and related contexts:

rust,ignore

// Inside a SimpleFlowNode's process_request() method
fn process_request(&self, request: Self::Request, ctx: &mut NodeCtx<'_>) {
    // Assume a single Request provided an input ReadVar and output WriteVar
    let input_var: ReadVar<String> = /* from one of the requests */;
    let output_var: WriteVar<i32> = /* from one of the requests */;

    // Declare a step (still build-time). This adds a node to the DAG.
    ctx.emit_rust_step("compute length", |step| {
        // step : StepCtx  (outer closure, build-time)
        // Claim dependencies so the graph knows: this step READS input_var, WRITES output_var.
        let input_var = input_var.claim(step);
        let output_var = output_var.claim(step);

        // Return the runtime closure.
        move |rt| {
            // rt : RustRuntimeServices (runtime phase)
            let input = rt.read(input_var);      // consume value
            let len = input.len() as i32;
            rt.write(output_var, &len);          // fulfill promise
            Ok(())
        }
    });
}

Why the nested closure dance?

The nested closure pattern is fundamental to flowey's two-phase execution model:

Build-Time (Outer Closure): When flowey constructs the DAG, the outer closure runs to:
- Claim variables, which registers dependencies in the graph
- Determine what this step depends on (reads) and produces (writes)
- Allow flowey determine execution order
- Returns an inner closure that gets invoked during the job's runtime
Runtime (Inner Closure): When the pipeline actually executes, the inner closure runs to:
- Read actual values from claimed ReadVars
- Perform the real work (computations, running commands, etc.)
- Write actual values to claimed WriteVars

NodeCtx: Used when emitting steps (during the build-time phase). Provides emit_* methods, new_var(), req(), etc.
StepCtx: Used inside step closures (during runtime execution). Provides access to claim() for variables, and basic environment info (backend(), platform()).

The type system enforces this separation: claim() requires StepCtx (only available in the outer closure), while read()/write() require RustRuntimeServices (only available in the inner closure).

ClaimedReadVar and ClaimedWriteVar

These are type aliases for claimed variables:

ClaimedReadVar<T> = ReadVar<T, VarClaimed>
ClaimedWriteVar<T> = WriteVar<T, VarClaimed>

Only claimed variables can be read/written at runtime.

Implementation Detail: Zero-Sized Types (ZSTs)

The claim state markers VarClaimed and VarNotClaimed are zero-sized types (ZSTs) - they exist purely at the type level. It allows Rust to statically verify that all variables used in a runtime block have been claimed by that block.

The type system ensures that claim() is the only way to convert from VarNotClaimed to VarClaimed, and this conversion can only happen within the outer closure where StepCtx is available.

Static Values vs Runtime Values

Sometimes you know a value at build-time:

rust,ignore

// Create a ReadVar with a static value
let version = ReadVar::from_static("1.2.3".to_string());

// This is encoded directly in the pipeline, not computed at runtime
// WARNING: Never use this for secrets!

This can be used as an escape hatch when you have a Request (that expects a value to be determined at runtime), but in a given instance you know the value at build-time.

Variable Operations

ReadVar provides operations for transforming and combining variables:

map(): Transform a ReadVar<T> into a ReadVar
zip(): Combine two ReadVars into ReadVar<(T, U)>
into_side_effect(): Convert ReadVar<T> to ReadVar<SideEffect> when you only care about ordering, not the value
depending_on(): Create a new ReadVar with an explicit dependency

For detailed examples, see the ReadVar documentation.

Guide Src Dev Guide Tests Fuzzing Running (`Guide/src/dev_guide/tests/fuzzing/running.md`)

Running Fuzzers Locally

Installing Dependencies

To begin fuzzing in OpenVMM, you'll need to install cargo-fuzz.

Installation should be as simple as:

bash

cargo install cargo-fuzz

For debugging crashes, a debugger such as lldb is useful:

bash

sudo apt-get install -y lldb

For coverage reports, install llvm-tools and lcov:

bash

rustup component add llvm-tools
sudo apt-get install -y lcov

admonish info

`cargo-fuzz` requires a nightly toolchain as it compiles targets with
[ASAN](https://learn.microsoft.com/en-us/cpp/sanitizers/asan?view=msvc-170) to
improve the likelihood of finding bugs and the reproducibility of testcases.

admonish warning

Fuzzing only works on **Linux**. libfuzzer-sys doesn't support Windows.

On **aarch64**, you must set `RUSTFLAGS="-Ctarget-feature=+lse,+neon"` before
any cargo-fuzz command (build, run, coverage), or builds will fail with
atomics errors. This is not needed on x86_64.

Running

While it's entirely possible to run the various fuzzers in the OpenVMM repo using
cargo fuzz directly, the OpenVMM repo includes additional tooling to streamline
working with fuzzers at "OpenVMM scale": cargo xtask fuzz

cargo xtask fuzz bridges the gap between cargo fuzz's "crate-oriented"
tooling, and OpenVMM's "repo-oriented" tooling.

e.g: instead of manually navigating to each individual crate/fuzz directory in
order to use cargo fuzz, with cargo xtask fuzz, you can list/run/build any
fuzzer in the OpenVMM repo, regardless where it happens to be in the repo!

Before you can run a fuzzer, you need to know its name. To see a list of all
fuzzers currently in the OpenVMM tree, you can run:

bash

cargo xtask fuzz list

The output will be a list of available "fuzz targets":

bash

$ cargo xtask fuzz list
fuzz_chipset_battery
fuzz_ide
fuzz_scsi_buffers

Once you've got a fuzzer you're interested in running (e.g: fuzz_ide),
starting a fuzzing session is as easy as running:

bash

cargo xtask fuzz run fuzz_ide

And you're off! If you see a whole bunch of terminal spew, congrats, you're
fuzzing!

When run locally using the above command, the fuzzer will run indefinitely until
a crash is discovered.

Building without running

To just build a fuzzer without starting it:

bash

cargo xtask fuzz build fuzz_ide

The binary lands at target//release/fuzz_ide.

Reproducing a crash

When LibFuzzer finds a crash, it saves the input as an artifact. Reproduce it:

bash

# Through xtask (sets XTASK_FUZZ_REPRO=1 automatically for tracing)
cargo xtask fuzz run fuzz_ide path/to/crash-artifact

# Or run the binary directly (faster for iteration)
./target/<triple>/release/fuzz_ide path/to/crash-artifact

# With tracing enabled (verbose — shows device state at each poll)
XTASK_FUZZ_REPRO=1 ./target/<triple>/release/fuzz_ide path/to/crash-artifact

Minimizing crash inputs

bash

cargo xtask fuzz tmin fuzz_ide path/to/crash-artifact

Corpus management

Corpus files live in <crate>/fuzz/corpus/<target>/.
Crash artifacts land in <crate>/fuzz/artifacts/<target>/.

To minimize the corpus (remove redundant inputs):

bash

cargo xtask fuzz cmin fuzz_ide

Parallel fuzzing

Use -fork=N to run N fuzzer processes in parallel across CPUs:

bash

cargo xtask fuzz run fuzz_ide -- -- -fork=20 -max_total_time=3600 -print_final_stats=1

To survive crashes/timeouts/OOMs during long campaigns:

bash

cargo xtask fuzz run fuzz_ide -- -- \
  -fork=20 -max_total_time=21600 \
  -ignore_crashes=1 -ignore_timeouts=1 -ignore_ooms=1 \
  -print_final_stats=1

If you need to tweak the runtime behavior of the command, all of libFuzzer's
commandline options are at your disposal. Alternatively you can
print the help of the fuzzer like so:

bash

# NOTE: The "-- --" is required to differentiate between `xtask fuzz`'s
# extra-args, and `cargo fuzz`'s extra-args
cargo xtask fuzz run fuzz_ide -- -- -help=1

Other Fuzzing Commands

The cargo xtask fuzz CLI includes plenty of docs via --help text. Don't be
afraid to dig into all the tools available via cargo xtask fuzz by using
--help at both the top-level, and for more details regarding the various
subcommands.

Note that most cargo xtask fuzz commands mirror those from cargo fuzz, so
for additional information on how certain commands work, check out the
cargo-fuzz book.

Coverage

The effectiveness of fuzzing can be measured with code coverage.

Code coverage can be analyzed to determine which branches in the target were
exercised and which were missed by the fuzzer. This can be used to determine if
the fuzzer needs improvements or is doing an adequate job.

Before you begin you'll need some additional dependencies to generate an html
report:

bash

rustup component add llvm-tools
apt install lcov

To generate a report with "sane defaults", you can simply run:

bash

cargo xtask fuzz coverage fuzz_ide --with-html-report

Simply navigate to the html/report/dir/index.html on your machine and inspect the coverage!

admonish tip

The overall coverage percentage in the HTML report covers *all* compiled code
including dependencies — it's typically 4–6% and meaningless. Focus on coverage
of the **target crate itself** (e.g., `ide/src/lib.rs`, `storvsp/src/lib.rs`).

admonish note

`--with-html-report` offers a quick-and-easy way for an individual user
generate a coverage report locally, but it may not be entirely appropriate for
more "industrial scale" fuzzing pipelines.

Manual Coverage Generation (Advanced)

The basic way this is done is by running all the discovered input testcases
through the fuzzer and merging all the coverage events together (remember, the
fuzzers only save testcases which generate new coverage). Cargo-fuzz provides a
way to do this with the coverage subcommand. This step generates a
coverage.profdata file which can be turned into a human-readable HTML report:

bash

# cargo xtask fuzz coverage <fuzzer name>
cargo xtask fuzz coverage fuzz_ide
# confirm coverage.profdata was created
ls -l coverage.profdata

OR if you have a large number of inputs (5k+) the below will collect and merge
coverage significantly faster:

bash

# rebuild the fuzzer with coverage instrumentation
RUSTFLAGS="-C instrument-coverage" cargo fuzz build
# set env var to rustup's llvm-tools
LLVM_TOOLS_PATH=$(dirname $(find $(rustc --print sysroot) -name 'llvm-profdata'))
# make an output directory for corpus minimation
mkdir min_corp
# run the minimizer putting the raw cov data into coverage.profraw
LLVM_PROFILE_FILE="coverage.profraw" ./fuzz/targets/<target-path>/release/fuzz_ide min_corp <path to input corpus directory> -merge=1
# merge the raw data into coverage.profdata
$LLVM_TOOLS_PATH/llvm-profdata merge -sparse coverage.profraw -o coverage.profdata

Next find the location of the llvm-tools you installed with rustup
(NOTE: rustup is used to install the LLVM tools to ensure that rust's llvm
version and the tool version are in sync), and convert the coverage data into
a report:

bash

# set env var to rustup's llvm-tools
LLVM_TOOLS_PATH=$(dirname $(find $(rustc --print sysroot) -name 'llvm-profdata'))
# convert the coverage data into an lcov format
$LLVM_TOOLS_PATH/llvm-cov export -instr-profile=coverage.profdata \
    -format=lcov \
    -object ./fuzz/targets/<target-triple>/coverage/<target-triple>/release/fuzz_ide \
    --ignore-filename-regex "rustc" > coverage.lcov
# summarize the coverage information
lcov --summary ./coverage.lcov
# make an output directory for the html report
mkdir -p lcov_html
# generate the html report
genhtml -o lcov_html --legend --highlight ./coverage.lcov

Guide Src Dev Guide Tests Fuzzing Writing (`Guide/src/dev_guide/tests/fuzzing/writing.md`)

Writing Fuzzers

Writing a new fuzzer in OpenVMM

The easiest way to get up and running is to look at the existing in-tree fuzzers
(which you can list using cargo xtask fuzz list), along with reading through
the cargo-fuzz book (the
book is fairly brief and shouldn't take more than 20 minutes to read through)

Some examples of in-tree fuzzers:

Simple device example: chipset/fuzz/battery.rs
More complex device example: ide/fuzz/fuzz_ide.rs
Abstraction over unsafe example: ide/fuzz/fuzz_scsi_buffer.rs

Once you're ready to take a stab at writing your own fuzzer, spinning up a new
fuzzer is as easy as running:

bash

cargo xtask fuzz init openvmm_crate_to_fuzz TEMPLATE

Use --help for more details on the available TEMPLATE types.

admonish caution

We don't suggest using `cargo fuzz init` (i.e: without `xtask`), as it
emits a template that isn't compatible with the OpenVMM repo style, and also
doesn't properly update the root Cargo.toml's `workspace.members` array.

Fuzzing an abstraction over unsafe code

Unsafe code is a prioritized target for fuzzing given its self-evident risks.

However, it can be hard to reason about how best to exercise unsafe code in OpenVMM
via fuzzing as often (and correctly) the unsafe code is not directly interacting
with guest-controlled data.

The approach taken with OpenVMM then is to target abstractions over unsafe code.
Such as interfaces and data structures like BounceBuffers, guest_memory, or
ucs2. A fuzzer for one of these will attempt to be a regular consumer of the
abstraction, calling those APIs declared safe and using any data structure in a
rust-safe way. This attempts to check that the safety guarantees the abstraction
is making are being upheld via the API.

For example, let's say we want to fuzz BounceBuffers, here's what we may
want to fuzz:

Fuzz logic might then allocate a BounceBuffer using new and call methods on
it such as as_mut_bytes and io_vecs. Then it could access the return result
of both those calls:

rust,ignore

use scsi_buffers::BounceBuffer;

#[derive(Arbitrary)]
enum BouneBufferAccess {
    AsMutBytes,
    IoVecs
}

#[derive(Arbitrary)]
struct FuzzCase {
    #[arbitrary(with = |u: &mut Unstructured| u.int_in_range(0..=0x40000))]
    size: usize,
    accesses: Vec<BounceBufferAccess>
}

fn access_mut_bytes(buf: &mut [u8]) {
    buf.fill(b'A');
    // access buf in other ways to test validity of underlying memory and slice
}

fn access_io_vecs(io_vecs: &[IoBuffer]) {
    // access each IoBuffer to test validity of ptr and len
}

fuzz_target!(|fuzz_case: FuzzCase| { do_fuzz(fuzz_case) });

fn do_fuzz(fuzz_case: FuzzCase) {
    let mut bb = BounceBuffer::new(fuzz_case.size);

    for access in fuzz_case.accesses {
        match access {
            AsMutBytes => {
                let buf = bb.as_mut_bytes();
                access_mut_bytes(buf);
            },
            IoVecs => {
                let io_vecs = bb.io_vecs();
                access_io_vecs(io_vecs)
            }
        }
    }
}

The fuzzer should work to ensure the safe members of the API cannot be misused
in any way that may result in memory corruption or unsoundness.

Fuzzing a chipset device

Writing a fuzzer for a chipset device (e.g: battery, ide, serial, pic,
etc...) involves targeting the API that is roughly exposed to guests: the
device's port IO, PCI config, and MMIO interfaces.

While it's entirely possible to hand-roll a fuzzer that is tailored to the
specific register configuration of a particular device, the in-repo
chipset_device_fuzz crate exports a FuzzChipset type that offers a
"plug-and-play" way to hook a chipset device up to a fuzzer:

text

/* Detailed source-code truncated for AI context efficiency. */

Fuzzing a vmbus device

VMBus device fuzzers exercise the guest-facing VMBus channel protocol. Unlike
chipset device fuzzers (which use FuzzChipset for PIO/MMIO dispatch), VMBus
fuzzers create a pair of connected async channels and send packets directly.

See storvsp/fuzz/fuzz_storvsp.rs for a complete example.

The general pattern:

Create connected channels using vmbus_channel::connected_async_channels
Set up the device (controller, disks, etc.) and start a TestWorker
Negotiate the protocol — most VMBus devices reject all packets until
initialization completes, so always negotiate upfront
Loop over arbitrary actions — send SCSI commands, reset packets, raw
packets, and read completions
Race teardown against the fuzz loop using futures::select!

rust,ignore

use futures::{select, FutureExt};
use std::pin::pin;

fn do_fuzz(u: &mut Unstructured<'_>) -> Result<(), anyhow::Error> {
    DefaultPool::run_with(|driver| async move {
        let (host, guest_channel) = connected_async_channels(4 * 1024);
        let guest_queue = Queue::new(guest_channel)?;

        // Set up the device under test
        let controller = ScsiController::new();
        // ... attach disks ...

        let test_worker = TestWorker::start(controller, driver.clone(), mem, host, None);
        let mut guest = TestGuest { queue: guest_queue, transaction_id: 0 };

        // Always negotiate — without init, the device rejects everything
        guest.perform_protocol_negotiation().await;

        // Fuzz loop
        let mut fuzz_loop = pin!(async {
            while !u.is_empty() {
                let action = u.arbitrary::<FuzzAction>()?;
                // ... execute action ...
            }
            Ok::<(), anyhow::Error>(())
        }.fuse());
        let mut teardown = pin!(test_worker.teardown_ignore().fuse());

        select! {
            _ = fuzz_loop => {},
            _ = teardown => {},
        }
        Ok(())
    })
}

Fuzzing `async` code

Depending on the nature of the 'async' code in question, there are two main
recommended approaches to fuzzing it:

now_or_never: The recommended approach for individual asynchronous calls
is to use the now_or_never method from the futures crate. This method
will poll the future to completion, but will not block if the future is
not ready to complete. This allows you to fuzz the future without needing
to run it to completion, which can be useful for testing the behavior of
the future in various states.
DefaultPool::run_with: The recommended approach for more intricate
asynchronous requirements is to use the DefaultPool::run_with method from
our pal_async crate. This method takes a custom async function and runs it
to completion. This allows you to write custom code using regular async/await
syntax, combinators, joins, selects, or whatever you wish.

Guide Src Dev Guide Tests Vmm Azure Images (`Guide/src/dev_guide/tests/vmm/azure_images.md`)

Azure-hosted Test Images

OpenVMM utilizes pre-made VHDs in order to run tests with multiple guest
operating systems. These images are as close to a "stock" installation as
possible, created from the Azure Marketplace or downloaded directly from a
trusted upstream source.

These VHDs are stored in Azure Blob Storage, and are downloaded when running VMM
tests in CI.

Lazy Fetching (Remote Artifacts)

When running tests with the OpenVMM backend, disk artifacts that are hosted on
Azure Blob Storage can be fetched lazily over HTTP instead of being
pre-downloaded. The VMM reads disk sectors on demand via HTTP Range requests,
with a local SQLite cache so that sectors are only fetched once.

This is controlled by two environment variables:

PETRI_REMOTE_ARTIFACTS — Set to 0 or false to force all artifacts to
be resolved locally, disabling lazy fetching. By default, remote access is
allowed for artifacts that opt in.
PETRI_CACHE_DIR — Override the directory used for the SQLite read cache.
Defaults to a platform-appropriate cache directory (e.g.
~/.cache/petri on Linux, ~/Library/Caches/petri on macOS,
%LOCALAPPDATA%\petri\cache on Windows).

admonish note

VHDX artifacts (e.g. `WINDOWS_11_ENTERPRISE_AARCH64`) do **not** support lazy
fetching and must be downloaded locally. The Hyper-V backend also requires all
artifacts to be local.

Downloading VHDs

The cargo xtask guest-test download-image command can be used to download vhds
to your machine.

By default it will download all available VHDs, however the --vhd option can
be used to only download select guests. After running it the tests can be run
just like any other. This command requires having
AzCopy
installed.

Guide Src Reference Architecture Devices Storage (`Guide/src/reference/architecture/devices/storage.md`)

Storage pipeline

The storage stack carries guest I/O requests from a guest-visible
controller to a backing store and back. It's shared between OpenVMM
and OpenHCL. Every disk backend implements the
DiskIo
trait, and frontends hold a
Disk
wrapper — a cheap, cloneable handle to any backend. For the DiskIo
trait surface, method contracts, and error model, see the
disk_backend rustdoc.

The pipeline

Every storage I/O flows through the same layered pipeline:

text

/* Detailed source-code truncated for AI context efficiency. */

Key vocabulary:

Frontend. Speaks a guest-visible storage protocol and translates
requests into DiskIo calls.
SCSI adapter. For the SCSI and IDE paths, an intermediate layer
(SimpleScsiDisk
or
SimpleScsiDvd)
that parses SCSI CDB opcodes before calling DiskIo.
Backend. A DiskIo implementation that reads and writes to a
specific backing store.
Decorator. A DiskIo implementation that wraps another Disk
and transforms I/O in transit (encryption, delay, persistent
reservations).
Layered disk. A DiskIo implementation composed of ordered
layers with per-sector presence tracking.

Frontends

Three frontends exist. Each speaks a different guest-visible protocol
but they all produce DiskIo calls on the backend side.

Frontend	Protocol	Transport	Crate
NVMe	NVMe 2.0	PCI MMIO + MSI-X	`nvme`
StorVSP	SCSI CDB over VMBus	VMBus ring buffers	`storvsp`
IDE	ATA / ATAPI	PCI/ISA I/O ports + DMA	`ide`

NVMe is the simplest path. The NVMe controller's namespace directly holds a Disk. NVM opcodes (READ, WRITE, FLUSH, DSM) map nearly 1:1 to DiskIo methods. The FUA bit from the NVMe write command is forwarded directly.

StorVSP / SCSI has a two-layer design. StorVSP handles the VMBus
transport — negotiation, ring buffer management,
sub-channel allocation. It
dispatches each SCSI request to an
AsyncScsiDisk
implementation. For hard drives, that's
SimpleScsiDisk,
which parses the SCSI CDB and translates it to DiskIo calls. For
optical drives, it's
SimpleScsiDvd.

IDE is the legacy path. ATA commands for hard drives call DiskIo directly. ATAPI commands for optical drives delegate to SimpleScsiDvd through an ATAPI-to-SCSI translation layer — the same DVD implementation that StorVSP uses. IDE also supports enlightened INT13 commands, a Microsoft-specific optimization that collapses the multi-exit register-programming sequence into a single VM exit.

Backends

A backend is a DiskIo implementation that reads and writes to a specific backing store. Backends are interchangeable — swap one for another without changing the frontend. The frontend holds a Disk and doesn't know what's behind it. See the storage backends page for the full catalog and platform details.

Decorators

A decorator is a DiskIo implementation that wraps another Disk and transforms I/O in transit. Features compose by stacking decorators without modifying backends:

text

CryptDisk
    └── BlockDeviceDisk

Three decorators exist: CryptDisk (XTS-AES-256 encryption), DelayDisk (injected latency), and DiskWithReservations (in-memory persistent reservation emulation). All three forward metadata (sector count, sector size, disk ID, wait_resize) to the inner disk unchanged. See the storage backends page for the decorator catalog.

The layered disk model

A LayeredDisk is a DiskIo implementation composed of multiple layers, ordered from top to bottom. Each layer is a block device with per-sector presence tracking. This model powers diff disks, RAM overlays, and caching.

Reads fall through

When a read arrives, the layered disk checks layers top-to-bottom. The first layer that has the requested sectors provides the data. Sectors not present in any layer are zeroed.

Writes go to the top

Writes always go to the topmost layer. If that layer is configured with write-through, the write also propagates to the next layer.

Read caching

A layer can be configured to cache read misses: when sectors are fetched from a lower layer, they're written back to the cache layer. This uses a write_no_overwrite operation to avoid overwriting sectors that were written between the read and the cache population.

Layer implementations

Two concrete layers exist today:

RamDiskLayer (disklayer_ram) — ephemeral, in-memory. Data is stored in a BTreeMap keyed by sector number. Fast, but lost when the VM stops.
SqliteDiskLayer (disklayer_sqlite) — persistent, backed by a SQLite database (.dbhd file). Designed for dev/test scenarios — no stability guarantees on the on-disk format.

A full Disk can appear at the bottom of the stack as a fully-present layer (DiskAsLayer). This is the typical case: a RAM or sqlite layer on top of a file or block device.

Worked example: `memdiff:file:disk.vhdx`

text

Layer 0: RamDiskLayer (empty, writable)
  Layer 1: DiskAsLayer wrapping FileDisk (fully present, read-only
           from the layered disk's perspective)

Guest write → sector goes to the RAM layer.
Guest read → check RAM; if the sector is present, return it. If absent, fall through to the file.
Sectors absent from both layers → zero-filled.

Changes are ephemeral — they live in the RAM layer and are lost when the VM stops. The Running OpenVMM page shows concrete memdiff: examples.

How configuration becomes a concrete stack

The resource resolver connects configuration (CLI flags, VTL2 settings) to concrete backends. A resource handle describes what backend to use; a resolver creates it.

The storage resolver chain is recursive. An NVMe controller resolves each namespace's disk, which may be a layered disk, which resolves each layer in parallel, which may itself be a disk that needs resolving.

Example: --vmbus-scsi id=scsi0 --disk memdiff:file:path/to/disk.vhdx,on=scsi0

CLI parses this into a LayeredDiskHandle with two layers:
- Layer 0: RamDiskLayerHandle { len: None, sector_size: None } (RAM diff, inherits size and sector size from backing disk)
- Layer 1: DiskLayerHandle(FileDiskHandle(...)) (the file)
The layered disk resolver resolves both layers in parallel.
The RAM layer attaches on top of the file layer, inheriting its sector size and capacity.
The resulting LayeredDisk is wrapped in a Disk and handed to the NVMe namespace or SCSI controller.

For the OpenHCL settings model (StorageController, Lun, PhysicalDevice), see Storage Translation and Storage Configuration Model.

Backend catalog

Backend	Crate	Wraps	Platform	Note
FileDisk	`disk_file`	Host file	Cross-platform	Simplest backend
Vhd1Disk	`disk_vhd1`	VHD1 fixed file	Cross-platform	Parses VHD footer
VhdmpDisk	`disk_vhdmp`	Windows vhdmp driver	Windows	Dynamic/differencing VHD/VHDX
VhdxDisk	`vhdx`	VHDX file	Cross-platform	Pure-Rust VHDX parser
BlobDisk	`disk_blob`	HTTP / Azure Blob	Cross-platform	Read-only, HTTP range requests
BlockDeviceDisk	`disk_blockdevice`	Linux block device	Linux	io_uring, resize via uevent, PR passthrough
NvmeDisk	`disk_nvme`	Physical NVMe (VFIO)	Linux/Windows	User-mode NVMe driver, resize via AEN
StripedDisk	`disk_striped`	Multiple Disks	Cross-platform	Data striping

Online disk resize

Disk resize is a cross-cutting concern that spans backends and frontends.

Backend detection

Only two backends detect capacity changes at runtime:

BlockDeviceDisk — listens for Linux uevent notifications on the block device. When the host resizes the device, a uevent fires, the backend re-queries the size via ioctl, and wait_resize completes.
NvmeDisk — the user-mode NVMe driver monitors Async Event Notifications (AEN) from the physical controller and rescans namespace capacity.

All other backends default to never signaling (wait_resize returns pending()). Decorators and layered disks delegate wait_resize to the inner backend.

admonish warning

`FileDisk` never signals resize. If you attach a file backend and resize the file at runtime, nothing happens — the guest won't be notified. Use `BlockDeviceDisk` or `NvmeDisk` for runtime resize.

Frontend notification

Once a backend detects a resize, the frontend notifies the guest:

Frontend	Mechanism	How it works
NVMe	Async Event Notification	Background task per namespace calls `wait_resize`. On change, completes a queued AER command with a changed-namespace-list log page. Guest re-identifies the namespace.
StorVSP / SCSI	UNIT_ATTENTION	On the next SCSI command after a resize, `SimpleScsiDisk` detects the capacity change and returns CHECK_CONDITION with UNIT_ATTENTION / CAPACITY_DATA_CHANGED. Guest retries and re-reads capacity.
IDE	Not supported	IDE has no capacity-change notification mechanism.

The resize path is the same in OpenHCL and standalone — BlockDeviceDisk detects the uevent from the host, wait_resize completes, and the frontend notifies the guest through the standard mechanism. No special paravisor-level interception.

Virtual optical / DVD

DVD and CD-ROM drives use a different model from disk devices.

SimpleScsiDvd implements AsyncScsiDisk and manages media state: a disk can be Loaded or Unloaded. Optical media always uses a 2048-byte sector size. The implementation handles optical-specific SCSI commands: GET_EVENT_STATUS_NOTIFICATION, GET_CONFIGURATION, START_STOP_UNIT (eject), and media change events.

Eject

Two eject paths exist:

Guest-initiated (SCSI START_STOP_UNIT with the load/eject flag): the DVD handler checks the prevent flag, replaces media with Unloaded, and calls disk.eject(). Once ejected via SCSI, the media is permanently removed for the VM lifetime.
Host-initiated (change_media via the resolver's background task): can insert new media or remove existing media dynamically.

Frontend support

Frontend	DVD support	How
StorVSP / SCSI	Yes	`SimpleScsiDvd` implements `AsyncScsiDisk` directly.
IDE	Yes	ATAPI wraps `SimpleScsiDvd` through the ATAPI-to-SCSI layer.
NVMe	No	NVMe has no removable media concept. Explicitly rejected.

CLI

--disk file:my.iso,dvd,on=scsi0 → SCSI optical drive.
--ide file:my.iso,dvd → IDE optical drive (ATAPI).

The dvd flag implicitly sets read_only = true.

`mem:` and `memdiff:` CLI mapping

Both CLI options map to the layered disk model:

mem:1G creates a single-layer LayeredDisk with a RamDiskLayer sized to 1 GB. No backing disk — the RAM layer is the entire disk.
memdiff:file:disk.vhdx creates a two-layer LayeredDisk: a RamDiskLayer (inheriting size from the backing disk) on top of the file. Writes go to the RAM layer; reads fall through to the file for sectors not yet written.

Both use RamDiskLayerHandle under the hood. The difference is len: Some(size) for mem: (standalone RAM disk with explicit size) vs. len: None for memdiff: (inherits from backing disk). The optional sector_size field (default None) lets you override the sector size; when None, it inherits from the lower layer or defaults to 512 bytes. The Running OpenVMM page shows concrete examples.

Controller identity and Azure disk classification

In Azure, which controller a disk sits on is a de facto compatibility boundary. Azure VMs present four SCSI controllers (this may change), each with a distinct instance ID. One controller carries the OS disk, resource (temporary) disk, and related infrastructure disks; a separate controller carries remote data disks. For Gen1 VMs, the IDE controllers logically replace that first SCSI controller, while data disks remain on SCSI.

Guest agents use controller identity to classify disks. The azure-vm-utils udev rules match on SCSI controller instance IDs to create stable symlinks under /dev/disk/azure/. Moving a disk from one StorVSP controller instance to another changes its classification and can break guest-side automation. For SCSI disk mapping details, see the Azure disk mapping docs.

For NVMe, the mapping uses namespace IDs: NSID 1 is the OS disk, NSID 2+ are data disks (portal LUN = NSID − 2). On newer VM sizes (v7+), disks are split across multiple NVMe controllers by caching policy. NVMe is Gen2-only. See the NVMe overview and NVMe disk identification FAQ for the full Azure perspective.

Implementation map

Component	Why read it	Source	Rustdoc
`disk_backend`	`DiskIo` trait, `Disk` wrapper, error model	source	rustdoc
`disk_layered`	Layered disk, `LayerIo` trait, bitmap tracking	source	rustdoc
`nvme`	NVMe controller emulator	source	rustdoc
`storvsp`	VMBus SCSI controller	source	rustdoc
`scsidisk`	SCSI CDB parser (`SimpleScsiDisk`, `SimpleScsiDvd`)	source	rustdoc
`ide`	IDE controller emulator	source	rustdoc
`scsi_core`	`AsyncScsiDisk` trait, `Request`, `ScsiResult`	source	rustdoc

Guide Src Reference Architecture Openhcl Ak Cert (`Guide/src/reference/architecture/openhcl/ak_cert.md`)

AK Certificate Renewal

This page describes how OpenHCL-managed virtual TPM attestation-key (AK)
certificates are requested, retained, and replaced.

AK certification is separate from VMGS key release. See
Attestation and VMGS Protection for the surrounding flows.
The IGVM Attestation Protocol defines the exact
metadata encoding and response contract.

Request lifecycle

When OpenHCL is configured to manage AK certificate renewal and the TPM permits
it, the TPM device requests a certificate at initialization and can request
renewal later. Only one certificate request is outstanding at a time.

For each request:

Read the current TPM_NV_INDEX_AIK_CERT contents, using the index's declared
size rather than a fixed legacy-size buffer.
Decode the leading DER certificate and extract renewal metadata. Trailing
NV padding or bytes left by an earlier, longer certificate are not part of
that certificate.
Include successfully decoded metadata in the runtime claims before hashing
them and obtaining the attestation report. Record whether metadata was
included in this particular request.
Send the request through the host IGVM agent and process its response
asynchronously.

An absent or uninitialized NV index supplies no metadata. Read or decode
failures are logged and omit the metadata, allowing normal issuance to proceed.
The decoder requires issuer and subject common names; the authority key
identifier is optional.

If including metadata would exceed GET's 4 KiB request limit, OpenHCL omits it
before hashing the claims and generating the attestation report. This is
unlikely with ordinary production certificates. A metadata-free request that
still exceeds the transport limit fails rather than being sent.

Metadata is read for each request, not cached separately. Certificate
replacement, TPM key refresh, and state restoration therefore do not require
maintaining a second copy of certificate metadata.

Renewal outcomes

A nonempty certificate response follows the existing NV replacement path.
OpenHCL writes the returned certificate to the certificate index and advances
the renewal timer only after the write succeeds. NV write failures are logged
and emit a certificate-renewal-failed event without advancing the timer, so the
next eligible guest NV read can retry. NV writes are not atomic: a failure
after some chunks have been written can leave a partially replaced certificate.

A valid no-renewal response is accepted only if this request included
successfully decoded existing-certificate metadata. OpenHCL keeps the NV
contents without a write and advances the renewal timer. Without that
request-specific metadata, the same response is logged as a renewal failure
and does not advance the timer.

An empty transport response or an empty certificate payload indicates service
unavailability, not approval of the existing certificate. OpenHCL logs it,
leaves NV unchanged, and advances the timer to avoid retrying on every read.
Transport or response-parsing errors log a renewal failure without advancing
the timer.

Subsequent renewal is driven by guest NV reads and the existing 24-hour
renewal interval, rather than by an expiry-time alarm. The interval uses
OpenHCL system time, which can diverge from wall-clock elapsed time across
suspension and resumption.

Trust and guest compatibility

Certificate metadata is a renewal hint. Decoding it does not validate the
certificate signature, chain, validity period, or relationship to the TPM AK.
The no-renewal signal is a host response, not a signed certificate-validation
result.

Guest-requested attestation reports remain separate from renewal requests.
They retain their V1 layout and do not read or include certificate renewal
metadata, preserving the existing guest-visible NV format.

Implementation references

tpm_device owns NV access, pending requests, and renewal
scheduling.
tpm_resources defines the request helper interface
and typed certificate/no-renewal outcomes.
openhcl/underhill_core/src/emuplat/tpm.rs connects the TPM helper to
metadata parsing, report generation, and GET.
underhill_attestation extracts certificate metadata
and builds and parses attestation messages.

Guide Src Reference Architecture Openhcl Attestation (`Guide/src/reference/architecture/openhcl/attestation.md`)

Attestation and VMGS Protection

This page introduces OpenHCL's attestation flows and directs you to their
lifecycle and protocol references.

Attestation flows

OpenHCL uses attestation for two distinct purposes:

VM Guest State (VMGS) protection: secure key release obtains key material
used to recover access to encrypted VMGS. Hardware sealing provides local
key protection where supported and enabled by the VM configuration.
TPM attestation-key (AK) certification: the virtual TPM requests a
certificate for its AK and can later request renewal. Issuing or retaining
this certificate is separate from releasing keys to unlock VMGS.

The host IGVM agent is reached through the Guest Emulation Transport (GET).
For hardware-attested requests, OpenHCL obtains a TEE report that binds a hash
of the serialized runtime claims. Binding the claims does not by itself
authenticate host-supplied metadata or establish certificate trust.

Reference pages

AK Certificate Renewal owns certificate storage, renewal
decisions, scheduling, and the guest-report boundary.
IGVM Attestation Protocol owns the AK request
claims, response framing, signal definitions, and compatibility rules.

Implementation references

underhill_attestation prepares attestation requests,
parses responses, and implements VMGS key protection.
tee_call provides access to hardware attestation and
supported key derivation.

Boot and key protection

Depending on the VM's configuration, platform-security initialization obtains
key material through secure key release (SKR), hardware unsealing, and guest
state protection services. Hardware sealing can provide either the sole means
of recovering the VMGS datastore key (DEK) or a recovery path alongside other
key protectors.

The hardware protector contains the sealed DEK and the security version numbers
(SVNs) needed to reproduce its hardware-derived key. It is stored in
HW_KEY_PROTECTOR without VMGS-level encryption so it can be read before VMGS
is unlocked. Its header, including SVN metadata, and encrypted DEK are covered
by a hardware-keyed HMAC.

Boot unsealing uses the protector's recorded SVN, which can be lower than the
current hardware SVN. Recovery requires hardware support for that derivation
and successful protector authentication. This differs from the minimum TCB
policy used for runtime resealing below.

Ordinary stateless mode bypasses VMGS encryption when hardware sealing is not
requested. Stateless mode with the HardwareSealing encryption policy instead
requires hardware sealing: it seals a new DEK on provisioning and unseals then
rotates it on later boots. Unsupported required sealing or failure to derive,
unseal, write, or finalize the required protector prevents boot from completing.

The main implementation is in
underhill_attestation,
with local hardware access provided by
tee_call.

Resiliency

Recovery after live migration

A hardware-derived sealing key may change when a VM migrates to another host.
OpenHCL retains the active DEK in protected memory, allowing it to create a
destination-compatible protector without recovering the DEK from the source
protector.

The GET NOTIFY_POST_LIVE_MIGRATION notification triggers this recovery:

Obtain a local report and check it against the runtime TCB floor.
Derive destination hardware keys and seal the unchanged active DEK.
Verify the candidate protector, then write and flush it through the VMGS
broker. A defensive active-key check prevents publishing a stale DEK if
concurrent key rotation is introduced.
Verify again after persistence to detect hardware or SVN changes during the
operation.

Recovery is enabled only when the successful boot-unlock attempt has sealed,
written, and flushed a hardware protector for the active DEK and established a
trusted runtime floor under a supported policy. This eligibility comes from
trusted operation results, not the presence of an entry on disk. An LM event
maintains an established recovery path; it does not perform first-time sealing.
Once enabled, recovery can repair a subsequently missing or corrupt protector.
It neither changes the policy nor rotates the DEK. Hardware calls run off the
VP and GET executors. GET coalesces notifications received before callback
registration into one pending event and delivers it when registration completes.
The worker also latches events received before startup or during recovery.
Failed attempts retry with bounded backoff. There is no periodic verification;
after success, the worker waits for another event. Restoring serialized resealer
state also schedules a durable rewrite, as described below.

Runtime TCB floor

The trusted computing base (TCB) floor prevents runtime resealing at a lower
accepted security version. Boot establishes it from local reports already
obtained during platform-security initialization, including reports obtained
before failed SKR callouts. It is held in protected OpenHCL memory, not loaded
from host-controlled VMGS metadata.

Compatible higher observations advance the floor. Retries, persistence failures,
reset, and normal stop/start cannot lower it. If boot cannot establish a usable
floor, runtime resealing stays disabled with a warning; existing boot recovery
behavior is unchanged.

If optional boot sealing does not succeed, no floor is exported to the runtime
worker. When hardware sealing is only a backup, finalization failure disables
runtime recovery without failing boot. When sealing is required, or boot used
hardware unsealing to recover the DEK, failure to confirm the active sealed key
or flush completed writes fails boot. The full unlock/key-rotation sequence is
not automatically retried after this failure because it may already have changed
the active DEK. Existing protector write/unlock errors retain their handling.

The comparison is a partial order, not a packed-integer or lexicographic order:

SNP: known TCB components must be non-decreasing within the same supported
report version and CPU family/model; reserved bytes must remain equal.
For report versions 3–5, Milan/Genoa (family 19h, models 00h–1Fh) and
Turin (family 1Ah, models 90h–AFh and C0h–CFh) use distinct typed
layouts, as defined by AMD 56860 section 2.3. Report version alone does not
select the layout. Version 2 and unknown domains require exact SVN equality;
cross-domain transitions are rejected. The floor tracks the reported SVN used
for key derivation; it does not enforce the extended TCB added in report v6.
TDX: CPU SVN components must be non-decreasing. TEE SVN is parsed using
the typed TeeTcbSvn ABI layout: major SVN (byte 1) identifies the module
and must match, including legacy identity 0. Minor SVN (byte 0) and SE_SVN
(byte 2) must be non-decreasing; reserved bytes 3–15 must remain equal.
Reserved metadata changes are incompatible, not upgrades. This minimum-TCB
comparison does not determine Intel security status, which requires
authenticated Intel TCB Info.

Candidate verification also checks a fresh report. Its SVN must match the
candidate exactly, even if the hardware could still derive an older SVN's key.

CPU SVN comparison uses all 16 unsigned bytes in their original positions:
every destination byte must be at least its resident floor byte. There is no
integer conversion or lexicographic ordering; an increase in one byte cannot
compensate for a decrease in another. Intel PCS
Get TDX TCB Info V4, step 3.a, uses component-wise minima for the
16 PCK certificate TCB components;
Appendix A describes the components and comparison metadata.
OpenHCL's raw-report comparison is a local no-decrease policy, not that complete
appraisal algorithm: it neither maps raw CPU SVN bytes to PCK component
identities nor establishes cross-platform equivalence or Intel security status.

For SNP, SnpDomain retains report version and CPU family/model because the raw
TCB bytes do not identify their own layout. CPU family/model selects the layout;
version gates supported report semantics. Exact domain equality is required even
for identical raw SVNs or CPUs sharing a layout. This conservative restriction
avoids interpreting one CPU's component bytes using another CPU's meanings.
See AMD 56860, revision 1.59, section 2.3, tables 4 and 5.

Lifecycle and saved state

Memory-preserving migration retains the worker and its floor. Stopping the
worker drains an in-flight recovery attempt. Reset does not request resealing;
it preserves the floor, existing retries and backoff, and latched notifications.

The stopped worker saves a versioned protobuf state unit named
hardware_reseal, containing:

The accepted TCB floor: raw SNP TCB and report version/CPU comparison domain,
or both raw TDX SVN arrays, including identity and reserved bytes.
The serialized VM configuration used by the sealing KDF, checked against the
destination configuration rather than used to replace it.
Pending recovery, the failure count, and the latched migration notification.
Remaining retry and minimum-interval delays, bounded to 60 seconds. These are
relative durations, not process-specific clock values.

No DEK, hardware secret, derived key, or attestation report is saved by this
worker. Saving does not consume a notification or clear pending work: if another
unit fails to save, the original worker can resume with its obligations intact.

Restore constructs the worker without a floor, then installs the validated saved
floor before starting. It does not obtain a fresh report to initialize a new
floor or trust a VMGS protector as its source. Malformed or unsupported state,
an incompatible TEE or sealing configuration, or a decrease relative to an
existing resident floor fails restore without replacing that floor. A saved
worker also requires compatible hardware sealing support and encrypted VMGS.

Successful restore preserves retry backoff and merges saved notifications with
notifications received by the new worker. It always schedules a durable rewrite
when the worker starts, even when the saved worker was idle: hardware can change
while the worker is absent, including at the same SVN. Fresh reports must
satisfy the restored floor before any new protector is created. A cached
matching protector does not discharge a failed-flush obligation.

If the saved VM has no hardware_reseal unit, runtime resealing stays disabled
after reconstruction. This accommodates state from runtimes that were not
enrolled or did not support resealer state, without inventing a replacement
floor. Binaries that cannot interpret this state unit cannot restore an enrolled
worker's snapshot.

admonish warning title="trusted state transfer"

Protobuf encoding does not authenticate the saved floor. The enclosing transfer
must preserve integrity, bind the state to the VM, and prevent rollback to an
older snapshot. The resealer must not load this state from VMGS or
unauthenticated host metadata. Host-returned servicing state cannot initialize
this worker with the current transport and is rejected.

Resealer serialization does not enable isolated-VM servicing by itself. The
host-servicing isolation guard and unsupported SNP/TDX VP save/restore paths
remain in place. A trusted isolated-servicing handoff must satisfy the above
requirements before enabling that path.

Recovery limits

Cold boot establishes a new runtime lifetime. The floor is not a persistent
anti-rollback counter and does not prevent replay of an entire VMGS snapshot.
Rejecting resealing on a lower-TCB destination also cannot undo delivery of the
resident DEK by migration; migration admission is a separate security boundary.

admonish warning

Recovery is best effort. A missed notification can leave the protector stale,
and a crash before durable destination resealing can leave hardware-only VMGS
unrecoverable. Report, derivation, and persistence operations are not atomic
with migration. Durability also depends on backing storage honoring flushes.

Guide Src Reference Architecture Openhcl Attestation Protocol (`Guide/src/reference/architecture/openhcl/attestation_protocol.md`)

IGVM Attestation Protocol

This page specifies the AK certificate request claims and response contract
used by OpenHCL over the Guest Emulation Transport (GET).

admonish note title="see also"

[Attestation and VMGS Protection](attestation.md) introduces the attestation
flows. [AK Certificate Renewal](ak_cert.md) describes NV storage, renewal
scheduling, and the trust boundary.

AK request and response versions

AK certificate requests use request-data version V2 and accept V1 or V2
responses. The parser selects the layout from the received response version;
other response versions are rejected.

Guest-requested attestation reports retain the V1 request layout exposed
through TPM NV storage. They do not include the renewal metadata below.

Existing-certificate claims

An AK request with successfully decoded existing-certificate metadata includes
an optional top-level akcert object in its runtime claims. If metadata is
unavailable, the entire object is omitted, not serialized as null.
Key-release requests and guest-requested reports omit it.

Field	Encoding
`issuer`	Certificate issuer common name, not its full distinguished name.
`subject`	Certificate subject common name.
`not_before`	ASN.1 time content: `YYMMDDhhmmssZ` for UTCTime or `YYYYMMDDhhmmssZ` for GeneralizedTime.
`not_after`	The same encoding rule, selected by this field's ASN.1 time type.
`authority`	Uppercase hex of the Authority Key Identifier's `keyIdentifier` bytes; empty string if absent.

The field names use underscores exactly as shown. The time strings are not
Unix timestamps or RFC 3339 strings. The authority identifier has no prefix or
separators and does not encode the complete extension.

For example, this is an illustrative claim object, not a usable certificate:

json

{
  "akcert": {
    "issuer": "Example AK issuer",
    "subject": "Example AK",
    "not_before": "260101000000Z",
    "not_after": "270101000000Z",
    "authority": "0123456789ABCDEF"
  }
}

Claims are serialized with JSON escaping before hashing. The SHA-256 digest
occupies the first 32 bytes of the 64-byte report-data field; the remaining
bytes are zero. Adding akcert therefore changes the report binding, not just
an unmeasured transport field.

AK response framing

Header integers are 32-bit little-endian values. data_size includes the
header and any payload.

Offset	Size	Field
0	4	`data_size`
4	4	Response `version`
8	4	V2 only: `IgvmErrorInfo.error_code`
12	4	V2 only: `IgvmErrorInfo.http_status_code`
16	4	V2 only: `IgvmErrorInfo.igvm_signal`
20	12	V2 only: reserved

V1 has an 8-byte header; V2 has a 32-byte header. Certificate bytes follow
the version-specific header directly, without a JSON response envelope.
The AK response buffer is 4 KiB, including the header.

The parser requires a complete version-specific header within data_size,
rejects a declared size larger than the supplied buffer, and ignores bytes
after the declared size. A nonzero V2 error_code is an error, even when
the no-renewal signal is also set.

No-renewal signal

V2 signal bit 5 (IGVM_SIGNAL_NO_RENEW_CERT_BIT, mask 0x20) means that
the service recommends retaining the existing AK certificate described in the
request. Bit 4 remains reserved.

A successful, validly framed response with bit 5 set returns a no-renewal
outcome instead of certificate bytes. The signal takes precedence over any
payload in that response. Other signal bits alone do not indicate no-renewal.
V1 has no signal field and cannot express this outcome.

The TPM caller additionally requires that the request included successfully
decoded certificate metadata before accepting no-renewal. See
renewal outcomes for timer, NV, and failure
handling. An empty response or certificate payload is distinct from this
explicit signal.

Implementation references

openhcl_attestation_protocol::igvm_attest::get defines the
request and response structures, signal bitmap, and runtime claims.
underhill_attestation implements claim hashing,
certificate metadata extraction, and response validation.

Guide Src Reference Architecture Openhcl Boot (`Guide/src/reference/architecture/openhcl/boot.md`)

OpenHCL Boot Flow

This page describes how the host loads OpenHCL, how openhcl_boot starts Linux,
and how the OpenHCL user-mode processes begin running.

text

/* Detailed source-code truncated for AI context efficiency. */

1. IGVM Loading

The host VMM follows the directives in the OpenHCL IGVM file to populate VTL2
memory and start the boot shim. The file contains the boot shim, Linux kernel,
initial ramdisk, and other components needed by the paravisor.

The IGVM file tells the host where to load each component and which initial
state contributes to the launch measurement. The host also supplies runtime
values such as processor topology, VTL2 memory, serial configuration, and
device settings.

2. Boot Shim Execution (`openhcl_boot`)

The host transfers control to openhcl_boot, which performs these steps:

Hardware initialization: Initialize CPU state and the memory management
unit (MMU).
Configuration validation: Validate imported regions and combine measured
build-time parameters with the runtime data permitted by the image policy.
Device tree construction: Build the hardware topology and the set of
devices passed to Linux.
Sidecar setup (x86_64): Assign processors to Linux or sidecar, initialize
their control structures, and start the sidecar processors.
Kernel handoff: Start Linux with the final device tree, command line, and
architecture-specific boot data.

Measured and host-provided inputs

The measured IGVM fixes component addresses, imported regions, initrd metadata,
and static command-line options. At launch, the host contributes topology and
resource data. For a hardware-isolated VM, the shim filters that host data
according to the policy in the measured image before exposing it to Linux.

3. Linux Kernel Boot

Linux initializes memory management, scheduling, and the drivers used by the
paravisor. It mounts the initial ramdisk as its root filesystem and starts
underhill_init as PID 1. Sidecar CPUs remain in their dispatch loop until
Linux hot-plugs them.

4. Userspace Initialization (`underhill_init`)

underhill_init mounts the required pseudo-filesystems, configures the process
environment and system limits, and then replaces itself with
/bin/openvmm_hcl.

5. Paravisor Startup (`openvmm_hcl`)

openvmm_hcl reads topology and configuration from /proc/device-tree and
other kernel interfaces. It initializes host communication and VTL0 management,
then starts the underhill_vm worker.

openvmm_hcl remains the policy and control-plane process after spawning the
worker. See the dedicated openvmm_hcl page for its
resource, worker, servicing, and diagnostic responsibilities.

6. VM Execution

The underhill_vm process runs the VTL0 virtual processors, handles exits, and
coordinates device emulation. Devices such as the virtual TPM can run in
dedicated worker processes, with underhill_vm proxying their I/O and guest
memory access. openvmm_hcl remains responsible for policy, lifecycle, and
host communication.

Guide Src Reference Architecture Openhcl Cpu Scheduling (`Guide/src/reference/architecture/openhcl/cpu_scheduling.md`)

CPU Scheduling

OpenHCL runs a cooperative async executor on each VP thread. This page explains
how VP threads split time between guest execution and device work, what happens
when things block, and how the sidecar kernel changes the picture.

admonish tip

This document uses "lower VTL" and "VTL0" to refer to what are similar
things. VTLs increase in privilege as the number gets higher. VTL2 is a
higher privilege level than VTL1, which is yet higher than VTL0.

Engineers focused on IO often think about the "guest" as "VTL0", since
that's the VTL that issues IO to storage and networking devices. When
this document discusses entering VTL0, though, it's more precise to say
that control returns to any VTL that is less privileged than VTL2. It
might be VTL0 or it might be VTL1.

Scope

This page covers the VM worker process — the main OpenHCL process that runs
device emulation and VP dispatch. OpenHCL also runs other processes (see
Processes and Components), but the cooperative executor model
described here applies specifically to the worker process and its per-VP
threadpool.

For background on Rust async executors, see the Asynchronous Programming in
Rust book (especially this Under the
Hood
section).

Thread model

OpenHCL's worker process runs one thread per VP in its
threadpool.
Each thread is CPU-affinitized — thread N is pinned to Linux CPU N,
which maps 1:1 to VP index N in current configurations.

text

VP 0 thread (CPU 0)    VP 1 thread (CPU 1)    VP 2 thread (CPU 2)
  ┌──────────────────┐   ┌──────────────────┐   ┌──────────────────┐
  │ cooperative      │   │ cooperative      │   │ cooperative      │
  │ async executor   │   │ async executor   │   │ async executor   │
  │                  │   │                  │   │                  │
  │ • async workers  │   │ • async workers  │   │ • async workers  │
  │   e.g. devices   │   │   e.g. devices   │   │   e.g. devices   │
  │ • when idle:     │   │ • when idle:     │   │ • when idle:     │
  │ enter lower VTL  │   │ enter lower VTL  │   │ enter lower VTL  │
  └──────────────────┘   └──────────────────┘   └──────────────────┘

The code calls VTL0 execution the thread's "idle task" — meaning the VP thread
enters VTL0 when there is no pending VTL2 async work. The thread itself is not
idle (the physical CPU is running guest code), but the VTL2 executor has nothing
to do.

Alongside the VP threads, the worker process runs a few additional threads:

GET (Guest Emulation Transport) worker — on a dedicated thread because
it issues blocking syscalls that would stall the VP executor. When a GET
message arrives, it is processed on this dedicated thread, not on a VP
thread. Results are dispatched back to VP threads via async channels.
Tracing thread — log collection.
CPU-online helper threads — temporary, used when bringing sidecar VPs into
Linux.

This list may not be exhaustive at the point that you're reading these docs, but
the point remains: most work happens on the same thread as the lower VTL VPs
and work that occurs on their behalf in OpenHCL.

Cooperative scheduling

Each VP thread runs an async executor that multiplexes all tasks targeted at
that VP. Tasks only yield to each other at .await points. If you're not
familiar with the Rust execution model, it may be tempting to think that the
system will time slice execution or blocking requests with other async tasks
running on the same VP thread. It won't.

What runs on a VP thread

All tasks with target_vp = N and run_on_target = true run on VP N's thread,
once the target VP is ready (i.e., the CPU is online and affinity is set).

All tasks with target_vp = N and run_on_target = false can be spawned on
any VP's thread. However, target_vp still matters for IO affinity: IOs
issued by the task use VP N's io-uring. When the IO completes, the task is
woken on VP N and will likely continue running there. In practice, these
tasks gravitate toward their target VP despite not requiring it for
scheduling. This keeps per-VP I/O flows localized: submission, completion,
and follow-up processing tend to happen on the same VP.

All tasks without a specific target_vp will fall into the thread pool's
untargeted path. These could run on any arbitrary VP. IO operations from
untargeted tasks currently use VP 0's io-uring. This is a compatibility
default and may change — see the
backend implementation
for details.

Blocking scenarios

Because the executor is cooperative and single-threaded per VP, several
situations can stall all tasks on a VP.

admonish note

In OpenVMM (not OpenHCL), device workers and VP execution run
on separate threads. Code authors *should not* issue a blocking wait inside
an async task. This is a general Rust rule, that emphatically applies in
OpenVMM and OpenHCL.

VTL0 guest execution

When there are no VTL2 tasks that can make forward progress, the VP thread
enters a lower VTL via an ioctl (hcl_return_to_lower_vtl). The thread is
in the kernel until a VM exit returns control to VTL2.

IO completions still wake VTL2. OpenHCL registers the io_uring fd with
the HCL kernel module via set_poll_file. When an io_uring completion fires
(e.g., a disk I/O completes via disk_blockdevice), the kernel cancels the
VM run, returning the thread to VTL2. This applies to any async work that
completes through io_uring — not just disk I/O.

For device interrupts that don't go through io_uring (e.g., the physical
NVMe driver in disk_nvme receives interrupts via an eventfd), the eventfd
is registered with io_uring as a poll operation, so it also triggers the
cancel path.

If the VTL0 guest traps into the hypervisor (e.g., for a hypercall or MMIO
(Memory-Mapped I/O) access that the hypervisor handles on behalf of the
root), the VP is in the hypervisor — not in VTL0 usermode — and the
io_uring cancel mechanism does not apply. The VP remains in the hypervisor
until the intercept completes. I/O completions may still occur but do not
preempt the intercept.

Kernel syscall blocking

If a device worker issues a blocking syscall (e.g., a disk backend falls back to
synchronous I/O), the thread is in the kernel. No .await yield is possible
because the thread itself is blocked. VTL0 cannot execute either.

New device backends should use the built-in io_uring primitives in OpenHCL to
create workers for those blocking tasks. If that is impossible, that device will
need to spawn a new thread. This should be rare, and you should discuss your
rationale with the community before implementing something that way.

Hypervisor intercepts

When VTL2 triggers an operation that requires root partition handling — for
example, an MMIO (Memory-Mapped I/O) write that traps to the hypervisor — the
VP can be stopped in the hypervisor while the root processes the intercept. Both
VTL2 and VTL0 are stalled on that VP.

This is not a software-level problem in OpenHCL — it's an artifact of the
hypervisor/root architecture. The VP physically cannot execute until the root
completes the intercept.

VTL2 blocking VTL0

The reverse of VTL0 blocking: while the VP thread is running VTL2 tasks, VTL0
cannot execute on that VP. A long burst of VTL2 device work (e.g., processing a
large batch of StorVSP completions) delays guest execution. During large
batches, ensure the code contains .await points so other VTL2 tasks (and
ultimately VTL0 execution) can make progress; there is no automatic preemption.

Timeline

A VP thread's execution over time:

text

RUNNING          STALLED          RUNNING        BLOCKED
  ┌──────────────┬─────────────────┬──────────────┬──────────────┐
  │▓▓▓▓▓▓▓▓▓▓▓▓▓▓│░░░░░░░░░░░░░░░░░│▓▓▓▓▓▓▓▓▓▓▓▓▓▓│██████████████│
  │  VTL2 tasks  │  VTL0 guest     │  VTL2 tasks  │   kernel     │
  │              │                 │              │   syscall    │
  │  storvsp,    │  all VTL2       │  storvsp,    │  ALL VTL2    │
  │  netvsp,     │  futures pending│  netvsp,     │  tasks wait  │
  │  relay       │  (exit → wake)  │  relay       │              │
  └──────────────┴─────────────────┴──────────────┴──────────────┘
  ▓ = VTL2 work active    ░ = VTL0 running    █ = kernel blocked

Each segment is mutually exclusive — only one of VTL2 tasks, VTL0 guest, or
kernel work can run at any instant on a given VP thread.

No work stealing

The OpenHCL threadpool does not implement work stealing. Targeted tasks always
run on their target VP's thread. For example: If VP 2's thread is blocked in
VTL0, a StorVSP worker targeted at VP 2 cannot be picked up by VP 3's thread.

Tasks without a specific target_vp run on the thread that wakes them — which
is not the same as work stealing. Similarly, tasks with run_on_target = false
may run on a thread other than their target VP's, but their IOs are still
directed to the target VP's io-uring.

Sidecar changes

On x64 non-isolated VMs, the sidecar kernel splits VP execution
from device work. Most VPs run in the sidecar — a minimal kernel that handles
VTL0 entry/exit without Linux. Only a few CPUs (typically one per NUMA node)
boot into Linux. This is to amortize the CPU startup cost until it becomes
necessary.

Device workers run only on the CPUs that are onlined in the OpenHCL Linux
kernel. Device workers that are CPU agnostic can run in this more limited set of
Linux CPUs.

When a sidecar VP hits an intercept that requires VTL2 processing (the first
handled VM exit), the sidecar CPU is hot-plugged into Linux. From that point,
the VP's device workers can run on its own thread instead.

Impact on device design

When writing device backends, keep these rules in mind:

Never block synchronously in a device worker on a VP thread. Use async
I/O (io_uring) or spawn a helper thread for blocking work. No VMBus devices
used in OpenHCL currently spawn helper threads. Instead, subsystems that need
blocking (GET, VMGS) run on their own dedicated threads outside the VP
threadpool.
Sidecar VPs run remotely first. Beware doing work on a targeted VP early
in boot. If a device worker is targeted at a sidecar VP, it initially runs on
the base CPU, not the target CPU. This can cause contention, but more
importantly: work that must occur on certain VP will cause that VP to exit
the sidecar and enter Linux. This hot-plug has non-trivial latency; sidecar
exists specifically to defer this cost until the VP actually needs VTL2
processing.
Use TaskControl for worker lifecycle. Device workers should implement
AsyncRun
and be managed via
TaskControl,
which provides start/stop/inspect integration. (This doesn't really apply to
the CPU scheduling model, but is general good advice for writing device
backends).

Guide Src Reference Architecture Openhcl Igvm (`Guide/src/reference/architecture/openhcl/igvm.md`)

IGVM Image

The Independent Guest Virtual Machine (IGVM) format describes an isolated VM's
initial state. OpenHCL is delivered as an IGVM image.

Note: For more details on the IGVM specification, see the IGVM repository.

Purpose

The IGVM file serves as the firmware image for the OpenHCL paravisor. It instructs
the host VMM on how to:

Load the OpenHCL components into VTL2 memory.
Place them at required physical addresses in a defined order so launch
measurements are reproducible.
Pass initial configuration data to the paravisor.

IGVM Image Contents

An OpenHCL IGVM image bundles the following artifacts:

Boot Shim (openhcl_boot): The entry point for VTL2 execution.
Linux Kernel: The operating system kernel.
Sidecar Kernel (x86_64): The lightweight kernel for APs.
Initial Ramdisk (initrd): The root filesystem containing userspace
binaries such as underhill_init and openvmm_hcl.
Memory Layout: Directives specifying where each component should be loaded in memory.
Measurements: Data used by the platform to confirm that the expected
image was loaded.
Configuration: Measured build-time parameters and permitted launch-time
data such as CPU topology and device settings.

See
ParavisorMeasuredVtl0Config
and
ParavisorMeasuredVtl2Config
for examples of measured configuration.

Build Process

The IGVM artifact is generated by the OpenHCL build system.
See Building OpenHCL for
build instructions.

The igvmfilegen page documents
manifest generation, image inspection, launch identity, and CoRIM signing.

Guide Src Reference Architecture Openhcl Openvmm Hcl (`Guide/src/reference/architecture/openhcl/openvmm_hcl.md`)

openvmm_hcl

openvmm_hcl is the main user-mode VMM and management process inside the
OpenHCL Linux VTL2 environment.

Position in the boot flow

OpenHCL reaches openvmm_hcl after several earlier stages:

text

host loads OpenHCL IGVM into VTL2
  `- openhcl_boot prepares memory, CPUs, and Linux boot data
       `- Linux kernel boots the initrd
            `- underhill_init prepares userspace
                 `- exec /bin/openvmm_hcl

Startup responsibilities

After Linux userspace starts, openvmm_hcl performs these broad tasks:

Read VTL2 configuration and topology from the device tree and Linux
interfaces established by the boot loader.
Initialize host communication, diagnostics, tracing, and servicing control.
Resolve the device and backend resources compiled into the image.
Start the VM worker that owns the VTL0 processor and device data path.
Start separate workers for devices whose isolation or lifecycle requires a
process boundary.
Coordinate pause, resume, save, restore, servicing, restart, and shutdown.

The process remains the policy and control-plane owner while worker processes
perform high-volume VM and device work.

Worker model

The VM worker runs VTL0 virtual processors, handles exits, and coordinates the
guest device model. Selected devices, including supported TPM configurations,
can run in separate worker processes. OpenHCL communicates with those workers
through typed channels rather than sharing arbitrary process state.

This process model allows a worker to be restarted or serviced while the main
paravisor process retains policy and host communication state. It also limits
which process directly handles sensitive device operations.

Trust boundaries

OpenHCL does not trust the VTL0 guest. In a confidential VM, it also cannot
blindly trust host-provided configuration. The measured IGVM and boot loader
establish which runtime inputs are acceptable before openvmm_hcl sta

Technical Documentation: microsoft/openvmm

Overview (`README.md`) Copy Section

OpenVMM

Getting Started Copy Chapter

Contributing Copy Chapter

Trademarks Copy Chapter

Overview (`guest_test_uefi/README.md`) Copy Section

`guest_test_uefi`

Overview (`opentmk/README.md`) Copy Section

OpenTMK

Overview (`support/README.md`) Copy Section

Overview (`vm/README.md`) Copy Section

Overview (`Guide/mdbook-openvmm-shim/README.md`) Copy Section

mdbook-openvmm-shim

Why is this its own workspace? Copy Chapter

Overview (`petri/guest-bootstrap/README.md`) Copy Section

Overview (`petri/logview/README.md`) Copy Section

Petri Logview Web Application

Prerequisites Copy Chapter

Initial Setup Copy Chapter

1. Install Dependencies Copy Section

2. Verify Installation Copy Section

3. Development Commands Copy Section

Project Structure Copy Chapter

Troubleshooting Copy Chapter

Common Issues Copy Section

Important: Build Before Pushing / Merging (Temporary Feature) Copy Chapter

Recommended: Prettier Setup for Consistent Formatting Copy Chapter

Housekeeping Stuff Copy Chapter

1. Updating branch quick-filters Copy Section

Overview (`vmm_tests/vmm_test_igvm_agent/README.md`) Copy Section

Use test IGVM Agent for local single-test runs

Overview (`vm/devices/support/README.md`) Copy Section

Overview (`vm/loader/manifests/README.md`) Copy Section

SNP Linux-direct profile Copy Chapter

Launching on MSHV Copy Section

Host-described PCIe on MSHV Copy Section

Overview (`vm/devices/tpm/test_data/README.md`) Copy Section

vTPMState Blob

Steps to Create the Blob Copy Chapter

Guide Src Index (`Guide/src/index.md`) Copy Section

Introduction

Cross-Platform Copy Chapter

Running in the OpenHCL paravisor Copy Chapter

Standalone VMM Copy Chapter

Relationship to other Rust-based VMMs Copy Chapter

Guest Compatibility Copy Chapter

Guide Src Reference Devices Firmware Overview (`Guide/src/reference/devices/firmware/overview.md`) Copy Section

Firmware and Boot Modes

Guide Src Reference Emulated NVMe Overview (`Guide/src/reference/emulated/NVMe/overview.md`) Copy Section

NVMe emulator

Guide Src Reference Emulated Pcie Overview (`Guide/src/reference/emulated/pcie/overview.md`) Copy Section

PCIe Emulation

Topology Copy Chapter

PCIe Hotplug Copy Chapter

How it works Copy Section

Runtime API Copy Section

ACPI _OSC Copy Section

MSI Interrupt Routing (aarch64) Copy Section

Implementation notes Copy Section

Guide Src SUMMARY (`Guide/src/SUMMARY.md`) Copy Section

Summary

Getting Started

Developer Guide

Reference

Guide Src Rustdoc (`Guide/src/rustdoc.md`) Copy Section

OpenVMM Rust Crate API Docs

Guide Src Dev Guide Contrib (`Guide/src/dev_guide/contrib.md`) Copy Section

Contributing

Guide Src Dev Guide Dev Tools (`Guide/src/dev_guide/dev_tools.md`) Copy Section

Dev Tools / Utilities

Rust-based Tooling Copy Chapter

Guide Src Dev Guide Getting Started (`Guide/src/dev_guide/getting_started.md`) Copy Section

Getting Started (for Developers)

_Aside:_ What is HvLite? Underhill? Copy Chapter

Guide Src Dev Guide Snapshot Format (`Guide/src/dev_guide/snapshot_format.md`) Copy Section

Snapshot Format

Directory layout Copy Chapter

Manifest format Copy Chapter

Device state (`state.bin`) Copy Chapter

Overview (`README.md`)

Getting Started

Contributing

Trademarks

Overview (`guest_test_uefi/README.md`)

Overview (`opentmk/README.md`)

Overview (`support/README.md`)

Overview (`vm/README.md`)

Overview (`Guide/mdbook-openvmm-shim/README.md`)

Why is this its own workspace?

Overview (`petri/guest-bootstrap/README.md`)

Overview (`petri/logview/README.md`)

Prerequisites

Initial Setup

1. Install Dependencies

2. Verify Installation

3. Development Commands

Project Structure

Troubleshooting

Common Issues

Important: Build Before Pushing / Merging (Temporary Feature)

Recommended: Prettier Setup for Consistent Formatting

Housekeeping Stuff

1. Updating branch quick-filters

Overview (`vmm_tests/vmm_test_igvm_agent/README.md`)

Overview (`vm/devices/support/README.md`)

Overview (`vm/loader/manifests/README.md`)

SNP Linux-direct profile

Launching on MSHV

Host-described PCIe on MSHV

Overview (`vm/devices/tpm/test_data/README.md`)

Steps to Create the Blob

Guide Src Index (`Guide/src/index.md`)

Cross-Platform

Running in the OpenHCL paravisor

Standalone VMM

Relationship to other Rust-based VMMs

Guest Compatibility

Guide Src Reference Devices Firmware Overview (`Guide/src/reference/devices/firmware/overview.md`)

Guide Src Reference Emulated NVMe Overview (`Guide/src/reference/emulated/NVMe/overview.md`)

Guide Src Reference Emulated Pcie Overview (`Guide/src/reference/emulated/pcie/overview.md`)

Topology

PCIe Hotplug

How it works

Runtime API

ACPI _OSC

MSI Interrupt Routing (aarch64)

Implementation notes

Guide Src SUMMARY (`Guide/src/SUMMARY.md`)

Guide Src Rustdoc (`Guide/src/rustdoc.md`)

Guide Src Dev Guide Contrib (`Guide/src/dev_guide/contrib.md`)

Guide Src Dev Guide Dev Tools (`Guide/src/dev_guide/dev_tools.md`)

Rust-based Tooling

Guide Src Dev Guide Getting Started (`Guide/src/dev_guide/getting_started.md`)

_Aside:_ What is HvLite? Underhill?

Guide Src Dev Guide Snapshot Format (`Guide/src/dev_guide/snapshot_format.md`)

Directory layout

Manifest format

Device state (`state.bin`)

Memory (`memory.bin`)

Same-file detection

Code references

Device state architecture

Restore matching rules

Unit naming conventions

Devices that do not support save/restore

Extending the format

Guide Src Dev Guide Tests (`Guide/src/dev_guide/tests.md`)

Guide Src Reference Architecture (`Guide/src/reference/architecture.md`)

Guide Src Reference Openhcl (`Guide/src/reference/openhcl.md`)

Guide Src Reference Openvmm (`Guide/src/reference/openvmm.md`)

Guide Src User Guide Openhcl (`Guide/src/user_guide/openhcl.md`)

Guide Src User Guide Openvmm Img (`Guide/src/user_guide/openvmm-img.md`)

Creating Images

Inspecting and Validating Images

Converting Images

Guide Src User Guide Openvmm (`Guide/src/user_guide/openvmm.md`)

Notable Features

Guide Src Dev Guide Contrib Code (`Guide/src/dev_guide/contrib/code.md`)

`rustfmt`