# Technical Documentation: microsoft/openvmm > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/microsoft/openvmm) ### Overview (`README.md`) # OpenVMM [](https://github.com/microsoft/openvmm/actions/workflows/openvmm-ci.yaml) OpenVMM is a modular, cross-platform Virtual Machine Monitor (VMM), written in Rust. Although it can function as a traditional VMM, OpenVMM's development is currently focused on its role in the OpenHCL paravisor. This repo is the home for both projects. For more information, read our [guide](https://openvmm.dev/) or our [introductory blog post](https://techcommunity.microsoft.com/t5/windows-os-platform-blog/openhcl-the-new-open-source-paravisor/ba-p/4273172). ## Getting Started For info on how to run, build, and use OpenVMM, check out the [The OpenVMM Guide][]. The guide is published out of this repo via [Markdown files](Guide/src/SUMMARY.md). Please keep them up-to-date. The maintainers of this project use Discord to collaborate with external contributors and users. Please [join](https://aka.ms/openvmmdiscord) if you have any questions about contributing to or using OpenVMM/OpenHCL. [The OpenVMM Guide]: https://aka.ms/openvmmguide ## Contributing This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit [Contributor License Agreements](https://cla.opensource.microsoft.com). When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. ## Trademarks This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies. --- ### Overview (`guest_test_uefi/README.md`) # `guest_test_uefi` See the guide for more info on how to build/run the code in this crate. --- ### Overview (`opentmk/README.md`) # OpenTMK See the guide for more info on how to build/run the code in this crate. --- ### Overview (`support/README.md`) This directory is for library and infrastructure crates that are VM agnostic. Or, in other words: - They contain no reference to VMs or VM concepts at all - They contain no dependencies to crates in the repo other than crates in this directory - They are general purpose enough that they could be open source and used by other projects One way to think of this folder is OpenVMM's private crates.io mirror for crates that are not yet ready to be open sourced, but could be. Please name crates based on what they provide, and not general catch all names like "util, base, common" based on what they contain. See the section in the OpenVMM guide on naming crates. --- ### Overview (`vm/README.md`) This directory is for library and infrastructure crates that provide VM-related functionality, _without_ being tied to any particular "VMM frontend" (e.g: OpenVMM, OpenHCL, etc...) Crates in this directory must not take any direct dependencies on any code outside of `vm/`, _except_ for crates pulled from crates.io, or crates under `../support/`. --- ### Overview (`Guide/mdbook-openvmm-shim/README.md`) # mdbook-openvmm-shim This is a small standalone Rust binary which implements a mdbook preprocessor "shim", used to give OpenVMM additional control over how `mdbook` finds and interacts with external preprocessors. Notably: it allows us to work around the following outstanding mdbook issue: Preprocessor cannot add folders/files directly to book directory [#1222](https://github.com/rust-lang/mdBook/issues/1222). Instead, this shim will dynamically invoke each preprocessor's `install` subcommand on-demand, avoiding the need to check-in the somewhat large third-party css/js/html blobs that these plugins depend on. In addition, this shim has the benefit of keeping the `mdbook` end-user experience as vanilla as possible, without requiring a project-specific wrapper script (e.g: `cargo xtask mdbook`). ## Why is this its own workspace? Since `book.toml` uses `cargo run` to invoke this script, having this crate live in the main OpenVMM Rust workspace would result in delays of ~0.5s on each command invocation (due to `cargo run` needing to scan the whole workspace on each invocation). Having it in its own workspace results in a `cargo run` overhead of roughly `0.01` seconds (on average), which is far more tolerable. In the future, when `cargo-script` finally lands (see ), this shim should simply be a standalone file in the `Guide/` directory. --- ### Overview (`petri/guest-bootstrap/README.md`) This directory contains files needed to bootstrap the guest with the pipette agent. * `meta-data` and `user-data`: cloud-init files for Linux guests * `imc.hiv`: an IMC hive for Windows guests To update `imc.hiv`, on a Windows machine run `cargo run -p make_imc_hive PATH/TO/imc.hiv` --- ### Overview (`petri/logview/README.md`) # Petri Logview Web Application A React-based web application for viewing and analyzing Petri logs, built with TypeScript, Vite, and modern React ecosystem tools. ## Prerequisites - **Node.js** (version `>=22.22.0`, as required by react-router 8) - **npm** (comes with Node.js) - **Browser**: Safari 16.4+, Chrome 111+, or Firefox 128+ (required by Tailwind CSS 4) Tailwind CSS 4 runs through the `@tailwindcss/vite` plugin in `vite.config.ts`. `tailwind.css` imports the theme and utilities without Preflight and retains the existing gray palette, preserving browser defaults and the site's custom styles. Source scanning is limited to TypeScript files under `src/`. ## Initial Setup Follow these steps to set up the project from scratch: ### 1. Install Dependencies Navigate to the project directory and install all required packages: ```powershell # Navigate to the logview directory cd .\petri\logview # Install all dependencies exactly as locked npm ci ``` `npm ci` installs the exact versions recorded in `package-lock.json`, which is what CI runs. Use `npm install` only when you intend to add or update a dependency, and commit the resulting `package-lock.json` changes. ### 2. Verify Installation After installation, you should have the following key dependencies: **Runtime Dependencies:** - `react` & `react-dom` - React framework - `react-router` - Client-side routing - `@tanstack/react-query` - Data fetching and caching - `@tanstack/react-table` - Table component library - `@tanstack/react-virtual` - Virtual scrolling **Development Dependencies:** - `typescript` - TypeScript compiler - `vite` - Build tool and dev server - `@vitejs/plugin-react` - Vite React plugin - `eslint` - Code linting - `@types/react` & `@types/react-dom` - TypeScript definitions ### 3. Development Commands ```powershell # Start development server (runs on http://localhost:3000) npm run dev # Build for production npm run build # Preview production build npm run preview # Run linting npx eslint . ``` ## Project Structure ```text logview/ ├── src/ │ ├── main.tsx # Application entry point │ ├── style/ # All CSS and global styles (CSS, PostCSS, or other style assets) │ │ └── ... # e.g. globals.css, themes, component styles │ └── table_defs/ # Column definitions for tables used across the app │ └── ... # e.g. logsColumns.ts, eventsColumns.ts, devicesColumns.ts ├── index.html # HTML template ├── package.json # Dependencies and scripts ├── tsconfig.json # TypeScript configuration ├── tsconfig.node.json # TypeScript config for build tools ├── vite.config.ts # Vite configuration ├── eslint.config.ts # ESLint configuration └── README.md # This file ``` ## Troubleshooting ### Common Issues 1. **Module not found errors**: Ensure all dependencies are installed with `npm ci` 2. **TypeScript errors**: Make sure both `tsconfig.json` and `tsconfig.node.json` are present 3. **Port already in use**: The dev server uses port 3000 by default. You can change this in `vite.config.ts` ## Important: Build Before Pushing / Merging (Temporary Feature) Please run a production build before pushing or opening a merge request to ensure the front-end assets are up to date. ```powershell npm run build ``` - Where the output goes: - The build artifacts will be written to the `assets/` folder at the project root (e.g. `logview/assets/`). - You can run `npm run preview` to verify the production build locally. Note: This is a temporary requirement and will be automated in the future. ## Recommended: Prettier Setup for Consistent Formatting To ensure consistent code formatting, it is recommended to install the [Prettier](https://prettier.io/) extension in your code editor (such as VS Code). For VS Code users, add the following to your settings to automatically format files on save: ```json "editor.defaultFormatter": "esbenp.prettier-vscode" ``` ## Housekeeping Stuff ### 1. Updating branch quick-filters The branches we want to track test and runs for are bound to frequently change as new releases come through. For convenience these filters are defined as a list in the `branch_quick_filters.tsx` file. `run_filters` will update the filters that show up on the Runs page. `test_filters` will update the filters that show up for the Test and TestDetails pages. `all` is a reserved branch filter name and will display all the branches. --- ### Overview (`vmm_tests/vmm_test_igvm_agent/README.md`) # Use test IGVM Agent for local single-test runs Windows-only helper to start `test_igvm_agent_rpc_server.exe` when it is not already running (flowey still owns CI). For local nextest runs we assume one test process at a time; the helper starts the server at test begin, streams stderr into the test output, and tears it down via a guard when the test ends. Concurrency is not supported or guaranteed. Local runs must explicitly opt-in to the helper by exporting `VMM_TEST_IGVM_AGENT_LOCAL_AUTOSTART=1` (or `true`/`yes`). CI never sets this variable so tests fail loudly if Flowey forgets to launch the server. The crate exposes this variable as the `LOCAL_AUTOSTART_ENV` constant for reuse in callers. --- ### Overview (`vm/devices/support/README.md`) These support crates are tailored to specific devices, and are a bit too specialized to be a part of the top level `support/` folder. --- ### Overview (`vm/loader/manifests/README.md`) This folder contains manifest recipes for building IGVM files. Create the resource file and run `igvmfilegen manifest` directly. ## SNP Linux-direct profile Flowey builds `snp-linux-direct-pipette.json` for the `linux_direct_x64[snp]` VMM test configuration. It injects the statically linked Linux pipette into the shared test initrd from `openvmm-deps`, then runs it as PID 1 with `rdinit=/pipette`. The test also uses the SNP guest kernel from `openvmm-deps` and the in-tree `snp_bootshim`. The manual `snp-linux-direct.json` bring-up profile instead starts the initrd shell. These profiles have these assumptions: - x64 and one VTL0 SEV-SNP guest that boots Linux directly - at least 160 MiB of RAM (40,960 4-KiB pages), which the image measures - DT-described CPUs, RAM, NUMA nodes, UARTs, and optional PCIe bridges, with the fixed enlightened x86 chipset - no shared GPA boundary, normal interrupt injection, and secure AVIC disabled - base SNP policy `0x30000`; `enable_debug` adds the debug bit to produce the current debug-capable policy `0xb0000` - an initrd and the kernel command line `console=ttyS0 earlyprintk=serial earlycon panic=-1` - SNP C-bit position 51; the value must be bit 32 or higher because the startup page tables identity-map the lower 4 GiB The normal-injection output is a shared artifact: the same binary is intended to boot on KVM and MSHV. Its `SnpVpContext` uses the SNP initial-VMSA GPA `0xffff_ffff_f000`. KVM synthesizes its measured VMSA at that GPA, while MSHV maps and imports the file-provided VMSA there. Both backends use the policy encoded in the file, but only MSHV submits its SNP ID block. The `snp-linux-direct-restricted.json` profile encodes restricted interrupt injection in its IGVM VMSA. It is intended only for MSHV bring-up. Every SNP Linux-direct image requests the host's device tree through an unmeasured IGVM parameter area. After launch, the measured bootshim reads the CPUs, RAM, PCIe host bridges, and COM ports from the tree and generates DSDT, FADT, MADT, SRAT, XSDT and RSDP. It adds MCFG and a PCIe SSDT only when bridges are present. Images contain zero-filled output storage, not prebuilt ACPI tables. One image therefore boots with any supported CPU count, RAM size, NUMA layout, and PCIe configuration. The runtime describes COM1/COM2 when any serial backend is configured, and COM3/COM4 only when their individual backend is configured. The shim emits only the UARTs in that description. OpenHCL still selects a described COM3 as its console. Both consumers use the existing IGVM PIO binding, in which the `reg` property of `pio-bus/serial@` starts with a big-endian 64-bit base. The shim finds each COM port by that base and uses the fixed COM I/O port and IRQ. The measured handoff supplies the end of the measured RAM and bounded workspace locations. Older experimental images and shims are incompatible. Rebuild both; there is no static-table fallback. The image contains a small measured bootshim. Only pages containing the kernel, initrd, boot metadata, SNP special pages, bootshim, or bootshim parameters are included as IGVM `PageData`, along with zero-filled permanent ACPI and RSDP storage. The DT uses private, already-imported parameter pages. The shim reads the DT without allocating before it accepts any omitted pages. Then it accepts the remaining private RAM with `PVALIDATE` and enters Linux. This avoids loading and measuring every configured RAM page, but still accepts all RAM before Linux starts. The shim trusts the RAM that the host device tree describes. Below the measured RAM end, it accepts only the gaps that the measured parameter page lists, so it never accepts a measured page again. It accepts all host RAM above the measured RAM end and adds that RAM to the Linux E820 map. The IGVM contains only the BSP VMSA, regardless of processor count. Backends are responsible for any AP launch state they require. Current KVM constructs and measures the initial VMSAs itself rather than accepting the IGVM VMSA page. Those KVM-created VMSAs are not part of the IGVM launch measurement, so the file's SNP ID block is not valid for KVM. KVM attestation against that ID block remains unsupported until KVM accepts userspace-provided VMSAs. To build it manually, create a resources file containing absolute paths: ```json { "resources": { "linux_kernel": "/absolute/path/to/vmlinux-or-bzImage", "linux_initrd": "/absolute/path/to/initrd", "snp_bootshim": "/absolute/path/to/snp_bootshim" } } ``` Build the bootshim first: ```bash MINIMAL_RT_BUILD=1 cargo build \ --profile boot-dev \ --target x86_64-unknown-none \ -p snp_bootshim ``` Build the host-native generator: ```bash cargo build -p igvmfilegen ``` Then generate the image: ```bash repo=/absolute/path/to/openvmm cargo run -p igvmfilegen -- manifest \ --manifest "$repo/vm/loader/manifests/snp-linux-direct.json" \ --resources /absolute/path/to/snp-linux-direct-resources.json \ --output /absolute/path/to/snp-linux-direct.bin ``` The standard outputs are: - `snp-linux-direct.bin` - `snp-linux-direct.bin.map` - `snp-linux-direct-snp.json` ### Launching on MSHV The guest needs at least `memory_page_count * 4096` bytes of RAM. Choose the CPU count, larger RAM sizes, and NUMA layout at launch: ```bash openvmm --hypervisor mshv --isolation snp \ --igvm firmware=path/to/snp-linux-direct.bin,personality=linux-direct \ --hv --no-vmbus \ --memory 2GB --processors 4 \ --com1 console ``` Keep COM1 for the profile's `console=ttyS0` kernel command line. If any serial backend is configured, the host describes COM1 and COM2. It describes COM3 and COM4 only when their individual backends are configured. The shim emits only described UARTs and supports shared IRQ4 for COM1/COM3 and IRQ3 for COM2/COM4. It does not use console metadata to infer UART presence. ### Host-described PCIe on MSHV Use the same image. The runtime supplies the resolved PCIe layout; no ECAM-base or MMIO-base overrides are needed. For example: ```bash openvmm --hypervisor mshv --isolation snp --hv --no-vmbus \ --igvm firmware=path/to/snp-linux-direct.bin,personality=linux-direct \ --memory 160MB --processors 2 --vps-per-socket 2 \ --com1 file=path/to/serial.log \ --pcie-root-complex rc0,segment=0,start_bus=0,end_bus=31,low_mmio=64M,high_mmio=1G,node=0 \ --pcie-ecam-below-4gb \ --pcie-root-port rc0:disk0 --pcie-root-port rc0:net0 \ --virtio-blk file:path/to/disk.raw,pcie_port=disk0 \ --virtio-net pcie_port=net0:consomme ``` The below-4-GiB option selects a placement class, not a fixed address. Some direct-boot Linux kernels reject MCFG entries above 4 GiB unless SMBIOS reports a sufficiently recent BIOS date. Keep this option for those kernels; the converter supports 64-bit addresses but cannot bypass that guest policy. This implementation supports at most 255 CPUs, 64 RAM ranges that end at or below 512 GiB, the four fixed COM ports, and eight generic ECAM bridges in distinct segments. The tables put every bridge on node 0, with native x86 MSI/MSI-X and identity low and high MMIO windows; either window may be omitted. The bootshim stops the guest when the device tree exceeds these limits or puts two bridges in one segment. The x86 device tree does not describe an IOMMU or CXL, so the guest does not see them. The image reserves 64 KiB for the device tree, 64 KiB for generated ACPI, a temporary 1-MiB heap, and a 32-KiB bootshim stack. The heap is omitted from page data and accepted with the RAM gaps before use. ACPI output, RSDP, DT, the shim, measured handoff pages, and SNP special pages are imported, not accepted again. No final table points into the DT, heap, or shim. The bootshim stops the guest if it cannot parse a PCIe bridge, or if a PCIe ECAM or MMIO window overlaps RAM or reaches the C-bit. The ACPI arena is ACPI-reclaimable in E820; the pinned RSDP remains reserved. The shim copies all tables to permanent storage before it publishes the RSDP and Linux zero-page pointer. It never enters Linux with partial ACPI. The shim ignores all other device-tree nodes and properties, including the `/chosen` and VMBus nodes. OpenVMM adds the `/openhcl` node only for `personality=openhcl`. Host-selected topology is unmeasured. Attestation policy for those values is deferred for bring-up; the shim does not bind them into the launch identity. When the shim stops the guest, it sends a GHCB termination request with reason-code set 0xF and a reason code from `Failure` in `vm/loader/snp_bootshim/src/main.rs`. --- ### Overview (`vm/devices/tpm/test_data/README.md`) # vTPMState Blob The blob is used for testing TPM with pre-provisioned states. Refer to the following tests for usage examples: - tpm_device::tests::test_fix_corrupted_vmgs - tpm_lib::tests::test_with_pre_provisioned_state - tpm_lib::tests::test_initialize_guest_secret_key ## Steps to Create the Blob 1. Initialize TPM (an `MsTpm20RefPlatform` instance) with callbacks that define the data store where TPM will write its NV states. 2. Configure the TPM with the desired commands to establish the required state. 3. Once the TPM state is prepared, export the data store to generate the blob. --- ### Guide Src Index (`Guide/src/index.md`) # Introduction OpenVMM is a modular, cross-platform Virtual Machine Monitor (VMM), written in Rust. OpenVMM can run as a traditional VMM and also serves as the VMM within the [OpenHCL paravisor][paravisor]. The project is open-source, MIT Licensed, and developed publicly at [microsoft/openvmm](https://github.com/microsoft/openvmm) on GitHub. ## Cross-Platform OpenVMM supports a variety of host operating systems, architectures, and virtualization backends: | Host OS | Architecture | Virtualization API | | ------------------- | ------------- | -------------------------------------- | | Linux ([paravisor]) | x64 / Aarch64 | MSHV (using [VSM] / [TDX] / [SEV-SNP]) | | Windows | x64 / Aarch64 | WHP (Windows Hypervisor Platform) | | Linux | x64 / Aarch64 | KVM | | | x64 / Aarch64 | MSHV (Microsoft Hypervisor) | | macOS | Aarch64 | Hypervisor.framework | ## Running in the OpenHCL paravisor OpenVMM is the VMM that runs in the [OpenHCL paravisor][paravisor]. Unlike in traditional virtualization, where a VMM runs in a privileged host/root partition and provides virtualization services to a unprivileged guest partition, the "paravisor" model enables a VMM to provide virtualization services from _within_ the guest partition itself. It can be considered a form of "virtual firmware", running at a higher privilege level than the primary guest OS. Paravisors are quite exciting, as they enable a wide variety of useful and novel virtualization scenarios! For example: at Microsoft, OpenHCL plays a key role in enabling several important Azure scenarios: - Enabling existing workloads to seamlessly leverage [Azure Boost] (Azure's next-generation hardware accelerator), without requiring any modifications to the guest VM image. - Enabling existing guest operating systems to run inside [Confidential VMs]. - Powering [Trusted Launch VMs] - VMs that support Secure Boot, and include a vTPM. ## Standalone VMM OpenVMM can also run as a general-purpose VMM on a Windows, Linux, or macOS host. This configuration shares most of its code with OpenVMM running in a paravisor and provides a convenient environment for running and testing virtual machines directly on a host. ## Relationship to other Rust-based VMMs OpenVMM's core security principles are aligned with those of the Rust-based Cloud Hypervisor, Firecracker, and crosvm projects, which is why we also chose to write OpenVMM in Rust. However, OpenVMM's unique goal of running efficiently in a paravisor environment made it difficult to leverage existing projects. OpenVMM requires fine-grained control over thread and task scheduling in order to avoid introducing jitter and other performance issues into guest VMs. It is difficult to achieve these requirements with traditional, thread-based designs. Instead, OpenVMM uses Rust's `async` support throughout its codebase, decoupling the policy details of _where_ code runs (which OS threads) from the mechanism of _what_ runs (device-specific emulators). In a paravisor or resource-constrained environment, OpenVMM can run with one thread per guest CPU and ensure that device work is cooperatively scheduled along with the guest OS. In more traditional virtualization host, OpenVMM can run with one thread per device to use host CPUs to fully parallelize guest CPU and IO processing. This approach has a significant impact on the design and implementation of the codebase, and bringing this model to an existing VMM would be a major undertaking. We came to the conclusion that a new project was the best way to achieve this goal. We are indebted to the Rust VMM community for their trailblazing work. Now that the OpenVMM project is open source, we hope to find ways to collaborate on shared code while maintaining the benefits of the OpenVMM architecture. ## Guest Compatibility Similar to other general-purpose VMMs (such as Hyper-V, QEMU, VirtualBox), OpenVMM is able to host a wide variety of both modern and legacy guest operating systems on-top of its flexible virtual hardware platform. - Modern operating systems can boot via UEFI, and interface with a wide selection of paravirtualized devices for services like networking, storage, and graphics. - Legacy x86 operating systems can boot via BIOS, and are presented with a PC-compatible emulated device platform which includes legacy hardware such as IDE hard-disk/optical drives, floppy disk drives, and VGA graphics cards. OpenVMM is regularly tested to ensure compatibility with popular operating systems (such as Windows, Linux, and FreeBSD), and strives to maintain reasonable compatibility with other, more niche/legacy operating systems as well. * * * To learn more about different facets of the OpenVMM project, check out the following links: | | | | ----------------------------------------------------------------------------- | ----------------------------------------- | | [Getting Started: OpenVMM](./user_guide/openvmm.md) | Running OpenVMM as traditional host VMM | | [Getting Started: OpenHCL](./user_guide/openhcl.md) | Running OpenVMM as a paravisor (OpenHCL) | | [Developer Guide: Getting Started](./dev_guide/getting_started.md) | Building OpenVMM / OpenHCL locally | | [[Github] microsoft/openvmm](https://github.com/microsoft/openvmm) | Viewing / Downloading OpenVMM source code | | [[Github] OpenVMM issue tracker](https://github.com/microsoft/openvmm/issues) | Reporting OpenVMM issues | [paravisor]: ./user_guide/openhcl.md [VSM]: https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/vsm [Azure Boost]: https://learn.microsoft.com/en-us/azure/azure-boost/overview [Confidential VMs]: https://azure.microsoft.com/en-us/solutions/confidential-compute [Trusted Launch VMs]: https://learn.microsoft.com/en-us/azure/virtual-machines/trusted-launch [TDX]: https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/overview.html [SEV-SNP]: https://www.amd.com/content/dam/amd/en/documents/epyc-business-docs/white-papers/SEV-SNP-strengthening-vm-isolation-with-integrity-protection-and-more.pdf --- ### Guide Src Reference Devices Firmware Overview (`Guide/src/reference/devices/firmware/overview.md`) # Firmware and Boot Modes OpenVMM supports several ways to boot a guest VM, each with different firmware requirements and guest OS compatibility: | Boot mode | Architecture | Firmware | Use case | |-----------|-------------|----------|----------| | **UEFI** | x86_64, AArch64 | [mu_msvm](./mu_msvm_uefi.md) | Windows, modern Linux, full UEFI environment | | **PCAT BIOS** | x86_64 | [Hyper-V PCAT BIOS](./pcat_bios.md) | Legacy OS, Gen1-style boot | | **Linux Direct** | x86_64, AArch64 | None (VMM is the bootloader) | [Fast Linux boot](./linux_direct.md), development, testing | | **IGVM** | x86_64, AArch64 | Packaged in IGVM file | OpenHCL paravisor, confidential VMs | The boot mode is selected by which `--kernel`, `--uefi`, `--pcat`, or `--igvm` flag is passed on the command line (or the equivalent ttrpc configuration). ```admonish note Not all boot modes are available on all architectures — see the table above for supported combinations. ``` --- ### Guide Src Reference Emulated NVMe Overview (`Guide/src/reference/emulated/NVMe/overview.md`) # NVMe emulator Among the devices that OpenVMM emulates, an NVMe controller is one. The OpenVMM NVMe emulator comes in two flavors: - An NVMe emulator that can be used to serve IO workloads (but pragmatically is only used by OpenVMM for test scenarios today) - An NVMe emulator used to test OpenHCL ([`nvme_test`](https://openvmm.dev/rustdoc/linux/nvme_test/index.html)), which allows test authors to inject faults and inspect the state of NVMe devices used by the guest. This guide provides a brief overview of the architecture shared by the NVMe emulators. For how NVMe fits into the broader storage pipeline — including how namespaces map to [`DiskIo`](https://openvmm.dev/rustdoc/linux/disk_backend/trait.DiskIo.html) backends, online disk resize via AEN, and the layered disk model — see the [storage pipeline](../../architecture/devices/storage.md) page. --- ### Guide Src Reference Emulated Pcie Overview (`Guide/src/reference/emulated/pcie/overview.md`) # PCIe Emulation OpenVMM emulates a PCIe topology that presents root complexes, root ports, and optional switches to the guest. Endpoint devices (NVMe, virtio, etc.) attach to root ports via a generic bus interface. ## Topology ```text Root Complex (GenericPcieRootComplex) ├── Root Port 0 (PcieDownstreamPort) → endpoint device ├── Root Port 1 (PcieDownstreamPort) → endpoint device or switch └── Root Port N (PcieDownstreamPort) → ... ``` The root complex owns the ECAM MMIO region. When the guest reads or writes a config space address, the root complex decodes the bus/device/function from the ECAM offset and routes the access to the correct port. Each port has a Type 1 (bridge) configuration space with PCIe Express and MSI capabilities. Ports may optionally be hotplug-capable. Devices behind non-hotplug ports are attached at VM construction time. Hotplug-capable ports start empty and devices can be added or removed at runtime. ## PCIe Hotplug Native PCIe hotplug follows the same interrupt-driven model as real hardware (PCIe Base Spec §6.7). No ACPI GPE, SCI, or custom protocol is needed — the guest's `pciehp` driver handles everything via config space registers and MSI. ### How it works 1. **VMM sets port state**: When a device is hot-added, the VMM atomically updates the port's Slot Status (`presence_detect_state`, `presence_detect_changed`, `data_link_layer_state_changed`) and Link Status (`data_link_layer_link_active`). 2. **MSI fires**: If the guest has enabled `hot_plug_interrupt_enable` in the port's Slot Control register, the VMM fires the port's MSI. 3. **Guest handles the event**: The guest's `pciehp` driver receives the MSI, reads Slot Status to see what changed, programs the bridge's bus numbers, scans the secondary bus for new devices, and clears the RW1C status bits. 4. **Device removal** follows the same flow in reverse — presence and link active are cleared, changed bits are set, and MSI fires. ### Runtime API Hot-add and hot-remove are triggered via `VmRpc::AddPcieDevice` and `VmRpc::RemovePcieDevice` messages. These resolve a device resource, create the device with MMIO registration, attach it to the named port, and fire the hotplug notification. See the [`PetriVmRuntime`](https://openvmm.dev/rustdoc/linux/petri/vm/trait.PetriVmRuntime.html) trait for the test API (`add_pcie_device` / `remove_pcie_device`) and the `pcie_hotplug` test in `vmm_tests/vmm_tests/tests/tests/multiarch/pcie.rs` for a working example. ### ACPI _OSC The SSDT includes an `_OSC` method on each PCIe root complex that grants native PCIe control to the OS (ACPI spec §6.2.11, PCI Firmware Spec §4.5.1). This tells the OS it can use native hotplug, PME, AER, and other PCIe features rather than ACPI-based fallbacks. Linux assumes native control regardless, but Windows requires `_OSC` to enable native hotplug. ### MSI Interrupt Routing (aarch64) On aarch64, PCIe MSI/MSI-X interrupts are routed through either a GICv3 ITS or a GICv2m MSI frame, depending on the platform: - **GICv3 ITS** (default on KVM with GICv3): The VMM creates a KVM in-kernel ITS device. Each MSI is delivered with a 32-bit ITS device ID of the form `(segment << 16) | BDF`. The BDF is resolved lazily at interrupt-delivery time by the device's `MsiTarget`, which combines the parent port's `AssignedBusRange` (updated by the guest as it programs secondary/subordinate bus numbers) with the device's `devfn`. Multi-function devices and root/switch ports that need a specific RID pass it explicitly. A single `ItsSignalMsi`/`ItsIrqFd` wrapper per segment then prepends the segment to produce the final ITS device ID — no per-device wrappers and no push-based RID synchronization are needed. ACPI boots emit an IORT with an ITS Group node and per-root-complex ID mappings; the device tree includes an `its` child node under the GIC with `msi-controller`. - **GICv2m**: MSI writes map to a fixed pool of 64 SPIs via a v2m doorbell register. The MADT includes a GICv2m MSI frame entry. The MSI controller can be overridden with the `--gic-msi` CLI option (`auto`, `its`, or `v2m`). ### Implementation notes ```admonish note title="No Command Completed support" Hotplug ports advertise `no_command_completed_support` in Slot Capabilities. Our emulation applies Slot Control changes instantly, so the guest does not need to wait for command completion. This avoids an interrupt storm that would occur if `command_completed` were set on every Slot Control write — see the comment in `with_hotplug_support()` in `pci_express.rs` for details. ``` ```admonish note title="Spurious PME interrupts" The `pcieport` driver's PME service shares the same MSI vector as `pciehp`. When a hotplug MSI fires, the PME handler also runs and may log "Spurious native interrupt!" since there is no PME event. These warnings are cosmetic and do not affect functionality. A future improvement could use MSI-X (multiple vectors) to give each service its own interrupt. ``` --- ### Guide Src SUMMARY (`Guide/src/SUMMARY.md`) # Summary [Introduction](./index.md) # Getting Started - [OpenVMM](./user_guide/openvmm.md) - [Running OpenVMM](./user_guide/openvmm/run.md) - [Alpine Linux](./user_guide/openvmm/alpine.md) - [VM Configurations](./user_guide/openvmm/vm_configurations.md) - [VFIO Device Assignment](./user_guide/openvmm/vfio.md) - [Troubleshooting](./user_guide/openvmm/troubleshooting.md) - [Snapshots](./user_guide/openvmm/snapshots.md) - [VM Memory Dumps](./user_guide/openvmm/vm_memory_dumps.md) - [Next Steps](./user_guide/openvmm/next_steps.md) - [OpenHCL](./user_guide/openhcl.md) - [Running OpenHCL](./user_guide/openhcl/run.md) - [On Windows - Hyper-V](./user_guide/openhcl/run/hyperv.md) - [On Windows - OpenVMM](./user_guide/openhcl/run/openvmm.md) - [On Linux](./user_guide/openhcl/run/openvmm_linux.md) - [Troubleshooting](./user_guide/openhcl/troubleshooting.md) - [Next Steps](./user_guide/openhcl/next_steps.md) - [`openvmm-img`](./user_guide/openvmm-img.md) # Developer Guide - [Getting Started](./dev_guide/getting_started.md) - [On Linux / WSL2](./dev_guide/getting_started/linux.md) - [On Windows](./dev_guide/getting_started/windows.md) - [Via Codespaces / Dev Container](./dev_guide/getting_started/devcontainer.md) - [Building OpenVMM](./dev_guide/getting_started/build_openvmm.md) - [Building OpenHCL](./dev_guide/getting_started/build_openhcl.md) - [Building a Custom Kernel](./dev_guide/getting_started/build_ohcl_kernel.md) - [Suggested Dev Environment](./dev_guide/getting_started/suggested_dev_env.md) - [Cross Compiling for Windows](./dev_guide/getting_started/cross_compile.md) - [Running from WSL2](./dev_guide/getting_started/running_from_wsl.md) - [Testing](./dev_guide/tests.md) - [Unit Tests](./dev_guide/tests/unit.md) - [VMM Tests](./dev_guide/tests/vmm.md) - [Azure-hosted Test Images](./dev_guide/tests/vmm/azure_images.md) - [Pipette Guest Agent](./dev_guide/tests/pipette.md) - [Windows Test Image Preparation](./dev_guide/tests/prep_steps.md) - [IMC Hive Generator](./dev_guide/tests/make_imc_hive.md) - [Incubator](./dev_guide/tests/incubator.md) - [TPM Guest Test Utility](./dev_guide/tests/tpm_guest_tests.md) - [IGVM Agent Test Server](./dev_guide/tests/test_igvm_agent_rpc_server.md) - [TMK VMM](./dev_guide/tests/tmk_vmm.md) - [simple_tmk](./dev_guide/tests/simple_tmk.md) - [OpenTMK](./dev_guide/tests/opentmk.md) - [OpenTMK Executor](./dev_guide/tests/opentmk_executor.md) - [Performance Benchmarks]() - [Performance Tests (burette)](./dev_guide/tests/perf.md) - [VMM.Perf Runner](./dev_guide/tests/vmm_perf.md) - [Fuzzing](./dev_guide/tests/fuzzing.md) - [Running Fuzzers](./dev_guide/tests/fuzzing/running.md) - [Writing Fuzzers](./dev_guide/tests/fuzzing/writing.md) - [Developer Tools / Utilities](./dev_guide/dev_tools.md) - [`flowey`](./dev_guide/dev_tools/flowey.md) - [`Flowey Fundamentals`](./dev_guide/dev_tools/flowey/flowey_fundamentals.md) - [`Steps`](./dev_guide/dev_tools/flowey/steps.md) - [`Variables`](./dev_guide/dev_tools/flowey/variables.md) - [`Nodes`](./dev_guide/dev_tools/flowey/nodes.md) - [`Artifacts`](./dev_guide/dev_tools/flowey/artifacts.md) - [`Pipelines`](./dev_guide/dev_tools/flowey/pipelines.md) - [`Nix`](./dev_guide/dev_tools/flowey/nix.md) - [`cargo xtask`](./dev_guide/dev_tools/xtask.md) - [`cargo xflowey`](./dev_guide/dev_tools/xflowey.md) - [VmgsTool](./dev_guide/dev_tools/vmgstool.md) - [update-rootfs.py]() - [`igvmfilegen`](./dev_guide/dev_tools/igvmfilegen.md) - [guest_test_uefi](./dev_guide/dev_tools/guest_test_uefi.md) - [hypestv](./dev_guide/dev_tools/hypestv.md) - [Contributing](./dev_guide/contrib.md) - [Coding Conventions](./dev_guide/contrib/code.md) - [Save State](./dev_guide/contrib/save-state.md) - [Snapshot Format](./dev_guide/snapshot_format.md) - [Releases & Code Flow](./dev_guide/contrib/release.md) - [Security Releases](./dev_guide/contrib/security_releases.md) - [Packaging OpenVMM for Linux](./dev_guide/contrib/openvmm_packaging.md) - [Cutting an OpenVMM Source Release](./dev_guide/contrib/openvmm_source_release.md) - [Submitting Changes](./dev_guide/contrib/pr.md) - [Code Review Process](./dev_guide/contrib/code_review.md) - [Guide Updates](./dev_guide/contrib/guide.md) - [Style Guide](./dev_guide/contrib/style_guide.md) - [Crypto Backends](./dev_guide/contrib/crypto_backends.md) # Reference - [OpenVMM Features](./reference/openvmm.md) - [Configuration and Management](./reference/openvmm/management.md) - [CLI](./reference/openvmm/management/cli.md) - [Interactive Console](./reference/openvmm/management/interactive_console.md) - [gRPC / ttrpc](./reference/openvmm/management/grpc.md) - [Graphical Console](./reference/openvmm/graphical_console.md) - [Logging](./reference/openvmm/logging.md) - [OpenHCL Features](./reference/openhcl.md) - [Dynamic Configuration]() - [Diagnostics](./reference/openhcl/diag.md) - [Preface: CVM restrictions](./reference/openhcl/diag/cvm_restrictions.md) - [Intro to ohcldiag-dev](./reference/openhcl/diag/ohcldiag_dev.md) - [Network packet capture (PCAP)](./reference/openhcl/diag/ohcldiag_dev/pcap.md) - [Performance analysis](./reference/openhcl/diag/ohcldiag_dev/perf.md) - [Tracing](./reference/openhcl/diag/tracing.md) - [Debugging](./reference/openhcl/debugging.md) - [Developer Features]() - [Hardware Debugging (gdbstub)](./reference/dev_feats/gdbstub.md) - [Kernel Debugging (KDNET)](./reference/dev_feats/kdnet.md) - [Firmware and Boot Modes](./reference/devices/firmware/overview.md) - [UEFI: mu_msvm](./reference/devices/firmware/mu_msvm_uefi.md) - [BIOS: Hyper-V PCAT BIOS](./reference/devices/firmware/pcat_bios.md) - [Linux Direct](./reference/devices/firmware/linux_direct.md) - [Devices]() - [PCI IDs](./reference/devices/pci_ids.md) - [Virtio]() - [virtio-fs]() - [virtio-9p]() - [virtio-net]() - [virtio-pmem]() - [VMBus]() - [storvsp](./reference/devices/vmbus/storvsp.md) - [Channels & Subchannels](./reference/devices/vmbus/storvsp_channels.md) - [netvsp]() - [vpci]() - [serial]() - [framebuffer]() - [input]() - [Emulated]() - [IPMI KCS](./reference/emulated/ipmi_kcs.md) - [vTPM]() - [NVMe]() - [Overview](./reference/emulated/NVMe/overview.md) - [Doorbells](./reference/emulated/NVMe/doorbells.md) - [Serial]() - [Legacy x86]() - [i440BX + PIIX4 chipset]() - [IDE HDD/Optical](./reference/emulated/legacy_x86/ide.md) - [Floppy](./reference/emulated/legacy_x86/floppy.md) - [PCI]() - [VGA]() - [PCIe]() - [Overview](./reference/emulated/pcie/overview.md) - [IOMMU]() - [Arm SMMUv3](./reference/emulated/iommu/smmuv3.md) - [Device Backends]() - [Serial]() - [Graphics and Input]() - [Storage](./reference/backends/storage.md) - [VHDX Parser](./reference/backends/vhdx.md) - [Networking](./reference/backends/networking.md) - [Consomme](./reference/backends/consomme.md) - [Architecture](./reference/architecture.md) - [Timekeeping](./reference/architecture/timekeeping.md) - [OpenVMM Architecture](./reference/architecture/openvmm.md) - [Memory Layout](./reference/architecture/openvmm/memory-layout.md) - [Memory Backing](./reference/architecture/openvmm/memory-backing.md) - [NUMA Topology](./reference/architecture/openvmm/numa.md) - [Device Tree](./reference/architecture/openvmm/device-tree.md) - [mesh](./reference/architecture/openvmm/mesh.md) - [Using mesh](./reference/architecture/openvmm/mesh/usage.md) - [How mesh works](./reference/architecture/openvmm/mesh/internals.md) - [inspect]() - [OpenHCL Architecture](./reference/architecture/openhcl.md) - [Processes and Components](./reference/architecture/openhcl/processes.md) - [`openvmm_hcl`](./reference/architecture/openhcl/openvmm_hcl.md) - [Attestation and VMGS Protection](./reference/architecture/openhcl/attestation.md) - [IGVM Attestation Protocol](./reference/architecture/openhcl/attestation_protocol.md) - [AK Certificate Renewal](./reference/architecture/openhcl/ak_cert.md) - [VMBus Relay and Device Interception](./reference/architecture/openhcl/vmbus.md) - [Storage Translation](./reference/architecture/openhcl/storage_translation.md) - [Storage Configuration Model](./reference/architecture/openhcl/storage_configuration.md) - [Boot Flow](./reference/architecture/openhcl/boot.md) - [Sidecar](./reference/architecture/openhcl/sidecar.md) - [CPU Scheduling](./reference/architecture/openhcl/cpu_scheduling.md) - [IGVM](./reference/architecture/openhcl/igvm.md) - [Device Architecture](./reference/architecture/devices.md) - [Storage Pipeline](./reference/architecture/devices/storage.md) --- [OpenVMM Rust Crate API Docs](./rustdoc.md) --- ### Guide Src Rustdoc (`Guide/src/rustdoc.md`) # OpenVMM Rust Crate API Docs Rust crate API documentation generated by `cargo doc` is available at the following URLs: - Linux x86 - - Windows x86 - --- ### Guide Src Dev Guide Contrib (`Guide/src/dev_guide/contrib.md`) # Contributing This chapter discusses various things developers should be aware of prior to submitting changes to the OpenVMM project. This includes both code-level [coding conventions](./contrib/code.md), as well as instructions on properly [submitting changes](./contrib/pr.md) to the OpenVMM GitHub repo, and expectations around [code reviews](./contrib/code_review.md). --- ### Guide Src Dev Guide Dev Tools (`Guide/src/dev_guide/dev_tools.md`) # Dev Tools / Utilities While most tasks in the OpenVMM repo can be accomplished directly via standard Rust tooling (i.e: `cargo run`, `cargo build`), there are many dev tasks that don't neatly fall under the `cargo` umbrella. e.g: - running code formatters / linters - orchestrating multi-stage, multi-component OpenHCL builds - running different kinds of test suites - building/downloading test images for VMM testing - setting up git hooks - etc... The following chapter discusses some of the various dev-facing tools / utilities you may encounter and/or find useful when working on OpenVMM. ## Rust-based Tooling As with many projects, OpenVMM initially took the simple approach of spinning up ad-hoc Bash/Python scripts, and hand-written YAML workflow automation. This worked for a while... but as the project continued to grow, our once small and focused set of scripts evolved into a mass of interconnected dependencies, magic strings, and global variables! To pay down mounting tech debt, and to foster a culture where all devs are empowered to contribute and maintain OpenVMM's project tooling, we have adopted a policy of migrating as much core tooling away from loosely-typed languages (like Bash, Python, and hand-written Workflow YAML), and towards new strongly-typed Rust-based tooling. --- ### Guide Src Dev Guide Getting Started (`Guide/src/dev_guide/getting_started.md`) # Getting Started (for Developers) This chapter discusses all the basic steps required to begin building code in the OpenVMM project. By the end of this chapter, you will have: - Cloned the OpenVMM git repo - Installed all required pre-build dependencies (e.g: Rust) - Built local copies of both OpenVMM and OpenHCL - (optionally) Set up a suggested VSCode-based development environment ## _Aside:_ What is HvLite? Underhill? As you explore the OpenVMM repo, you may find references to things called **HvLite** and **Underhill**. Simply put: - **OpenVMM** is synonymous with **HvLite** - **OpenHCL** is synonymous with **Underhill** HvLite and Underhill were former Microsoft-internal codenames for OpenVMM and OpenHCL. Migrating all existing code and documentation away from these codewords is not an overnight process, and it's quite likely these terms will linger in various code comments, variable names, library names, etc... the forseeable future. --- ### Guide Src Dev Guide Snapshot Format (`Guide/src/dev_guide/snapshot_format.md`) # Snapshot Format This page documents the on-disk format used by OpenVMM snapshots, intended for developers working on the save/restore subsystem. ## Directory layout A snapshot is stored as a directory containing three files: ```text snapshot-dir/ ├── manifest.bin # Protobuf-encoded SnapshotManifest ├── state.bin # Protobuf-encoded device saved state └── memory.bin # Hard link to the guest memory backing file ``` ## Manifest format The manifest is a protobuf message defined as [`SnapshotManifest`](https://openvmm.dev/rustdoc/linux/openvmm_helpers/snapshot/struct.SnapshotManifest.html) in `openvmm/openvmm_helpers/src/snapshot.rs`, encoded using the `mesh` crate's protobuf encoding. ## Device state (`state.bin`) The device state contains every device's saved state, collected via the `SaveRestore` trait and encoded as a `mesh` protobuf message. The [Save State](contrib/save-state.md) compatibility rules (mesh tag stability, default values, forward/backward compatibility) apply. ## Memory (`memory.bin`) `memory.bin` is a hard link to the file-backed guest RAM file. During a save, `write_snapshot()` creates this hard link using `std::fs::hard_link`. ```admonish note The hard-link approach means the memory backing file and snapshot directory must reside on the same filesystem. If they are on different filesystems, `write_snapshot` returns an error with a suggestion to place the backing file inside the snapshot directory. ``` ### Same-file detection If the user passes `--memory file=/memory.bin`, the source and target of the hard link are the same file. The code detects this by canonicalizing both paths and comparing them. When they match, the hard-link step is skipped. ## Code references - Manifest type and I/O: `openvmm/openvmm_helpers/src/snapshot.rs` - Restore entry point: `prepare_snapshot_restore()` in `openvmm/openvmm_entry/src/lib.rs` - File-backed memory: `SharedMemoryFd` type alias in `openvmm/openvmm_defs/src/worker.rs` ## Device state architecture Each VM component that participates in save/restore is registered as a "state unit" with a unique string name via `StateUnits::add("name")`. During save, every state unit receives a `StateRequest::Save`. Units that have state return `Ok(Some(blob))`; units with no persistent state (e.g. the input distributor) return `Ok(None)` and are omitted from `state.bin`. The resulting `state.bin` contains a `Vec`, where each entry pairs a unit name with its opaque protobuf-encoded state blob. ### Restore matching rules During restore, `StateUnits::restore()` matches saved-state entries to currently registered units **by name**: | Scenario | Result | |---|---| | Names match exactly | State is dispatched to the unit | | Saved entry has no matching unit | **Error** — `unknown unit name` | | Unit exists with no saved entry | Unit is skipped (keeps default state) | | Duplicate name in saved state | **Error** — `duplicate unit name` | This means removing a device between save and restore will fail, but adding a new device is allowed (it initialises to its power-on defaults). ### Unit naming conventions - **Chipset devices** — registered via `arc_mutex_device("name")` in `vmotherboard`, e.g. `"pit"`, `"rtc"`, `"uefi"`, `"ide"`. - **VMBus devices** — named `"{interface_name}:{instance_id}"`, e.g. `"StorageVsp:ba6163d9-..."`. The instance GUID makes each offer unique. - **Infrastructure units** — `"vmtime"`, `"input"`, `"vmbus"`. ### Devices that do not support save/restore Not all devices implement save/restore. Devices signal this in one of two ways: 1. **`SaveError::NotSupported`** — the `save()` method returns this error. If any state unit does this, the entire save operation fails. 2. **`supports_save_restore() -> false`** (virtio) or `supports_save_restore() -> None` (VMBus) — transport-level check that causes the transport's `save()` to return `SaveError::NotSupported`. Key unsupported categories: - **PCIe** — `GenericPcieRootComplex`, `GenericPcieSwitch` return `SaveError::NotSupported`. - **NVMe** — `NvmeController` returns `SaveError::NotSupported`. - **Pass-through PCI** — `AssignedPciDevice`, `RelayedVpciDevice`. - **VGA / GDMA** — marked `todo!()` (will panic on save). - **Virtio devices** — the `VirtioDevice` trait defaults `supports_save_restore()` to `false`. Only `virtio-blk`, `virtio-net`, `virtio-pmem`, and `virtio-rng` override it to `true`. Devices with host-side session state (`virtio-9p`, `virtiofs`, `virtio-console`) intentionally leave it `false`. - **Some VMBus devices** — `GuestCrashDevice`, `GuestEmulationDevice`, `VmbusSerialHost`, `Vmbfs` return `None` from `supports_save_restore()`. ## Extending the format When adding new fields to `SnapshotManifest`, use the next available mesh tag number. The protobuf encoding is forward-compatible: older readers will ignore unknown fields. However, removing or reordering existing fields is a breaking change. See [Save State](contrib/save-state.md) for the full set of compatibility rules. ```admonish warning Changing the mesh tag numbers of existing fields will break compatibility with previously saved snapshots. ``` --- ### Guide Src Dev Guide Tests (`Guide/src/dev_guide/tests.md`) # Testing This chapter discusses the various kinds of tests you will encounter as an OpenVMM developer. --- ### Guide Src Reference Architecture (`Guide/src/reference/architecture.md`) # Architecture This chapter discusses the architecture of various key parts of the OpenVMM codebase. [Timekeeping](./architecture/timekeeping.md) covers the clock domains, timers, and lifecycle behavior shared by OpenVMM and OpenHCL. --- ### Guide Src Reference Openhcl (`Guide/src/reference/openhcl.md`) # OpenHCL Features This section discuss various features that are exclusive to OpenHCL. --- ### Guide Src Reference Openvmm (`Guide/src/reference/openvmm.md`) # OpenVMM Features This section discuss various features that are exclusive to OpenVMM. --- ### Guide Src User Guide Openhcl (`Guide/src/user_guide/openhcl.md`) # OpenHCL OpenHCL is an execution environment which runs OpenVMM as a **paravisor**. Unlike in traditional virtualization, where a VMM runs in a privileged host/root partition and provides virtualization services to a unprivileged guest partition, the "paravisor" model enables a VMM to provide virtualization services from _within_ the guest partition itself. It can be considered a form of "virtual firmware", running at a higher privilege level than the primary guest OS. Paravisors are quite exciting, as they enable a wide variety of useful and novel virtualization scenarios! For example: at Microsoft, OpenHCL plays a key role in enabling several important Azure scenarios: - Enabling existing workloads to seamlessly leverage [Azure Boost] (Azure's next-generation hardware accelerator), without requiring any modifications to the guest VM image. - Enabling existing guest operating systems to run inside [Confidential VMs]. - Powering [Trusted Launch VMs] - VMs that support Secure Boot, and include a vTPM. * * * To learn more about OpenHCL's architecture, please refer to [OpenHCL Architecture](../reference/architecture/openhcl.md). ```admonish note _Note:_ As you explore the OpenVMM repo, you may find references to the term **Underhill**. Underhill was the former codename for OpenHCL, so whenever you see the term "Underhill", you can treat it as synonymous to "OpenHCL". We are actively migrating existing code and docs away from using the term "Underhill". ``` [VSM]: https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/vsm [Virtual Trust Levels]: https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/vsm [Azure Boost]: https://learn.microsoft.com/en-us/azure/azure-boost/overview [Confidential VMs]: https://azure.microsoft.com/en-us/solutions/confidential-compute [Trusted Launch VMs]: https://learn.microsoft.com/en-us/azure/virtual-machines/trusted-launch [TDX]: https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/overview.html [SEV-SNP]: https://www.amd.com/content/dam/amd/en/documents/epyc-business-docs/white-papers/SEV-SNP-strengthening-vm-isolation-with-integrity-protection-and-more.pdf --- ### Guide Src User Guide Openvmm Img (`Guide/src/user_guide/openvmm-img.md`) # openvmm-img `openvmm-img` creates, inspects, validates, and converts disk images using OpenVMM's native storage implementations. It is part of the OpenVMM software suite, but can be used independently of the virtual machine monitor. The tool currently supports VHDX image management and conversion between raw and VHDX images. Run it from the repository with Cargo: ```bash cargo run -p openvmm-img -- --help ``` ## Creating Images Create a dynamic VHDX image by naming it with a `.vhdx` extension or by specifying the format explicitly: ```bash cargo run -p openvmm-img -- create path/to/disk.vhdx --size 64G cargo run -p openvmm-img -- create path/to/disk --format vhdx --size 64G ``` Format-specific options are passed as comma-separated key-value pairs. Use `allocation=fixed` to provision all VHDX payload blocks during creation: ```bash cargo run -p openvmm-img -- create path/to/disk.vhdx --size 64G \ --format-options allocation=fixed ``` Use `--parent` to create a differencing image. The parent implies dynamic allocation, so it cannot be combined with `allocation=fixed`: ```bash cargo run -p openvmm-img -- create path/to/child.vhdx \ --size 64G --parent path/to/parent.vhdx ``` A differencing child inherits its parent's logical sector size unless one is specified explicitly. An explicit logical sector size must match the parent. The child and parent may have different virtual disk sizes. Run `format-options` without a format to list the supported formats, or pass a format to list its accepted options: ```bash cargo run -p openvmm-img -- format-options cargo run -p openvmm-img -- format-options vhdx ``` ## Inspecting and Validating Images Use `info` for image metadata, `map` for allocated virtual ranges, and `check` to validate VHDX metadata and its differencing parent chain: ```bash cargo run -p openvmm-img -- info path/to/disk.vhdx cargo run -p openvmm-img -- map path/to/disk.vhdx cargo run -p openvmm-img -- check path/to/disk.vhdx ``` Pass `--json` to `info` or `map` for machine-readable output. `check` exits with status 2 for an inconsistent image and status 1 for other failures. Use `replay` to repair an image with a dirty VHDX write-ahead log. Pass `--dry-run` to report whether replay is required without changing the image. ```bash cargo run -p openvmm-img -- replay path/to/disk.vhdx --dry-run cargo run -p openvmm-img -- replay path/to/disk.vhdx ``` ## Converting Images Convert between raw and VHDX images by specifying the output format: ```bash cargo run -p openvmm-img -- convert path/to/disk.raw \ --output path/to/disk.vhdx --output-format vhdx cargo run -p openvmm-img -- convert path/to/disk.vhdx \ --output path/to/disk.raw --output-format raw ``` VHDX output options use the same key-value syntax. For example, this creates a fixed VHDX with 4 MiB payload blocks: ```bash cargo run -p openvmm-img -- convert path/to/disk.raw \ --output path/to/disk.vhdx --output-format vhdx \ --format-options allocation=fixed,block_size=4M ``` The input format is inferred from `.raw`, `.img`, or `.vhdx`. Use `--input-format` when the input has another extension. Differencing VHDX inputs are not supported by `convert`. All-zero copy chunks are skipped so dynamic VHDX and raw outputs remain sparse when the host filesystem supports sparse files. --- ### Guide Src User Guide Openvmm (`Guide/src/user_guide/openvmm.md`) # OpenVMM OpenVMM can be configured to run as a conventional [hosted, or "type-2"](https://en.wikipedia.org/wiki/Hypervisor#Classification) Virtual Machine Monitor (VMM). At the moment, OpenVMM can be built and run on the following host platforms: | Host OS | Architecture | Virtualization API | | ------- | ------------- | --------------------------------- | | Windows | x64 / Aarch64 | WHP (Windows Hypervisor Platform) | | Linux | x64 / Aarch64 | KVM | | | x64 / Aarch64 | MSHV (Microsoft Hypervisor) | | macOS | Aarch64 | Hypervisor.framework | When compiled, OpenVMM consists of a single standalone `openvmm` / `openvmm.exe` executable.[^dlls] For practical launch commands, see [Running OpenVMM](./openvmm/run.md). ```admonish note As you explore the OpenVMM repo, you may find references to the term **HvLite**. HvLite was the former codename for OpenVMM, so whenever you see the term "HvLite", you can treat it as synonymous to "OpenVMM". We are actively migrating existing code and docs away from using the term "HvLite". ``` ## Notable Features This *non-exhaustive* list provides a broad overview of some notable features, devices, and scenarios OpenVMM currently supports. - Boot modes - UEFI - via [`microsoft/mu_msvm`](https://github.com/microsoft/mu_msvm) firmware - BIOS - via the [Hyper-V PCAT BIOS](../reference/devices/firmware/pcat_bios.md) firmware - Linux Direct Boot - Devices - Paravirtualized - [Virtio](https://wiki.osdev.org/Virtio) - virtio-fs - virtio-9p - virtio-net - virtio-pmem - [VMBus](https://docs.kernel.org/virt/hyperv/vmbus.html) - storvsp - netvsp - serial - framebuffer - keyboard / mouse - vpci - Direct Assigned (experimental, WHP only) - Emulated - vTPM - NVMe - Serial UARTs (both 16550, and PL011) - Legacy x86 - i440BX + PIIX4 chipset (PS/2 kbd/mouse, RTC, PIT, etc) - IDE HDD/Optical, Floppy - PCI - VGA graphics (experimental) - Device backends - Graphics / Mouse / Keyboard (VNC) - Serial (term, socket, tcp) - Storage (raw img, VHD/VHDx, Linux blockdev, HTTP) - Networking (various) - Management interfaces (evolving) - CLI - Interactive console - gRPC - ttrpc For more information on any / all of these features, see their corresponding pages under the **Reference** section of the OpenVMM Guide. ```admonish note title="Management interface compatibility" OpenVMM's management interfaces continue to evolve and may change between releases. Consult the documentation for your OpenVMM version, especially when upgrading. ``` [^dlls]: though, depending on the platform and compiled-in feature-set, some additional DLLs and/or system libraries may need to be installed. --- ### Guide Src Dev Guide Contrib Code (`Guide/src/dev_guide/contrib/code.md`) # Coding conventions One of our major goals with OpenVMM is to provide a high quality coding experience for contributors, starting first-and-foremost by having a consistent set of coding conventions in the project. [Do your part](https://youtu.be/-_7FaWnlhS4?t=10) and keep OpenVMM clean! ## `rustfmt` _Checked Automatically:_ **Yes** (via [`cargo xtask fmt --pass rustfmt`](../dev_tools/xtask.md)) OpenVMM source must be formatted using [rustfmt](https://github.com/rust-lang/rustfmt), which automatically and mechanically applies standard formatting to all the code. This eliminates time spent discussing or reviewing stylistic issues in pull requests. The CI will run `rustfmt --check` to enforce consistent formatting, and will fail if it notices any discrepancies. Unfortunately, `rustfmt` isn't infinitely customizable, and there are several rules that are must be manually enforced: - All lines must end with LF, not CRLF. - Top-level `use` imports should be non-nested. ```admonish note Some of these manually-enforced conventions were introduced late into OpenVMM's development, and there may be chunks of the codebase that do not adhere to these conventions. If you're working in a file and notice that it isn't following a certain convention, please take a moment to fix it! ``` Assuming you've followed the [suggested dev env setup](../getting_started/suggested_dev_env.md) and set up `rust-analyzer` to format-on-save, you should rarely have to think about formatting in `.rs` files. ## Custom Lints _Checked Automatically:_ **Yes** (via [`cargo xtask fmt --pass lints`](../dev_tools/xtask.md)) Custom lints are a set of miscellaneous code lints that are specific to OpenVMM, which are enforced using a custom in-house tool: - enforce the presence of the standard Microsoft copyright header - enforce in-repo crate names don't use '-' in their name (use '_' instead) - enforce Cargo.toml files don't include autogenerated "see more keys" comments - enforce Cargo.toml files don't contain author or version fields - enforce files end with a single trailing newline - deny usage of `#[repr(packed)]` (you want `#[repr(C, packed)]`) - justify usage of `cfg(target_arch = ...)` (use `guest_arch` instead!) - justify usage of `expect(unsafe_code)` with an UNSAFETY comment - and more! Some of these lints are self explanatory, whereas others are described in more detail elsewhere on this page. ## Unused `Cargo.toml` Dependencies _Checked Automatically:_ **Yes** (via [`cargo xtask fmt --pass lints`](../dev_tools/xtask.md)) We have an in-repo fork of [`cargo-machete`](https://github.com/bnjbvr/cargo-machete) that ensures `Cargo.toml` files only include dependencies that are actually being used. Avoiding unused dependencies makes it easier to reason about what a crate is doing just by looking at its dependencies, and also helps cut-down on incremental compile times. ## Formatting (`Cargo.toml`) _Checked Automatically:_ **No** When defining dependencies in `Cargo.toml` files, please organize dependencies into the following groups: - crate-specific "subcrates" - crates under vm/ - crates under vm/vmcore/ - crates under support/ - external dependencies The rationale here is that crates should be grouped according to how related to how _widely applicable_ they are. i.e: crates from crates.io and support are widely applicable outside of OpenVMM, whereas crates under vmcore/ only make sense within the context of OpenVMM, and crate-specific subcrates are - by definition - only applicable to the crate they are being imported from. Additionally, we make use of the [workspace dependencies](https://doc.rust-lang.org/nightly/cargo/reference/workspaces.html#the-dependencies-table) feature to ensure that all our dependencies stay in sync. This requires defining dependencies in your crate's `Cargo.toml` file and in the project's root `Cargo.toml`. So, for example: ```toml [package] name = "openvmm" [dependencies] # crate-specific subcrates openvmm_core.workspace = true ... # /vmcore vmcore.workspace = true ... # /vm/devices firmware_uefi_custom_vars.workspace = true storvsp.workspace = true ... # /support guid.workspace = true inspect.workspace = true inspect_proto.workspace = true ... # external dependencies anyhow.workspace = true cfg-if.workspace = true clap.workspace = true ... ``` ## Linting (via `clippy`) _Checked Automatically:_ **Yes** OpenVMM uses [`cargo clippy`](https://github.com/rust-lang/rust-clippy) to supplement rustc's built-in lints. Assuming you've followed the guide and set up `rust-analyzer` to [use clippy](../getting_started/suggested_dev_env.md#enabling-clippy), you should see clippy lints appear inline when working on Rust code. The CI runs `cargo clippy` on every crate in the repo prior to building the project, and will fast-fail if it catches any warnings / errors. ### Suppressing Lints In general, lints should be fixed by modifying the code to satisfy the lint. However, there are cases where a lint may need to be `expect`'d inline. In these cases, you _must_ provide a inline comment providing reasonable justification for the suppressed lint. e.g: ```rust,ignore // x86_64-unknown-linux-musl targets have a different type defn for // `libc::cmsghdr`, hence why these lints are being suppressed. #[expect(clippy::needless_update, clippy::useless_conversion)] libc::cmsghdr { cmsg_level: libc::SOL_SOCKET, cmsg_type: libc::SCM_RIGHTS, cmsg_len: (size_of::() + size_of_val(fds)) .try_into() .unwrap(), ..std::mem::zeroed() } ``` ### OpenVMM's `clippy` Configuration We stick fairly close to the default set of rustc / clippy lints, though there are some default lints that we've decided to disabled project wide, and other non-default lints which we've explicitly opted into. See the `[workspace.lints]` sections of OpenVMM's root [`Cargo.toml`](https://github.com/microsoft/openvmm?path=/Cargo.toml) for a list of globally enabled/disabled lints, along with justification as to why certain lints have been enabled/disabled. ## Unsafe Code Policy ```admonish danger When possible, try to avoid introducing new `unsafe` code! Before rolling your own `unsafe` code, check to see if a safe abstraction already exists, either in-tree, on crates.io\*, or in the standard library. \*subject to an unsafe-code audit ``` Rather than synthesizing our own unsafe code conventions, we follow the guidelines outlined in the following two resources: - [The Rust `unsafe` docs](https://doc.rust-lang.org/std/keyword.unsafe.html) - [The Fuchsia "Unsafe code in Rust" Guidelines](https://fuchsia.googlesource.com/fuchsia/+/refs/heads/main/docs/development/languages/rust/unsafe.md) In a nutshell: - `unsafe` functions are required to include `/// # Safety` documentation describing any preconditions the caller must uphold when calling the function. - `unsafe {}` blocks are required to include a `// SAFETY:` comment describing how the preconditions for calling the `unsafe` function(s) within the block are being satisfied. - `expect(unsafe_code)` annotations are required to include an `// UNSAFETY:` comment justifying why the code in question needs to use `unsafe`. This annotation must be placed at the module or crate level. These requirements are enforced by CI, and will cause the build to fail if required documentation is missing. Editing a file containing unsafe code will trigger CI to label your PR and warn reviewers that the PR touches unsafe code. This is to ensure that all unsafe code is audited for correctness by area experts. ## Uses of `cfg(target_arch = ...)` must be justified _Checked Automatically:_ **Yes** (via [`cargo xtask fmt --pass lints`](../dev_tools/xtask.md)) Unless you're working on something that's genuinely tied to the host's CPU architecture, you should use `cfg(guest_arch = ...)` instead of `cfg(target_arch = ...)`. OpenVMM is a multi-architecture VMM framework, capable of running running x64 guests on a x64 host, as well as Aarch64 guests on Aarch64 hosts (with the potential of adding additional platforms in the future). At the moment, OpenVMM requires that the host architecture and guest architecture match. That said, it's possible that at some point in the future, OpenVMM may also support _mismatched_ guest/host architectures, via an emulated CPU virtualization backend (akin to QEMU). Having an emulated CPU backend would enable OpenVMM to support such useful scenarios as: - running Arch64 guests on x86 machines - running x86 guests on ARM - running something exotic (e.g: RISC-V) on x86/ARM machines - ...assuming we had the bandwidth to implement + maintain something like that - running OpenVMM on systems without hardware-accelerated CPU virtualization enabled With these scenarios in mind, it would be short-sighted to rely entirely on `cfg(target_arch = ...)` to gate guest-facing arch-specific functionality, as it would inexorably tie the guest's arch to the host's arch, making any future initiatives to pry the two apart significantly more difficult! As such, the `OpenVMM` repo includes infrastructure to specify a custom, OpenVMM-specific `cfg(guest_arch = ...)` cfg parameter. By default, `cfg(guest_arch = ...)` will act the same way as `cfg(target_arch = ...)`, but it can be swapped to a different architecture by setting the `OPENVMM_GUEST_TARGET` env var at compile-time. There are very few reasons to use `cfg(target_arch = ...)` within the OpenVMM repo, and to enforce this rule, we have an in-house `xtask fmt --pass lints` check that lints each use of `cfg(target_arch = ...)` to include a "justification" for why it's being used. e.g: `cfg(target_arch = ...)` would be applicable when feature-gating a CPU intrinsic (such as CPUID, or a SIMD instruction), or when implementing a `*-sys` crate where the underlying C API/ABI varies between architectures. ...otherwise, use `cfg(guest_arch = ...)`! ## Avoid `Default` when using `zerocopy::FromZeros` _Checked Automatically:_ **No** The rule: - A type can `derive(Default)` **XOR** `derive(FromZeros)`. - A type that is `FromZeros` can also `impl Default`, but it must be a conscious, explicit choice, with justification (read: inline comment) as to why that particular default value was chosen. - N.B. `derive(IntoBytes)` or `derive(FromBytes)` imply `derive(FromZeros)`. That is: this convention applies to those types as well. The why: - The all-zero type is often not a semantically valid `Default` value for a type - There are plenty of types that don't have a `Default` value, but _do_ have a valid all-zero repr ### Additional context As per the Rust docs for [`Default::default`](https://doc.rust-lang.org/std/default/trait.Default.html#tymethod.default): ```admonish quote `fn default() -> Self` Returns the “default value” for a type. Default values are often some kind of initial value, identity value, or anything else that may make sense as a default. ``` Notably, **default should _not_ be used as some shorthand to "zero initialize" values!** For most non-trivial structs, the all-zero representation is _not_ a semantically valid `Default`! This is true in many contexts... but one that's particularly relevant in OpenVMM is that of FFI via C-style APIs and ABIs. In C, it's very common for types to undergo multi-stage instantiation, where they are initially allocated as all-zero, and then get "filled in" by some secondary init code. Notably: it's quite rare for that initial "all-zero" struct to be a valid instance of the type! #### Example: C FFI For example, a common pattern in C libraries might look something like: ```c struct Handle { uint16_t opaque_handle; } struct Handle handle = {0}; init_handle(&handle); update_handle(handle, options); do_thing(handle); ``` In this example: it would be an error to invoke `update_handle` or `do_thing` with an all-zero handle, as the type hadn't finished being fully initialized. If we wanted to use this library from Rust, a "naive" approach would be to do something like: ```rust,ignore #[repr(C)] #[derive(Default)] struct Handle { opaque_handle: u16, } let mut handle = Handle::default(); // BAD! unsafe { init_handle(&handle) }; unsafe { update_handle(handle, options) }; unsafe { do_thing(handle) }; ``` While this _technically_ works... using `Default` here is kinda bogus! After all - `Handle::default()` doesn't actually call `init_handle`, which means the value returned by `default()` doesn't match the "promise" of the trait! Namely: the returned value is not semantically valid yet! In many other Rust codebase, this "overloading" of `Default` to represent both semantically valid value _and_ all-zero values (in FFI) is par for the course, as once structs get more complicated, having a `derive` that is able to fully init a "uninitialized" struct in-memory is quite handy... In OpenVMM, we don't do this. Instead, we use a separate trait to init all-zero structs. **In OpenVMM, we use `FromZeros` and `FromZeros::new_zeroed()` to work with types that have valid all-zero representations, _without_ implying that those types also have valid all-zero _default_ values!** So, for the example above: ```rust,ignore #[repr(C)] #[derive(zerocopy::FromZeros)] struct Handle { opaque_handle: u16 } let mut handle = Handle::new_zeroed(); // GOOD! unsafe { init_handle(&handle) }; ``` Now, it's impossible for code elsewhere to obtain a `Handle` via `Handle::default`, and mistakenly forget to invoke `init_handle` on it. ...but if it so happens that we _do_ want a `Default` impl for `Handle`, we can do so by _manually_ implementing `derive(Default)` ourselves: ```rust,ignore // Default + FromZeros: `default` returns fully initialized handle impl Default for Handle { fn default() -> Handle { let mut handle = Handle::new_zeroed(); unsafe { init_handle(&handle) }; handle } } ``` ## Avoid Requiring `Debug` on Traits _Checked Automatically:_ **No** **TL;DR:** Don't do this: ```rust,ignore trait MyTrait: std::fmt::Debug ``` Implementations of the standard library's `Debug` trait can be surprisingly large, and the final binary size of OpenHCL and related binaries is a major concern for us. Unused implementations of this trait are usually removed during the optimization process (like all dead code), making this a non-issue. However when traits, and more specifically trait objects, are involved, the compiler has a much more difficult time proving that implementations are unused. This can result in large amounts of functionally dead code ending up in the final binaries. If you need to implement `Debug` for a struct containing such a trait object, you will need to do so manually, so that you can skip over that field. Moreover, it's usually not good form to leave tracing statements that log a struct's `Debug` representation in production. Prefer tracing just the fields you're interested in, and/or connecting objects to the `Inspect` graph. ## Crate Naming _Checked Automatically:_ **Yes** (via [`cargo xtask fmt --pass lints`](../dev_tools/xtask.md)) **Crates must be named with underscores, not dashes and underscores used in folder names.** - Bad: `my-cool-crate` - Good: `my_cool_crate` Rust does not allow dashes in imports, with any dashes getting replaced with underscores when used in the code. Avoiding dashes altogether makes it easier to grep for crate names, and makes things more consistent across the repo. This convention is enforced by CI **Do not name crates with the words "base, util, common" or other terms that are overly general.** For example, consider a crate that provides a common data structure used by multiple devices: - Bad: `devices_common` - Good: `range_map` Libraries that contain the following eventually become a mishmash of unrelated functionality that is located there for convenience. This [blog post](https://dave.cheney.net/2019/01/08/avoid-package-names-like-base-util-or-common) goes more in-depth as to why. Instead, name things based on what they logically provide, like functionality or data types. ## Release Gates Workflow _Triggered Manually:_ **Yes** (via GitHub labels) In addition to the standard PR gates that run in debug mode, OpenVMM also provides a "Release Gates" workflow that runs the same checks but compiled in release mode. This workflow takes significantly longer to run but can catch issues that only manifest in optimized builds. ### When to Use Release Gates The release gates workflow should be used when: - Making changes to performance-critical code paths - Modifying compiler flags or build configuration - Implementing low-level optimizations - Debugging issues that only appear in release builds - Before merging large refactoring changes ### How to Trigger Release Gates To run the release gates on your PR: 1. Ensure your PR is ready for review (not in draft mode) 2. Add the `release-ci-required` label to your PR 3. The workflow will automatically trigger and run all checks in release mode The workflow will only run when the specific label is present, so you have full control over when to use this more resource-intensive testing. ### Label Management Only repository maintainers can add labels to PRs. If you need release gates run on your PR, ask a maintainer to add the `release-ci-required` label for you. --- ### Guide Src Dev Guide Contrib Code Review (`Guide/src/dev_guide/contrib/code_review.md`) # Code Review Guidance ## The Code Review Process Although the OpenVMM repository requires a maintainer to sign off before a pull request can be merged, this does not mean non-maintainer reviews are not valuable. It is extremely important for the health of the project for people to review areas of code they are familiar with. For example, for a change that touches multiple components, somebody familiar with one component could leave a review that says, "The changes to component A look good, but I'm not familiar with the rest of the code." These types of reviews are still extremely useful and help move changes forward. Maintainers will also feel more confident merging changes once subject-area experts have weighed in. ## Merge Permissions Approval and merge access are separate. A maintainer must approve a pull request before it is merged. Contributors who need permission to merge pull requests should be added to the `microsoft/openvmm-write` GitHub team, which grants general write access to the repository. Contact an OpenVMM maintainer to request membership. Membership in the team allows a contributor to perform the merge after the required reviews and status checks have completed. It does not replace the maintainer approval requirement. ## Submitter's Responsibility If your patch or change isn’t gaining traction, it is your responsibility as the submitter to follow up and ensure the right people review your changes. This means sending polite reminders to potential reviewers. Do not rely on GitHub notifications for this because people may have them filtered or disabled. The OpenVMM repository has a [CODEOWNERS](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) file, which will automatically add teams to the review for some components. If you are unsure who the correct audience for the review is, please ask for guidance from maintainers or other contributors. --- ### Guide Src Dev Guide Contrib Crypto Backends (`Guide/src/dev_guide/contrib/crypto_backends.md`) # Crypto Backends The `crypto` crate (`support/crypto`) abstracts over several backend implementations of the cryptographic primitives OpenVMM and OpenHCL need. Exactly one backend must be selected per binary, chosen via Cargo features: | Feature | Backend | | ---------- | -------------------------------------------------------------------------------- | | `openssl` | OpenSSL (typical for Linux / OpenHCL). | | `symcrypt` | [SymCrypt](https://github.com/microsoft/SymCrypt). | | `rust` | Pure-Rust implementations from [RustCrypto](https://github.com/RustCrypto). | | `native` | Platform default — OpenSSL on Linux, BCrypt/CNG on Windows, Security.framework on macOS. | Selection happens in [`support/crypto/build.rs`](https://github.com/microsoft/openvmm/blob/main/support/crypto/build.rs), which emits a `cfg` (`openssl`, `symcrypt`, `rust`, or `native`) based on the enabled features. ## The "wrong number of backends" error Because Cargo unifies features across a workspace, building two binaries in the same `cargo` invocation that ask for *different* `crypto` backends will result in the `crypto` crate being compiled with *all* of those features enabled at once. Conversely, building a library crate that transitively depends on `crypto` without itself selecting a backend (e.g. running its unit tests) results in `crypto` being compiled with *zero* backends enabled. In either case there is no sensible single backend to pick. To keep workspace-wide `cargo check` and `cargo test` usable, the build script does not panic in this situation — it picks a placeholder backend so the crate still compiles. To still guarantee that a *shipping binary* has linked exactly one backend, each binary opts into a link-time check by invoking the `crypto::ensure_single_backend!()` macro (typically gated on `#[cfg(not(test))]`). The macro emits a `#[used]` reference to a symbol that is only defined by `crypto` when exactly one backend is selected. The result: - `cargo check --workspace` — succeeds. No linking happens. - `cargo test -p ` — succeeds, even when that crate transitively depends on `crypto` without selecting a backend. - `cargo build -p ` for a binary that invokes the macro — fails at link time if zero or multiple backends end up enabled, with: ```text rust-lld: error: undefined symbol: __openvmm_crypto_ensure_single_backend__enable_exactly_one__see_support_crypto ``` The symbol name *is* the diagnostic: the offending binary has the wrong number of `crypto` backend features enabled and needs to pick exactly one. ### Fixing it 1. Identify which binary you are building and which `crypto` features it (transitively) enables. `cargo tree -e features -p -i crypto` is usually the fastest way. 2. Either narrow your `cargo build` invocation (e.g. `-p ` instead of `--workspace`), or adjust the offending crate's `Cargo.toml` so it stops enabling a backend it shouldn't. 3. Remember that adding `features = ["native"]` (or any other backend) to a *library* crate's dependency on `crypto` will force that backend on every binary that links the library. Backends should normally be selected by binary crates only. --- ### Guide Src Dev Guide Contrib Guide (`Guide/src/dev_guide/contrib/guide.md`) # Updating this Guide We gladly welcome PRs that improve the quality of the OpenVMM guide! The OpenVMM Guide is written in Markdown, and rendered to HTML using [mdbook](https://github.com/rust-lang/mdBook). You can find the source-code of this Guide in the main OpenVMM GitHub repo, in the [`Guide/`](https://github.com/microsoft/openvmm/tree/main/Guide) folder. ## Guide conventions * Line Length: Wrap lines at 80 characters. ## Editing the Guide ### Small Changes For small changes, you can simply click the "Suggest an Edit" button in the top-right corner of any page to automatically open up a GitHub Edit page. ### Medium Changes For medium changes, we suggest cloning the repo locally, and previewing changes to Markdown in your editor (Visual Studio Code has good support for this). ### Large Changes For large changes, we suggest cloning the repo locally, and building a fully rendered copy of the Guide using `mdbook`. This is very useful when making changes that leverage mdbook preprocessors, such as using mermaid diagrams, or previewing admonishments. ```admonish info For example, the `mdbook-admonish` preprocessor is what powers this nice looking "Info" box! ``` Building the Guide locally is quite straightforward: 1. Install `mdbook` and the additional preprocessors we use locally: ```bash cargo install mdbook cargo install mdbook-admonish cargo install mdbook-mermaid ``` 2. Navigate into the `Guide/` directory, and run `mdbook`: ```bash cd Guide/ # must be run inside the `Guide/` directory! mdbook serve ``` 3. Navigate to the localhost URL in your web browser (typically `http://127.0.0.1:3000/`) ### Troubleshooting #### Running `mdbook serve` outside the `Guide/` directory **Error:** ```text 2024-10-29 16:26:22 [INFO] (mdbook::book): Book building has started error: manifest path `./mdbook-openvmm-shim/Cargo.toml` does not exist ``` **Solution:** Ensure you have changed your working-directory to the `Guide/` folder (e.g: via `cd Guide/`), and then run `mdbook serve`. #### Rust is not installed **Error:** ```text 2024-10-29 16:35:49 [INFO] (mdbook::book): Book building has started 2024-10-29 16:35:49 [WARN] (mdbook::preprocess::cmd): The command wasn't found, is the "admonish" preprocessor installed? 2024-10-29 16:35:49 [WARN] (mdbook::preprocess::cmd): Command: cargo run --quiet --manifest-path ./mdbook-openvmm-shim/Cargo.toml mdbook-admonish ``` **Solution:** The OpenVMM Guide hooks into a custom Rust utility called `mdbook-openvmm-shim`, which must be compiled in order for `mdbook` to successfully build the OpenVMM guide. Please ensure you have [installed Rust](../getting_started/linux.md#installing-rust). --- ### Guide Src Dev Guide Contrib Openvmm Packaging (`Guide/src/dev_guide/contrib/openvmm_packaging.md`) # Packaging OpenVMM for Linux This page describes the source inputs and build configuration OpenVMM provides for Linux distribution packages. ## Release inputs Releases are tagged `openvmm-v`. GitHub automatically generates the source archive for every release tag. The `.tar.gz` URL is: ```text https://github.com/microsoft/openvmm/archive/refs/tags/openvmm-v.tar.gz ``` OpenVMM also uploads one release asset: `openvmm--vendor.tar.gz`. `` is the canonical `[workspace.package] version` in the root `Cargo.toml`. GitHub derives the source archive root from the repository and tag names. For the release tag above, expect `openvmm-openvmm-v/`. Confirm the root against the downloaded archive before finalizing distribution setup macros. The source archive contains no `.git` directory. The workspace version is already in `Cargo.toml`, so a binary built from the extracted archive reports the plain workspace version. The source archive covers the entire tracked tree, including components a Linux package does not build. Build `-p openvmm` and package only that binary. The vendor archive contains exactly these top-level paths: ```text cargo_config vendor/ ``` `cargo_config` is the exact stdout of `cargo vendor --locked --versioned-dirs vendor`. The `vendor/` tree comes from the same invocation, so it includes both crates.io and locked git dependencies needed by the workspace. `cargo vendor` operates on the workspace, so the vendor tree includes dependencies not compiled by the OpenVMM Linux binary. ```admonish note GitHub guarantees stable extracted source contents when a tag continues to name the same commit, but its compression settings may change. Distribution lookaside caches should retain the exact downloaded source archive and vendor archive used by the package. ``` ## Verifying the release Confirm that the release tag points to the expected commit: ```bash git ls-remote https://github.com/microsoft/openvmm.git \ refs/tags/openvmm-v ``` Download the source archive and vendor archive once when updating the distribution package, record their digests in the distribution's source metadata, and retain those exact bytes in the distribution's lookaside cache. OpenVMM does not upload a checksum file, an OpenPGP signature, or a provenance attestation for either archive. ## Build requirements The distribution build requires: - the Rust toolchain required by the workspace; - a C compiler and linker; - glibc development headers; - Linux UAPI headers; - OpenSSL development headers; - `pkg-config`; - a Protocol Buffers compiler providing `protoc`. These map to the following distribution packages, which is what OpenVMM's release validation job installs: | Requirement | Debian / Ubuntu | Fedora | | --- | --- | --- | | C compiler and linker | `build-essential` | `gcc`, `binutils` | | Linux UAPI headers | `linux-libc-dev` | `kernel-headers` | | OpenSSL development headers | `libssl-dev` | `openssl-devel` | | `pkg-config` | `pkg-config` | `pkgconf-pkg-config` | | `protoc` | `protobuf-compiler` | `protobuf-compiler` | Building with the optional `tpm` feature additionally requires CMake, provided by the `cmake` package on both Debian / Ubuntu and Fedora. Do not use `cargo xflowey restore-packages` when building a distribution package. That command restores prebuilt native dependencies intended for repository development. ## Offline build configuration Extract the GitHub source archive, then extract the vendor archive into that source tree: ```bash tar -xzf openvmm-v.tar.gz cd openvmm-openvmm-v tar -xzf ../openvmm--vendor.tar.gz ``` Append the shipped `cargo_config` bytes to `.cargo/config.toml`. Do not overwrite the existing file: ```bash printf '\n' >> .cargo/config.toml cat cargo_config >> .cargo/config.toml ``` Then build the host `x86_64-unknown-linux-gnu` target with vendored dependencies: ```bash export PROTOC="$(command -v protoc)" export OPENSSL_NO_VENDOR=1 cargo build --release --locked --offline -p openvmm \ --target x86_64-unknown-linux-gnu ``` ```admonish warning The shipped `.cargo/config.toml` must keep its existing settings. If your packaging flow has already added a `[source]` table, merge the vendored-source configuration manually instead of overwriting the file. ``` The vendor tree naturally preserves license files shipped by upstream dependencies. OpenVMM does not add an aggregate license inventory to the vendor archive. Downstream tooling may generate feature-aware manifests from the extracted vendor tree when distribution policy requires them. ## Distribution integration Mapping GitHub's generated source archive onto a distribution's conventions is the packager's job, but the points below are the ones OpenVMM's layout affects directly. ### RPM GitHub's generated source archive is expected to unpack into `openvmm-openvmm-v%{version}` rather than `%{name}-%{version}`. Confirm the root directory, then pass it explicitly: ```spec %autosetup -n openvmm-openvmm-v%{version} ``` Fedora's [Rust packaging guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Rust/) additionally require, for a package like OpenVMM that ships an executable with statically linked Rust dependencies: - `%cargo_prep` in `%prep`, after the source and vendor archives are unpacked and the vendored source configuration is appended. - `%cargo_license_summary` (and `%cargo_license`) in `%build` after the build, because the package includes a binary target. Any feature flags passed to the build must also be passed to these macros, or the recorded licenses will not match what was compiled. - `%cargo_vendor_manifest` in `%build` when policy requires a generated manifest for the vendored sources. It writes `cargo-vendor.txt`, which can then be shipped as a `%license` file. - `%cargo_generate_buildrequires` in `%generate_buildrequires` except when building only from vendored dependencies. ### Debian Configure `debian/watch` for the GitHub-generated source archive and fetch the matching vendor archive as an additional source. See the [Debian watch documentation](https://wiki.debian.org/debian/watch) for the GitHub-specific patterns. Record the vendored crate licenses in `debian/copyright` for the same reason Fedora may generate a vendor manifest from the extracted tree. ```admonish note OpenVMM does not publish an OpenPGP signature or `SHA256SUMS` alongside either archive, so `uscan` signature verification cannot be enabled. Confirm the release tag and retain the exact downloaded archives used for the package. ``` ## Package identity The OpenVMM binary reports the upstream product version committed in the source tree. Record a distribution-specific package revision in the distribution package metadata rather than replacing the binary version. OpenVMM deliberately offers no environment variable or build flag that overrides the reported version. The version is chosen by a reviewed commit to the root `Cargo.toml`, which keeps a binary's self-reported identity tied to the source it was built from. ## Runtime dependencies Confirm the exact dependencies for the packaged executable with `readelf` or the distribution's automatic dependency generator. The expected shared libraries include glibc, OpenSSL (`libssl` and `libcrypto`), and `libgcc_s`. SQLite is compiled into the binary and does not add a shared runtime dependency. --- ### Guide Src Dev Guide Contrib Openvmm Source Release (`Guide/src/dev_guide/contrib/openvmm_source_release.md`) # Cutting an OpenVMM source release This page explains how maintainers create an OpenVMM source release on GitHub. For the packager's view of the shipped inputs, see [Packaging OpenVMM for Linux](./openvmm_packaging.md). ## What a release is An OpenVMM source release is a GitHub release tagged `openvmm-v`. GitHub automatically provides the source archive for that tag. OpenVMM uploads exactly one additional asset: `openvmm--vendor.tar.gz`. That vendor archive contains the vendored Cargo dependency tree and the exact `cargo vendor` source-replacement snippet needed for offline builds. No custom source archive is assembled or uploaded. A release publishes source. Prebuilt binaries, and any commitment to service a published version, are outside this process. `` is the `version` field under `[workspace.package]` in the OpenVMM repository's root `Cargo.toml`. Nothing is stamped into the source archive, and the tree is never rewritten, so the version a release publishes is the version that was already reviewed and merged. ## Selecting the version Open an ordinary pull request that sets that field to the version being released, and merge it. That review *is* the decision to release that version; the release workflow only ever publishes what the tree already says. If the committed version has never been released and is already the intended value, the pull request may instead state explicitly that it selects that existing version. A no-op edit to `Cargo.toml` is not required. ```admonish note The version stays at the released value after publication. Commits made afterwards report `+g` and are identifiable by the appended commit, so there is no second commit to "reopen" the version. ``` ## Running the workflow Dispatch the **OpenVMM Source Release** workflow against the commit to release. It has no automatic triggers; it only runs when someone asks for it. The workflow: 1. reads `[workspace.package] version` and the commit from the checkout; 2. assembles `openvmm--vendor.tar.gz` once with `cargo vendor --locked --versioned-dirs`; 3. checks out the same revision, appends the generated `cargo_config` to `.cargo/config.toml`, and builds `openvmm` with `--locked --offline`; 4. creates or verifies `openvmm-v` at that commit, after confirming no release already exists for it; 5. creates a **draft** release for that tag, relies on GitHub's automatic source archive, and uploads only the vendor archive. The workflow does not upload `SHA256SUMS` or a provenance attestation. ## Publishing Before publishing, confirm the draft still targets the commit the workflow pinned. The workflow logs that commit when it creates `openvmm-v`, and `git ls-remote origin refs/tags/openvmm-v` reports where the tag points now. They must match. That check is the one thing publication cannot do for you. A draft release tracks a tag *name*, not a commit, and GitHub enforces immutability only after publication. Creating the tag up front means publication cannot invent a tag at some other commit, but nothing in the workflow prevents someone from moving an existing tag while the draft sits in review. Then review the draft, write its notes, and click **Publish release**. ```admonish warning The tag is publicly visible while the release is still a draft. Do not move or delete it during review. Publishing is the irreversible step: a published tag is not moved or deleted. Correcting a release means merging a pull request that selects a new patch version and running the workflow again. ``` The workflow fails rather than overwriting an existing release for the same tag, since that release may already have been reviewed or published. It checks for that release before creating the tag, so a refused run does not leave a tag behind. If a run must be retried before publication, it reuses the tag only when the tag still names the exact pinned commit. A maintainer may delete the draft and keep the matching tag. Deleting the tag is reserved for exceptional pre-publication cleanup. ## Limitations The workflow validates one Linux `openvmm` build with the uploaded vendor archive. Distribution-specific packaging steps still belong to the downstream package. ```admonish note OpenVMM does not publish an OpenPGP signature, `SHA256SUMS`, or a provenance attestation for either the GitHub-generated source archive or the uploaded vendor archive. Consumers can confirm that the release tag names the expected commit and retain the exact downloaded bytes they package. ``` --- ### Guide Src Dev Guide Contrib Pr (`Guide/src/dev_guide/contrib/pr.md`) # Submitting Changes Follow the [CONTRIBUTING](https://github.com/microsoft/openvmm/blob/main/CONTRIBUTE.MD) guide. --- ### Guide Src Dev Guide Contrib Release (`Guide/src/dev_guide/contrib/release.md`) # Release Management The OpenVMM repository periodically creates releases from `main`, typically about once a month. Releases are named for the year and month in which they were forked, using `YYMM`. For example, a release forked in September 2026 is release `2609`. Each monthly release is represented by a branch named `release/`, such as `release/2609`. The branch remains available as the release baseline and receives backported fixes if that release needs stabilization or servicing. The repository produces both the OpenVMM VMM and the OpenHCL paravisor. OpenVMM and OpenHCL retain their own product versions; the `YYMM` release identifies their repository baseline. OpenHCL currently ships about twice a year and selects one of the repository releases as the baseline for each product release. Before publishing an OpenVMM source release from `main` or a release branch, ensure that the workspace version is the intended SemVer and has not previously been published. If the current version has already been published, update it in a normal pull request. A new feature release from `main` typically increments the minor version, such as `1.1.0` to `1.2.0`. A servicing release from a release branch increments the patch version, such as `1.1.0` to `1.1.1`. An already committed, unpublished version can be selected without changing `Cargo.toml`. The same workspace version may exist in multiple branches during development, but it can be published as an `openvmm-v` release only once. Continuing development or creating a release branch does not by itself require a version bump. See [Cutting an OpenVMM source release](openvmm_source_release.md) for the publication process. We expect a high quality bar for all code that goes into the OpenVMM main branch, and we ask developers to hold release branches to the highest quality standards. The OpenVMM maintainers will gradually slow the rate of churn into these branches as we get closer to a shipping date. ```admonish note title="See also" [Security Releases](security_releases.md) describes private reporting and coordinated disclosure for fixes that cannot be developed publicly. ``` ```admonish note Some existing release branches use the older `release/..` format, such as `release/1.8.2607`. ``` This process should not impact your typical workflow; all new work should go into the `main` branch. But, to ease the cherry-picks, we may ask that you hold off from making breaking or large refactoring changes at points in this process. ## Marking, Approval Process, Code Flow The OpenVMM maintainers will publish various dates for the upcoming releases. Currently, these dates are driven by a Microsoft-internal process and can, and do, often change. Microsoft does not mean to convey any new product launches by choices of these dates. Releases naturally fall into several phases: | Phase | Meaning | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | Active Development | Regular development phase where new features and fixes are added. | | Stabilization | Phase focused on stabilizing the release by fixing bugs. | | Ask Mode | Only critical fixes are allowed; changes are scrutinized. No new features. This is the last phase before a release is closed. | | Servicing | Only essential fixes are made to support the release (a.k.a. maintenance mode). | | Out of service | A previous release which is no longer receiving updates. | By default, a monthly release branch receives short-term servicing until the next monthly release supersedes it. Branches selected for an OpenHCL product release receive extended servicing. For OpenHCL, the plan of record is to service the current product release and the two preceding product releases. Because OpenHCL currently releases about twice a year, this is approximately 18 months of servicing. The release-count policy is authoritative if the release cadence changes. When a new OpenHCL release ships, the release that is now three product releases behind moves out of service. This servicing policy applies prospectively to releases created under the new model. Existing release branches retain the lifecycle states listed below. ### Backport acceptance bar Release branches accept security fixes and critical bug fixes only. A critical bug fix addresses one of the following: - A crash, data corruption, or data loss. - A severe reliability or availability regression. - A critical compatibility or servicing failure. - An issue that prevents a supported scenario from functioning. New features, routine performance improvements, cleanup, and refactoring are not accepted. A narrowly scoped prerequisite for an approved fix may be considered with the same level of scrutiny. Changes must land in `main` first and are selectively backported after assessing their risk to the serviced release. ### Release branch process In the label and tooling examples below, `` is the suffix of the release branch. For `release/2609`, `` is `2609`. When creating a monthly release branch: 1. Create the `release_`, `backport_`, and `backported_` labels in GitHub. 2. Add the `release_` base-branch rule to `.github/labeler.yml` so pull requests targeting the branch are labeled consistently. 3. Add the branch to both `run_filters` and `test_filters` in `petri/logview/src/branch_quick_filters.tsx` so it appears in the Runs, Tests, and TestDetails quick filters. We track the state of candidates for a given release by tagging the PRs with the following labels: * `backport_`: This PR (to `main`) is a candidate to be included in the release. * N.B.: A maintainer will _remove_ this tag if the fix is not accepted into the release. * `backported_`: This PR (to `main`) has been cherry-picked to the release branch. The [`repo_support/relabel_backported.py`](https://github.com/microsoft/openvmm/blob/main/repo_support/relabel_backported.py) script can be used to automatically transition PRs from `backport_` to `backported_` once they have been cherry-picked to the release branch. #### Seeking Approval for Backport To seek approval to include a change in a release branch, follow these steps: 1. Tag your PR to `main` with the `backport_` label. 2. Wait for the PR to be merged to `main`. 3. Cherry-pick the change to the appropriate release branch in your fork and stage a PR to that same branch in the main repository. Please reach out to the maintainers before staging that PR if you have any doubts. #### Backport PR Best Practices When creating a backport PR to a release branch: * **Clean cherry-picks are strongly preferred.** A clean cherry-pick minimizes reviewer effort and reduces the risk of introducing regressions. * **If the backport is not a clean cherry-pick** (e.g., requires conflict resolution or additional modifications), clearly indicate this in the PR description. This signals to the reviewer that extra care is needed during the review process. ## Existing Release Branches | Release | Phase | Notes | | ---------------- | ------------------ | ------------------------------------------------------------------------------------ | | release/2411 | Out of service | | | release/2505 | Out of service | Supports runtime servicing from release/2411. | | release/1.7.2511 | Servicing | Supports runtime servicing from release/2411 and release/2505. | | release/1.8.2607 | Ask Mode | Supports runtime servicing from release/2411, release/2505, and release/1.7.2511. | | _tbd, in main_ | Active Development | Supports runtime servicing from release/2411, release/2505, and release/1.7.2511. | ## Taking a Dependency on a Release We welcome feedback, especially if you would like to depend on a reliable release process. Please reach out! --- ### Guide Src Dev Guide Contrib Save State (`Guide/src/dev_guide/contrib/save-state.md`) # Save State OpenHCL supports the mechanism of saving & restoring state. This primitive can be used for various VM operations, but a key use case is for updating OpenHCL at runtime (a.k.a. "Servicing"). This save state can be stored in memory or on durable media, read back at a later time. ## Save State First Principles Here are the principles you must maintain when adding new save & restore code: 1. **Save & Restore is Forward & Backwards Compatible**: A newer version of OpenHCL must understand save state from a prior version, and an older version must not crash when reading save state from a newer version. 2. **Do not break save state after that state is in use**: Save state must be compatible from *any* commit to any other commit, once the product has shipped and started using that save state.[^1] 3. **All Save State is Protocol Buffers**: All save state is encoded as `ProtoBuf`, using `mesh`. ## Best practices 1. **Put save state in it's own module**: This makes PR reviews easier, to catch any mistakes updating save state. 2. **Create a unique package per crate**: A logical grouping of saved state should have the same `package` 3. **Avoid unsupported types, when possible**: These types don't support the default values needed for safely extending save state: * Arrays: if you need to add an array, consider a `vec` or `Option<[T; N]>` instead. * Enum: if you need to add an enum, add it as `Option` instead. 4. **Be particularly careful when updating saved state**: See below. ### Updating (Extending) Saved State Since saved state is just Protocol Buffers, use the [guide to updating Protocol Buffers messages](https://protobuf.dev/programming-guides/proto3/#updating) as a starting point, with the following caveats: 1. OpenVMM uses `sint32` to represent the `i32` native type. Therefore, changing `i32` to `u32` is a breaking change, for example. 2. The Protocol Buffers docs mention what happens for newly added fields, but it bears adding some nuance here: 1. Plain `arrays` and `enums` are not supported. Reading newer save state with an `array` or `enum` will fail on an older build. Instead, wrap these in an `Option`. 2. Old -> New Save State: Save state from a prior revision will not contain some newly added fields. Those fields will get the [default values](https://protobuf.dev/programming-guides/proto3/#default). This is how that breaks down for the rust types: * `Option` => `None` * Structs => each field gets that field's default value * Vecs => empty vec * Numbers => 0 * Strings => `""` 3. New -> Old Save State: Unknown fields are ignored. 3. Ensure that default values for new saved state fields make semantic sense. Here is the heuristic: if you need a conditional to check if the restored value is the default because it was not included in save state, you should use an `Option`. If you wouldn't otherwise need a conditional, you should not use an `Option`. 4. Define your saved state so that the natural default of `bool` types is `false`. ## Defining Saved State Saved state is defined as a `struct` that has `#[derive(Protobuf)]` and `#[mesh(package = "package_name")]` attributes. Here is an example, taken from the `nvme_driver`: ```rust,ignore pub mod save_restore { use super::*; /// Save/restore state for IoQueue. #[derive(Protobuf, Clone, Debug)] #[mesh(package = "nvme_driver")] pub struct IoQueueSavedState { #[mesh(1)] /// Which CPU handles requests. pub cpu: u32, #[mesh(2)] /// Interrupt vector (MSI-X) pub iv: u32, #[mesh(3)] pub queue_data: QueuePairSavedState, } } ``` [^1]: Saved state is in use when it reaches a release branch that is in tell mode. See [release management](./release.md) for details. --- ### Guide Src Dev Guide Contrib Security Releases (`Guide/src/dev_guide/contrib/security_releases.md`) # Security Releases This page describes how OpenVMM coordinates fixes that cannot be developed in public before disclosure. ## Report vulnerabilities privately Report potential vulnerabilities through the [Microsoft Security Response Center](https://aka.ms/SECURITY.md). Do not open a public issue, pull request, or discussion for a suspected vulnerability. MSRC coordinates severity assessment, CVE handling, disclosure timing, and communication with affected parties. Not every security fix requires an embargo. When public development would not expose users to additional risk, the maintainers may use the normal public contribution process. ## Embargoed fix development When an embargo is warranted, a small need-to-know incident team develops and reviews the fix in a restricted workspace containing the public OpenVMM source and the embargoed changes. The workspace must not introduce proprietary source or dependencies into the eventual public change. The team prepares the public patch, tests, release notes, advisory, and CVE material before the coordinated disclosure time. Embargoes should be no longer than necessary to produce and validate fixes and give affected downstreams a reasonable opportunity to prepare. ## Coordinating with downstreams MSRC may authorize advance sharing with qualified downstream security teams when they need time to validate or integrate a fix. Advance sharing is case-by-case, not a general early-access program. An advance package should identify the exact public base commit and affected versions, include a reviewable patch series and its hashes, and provide build, test, disclosure, and handling instructions. Recipients must keep the material need-to-know, use approved secure channels, and must not redistribute the patches or patched artifacts before disclosure unless MSRC explicitly authorizes it. An operator may be authorized to deploy early to a hosted service when users cannot retrieve the patched software or infer the vulnerability from the deployment. This also requires explicit incident-specific approval. ```admonish warning Receiving an embargoed patch does not grant permission to publish it, ship a patched product, or disclose the vulnerability before the coordinated time. The instructions supplied with the incident are authoritative. ``` ## Coordinated disclosure At the agreed disclosure time, the incident team coordinates: 1. Publishing the fix to each affected public branch. 2. Publishing a new patch-version OpenVMM source release. 3. Publishing the security advisory and CVE information. 4. Notifying coordinated downstreams that the embargo has lifted. The source release, advisory, and fix should become public in the same release window. Detailed exploit material may be delayed when MSRC determines that doing so reduces risk without preventing users from understanding and applying the fix. Support for older OpenVMM releases is decided per incident. A fix normally targets supported releases; an affected unsupported release does not automatically become supported. ## After disclosure After the public fix is available, development and review continue through the normal public process. Embargo recipients should replace private patches with the public fix when practical and securely delete pre-disclosure material when their incident instructions require it. --- ### Guide Src Dev Guide Contrib Style Guide (`Guide/src/dev_guide/contrib/style_guide.md`) # Style Guide This page defines the writing conventions for the OpenVMM developer guide. Follow these standards when creating or editing any page. ## Voice and tone Write as if explaining to a smart colleague who is new to the project. Use second person ("you") for instructions: "To build OpenHCL, run..." not "One can build OpenHCL by running..." State facts. Avoid hedging ("I believe", "some reason", "maybe we can"). If something is uncertain, say "As of [date], verify that..." ## Page structure Open every page with a one-sentence summary of what it covers. Use heading levels consistently: `# Page Title`, `## Major Section`, `### Subsection`. Keep pages under 400 lines. If a page grows beyond that, consider splitting it into sub-pages. ## Callouts This guide uses [mdbook-admonish](https://github.com/tommilligan/mdbook-admonish) for callouts. Use fenced code block syntax with the `admonish` keyword: ~~~markdown ```admonish note Supplementary information the reader should be aware of. ``` ```admonish tip Helpful suggestions or shortcuts. ``` ```admonish warning Potential pitfalls or easy mistakes. ``` ```admonish danger Actions that could cause data loss or security issues. ``` ~~~ You can add a custom title with the `title` attribute: ~~~markdown ```admonish warning title="Breaking change" This API was removed in v1.8. ``` ~~~ For cross-links to related pages, use a **See also** callout: ~~~markdown ```admonish note title="See also" [Debugging OpenHCL](../../reference/openhcl/debugging.md) for serial logs, crash dumps, and diagnostic tools. ``` ~~~ ## Code blocks ### Shell labeling Always specify the language. Never leave a code fence unlabeled. | Label | When | |-------|------| | `bash` | Linux / WSL / macOS shell commands | | `powershell` | PowerShell commands and scripts | | `rust` | Rust source code | | `toml` | TOML configuration | | `json` | JSON data | | `text` | Plain text output, logs, or paths | ### Placeholders Two conventions are used for placeholder values. Use whichever is clearest in context: - **`path/to/...` style** — for file paths where the structure is obvious. This is the predominant style in the existing guide. ```bash cargo run -- --uefi --vmbus-scsi id=scsi0 --disk memdiff:file:path/to/disk.vhdx,on=scsi0 ``` - **`` style** — for values that need extra emphasis or where the expected format isn't obvious (build numbers, VM names, keys). ```powershell Set-VMComPort -VMName -Number 3 -Path \\.\pipe\ ``` Bad (hardcoded path that won't work for the reader): ~~~bash cargo run -- --uefi --vmbus-scsi id=scsi0 --disk memdiff:file:/home/alice/disks/myvm.vhdx,on=scsi0 ~~~ ### File paths in code examples Use **forward slashes** (Unix-style) for file paths in `bash` code blocks, even when the path refers to a Windows filesystem location accessed via WSL: ```bash # Good — forward slashes in bash cargo run -- --vmbus-scsi id=scsi0 --disk memdiff:file:path/to/disk.vhdx,on=scsi0 # Good — wslpath output uses backslashes, but that's a Windows path cargo run -- --vmbus-scsi id=scsi0 --disk "memdiff:file:$(wslpath -w /mnt/c/vhds/disk.vhdx),on=scsi0" ``` Use **backslashes** for paths in `powershell` code blocks: ```powershell # Good — backslashes in PowerShell cargo run -- --vmbus-scsi id=scsi0 --disk memdiff:file:C:\vhds\disk.vhdx,on=scsi0 ``` ### Length Keep code blocks under 30 lines. If longer, split with explanatory text between blocks. Diagrams (ASCII art in `text` fences) are exempt from this limit — keep them as a single block so the visual structure isn't broken. Comments inside code blocks should explain *why*, not *what*. ## Line wrapping Wrap prose lines at approximately 80 characters. This keeps diffs readable and makes review easier. Lines inside tables and code blocks are exempt. ## Links Use relative links to other guide pages. When linking to external documentation, prefer stable URLs (Microsoft Learn, GitHub, official docs) over internal wikis or SharePoint. --- ### Guide Src Dev Guide Dev Tools Flowey (`Guide/src/dev_guide/dev_tools/flowey.md`) # Flowey Flowey is an in-house, Rust library for writing maintainable, cross-platform automation. It enables developers to define CI/CD pipelines and local workflows as type-safe Rust code that can generate backend-specific YAML (Azure DevOps, GitHub Actions) or execute directly on a local machine. Rather than writing automation logic in YAML with implicit dependencies, flowey treats automation as first-class Rust code with explicit, typed dependencies tracked through a directed acyclic graph (DAG). ## Why Flowey? Traditional CI/CD pipelines using YAML-based configuration (e.g., Azure DevOps Pipelines, GitHub Actions workflows) have several fundamental limitations that become increasingly problematic as projects grow in complexity: ### The Problems with Traditional YAML Pipelines #### Non-Local Reasoning and Global State - YAML pipelines heavily rely on global state and implicit dependencies (environment variables, file system state, installed tools) - Understanding what a step does often requires mentally tracking state mutations across the entire pipeline - Debugging requires reasoning about the entire pipeline context rather than isolated units of work - Changes in one part of the pipeline can have unexpected effects in distant, seemingly unrelated parts #### Maintainability Challenges - YAML lacks type safety, making it easy to introduce subtle bugs (typos in variable names, incorrect data types, etc.) - No compile-time validation means errors only surface at runtime - Refactoring is risky and error-prone without automated tools to catch breaking changes - Code duplication is common because YAML lacks good abstraction mechanisms - Testing pipeline logic requires actually running the pipeline, making iteration slow and expensive #### Platform Lock-In - Pipelines are tightly coupled to their specific CI backend (ADO, GitHub Actions, etc.) - Multi-platform support means maintaining multiple, divergent YAML files #### Local Development Gaps - Developers can't easily test pipeline changes before pushing to CI - Reproducing CI failures locally is difficult or impossible - The feedback loop is slow: push → wait for CI → debug → repeat ### Flowey's Solution Flowey addresses these issues by treating automation as **first-class Rust code**: - **Type Safety**: Rust's type system catches errors at compile-time rather than runtime - **Local Reasoning**: Dependencies are explicit through typed variables, not implicit through global state - **Portability**: Write once, generate YAML for any backend (ADO, GitHub Actions, or run locally) - **Reusability**: Nodes are composable building blocks that can be shared across pipelines ## Flowey's Directory Structure Flowey is architected as a standalone tool with a layered crate structure that separates project-agnostic core functionality from project-specific implementations: - **`flowey_core`**: Provides the core types and traits shared between user-facing and internal Flowey code, such as the essential abstractions for nodes and pipelines. - **`flowey`**: Thin wrapper around `flowey_core` that exposes the public API for defining nodes and pipelines. - **`flowey_cli`**: Command-line interface for running flowey - handles YAML generation, local execution, and pipeline orchestration. - **`schema_ado_yaml`**: Rust types for Azure DevOps YAML schemas used during pipeline generation. - **`flowey_lib_common`**: Ecosystem-wide reusable nodes (installing Rust, running Cargo, downloading tools, etc.) that could be useful across projects outside of OpenVMM. - **`flowey_lib_hvlite`**: OpenVMM-specific nodes and workflows that build on the common library primitives. - **`flowey_hvlite`**: The OpenVMM pipeline definitions that compose nodes from the libraries above into complete CI/CD workflows. --- ### Guide Src Dev Guide Dev Tools Guest Test Uefi (`Guide/src/dev_guide/dev_tools/guest_test_uefi.md`) # guest_test_uefi `guest_test_uefi` is a minimal `no_std` + `alloc` UEFI application for OpenVMM-specific bare-metal guest tests. The application runs without a general-purpose guest OS, so tests can exercise firmware variables, watchdog behavior, MMIO, and port I/O with little intervening software. ## Runtime and action selection Firmware launches the EFI executable from a boot disk. On startup, the application reads the `PetriBootAction` UEFI variable from the global variable namespace. The current actions are: | Value | Behavior | | --- | --- | | Missing, unreadable, or unknown | Run the normal UEFI test suite | | `hibernate` | Request platform hibernation and do not return | Petri seeds this variable through the VM's UEFI NVRAM state when a test needs a non-default action. The default makes a manually booted image run its ordinary test suite without additional configuration. ## Current tests The default suite runs tests in a fixed order: 1. Allocate and print a heap-backed string, validating global allocation. 2. Read the Secure Boot `db` and `dbDefault` variables and compare their data. 3. Arm the UEFI watchdog for five seconds and wait long enough for it to reset the VM. The watchdog test runs last because success destroys the current VM execution. A write-protection test for `dbDefault` exists but is disabled until the firmware behavior it depends on is available. The application draws changing splash values between tests and writes progress to the UEFI console. Capture both console and VM lifecycle events when diagnosing a failure. ## Building and running `guest_test_uefi` must be built for `*-unknown-uefi` targets. These are not installed by default, so you'll need to install the correct target via `rustup`. For example: ```bash rustup target add x86_64-unknown-uefi ``` Since this code runs in the guest, package the built `.efi` binary into a disk image that UEFI can read. To streamline the process of obtaining such a disk image, `cargo xtask` includes a helper to generate properly formatted `.img` files containing a given `.efi` image. e.g: ```bash # build the UEFI test application cargo build -p guest_test_uefi --target x86_64-unknown-uefi # create the disk image cargo xtask guest-test uefi \ --bootx64 target/x86_64-unknown-uefi/debug/guest_test_uefi.efi # test in OpenVMM cargo run -- --uefi --gfx --hv --processors 1 \ --vmbus-scsi id=scsi0 \ --disk memdiff:target/x86_64-unknown-uefi/debug/guest_test_uefi.img,on=scsi0 ``` The output is a generic UEFI boot image and can also run under Hyper-V, QEMU, or another VMM with compatible UEFI firmware. ```admonish note When launching a prebuilt `openvmm` executable instead of `cargo run`, provide the UEFI firmware path explicitly. The Cargo environment normally supplies it from restored packages. ``` ## Selecting hibernation in Petri Petri tests should use the guest-action seeding helper rather than manually editing the VMGS. The helper writes `PetriBootAction=hibernate` into the custom UEFI NVRAM delta before boot and then observes the VM's hibernation behavior. Manual launches run the default suite unless you prepare equivalent NVRAM state. ## Converting the image To convert the raw `.img` into other formats, `qemu-img` is very helpful: ```bash # Adjust for the target architecture and build profile. IMAGE_DIR=target/x86_64-unknown-uefi/debug # VMware qemu-img convert -f raw -O vmdk \ ${IMAGE_DIR}/guest_test_uefi.img ${IMAGE_DIR}/guest_test_uefi.vmdk # Hyper-V qemu-img convert -f raw -O vhdx \ ${IMAGE_DIR}/guest_test_uefi.img ${IMAGE_DIR}/guest_test_uefi.vhdx ``` ## Troubleshooting - If firmware does not find the application, verify the target architecture and the `--bootx64` or corresponding image-builder input. - If `db` or `dbDefault` is missing, use firmware containing the expected Secure Boot variable defaults. - If the watchdog test returns and panics, the firmware did not honor the watchdog reset request. - If Petri performs the wrong action, inspect the custom UEFI NVRAM delta and confirm the variable name, namespace, and UTF-8 value. - If no output appears, attach a graphical or serial console supported by the selected firmware and inspect early firmware boot logs. --- ### Guide Src Dev Guide Dev Tools Hypestv (`Guide/src/dev_guide/dev_tools/hypestv.md`) # hypestv `hypestv` is an interactive command-line interface for Hyper-V VMs, designed for making OpenHCL developers' lives easier. Similar to [`ohcldiag-dev`][], it can interact with the OpenHCL paravisor running inside a Hyper-V VM. But unlike `ohcldiag-dev`, it sports an interactive terminal interface (with history and tab completion), and it is specifically designed to interact with Hyper-VMs. [`ohcldiag-dev`]: ../../reference/openhcl/diag/ohcldiag_dev.md ## Platform and build Hypestv runs on Windows and controls VMs registered with the local Hyper-V service. It is not a remote management client and has no non-Windows support. Build it from a Windows checkout or a configured WSL cross environment: ```powershell cargo build -p hypestv ``` Start detached or select an initial VM by name: ```powershell .\target\debug\hypestv.exe .\target\debug\hypestv.exe ``` The interface is deliberately interactive, it is not intended for use in automation. In many ways, it is similar to the OpenVMM interactive console. In time, it may end up sharing code and capabilities with it and with `ohcldiag-dev`, but it will always be a Hyper-V specific tool. Currently, it can: * Change VM state (starting/stopping/resetting) * Enable serial port output to standard output, or input/output to another terminal window * Enable paravisor log output to standard output or another terminal window * Inspect paravisor state * Modify the paravisor (update the paravisor command line, reload the paravisor) In the future, it might be able to: * Enable Hyper-V log output * Capture serial port output to a file * Inspect host state * Persistence workspaces (save/restore configured serial ports and logs) ## Example session `hypestv` launches into a detached mode, unless you specify a VM name on the command line. To select a VM to work on, the VM named `tdxvm` in this example, use the `select` command. If successful, you will now see the name and VM state in the prompt: ```text > select tdxvm tdxvm [off]> ``` After this, all commands will implicitly operate on `tdxvm`. Use `select` again to work on another VM. To enable serial port output, use the `serial` command. This can be used at any time, even while the VM is not running. E.g., to open a separate window for interactive use of COM1 and enable logging serial port output for COM2: ```text tdxvm [off]> serial 1 term tdxvm [off]> serial 2 log ``` You can also enable paravisor log output at any time: ```text tdxvm [off]> paravisor kmsg log ``` Start a VM with `start`. This is an asynchronous command: you can continue to type other commands at the prompt while the VM starts. You should see an output message when the VM finishes starting, as well as output about any configured serial ports connecting. Note that, due to limitations of the `rustyline` crate, the displayed VM state on the prompt may not be accurate until you type another command or press Enter. ```text tdxvm [off]> start com1 connected com2 connected VM started tdxvm [off]> tdxvm [running]> ``` At this point, the VM is running, including the paravisor (if one is configured). As in the OpenVMM interactive console, you can inspect paravisor state with the `inspect` or `x` command, but under the `paravisor`/`pv` command: ```text tdxvm [running]> pv x { build_info: _, control_state: "started", mesh: _, proc: _, trace: _, uhdiag: _, vm: _, } ``` You can terminate the VM with `kill`. This will disconnect any connected serial ports as well, but they will reconnect next time the VM starts. Killing a VM does not detach/deselect it; subsequent commands will continue to operate on the VM. ```text tdxvm [running]> kill com1 disconnected com2 disconnected VM killed tdxvm [stopping]> tdxvm [off]> ``` ## Asynchronous event model Power operations and endpoint connections can complete after the prompt is redrawn. Hypestv prints completion, disconnection, and failure events as they arrive. The prompt's state is therefore a recent observation rather than a transactional lock on Hyper-V state. Press Enter to refresh the prompt after an asynchronous transition. Avoid starting a conflicting operation while the VM is still starting, stopping, or reloading. ## Troubleshooting - If `select` fails, use `list` and match the registered VM name exactly. - If a WMI `kill` fails while Hyper-V believes the VM is transitioning, retry with `kill --force` only when abrupt power loss is acceptable. - If guest `shutdown` has no effect, verify that the guest shutdown service and integration path are running. - If `pv` commands cannot connect, confirm that the VM contains OpenHCL and that the paravisor reached its diagnostic service. - For a failed reload, enable `pv kmsg log` and the relevant serial ports before trying again so early output is retained. - If displayed state appears stale, wait for the asynchronous completion event and press Enter. --- ### Guide Src Dev Guide Dev Tools Igvmfilegen (`Guide/src/dev_guide/dev_tools/igvmfilegen.md`) # igvmfilegen `igvmfilegen` constructs, measures, inspects, and signs metadata for Isolated Guest Virtual Machine (IGVM) files. ## Role in an OpenHCL build An OpenHCL IGVM combines several independently built inputs, including the boot loader, Linux kernel, initrd, `openvmm_hcl`, optional sidecar, and guest boot resources. A manifest describes how to place those resources and which VBS, SEV-SNP, or Intel TDX platform definitions the file supports. `igvmfilegen` turns that description into the final binary layout and computes the platform launch measurements and endorsement material associated with it. Use `cargo xflowey build-igvm ` for normal OpenHCL builds. Flowey selects a supported manifest and produces the resource file from concrete build artifacts. Invoke `igvmfilegen` directly when developing manifests, inspecting an image, or completing a signing workflow. ## Build Build the host utility with: ```bash cargo build -p igvmfilegen ``` Display the current top-level interface with: ```bash cargo run -p igvmfilegen -- --help ``` ## Build from a manifest The `manifest` command consumes two JSON files: - The manifest describes images, platform policies, memory layout, command lines, and the relationship between components. - The resources file maps symbolic manifest resources to concrete files. Build an image with: ```bash cargo run -p igvmfilegen -- manifest \ --manifest path/to/manifest.json \ --resources path/to/resources.json \ --output path/to/openhcl.bin ``` `--debug-validation` enables additional build-time checks. `--disable-secure-avic` overrides secure AVIC for supported debug SNP configurations. ```admonish danger `--confidential-debug` changes the measured OpenHCL command line so a confidential guest permits diagnostics and trusts host-provided debug options. Use it only for an intentionally debuggable development image. It weakens the security assumptions expected from a production confidential image. ``` For each measurable platform, the command also writes sibling identity and endorsement files next to the output: ```text -snp.json -snp.cbor -tdx.json -tdx.cbor -vbs.json -vbs.cbor -snp.idblock ``` Only files corresponding to platforms present in the manifest are emitted. The `.idblock` file is the SNP signing payload used by the production signing mode described below. ## Inspect an IGVM Dump headers and directives in a human-readable form: ```bash cargo run -p igvmfilegen -- dump \ --filepath path/to/openhcl.bin ``` The input may be a raw IGVM or a supported `vmfirmwareigvm` resource DLL. For a DLL, the embedded IGVM is selected automatically. Inspect CoRIM entries and optionally extract their payloads: ```bash cargo run -p igvmfilegen -- dump-corim \ --filepath path/to/openhcl.bin \ --platform snp \ --output path/to/corim-output ``` `--header-type document|signature` narrows the result. Without filters, the command reports every supported CoRIM entry in the file. ## Add an SNP ID block An SNP ID block binds an identity key and guest SVN to the image launch measurement. The IGVM must already contain a compatible SNP platform and guest policy, and a file that already has an ID block is rejected. For development, generate an ephemeral P-384 key and specify an SVN: ```bash cargo run -p igvmfilegen -- add-snp-id-block \ --input path/to/input.bin \ --output path/to/output.bin \ --guest-svn ``` Alternatively, `--manifest path/to/manifest.json` sources the guest SVN and image identity from the manifest's SNP configuration. Production signing keeps the private key outside this process. Sign the exact `.idblock` payload emitted by `manifest`, then attach the payload, DER ECDSA signature, and signer public key or certificate: ```bash cargo run -p igvmfilegen -- add-snp-id-block \ --input path/to/input.bin \ --output path/to/output.bin \ --id-block path/to/image-snp.idblock \ --id-signature path/to/signature.der \ --id-public-key path/to/public-key.pem ``` The output path may equal the input path, but a separate output is easier to recover and compare during development. ## Patch a CoRIM signature `manifest` writes the CoRIM document into the IGVM. After an external signer produces a signed bundle or detached signature, attach it for one platform: ```bash cargo run -p igvmfilegen -- patch-corim-signature \ --input path/to/input.bin \ --output path/to/output.bin \ --platform snp \ --corim-bundle path/to/signed-corim.cose ``` Use `--corim-signature` instead when the input is already a detached COSE_Sign1 signature. The tool verifies a well-formed PS384 envelope and verifies its signature over the IGVM-embedded document using the key carried in `x5chain` or `x5bag`. ```admonish warning title="Signature trust" This verification checks signature integrity and algorithm compatibility. It does not establish certificate-chain trust, check revocation, or enforce certificate policy. The caller must obtain the signature from a trusted signing pipeline and validate that pipeline's identity separately. ``` ## Troubleshooting - A missing resource names the symbolic manifest entry that could not be resolved; check the resources JSON and the referenced build output. - A platform-policy error usually indicates incompatible manifest directives, guest policy, architecture, or isolation type. - A measurement mismatch should be investigated as a layout or input change, not bypassed. - ID-block errors can indicate a signing payload from a different image, an unsupported key, or an IGVM that already contains a block. - CoRIM errors distinguish document mismatch, unsupported algorithms, malformed COSE, and failed signature math. None of these establish signer trust. --- ### Guide Src Dev Guide Dev Tools Vmgstool (`Guide/src/dev_guide/dev_tools/vmgstool.md`) # VmgsTool `vmgstool` creates, inspects, and modifies version 3 VM Guest State (VMGS) files for provisioning and debugging. OpenHCL uses VMGS v3 on both Hyper-V and OpenVMM. OpenVMM also uses VMGS v3 for VMs without OpenHCL. Hyper-V VMs without an HCL use a different state format. VMGS persists firmware variables and security state on behalf of a VM. The store is packaged as a VHD and contains numbered logical files. These are data slots rather than host filesystem files; for example, BIOS NVRAM and vTPM state occupy separate well-known file IDs. Confidential VMs can encrypt selected VMGS contents before deployment so the host handles encrypted state rather than plaintext guest secrets. ## Data model A VMGS store contains redundant headers, a file table, and allocated data blocks. Commands identify logical data through either a numeric file ID or a known symbolic name. See [`vmgs_format::FileId`][] for the authoritative list of IDs and names. [`vmgs_format::FileId`]: https://github.com/microsoft/openvmm/blob/main/vm/vmgs/vmgs_format/src/lib.rs#L41-L65 Use `dump-file-table` to discover which IDs are allocated before modifying a store: ```powershell vmgstool.exe dump-file-table --file-path path\to\guest.vmgs ``` ```admonish note `dump-file-table` can inspect an encrypted store without its key. However, its per-file encryption information can be inaccurate when the store was last modified by a pre-1.8 version of OpenHCL, VmgsTool, or OpenVMM. ``` ## Alternatively: Pre-Built Binaries If you would prefer to use VmgsTool without building it from scratch, you can download pre-built copies of the binary from [OpenVMM CI](https://github.com/microsoft/openvmm/actions/workflows/openvmm-ci.yaml). Simply select a successful pipeline run (should have a Green checkbox), and scroll down to select an appropriate `*-vmgstool` artifact for your particular architecture and operating system. ## Running ```admonish tip Note: The examples in this section use the Windows executable `vmgstool.exe`, which can be replaced with the Linux executable `vmgstool`. Developers who have already setup their development environment may also use the appropriate `cargo run` command. For more details on building, see the [build](#building) section below. ``` VmgsTool commands continue to evolve, so use `vmgstool.exe --help` for the current interface. Every command and nested operation also has help: ```powershell vmgstool.exe uefi-nvram dump --help ``` Except where explicitly documented, stdout and stderr are for humans and are not stable automation formats. ### Read and Write Raw Data To read raw data from a VMGS file, use the `dump` command. For example, to export the decrypted binary contents of the BIOS_NVRAM (`--fileid 1`) to a file: ```powershell vmgstool.exe dump --filepath path\to\guest.vmgs ` --keypath path\to\key.bin --datapath path\to\nvram.bin --fileid 1 ``` To write raw data to a VMGS file, use the `write` command. For example, to write those NVRAM variables to a different, unencrypted VMGS file: ```powershell vmgstool.exe write --filepath path\to\guest.vmgs ` --datapath path\to\nvram.bin --fileid 1 ``` By default, `write` refuses to replace a nonempty slot. Add `--allow-overwrite` only after confirming the destination ID. If `dump` has no `--data-path`, it writes an ASCII hexadecimal representation to stdout. `--raw-stdout` selects raw bytes and cannot be combined with an output path. ### Read and Parse UEFI NVRAM Variables Furthermore, VmgsTool contains parsers to help debug UEFI NVRAM variables in VMGS file ID 1 (`BIOS_NVRAM`). To dump the variables from an encrypted VMGS and truncate binary data without a parser: ```powershell vmgstool.exe uefi-nvram dump --filepath path\to\guest.vmgs ` --keypath path\to\key.bin --truncate ``` ### Read DLL File to Write IGVMfile to VMGS VmgsTool can extract an IGVM from a resource DLL and write it to VMGS file ID 8 (`GUEST_FIRMWARE`). Select one of `NONCONFIDENTIAL`, `SNP`, `TDX`, `SNP_NO_HCL`, `TDX_NO_HCL`, or `OPENHCL`: ```powershell vmgstool.exe copy-igvmfile --filepath path\to\guest.vmgs ` --datapath path\to\vmfirmwareigvm.dll --resource-code SNP ``` ### Delete Boot Variables to Recover a VM that Fails to Boot A VM may fail to boot if the disk configuration changes and UEFI's `DefaultBootAlwaysAttempt` setting is disabled. Deleting the existing (invalid) boot entries using VmgsTool will trigger a default boot (which attempts to boot all available partitions and devices). To print the boot entries in an encrypted VMGS file: ```powershell vmgstool.exe uefi-nvram remove-boot-entries ` --filepath path\to\guest.vmgs --keypath path\to\key.bin --dry-run ``` To actually remove the boot entries from the VMGS file, remove `--dry-run`. This will remove all `Boot####` variables and the `BootOrder` variable. To remove a specific boot entry or another UEFI NVRAM variable, use `remove-entry`. For example, to remove `Boot0000`: ```powershell vmgstool.exe uefi-nvram remove-entry --filepath path\to\guest.vmgs ` --keypath path\to\key.bin --name Boot0000 ` --vendor 8be4df61-93ca-11d2-aa0d-00e098032b8c ``` Always run `remove-boot-entries --dry-run` first. Removing all boot variables causes firmware to fall back to default boot enumeration and can change which disk starts. ## Troubleshooting ### Expected at least N more bytes, but only found M If you get an error similar to the one below, it is likely that you are trying to read an encrypted VMGS file and haven't provided the correct decryption key. ```text ERROR: remove_boot_entries error Caused by: 0: error loading data from Nvram storage 1: unexpected EOF. expected at least 2330702412 more bytes, but only found 30067 ``` ## Building Prior to building VmgsTool, please ensure you have built either OpenVMM or OpenHCL at least once, to ensure you have all necessary build dependencies installed. VmgsTool can be built with `cargo build -p vmgstool` for Windows and Linux. To interact with encrypted VMGS files, you will need to compile with the encryption feature: `cargo build --features encryption -p vmgstool` --- ### Guide Src Dev Guide Dev Tools Xflowey (`Guide/src/dev_guide/dev_tools/xflowey.md`) # cargo xflowey `cargo xflowey` runs OpenVMM's local Flowey pipelines for builds, dependency restoration, test execution, and generated CI workflows. ## Pipeline overview Important local pipelines include: | Pipeline | Purpose | | --- | --- | | `restore-packages` | Download and unpack external build dependencies | | `build-igvm` | Build a supported or customized OpenHCL IGVM image | | `vmm-tests-run` | Discover artifacts, build them, and run selected VMM tests | | `regen` | Regenerate checked-in CI definitions from Flowey pipelines | List the current surface with: ```bash cargo xflowey --help ``` ## How a pipeline runs Flowey separates planning from execution. A pipeline declares jobs and their node dependencies, and requests typed variables or artifacts from those nodes. The runtime resolves the resulting graph, schedules ready steps, and records enough state to avoid repeating valid work. Local runs commonly write to: - `flowey-out` for requested artifacts and user-facing output. - `flowey-persist` for cached node state and downloaded inputs. - `target` for Cargo build products and test-specific scratch roots. ## Restoring packages Restore external dependencies needed by common builds: ```bash cargo xflowey restore-packages ``` This can include firmware, kernels, sysroots, protocol compilers, and other versioned packages. The exact set is selected from the build configuration and host platform. Some pipelines support `--install-missing-deps`, allowing Flowey to install reported host prerequisites. Without it, the failure should explain the manual installation needed. ## Building OpenHCL Build a standard x64 image: ```bash cargo xflowey build-igvm x64 ``` Select release output or another supported recipe: ```bash cargo xflowey build-igvm x64-cvm --release ``` The pipeline builds and stages `openvmm_hcl`, the boot loader, sidecar, kernel, initrd, manifest resources, and final IGVM. Output is placed below `flowey-out/artifacts/build-igvm` by recipe and build profile. ```admonish warning The local `build-igvm` CLI is a developer interface and does not promise stable automation syntax. Checked-in automation should depend on Flowey nodes and typed artifacts rather than shelling out to this local command. ``` ## Running VMM tests Run one test or group with a nextest filter expression: ```bash cargo xflowey vmm-tests-run --filter "test(my_test)" ``` The pipeline first enumerates matching tests and their static artifact requirements. It then selects builds and downloads, initializes a test-content directory, performs required image preparation, and invokes nextest with the resolved environment. Use `--build-only` to produce the selected artifacts without executing tests. Cross-compilation is selected through `--target` and related options described in the VMM-test guide. ## Regenerating CI Flowey pipeline definitions are the source of truth for generated CI YAML. After changing those definitions, regenerate them with: ```bash cargo xflowey regen ``` Do not hand-edit generated files under `ci-flowey` or generated workflow paths. The `verify-flowey` xtask pass checks whether regeneration would change them. The same graph definitions used for local pipelines are adapted to CI backends. CI adds pools, triggers, permissions, and published artifacts around the shared nodes rather than maintaining an unrelated shell implementation. ## `xflowey` vs `xtask` Use the ownership boundary: - `cargo xtask`: implements novel, standalone tools/utilities - `cargo xflowey`: orchestrates invoking a sequence of tools/utilities, without doing any non-trivial data processing itself ## Troubleshooting - Read the first failed node's context rather than only the final pipeline summary; later requested artifacts often fail because of that dependency. - Use the pipeline's `--help` to find supported verbosity and dependency installation options. - A stale or corrupt cached download can be isolated by cleaning the specific state reported in `flowey-persist`; avoid deleting every package by default. - When cross-compiling from WSL, source the Windows cross environment before invoking a Windows target. - If generated CI verification fails, run `cargo xflowey regen` and review the generated diff rather than editing YAML manually. --- ### Guide Src Dev Guide Dev Tools Xtask (`Guide/src/dev_guide/dev_tools/xtask.md`) # cargo xtask `cargo xtask` runs standalone, repository-specific development utilities implemented by the `xtask` binary. ## Invocation Run commands from the repository root so task-relative paths resolve consistently: ```bash cargo xtask --help ``` ## Command overview | Command | Purpose | | --- | --- | | `clean` | Remove generated build-system, package, and test artifacts | | `fmt` | Run repository formatting and structural validation passes | | `fuzz` | Discover, build, execute, minimize, and inspect fuzz targets | | `guest-test` | Create or download guest test images | | `install-git-hooks` | Install repository-managed git hooks | | `verify-size` | Compare ELF section sizes between two binaries | Run `cargo xtask --help` for the current command-specific interface. ## Cleaning generated state The default clean removes directories that `cargo clean` does not own: - `flowey-out` - `flowey-persist` - `.packages` - `vmm_test_results` Preview the removal first: ```bash cargo xtask clean --dry-run ``` Add `--cargo` to run `cargo clean` and remove `target` as well. ```admonish warning The default clean removes restored packages and test results in addition to generated Flowey state. Use `--dry-run` when you only intend to reclaim Cargo output or need to retain diagnostic artifacts. ``` ## Repository formatting `cargo xtask fmt` is the canonical repository formatter and validator. It runs four pass families: | Pass | Coverage | | --- | --- | | `rustfmt` | Rust formatting and generated formatting expectations | | `lints` | Repository source, metadata, documentation, and policy lints | | `verify-fuzzers` | Fuzz-target manifest and registration consistency | | `verify-flowey` | Flowey pipeline and generated-definition consistency | Run every pass and apply supported fixes: ```bash cargo xtask fmt --fix ``` Run only selected passes against changed files: ```bash cargo xtask fmt --only-diffed \ --pass rustfmt --pass lints ``` Without `--fix`, independent passes run in parallel unless `--no-parallel` is set. A failed pass prints the corresponding fixing command where applicable. ## Fuzzing commands The `fuzz` command wraps `cargo-fuzz` with workspace target discovery and OpenVMM-specific build configuration. Start by listing registered targets: ```bash cargo xtask fuzz list ``` Run a target or reproduce one artifact: ```bash cargo xtask fuzz run fuzz_ide cargo xtask fuzz run fuzz_ide path/to/crash-artifact ``` The fuzzing section documents prerequisites, corpora, minimization, and coverage workflows in depth. ## Guest test images `guest-test uefi` places a UEFI executable into a bootable raw image. For example: ```bash cargo xtask guest-test uefi \ --bootx64 path/to/guest_test_uefi.efi ``` `guest-test download-image` obtains the repository's supported guest test image. Use each subcommand's help for architecture and output options. ## Git hooks Install the repository-managed hooks with: ```bash cargo xtask install-git-hooks ``` The installed hook locates the built automation binary through `target/xtask-path`, avoiding an unnecessary Cargo invocation when possible. Hook behavior still comes from the current checkout, so rerun installation if the hook setup changes. Fix a hook-reported issue with the same `cargo xtask` command shown in its output. Do not bypass a failing hook without understanding which required check is being skipped. ## Binary size comparison Compare section sizes of two ELF binaries: ```bash cargo xtask verify-size \ --original path/to/original \ --new path/to/new ``` The command reports changed sections, net size difference, and total absolute section difference. It fails when the total difference exceeds its current 50-KiB threshold. This is a coarse guard for unexpected layout changes, not a replacement for scenario-specific image size limits. ## `xtask` versus `xflowey` Use `xtask` for a tool that performs project-specific data processing or a focused repository operation. Use `xflowey` to orchestrate a dependency graph of builds, downloads, and existing tools across platforms. Examples: - Formatting Rust and validating fuzzer manifests belong in `xtask`. - Building OpenHCL from many components and resolving VMM-test artifacts belong in `xflowey`. ## Troubleshooting - Run from the repository root unless using `--custom-root`. - If an automation change is not reflected, allow Cargo to rebuild the `light` profile instead of invoking a stale path directly. - On Windows, locked files can prevent cleanup; stop VMs, debuggers, and helper processes before retrying. - Use `--no-parallel` when interleaved formatter output obscures the failing pass. - A missing `cargo-fuzz` or target toolchain is a fuzzing prerequisite error, not an xtask registration failure. The command follows the general [`cargo-xtask`](https://github.com/matklad/cargo-xtask) convention. --- ### Guide Src Dev Guide Getting Started Build Ohcl Kernel (`Guide/src/dev_guide/getting_started/build_ohcl_kernel.md`) # Building a custom OpenHCL Linux Kernel ```admonish note This step is NOT required in order to build OpenHCL! Unless you have a specific reason, it is _strongly_ recommended to stick to the pre-built Kernel image which is automatically downloaded as part of the OpenHCL build process. ``` ## Cloning the kernel repository If you need to rebuild the kernel, the sources are available in the [OHCL-Linux-Kernel](https://github.com/microsoft/OHCL-Linux-Kernel) repo : * the main branch is [product/hcl-main/6.6](https://github.com/microsoft/OHCL-Linux-Kernel/tree/product/hcl-main/6.6), * the dev branch is [project/hcl-dev/6.6](https://github.com/microsoft/OHCL-Linux-Kernel/tree/project/hcl-dev/6.6). Unless you need the entire repo history, cloning just one branch and `depth = 1`, saves significant time and disk space: ```sh git clone https://github.com/microsoft/OHCL-Linux-Kernel.git -b product/hcl-main/6.6 --depth=1 ``` Cloning under Windows is likely to fail due to some files using names that Windows reserves, e.g. `aux.c`, and NTFS being non case-sensitive by default as there are few files in the Linux kernel repo whose names differ in their case only. To clone successfully under Windows, need a fix in `ntdll` (merged in `Ni`?), and a case-sensitive NTFS partition. Best to start with the default WSL2 if there is no existing working setup. ## Building the kernel locally The following instructions for building the kernel locally target the Ubuntu distributions. In order to build on rpm-based systems the only changes are likely to be the way the package manager is invoked and installing the `kernel-devel` package instead of what is needed for Ubuntu. The paths below are relative to the cloned kernel repository root. Do once for every machine that hasn't run this step successfully: ```sh ./Microsoft/install-deps.sh ``` If you plan to use your custom kernel to work with confidential VMs, you need to enable a few more options in the kernel. Do once after cloning the kernel repository or every time you remove local changes from your kernel configuration file: ```sh ./Microsoft/merge-cvm-config.sh ``` Every time the kernel needs to be rebuilt: ```sh ./Microsoft/build-hcl-kernel.sh ``` The output directory is `./out`, it contains the kernel binary, the kernel modules, and the debug symbols, and `cargo xflowey build-igvm` can be pointed to it to use as a source of the kernel binary and the modules. In the case you are iterating on a change, install [`ccache`](https://ccache.dev/) for decreasing the kernel build time significantly, might be close to an order of magnitude. To use the compiler cache, prepend `CC="ccache gcc"` to the build command. To see if the cache integrates itself into the toolchain: ```sh host:~$ which gcc /usr/lib64/ccache/gcc ``` --- ### Guide Src Dev Guide Getting Started Build Openhcl (`Guide/src/dev_guide/getting_started/build_openhcl.md`) # Building OpenHCL **Prerequisites:** - [Getting started on Linux / WSL2](./linux.md). Reminder: OpenHCL cannot currently be built on Windows hosts! * * * An OpenHCL IGVM firmware image is composed of several distinct binaries and artifacts. For example: the `openvmm_hcl` usermode binary, the OpenHCL boot shim, the OpenHCL Linux kernel and initrd, etc.... Some of these components are built directly out of the OpenVMM repo, whereas others must be downloaded as pre-built artifacts from other associated repos. Various tools and scripts will then transform, package, and re-package these artifacts into a final OpenHCL IGVM firmware binary. Fortunately, we don't expect you do to all those steps manually! All the complexity of installing the correct system dependencies, building the right binaries, downloading the right artifacts, etc... is neatly encapsulated behind a single `cargo xflowey build-igvm` command, which orchestrates the entire end-to-end OpenHCL build process. Using the `build-igvm` flow is as simple as running: ```bash cargo xflowey build-igvm [RECIPE] ``` The first build will take some time as all the dependencies are installed/downloaded/built. **Note: At this time, OpenHCL can only be built on Linux (or WSL2)!** A "recipe" corresponds to one of the pre-defined IGVM SKUs that are actively supported and tested in OpenVMM's build infrastructure. A single recipe encodes _all_ the details of what goes into an individual IGVM file, such as what build flags `openvmm_hcl` should be built with, what goes into a VTL2 initrd, what `igvmfilegen` manifest is being used, etc... - e.g: `x64`, for a "standard" x64 IGVM - e.g: `aarch64`, for a "standard" aarch64 IGVM - e.g: `x64-cvm`, for a x64 CVM IGVM - e.g: `x64-test-linux-direct`, for x64 IGVM booting a test linux direct image - _for a full list of available recipes, please run `cargo xflowey build-igvm --help`_ New recipes can be added by modifying the `build-igvm` source code. Build output is then binplaced to: `flowey-out/artifacts/build-igvm/{release-mode}/{recipe}/openhcl-{recipe}.bin` So, for example: ```bash cargo xflowey build-igvm x64-cvm # output: flowey-out/artifacts/build-igvm/debug/x64-cvm/openhcl-x64-cvm.bin cargo xflowey build-igvm x64 --release # output: flowey-out/artifacts/build-igvm/release/x64/openhcl-x64.bin ``` ```admonish warning `cargo xflowey build-igvm` is designed to be used as part of the developer inner-loop, and does _NOT_ have a stable CLI suitable for CI or any other form of production automation! In-tree pipelines and automation should interface with the underlying `flowey` infrastructure that powers `cargo xflowey build-igvm`, _without_ relying on the details of its CLI. ``` ## Building ohcldiag-dev `ohcldiag-dev` is typically built as a Windows binary. This can be done directly from Windows, or using [cross-compilation from WSL2](../getting_started/cross_compile.md). The command to build `ohcldiag-dev` is simply: ```sh # you may need to run `rustup target add x86_64-pc-windows-msvc` first cargo build -p ohcldiag-dev --target x86_64-pc-windows-msvc ``` **Note:** Thanks to x86 emulation built into Windows, `ohcldiag-dev.exe` that is built for x64 Windows will work on Aarch64 Windows as well. ## Troubleshooting This section documents some common errors you may encounter while building OpenHCL. If you are still running into issues, consider filing an issue on the OpenVMM GitHub Issue tracker. ### Help! The build failed due to a missing dependency If you don't mind having `xflowey` install some dependencies globally on your machine (i.e: via `apt install`, or `rustup toolchain add`), you can pass `--install-missing-deps` to your invocation of `build-igvm`: ```bash cargo xflowey build-igvm x64 --install-missing-deps ``` This will automatically install all required dependencies, including the .NET SDK, Rust toolchains, Node.js, and any necessary system packages. Alternatively - `build-igvm` _should_ emit useful human-readable error messages when it encounters a dependency that isn't installed, with a suggestion on how to install it. If it doesn't - please file an Issue! ### Help! Everything is rebuilding even though I only made a small change Cargo's target triple handling can be a bit buggy. Try running with: ```bash CARGO_BUILD_TARGET=x86_64-unknown-linux-gnu cargo build-igvm [RECIPE] ``` or adding the below to your .bashrc: ```bash export CARGO_BUILD_TARGET=x86_64-unknown-linux-gnu ``` ## Build Customization Aside from building IGVM files corresponding the the built-in IGVM recipes, `build-igvm` also offers a plethora of customization options for developers who wish to build specialized custom IGVM files for local testing. Some examples of potentially useful customization include: - `--override-manifest`: Override the recipe's `igvmfilegen` manifest file via, in order to tweak different kernel command line options, different VTL0 boot configuration, or different VTL2 memory sizes. - `--custom-openvmm-hcl`: Specify a pre-built `openvmm_hcl` binary. This is useful in case you have already built it with some custom settings, e.g.: ```bash cargo build --target x86_64-unknown-linux-musl -p openvmm_hcl --features myfeature cargo xflowey build-igvm x64 --custom-openvmm-hcl target/x86_64-unknown-linux-musl/debug/openvmm_hcl ``` - Specify a custom VTL2 kernel `vmlinux` / `Image`, instead of using the packaged main kernel. ```bash cargo xflowey build-igvm x64 --custom-kernel path/to/my/prebuilt/vmlinux ``` The packaged dev kernel variants are disabled by default. A custom kernel remains available for local kernel development without enabling those variants. For a full list of available customizations, refer to `build-igvm --help`. ### Advanced Depending on what you're doing, you may need to build the individual components that go into an OpenHCL IGVM build. Our `flowey`-based pipelines handle the complexities of properly invoking and orchestrating the various individual build tools / scripts used to construct IGVM files, but a sufficiently motivated user can go through these steps manually. Please consult the source code for `cargo xflowey build-igvm` for a breakdown of all build steps and available customization options. Note that the canonical "source of truth" for how to build end-to-end OpenHCL IGVM files are these build scripts themselves, and the specific flow is subject to change over time! --- ### Guide Src Dev Guide Getting Started Build Openvmm (`Guide/src/dev_guide/getting_started/build_openvmm.md`) # Building OpenVMM **Prerequisites:** - One of: - [Getting started on Windows](./windows.md) - [Getting started on Linux / WSL2](./linux.md). ```admonish tip It is strongly suggested that you use WSL2, and [cross compile](./cross_compile.md) for Windows when necessary. ``` ## Build Dependencies OpenVMM currently requires a handful of external dependencies to be present in order to properly build / run. e.g: a copy of `protoc` to compile Protobuf files, a copy of the `mu_msvm` UEFI firmware, some test linux kernels, etc... Running the following command will fetch and unpack these various artifacts into the correct locations within the repo: ```sh cargo xflowey restore-packages ``` If you intend to cross-compile, refer to the command's `--help` for additional options related to downloading packages for other architectures. ## Building OpenVMM uses the standard Rust build system, `cargo`. To build OpenVMM, simply run: ```sh cargo build ``` Note that certain features may require compiling with additional `--feature` flags. ## Troubleshooting This section documents some common errors you may encounter while building OpenVMM. If you are still running into issues, consider filing an issue on the OpenVMM GitHub Issue tracker. ### failed to invoke protoc **Error:** ```text error: failed to run custom build command for `inspect_proto v0.0.0 (/home/daprilik/src/openvmm/support/inspect_proto)` Caused by: process didn't exit successfully: `/home/daprilik/src/openvmm/target/debug/build/inspect_proto-e959f9d63c672ccc/build-script-build` (exit status: 101) --- stderr thread 'main' panicked at support/inspect_proto/build.rs:23:10: called `Result::unwrap()` on an `Err` value: Custom { kind: NotFound, error: "failed to invoke protoc (hint: https://docs.rs/prost-build/#sourcing-protoc): (path: \"/home/daprilik/src/openvmm/.packages/Google.Protobuf.Tools/tools/protoc\"): No such file or directory (os error 2)" } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace warning: build failed, waiting for other jobs to finish... ``` Note: the specific package that throws this error may vary, and may not always be `inspect_proto` **Solution:** You attempted to build OpenVMM without first restoring necessary packages. Please run `cargo xflowey restore-packages`, and try again. ### use of unstable library feature **Error:** ```text error[E0658]: use of unstable library feature 'absolute_path' --> flowey/flowey/src/lib.rs:37:17 | 37 | std::path::absolute(self) | ^^^^^^^^^^^^^^^^^^^ | = note: see issue #92750 for more information For more information about this error, try `rustc --explain E0658`. error: could not compile `flowey` (lib) due to previous error ``` **Solution:** Install Rust using the official instructions for [Linux](https://openvmm.dev/guide/dev_guide/getting_started/linux.html#installing-rust) or [Windows](https://openvmm.dev/guide/dev_guide/getting_started/windows.html#installing-rust). --- ### Guide Src Dev Guide Getting Started Cross Compile (`Guide/src/dev_guide/getting_started/cross_compile.md`) # \[WSL2] Cross Compiling from WSL2 to Windows Setting up cross compilation is very useful, as it allows using the same repo cloned in WSL2 to both develop OpenHCL, as well as launch it via OpenVMM via the WHP backend. ## Required Dependencies Note that this requires some additional dependencies, described below. ### Windows deps Visual Studio build tools must be installed, along with the Windows SDK. [This is the same as what's required to build OpenVMM on windows.](./windows.md#installing-rust) ### WSL deps The msvc target `x86_64-pc-windows-msvc` must be installed for the toolchain being used in WSL. This can be added by doing the following: ```bash rustup target add x86_64-pc-windows-msvc ``` Additional build tools must be installed as well. If your distro has LLVM 14 available (Ubuntu 22.04 or newer): ```bash sudo apt install clang-tools lld llvm-dev ``` Otherwise, follow the steps at to install a specific version, by adding the correct apt repos. Note that you must install `clang-tools` as default `clang` uses gcc style arguments, where `clang-cl` uses msvc style arguments. You can use their helper script as well: ```bash wget https://apt.llvm.org/llvm.sh chmod +x llvm.sh sudo ./llvm.sh sudo apt install clang-tools ``` ## Setting up the terminal environment Source the `build_support/setup_windows_cross.sh` script from your terminal instance. For example, the following script will do this along with setting a default cargo build target: ```bash #!/bin/bash # Setup environment and windows cross tooling. export CARGO_BUILD_TARGET=x86_64-unknown-linux-gnu cd path/to/openvmm || exit . build_support/setup_windows_cross.sh exec "$SHELL" ``` For developers using shells other than bash, you may need to run the `setup_windows_cross.sh` script in bash then launch your shell in order to get the correct environment variables. ## Editing with vscode You can have rust-analyzer target Windows, which will allow you to use the same repo for OpenHCL, Linux, and Windows changes, but the vscode remote server must be launched from the terminal window that sourced the setup script. You can do this by closing all vscode windows then opening your workspace with `code ` in your terminal. Add the following to your workspace settings for a vscode workspace dedicated to Windows: ```json "settings": { "rust-analyzer.cargo.target": "x86_64-pc-windows-msvc" } ``` ## Running Windows OpenVMM from within WSL You can build and run the windows version of OpenVMM by overriding the target field of cargo commands, via `--target x86_64-pc-windows-msvc`. For example, the following command will run OpenVMM with WHP: ```bash cargo run --target x86_64-pc-windows-msvc ``` You can optionally set cargo aliases for this so that you don't have to type out the full target every time. Add the following to your `~/.cargo/config.toml`: ```toml [alias] winbuild = "build --target x86_64-pc-windows-msvc" wincheck = "check --target x86_64-pc-windows-msvc" winclippy = "clippy --target x86_64-pc-windows-msvc" windoc = "doc --target x86_64-pc-windows-msvc" winrun = "run --target x86_64-pc-windows-msvc" wintest = "test --target x86_64-pc-windows-msvc" ``` You can then run the windows version of OpenVMM by running: ```bash cargo winrun ``` OpenVMM configures some environment variables that specify the default Linux kernel, initrd, and UEFI firmware. To make those variables available in Windows, run the following: ```bash export WSLENV=$WSLENV:X86_64_OPENVMM_LINUX_DIRECT_KERNEL:X86_64_OPENVMM_LINUX_DIRECT_INITRD:AARCH64_OPENVMM_LINUX_DIRECT_KERNEL:AARCH64_OPENVMM_LINUX_DIRECT_INITRD:X86_64_OPENVMM_UEFI_FIRMWARE:AARCH64_OPENVMM_UEFI_FIRMWARE:RUST_BACKTRACE ``` ### Speeding up Windows OpenVMM launch Due to filesystem limitations on WSL, launching OpenVMM directly will be somewhat slow. Instead, you can copy the built binaries to a location on in the Windows filesystem and then launch them via WSL. Quite a few folks working on the OpenVMM project have hacked together personal helper scripts to automate this process. Here is one example: (update the variables at the top of the file as necessary.) ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Running Windows binaries from WSL2 When you cross-compile to `x86_64-pc-windows-msvc` and run the resulting `.exe` from WSL, it runs as a **native Windows process**. This means it sees Windows paths (`C:\...`), not Linux paths (`/mnt/c/...`). You must translate paths at the command line. Use `wslpath -w` to convert a WSL path to a Windows path: ```bash # Convert a WSL path to a Windows path wslpath -w /mnt/c/vhds/server25.vhdx # Output: C:\vhds\server25.vhdx ``` Here is a full working example that launches OpenVMM with a VHDX disk: ```bash cargo run --target x86_64-pc-windows-msvc -- \ --vmbus-scsi id=scsi0 \ --disk "memdiff:file:$(wslpath -w /mnt/c/vhds/server25.vhdx),on=scsi0" \ --uefi "firmware=$(wslpath -w .packages/hyperv.uefi.mscoreuefi.x64.RELEASE/MsvmX64/RELEASE_VS2022/FV/MSVM.fd)" \ --gfx -p 6 -m 8GB ``` ~~~admonish tip If you have the [WSLENV export](#running-windows-openvmm-from-within-wsl) active, you can omit the `firmware` option. The `X86_64_OPENVMM_UEFI_FIRMWARE` env var (set in `.cargo/config.toml`) is automatically translated to a Windows path: ```bash export WSLENV=$WSLENV:X86_64_OPENVMM_UEFI_FIRMWARE/p cargo run --target x86_64-pc-windows-msvc -- \ --vmbus-scsi id=scsi0 \ --disk "memdiff:file:$(wslpath -w /mnt/c/vhds/server25.vhdx),on=scsi0" \ --uefi --gfx -p 6 -m 8GB ``` ~~~ ```admonish warning A common mistake is passing `/mnt/c/...` paths directly. The Windows binary does not understand Linux paths, so you'll get "Access is denied" or "file not found" errors. ``` ```admonish tip Keep VHDX files on the Windows filesystem (e.g., `C:\vhds\`), not on the WSL ext4 volume. The WSL volume is slow for large I/O and has limited size. The same performance note from [Speeding up Windows OpenVMM launch](#speeding-up-windows-openvmm-launch) applies to disk images — large files should live on the Windows side. ``` --- ### Guide Src Dev Guide Getting Started Devcontainer (`Guide/src/dev_guide/getting_started/devcontainer.md`) # Getting started via Dev Container This page provides instructions for setting up a development environment using the repo provided Dev Container configuration for local development or development using GitHub Codespaces. The repo provides an Ubuntu `devcontainer.json` that installs Rust and the supported targets for the project. ## Developing using a GitHub Codespace Either create a GitHub Codespace via your fork by clicking on the `Code` box, or visit the link [here](https://github.com/codespaces/new) and select your fork and branch. If you plan on using rust-analyzer or doing any sort of dev work, it's recommended to use an 8 core SKU or beefier. More documentation can be found at the official GitHub [docs](https://docs.github.com/en/codespaces). ## Developing using a local dev container This will use Docker + the dev container vscode extension to launch the repo provided `devcontainer.json` on your local machine. Follow the install instructions outlined [here](https://code.visualstudio.com/docs/devcontainers/containers#_installation). From there, use the dev container extension in vscode to create a new dev container for the repository. It's recommended to clone the repo _inside_ the dev container using the `Dev Containers: Clone Repository Inside Container Volume...` command, as the filesystem otherwise will be very slow over the bind mount, which will make your builds & rust-analyzer very slow. More documentation can be found at the official vscode [docs](https://code.visualstudio.com/docs/devcontainers/containers). ## Customizing your dev container Both GitHub codespaces and local dev containers support dotfile repos which can be used to run personalized install scripts like installing your favorite tools and shells, and copying over your dotfiles and configuration. For codespaces, see the documentation [here](https://docs.github.com/en/codespaces/setting-your-user-preferences/personalizing-github-codespaces-for-your-account#dotfiles). For dev containers, see the documentation [here](https://code.visualstudio.com/docs/devcontainers/containers#_personalizing-with-dotfile-repositories). You can use the same dotfiles repo for both, but note that codespaces has a few more limitations outlined in their documentation. --- ### Guide Src Dev Guide Getting Started Linux (`Guide/src/dev_guide/getting_started/linux.md`) # Getting started on Linux / WSL2 This page provides instructions for installing the necessary dependencies to build OpenVMM or OpenHCL on Linux / WSL2. ## \[WSL2] Installing WSL2 To install Windows Subsystem for Linux, run the following command in an elevated Powershell window: ```powershell PS> wsl --install ``` This should install WSL2 using the default Ubuntu linux distribution. You can check that the installation completed successfully by running the following command in a Powershell window. ```powershell PS> wsl -l -v NAME STATE VERSION * Ubuntu Running 2 ``` Once that command has completed, you will need to open WSL to complete the installation and set your password. You can open WSL by typing `wsl` or `bash` into Command Prompt or Powershell, or by opening the "Ubuntu" Windows Terminal profile that should have been created. ```admonish info If you intend to cross-compile OpenVMM for Windows, please ensure you are running a recent version of Windows 11. Windows 10 is no longer supported as a development platform, due to needed WHP APIs. For the best OpenHCL development experience, we recommend **Windows 11 26H1** (Insider Canary channel, build 28000+) if available for your device — this enables COM3 serial output for kernel logs and matches the CI runners. See [Debugging OpenHCL](../../reference/openhcl/debugging.md#recommended-host-os-for-openhcl-development) for details. ``` All subsequent commands on this page must be run within WSL2. ## Installing Rust To build OpenVMM or OpenHCL, you first need to install Rust. The OpenVMM project actively tracks the latest stable release of Rust, though it may take a week or two after a new stable is released until OpenVMM switches over to it. Please follow the [official instructions](https://www.rust-lang.org/tools/install) to do so. ## \[Linux] Additional Dependencies On Linux, there are various other dependencies you will need depending on what you're working on. On Debian-based distros such as Ubuntu, running the following command within WSL will install these dependencies. In the future, it is likely that this step will be folded into the `cargo xflowey restore-packages` command. ```bash $ sudo apt install \ binutils \ build-essential \ cmake \ gcc-aarch64-linux-gnu \ libssl-dev \ pkg-config ``` ## Cloning the OpenVMM source **If using WSL2:** Do NOT clone the repo into Windows then try to access said clone from Linux! It will result in serious performance issues, and may break certain functionality (e.g: cross-compiling Windows binaries). ```bash cd ~/src/ git clone https://github.com/microsoft/openvmm.git ``` ## Automatic Dependency Installation When building OpenHCL, you can pass `--install-missing-deps` to have the build system automatically install any missing dependencies (e.g., .NET SDK, cross-compilation toolchains, system packages): ```bash cargo xflowey build-igvm x64 --install-missing-deps ``` ```admonish warning This flag may install packages and toolchains globally on your system (e.g., via `apt install` or `rustup toolchain add`). ``` This is the easiest way to get a working build environment without manually tracking down each dependency. ## Next Steps You are now ready to build [OpenVMM](./build_openvmm.md) or [OpenHCL](./build_openhcl.md)! --- ### Guide Src Dev Guide Getting Started Running From Wsl (`Guide/src/dev_guide/getting_started/running_from_wsl.md`) # Running OpenVMM from WSL2 WSL2 runs inside a Hyper-V VM, so when you run OpenVMM from WSL you have two options with very different behavior: cross-compile to Windows, or build natively for Linux. ## Option A: Cross-compile to Windows (recommended) Cross-compiling produces a Windows `.exe` that runs natively on the host, outside the WSL2 VM. This is the recommended approach for most development. ```bash cargo run --target x86_64-pc-windows-msvc ``` - Uses **WHP** (Windows Hypervisor Platform) — runs directly on the host hypervisor with no nesting overhead. - **Supports OpenHCL / VTL2** — the full Hyper-V paravisor stack works because WHP provides VTL support. - Best performance — your guest VM runs at native speed on the host. **Setup:** ```bash rustup target add x86_64-pc-windows-msvc ``` You also need Visual Studio Build Tools installed on the Windows side. See the [Cross Compiling](./cross_compile.md) page for full setup instructions. Because the resulting binary is a Windows process, you must translate WSL paths to Windows paths with `wslpath -w`. See [Running Windows binaries from WSL2](./cross_compile.md#running-windows-binaries-from-wsl2) for details. ## Option B: Build natively for Linux Building with the default Linux target produces a Linux binary that runs inside the WSL2 VM. ```bash cargo run ``` - Uses **KVM** as the hypervisor backend. (MSHV — the Microsoft Hypervisor for Linux — is tried first but requires a specialized kernel module that is not publicly available. In WSL2, KVM is what you'll get.) - Does **not** support OpenHCL. OpenHCL requires Hyper-V VTL support, which KVM does not provide. - Performance suffers from nested virtualization — your guest VM runs inside WSL2's Hyper-V VM, adding overhead. - Fine for testing OpenVMM without VTL2, for example booting a simple guest VM with `--uefi`. ```admonish note KVM requires access to `/dev/kvm`. Add your user to the `kvm` group: ~~~bash sudo usermod -aG kvm $USER ~~~ Then log out and back in. Verify with `groups | grep kvm`. ``` ## Backend auto-detection OpenVMM auto-selects the hypervisor backend based on the platform. On Linux it tries MSHV first (requires a specialized kernel module not publicly available), then falls back to KVM. On Windows it uses WHP. You can override with `--hypervisor` but rarely need to. ## Summary | Approach | Binary | Backend | OpenHCL? | Performance | |----------|--------|---------|----------|-------------| | Cross-compile (`--target x86_64-pc-windows-msvc`) | Windows | WHP | ✅ Yes | Best — native, no nesting | | Native Linux build | Linux | KVM/MSHV | ❌ No | Nested — may be slow | ```admonish tip Use cross-compilation for most development. It's what the project's CI uses, it supports OpenHCL, and it avoids nested virtualization overhead. ``` --- ### Guide Src Dev Guide Getting Started Suggested Dev Env (`Guide/src/dev_guide/getting_started/suggested_dev_env.md`) # Suggested Dev Environment **Prerequisites:** - One of: - [Getting started on Windows](./windows.md) - [Getting started on Linux / WSL2](./linux.md). - [Getting started via Dev Container](./devcontainer.md) - One of: - [Building OpenVMM](./build_openvmm.md) - [Building OpenHCL](./build_openhcl.md) * * * This page is for those interested in actively iterating on OpenVMM or OpenHCL. ## Setting up VSCode These instructions assume you're using [VSCode](https://code.visualstudio.com/). If you're using a different development environment, we nonetheless suggest reading through this section, so you can enable similar settings in whatever editor / IDE you happen to be using. ~~~admonish tip Just want the recommended editor settings? Put this in `openvmm/.vscode/settings.json`: ```json { "rust-analyzer.linkedProjects": [ "Cargo.toml", ], "rust-analyzer.cargo.targetDir": true, "rust-analyzer.imports.granularity.group": "item", "rust-analyzer.imports.group.enable": false, "[rust]": { "editor.formatOnSave": true }, } ``` ~~~ ### \[WSL2] Connecting to WSL using VSCode When using Visual Studio Code with WSL, be sure to use the [WSL extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-wsl) instead of accessing your files using the `\\wsl.localhost` share (the repo should be cloned in the WSL filesystem, as mentioned in the WSL getting started guide). This will ensure that all VSCode extensions and features to work properly. Once the extension is installed, click the blue arrows in the bottom left corner and select "Connect to WSL". Then open the folder you cloned the repository into. More information is available [here](https://learn.microsoft.com/en-us/windows/wsl/tutorials/wsl-vscode). ### Configuring `rust-analyzer` [rust-analyzer](https://marketplace.visualstudio.com/items?itemName=matklad.rust-analyzer) provides IDE-like functionality when writing Rust code (e.g: autocomplete, jump to definition, refactoring, etc...). It is a massive productivity multiplier when working with Rust code, and it would be a _very bad idea_ to work in the OpenVMM repo without having it set up correctly. Check out the [rust-analyzer manual](https://rust-analyzer.github.io/manual.html) for a comprehensive overview of rust-analyzer's features. Once installed, we suggest you specify the following additional configuration options in the OpenVMM workspace's `.vscode/settings.json` file: ```json { "rust-analyzer.linkedProjects": [ "Cargo.toml", ] } ``` #### (Strongly Suggested) Avoiding cache invalidation To avoid unnecessary re-builds or lock-contention in the build directory between rust-analyzer and manual builds, set the following configuration option to give rust-analyzer a separate target directory: ```json { "rust-analyzer.cargo.targetDir": true, } ``` #### (Strongly Suggested) Disable nested imports When auto-importing deps, rust-analyzer defaults to nesting imports, which isn't the OpenVMM convention. This can be changed to one-dep-per-line by specifying the following settings: ```json { "rust-analyzer.imports.granularity.group": "item", "rust-analyzer.imports.group.enable": false, } ``` #### Enabling `clippy` CI will fail if the code is not clippy-clean. [Clippy](https://doc.rust-lang.org/stable/clippy/) is a linter that helps catch common mistakes and improves the quality of our Rust code. By default, rust-analyzer will use `cargo check` to lint code, but it can be configured to use `cargo clippy` instead: ```json { "rust-analyzer.check.command": "clippy", } ``` #### Enabling Format on Save CI will fail if code is not formatted with `rustfmt`. You can enable the "format on save" option in VSCode to automatically run `rustfmt` whenever you save a file: ```json { "[rust]": { "editor.formatOnSave": true }, } ``` #### Enhanced "Enter" `rust-analyzer` can override the "Enter" key to make it smarter: - "Enter" inside triple-slash comments automatically inserts `///` - "Enter" in the middle or after a trailing space in `//` inserts `//` - "Enter" inside `//!` doc comments automatically inserts `//!` - "Enter" after `{` indents contents and closing `}` of single-line block This action needs to be assigned to shortcut explicitly, which can be done by adding the following line to `keybindings.json`: ```json // must be put into keybindings.json, NOT .vscode/settings.json! { "key": "Enter", "command": "rust-analyzer.onEnter", "when": "editorTextFocus && !suggestWidgetVisible && editorLangId == rust" } ``` ### GitHub Pull Request Integration As the repo is hosted on GitHub, you might find convenient to use the [GitHub Pull Request](https://marketplace.visualstudio.com/items?itemName=GitHub.vscode-pull-request-github) VSCode extension. That allows working through the PR feedback and issues without leaving the comfort of VSCode. ### (Possibly Useful) Enabling 'lints' formatting on-save Aside from using `rustfmt`, the OpenVMM project also relies on a handful of extra formatting custom lints. e.g: enforcing the presence of copyright headers, enforcing single-trailing newlines, etc... CI will fail if files are not formatted with `cargo xtask fmt --pass lints`. In general, there are 3 ways to fix our custom lints: 1. Manually fixing issues in response to automated feedback 2. Invoking `cargo xtask fmt --pass lints --fix` to fix the whole project 3. Invoking `cargo xtask fmt --pass lints --fix --only-diffed` to fix only changes with a git diff If you would prefer having lints enforced whenever you save a file in VSCode, you can install the [RunOnSave](https://marketplace.visualstudio.com/items?itemName=emeraldwalk.RunOnSave) extension, and add the following configuration to `.vscode/settings.json`: ```json { "emeraldwalk.runonsave": { "commands": [ { "match": ".*", "cmd": "cd ${workspaceFolder}" }, { "match": ".*", "isAsync": true, "cmd": "$(cat ./target/xtask-path) --run-on-save fmt --pass lints --fix --only-diffed" } ] }, } ``` ## Setting up pre-commit and pre-push hooks It's never fun having CI reject your changes due to some minor formatting issue, especially when it's super quick to run those formatting checks locally. Running `cargo xtask fmt` before pushing up your code is quick and easy, and will save you the annoyance of wrestling with formatting check-in gates! Of course, it's very easy to forget to run `cargo xtask fmt` after making code changes, but thankfully, you can set up some [git hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) that will do this for you automatically! You can run `cargo xtask install-git-hooks --help` for more details on what hooks are available and their various configuration options, but for most users, we suggest the following config: ```bash cargo xtask install-git-hooks --pre-push --with-fmt=yes ``` And you'll be all set! ```admonish success If you're worried about time, the `pre-push` hook should only take ~5 seconds to run locally. That's far better than waiting ~20+ minutes only for CI to fail on your pull request. ``` ## \[WSL2] Cross Compiling from WSL2 to Windows You may also want to [set up cross compiling in WSL2](./cross_compile.md) to you can build for both Windows and Linux in one dev environemnt. --- ### Guide Src Dev Guide Getting Started Windows (`Guide/src/dev_guide/getting_started/windows.md`) # Getting started on Windows This page provides instructions for installing the necessary dependencies to build OpenVMM on Windows. ```admonish tip We _strongly_ suggest using [WSL2](./linux.md) for OpenVMM development, rather than developing on Windows directly. Developing in WSL2 offers a smoother development experience, while still allowing you to build and run OpenVMM on Windows through the use of [cross compilation](./cross_compile.md). Additionally, it allows you to have a single clone of the OpenVMM repo suitable for both OpenVMM and OpenHCL development. ``` You must be running a recent version of Windows 11. Windows 10 is no longer supported as a development platform, due to needed WHP APIs. ```admonish note title="Recommended: Windows 11 26H1" For the best OpenHCL development experience, we recommend running a **Windows 11 26H1** Insider flight (build 28000+) if it is available for your device. This enables COM3 serial output for OpenHCL kernel logs and matches the OS used on the project's CI runners. See [What to know about Windows 11 version 26H1](https://techcommunity.microsoft.com/blog/windows-itpro-blog/what-to-know-about-windows-11-version-26h1/4491941) and the [Windows Insider Flight Hub](https://learn.microsoft.com/en-us/windows-insider/flight-hub/) for availability. If 26H1 is not available for your device, Windows 11 24H2/25H2 works — see [Debugging OpenHCL](../../reference/openhcl/debugging.md#recommended-host-os-for-openhcl-development) for details on the tradeoffs. ``` **NOTE: OpenHCL does NOT build on Windows.** If you are interested in building OpenHCL, please follow the getting started guide for [Linux / WSL2](./linux.md). ## Installing Rust To build OpenVMM, you first need to install Rust. The OpenVMM project actively tracks the latest stable release of Rust, though it may take a week or two after a new stable is released until OpenVMM switches over to it. Please follow the [official instructions](https://www.rust-lang.org/tools/install) to do so. ### Visual Studio C++ Build Tools and Windows SDK If you don't already have it, you will need to install [Visual Studio C++ Build tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) or [Visual Studio](https://visualstudio.microsoft.com/vs/) with the components "Desktop Development for C++" and the Windows SDK. Windows 11 SDK version 26100 is the latest version as of this update, but you should install the newest version if there is a newer version. The C++ build tools can be installed via `Visual Studio Installer` -> `Modify` -> `Individual Components` -> `MSVC v143 - VS 2022 C++ x64/x86 build tools (latest)`. The Windows SDK can be installed via `Visual Studio Installer` -> `Modify` -> `Individual Components` -> `Windows 11 SDK (10.0.26100.0)`. The C++ build tools and Windows SDK can be installed at the same time. Or, you can install the tool via the powershell command below. ```powershell PS> winget install Microsoft.VisualStudio.2022.Community --override "--quiet --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.Windows11SDK.26100" ``` ### Aarch64 support To build ARM64, you need an additional dependency. This can be installed via `Visual Studio Installer` -> `Modify` -> `Individual Components` -> `MSVC v143 - VS 2022 C++ ARM64/ARM64EC build tools (latest)`. Or, you can install the tool via the powershell command below. ```powershell PS> winget install Microsoft.VisualStudio.2022.Community --override "--quiet --add Microsoft.VisualStudio.Component.VC.Tools.ARM64" ``` ## Cloning the OpenVMM source If you haven't already installed `git`, you can download it [here](https://git-scm.com/downloads). ```powershell PS> git clone https://github.com/microsoft/openvmm.git ``` ## Next Steps You are now ready to build [OpenVMM](./build_openvmm.md)! --- ### Guide Src Dev Guide Tests Fuzzing (`Guide/src/dev_guide/tests/fuzzing.md`) # Fuzzing in OpenVMM Fuzzing infrastructure in OpenVMM is based on the excellent [cargo-fuzz](https://github.com/rust-fuzz/cargo-fuzz) project, which makes it super easy to get up-and-running with fuzzing in Rust projects. Under-the-hood, `cargo-fuzz` hooks into LLVM's [libFuzzer](https://www.llvm.org/docs/LibFuzzer.html) to do the actual fuzzing. ```admonish important Fuzzing only works on **Linux**. libfuzzer-sys doesn't support Windows. On **aarch64**, set `RUSTFLAGS="-Ctarget-feature=+lse,+neon"` before any cargo-fuzz command, or builds will fail with atomics errors. ``` OpenVMM fuzzers target several categories of code: - **Chipset devices** (battery, CMOS/RTC, IDE) — PIO, MMIO, and PCI config interfaces exposed to guests - **VMBus devices** (storvsp) — VMBus channel protocol and SCSI command processing - **Driver stacks** (NVMe) — driver-side fuzzing against fuzzed device responses - **Unsafe abstractions** (scsi_buffers, guestmem, sparse_mmap) — safe API surface over unsafe internals - **Protocol parsers** (UEFI NVRAM, mesh ttrpc, UCS-2) — parsing and validation of structured data --- ### Guide Src Dev Guide Tests Incubator (`Guide/src/dev_guide/tests/incubator.md`) # Incubator Incubator runs cross-compiled test executables inside a QEMU-emulated Linux environment when the host cannot provide the required architecture or hardware model. ## When to use Incubator Use Incubator for tests that need hardware behavior outside the normal OpenVMM host test environment. QEMU TCG is significantly slower than native execution, so Incubator is reserved for tests that need its emulated platform. ## Execution model Incubator introduces an outer VM around the normal Petri test process: ```text x64 Linux host `- Incubator process `- QEMU TCG AArch64 Linux VM (L1) |- Pipette agent `- VMM test executable `- OpenVMM using KVM `- test VM (L2) ``` The VMM test executable is cross-compiled for `*-unknown-linux-musl`. It runs inside the QEMU VM and starts OpenVMM there. OpenVMM then uses the emulated KVM interface to create the test VM. Incubator is therefore not an OpenVMM backend. It is a Cargo target runner that places the existing test executable in a machine capable of running it. ## Direct invocation The binary also has a direct CLI for debugging the runner itself: ```bash cargo run -p incubator -- \ --profile petri/incubator/profiles/aarch64-tcg-pcie.toml \ --share path/to/shared-root \ --map-command-path \ path/to/shared-root/test-binary ``` Direct use requires a suitable kernel, initrd, Pipette binary, QEMU, and shared test artifacts. Running through `cargo xflowey vmm-tests-run` is preferred because Flowey resolves and connects those inputs automatically. ## Output and failures Incubator writes a per-process serial log named `incubator-serial..log` under the test output directory. Guest command stdout and stderr flow through Pipette to nextest. When a run fails, check the serial log first. It distinguishes a guest boot or Pipette startup failure from a failure in the nested VMM test itself. QEMU stderr is also captured and reported when the process exits. --- ### Guide Src Dev Guide Tests Make Imc Hive (`Guide/src/dev_guide/tests/make_imc_hive.md`) # IMC Hive Generator `make_imc_hive` regenerates the Windows registry hive used to configure Petri test guests during image preparation. ## Purpose The checked-in IMC hive is applied to an offline Windows installation by `prep_steps`. It describes settings that must exist before the prepared test guest first boots, most importantly the Pipette service. This is a generator for a repository artifact, not part of every VMM test run. Normal tests consume the already-generated hive. ## Platform requirement The implementation uses Windows offline-registry APIs and only runs on Windows. The workspace builds a stub on other platforms, but that stub exits with an unsupported-platform error. ## Regenerating the hive Run the generator from Windows and pass the output path as its single positional argument: ```powershell cargo run -p make_imc_hive -- petri\guest-bootstrap\imc.hiv ``` The tool removes an existing output file before saving the new hive. Regenerate the hive when changing Pipette's service path, startup account, dependencies, isolation policy, or initial crash-dump configuration. Changes that can be made after Pipette connects usually belong in a Petri test instead. ## How the hive is consumed `prep_steps` embeds or resolves the generated hive and makes it available to the temporary utility guest. Windows applies the hive to the offline SYSTEM configuration on the target test disk. The prepared disk is then cached for later VMM tests. ## Troubleshooting An offline-registry error generally means the destination cannot be created or the platform APIs rejected a key operation. A later guest failure can indicate that the hive was valid structurally but contains an incorrect service path or startup dependency. Inspect the prepared guest's SYSTEM hive and Windows service logs in that case. --- ### Guide Src Dev Guide Tests Opentmk (`Guide/src/dev_guide/tests/opentmk.md`) # OpenTMK `opentmk` is a configurable UEFI Test Microkernel payload for low-level Hyper-V and OpenHCL test scenarios. ## Distinction from `simple_tmk` The repository contains two separate microkernel systems: | System | Image | Selection model | Executor | | --- | --- | --- | --- | | `tmk/simple_tmk` | Bare-metal ELF | Static `tmk_tests` descriptors | Host `tmk_vmm` process | | `opentmk` | UEFI executable in a VHD | Embedded JSON configuration | Payload dispatches its own test | Use OpenTMK when the test needs its UEFI-based runtime, Hyper-V test contexts, multiple VTLs, or a configuration patched into the boot image. Use `simple_tmk` for small tests that should run through the multi-backend `tmk_vmm` executor. ## Build and package The supported local entry point builds the UEFI executable and packages it in a bootable VHD: ```bash cargo xflowey build-opentmk ``` ## Embedded configuration OpenTMK contains a fixed configuration region that host tooling can locate and patch without relinking the UEFI executable. Supply a JSON selection file to the build pipeline: ```bash cargo xflowey build-opentmk x86-64 \ --config path/to/test-config.json ``` The configuration identifies the backend and test to dispatch and carries test-specific parameters. The exact schema follows `opentmk_protocol`; use configurations from the test being developed as the starting point. If `--config` is omitted, the VHD contains the unconfigured OpenTMK executable. That artifact is useful as an intermediate build output but cannot select a test until a harness patches it. ## Guest lifecycle When firmware launches the image, OpenTMK: 1. Initializes UEFI services and the OpenTMK runtime. 2. Reads and validates the embedded configuration region. 3. Selects the registered backend. 4. Dispatches the configured test with its parameters. 5. Reports progress through its logging and assertion infrastructure. ## Running the VHD The Flowey output is a guest disk. Attach it to a UEFI VM using the environment required by the selected test. OpenTMK itself has no host command line. A generic OpenVMM launch has this shape: ```bash cargo run -p openvmm -- --uefi \ --vmbus-scsi id=scsi0 \ --disk memdiff:path/to/opentmk.vhd,on=scsi0 ``` Most OpenTMK tests require additional Hyper-V or OpenHCL configuration. Copy the VM setup from the owning test rather than assuming the generic launch is sufficient. ## Failures and diagnostics An unknown backend or test indicates a mismatch between the JSON configuration and the payload's registered dispatch table. A missing or malformed config usually fails before test code starts. Capture the guest serial output from the beginning of boot. It distinguishes UEFI/runtime initialization failures from assertions inside the selected test. Because the payload runs without a general-purpose OS, there is no Pipette or shell to inspect after a fatal failure. --- ### Guide Src Dev Guide Tests Opentmk Executor (`Guide/src/dev_guide/tests/opentmk_executor.md`) # OpenTMK Executor `opentmk_executor` is a UEFI payload that receives generated programs over a serial protocol and executes registered low-level operations. ## Intended use The executor provides a small, deterministic environment for an external test generator such as syzkaller. Instead of rebuilding the guest for each case, the host boots the executor once, sends encoded programs, and receives an acknowledgement or error packet for each case. It is distinct from the configuration-driven `opentmk` test payload. The executor has no embedded test selection. ## Build Install the architecture's Rust UEFI target, then build the binary. For x64: ```bash rustup target add x86_64-unknown-uefi cargo build -p opentmk_executor --target x86_64-unknown-uefi ``` The output is a UEFI executable. Package it in a bootable image using the same UEFI image tooling used by other guest test applications, or integrate it into the external harness that owns the VM and serial channel. ## Serial channel COM1 carries the executor protocol. After OpenTMK runtime initialization, the guest performs a three-way handshake with fixed SYN, SYN-ACK, and ACK values. Normal packets are framed with header and footer magic values and carry a serialized `OpenTMKPacket`. Do not use COM1 for human-readable logging while driving the protocol. The runtime uses another serial path for logs so arbitrary text does not corrupt packet framing. ## Packet sequence A host session follows this order: 1. Complete the serial handshake. 2. Send a `Configuration` packet selecting a grammar deserializer and providing its function mapping. 3. Wait for an `Ack` packet. 4. Send one or more `FuzzTest` packets containing encoded programs. 5. Read an `Ack` with the program result code after each test. 6. Treat an `Error` packet as a terminal protocol or execution failure. ## Memory safety boundary Decoded programs refer to generated memory through the `SafeMemoryMap` interface. Read operations place their result into that map rather than treating generated addresses as native pointers. Function implementations validate arity and variable types before issuing privileged operations. The environment still deliberately performs low-level I/O and hypercalls. Run it only in a disposable test VM controlled by the fuzzing harness. ## Failure behavior Bad handshake magic, packet framing, serialization, mappings, or packet order produces an `OpenTMKErrorPacket` when possible and ends the executor loop. Individual generated operations can return a nonzero acknowledgement code without necessarily invalidating the session. If the host appears to hang: - Verify that the UEFI payload reached OpenTMK runtime initialization. - Confirm both sides use the same packet and grammar definitions. - Check that COM1 is a clean binary channel with no terminal translation. - Ensure the host waits for configuration acknowledgement before sending a test case. ```admonish note title="See also" [OpenTMK](opentmk.md) describes the configuration-driven UEFI test payload. ``` --- ### Guide Src Dev Guide Tests Perf (`Guide/src/dev_guide/tests/perf.md`) # Performance Tests (burette) `burette` is a standalone binary that runs performance benchmarks against OpenVMM using the `petri` test framework. It measures boot time, memory overhead, and concurrent VM scaling behavior, producing JSON reports that can be compared across builds. Use Burette for developer-controlled experiments, report comparison, and self-contained remote runs. ## Prerequisites - Linux host with `/dev/kvm` access (or Windows with Hyper-V) - Built `openvmm` and `burette` built for the host target, and `pipette` built for `-unknown-linux-musl`. - Test kernel/initrd artifacts for the appropriate architecture installed to their magic paths. - `iperf3` package installed (for networking tests). For example, to download and build everything on an x64 Ubuntu system: ```bash cargo xflowey restore-packages sudo apt update sudo apt install iperf3 cargo build --release -p burette -p openvmm cargo build --release --target x86_64-unknown-linux-musl -p pipette ``` ## Architecture Each benchmark controls an OpenVMM process through Petri: ```text burette |- resolve host and guest artifacts |- launch OpenVMM | `- boot Linux test guest |- connect to Pipette in the guest |- run or observe the workload |- collect host and guest measurements `- stop the VM and write JSON results ``` Some tests rebuild the VM for every sample, while throughput tests can retain a warm VM and run repeated guest workloads. Read the benchmark description when comparing first-run and steady-state results. ## Running Tests ```admonish note In all of the below commands, replace `burette` with the path to your built burette executable, or `cargo run --release -p burette` to build and run burette within a local clone. ``` ### Boot time Measures launch-to-pipette-connect time using Linux direct boot: ```bash burette run --test boot-time -o report.json ``` Available profiles control the VM configuration: | Profile | Description | |---|---| | `standard` | Full device set, serial agent, shared memory | | `quiet-serial` | Like standard but suppresses kernel console | | `minimal` | Pipette-as-init, minimal devices, shared memory | | `minimal-private` | Same as minimal but with private memory (fastest) | ```bash # Use a specific profile burette run --test boot-time --profile minimal -o report.json # Custom iteration count and guest RAM burette run --test boot-time --iterations 20 --mem-mb 1024 ``` ### Memory overhead Boots a single VM and measures host-side memory consumption for the openvmm process tree: ```bash burette run --test memory -o memory.json ``` On Windows, the available process accounting differs and VMM overhead follows the private-memory measurement. ### Network throughput Measures TCP throughput (Gbps) and UDP packet rate (pps) using iperf3 with a linux_direct VM, an erofs tool image, and Consomme networking: ```bash burette run --test network -o network.json # Test with virtio-net instead of VMBus burette run --test network --nic virtio-net -o network.json ``` The report records TCP throughput and UDP packet-rate metrics. Keep the NIC, network backend, host CPU placement, and MTU constant between compared runs. Compare shared vs. private memory overhead: ```bash burette run --test memory --profile minimal -o shared.json burette run --test memory --profile minimal-private -o private.json burette compare shared.json private.json ``` ### Scale boot Launches N VMs concurrently to measure boot time under contention and per-VM memory overhead. Default sweep: N = 1, 2, 4, 8, 16, 32, 64. ```bash # Full geometric sweep (auto-stops at 90% host memory) burette run --test scale-boot --mem-mb 256 -o scale.json # Single data point burette run --test scale-boot --vms 16 --mem-mb 256 # Custom sweep burette run --test scale-boot --vms 1,2,4,8 --max-vms 32 ``` Per-N metrics include `scale_{N}_mean_boot_ms`, `scale_{N}_p99_boot_ms`, `scale_{N}_last_ready_ms`, `scale_{N}_per_vm_memory_mib`, and others. ### Disk I/O Measures block I/O throughput (MiB/s) and IOPS using fio in a linux_direct VM with an erofs tool image and a data disk. Supports virtio-blk and storvsc (synthetic SCSI) backends: ```bash # Virtio-blk with RAM-backed disk (measures virtio overhead) burette run --test disk-io -o disk.json # Storvsc backend burette run --test disk-io --disk-backend storvsc -o disk.json # File-backed disk for realistic host I/O latency burette run --test disk-io --data-disk /tmp/test.raw --data-disk-size-gib 8 ``` Reported metrics per backend: - `fio_{backend}_seq_read_bw` / `fio_{backend}_seq_write_bw` — sequential bandwidth (MiB/s) - `fio_{backend}_rand_read_bw` / `fio_{backend}_rand_write_bw` — random bandwidth (MiB/s) - `fio_{backend}_rand_read_iops` / `fio_{backend}_rand_write_iops` — random IOPS By default a RAM-backed disk is used to isolate virtio/storvsc overhead without host filesystem noise. Pass `--data-disk` with a path on fast storage (e.g., NVMe) for end-to-end latency measurements. ### virtio-fs The virtio-fs benchmark boots a minimal Linux guest, mounts a host-exported filesystem, and runs `fio` against a test file through the virtio-fs data path: ```bash burette run --test virtio-fs -o virtio-fs.json ``` Use `--virtiofs-file-size-mib` to change the generated test-file size. Ensure the host filesystem has enough free space and keep its storage medium constant between runs. ## Test lifecycle and cleanup Burette creates a Petri log source, resolves only the artifacts required by the selected test, and then runs that benchmark's harness. VM teardown is part of the measured test's cleanup path. A failed workload can therefore leave useful OpenVMM and guest logs even when no JSON sample is recorded. Use `--log-dir` to retain logs outside the default `vmm_test_results/burette` location. On Linux, `--perf-dir` can retain host performance traces for supported tests. ## Comparing Reports ```bash burette compare baseline.json candidate.json ``` Prints a table of deltas and percentage changes for each metric. Optionally write the comparison to JSON: ```bash burette compare baseline.json candidate.json -o diff.json ``` The comparison displays absolute and percentage deltas. A positive percentage is not universally better: higher throughput is desirable, while higher boot latency or memory overhead is a regression. Interpret the metric's unit and direction before drawing a conclusion. ## Remote Deployment Package all binaries and artifacts into a self-contained tarball: ```bash burette package -o burette_bundle.tar.gz ``` On the remote machine: ```bash tar xzf burette_bundle.tar.gz cd burette_bundle VMM_TESTS_CONTENT_DIR=$PWD ./burette run -o report.json ``` The bundle includes `burette`, `openvmm`, `pipette`, the test kernel, and initrd — no Rust toolchain or repo checkout needed. The remote host still needs a supported virtualization interface and any host facilities required by the selected network, storage, or filesystem backend. ## Running All Tests Omit `--test` to run every test: ```bash burette run -o full_report.json ``` ## JSON Report Format Reports are JSON files with git revision info, timestamps, and per-metric statistics: ```json { "git_revision": "abc123", "git_commit_date": "2026-03-18T00:00:00Z", "date": "2026-03-18T01:00:00Z", "results": [ { "name": "boot_time_ms", "unit": "ms", "iterations": 10, "mean": 126.3, "std_dev": 1.5, "min": 124.4, "max": 128.1 } ] } ``` The report records source revision and time alongside per-metric summary statistics. Preserve the complete report rather than extracting only the mean; sample count, standard deviation, minimum, and maximum help distinguish a real change from host noise. ## Measurement practice For useful before/after comparisons: 1. Use the same host, power policy, kernel, firmware, OpenVMM features, and guest artifacts. 2. Stop unrelated CPU, disk, and network workloads. 3. Use release builds for representative performance. 4. Run enough iterations to expose variance. 5. Keep raw reports and logs for both revisions. 6. Repeat surprising results before attributing them to a code change. The scale test deliberately applies host-memory limits. A truncated sweep is not directly comparable with a sweep that reached larger VM counts. ## Troubleshooting - A missing-artifact error names the Petri artifact that could not be resolved; build or restore that exact target rather than substituting a host binary. - A launch-to-Pipette timeout usually means the guest failed to boot or the selected transport did not connect. Read the Petri and guest serial logs. - Network failures can come from the helper process, host backend, guest tool image, or firewall. Verify a single warm-up run before collecting samples. - File-backed disk and virtio-fs results are sensitive to host caching and free space. Use a dedicated fast filesystem when measuring end-to-end I/O. - A report with large variance is not made reliable by `compare`; remove the source of noise or increase the iteration count. --- ### Guide Src Dev Guide Tests Pipette (`Guide/src/dev_guide/tests/pipette.md`) # Pipette Guest Agent `pipette` is the in-guest agent through which Petri tests execute commands and perform operating-system operations. ## Role in a VMM test Petri divides guest control between a host client and the Pipette agent: ```text VMM test process `- Pipette client `- VSocket or TCP connection `- Pipette agent in the guest |- execute process |- transfer files |- collect output `- shut down the guest ``` The test obtains a `PipetteClient` after the VM boots. Commands created through that client are serialized to the guest, executed there, and represented by a remote child process whose status and standard streams are returned to the host. Pipette is test infrastructure, not an OpenVMM management protocol. Its API and wire format are shared by Petri components and can change with the repository. ## Supported guests Pipette runs in Linux and Windows guests on the architectures built by the VMM test pipeline. The binary is compiled for the guest, not necessarily for the developer's host. The default transport is VSocket. TCP is used when the guest configuration has no usable VSocket path, including the current Windows `no-vmbus` image. Windows builds also accept `--service`. The generated test-image registry hive uses this mode so Pipette starts automatically as `LocalSystem`. ## Linux PID 1 mode Some Linux direct-boot configurations use Pipette as `rdinit`. When its process ID is 1, Pipette performs the minimal init responsibilities needed by the test guest before starting the request agent. These include setting up essential mounts and reaping child processes. This mode deliberately does not try to become a general init system. The guest image should contain only the services required by its test workload. ## Connection lifecycle At startup, Pipette initializes its selected transport and establishes the protocol connection. It then serves requests until the host disconnects. A disconnect is not fatal: the process creates a new agent connection and waits for the host again. The exact request set is defined by the `pipette_protocol` crate. Tests should normally use `pipette_client` rather than manually constructing protocol messages. ## Building Pipette The supported path is to let `cargo xflowey vmm-tests-run` discover and build the guest artifact required by the selected tests: ```bash cargo xflowey vmm-tests-run --filter "test(my_test)" --build-only ``` For a focused build, choose a target matching the guest image. For example: ```bash cargo build -p pipette --target x86_64-unknown-linux-musl ``` ```bash cargo build -p pipette --target x86_64-pc-windows-msvc ``` ```admonish warning A host-native `cargo build -p pipette` is only useful when the host target also matches the intended guest. Petri resolves Pipette by guest OS and architecture, so placing a binary under the wrong target directory does not satisfy the artifact requirement. ``` ## Image integration Linux test images either include Pipette in an initrd or expose it through a shared test-content disk. Windows images are prepared with a registry hive that defines the Pipette service and points it at the injected executable. The `prep_steps` tool prepares reusable Windows images. The `make_imc_hive` tool regenerates the source hive when the service definition changes. ## Troubleshooting If a test times out waiting for Pipette: 1. Check the guest serial log for Pipette startup or panic messages. 2. Confirm that the binary matches the guest architecture and operating system. 3. Confirm that the configured transport matches the VM's available VSocket or TCP path. 4. For Linux direct boot, verify that the kernel command line starts the intended init and that essential mounts succeeded. 5. For Windows, verify that the service image path matches the drive containing `pipette.exe`. Incubator additionally waits for a `PIPETTE READY` serial marker before opening its forwarded TCP connection. A missing marker generally means the outer guest failed before Pipette initialized. --- ### Guide Src Dev Guide Tests Prep Steps (`Guide/src/dev_guide/tests/prep_steps.md`) # Windows Test Image Preparation `prep_steps` prepares reusable Windows disks for Petri VMM tests by modifying them from a temporary OpenVMM guest. ## Why preparation is separate Windows guest images require changes that are awkward or unsafe to perform by editing the VHD directly. `prep_steps` boots a known Windows utility VM, attaches a copy of the target image as a data disk, and performs the changes through Windows itself. Keeping this work outside individual tests has two benefits: - Every test starts from the same prepared image. - Expensive service and driver installation can be cached and reused. The tool remains intentionally small. Changes that can be made after the test guest boots should be performed through Pipette in the test instead. ## Normal use You normally do not invoke `prep_steps` yourself. Artifact discovery in `cargo xflowey vmm-tests-run` determines whether a selected test needs a prepared Windows image, builds the helper, and runs the required preparation mode before nextest starts. To force regeneration while developing preparation logic, disable reuse: ```bash cargo xflowey vmm-tests-run \ --no-reuse-prepped-vhds \ --filter "test(my_windows_test)" ``` The resulting images and preparation logs live below the VMM test content directory selected by `--dir`. ## Standard image flow The standard path performs the following work: 1. Resolve the utility boot image, source test VHD, firmware, OpenVMM, Pipette, and output directory. 2. Copy the source disk to its prepared output path. 3. Randomize disk and partition identifiers so the utility VM can attach the source and result disks without Windows treating them as duplicates. 4. Boot the utility Windows VM with the result disk attached. 5. Connect to the utility VM's Pipette agent. 6. Apply the IMC registry hive to the offline Windows installation on the attached disk. 7. Shut down the utility VM and retain the prepared disk. The hive configures Pipette as an automatically started Windows service and enables the VMBus behavior expected by isolated test guests. ## Outputs and caching Prepared images are derived artifacts. Reusing one is correct only while its source image and preparation inputs remain compatible. Use `--no-reuse-prepped-vhds` after changing: - `prep_steps` behavior. - The Pipette service hive. - Drivers installed by the `no-vmbus` mode. - The source Windows image assumptions. Do not edit a shared source VHD in place. Preparation works on a result copy so tests remain repeatable. ## Troubleshooting Preparation failures usually occur before the actual nextest run. Check the preparation log for the stage that failed: - Artifact errors indicate a missing source VHD, firmware, Pipette, or driver build. - Disk errors can indicate a corrupt image or failure while rewriting disk identifiers. - A Pipette timeout indicates that the utility boot VM did not start its agent. - Windows servicing failures in `no-vmbus` usually identify a missing or incompatible `virtio-win` file. When cross-running Windows tests from WSL, Flowey reports if the output directory must be placed on a Windows filesystem. --- ### Guide Src Dev Guide Tests Simple Tmk (`Guide/src/dev_guide/tests/simple_tmk.md`) # simple_tmk `simple_tmk` is the bare-metal guest payload containing the repository's basic Test Microkernel processor and interrupt tests. ## Runtime model The payload is a `no_std`, `minimal_rt` executable. It has no bootloader, firmware interface, process model, or command line. `tmk_vmm` loads its ELF directly, initializes the processor, and selects one test by index. The shared `tmk_core` runtime provides: - A test entry point and panic handler. - Guest-to-host logging and completion commands. - Scoped interrupt-handler registration. - Architecture-specific helpers such as x86 MSR access. The payload is useful for code that must touch processor state directly. Use a normal VMM test when the behavior requires firmware, devices, or a guest OS. ## Building Build the x86_64 payload with the same command emitted by Petri's artifact resolver: ```bash RUSTC_BOOTSTRAP=1 cargo build -p simple_tmk \ --config openhcl/minimal_rt/x86_64-config.toml ``` Build AArch64 with: ```bash RUSTC_BOOTSTRAP=1 cargo build -p simple_tmk \ --config openhcl/minimal_rt/aarch64-config.toml ``` The output is an ELF consumed by `tmk_vmm`. Running it as a host process is not supported; non-`minimal_rt` builds contain only a rejecting stub. Flowey builds the correct payload automatically when selected Petri tests require a `SIMPLE_TMK_*` artifact. ## Defining a test A test is a function annotated with `#[tmk_test]`: ```rust,ignore #[tmk_test] fn example(_: TestContext<'_>) { log!("example started"); assert_eq!(2 + 2, 4); } ``` The macro emits a static descriptor into the ELF `tmk_tests` section. The descriptor gives `tmk_vmm` the test name, entry point, and flags needed for discovery and execution. Tests that intentionally fault can be marked as expected failures: ```rust,ignore #[tmk_test(expected_failure, linux_only)] fn expected_abort(_: TestContext<'_>) { // Trigger the architecture condition under test. } ``` ## Scoped interrupt handlers On x86_64, `TestContext` exposes a `Scope`. A test can create a nested scope, install an ISR for a vector, enable interrupts, and let the scope restore its handler state before returning. This keeps one test's interrupt setup from leaking into another test implementation. ## Running and listing After building both binaries: ```bash target/debug/tmk_vmm --tmk \ target/x86_64-unknown-none/debug/simple_tmk --list ``` Run one test by passing its discovered name: ```bash target/debug/tmk_vmm --tmk \ target/x86_64-unknown-none/debug/simple_tmk ud2 ``` Output appears in the `tmk_vmm` tracing stream. A panic becomes a structured protocol message when the runtime remains operational; an earlier processor fault is reported from the host VP loop. ## Adding coverage safely When adding a test: 1. Keep the initial machine assumptions explicit. 2. Install handlers before enabling or triggering interrupts. 3. Bound polling loops so a missing hardware event becomes a failure. 4. Restore mutable processor state when later operations in the same test need a known configuration. 5. Run through every backend whose behavior the test claims to cover. Tests share the image but receive fresh VM state from `tmk_vmm`, so global state does not need to survive between test cases. ```admonish note title="See also" [TMK VMM](tmk_vmm.md) documents the host executor, hypervisor backends, and result protocol. ``` --- ### Guide Src Dev Guide Tests Test Igvm Agent Rpc Server (`Guide/src/dev_guide/tests/test_igvm_agent_rpc_server.md`) # IGVM Agent Test Server `test_igvm_agent_rpc_server` is a Windows-only test double for the host IGVM agent RPC interface used by guest attestation flows. ## Purpose Attestation tests need deterministic host responses for conditions that are difficult to reproduce with a real service. This executable hosts the expected Windows RPC facade and can install predefined response plans before accepting requests. It is test infrastructure, not an IGVM agent suitable for deployment. The implementation and scenario names follow the in-tree tests rather than a stable external API. ## Normal orchestration For tests that declare this artifact, Flowey: 1. Builds the server for Windows MSVC. 2. Copies it into the VMM-test content directory. 3. Starts it before nextest and redirects output to `test_igvm_agent_rpc_server.log`. 4. Verifies that it did not exit immediately. 5. Runs the selected attestation tests. 6. Terminates the background server during cleanup. This lifecycle is part of `cargo xflowey vmm-tests-run`; most developers do not need to start the process themselves. ## Direct use For focused Windows debugging, build and launch it directly: ```powershell cargo build -p test_igvm_agent_rpc_server ` --target x86_64-pc-windows-msvc ``` ```powershell .\target\x86_64-pc-windows-msvc\debug\test_igvm_agent_rpc_server.exe ` --test-config AkCertRequestFailureAndRetry ``` The server continues running while it services RPC requests. Stop it after the test so it does not affect a later scenario. Some local test paths support an explicit autostart environment variable. Use the instructions emitted by the test or its local-autostart helper rather than running multiple server instances. ## Troubleshooting - An immediate unsupported-platform failure means the executable was launched outside Windows. - A startup failure can indicate that another instance owns the RPC endpoint. - Check `test_igvm_agent_rpc_server.log` before guest logs when no host request reaches the planned scenario. - Confirm that the selected test configuration matches the scenario expected by the Petri test. - Stop stale instances before rerunning a different plan. --- ### Guide Src Dev Guide Tests Tmk Vmm (`Guide/src/dev_guide/tests/tmk_vmm.md`) # TMK VMM `tmk_vmm` is a small host-side VMM that discovers and executes tests from a Test Microkernel ELF image. ## Why use a TMK A full OpenVMM VM includes firmware, an operating system, a broad device model, and guest-control infrastructure. That is useful for integration coverage but makes a poor fixture for a focused processor or hypervisor-interface test. `tmk_vmm` creates only the VM state needed by the Test Microkernel: - A single virtual processor. - 4 MiB of guest RAM. - Architecture-specific initial register state. - A fixed MMIO command address for guest-to-host reporting. This keeps failures close to the hypervisor behavior under test. `tmk_vmm` does not support general-purpose VMs. ## Host and guest split ```text tmk_vmm host process |- parse the TMK ELF |- enumerate test descriptors |- create a minimal partition and VP |- handle guest MMIO commands `- report test results simple_tmk guest image |- initialize the minimal runtime |- invoke one selected test `- send log, panic, or completion commands ``` The host reloads the TMK and creates fresh execution state for each selected test. ## Building the executor Build a native executor with Cargo: ```bash cargo build -p tmk_vmm ``` The available hypervisor backend is determined by the build platform and guest architecture. Flowey builds additional native and static Linux variants for VMM-test artifacts. ## Running a TMK List tests without creating a VM: ```bash target/debug/tmk_vmm --tmk path/to/simple_tmk --list ``` Run every test using the automatically selected hypervisor: ```bash target/debug/tmk_vmm --tmk path/to/simple_tmk ``` Pass test names as positional arguments to run a subset: ```bash target/debug/tmk_vmm --tmk path/to/simple_tmk boot apic_timer ``` ## ELF test discovery The input is a 64-bit ELF containing a `tmk_tests` section. Each descriptor stores: - A pointer and length for the UTF-8 test name. - The guest entry point. - An `expected_failure` flag. - A `linux_only` flag. `tmk_vmm --list` parses and relocates this metadata without entering the guest. When running a test, the loader maps the ELF, prepares page tables and initial registers, and passes a `StartInput` structure to the TMK. ## Result protocol The host reserves MMIO address `0xffff0000`. The guest writes a pointer to a `tmk_protocol::Command` at that address. Commands can: - Log a UTF-8 string held in guest memory. - Report a panic with message, file, and line. - Complete explicitly with a success value. If the VP halts or faults before completion, `tmk_vmm` reports the halt reason and available register state. A normal failure makes the process exit nonzero. For a descriptor marked `expected_failure`, a failure or fault is success and a normal pass is an error. ## Troubleshooting - `no hypervisor available` means no compiled native backend passed its availability check. Verify host permissions or pass `--hv` explicitly. - An empty `--list` result usually means the image was not built with the TMK linker configuration or contains no registered tests. - A protocol error indicates that guest pointers or command contents were not valid for the mapped memory. - A `Faulted` result includes processor state; use it before adding logging to code that may fail before the command channel works. ```admonish note title="See also" [simple_tmk](simple_tmk.md) describes the guest payload and test authoring model. ``` --- ### Guide Src Dev Guide Tests Tpm Guest Tests (`Guide/src/dev_guide/tests/tpm_guest_tests.md`) # TPM Guest Test Utility `tpm_guest_tests` is an in-guest CLI for probing TPM 2.0 and OpenHCL attestation behavior from Petri tests. ## Execution environment The binary runs inside a Linux or Windows guest with a visible TPM, normally an OpenVMM or OpenHCL vTPM. It does not operate on a VMGS file and is not a host-side TPM provisioning utility. On Unix, it opens `/dev/tpmrm0` first and falls back to `/dev/tpm0`. On Windows, it sends commands through TPM Base Services. The tool adapts those transports to the shared `tpm_lib` command implementation. Petri usually copies the architecture-appropriate executable into the guest and runs it through Pipette. You can also invoke it from an interactive guest shell. ## Building and deployment Let VMM-test artifact discovery build the correct Linux or Windows guest binary: ```bash cargo xflowey vmm-tests-run \ --build-only \ --filter "test(tpm)" ``` A focused Linux build can be produced with: ```bash cargo build -p tpm_guest_tests \ --target x86_64-unknown-linux-musl ``` The binary must still be copied into a guest with the corresponding architecture and OS before it is useful. ## Exit status and diagnostics Successful commands exit with status zero. CLI errors, TPM transport failures, malformed responses, oversized data, and comparison failures print an error chain and exit nonzero. NV reads include a bounded hexadecimal and ASCII dump to aid diagnosis without flooding the test log. If opening the TPM fails, confirm that the VM configuration exposed a vTPM and that the guest user can access its device or TBS context. An AK certificate may appear asynchronously during attestation initialization; use comparison retry only when the test expects that transition. --- ### Guide Src Dev Guide Tests Unit (`Guide/src/dev_guide/tests/unit.md`) # Unit Tests ```admonish tip Note: We recommend using [cargo-nextest](https://nexte.st/) to run unit / VMM tests. It is a significant improvement over the built-in `cargo test` runner, and is the test runner we use in all our CI pipelines. You can install it locally by running: `cargo install cargo-nextest --locked` See the [cargo-nextest](https://nexte.st/) documentation for more info. ``` Unit tests test individual functions or components without pulling in lots of ambient infrastructure. In Rust, these are usually written in the same file as the product code--this ensures that the test has access to any internal methods or state it requires, and it makes it easier to ensure that tests and code are updated at the same time. A typical module with unit tests might look something like this: ```rust fn add_5(n: u32) -> u32 { n + 5 } #[cfg(test)] mod tests { #[test] fn test_add_5() { assert_eq!(add_5(3), 8); } } ``` In the OpenVMM repo, all the unit tests are run on every pull request, on an arbitrary build machine. As a result of this approach, it's important that unit tests: - run quickly - do not affect the state of the machine that runs them - do not take a dependency on machine configuration - e.g: no root/administrator access or virtualization requirement We may loosen these guidelines over time if it becomes necessary. You can also mark tests with `#[ignore]` if they do not meet these guidelines but are useful for manual testing. See the [unit testing section](https://doc.rust-lang.org/rust-by-example/testing/unit_testing.html) in "Rust by example" for more details. ## Doc tests Rust has another type of unit tests known as doc tests. These are unit tests that are written in the API documentation comments of public functions. They will be run automatically along with the unit tests, so the same guidelines apply. When do you choose a doc test over a unit test? Doc tests can only access public functionality, and they are intended to document the usage of a function or method, not to exhaustively check every case. So write doc tests primarily as examples for other developers, and rely on unit tests for your main coverage. An example might look like this: ```rust /// Adds 5 to `n`. /// /// ``` /// assert_eq!(mycrate::add_5(3), 8); /// ``` pub fn add_5(n: u32) -> u32 { n + 5 } ``` See the [documentation testing section](https://doc.rust-lang.org/rust-by-example/testing/doc_testing.html) in Rust by example for more info. --- ### Guide Src Dev Guide Tests Vmm (`Guide/src/dev_guide/tests/vmm.md`) # VMM Tests The OpenVMM repo contains a set of "heavyweight" VMM tests that fully boot a virtual machine and run validation against it. Unlike Unit tests, these are all centralized in a single top-level `vmm_tests` directory. The OpenVMM PR and CI pipelines will run the full test suite on all supported platforms; you'd typically run only the tests relevant to the changes you're working on. ## Scheduled Pipelines The **OpenVMM Burn In** workflow repeatedly runs Windows x64 VMM tests using artifacts from a successful CI run on `main` to measure failure frequency. The **OpenVMM Patina Nightly** workflow runs the release check-in gates using the configured `mu_msvm` RELEASE ClangPDB Patina firmware for standalone OpenVMM UEFI tests and newly built OpenHCL images. You can also start it manually using GitHub Actions' **Run workflow**. ## Writing VMM Tests To streamline the process of booting and interacting with VMs during VMM tests, the OpenVMM project uses an in-house test framework/library called `petri`. The library does not yet have a stable API, so at this time, the best way to learn how to write new VMM tests is by reading through the existing corpus of tests (start with vmm_tests/vmm_tests/tests/tests/multiarch.rs), as well as reading through `petri`'s rustdoc-generated API docs. The tests are currently generated using a macro (`#[vmm_test]`) that allows the same test body to be run in a variety of scenarios, with different guest operating systems, firmwares, and VMMs (including Hyper-V, which is useful for testing certain OpenHCL features that aren't supported when using OpenVMM as the host VMM). ### "heavy" tests The global [nextest.toml](https://github.com/microsoft/openvmm/blob/main/.config/nextest.toml) configures how tests run in our test environments. The `default` and `ci` profiles control things like timeouts, and how many resources we allocate to a given test. The number of required threads is a fuzzy requirement relative to the number of VPs consumed by the VM under test, the amount of memory your test needs, the host test framework (petri itself), and so on. We have some pre-defined overrides that perform filterset matching on test name. These overrides are curated to balance individual trial (test case) performance against overall concurrency on engineers' local machines and in CI. Put these special words in your test to opt in to that override: - `heavy` - if your test is heavier than the typical vmm_test. E.g., your test explicitly requests 16 virtual processors. - `very_heavy` if your test is heavier than a `heavy` test. E.g., your test explicitly requests 32 virtual processors. ### "unstable" tests If a test is not yet reliable enough to gate PRs, mark it `unstable`. Every `unstable` marker requires a `reason` string documenting why the test is unstable (ideally referencing a tracking issue). To mark a single variant unstable, wrap it in `unstable(reason = "...", )`: ```rust,ignore #[vmm_test( // unstable variant: unstable(reason = "flaky on aarch64 in CI (microsoft/openvmm#1234)", hyperv_openhcl_uefi_aarch64(vhd(windows_11_enterprise_aarch64))), // other reliable variants: hyperv_openhcl_uefi_aarch64(vhd(ubuntu_2404_server_aarch64)) // ... )] async fn my_test(config: PetriVmBuilder) -> anyhow::Result<()> { // ... } ``` To mark all variants of the test unstable, apply `unstable(reason = "...")` as a test-wide attribute alongside `configs(...)`: ```rust,ignore #[vmm_test_with(unstable(reason = "flaky in CI (microsoft/openvmm#1234)"), configs( hyperv_openhcl_uefi_aarch64(vhd(windows_11_enterprise_aarch64)), hyperv_openhcl_uefi_aarch64(vhd(ubuntu_2404_server_aarch64)) // ... ))] async fn my_test(config: PetriVmBuilder) -> anyhow::Result<()> { // ... } ``` Unstable tests run in the same CI job as stable tests. In CI the `PETRI_IGNORE_UNSTABLE_FAILURES` environment variable is set, so when an unstable test fails the CI run passes with a warning. Outside of CI that variable is not set by default, so an unstable test failure is reported as a failure like any other test. To promote an unstable test to stable, remove the `unstable(...)` wrapper. This is a single-place change — no CI or configuration updates are required. To suppress failures from `unstable` tests when running locally (matching the CI behavior), set the following environment variable: `PETRI_IGNORE_UNSTABLE_FAILURES=1` ### "ignore" tests If a test (or a specific variant) should not run by default at all — for example it depends on functionality that isn't ready yet — mark it `ignore`, analogous to libtest's `#[ignore]`. Like `unstable`, every `ignore` marker requires a `reason` string (ideally referencing a tracking issue). Ignored tests are skipped unless explicitly requested. To ignore a single variant, wrap it in `ignore(reason = "...", )`: ```rust,ignore #[vmm_test( ignore(reason = "OpenVMM VBS boot on Ubuntu is unreliable (microsoft/openvmm#2139)", openvmm_openhcl_uefi_x64[vbs](vhd(ubuntu_2504_server_x64))), hyperv_openhcl_uefi_aarch64(vhd(ubuntu_2404_server_aarch64)) // ... )] async fn my_test(config: PetriVmBuilder) -> anyhow::Result<()> { // ... } ``` To ignore all variants of the test, apply `ignore(reason = "...")` as a test-wide attribute alongside `configs(...)`: ```rust,ignore #[vmm_test_with(ignore(reason = "needs functionality not yet implemented (microsoft/openvmm#1234)"), configs( hyperv_openhcl_uefi_aarch64(vhd(windows_11_enterprise_aarch64)), hyperv_openhcl_uefi_aarch64(vhd(ubuntu_2404_server_aarch64)) // ... ))] async fn my_test(config: PetriVmBuilder) -> anyhow::Result<()> { // ... } ``` ### Nested Tests Petri supports writing nested tests by defining a function to create the L1 VM host using Petri and annotating the L2 test definition as follows: ```rust,ignore #[vmm_test( nested( (my_vm_host, l1_config), (NEXTEST_ARCHIVE_WITH_NESTED_TEST, "test_binary", "test::module::path", l2_config), ), )] async fn my_test(config: PetriVmBuilder) -> anyhow::Result<()> { // ... } async fn my_vm_host(config: PetriVmBuilder) -> anyhow::Result<()> { let (vm, agent) = config .with_nested_virt() // other l1 configuration .run().await?; // use the pipette agent to configure the l1 if necessary // run the nested test, optionally with custom environment variables. vm.run_nested_test(&agent, Default::default()).await?; // power off and clean up agent.power_off().await?; vm.wait_for_clean_teardown().await?; Ok(()) } ``` ## Running VMM Tests (Flowey) The easiest way to run VMM tests locally is `cargo xflowey vmm-tests-run`. It automatically discovers required artifacts, builds dependencies, and runs your tests in a single command. To run a **specific test** (or set of tests), use `--filter` with a [nextest filter](https://nexte.st/docs/filtersets/) expression: ```bash cargo xflowey vmm-tests-run --filter "test(my_test_name)" --dir /tmp/vmm-tests-run ``` ### Targeting a Platform By default, `vmm-tests-run` builds for the current host. Use `--target` to build for a different platform. The supported targets are: | Target | Description | |--------|-------------| | `windows-x64` | Windows x86_64 (Hyper-V / WHP) | | `windows-aarch64` | Windows ARM64 (Hyper-V / WHP) | | `linux-x64` | Linux x86_64 | **Cross-compiling for Windows from WSL2** is fully supported, you can build and run Windows VMM tests directly from your WSL2 shell. This requires the cross-compilation environment to be set up first. Then target Windows as usual: ```bash cargo xflowey vmm-tests-run --target windows-x64 ``` Most tests work with the default output directory (on the WSL filesystem), because their disk images are either streamed on demand or opened as plain files (fixed VHD1, VMGS, ISO), which work fine over the `\\wsl$` path. You only need to override `--dir` to a Windows filesystem path (a DrvFs mount like `/mnt/d/...`) when the selected tests use disk images that require a Windows filesystem — namely **Hyper-V tests** (their VHDs are attached to a real Hyper-V VM) or tests using **VHDX / dynamic VHD1** images (opened via the Windows virtual-disk mount API): ```bash cargo xflowey vmm-tests-run --target windows-x64 --dir /mnt/d/vmm_tests ``` `vmm-tests-run` detects this automatically and will tell you if `--dir` is required for your selection. For full cross-compilation setup instructions, see [Cross Compiling for Windows](../getting_started/cross_compile.md). When running Hyper-V tests, your user account must be a member of the Hyper-V Administrators group. To see all available options: `cargo xflowey vmm-tests-run --help`. ### Lazy Fetch (Default) By default, VHD/ISO disk images are streamed on demand via HTTP and cached locally in a SQLite database. This avoids multi-GB upfront downloads and significantly speeds up the dev inner loop once the cache is warm. Lazy fetch is automatically disabled for Hyper-V tests, which require local files. To force all images to be downloaded upfront, pass `--no-lazy-fetch`: ```bash cargo xflowey vmm-tests-run --filter "test(my_test)" --dir /tmp/vmm-tests-run --no-lazy-fetch ``` ## Running VMM Tests (Manual) ```admonish tip Note: We recommend using [cargo-nextest](https://nexte.st/) to run unit / VMM tests. It is a significant improvement over the built-in `cargo test` runner, and is the test runner we use in all our CI pipelines. You can install it locally by running: `cargo install cargo-nextest --locked` See the [cargo-nextest](https://nexte.st/) documentation for more info. ``` You can directly invoke `cargo test` or `cargo nextest` to run the vmm tests manually. Unlike Unit Tests, VMM tests may rely on additional external artifacts in order to run. e.g: Virtual Disk Images, pre-built OpenHCL binaries, UEFI / PCAT firmware blobs, etc. As such, the first step in running a VMM test is to ensure you have acquired all external test artifacts it may depend upon. The VMM test infrastructure does not automatically fetch / rebuild necessary artifacts unless you are using [flowey](#running-vmm-tests-flowey). However, the test infrastructure is designed to report clear and actionable error messages whenever a required test artifact cannot be found, which provide detailed instructions on how to build / acquire the missing artifact. Some dependencies can only be built on Linux (OpenHCL and Linux pipette, for example). If you are building on Linux and want to run Windows guest tests, pipette will need to be [cross compiled for Windows](#linux-cross-compiling-pipetteexe). ```admonish warning `cargo nextest run` won't rebuild any of your changes. Make sure you `cargo build` or `cargo xflowey build-igvm [RECIPE]` first! ``` VMM tests are run using standard Rust test infrastructure, and are invoked via `cargo test` / `cargo nextest`. ```bash cargo nextest run -p vmm_tests [TEST_FILTERS] ``` For example, to run a simple VMM test that simply boots using UEFI: ```bash cargo nextest run -p vmm_tests multiarch::openvmm_uefi_x64_frontpage ``` And, for further example, to rebuild everything* and run all* the tests (see below for details on these steps): *This will not work for Hyper-V tests. TMK tests need additional build steps. ```bash # Install (most) of the dependencies; cargo nextest run may tell you # about other deps. rustup target add x86_64-unknown-none rustup target add x86_64-unknown-uefi rustup target add x86_64-pc-windows-msvc sudo apt install clang-tools lld cargo install cargo-nextest --locked cargo xtask guest-test download-image cargo xtask guest-test uefi --bootx64 # Rebuild all, and run all tests cargo build --target x86_64-pc-windows-msvc -p pipette cargo build --target x86_64-unknown-linux-musl -p pipette cargo build --target x86_64-pc-windows-msvc -p openvmm cargo xflowey build-igvm x64-test-linux-direct cargo xflowey build-igvm x64-cvm cargo xflowey build-igvm x64 cargo nextest run --target x86_64-pc-windows-msvc -p vmm_tests --filter-expr 'all() & !test(hyperv) & !test(tmk)' ``` ### \[Linux] Cross-compiling `pipette.exe` These commands might use the test agent (`pipette`) that is put inside the VM, and if the host machine OS and the guest machine OS are different, a setup is required for cross-building. The recommended approach is to use WSL2 and cross-compile using the freely available Microsoft Visual Studio Build Tools or Microsoft Visual Studio Community Edition as described in [\[WSL2\] Cross Compiling from WSL2 to Windows](../getting_started/cross_compile.md) If that is not possible, here is another option that relies on [MinGW-w64](https://www.mingw-w64.org/) and doesn't require installing Windows: ```bash # Do 1 once, do 2 as needed. # # 1. Setup the toolchain rustup target add x86_64-pc-windows-gnu sudo apt-get install mingw-w64-x86-64-dev mingw-genlib -a x86_64 ./support/pal/api-ms-win-security-base-private-l1-1-1.def sudo mv libapi-ms-win-security-base-private-l1-1-1.a /usr/x86_64-w64-mingw32/lib # 2. Build Pipette (builds target/x86_64-pc-windows-gnu/debug/pipette.exe first) cargo build --target x86_64-pc-windows-gnu -p pipette ``` ```bash # Run a test cargo nextest run -p vmm_tests multiarch::openvmm_uefi_x64_windows_datacenter_core_2022_x64_boot ``` ### Printing logs for VMM Tests In order to see the OpenVMM logs while running a VMM test, do the following: 1. Add the `--no-capture` flag to your `cargo nextest` command. 2. Set `OPENVMM_LOG=trace`, replacing `trace` with the log level you want to view. --- ### Guide Src Dev Guide Tests Vmm Perf (`Guide/src/dev_guide/tests/vmm_perf.md`) # VMM.Perf Runner `vmm_perf` uses Microsoft's [VirtualClient][] workload runner to execute packaged performance profiles against an OpenVMM build and collect their metrics and diagnostic files. [VirtualClient]: https://github.com/microsoft/VirtualClient ## When to use VMM.Perf Use VMM.Perf for the standardized `fio`, `iperf3`, and boot-time workloads assembled by the performance pipeline. It is suitable for repeatable local or CI runs where the runtime archive and profile definitions must remain pinned. ## Run through Flowey The preferred command is: ```bash cargo xflowey vmm-perf ``` Flowey builds OpenVMM and `vmm_perf`, restores UEFI firmware, downloads the pinned runtime archive, chooses a native target, and supplies all low-level paths. Run one profile: ```bash cargo xflowey vmm-perf --profile fio ``` Profile selections use `WORKLOAD[:MODIFIER...]`. Bare `fio` and `iperf3` select their VMBus profiles for compatibility, while `boot-time` has no default modifier. Select VIRTIO explicitly, or repeat `--profile` to run both device variants: ```bash cargo xflowey vmm-perf \ --profile fio:vmbus \ --profile fio:virtio ``` Modifiers are colon-delimited ASCII alphanumeric tokens. Multiple ordered modifiers select a profile with the same suffix order, such as `fio:ssh:virtio` selecting `FIO-SSH-VIRTIO`, when that profile exists in the pinned runtime. The pipeline selects a GNU Linux x64 target on Linux and Windows x64 on Windows. A static Linux runner can be selected with `--target linux-x64-musl` when required by the host environment. ## VM sizing The default configuration requests 16 virtual processors and 64 GiB of guest memory. The runner validates requested CPU and memory against host capacity and fails before starting a workload when the host is too small. Select smaller or additional shapes through Flowey: ```bash cargo xflowey vmm-perf \ --profile fio \ --vmm-perf-vmsizes 'CpuCount=2,MemoryMB=4096' \ --vmm-perf-vmsizes 'CpuCount=8,MemoryMB=16384' ``` Each shape becomes a separate named configuration. Running multiple profiles multiplies the number of VirtualClient executions by the number of shapes. ## Direct runner interface Direct invocation is useful when a pipeline has already produced all inputs: ```bash vmm_perf \ --openvmm path/to/openvmm \ --firmware path/to/MSVM.fd \ --runtime-archive path/to/vmm-perf-runtime.tar.gz \ --output-dir path/to/results \ --profile fio ``` The required inputs are: - An OpenVMM executable for the current host. - Matching UEFI firmware. - A supported VirtualClient runtime archive. - A destination for retained diagnostics. `--temp-dir` changes the scratch root. `--vm-sizes-json` accepts an array of named or unnamed configuration objects, while `--parameters-json` applies additional scalar VirtualClient parameters to every selected configuration. For example: ```bash vmm_perf \ --vm-sizes-json \ '[{"name":"small","parameters":{"CpuCount":2,"MemoryMB":4096}}]' ``` Use raw parameter overrides only when you understand the selected VirtualClient profile. They can change the workload as well as the VM shape. ## Execution lifecycle For each profile and configuration, the runner: 1. Validates input files and host capacity. 2. Extracts or reuses a runtime cache adjacent to the archive. 3. Verifies that the platform-specific profile exists in that runtime. 4. Creates isolated temporary, work, and output directories. 5. Constructs a VirtualClient command with OpenVMM, firmware, shape, and metadata parameters. 6. Executes one VirtualClient iteration. 7. Restores file ownership where required and gathers diagnostics. The runner continues through remaining profile/configuration pairs after an individual failure, then returns one combined failure summary. ## Output layout Flowey defaults to `target/vmm_perf`, with scratch files under `t` and retained results under `results`. Each profile/configuration output can contain: - `results/` for workload `metrics.csv` files. - `openvmm-logs/` for collected VMM logs. - `virtual-client/logs/` for `console.log`, `metrics.csv`, `vc.metrics`, `vc.traces`, and split metric files. - `virtual-client/runtime/` for publishable files emitted by the runtime. A successful VirtualClient process that does not produce `vc.metrics` is treated as a failed run. ## Troubleshooting - A capacity error should be fixed by selecting a smaller VM shape, not by bypassing the check. - A missing profile usually means the runtime archive does not match the runner's OS, architecture, or expected package version. - An archive error can indicate an unsupported extension or multiple runtime roots inside the archive. - Read `virtual-client/logs/console.log` for workload startup failures and the collected OpenVMM logs for VM failures. - On Linux, ownership restoration errors can follow a workload that created root-owned output; retain the first workload error as the primary signal. --- ### Guide Src Reference Architecture Devices (`Guide/src/reference/architecture/devices.md`) # Device architecture This section covers the internal architecture of device emulators and their backends — the shared machinery that both OpenVMM and OpenHCL use to connect guest-visible storage, networking, and other devices to their backing implementations. ## Pages - [Storage pipeline](./devices/storage.md) — how guest I/O flows from a storage frontend (NVMe, SCSI, IDE) through the [`DiskIo`](https://openvmm.dev/rustdoc/linux/disk_backend/trait.DiskIo.html) abstraction to a concrete backing store. --- ### Guide Src Reference Architecture Openhcl (`Guide/src/reference/architecture/openhcl.md`) # OpenHCL Architecture **Prerequisites:** - [Getting Started: OpenHCL](../../user_guide/openhcl.md). - [Architecture: OpenVMM Architecture](./openvmm.md). * * * ## Overview OpenHCL is a paravisor execution environment that runs within the guest partition of a virtual machine. It provides virtualization services to the guest OS from within the guest partition itself, rather than from the host as is traditionally done. The following diagram offers a brief, high-level overview of the OpenHCL Architecture. ## Virtual Trust Levels (VTLs) OpenHCL relies on [Virtual Trust Levels] (VTLs) to establish a security boundary between itself and the guest OS. - **VTL2:** OpenHCL runs here[^sk]. It has higher privileges and is isolated from VTL0. - **VTL0 (and sometimes VTL1):** The Guest OS (e.g., Windows, Linux) runs here. It cannot access VTL2 memory or resources. This isolation is enforced by the system configured by the underlying virtual machine monitor (Hyper-V) and can be backed by: - Hardware [TEEs], like Intel [TDX] and AMD [SEV-SNP]. - Software-based constructs, like Hyper-V [VSM]. ## Scenarios OpenHCL enables several key scenarios by providing a trusted execution environment within the VM: ### Azure Boost OpenHCL acts as a compatibility layer for Azure Boost. It translates legacy synthetic device interfaces (like VMBus networking and storage) used by the guest OS into the hardware-accelerated interfaces (proprietary [Microsoft Azure Network Adapter] (MANA) and NVMe) provided by the Azure Boost infrastructure. This allows unmodified guests to take advantage of next-generation hardware. The following diagram shows a high level overview of how synthetic networking is supported in OpenHCL over Microsoft Azure Network Adapter (MANA) The following diagram shows a high level overview of how accelerated networking is supported in OpenHCL over MANA ### Confidential Computing In Confidential VMs (CVMs), the host is not trusted. OpenHCL runs inside the encrypted VM context (VTL2) and provides necessary services (like device emulation and TPM) that the untrusted host cannot securely provide. Security-sensitive devices, such as the virtual TPM, can be further isolated by running them in separate worker processes within VTL2 for defense-in-depth protection. ### Trusted Launch OpenHCL hosts a virtual TPM (vTPM) and enforces Secure Boot policies, ensuring the integrity of the guest boot process. ## Architecture Components OpenHCL is built on top of a specialized Linux kernel and consists of several userspace processes that work together to provide these services. For more details on the internal components and selected OpenHCL device paths, see: - [Processes and Components](./openhcl/processes.md) - [Timekeeping](./timekeeping.md) - [Attestation and VMGS Protection](./openhcl/attestation.md) - [VMBus Relay and Device Interception](./openhcl/vmbus.md) - [Storage Translation](./openhcl/storage_translation.md) - [Storage Configuration Model](./openhcl/storage_configuration.md) - [Boot Flow](./openhcl/boot.md) - [Sidecar](./openhcl/sidecar.md) - [IGVM Artifact](./openhcl/igvm.md) [^sk]: Why not VTL1? Windows already uses VTL1 in order to host the [Secure Kernel]. [VSM]: https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/vsm [Virtual Trust Levels]: https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/vsm [TDX]: https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/overview.html [SEV-SNP]: https://www.amd.com/content/dam/amd/en/documents/epyc-business-docs/white-papers/SEV-SNP-strengthening-vm-isolation-with-integrity-protection-and-more.pdf [TEEs]: https://en.wikipedia.org/wiki/Trusted_execution_environment [Secure Kernel]: https://www.microsoft.com/en-us/security/blog/2020/07/08/introducing-kernel-data-protection-a-new-platform-security-technology-for-preventing-data-corruption/ [Microsoft Azure Network Adapter]: https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-mana-overview --- ### Guide Src Reference Architecture Openvmm (`Guide/src/reference/architecture/openvmm.md`) # OpenVMM Architecture This section describes the architecture of OpenVMM when it runs as a hosted VMM. - [Memory Layout](./openvmm/memory-layout.md) describes the guest physical address space. - [Memory Backing](./openvmm/memory-backing.md) explains how guest RAM is allocated and shared. - [NUMA Topology](./openvmm/numa.md) covers guest NUMA configuration and host memory placement. - [Device Tree](./openvmm/device-tree.md) describes the device trees that OpenVMM builds for IGVM files and ARM64 Linux direct boot. - [Timekeeping](./timekeeping.md) explains clocks, timers, and their behavior across VM lifecycle changes. - [mesh](./openvmm/mesh.md) describes OpenVMM's inter-process communication framework. --- ### Guide Src Reference Architecture Timekeeping (`Guide/src/reference/architecture/timekeeping.md`) # Timekeeping This page explains the clocks and timers used by OpenVMM and OpenHCL, how they relate to guest-visible time, and what happens when execution stops. There is no single "VM clock". A guest can read a processor counter, read a hypervisor reference counter, ask firmware for the date, and receive a timesync message. These interfaces have different origins, owners, and pause behavior. A timer also has state beyond the clock it reads: its deadline, periodic phase, and any interrupt or message awaiting delivery. ## Clock domains | Domain | Representation | Purpose | | --- | --- | --- | | Host monotonic time | `pal_async::timer::Instant`, nanoseconds | Scheduling host tasks and implementing elapsed-time clocks. Not a date. | | Host system time | `SystemTime` or `jiff::Timestamp` | Calendar time, subject to host clock adjustments. | | Software VM time | `vmcore::vmtime::VmTime`, 100 ns ticks | Pausable timeline used by software device emulation. | | Partition reference time | `ReferenceTimeSource`, 100 ns ticks | Hyper-V reference-counter reads, synthetic timer deadlines, and correlation with UTC. Its source depends on the backend. | | Architectural counters | x86 TSC; Arm physical/virtual counters | Guest instruction-visible counters, with architecture-specific frequencies and virtualization offsets. | | Device wall-clock time | `LocalClockTime`, milliseconds since the Unix epoch | Reprogrammable calendar time for RTC and firmware services. | | Guest OS time | Maintained by the guest | The OS derives monotonic and wall-clock time from available counters and synchronization services. | Equal units do not imply equal clocks. In particular, `VmTime` and partition reference time both use 100 ns ticks, but generally have different origins. TSC ticks are not nanoseconds or CPU instruction cycles; you need the counter's frequency to convert them to a duration. Here, "host" means the OS running the relevant userspace component. For hosted OpenVMM that is the host OS; for OpenHCL userspace it is the Linux environment in VTL2, not the root partition. ```text Host monotonic clock +-- host task deadlines (PolledTimer) +-- software VmTime, with a running/stopped mapping +-- emulated device counters and timer deadlines +-- Hyper-V reference time in some emulated configurations Architectural counter / hypervisor clock +-- guest TSC or Arm counter +-- partition reference time in native and counter-backed configurations System time / platform-provided UTC +-- LocalClock --> RTC or UEFI calendar time +-- UTC + partition-reference sample --> timesync message --> guest OS ``` The branches describe possible sources, not universal wiring. Clock offsets and scales cannot be inferred from the counters' numeric values. ## Software VM time [`VmTimeKeeper`][vmtime] owns a software timeline. Hosted OpenVMM creates it stopped at zero. While running, its value is the VM time at the last start plus elapsed host monotonic time. Stopping captures a fixed value; starting again establishes a new host-time anchor without adding the stopped interval. Reset sets it to zero, and restore installs a saved VM time rather than an old host timestamp. This measures elapsed time, not scheduled VP runtime: it advances while guest CPUs are idle or descheduled unless the keeper is stopped. The underlying [`pal_async` clock][host-time] uses `CLOCK_MONOTONIC` on Unix and `QueryUnbiasedInterruptTimePrecise` on Windows. A VM pause, host-system sleep, and guest CPU idle are different events: VM pause is handled by the keeper, while host sleep behavior follows the underlying OS clock. Do not use these timestamps as UTC. Devices obtain a `VmTimeAccess` from a `VmTimeSource`. Each accessor can read time and register a deadline; the backing task translates VM deadlines into host timer deadlines while running. This lets a virtual device wait without advancing its guest-visible timeline during a pause. An overdue deadline still needs the consumer to run and process it: deadline expiration is not the same as guest interrupt delivery. `VmTimeSourceBuilder` provides synchronized cross-process access over mesh, but requires a shared OS monotonic clock. It cannot synchronize different machines or an OpenVMM host with OpenHCL. Stop reconciles the secondary keepers to a common value without moving accessors backward. The keeper saves the timeline value; devices save their own timer state. Monotonicity does not extend across reset or restore of an older snapshot. ## Partition reference time and processor counters [`ReferenceTimeSource`][reference-time] is a read interface, not a clock controller. A read returns a reference count and may also return a UTC timestamp captured at the same instant, allowing the two values to be correlated. The reference count is not itself UTC. A missing UTC value means that the source cannot cheaply provide the correlated timestamp. For Hyper-V-compatible guests, the reference counter is a partition-wide elapsed-time interface. On x86, guests can read `HV_X64_MSR_TIME_REF_COUNT`. On Arm, the corresponding synthetic register is `HvRegisterTimeRefCount`, accessed through a hypercall when the backend supports that interface. The reference-TSC page provides an alternative that avoids an intercept: ```text reference_time = high64(virtual_tsc * scale) + offset ``` The guest must follow the [sequence-number protocol][tlfs-timers]. A zero sequence requires fallback to the reference counter: advertising the facility does not mean that the page is usable at every instant. The shared `hv1_emulator` accepts an explicit reference-time source. It publishes a usable reference-TSC page only when configured for its TSC-backed, zero-offset model; otherwise it leaves the sequence invalid. A `VmTime`-backed reference counter therefore does not automatically provide a direct TSC fast path. Architectural counters have their own state. Changing reference time does not inherently change x86 `RDTSC`, an Arm counter offset, or a timer deadline expressed in counter ticks. Similarly, stopping a VP thread or leaving a hypervisor run call does not by itself freeze those counters. ## Timer implementations Choose the clock by the guest-visible contract, not by whichever timer API is easiest to call. | Timer or counter | Implementation and clock | | --- | --- | | PIT | `chipset::pit` advances counting elements from elapsed `VmTime` and schedules wakeups for interrupt generation. | | ACPI PM timer | Software reads scale `VmTime` to 3.579545 MHz. With PM timer assist, the hypervisor handles reads using its own time source instead. | | CMOS RTC | Calendar registers use `LocalClock`; alarm, periodic, and update interrupt scheduling uses `VmTime`. | | UEFI watchdog | `watchdog_core` receives a `VmTimeAccess`, so its deadline belongs to the software VM timeline. | | Software local APIC | `virt_support_apic` counts down using `VmTime`; backend-native APIC emulation instead has backend-owned timer state. | | Hyper-V synthetic timers | Deadlines use partition reference time. Native implementations belong to the hypervisor; `hv1_emulator::synic` evaluates them in software. | | Arm architectural timers | Compare architectural counter values with programmed deadlines; backend-specific code handles virtualization and interrupt delivery. | | Host work timers | `PolledTimer` waits on host monotonic time. It has no implicit association with VM pause. | Synthetic timers can be one-shot or periodic. In the shared emulator, a one-shot count is an absolute reference-time deadline; a periodic count is an interval. Expiration can request a direct interrupt or post a SynIC message. Masking, a busy message slot, and VP scheduling can delay delivery after the deadline. This distinction matters during save/restore. Restoring a clock value alone does not restore a periodic timer's phase, retract an already pending interrupt, or preserve an undelivered message. Conversely, blindly reloading a timer can discard events that were already pending. Backend state interfaces and device saved state must be considered together. ## Calendar time and synchronization ### RTC and UEFI [`LocalClock`][local-clock] is an instance-local, reprogrammable calendar clock. Setting it changes that instance, not the host's system clock. Unlike `VmTime`, its intended behavior is to account for elapsed wall-clock time while the VM is paused. Hosted RTC devices use `SystemTimeClock`, which returns host `SystemTime` plus an offset. Guest writes change that offset. Host calendar-clock adjustments, including backward jumps, remain visible. The default UEFI time source is also a `SystemTimeClock`; independently created clock instances do not share guest-written offsets. RTC dates can advance during a pause while `VmTime`-based interrupt machinery is stopped. Periodic interrupt rates do not define wall time. On Arm, the UEFI time service exposes the supplied local clock through `GetTime`/`SetTime`. It retains timezone and daylight fields for readback but does not apply them as timezone conversions. Its saved state stores those fields; persistence of the backing clock is the platform's responsibility. Similarly, CMOS register saved state is not a serialized `LocalClock` implementation. ### Hyper-V timesync The `hyperv_ic` timesync device sends a UTC sample paired with partition reference time. The pair allows the guest to account for elapsed reference time between sampling and consuming the message. This is separate from setting the RTC or changing the partition clock. It uses correlated UTC from `ReferenceTimeSource` when available; otherwise it samples local system time immediately after reference time. It waits for ring-buffer space before sampling to reduce delivery delay. The current implementation negotiates timesync version 4, sends an initial synchronization message, and then sends samples on a five-second host-timer cadence while the channel runs. On snapshot restore it renegotiates and sends a fresh synchronization message. Whether and how the guest OS adjusts its time is a guest policy decision. After resume, elapsed time can exclude the pause while RTC reads and timesync report the current date. Resynchronizing calendar time does not mean a frozen elapsed-time clock was restored incorrectly. ## Stop, resume, and saved state The optional [`PartitionTimeControl`][time-control] interface controls backend time separately from `VmTimeKeeper`. Supporting backends create partitions frozen. The lifecycle contract requires stopped VPs for both operations, makes repeated freeze/thaw calls harmless, and treats an unexpected transition failure as fatal inside the backend. For hosted OpenVMM, the relevant full-stop ordering is: ```text Stop: stop all VPs --> freeze supported backend time --> stop dependent device work --> stop VmTime Start: start VmTime --> start dependent device work --> thaw supported backend time --> allow VPs to run ``` The state-unit dependency graph establishes this ordering; it is not an atomic freeze of every clock. Backend time and software device time can have different stop/start anchors. | Operation | Timekeeping consequence | | --- | --- | | Full VM stop | Stops software VM time and requests backend freeze where supported. Does not freeze host clocks or calendar time. | | Temporary VP stop or debugger halt | Does not itself stop `VmTime` or freeze backend time. Timers can become due while VPs cannot execute. | | Full VM start | Thaws backend time even if a debugger halt or outstanding temporary-stop guard still prevents VP execution. | | Reset | Resets software VM time to zero. Supporting backends remain frozen while partition and VP state are reset. | | Snapshot restore | Restores stopped software time and the state supported by each backend/device. A time-state setter must not implicitly thaw a supporting backend. | | VTL scrub | May reset and freeze a backing VTL clock. The partition unit thaws it before restarting VPs; this is not a freeze of every VTL. | The x86 `virt` model separates reference time/page, TSC, APIC, and TSC deadline. Deadline state requires guest/backend support and restores after APIC/TSC: zero disarms; absent state is not restored. Native MSHV/WHP and KVM support it, unlike software APICs. Reference time is included only when Hyper-V enlightenments are enabled; lifecycle freezing cannot be hidden in its setter. Lifecycle freeze is separate; `can_freeze_time` gates exact state comparison. A reference-clock-only implementation is not sufficient to claim that capability. Arm state coverage differs: the generic `virt` partition-state schema is empty and its VP schema does not currently include architectural counter/timer state. A native freeze API is not, by itself, evidence of complete portable snapshot support. ## Hosted backend configurations These are current implementations, not uniform `virt` guarantees. Guest-visible Hyper-V facilities require the relevant enlightenments. | Backend/configuration | Reference-time source | Backend freeze on full stop | | --- | --- | --- | | MSHV, x86 or Arm | Hyper-V partition `ReferenceTime` property | Yes, through `TimeFreeze`. | | WHP, offloaded Hyper-V | Native WHP reference time | Yes, separately for each backing partition. | | WHP, emulated Hyper-V on x86 | Software `VmTime` | Yes for native WHP time; `VmTime` stops separately. | | KVM, x86 | KVM clock, converted from ns to 100 ns | No; stopping software VM time does not freeze KVM time. | | KVM, Arm | No Hyper-V reference-time source exposed | No; architectural timers remain KVM-owned. | | HVF, Arm | Software `VmTime` | No architectural-counter freeze; software time still stops. | MSHV and WHP reference-time sources return no correlated UTC timestamp. KVM x86 returns one when `KVM_GET_CLOCK` supplies `KVM_CLOCK_REALTIME`. HVF and WHP's software source return only the VM-time count. ### MSHV and WHP `virt_mshv` uses native Hyper-V time and interrupt-controller facilities. It reads reference time through a partition property rather than a VP register, avoiding a query that would need to wait for a running VP. Creation freezes the partition before guest initialization; runtime freeze/thaw updates `TimeFreeze`. Guest reference-TSC-page exposure is capability-dependent and is currently disabled for MSHV SNP partitions. That restriction is separate from freezing the partition. For non-isolated VM reset, `virt_mshv` uses `MSHV_SCRUB_PARTITION` when the kernel implements it, then explicitly reasserts `TimeFreeze`. An `ENOTTY` response selects the existing state-by-state reset path; other scrub errors fail reset. Per-VP user-mode state is reset on both paths, while a successful scrub replaces bulk hypervisor-state writes. `virt_whp` offloads Hyper-V facilities when offload is enabled, the required WHP features are available, and a user-mode APIC is not selected. Otherwise, its x86 software path uses `hv1_emulator` with a `VmTime`-backed reference counter and software synthetic timers. The user-mode APIC also uses `VmTime`. Arm does not support the user-mode-APIC or disabled-offload options; its virtual timer PPI is configured in WHP's GIC parameters. WHP uses [`WHvSuspendPartitionTime`][whp-suspend] and explicit resume, not the implicit resume performed by entering a VP. With VTL2 emulation, VTL0 and VTL2 have distinct WHP backing partitions and frozen-state bookkeeping. A full stop freezes both. Scrubbing VTL2 freezes only its backing clock and, on x86, preserves its reference count around native reset. This simulation excludes the time spent in that reset; it is not an exact model of servicing downtime. The WHP x86 partition-state accessors still have TODOs for locally emulated Hyper-V state: they access native WHP state even in that configuration. Do not infer full emulated-enlightenment snapshot coverage from the presence of the freeze interface. ### KVM On x86, KVM owns the virtual TSC, LAPIC timers, and enabled Hyper-V synthetic timers. OpenVMM reads the KVM clock for its reference-time source. Saving reference time rounds the nanosecond clock upward to 100 ns so conversion does not move it backward on restore. Setting reference time uses `KVM_SET_CLOCK` with flags zero; it does not request KVM's optional UTC-based addition of elapsed downtime. That setter rebases the KVM clock; it does not freeze it, rewind raw guest TSC, or undo a LAPIC timer expiration. KVM can record timer work while userspace has stopped running VPs. The backend currently exposes no `PartitionTimeControl`, and its x86 `can_freeze_time` is false. Rebasing reference time alone would not implement full counter and timer freeze semantics. KVM's synthetic-timer snapshot path saves config/count MSRs, but cannot preserve timer adjustment or undelivered expiration-message state. On Arm, KVM manages architectural counters/timers and the in-kernel interrupt controller. OpenVMM configures the virtual timer PPI from the topology; the physical timer PPI required by KVM is not advertised to the guest. There is no Hyper-V reference-time source or SynIC support in this backend, and no software pause compensation for the counters. ### HVF On macOS/Arm, HVF's synthetic reference counter and software SynIC timers use `VmTime`. Architectural counters use a different basis: `CNTPCT_EL0` reflects the host counter and `CNTVCT_EL0` reflects `mach_absolute_time()` minus HVF's virtual-timer offset. The backend does not adjust that offset to remove VM pause duration. For a VP waiting after WFI, the backend converts the remaining architectural virtual-timer interval into a `VmTime` wakeup. It then re-enters HVF, whose timer-activation exit raises the emulated GIC PPI. Using `VmTime` to arrange this wakeup does not make the architectural counter itself a `VmTime` clock. ## OpenHCL: time inside a partition OpenHCL runs in VTL2 inside a partition whose lifecycle is controlled outside OpenHCL. Stopping lower-VTL execution or servicing OpenHCL is not equivalent to freezing that surrounding partition. Its `VmPartition` wrapper therefore leaves the backend time-control hooks as no-ops. OpenHCL does have its own `VmTimeKeeper` for software device emulation. It starts at zero, participates in state-unit stop/start, and has saved state. Servicing also records partition reference time at stop and uses it to report blackout duration on restart. That measurement is separate from restoring the software VM-time value; it is not an instruction to rewind the surrounding partition clock or add downtime to `VmTime`. For non-hardware-isolated configurations, `virt_mshv_vtl` obtains native reference time through HCL's VTL2 `TimeRefCount` register access. In x86 SNP and TDX configurations, it emulates lower-VTL Hyper-V facilities with a reference source derived from `RDTSC` and a frequency-based scale. This source is not `VmTime`. The reference-TSC page uses the same zero-offset model. The frequency comes from the hypervisor; TDX also validates it against hardware-advertised frequency information. The emulated SynIC compares deadlines against that reference source. To wait using software VM time, the implementation converts a *relative interval*, `next_reference_time - current_reference_time`, into a `VmTime` deadline instead of equating the clocks' origins. SNP also reconciles kernel-handled STIMER0 writes with the userspace SynIC; kernel expiry wakes VTL2, while the SynIC scan performs delivery. TDX uses an L2 virtual-TSC execution deadline when supported, with a `VmTime`-based fallback. The shared software APIC timer still uses `VmTime`; hardware acceleration and interrupt delivery are distinct from the reference-clock choice. The x86 CVM TSC model must not be assumed to describe Arm hardware-isolated configurations. OpenHCL's calendar clock is different again. `UnderhillLocalClock` queries root-provided time over Guest Emulation Transport, caching samples for up to one second, and adds a guest-programmable offset. It persists the offset and saves it for servicing; without a saved offset it uses the host-provided timezone offset. Thus RTC calendar values need not represent UTC even though the internal type uses an epoch-based numeric representation. ```admonish warning Clock availability does not establish trust in UTC. In a confidential VM, root-provided calendar time is an external input, not authenticated time. A processor-derived reference counter is not a trusted calendar clock either. These mechanisms do not by themselves provide rollback-resistant timestamps or guarantees about host scheduling and timer delivery. ``` ## Implementation map Start with these repository paths: - `vm/vmcore/src/{vmtime,reference_time}.rs`: timelines and accessors. - `vmm_core/src/{partition_unit,vmtime_unit}.rs`: lifecycle operations; `openvmm/openvmm_core/src/worker/dispatch.rs`: hosted dependency wiring. - `vm/hv1/hv1_emulator/src/{hv,synic}.rs`: reference pages and timers. - `vmm_core/virt_{mshv,whp,kvm,hvf}/src/`: backend state and VP loops. - `vm/devices/chipset/src/` (`pit`, `pm`, `cmos_rtc.rs`) and `support/local_clock/src/`: device timers and calendar clocks. - `vm/devices/hyperv_ic/src/timesync.rs`: UTC/reference-time samples. - `openhcl/virt_mshv_vtl/src/`: native and confidential-VM time sources; `openhcl/underhill_core/src/emuplat/local_clock.rs`: calendar time. See also [snapshot behavior](../../user_guide/openvmm/snapshots.md) and the [snapshot format](../../dev_guide/snapshot_format.md). [vmtime]: https://openvmm.dev/rustdoc/vmcore/vmtime/index.html [host-time]: https://openvmm.dev/rustdoc/pal_async/timer/index.html [reference-time]: https://openvmm.dev/rustdoc/vmcore/reference_time/index.html [local-clock]: https://openvmm.dev/rustdoc/local_clock/index.html [time-control]: https://openvmm.dev/rustdoc/virt/trait.PartitionTimeControl.html [tlfs-timers]: https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/timers [whp-suspend]: https://learn.microsoft.com/en-us/virtualization/api/hypervisor-platform/funcs/whvsuspendpartitiontime --- ### Guide Src Reference Backends Consomme (`Guide/src/reference/backends/consomme.md`) # Consomme Consomme is a user-mode NAT that provides guest VM networking through ordinary host sockets — no driver installation and no system-wide network configuration. Because all traffic flows through standard socket APIs, host networking policies (firewalls, VPN routing, proxy settings) apply naturally, just as they would for any other application. The guest sees a normal network: it gets an IP address via DHCP, has a default gateway, and can resolve DNS names. Behind the scenes, Consomme intercepts the raw Ethernet frames from the virtual NIC, parses them, and translates them into socket calls on the host. ```mermaid flowchart TB subgraph VM ["Guest VM"] App["Application"] --> GuestStack["TCP/IP Stack"] --> VNIC["Virtual NIC"] end subgraph UserMode ["Host User Mode"] VNIC -- "Ethernet frames" --> Parse["Parse & Dispatch"] Parse --> TCPProxy["TCP proxy"] Parse --> UDPProxy["UDP proxy"] Parse --> ICMPFwd["ICMP forwarder"] Parse --> DNSFwd["DNS forwarder"] Parse --> DHCPSrv["DHCP / ARP / NDP"] TCPProxy & UDPProxy & ICMPFwd --> Sockets["Host sockets"] DNSFwd -- "DNS API call" --> HostDNS["Platform DNS resolver"] DHCPSrv -. "replies directly to guest" .-> VNIC end subgraph Kernel ["Host Kernel"] Sockets --> HostStack["Host TCP/IP stack + firewall + VPN"] HostDNS --> HostStack HostStack --> NIC["Physical NIC"] end NIC --> Internet((Internet)) ``` ## Default network topology | Role | IPv4 Address | |------|-------------| | Guest | `10.0.0.2` | | Gateway | `10.0.0.1` | | Subnet mask | `255.255.255.0` | The subnet is configurable via the `--net consomme:` CLI option. The gateway is always `.1` and the guest is always `.2`. ## Port forwarding By default, Consomme only provides outbound connectivity from the guest. To accept inbound connections, use the `hostfwd=` option to forward host ports into the guest: ```bash --net consomme:hostfwd=tcp::2222-:22 ``` This binds host port 2222 and forwards incoming TCP connections to guest port 22. The general syntax is: ```text hostfwd=:[]:-[]: ``` | Field | Description | |-------|-------------| | `proto` | `tcp` or `udp` | | `hostaddr` | Host IP to bind (default: all IPv4 interfaces, `0.0.0.0`). Use `[addr]` brackets for IPv6 addresses | | `hostport` | Host port to listen on. Use `0` for any available port | | `guestaddr` | Currently ignored — traffic always forwards to the guest IP | | `guestport` | Guest port to forward to | Multiple forwards can be specified with commas: ```bash --net consomme:hostfwd=tcp::2222-:22,hostfwd=tcp::3389-:3389 ``` Port forwards can also be combined with a CIDR: ```bash --net consomme:10.0.0.0/24,hostfwd=tcp::2222-:22 ``` Examples: ```bash # Forward host port 8080 to guest port 80 --net consomme:hostfwd=tcp::8080-:80 # Bind only to localhost --net consomme:hostfwd=tcp:127.0.0.1:8080-:80 # IPv6 localhost --net consomme:hostfwd=tcp:[::1]:8080-:80 # UDP forwarding --net consomme:hostfwd=udp::5353-:53 ``` IPv6 is enabled when the host has a routable IPv6 address. Consomme advertises a prefix via SLAAC and the guest auto-configures its own address. IPv6 DNS servers are advertised via RDNSS (in Router Advertisements) and DHCPv6 Information-Request. ## Protocol handling ### TCP Consomme maintains a full TCP state machine on the guest-facing side and opens a real host socket per connection. Data flows through ring buffers between the two sides. This is a split-TCP design: there are two independent TCP connections (guest↔Consomme and host↔remote server) stitched together. Key implications of the split-TCP design: - **Congestion control is disconnected from the real network.** The guest sees a virtual link with zero packet loss and sub-microsecond RTT. Its congestion window opens wide almost immediately. The host kernel's TCP stack handles all real congestion and retransmission. - **Another TCP stack to debug.** When a connection stalls, there are three TCP implementations in play: the guest's, Consomme's, and the host kernel's. Diagnosing which one is misbehaving requires visibility into Consomme's internal state. Consomme supports TCP Segmentation Offload (TSO) from the guest NIC, window scaling, and MSS negotiation. The per-connection ring buffers autotune. Each connection starts with a small buffer (16 KiB by default) and grows on demand — doubling under sustained throughput pressure up to a ceiling (4 MiB by default) — so idle and short-lived connections stay cheap while bulk transfers can ramp up to a large window. Embedders can override the bounds via the `tcp_rx_buffer` and `tcp_tx_buffer` fields on `ConsommeParams` (each a `TcpBufferBounds { initial, max }`); the values are clamped to `[16 KiB, 4 MiB]` and rounded up to a power of two. Setting `initial == max` pins a fixed size and disables growth. Connections waiting for the peer to advance a TCP close handshake use a 60-second inactivity timeout by default, including connections closed before their initial handshake completes. `CloseWait` is excluded because the local backend can still produce data; its timeout starts when the backend closes and the connection enters `LastAck`. ACKs and newly accepted data restart the timeout. Embedders can override it with the `tcp_close_timeout` field on `ConsommeParams`. The same duration is used for `TimeWait`, where it represents the 2*MSL wait required by RFC 9293. A separate fixed 60-second timeout bounds initial handshakes that receive no close request. Inbound connections require explicit port forwarding — OpenVMM can programmatically bind host ports and forward them into the guest. ### UDP Each unique guest source socket (guest IP + source port) gets a host-side UDP socket bound to an ephemeral port. Datagrams are forwarded in both directions. Idle bindings are cleaned up after a configurable timeout (5 minutes by default, per RFC 4787). UDP packets to the gateway on well-known ports are intercepted: - **Port 67** — handled by the built-in DHCP server - **Port 53** — handled by the DNS forwarder UDP segmentation offload and receive coalescing are not currently supported. Each datagram requires a separate socket call, which may matter for QUIC/HTTP/3 performance. ### DNS The guest is told (via DHCP) to use the gateway as its DNS server. Queries to the gateway on port 53 are intercepted and forwarded to the host's DNS resolver. On Windows, this uses the `DnsQueryRaw` API, which forwards the raw DNS packet without parsing — the host resolver handles all the logic and returns a raw response. On Unix-like hosts, Consomme uses the system resolver configuration (for example `/etc/resolv.conf`) and forwards queries via host UDP/TCP sockets. This means guest DNS resolution matches the host's behavior, including corporate VPNs and split-tunnel DNS configurations. DNS over TCP is also supported for large responses. ### ICMP Echo requests (ping) are forwarded through a host-side ICMP socket. The guest's TTL is preserved, so ping works and traceroute partially works (the echo reply comes back, but intermediate TTL Exceeded hops are not relayed — see limitations). ### ARP and NDP - **ARP** — Responds to requests for the gateway MAC address. All other ARP traffic is dropped. - **NDP** — Responds to Router Solicitations (advertising the IPv6 prefix and DNS servers) and Neighbor Solicitations for the gateway's link-local address. DAD is silently ignored. ### DHCP Built-in DHCPv4 server assigns the guest its IP address, gateway, and DNS servers. Lease duration is 24 hours. DHCPv6 Information-Request is also handled (stateless — provides DNS servers to SLAAC-configured guests). ## Checksum and segmentation offload Consomme supports the following offloads from the guest NIC: - IPv4 header checksum offload - TCP checksum offload - UDP checksum offload - TCP Segmentation Offload (TSO) ## Limitations ### No multicast or broadcast Beyond ARP, DHCP, and the NDP multicast messages required for IPv6, Consomme only processes unicast traffic. Protocols that rely on multicast discovery do not work: - **mDNS / Bonjour** — printer, AirPlay, Chromecast discovery - **SSDP / UPnP** — device discovery, NAT-PMP port mapping - **LLMNR** — Windows local name resolution fallback - **NetBIOS** — legacy Windows browsing ### No ICMP error relay ICMP error messages (Destination Unreachable, TTL Exceeded, etc.) are not forwarded to the guest: - **TCP** — unreachable hosts produce a TCP RST instead of an ICMP error. The connection fails, but the guest can't distinguish "host unreachable" from "connection refused." - **UDP** — Port Unreachable errors are silently swallowed. - **Traceroute** — TTL Exceeded replies from intermediate routers are not relayed, so the hop-by-hop path is not visible. ### No IP fragment reassembly Fragmented IP packets from the guest are silently dropped. This only affects UDP datagrams larger than ~1472 bytes. TCP is never affected because the guest kernel segments at the TCP layer. On the host side, fragmentation is handled transparently by the kernel socket layer. ### No jumbo frames MTU is fixed at 1500 bytes. ### No ICMPv6 Echo IPv6 ping does not work. IPv6 TCP and UDP work normally. ### No non-TCP/UDP/ICMP protocols GRE, ESP, SCTP, and other IP protocols are silently dropped. This means VPNs that terminate inside the guest and use IPsec (ESP) or PPTP (GRE) will not work. UDP-based VPNs like WireGuard work fine. IKEv2 with NAT-T (UDP-encapsulated ESP on port 4500) also works. VPNs running on the host work transparently — Consomme's traffic flows through the host network stack like any other application. ### IPv6 depends on host IPv6 is only enabled if the host has a routable IPv6 address at VM start time. ### Static DNS records only support the `A` record type Only `A` records are currently supported. `AAAA` queries for names with a static `A` record return `NOERROR` with no answers, rather than being forwarded to the upstream resolver. ### Static DNS records don't support DNSSEC All static DNS replies are returned with the AD flag unset. ## Crate structure | Crate | Path | Purpose | |-------|------|---------| | `consomme` | `vm/devices/net/net_consomme/consomme/` | Core user-mode TCP/IP stack | | `net_consomme` | `vm/devices/net/net_consomme/` | [`net_backend::Endpoint`](https://openvmm.dev/rustdoc/linux/net_backend/trait.Endpoint.html) integration | --- ### Guide Src Reference Backends Networking (`Guide/src/reference/backends/networking.md`) # Networking backends The networking backend system connects guest-facing NICs (frontends) to host-side packet I/O (backends) through a shared trait interface defined in the `net_backend` crate. This page explains how the pieces fit together, how packets flow, and how to navigate the code. ## Architecture overview ```text ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ virtio_net │ │ netvsp │ │ gdma/bnic │ │ (frontend) │ │ (frontend) │ │ (frontend) │ └───────┬──────┘ └───────┬──────┘ └───────┬──────┘ │ │ │ │ &mut dyn BufferAccess │ │ (owned by frontend) │ │ │ │ ▼ ▼ ▼ ┌──────────────────────────────────────────────────┐ │ dyn Queue (per-queue) │ │ poll_ready · rx_avail · rx_poll │ │ tx_avail · tx_poll │ └──────────────────────────────────────────────────┘ ▲ ▲ ▲ │ │ │ ┌───────┴──────┐ ┌───────┴──────┐ ┌───────┴──────┐ │ TapQueue │ │ConsommeQueue │ │ ManaQueue │ │ DioQueue │ │LoopbackQueue │ │ (hardware) │ │ ... │ │ NullQueue │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ ``` There are three layers: - **Frontend** — the guest-visible NIC device (`virtio_net`, `netvsp`, or `gdma`). Owns the `BufferAccess` implementation (no `Arc` or `Mutex` needed — each queue is driven from a single async task), translates between the guest-specific descriptor format and the generic `Queue` interface, and drives the poll loop. - **Queue** — a single TX/RX data path created by the backend. Frontends interact with it entirely through the [`Queue`](https://openvmm.dev/rustdoc/net_backend/trait.Queue.html) trait. A device may have multiple queues for RSS. - **Endpoint** — a backend factory. One per NIC. The frontend calls [`Endpoint::get_queues`](https://openvmm.dev/rustdoc/net_backend/trait.Endpoint.html#tymethod.get_queues) when the guest activates the NIC and [`Endpoint::stop`](https://openvmm.dev/rustdoc/net_backend/trait.Endpoint.html#tymethod.stop) on teardown. See the [`net_backend` rustdoc](https://openvmm.dev/rustdoc/net_backend/) for the full trait signatures and type definitions. ## Packet flow ### Transmit (guest → host) 1. The guest posts a TX descriptor (e.g. a virtio descriptor chain or a VMBus RNDIS message). 2. The frontend reads the descriptor from guest memory, extracts any offload metadata (checksum, TSO, USO), and builds a `TxSegment` array. Each segment carries a guest physical address and a length — **no data is copied** at this point. 3. The frontend calls `queue.tx_avail(&mut pool, &segments)`. The backend reads data directly from guest memory via `pool.guest_memory()` and transmits it (e.g. writes to a TAP fd, posts to hardware, or feeds it to a user-space TCP stack). 4. If the backend completes synchronously (`tx_avail` returns `sync = true`), the frontend can immediately mark the descriptor done. Otherwise, it polls `tx_poll` later for async completions. ### Receive (host → guest) 1. The frontend pre-populates the backend with receive buffers by calling `queue.rx_avail(&mut pool, &buffer_ids)`. 2. When `queue.poll_ready(cx, &mut pool)` signals readiness, the backend has received a packet. It writes the packet data into guest memory through `pool.write_packet(rx_id, metadata, data)`. 3. The frontend calls `queue.rx_poll(&mut pool, &mut ids)` to collect the IDs of completed buffers, then delivers them to the guest (e.g. by completing virtio descriptors or sending VMBus completion packets). 4. The guest eventually returns the buffer, and the frontend recycles it via `rx_avail`. ### Guest memory access The `Queue` interface works with guest physical addresses rather than host buffers, giving each backend flexibility in how it accesses packet data. The patterns fall into three categories: **GPA pass-through (hardware DMA).** `net_mana` converts guest physical addresses into IO virtual addresses (`GuestMemory::iova`) and posts them as scatter-gather entries directly to GDMA hardware. The NIC DMAs packet data to/from guest memory without any host-side copy. This is the fastest path, but requires IOMMU mappings and contiguous-enough buffers; when those conditions aren't met, MANA falls back to bounce buffers. **Host-mediated copy.** Software backends like `net_consomme` and `net_dio` read TX data from guest memory with `GuestMemory::read_at`, process or forward it, and write RX data back with `BufferAccess::write_packet`. The data passes through host memory, but the `Queue` interface avoids any extra copies between the frontend and backend layers — the backend reads/writes guest RAM directly. ## Lifecycle 1. The frontend creates a [`BufferAccess`](https://openvmm.dev/rustdoc/net_backend/trait.BufferAccess.html) implementation and one `QueueConfig` per queue. 2. It calls `endpoint.get_queues(configs, rss, &mut queues)`. 3. It enters the poll loop: `poll_ready` → `rx_avail` / `rx_poll` / `tx_avail` / `tx_poll`. 4. On shutdown, it drops the queues and calls `endpoint.stop()`. ## Backends | Backend | Crate | Transport | Platform | |---------|-------|-----------|----------| | TAP | `net_tap` | Linux TAP device | Linux | | DirectIO | `net_dio` | Windows vmswitch | Windows | | Consomme | `net_consomme` | User-space TCP/IP stack | Any | | MANA | `net_mana` | Azure hardware NIC (MANA/GDMA) | Linux | | Loopback | `net_backend` | Reflects TX → RX | Any | | Null | `net_backend` | Drops everything | Any | ## Frontends | Frontend | Crate | Guest interface | |----------|-------|-----------------| | virtio-net | `virtio_net` | Virtio network device | | netvsp | `netvsp` | VMBus synthetic NIC | | GDMA/BNIC | `gdma` | MANA Basic NIC (emulated GDMA) | ## Wrappers Wrappers implement `Endpoint` by delegating to an inner endpoint, adding cross-cutting behavior: - **PacketCapture** (`net_packet_capture`) — intercepts `rx_poll` and `tx_avail` to write PCAP-format packet traces. The capture path reads packet data from guest memory via `BufferAccess` and writes enhanced packet blocks to a ring buffer. Capture can be toggled at runtime; when disabled, the wrapper adds only an atomic load per call. - **Disconnectable** (`net_backend`) — supports hot-plug and hot-unplug by swapping the inner endpoint at runtime. ## RSS and multi-queue When a frontend supports Receive Side Scaling (RSS), it passes multiple `QueueConfig` entries and an `RssConfig` (hash key + indirection table) to `get_queues`. The backend creates one `Queue` per entry and uses the RSS configuration to steer incoming packets to the appropriate queue. Each queue is driven independently by its own async task. Currently `netvsp` and `net_mana` support multi-queue; `virtio_net` is limited to a single queue pair. --- ### Guide Src Reference Backends Storage (`Guide/src/reference/backends/storage.md`) # Storage backends Storage backends implement the [`DiskIo`](https://openvmm.dev/rustdoc/linux/disk_backend/trait.DiskIo.html) trait, the shared abstraction that all storage frontends use to read and write data. A frontend holds a [`Disk`](https://openvmm.dev/rustdoc/linux/disk_backend/struct.Disk.html) handle and doesn't know what kind of backend is behind it — the same frontend code works with a local file, a Linux block device, a remote blob, or a layered composition of multiple backends. ## Backend catalog | Backend | Crate | Wraps | Platform | Key characteristic | |---------|-------|-------|----------|--------------------| | FileDisk | [`disk_file`](https://openvmm.dev/rustdoc/linux/disk_file/index.html) | Host file | Cross-platform | Simplest backend. Blocking I/O via `unblock()`. | | Vhd1Disk | [`disk_vhd1`](https://openvmm.dev/rustdoc/linux/disk_vhd1/index.html) | VHD1 fixed file | Cross-platform | Parses VHD footer for geometry. | | VhdmpDisk | `disk_vhdmp` | Windows vhdmp driver | Windows | Dynamic and differencing VHD/VHDX. | | VhdxDisk | [`vhdx`](vhdx.md) | VHDX file | Cross-platform | Pure-Rust VHDX parser. Dynamic, fixed, and differencing. | | BlobDisk | [`disk_blob`](https://openvmm.dev/rustdoc/linux/disk_blob/index.html) | HTTP / Azure Blob | Cross-platform | Read-only. HTTP range requests. | | BlockDeviceDisk | [`disk_blockdevice`](https://openvmm.dev/rustdoc/linux/disk_blockdevice/index.html) | Linux block device or file | Linux | io_uring, resize via uevent, PR passthrough. Default for raw files on Linux in both OpenHCL and OpenVMM. | | NvmeDisk | [`disk_nvme`](https://openvmm.dev/rustdoc/linux/disk_nvme/index.html) | Physical NVMe (VFIO) | Linux/Windows | User-mode NVMe driver. Resize via AEN. | | StripedDisk | [`disk_striped`](https://openvmm.dev/rustdoc/linux/disk_striped/index.html) | Multiple Disks | Cross-platform | Stripes data across underlying disks. | ## Decorators Decorators wrap another [`Disk`](https://openvmm.dev/rustdoc/linux/disk_backend/struct.Disk.html) and transform I/O in transit. Features compose by stacking decorators without modifying the backends underneath. | Decorator | Crate | Transform | |-----------|-------|-----------| | CryptDisk | [`disk_crypt`](https://openvmm.dev/rustdoc/linux/disk_crypt/index.html) | XTS-AES-256 encryption. Encrypts on write, decrypts on read. | | DelayDisk | [`disk_delay`](https://openvmm.dev/rustdoc/linux/disk_delay/index.html) | Adds configurable latency to each I/O operation. | | DiskWithReservations | [`disk_prwrap`](https://openvmm.dev/rustdoc/linux/disk_prwrap/index.html) | In-memory SCSI persistent reservation emulation. | ## Layered disks A [`LayeredDisk`](https://openvmm.dev/rustdoc/linux/disk_layered/index.html) composes multiple layers into a single `DiskIo` implementation. Each layer tracks which sectors it has; reads fall through from top to bottom until a layer has the requested data. This powers the `memdiff:` and `mem:` CLI options. Two layer implementations exist today: - **RamDiskLayer** ([`disklayer_ram`](https://openvmm.dev/rustdoc/linux/disklayer_ram/index.html)) — ephemeral, in-memory. - **SqliteDiskLayer** ([`disklayer_sqlite`](https://openvmm.dev/rustdoc/linux/disklayer_sqlite/index.html)) — persistent, file-backed (dev/test only). The [storage pipeline](../architecture/devices/storage.md) page covers the full architecture: how frontends, backends, decorators, and the layered disk model connect, plus cross-cutting concerns like online disk resize and virtual optical media. --- ### Guide Src Reference Backends Vhdx (`Guide/src/reference/backends/vhdx.md`) # VHDX parser The `vhdx` crate (`vm/devices/storage/vhdx/`) is a pure-Rust implementation of the [VHDX format specification](https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-vhdx/). It supports dynamic, fixed, and differencing VHDX virtual hard disk files on all platforms — no Windows APIs or kernel drivers required. ## Features - **Create** and **open** VHDX files (read-only or writable) - Dynamic block allocation with four-priority free space management - Write-ahead log (WAL) for crash-consistent metadata updates - Sector bitmap tracking for partially-present (differencing) blocks - Block trim/unmap with multiple modes (file space, free space, zero, transparent, soft-anchor removal) - Concurrent flush coalescing - Parent locator parsing for differencing disk chains ## Architecture A VHDX file stores a virtual disk as a collection of fixed-size data blocks (default 2 MiB) tracked by a Block Allocation Table (BAT). The crate's write path uses a three-stage pipeline for crash consistency: ```text ┌───────────┐ commit ┌──────────┐ apply ┌────────────┐ │ Cache │ ──────────►│ Log Task │ ─────────►│ Apply Task │ │ (dirty │ dirty │ (WAL │ logged │ (final │ │ pages) │ pages │ writer) │ pages │ offsets) │ └───────────┘ └──────────┘ └────────────┘ ``` 1. The **cache** accumulates dirty 4 KiB metadata pages (BAT entries, sector bitmap bits). When the dirty count reaches a threshold or `flush()` is called, pages are committed to the log task. 2. The **log task** writes WAL entries to the circular log region in the VHDX file. On crash, `replay_log()` restores metadata from the WAL. 3. The **apply task** writes logged pages to their final file offsets. Backpressure is managed by a permit semaphore that limits in-flight pages. A flush sequencer coalesces concurrent flush requests so at most one file flush is in progress at a time. ## Lifecycle ```rust,ignore // Create a new empty VHDX file. create::create(&file, &mut params).await?; // Open for writing. let vhdx = VhdxFile::open(file) .block_alignment(2 * 1024 * 1024) .writable(&spawner) .await?; // Resolve a read — returns file-level ranges. let mut ranges = Vec::new(); let guard = vhdx.resolve_read(offset, len, &mut ranges).await?; // ... perform file I/O at the returned offsets ... drop(guard); // Resolve a write — returns file-level ranges + I/O guard. let mut ranges = Vec::new(); let guard = vhdx.resolve_write(offset, len, &mut ranges).await?; // ... write data at the returned offsets ... guard.complete().await?; // Flush to stable storage. vhdx.flush().await?; // Clean close (clears log GUID). vhdx.close().await?; ``` ## I/O model The crate separates **metadata I/O** from **payload I/O**. Metadata I/O (headers, BAT pages, sector bitmaps, WAL entries) is handled internally through the `AsyncFile` trait — the caller provides an `AsyncFile` implementation at open time and never thinks about metadata again. Payload I/O (guest data reads and writes) is the caller's responsibility. `resolve_read()` and `resolve_write()` translate virtual disk offsets into file-level byte ranges (`ReadRange` / `WriteRange`). The caller performs its own data I/O at those offsets using whatever mechanism it prefers (io_uring, standard file I/O, etc.), then finalizes metadata via the returned I/O guard. This separation lets the caller use a different, potentially more performant I/O path for bulk data without the crate imposing any particular strategy. - The `vhdx` crate provides the low-level VHDX format implementation and I/O resolution API. For OpenVMM integration, the `disklayer_vhdx` crate supplies a `LayerIo`-compatible backend used in the layered disk storage pipeline. - For differencing disks, the `vhdx` crate parses parent locator metadata, while `disklayer_vhdx::chain::open_vhdx_chain` walks and opens parent chains automatically. ## Parent lookup The Rust chain opener and `openvmm-img check` interpret parent metadata through `ParentLocator::vhdx_parent()`. This validates the locator type and requires a parseable parent linkage GUID. Both verify that the selected parent's data-write GUID matches the child's recorded linkage. Both callers use `VhdxParent::candidate_paths()` to enumerate paths. On Windows, the order is relative path, volume GUID path, then absolute Win32 path, as specified by MS-VHDX section 2.6.2.6.3. On Unix, only the relative locator is used, with backslashes translated to native separators. Windows-only locators are ignored even if their strings happen to name existing Unix files. `VhdxParent` stores locator strings without host-specific path validation or normalization. These are VHDX locator strings, not native filesystem paths; conversion to native paths happens during candidate enumeration. Path syntax is not validated by these helpers. Candidate enumeration does not open files. The callers select an existing candidate before verifying linkage; they do not retry later candidates on a linkage mismatch or repair stale locators. OpenVMM's normal Windows `.vhdx` path uses native VHDMP instead of this Rust chain opener. ## Related pages - [Storage backends](./storage.md) — catalog of all storage backends - [Storage pipeline](../architecture/devices/storage.md) — how frontends, backends, and layers connect --- ### Guide Src Reference Dev Feats Gdbstub (`Guide/src/reference/dev_feats/gdbstub.md`) # Hardware Debugging (gdbstub) Think EXDI from Hyper-V, except instead of using the EXDI interface, we use the [GDB Remote Serial Protocol](https://sourceware.org/gdb/onlinedocs/gdb/Remote-Protocol.html) (via the [`gdbstub`](https://github.com/daniel5151/gdbstub/) Rust library). Hardware debugging has several benefits over using an in-guest / in-kernel debugger: - Debugging early-boot scenarios (before UEFI / Windows / Linux debuggers are set up) - Debugging low-level ISRs - Non-intrusive debugging = easier to repro certain bugs - Debugging SNP/TDX/VBS Confidential VMs ## Enabling the Debugger ### OpenVMM 1. Pass the `--gdb ` flag at startup to enable the debug worker. e.g., `--gdb 9001` To pause the VM until the debugger has been attached, pass `--paused` at startup. ### OpenHCL 1. Pass the `OPENHCL_GDBSTUB=1` `OPENHCL_GDBSTUB_PORT=` parameters to enable gdbstub. e.g., `Set-VmFirmwareParameters -Name UhVM -CommandLine OPENHCL_GDBSTUB=1 OPENHCL_GDBSTUB_PORT=5900`. 2. To expose a TCP port, run `ohcldiag-dev.exe vsock-tcp-relay --allow-remote --reconnect `. To pause VTL0 boot until desired, pass `OPENHCL_VTL0_STARTS_PAUSED=1` as a parameter. Then once the debugger is attached, you can start VTL0 with `ohcldiag-dev.exe resume`. ## Connecting via GDB The quickest way to get connected to a OpenVMM VM is via `gdb` directly. Note that GDB does _not_ support debugging PDBs, so if you're trying to debug Windows, you'll be limited to plain disassembly. See the [`Connecting via WinDbg`](#connecting-via-windbg) section below if this is your use-case. On the flipside, if you're trying to debug ELF images with DWARF debug info (e.g., a vmlinux binary), then you'll likely want to use `gdb` directly, as it will support source-level debugging with symbols, whereas WinDbg will not. You can install `gdb` via your distro's package manager. e.g., on Ubuntu: ```bash sudo apt install gdb ``` Once `gdb` is installed, run it, and enter the following `gdb` command (swapping `9001` for whatever port you specified at the CLI) ```text target remote :9001 ``` If all goes well, you should get output similar to this: ```text (gdb) target remote :9001 Remote debugging using :9001 warning: No executable has been specified and target does not support determining executable automatically. Try using the "file" command. 0xfffff8015c054c1f in ?? () (gdb) ``` At this point, you can try some basic GDB commands to make sure things are working. e.g., start / interrupt the VM's execution using `cont` and `ctrl-c` ```text (gdb) cont Continuing. ^C # <-- hit ctrl-c in the terminal Thread 1 received signal SIGINT, Interrupt. 0xfffff8015c054c1f in ?? () (gdb) ``` e.g., inspecting register state ```text (gdb) info registers rax 0x0 0 rbx 0x0 0 rcx 0x40086 262278 rdx 0x0 0 rsi 0xffff960d4eea5010 -116491073990640 rdi 0x0 0 rbp 0x0 0x0 rsp 0xfffff8015b3f5ec8 0xfffff8015b3f5ec8 r8 0x0 0 r9 0xffffffff 4294967295 r10 0xfffff8015bfff1f0 -8790254554640 r11 0x0 0 r12 0xffffffff 4294967295 ...etc... ``` e.g., setting data breakpoints ```text (gdb) awatch *0xfffff804683190e0 Hardware access (read/write) watchpoint 1: *0xfffff804683190e0 ``` e.g., single stepping ```text 0xfffff8047a309686 in ?? () (gdb) si 0xfffff8047a309689 in ?? () ``` You may find [this blog post](https://blog.mattjustice.com/2018/08/24/gdb-for-windbg-users/) useful, as it includes a table of common `gdb` commands along with their WinDbg counterparts. ## Connecting via WinDbg WinDbg doesn't understand the GDB Remote Serial Protocol directly, but thankfully, some smart folks over on the WinDbg team have developed a GDB Remote Serial Protocol <-> WinDbg translation layer! For more information, see [Setting Up QEMU Kernel-Mode Debugging using EXDI](https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/setting-up-qemu-kernel-mode-debugging-using-exdi) Getting this working with OpenVMM or OpenHCL is as easy as following the guide, except you'll need to [enable our debugger](#enabling-the-debugger) instead of running QEMU. It's easiest to connect through the GUI. The steps are relatively simple: Open Windbgx -> File -> Attach to kernel -> EXDI. On the form, fill out: - Target Type: `QEMU` - Target Architecture: `X64` - Target OS: `Windows` - Image Screening heuristic size: `0xFFFE - NT` - Gdb server and port: `:` e.g., `127.0.0.1:1337` (use whatever port you set above) ### Known WinDbg Bugs - Hardware breakpoints are issued with [`ba`](https://learn.microsoft.com/en-us/windows-hardware/drivers/debuggercmds/ba--break-on-access-). The `Access Size` parameter is incorrectly multiplied by 8 when sent to the stub. Consequently, it _must_ be set to 1. - Unlike GDB, WinDbg doesn't implicitly set software breakpoints via our offered write_addrs implementation. --- ## Supported Features At the time of writing (8/16/24) the debugger supports the following operations: - read/write guest memory - read guest registers \* - start/interrupt execution - watchpoints - hardware breakpoints - single stepping ## TODO Features If you're looking for work, and want to improve the debugging experience for everyone, consider implementing one or more of the following features: - \* reading _all_ guest registers, including fpu, xmm, and various key msrs - software breakpoints: - Intercept guest breakpoint exceptions into VTL2 - writing guest registers - exposing the OpenVMM interactive console via the [`MonitorCmd`](https://docs.rs/gdbstub/latest/gdbstub/target/ext/monitor_cmd/trait.MonitorCmd.html) interface - Custom commands sent using `monitor` (gdb) / `.exdicmd` (WinDbg) - e.g., being able to invoke `x device/to/inspect` directly from the debugger - [any other features supported by the `gdbstub` library](https://github.com/daniel5151/gdbstub#debugging-features) --- ### Guide Src Reference Dev Feats Kdnet (`Guide/src/reference/dev_feats/kdnet.md`) # Kernel Debugging (KDNET) Kernel Debugging is available for Windows guests via KDNET over VMBus. ## Enabling and Starting the Debugger Set up KDNET on the guest and start the debugger as described on [Set up KDNET network kernel debugging manually | Microsoft Learn](https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/setting-up-a-network-debugging-connection). Setting `busparams` is not necessary. ```admonish warning If you're using a `memdiff:` disk (the default in most examples), guest state is not persisted between runs. You'll need to configure KDNET in the guest each time you launch — or configure it once on the base disk image before wrapping it with `memdiff:`. ``` ## With OpenVMM and WHP as Host Launch the VM with UEFI and networking enabled. KDNET requires `--net consomme` for the network transport: ```bash # Without OpenHCL cargo run -- --uefi --hv --net consomme \ --vmbus-scsi id=scsi0 \ --disk memdiff:file:path/to/windows.vhdx,on=scsi0 # With OpenHCL cargo run -- --uefi --hv --vtl2 --net consomme \ --igvm firmware=path/to/openhcl.igvm,personality=openhcl \ --vmbus-scsi id=scsi0 \ --disk memdiff:file:path/to/windows.vhdx,on=scsi0 ``` ### Known Issues - **Networking backend:** KDNET currently requires `--net consomme`. The `consomme` backend creates a new network adapter in the guest on each OpenVMM restart — this is harmless and can be ignored. Using `--net vmnic:` connects but hangs immediately due to an undiagnosed vmbusproxy issue. - **Debugger reconnection:** If you quit OpenVMM without shutting down the guest first, the same WinDbg instance cannot reconnect on next boot. Close and relaunch WinDbg to reconnect. - **Synic warnings with OpenHCL:** When launching an OpenHCL VM with KDNET, you may see a stream of `failed to signal synic` errors from `virt_whp::synic` for several seconds. These do not affect VM functionality and can be ignored. ## Debugging workflow Once KDNET is connected: 1. In WinDbg, go to **File → Kernel Debug → Net** and enter the key and port from the guest setup. 2. Start the VM. WinDbg should connect during Windows boot. 3. To break into the debugger: **Debug → Break** (or Ctrl+Break). ```admonish note KDNET debugs the VTL0 (guest) Windows kernel. For debugging the VTL2 (OpenHCL) side, see [Debugging OpenHCL](../openhcl/debugging.md) which covers serial logs, crash dumps, and `ohcldiag-dev`. ``` --- ### Guide Src Reference Devices Pci Ids (`Guide/src/reference/devices/pci_ids.md`) # PCI IDs This page records the PCI identifiers that OpenVMM reports for the PCI and PCIe devices it emulates. ```admonish note These IDs describe how a device advertises itself to the guest in PCI config space. They are distinct from VMBus/VPCI instance GUIDs, which identify device *instances* at runtime and are typically randomly generated per VM. ``` ## Microsoft vendor ID (`0x1414`) Most first-party OpenVMM devices report Microsoft's PCI vendor ID, `0x1414`. Device IDs under this vendor ID are **allocated and centrally managed by Microsoft**. You cannot pick an arbitrary device ID when adding a new emulated device: a device ID has to be formally assigned so that it does not collide with any other Microsoft product that shares this vendor ID. The values below are the assignments OpenVMM currently uses. These identifiers — the vendor ID, each device ID, and the default subsystem ID — are all defined in one place, the `microsoft` module of the `pci_core` crate (`vm/devices/pci/pci_core/src/microsoft.rs`), and referenced from each device. | Device ID | Device | Crate | | --------- | --------------------------- | ------------------ | | `0x00BA` | MANA/GDMA network adapter | `gdma` | | `0x5353` | VGA adapter | `vga` | | `0xC030` | PCIe root port | `pcie` | | `0xC031` | PCIe upstream switch port | `pcie` | | `0xC032` | PCIe downstream switch port | `pcie` | | `0xC03E` | NVMe controller | `nvme`, `nvme_test`| These devices all report a subsystem vendor ID and subsystem ID of `0`. ## Devices that report standardized IDs Some devices emulate a real, standardized controller and therefore report the hardware vendor/device ID that stock guest drivers bind to, rather than a Microsoft-allocated ID: - **virtio** devices use the virtio-over-PCI scheme: vendor ID `0x1AF4` and device ID `0x1040` plus the virtio device type. They report Microsoft's `0x1414` as the *subsystem vendor ID* to identify OpenVMM as the host environment, with a subsystem ID of `0x2000`. Source: `vm/devices/virtio/virtio/src/transport/pci.rs`. - **AMD IOMMU** reports AMD's vendor ID `0x1022` and device ID `0x1451` (a family 17h / Zen IOMMU) so that the guest's AMD IOMMU driver binds to it. It reports Microsoft's `0x1414` as its subsystem vendor ID and `0x2000` as its subsystem ID to identify OpenVMM as the host environment. Source: `vm/devices/iommu/amd_iommu/src/lib.rs`. --- ### Guide Src Reference Emulated Ipmi Kcs (`Guide/src/reference/emulated/ipmi_kcs.md`) # IPMI KCS OpenVMM provides a minimal virtual IPMI baseboard management controller (BMC) using the Keyboard Controller Style (KCS) system interface. The device allows UEFI guests to write and query a bounded System Event Log (SEL). The implementation is provided by the `ipmi_kcs` crate. Wire-level command and record definitions are in the `ipmi_protocol` crate. ## Configuration The device is created when the platform configuration enables IPMI. OpenHCL accepts this setting only for UEFI guests; PCAT and Linux-direct boot modes are not supported. The guest-visible register interface depends on the architecture: | Architecture | Interface | Registers | | --- | --- | --- | | x86-64 | Port I/O | Data at `0xCA2`; status/command at `0xCA3` | | AArch64 | MMIO | Data at `0xEFFE7000`; status/command at `0xEFFE7004` | Only one-byte register accesses are supported. ## Supported commands The virtual BMC implements the IPMI application `Get Device ID` command and these SEL storage commands: - Get SEL Info - Reserve SEL - Get SEL Entry - Add SEL Entry - Clear SEL - Get SEL Time - Set SEL Time The SEL stores at most 128 records. Reservations are exposed for guest software compatibility but are not enforced because the device has one serialized KCS requestor and no independent SEL mutators. Completed SEL records are retained by the device and forwarded to the host on a best-effort basis. Forwarding is limited to 256 records per trusted wall-clock second; reaching that limit does not remove records from the SEL. ## Servicing Saved state includes the KCS transaction, SEL records, record allocation state, reservation identifier, and guest-selected SEL time offset. Diagnostic forwarding counters and rate-limiter state are reset after restore. See the [`ipmi_kcs` rustdoc](https://openvmm.dev/rustdoc/ipmi_kcs/index.html) for the device API. --- ### Guide Src Reference Openhcl Debugging (`Guide/src/reference/openhcl/debugging.md`) # Debugging OpenHCL OpenHCL provides several debugging tools for investigating issues at the user-mode and kernel level. See [ohcldiag-dev](./diag/ohcldiag_dev.md) for the diagnostic client and [Tracing](./diag/tracing.md) for serial and event log tracing. ## On-demand memory dumps Use `ohcldiag-dev dump` to capture a live user-mode memory dump of the OpenHCL process at any time. The dump is an ELF core file that you can analyze with `lldb`, `gdb`, or `rust-lldb`. See the [ohcldiag-dev](./diag/ohcldiag_dev.md) page for the full command reference. ## User-mode crash dumps When an OpenHCL user-mode process crashes, a crash dump is automatically generated via the `underhill-crash` infrastructure and sent to the host over VMBus. On Windows hosts, these dumps are collected by Windows Error Reporting (WER). Use `lldb` or `gdb` to analyze the resulting ELF core dump. ## Kernel crash dumps Kernel-mode crash dumps (kdump) are **not currently supported** in OpenHCL. The OpenHCL kernel does not have `CONFIG_KDUMP` or `CONFIG_KEXEC` compiled in. If the kernel panics, no dump is generated. The only diagnostic output is COM3 serial (if enabled), which captures the panic message in real time. If the diagnostic service was running before the panic, `ohcldiag-dev` may have ring buffer messages up to that point, but it cannot capture the panic itself since the service is terminated by the panic. For debugging kernel-level issues, the best approach is to enable serial output via COM3 (see below) — it captures output from the very first instruction of kernel boot. ## Getting OpenHCL kernel logs (COM3 vs ohcldiag-dev) Two methods exist for capturing OpenHCL kernel (`kmsg`) output: **COM3 serial** uses direct UART I/O — it streams output from the very first instruction of OpenHCL boot in real time. **ohcldiag-dev** connects over vsock to the diagnostic service, which reads `/dev/kmsg`. Because `/dev/kmsg` preserves the kernel ring buffer, early boot messages are **replayed** when you connect — you get them even if you connect late. However, `ohcldiag-dev` only works if the diagnostic service successfully starts. | Boot phase | COM3 serial | ohcldiag-dev | |------------|:-----------:|:------------:| | Very early kernel (entry → memory setup) | ✅ live | ✅ replayed from ring buffer | | Device initialization (VMBus, etc.) | ✅ live | ✅ replayed from ring buffer | | Kernel panic before userspace | ✅ live | ❌ service never starts | | Boot hang (kernel stuck) | ✅ live | ❌ service never starts | | After diagnostic service starts | ✅ live | ✅ live | For most development, `ohcldiag-dev` is sufficient — boot succeeds and you get logs. COM3 is essential for debugging early boot failures, kernel panics, and init crashes. ## Enabling COM3 on Hyper-V COM3 support requires a host OS build that includes the `EnableAdditionalComPorts` code path. This was added in Windows 11 26H1 (build 28000+, Insider Canary channel). It is **not available** on Windows 11 24H2, 25H2, or Windows Server 2025. To enable COM3 on a supported build: ```powershell # Enable additional COM ports (requires reboot or VMMS restart) reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Virtualization" /v EnableAdditionalComPorts /t REG_DWORD /d 1 /f # Attach COM3 to a named pipe for a VM Set-VMComPort -VMName $VmName -Number 3 -Path "\\.\pipe\openhcl-com3" # Read the serial output hvc serial -c -p 3 -r $VmName ``` ```admonish note The `flowey` test runner (`install_vmm_tests_external_deps`) sets this registry key automatically when running VMM tests. If you run `cargo xflowey` to execute tests, you'll be prompted to allow the registry change. ``` ## Recommended host OS for OpenHCL development We recommend running a **Windows 11 26H1** Insider flight (Canary channel, build 28000+) on your development machine, if it is available for your device. This gives you COM3 support via the registry key above, plus access to the latest Hyper-V features. This matches the OS used on the project's self-hosted CI runners. Note that 26H1 Insider builds may not be available for all hardware — see [What to know about Windows 11 version 26H1](https://techcommunity.microsoft.com/blog/windows-itpro-blog/what-to-know-about-windows-11-version-26h1/4491941) for details and the [Windows Insider Flight Hub](https://learn.microsoft.com/en-us/windows-insider/flight-hub/) for availability. If you're on Windows 11 24H2/25H2 (builds 26100/26200), COM3 is not available via the registry key. Use `ohcldiag-dev` for kernel logs instead. ## ARM64 limitation On Hyper-V, additional serial ports (COM3+) are **not supported on ARM64**. The Hyper-V serial device for ARM64 does not support ports beyond COM1 and COM2. On ARM64 hosts, use `ohcldiag-dev` for OpenHCL kernel logs. This limitation is Hyper-V-specific — when running OpenVMM directly (without Hyper-V), ARM64 serial output works via PL011 UART. --- ### Guide Src Reference Openhcl Diag (`Guide/src/reference/openhcl/diag.md`) # Diagnostics This chapter discusses several of the diagnostic tools available when working with OpenHCL. --- ### Guide Src Reference Openvmm Graphical Console (`Guide/src/reference/openvmm/graphical_console.md`) # Graphical Console OpenVMM supports a graphical console exposed via VNC. To enable it, pass `--gfx` on the command line--this will start a VNC server on localhost port 5900. The port value can be changed with the `--vnc-port ` option. The bind address can be changed with `--vnc-listen
` (default: `127.0.0.1`). Use `--vnc-listen 0.0.0.0` for all IPv4 interfaces, or `--vnc-listen ::` for dual-stack IPv4+IPv6. The maximum number of concurrent clients can be set with `--vnc-max-clients ` (default: 16). With `--vnc-evict-oldest`, the oldest client is disconnected to make room for new connections instead of rejecting them. OpenVMM's VNC server also includes "pseudo" client-clipboard support, whereby the "Ctrl-Alt-P" key sequence will be intercepted by the server to type out the contents of the VNC clipboard. The VNC server supports RFB protocol versions 3.3, 3.7, and 3.8, with no authentication (security type "None"). It negotiates the following optional features based on client capabilities: * **Zlib compression** (encoding type 6) -- when the client advertises support, tile data is zlib-compressed to reduce bandwidth. * **Cursor pseudo-encoding** -- a local arrow cursor is rendered client-side when supported, eliminating server-side cursor compositing. * **DesktopSize pseudo-encoding** -- resolution changes are relayed to the client. Clients that do not advertise this encoding will be disconnected on a resolution change. * **QEMU extended key events** -- when available, the server uses scancode-based key input instead of xkeysym translation. * **Client reconnection** -- a new VNC client connecting will cleanly disconnect the previous session and take over. Once OpenVMM starts, you can connect to the VNC server using any supported VNC client. The following clients have been tested working with OpenVMM: * [TightVNC](https://www.tightvnc.com/download.php) * [TigerVNC](https://github.com/TigerVNC/tigervnc) * [RealVNC](https://www.realvnc.com/en/?lai_sr=0-4&lai_sl=l) * [noVNC](https://novnc.com/) (browser-based) * [MobaXterm](https://mobaxterm.mobatek.net/) (Windows terminal with built-in VNC) Once you have downloaded and installed it you can connect to `localhost` with the appropriate port to see your VM. --- ### Guide Src Reference Openvmm Logging (`Guide/src/reference/openvmm/logging.md`) # OpenVMM Logging ## Configuring OpenVMM logging messages to emit To configure logging, set the `OPENVMM_LOG` environment variable. The default level is `info`. For example: - `OPENVMM_LOG=debug` — enable debug events from all modules - `OPENVMM_LOG=info,mesh=trace` — enable trace events from the `mesh` crate and info events from everything else This is backed by the [`EnvFilter`](https://docs.rs/tracing-subscriber/0.2.17/tracing_subscriber/struct.EnvFilter.html) type; see the associated documentation for more details. ### Span events By default, OpenVMM does not log span enter/exit events. To enable them, set `OPENVMM_LOG_SPANS=1`. ### Rate limiting Trace events that can be triggered repeatedly by guest interactions are rate-limited by default. To disable rate limiting (useful for debugging), set `OPENVMM_DISABLE_TRACING_RATELIMITS=1`. ## Configuring OpenHCL Trace Logging OpenHCL also supports `EnvFilter`-style trace logging, configured via the `-c OPENVMM_LOG=` command line argument. The `-c` flag passes arguments to OpenHCL initialization. The filter syntax is the same as for OpenVMM. OpenHCL tracing can also be configured and dumped at runtime with `ohcldiag-dev`. See: [OpenHCL Diagnostics](../openhcl/diag/ohcldiag_dev.md) To retrieve OpenHCL log output at runtime, an output console or file must attach to the OpenHCL logging COM port. By default, OpenHCL outputs to `COM3`. To open a new terminal window with global OpenHCL debug level tracing enabled: ```shell openvmm -c "OPENVMM_LOG=debug" --com3 "term,name=VTL2 OpenHCL" [...] ``` Configure log levels of only a given module name: ```shell openvmm -c "OPENVMM_LOG=mesh=trace" --com3 "term,name=VTL2 OpenHCL" [...] ``` Multiple modules can be specified by separating them with a comma: ```shell openvmm -c "OPENVMM_LOG=mesh=trace,nvme_driver=trace" \ --com3 "term,name=VTL2 OpenHCL" [...] ``` ```admonish tip For more configuration examples of serial ports and the OpenVMM CLI, see the [Running OpenHCL Guide](../../../user_guide/openhcl/run/openvmm.md) and CLI `--help` output. ``` ## Capturing the ETW traces on the host On Windows, OpenVMM also logs to ETW, via the Microsoft.HvLite provider. To capture the trace, start a session: ```powershell logman start trace -ow -o trace.etl ` -p "{22bc55fe-2116-5adc-12fb-3fadfd7e360c}" 0xffffffffffffffff 0xff ` -nb 16 16 -bs 16 -mode 0x2 -ets ``` ```admonish note For OpenHCL traces, use `{AA5DE534-D149-487A-9053-05972BA20A7C}` as the provider GUID. ``` To flush: ```powershell logman update -ets -fd ``` To stop: ```powershell logman stop -ets ``` To decode as CSV: ```powershell tracerpt trace.etl -y -of csv -o trace.csv -summary trace-summary.txt ``` --- ### Guide Src Reference Openvmm Management (`Guide/src/reference/openvmm/management.md`) # Configuration and Management OpenVMM exposes three configuration and management interfaces. The CLI and interactive console support direct VM configuration and lifecycle operations, while gRPC and ttrpc provide programmatic control. ```admonish note title="Interface compatibility" These interfaces continue to evolve and may change between releases. Consult the documentation for your OpenVMM version, especially when upgrading. ``` - **CLI**: Used to configure and launch a single VM - This allows configuring static VM resource assignments, such as the number of processors, RAM size, UEFI, graphic console, etc.. as well as what devices are exposed to the Guest, such as a virtual NIC, Storage, vTPM, etc.. - **Interactive console**: Used to interact with a VM at runtime - This interface allows users to perform core VM operations such as stop, restart, save, restore, pause, resume, etc.. as well as things like storage hot-add, VTL2 servicing, running Inspect queries, etc.. - **gRPC / ttrpc**: Programmatic APIs for configuring and interacting with VMs. API compatibility and implementation coverage continue to evolve. ## Management Capabilities The following table summarizes the status of selected management features.
Feature Status
Suspend / Resume a VM OpenVMM's existing save/restore infrastructure theoretically supports this, but the end-to-end flow has not been wired up to any management interface at the moment.
Snapshots OpenVMM has core infrastructure for performing save/restore operations, but there are gaps in device support (notably: no support for storage snapshots).
Managing multiple running VMs

OpenVMM currently runs a single VM per-process.

It is not yet clear whether OpenVMM will support managing multiple VMs via a single OpenVMM process, or if OpenVMM will rely on external management tools (e.g: `libvirt`) interfacing with its existing APIs in order to launch and manage multiple VMs.
For additional feature requests and status information, search the [OpenVMM GitHub issues](https://github.com/microsoft/openvmm/issues). --- ### Guide Src User Guide Openhcl Next Steps (`Guide/src/user_guide/openhcl/next_steps.md`) # Next Steps To learn more about how OpenHCL works, and how to interact with it: | | | | --------------------------------- | --------------------------------------------- | | [Reference: OpenHCL Features] | Configure and interact with OpenHCL | | [Dev Guide: Building OpenHCL] | Build OpenHCL locally | | [Reference: OpenHCL Architecture] | Understand how OpenHCL works "under the hood" | [Reference: OpenHCL Features]: ../../reference/openhcl.md [Dev Guide: Building OpenHCL]: ../../dev_guide/getting_started/build_openhcl.md [Reference: OpenHCL Architecture]: ../../reference/architecture/openhcl.md --- ### Guide Src User Guide Openhcl Run (`Guide/src/user_guide/openhcl/run.md`) # Running OpenHCL This chapter provides a high-level overview of different ways to launch and interact with OpenHCL. ## OpenHCL Platforms - [On Windows - Hyper-V](./run/hyperv.md) - [On Windows - OpenVMM](./run/openvmm.md) - [On Linux](./run/openvmm_linux.md) ## High-level Overview In order to run OpenHCL, an existing host VMM must first load the OpenHCL environment into a VM, much akin to existing virtual firmware layers, like UEFI, or BIOS[^vtls]. OpenHCL is distributed as an [IGVM] file (Independent Guest Virtual Machine), which encapsulates all the directives and data required to launch a particular virtual machine configuration on any given virtualization stack. At this time, the only VMMs which are able to load and host OpenHCL IGVM files are Hyper-V, and OpenVMM. ## Obtaining a copy of OpenHCL To get started, ensure you have a copy of an OpenHCL IGVM firmware image, via one of the following options: ### Building OpenHCL Locally Follow the instructions on: [Building OpenHCL](../../dev_guide/getting_started/build_openhcl.md). Note: At this time, OpenHCL can only be built on Linux / WSL2. ### Pre-Built Binaries If you would prefer to try OpenHCL without building it from scratch, you can download pre-built copies of OpenHCL IGVM files from [OpenVMM CI](https://github.com/microsoft/openvmm/actions/workflows/openvmm-ci.yaml). Simply select a successful pipeline run (should have a Green checkbox), and scroll down to select an appropriate `*-openhcl-igvm` artifact for your particular architecture and operating system. [IGVM]: https://github.com/microsoft/igvm [^vtls]: Though, unlike UEFI / BIOS, OpenHCL is loaded into a distinct, higher privilege execution context within the VM, called [VTL2](../../reference/architecture/openhcl.md#vtls). --- ### Guide Src User Guide Openhcl Troubleshooting (`Guide/src/user_guide/openhcl/troubleshooting.md`) # OpenHCL Troubleshooting This page includes a miscellaneous collection of troubleshooting tips for common issues you may encounter when running OpenHCL. If you are still running into issues, consider filing an issue on the OpenVMM GitHub Issue tracker. ## \[Hyper-V] VTL2/VTL0 failed to start VTL2/VTL0 fails to boot is when either VTL2 or VTL0 has crashed. When the crash happens, they will emit an event to the Hyper-V worker channel. First, check `Hyper-V worker events` at `Applications and Services Logs -> Microsoft -> Windows -> Hyper-V-Worker-Admin` Alternatively, some queries you can use to get Hyper-V-Worker logs: - Display the `{n}` most recent events - `wevtutil qe Microsoft-Windows-Hyper-V-Worker-Admin /c:{n} /rd:true /f:text` - Export events to file - `wevtutil epl Microsoft-Windows-Hyper-V-Worker-Admin C:\vtl2_0_crash.evtx` ## Checking OpenHCL logging output OpenHCL logging output can be useful for debugging issues with startup or runtime behavior. See [OpenHCL Tracing](../../reference/openhcl/diag/tracing.md) for more details about how to enable OpenHCL logging. ## DeviceTree errors or warnings in the VTL2 kernel log 1. Retrieve the DeviceTree blob from OpenHCL: ```powershell uhdiag-dev.exe linux-uhvm00 file --file-path "/sys/firmware/fdt" > uh.dtb ``` 1. Install the DeviceTree compiler and convert the blob to the textual representation: ```sh sudo apt-get install dtc dtc -I dtb -o uh.dts uh.dtb ``` Check on the errors and warnings and should any have been produced, fix them in the DeviceTree generation code. If that doesn't resolve the issues, inspect the DT parsing code in the Linux kernel. --- ### Guide Src User Guide Openvmm Alpine (`Guide/src/user_guide/openvmm/alpine.md`) # Alpine Linux, via Direct Boot This guide boots Alpine Linux from a cloud disk image using Linux direct boot (no UEFI or BIOS firmware required). It demonstrates how to set up PCIe, attach virtio-blk disks, and use virtio-net for networking. ## Quick start Run the setup script from the openvmm repo: ```bash ./scripts/setup-alpine.sh [output-dir] ``` The default output directory is `./alpine-direct-boot`. The script prints the boot command when it finishes. It also writes a `README` file to the output directory with the same command for future reference. Login: `root` / `alpine`. Quit OpenVMM with `ctrl-q` then `q`. ```admonish warning The default password is intended for local development only. Change it or configure SSH keys before using a networking backend that exposes the VM to other hosts. ``` ## What the script does 1. **Downloads** the Alpine v3.21 "nocloud" UEFI tiny cloud image (qcow2). 2. **Converts** it to a raw disk image (`disk.raw`) with `qemu-img`. 3. **Extracts the kernel and initramfs** (`vmlinuz-virt`, `initramfs-virt`) from the ext4 root partition inside the image. This step requires `sudo` for the loopback mount. 4. **Extracts an ELF kernel** (`vmlinux-virt`) from the bzImage. OpenVMM's direct boot loader requires an uncompressed ELF, not a compressed bzImage. The script uses Python to find and decompress the gzip-embedded ELF. 5. **Creates a cloud-init data disk** (`cidata.img`), a small FAT image labeled `cidata` containing `user-data` and `meta-data` files. Alpine's `tiny-cloud` service reads these on first boot. The `user-data` uses `runcmd` to set the root password and add a getty on `hvc0` (Alpine does not spawn one by default). ### Required host tools `curl`, `qemu-img`, `python3`, `mkfs.vfat`, `mcopy` (from mtools), and `sudo` (for the loopback mount). ## Key points about the boot command - **`--hv`**: Enables Hyper-V enlightenments for better performance. - **`--pcie-root-complex` and `--pcie-root-port`**: Required for PCI device visibility. The default direct boot DSDT does not include a PCI bus, so without these flags, virtio devices will not be detected by the kernel. - **`--virtio-blk ...,pcie_port=disk`**: Attaches the raw disk image as a virtio-blk device on a PCIe root port. - **`--virtio-blk ...,ro,pcie_port=cidata`**: Attaches the cloud-init data image read-only on a second PCIe root port. - **`--virtio-net pcie_port=net:consomme`**: Adds a virtio-net NIC using the consomme user-mode NAT backend, on a third PCIe root port. - **`--com1 none`**: Disables the default COM1 serial port (which would otherwise claim the console). - **`--virtio-console console --virtio-console-pcie-port console`**: Adds a virtio-console device (`/dev/hvc0` in the guest) on a fourth PCIe root port. This replaces COM1 as the interactive console. - **`root=/dev/vda2`**: The root filesystem is on partition 2 of the virtio disk. - **`modules=virtio_pci,virtio_blk,ext4`**: Tells the Alpine initramfs to load these modules early, before attempting to mount the root filesystem. - **`console=hvc0`**: Directs the kernel to use the virtio-console (`/dev/hvc0`) for console output instead of the legacy serial port. ```admonish tip Use `ctrl-q` then `q` to quit OpenVMM (not `ctrl-c`). ``` --- ### Guide Src User Guide Openvmm Next Steps (`Guide/src/user_guide/openvmm/next_steps.md`) # Next Steps To learn more about how OpenVMM works, and how to interact with it: | | | | --------------------------------- | --------------------------------------------- | | [Reference: OpenVMM Features] | Configure and interact with OpenVMM | | [Dev Guide: Building OpenVMM] | Build OpenVMM locally | | [Reference: OpenVMM Architecture] | Understand how OpenVMM works "under the hood" | [Reference: OpenVMM Features]: ../../reference/openvmm.md [Dev Guide: Building OpenVMM]: ../../dev_guide/getting_started/build_openvmm.md [Reference: OpenVMM Architecture]: ../../reference/architecture/openvmm.md --- ### Guide Src User Guide Openvmm Run (`Guide/src/user_guide/openvmm/run.md`) # Running OpenVMM This page offers a high-level overview of different ways to launch and interact with OpenVMM. These examples provide a starting point for launching and configuring OpenVMM. ## Obtaining a copy of OpenVMM To get started, ensure you have a copy of the OpenVMM executable and its runtime dependencies, via one of the following options: ### Building OpenVMM Locally Follow the instructions on: [Building OpenVMM](../../dev_guide/getting_started/build_openvmm.md). ### Pre-Built Binaries If you would prefer to try OpenVMM without building it from scratch, you can download pre-built copies of the binary from [OpenVMM CI](https://github.com/microsoft/openvmm/actions/workflows/openvmm-ci.yaml). Simply select a successful pipeline run (should have a Green checkbox), and scroll down to select an appropriate `*-openvmm` artifact for your particular architecture and operating system. **You must be signed into GitHub in order to download artifacts**. ## Examples ```admonish tip These examples all use `cargo run --`, with the assumption that you are a developer building your own copy of OpenVMM locally! To run these examples using a pre-compiled copy of OpenVMM, swap `cargo run --` with `/path/to/openvmm`. ``` ~~~admonish tip title="UEFI firmware required when running outside cargo" When running via `cargo run`, environment variables in `.cargo/config.toml` automatically point OpenVMM to the `mu_msvm` UEFI firmware (`MSVM.fd`) downloaded by `cargo xflowey restore-packages`. When running the `openvmm` binary directly, these environment variables are **not set**, and you will get: > fatal error: must provide uefi firmware when booting with uefi To fix this, **explicitly pass the firmware** using the `firmware` option: ```shell openvmm --uefi firmware=path/to/MSVM.fd \ --vmbus-scsi id=scsi0 \ --disk memdiff:path/to/disk.vhdx,on=scsi0 ``` If you ran `cargo xflowey restore-packages`, the firmware is at: ```text .packages/hyperv.uefi.mscoreuefi.x64.RELEASE/MsvmX64/RELEASE_VS2022/FV/MSVM.fd # x64 .packages/hyperv.uefi.mscoreuefi.AARCH64.RELEASE/MsvmAARCH64/RELEASE_CLANGPDB/FV/MSVM.fd # aarch64 ``` If you used `cargo xflowey vmm-tests-run --build-only --dir `, the firmware is copied into that output directory under the same relative path. Alternatively, set the environment variable so you don't need the flag each time: ```shell # x64 export X86_64_OPENVMM_UEFI_FIRMWARE=path/to/MSVM.fd # aarch64 export AARCH64_OPENVMM_UEFI_FIRMWARE=path/to/MSVM.fd ``` ~~~ If you run into any issues, please refer to [Troubleshooting](./troubleshooting.md). ### _Preface:_ Quitting OpenVMM By default, OpenVMM will connect the guests's COM1 serial port to the current terminal session, forwarding all keystrokes directly to the VM. As such, a simple `ctrl-c` does not suffice to quit OpenVMM! Instead, you can type `crtl-q` to enter OpenVMM's [interactive console](../../reference/openvmm/management/interactive_console.md), and enter `q` to quit. ### Sample Linux Kernel, via direct-boot This example will launch Linux via direct boot (i.e: without going through UEFI or BIOS), and appends `single` to the kernel command line. The Linux guest's console will be hooked up to COM1, and is relayed to the host terminal by default. To launch Linux with an interactive console into the shell within initrd, simply run: ```shell cargo run ``` This works by setting the default `[env]` vars in `.cargo/config.toml` to configure OpenVMM to use a set of pre-compiled test kernel + initrd images, which are downloaded as part of the `cargo xflowey restore-packages` command. Note that this behavior only happens when run via `cargo run` (as `cargo` is the tool which ensures the required env-vars are set). The source for the sample kernel + initrd can be found on the [microsoft/openvmm-deps](https://github.com/microsoft/openvmm-deps) repo. The kernel and initrd can be controlled via options: * `--kernel `: The kernel image. Must be an uncompressed kernel (vmlinux, not bzImage). * `--initrd `: The initial ramdisk image. * `-c ` or `--cmdline `: Extra kernel command line options, such as `root=/dev/sda`. ### Windows, via UEFI This example will launch a modern copy of Windows via UEFI, using the `mu_msvm` firmware package. A copy of the `mu_msvm` UEFI firmware is automatically downloaded via `cargo xflowey restore-packages`. ```shell cargo run -- --uefi \ --vmbus-scsi id=scsi0 \ --disk memdiff:path/to/windows.vhdx,on=scsi0 \ --gfx ``` For more info on `--gfx`, and how to actually interact with the VM using a mouse/keyboard/video, see the [Graphical Console](../../reference/openvmm/graphical_console.md) docs. The file `windows.vhdx` can be any format of VHD(X). VHDX files (dynamic, fixed, and differencing) are supported on non-Windows platforms via the pure-Rust [`vhdx`](../../reference/backends/vhdx.md) parser. On Windows, `.vhdx` files use the native kernel-mode VHD path instead. Fixed VHD1 images work on all platforms. Dynamic and differencing VHD1 files are **not** supported — convert them to VHDX first: ```bash qemu-img convert -f vpc -O vhdx dynamic.vhd converted.vhdx ``` Also, note the use of `memdiff`, which creates a memory-backed "differencing disk" shim between the VMM and the backing disk image, which ensures that any writes the VM makes to the VHD are not persisted between runs. This is very useful when iterating on OpenVMM code, since booting the VM becomes repeatable and you don't have to worry about shutting down properly. Use `file` instead for normal persistent storage. ### OpenHCL, via Linux Direct Boot This example will boot OpenHCL in Linux direct mode, running a minimal shell inside VTL2. This is the same configuration used by the `openhcl_linux_direct_x64` integration tests. First, build the test artifacts from Linux or WSL using `vmm-tests-run --build-only`. The IGVM must be built on Linux: ```shell cargo xflowey vmm-tests-run --build-only --dir --target windows-x64 ``` ```admonish tip If you only need the IGVM binary (and already have `openvmm.exe`), you can use `cargo xflowey build-igvm` instead — it's faster than building the full test suite. ``` This places `openvmm.exe` and `openhcl-x64-test-linux-direct.bin` in the `` directory. Then, on Windows, from the `` directory: ```powershell .\openvmm.exe ` --hv ` --vtl2 ` --igvm "firmware=openhcl-x64-test-linux-direct.bin,personality=openhcl" ` -c "panic=-1 reboot=triple UNDERHILL_SERIAL_WAIT_FOR_RTS=1 UNDERHILL_CMDLINE_APPEND=rdinit=/bin/sh" ` -m 2GB ` --vmbus-com1-serial "term,name=VTL0 Linux" ` --com3 "term,name=VTL2 OpenHCL" ` --vmbus-vtl2-vsock-path $env:temp\ohcldiag-dev ``` ```admonish warning The `--vmbus-com1-serial` flag is **required** when using `rdinit=/bin/sh`. The shell running as PID 1 needs a controlling terminal (tty) — without one it exits immediately, causing a kernel panic and infinite reboot loop. The `--com3` flag is optional but recommended — it gives you VTL2 (OpenHCL) kernel console output for debugging. ``` For more details on running OpenHCL on OpenVMM, including [VMBus relay](../../reference/architecture/openhcl/vmbus.md) and device assignment, see [Running OpenHCL: OpenVMM](../openhcl/run/openvmm.md). ### Alpine Linux, via Direct Boot See the dedicated [Alpine Linux](./alpine.md) guide for a full walkthrough of booting Alpine from a cloud disk image using direct boot with PCIe and virtio-blk. ### DOS, via PCAT BIOS While DOS in particular is not a scenario that the OpenVMM has heavily invested in, the fact DOS is able to boot in OpenVMM serves as a testament to OpenVMM's solid support of legacy x86 devices and infrastructure. The following command will boot a copy of DOS from a virtual floppy disk, using the [Hyper-V PCAT BIOS](../../reference/devices/firmware/pcat_bios.md). Booting via PCAT is not just for DOS though! Many older operating systems, including older copies of Windows / Linux, require booting via BIOS. ```bash cargo run -- --pcat --gfx --floppy memdiff:/path/to/msdos.vfd --pcat-boot-order=floppy,optical,hdd ``` --- ### Guide Src User Guide Openvmm Snapshots (`Guide/src/user_guide/openvmm/snapshots.md`) # Snapshots OpenVMM supports saving and restoring VM snapshots, allowing you to capture the complete state of a running VM and resume it later. ## Overview A snapshot captures three pieces of state: - **Guest RAM** — the full contents of guest memory - **Device state** — the saved state of all emulated devices - **Manifest** — metadata describing the snapshot (architecture, memory size, VP count, page size, etc.) These are stored as three files in a snapshot directory: | File | Contents | |-----------------|---------------------------------------------| | `manifest.bin` | Protobuf-encoded snapshot metadata | | `state.bin` | Serialized device state | | `memory.bin` | Memory backing file | ## Prerequisites Snapshots require **file-backed guest memory**. Pass `file=` in the `--memory` option when launching the VM so that guest RAM is written to a file on disk rather than held in anonymous memory. ```admonish warning The memory backing file and the snapshot directory must be on the **same filesystem**. OpenVMM creates a hard link from the backing file to `memory.bin` inside the snapshot directory, which does not work across filesystem boundaries. ``` ## Saving a snapshot Start a VM with file-backed memory: ```bash cargo run -- \ --uefi \ --vmbus-scsi id=scsi0 \ --disk memdiff:file:path/to/disk.vhdx,on=scsi0 \ --memory size=4096M,file=path/to/memory.bin ``` Once the VM is running, open the interactive console and issue a save command, specifying the output directory: ```text save-snapshot path/to/snapshot-dir ``` OpenVMM writes `manifest.bin`, `state.bin`, and a hard link to `memory.bin` into the specified directory. ```admonish warning After saving, the VM remains **paused** and resume is blocked. Resuming would mutate guest RAM through `memory.bin`, corrupting the snapshot. Use `shutdown` to exit OpenVMM after saving. ``` ## Restoring a snapshot To restore, pass the snapshot directory with `--restore-snapshot`: ```bash cargo run -- \ --uefi \ --vmbus-scsi id=scsi0 \ --disk memdiff:file:path/to/disk.vhdx,on=scsi0 \ --memory size=4096M \ --processors 4 \ --restore-snapshot path/to/snapshot-dir ``` `--restore-snapshot` automatically opens `memory.bin` from the snapshot directory, so `file=...` should not be specified in `--memory` (the two options are mutually exclusive). ```admonish note The `--memory` and `--processors` values must match the values recorded in the snapshot manifest. If they do not match, OpenVMM will report a validation error and refuse to start. ``` ## Device configuration on restore The snapshot only stores device *state*, not device *configuration*. All device flags (e.g. `--disk`, `--nic`, `--serial`, `--virtio-blk`, etc.) must be specified on the restore command line exactly as they were when the snapshot was saved — they are not read from the snapshot. The snapshot manifest validates that `--memory`, `--processors`, architecture, and page size match the values recorded at save time. However, it does **not** record the list of CLI device flags. Instead, device configuration compatibility is enforced at the state-unit level: each emulated device saves its state under a unique name (e.g. `"pit"`, `"vmbus"`, `"ide"`), and restore matches saved-state entries to the currently instantiated devices by name. The rules are: | Scenario | Result | |---|---| | Device set matches exactly | Restore succeeds | | Snapshot contains a device not in current config | **Restore fails** — unknown unit name | | Current config has a device not in snapshot | Restore succeeds — device starts in its default/initial state | In practice this means: - You must pass the **same device flags** on restore as you did on save. Removing a device that was present at save time will cause restore to fail. - Adding a *new* device that was not present at save time is technically allowed — the new device will start in its power-on default state. This is not tested and the device may not be functional, since the guest OS will not have enumerated or initialised it during boot. The supported path is to use the same device flags on save and restore. ```admonish warning There is no single error message that tells you "your device configuration changed". Instead you will see errors like `restore failed: unknown unit name` when saved-state entries cannot be matched. If you see this, compare your restore command line with the one used at save time. ``` ## Device save/restore support Not all devices support save/restore. If a VM includes a device that does not support saving, the `save-snapshot` command will fail with `SaveError::NotSupported`. The following table summarises support for the device types relevant to OpenVMM snapshots: | Device | Bus | Save/Restore | |---|---|---| | PIT, PIC, I/O APIC, DMA | Chipset (ISA) | Yes | | CMOS RTC, Power Management | Chipset (ISA) | Yes | | i8042 (PS/2 keyboard/mouse) | Chipset (ISA) | Yes | | Serial 16550 | Chipset (ISA) / PCI | Yes | | UEFI firmware | Chipset (MMIO) | Yes | | Framebuffer | Chipset (MMIO) | Yes | | TPM | Chipset (MMIO) | Yes | | IDE controller | PCI | Yes | | PIIX4 bridges, bus, PM, RTC | PCI | Yes | | Generic PCI bus | PCI | Yes | | StorVsp (SCSI) | VMBus | Yes | | NetVsp (NIC) | VMBus | Yes | | Shutdown / Timesync / KVP ICs | VMBus | Yes | | VMBus Keyboard / Mouse / Video | VMBus | Yes | | Guest Emulation Log | VMBus | Yes | | virtio-blk | Virtio (PCI/MMIO) | Yes | | virtio-net | Virtio (PCI/MMIO) | Yes | | virtio-pmem | Virtio (PCI/MMIO) | Yes | | virtio-rng | Virtio (PCI/MMIO) | Yes | | NVMe | PCI | **No** | | VGA | PCI | **No** (`todo!()`) | | GDMA (MANA network) | PCI | **No** (`todo!()`) | | PCIe root complex / switch | PCIe | **No** | | Assigned PCI (pass-through) | PCI | **No** | | Relayed vPCI | PCI | **No** | | PCAT BIOS firmware | Chipset (ISA) | **No** (see limitations) | | virtio-9p, virtiofs | Virtio (PCI/MMIO) | **No** | | virtio-console | Virtio (PCI/MMIO) | **No** | | Guest Crash Device | VMBus | **No** | | Guest Emulation Device (GED) | VMBus | **No** | | VMBus serial (host) | VMBus | **No** | | Vmbfs | VMBus | **No** | ```admonish tip If you are unsure whether your VM configuration supports snapshots, try issuing `save-snapshot` to a scratch directory. The save will fail immediately with a clear error if any active device does not support it. ``` ## Limitations - Snapshots are **not portable** across architectures (e.g., you cannot restore an x86_64 snapshot on aarch64) - After restoring, `memory.bin` in the snapshot directory becomes the live guest RAM backing file and will be modified as the VM runs. To restore from the same snapshot multiple times, copy the snapshot directory before each restore. - VMs using VPCI or PCIe devices do not currently support save/restore - OpenHCL-based VMs do not currently support this snapshot mechanism - VMs using PCAT firmware do not support save/restore - `--memory` and `--processors` must be specified on restore and match the snapshot manifest values. A future version may read these from the snapshot automatically. --- ### Guide Src User Guide Openvmm Troubleshooting (`Guide/src/user_guide/openvmm/troubleshooting.md`) # OpenVMM Troubleshooting This page includes a miscellaneous collection of troubleshooting tips for common issues you may encounter when running OpenVMM. If you are still running into issues, consider filing an issue on the OpenVMM GitHub Issue tracker. ### failed to open `/dev/kvm/` **Error:** ```text fatal error: failed to launch vm worker Caused by: 0: failed to launch worker 1: failed to create the prototype partition 2: kvm error 3: failed to open /dev/kvm 4: Permission denied (os error 13) ``` **Solution:** When launching from a Linux/WSL host, your user account will need permission to interact with `/dev/kvm`. For example, you could add yourself to the group that owns that file: ```bash sudo usermod -a -G ``` For this change to take effect, you may need to restart. If using WSL2, you can simply restart WSL2 (run `wsl --shutdown` from Powershell and reopen the WSL window). Alternatively, for a quick-and-dirty solution that will only persist for the duration of the current user session: ```bash sudo chown /dev/kvm ``` --- ### Guide Src User Guide Openvmm Vfio (`Guide/src/user_guide/openvmm/vfio.md`) # VFIO Device Assignment This page explains how to assign a physical PCI device to an OpenVMM guest using Linux VFIO. VFIO device assignment lets a guest VM directly access a physical PCI device (such as an NVMe controller or GPU). The guest sees the real device in its PCI bus and can interact with its config space. ```admonish warning VFIO device assignment is experimental. PCI config space, BAR MMIO passthrough, MSI-X interrupts (via irqfd), and DMA are functional. Devices such as NVMe controllers work end-to-end. ``` ## Overview OpenVMM running on a Linux host can assign physical PCI devices to guest VMs using VFIO. The device is bound to the `vfio-pci` kernel driver, then OpenVMM opens it via VFIO and presents it to the guest as a PCIe endpoint. The guest sees the device's real config space with some filtering applied: each assigned device appears as a single-function device (the multi-function bit is cleared), and certain PCIe extended capabilities are hidden because they don't make sense in a virtual topology. Currently filtered extended capabilities are SR-IOV, ARI, and Resizable BAR. ```text Linux Host └── OpenVMM └── Guest VM └── sees physical PCI device on its PCI bus ``` ## Prerequisites - A Linux host with IOMMU support enabled (Intel VT-d or AMD-Vi) - A PCI device available for passthrough - The `vfio-pci` kernel module loaded - The `vfio_iommu_type1` kernel module loaded ## Step 1: Identify the device Find the PCI device you want to assign: ```bash lspci -D ``` Look for the device's PCI address in `domain:bus:device.function` format, for example `0000:01:00.0`. ## Step 2: Enable unsafe interrupts Some IOMMU implementations do not support interrupt remapping. If VFIO fails to set up the IOMMU with an "interrupt remapping" error, allow it to proceed without: ```bash echo 1 | sudo tee /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts ``` ```admonish note This flag is required in environments where the IOMMU does not support interrupt remapping (e.g., some nested virtualization setups). The "unsafe" label means a device could theoretically forge interrupt messages. In practice, this is acceptable when the host platform already constrains device behavior. ``` To make this persistent across reboots, add a modprobe config: ```bash echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" | sudo tee /etc/modprobe.d/vfio.conf ``` ## Step 3: Bind the device to vfio-pci If the device is currently bound to another driver (e.g., `nvme`), unbind it first: ```bash echo "0000:01:00.0" | sudo tee /sys/bus/pci/devices/0000:01:00.0/driver/unbind ``` Then bind to `vfio-pci`: ```bash echo "vfio-pci" | sudo tee /sys/bus/pci/devices/0000:01:00.0/driver_override echo "0000:01:00.0" | sudo tee /sys/bus/pci/drivers/vfio-pci/bind ``` Verify the binding: ```bash ls -la /sys/bus/pci/devices/0000:01:00.0/driver # Should show: ... -> ../../../../bus/pci/drivers/vfio-pci ``` ```admonish warning If the device is an NVMe controller backing a mounted filesystem, unbinding it will cause data loss. Make sure you are not using the device before unbinding. ``` ## Step 4: Verify VFIO group Check that a VFIO group device was created: ```bash ls /dev/vfio/ ``` You should see a numbered group (e.g., `/dev/vfio/0`). An IOMMU group is required — NoIommu mode is not supported. ## Step 5: Launch OpenVMM with the VFIO device Use the `--vfio` flag to assign the device to a PCIe root port. You also need to create a PCIe root complex and root port for the device to attach to: ```bash sudo openvmm \ --pcie-root-complex rc0 \ --pcie-root-port rc0:rp0 \ --vfio host=0000:01:00.0,port=rp0 \ --kernel /path/to/vmlinux \ --initrd /path/to/initrd \ --cmdline "console=ttyS0" \ --com1 console \ --memory 256M \ --processors 2 ``` The `--vfio` value is a comma-separated list of `key=value` pairs: - `host=` (required) — the PCI BDF of the VFIO device on the host (e.g., `0000:01:00.0`) - `port=` (required) — the name of the PCIe root port to attach the device to (must match a `--pcie-root-port` name) - `iommu=` (optional) — reference to an `--iommu` context; see [Using iommufd (cdev path)](#using-iommufd-cdev-path) below - `bar0=host` through `bar5=host` (optional) — pin the specified BAR to its physical host address (GPA = HPA); see [Peer-to-peer DMA](#peer-to-peer-dma) below - `bar0=0x` through `bar5=0x` (optional) — pin the specified BAR to an explicit host physical address; use this for BARs synthesized by a VFIO variant driver whose address is not reported through sysfs ```admonish tip You can assign multiple devices by adding more root ports and `--vfio` flags: --pcie-root-port rc0:rp0 \ --pcie-root-port rc0:rp1 \ --vfio host=0000:01:00.0,port=rp0 \ --vfio host=334c:00:00.0,port=rp1 ``` ### Using iommufd (cdev path) By default, `--vfio` uses the legacy VFIO group/container interface with the Type1v2 IOMMU driver. On hosts with Linux kernel 6.6 or newer, OpenVMM can instead use the modern VFIO cdev (per-device fd) + iommufd interface. Enable it by declaring an `--iommu` context and referencing it from each `--vfio` device with the `iommu=` key: ```bash sudo openvmm \ --pcie-root-complex rc0 \ --pcie-root-port rc0:rp0 \ --iommu id=iommu0 \ --vfio host=0000:01:00.0,port=rp0,iommu=iommu0 \ ... ``` The `--iommu` syntax is `id=`. All `--vfio` devices that reference the same `id` share a single iommufd IOAS (one set of IOMMU page tables and one DMA mapper registration). The IOAS is allocated on demand the first time a device referencing the id is opened. Devices opened via the cdev path read their device node from `/sys/bus/pci/devices//vfio-dev/vfioN` and open `/dev/vfio/devices/vfioN` instead of `/dev/vfio/`. ### Assigning a device behind a guest IOMMU (aarch64) By default an assigned device sees guest physical addresses directly: the guest programs DMA with GPAs and the host IOMMU maps them to host physical addresses. To instead expose an IOMMU *to the guest* — so the guest builds its own IOVA→GPA translations — put the device's root complex behind an accelerated SMMUv3 with `--smmu rc=,accel`: ```bash sudo openvmm \ --pcie-root-complex rc0 \ --pcie-root-port rc0:rp0 \ --smmu rc=rc0,accel \ --iommu id=iommu0 \ --vfio host=0000:01:00.0,port=rp0,iommu=iommu0 \ ... ``` The guest's stage-1 page tables are installed into the host IOMMU as a nested domain, so the physical SMMU performs the translation. This needs a host SMMUv3 with iommufd nesting support, every device behind the same SMMU must reference the same `--iommu` id, and the guest must boot with ACPI (`--device-tree` is rejected). Omitting `accel` leaves the SMMU translating in software, which cannot program the host IOMMU — assignment then fails rather than silently bypassing the guest's translations. See [Arm SMMUv3](../../reference/emulated/iommu/smmuv3.md) for the device reference. ## Step 6: Verify in the guest If the guest boots with PCI support, the assigned device should be visible: ```bash lspci ``` The device will appear with its real vendor and device ID from the physical hardware. ## Peer-to-peer DMA Normally, peer-to-peer (P2P) DMA between two passthrough devices works via ATS (Address Translation Services): each device translates DMA addresses through the IOMMU, so guest BAR placement doesn't matter. Some platforms — notably NVIDIA GB200 and GB300 — do not support ATS in their root complex. On these machines, P2P DMA between devices (e.g., GPU and NIC) works by disabling ACS on the physical PCIe switch so that TLPs route directly between devices without going through the IOMMU. Since there is no translation layer, the devices use raw physical addresses for P2P DMA. This means the guest BAR addresses must be identity-mapped to the host BAR addresses (GPA = HPA), or P2P DMA will target the wrong location. To enable this, pin the relevant BARs with `bar=host` on each `--vfio` device and set `preserve_bars` on the root complex so the PCI resource allocator keeps pinned BARs at their physical addresses: ```bash sudo openvmm \ --pcie-root-complex \ rc0,preserve_bars,low_mmio_base=0xc0000000,high_mmio_base=0x100000000 \ --pcie-root-port rc0:rp0 \ --pcie-root-port rc0:rp1 \ --vfio host=0000:01:00.0,port=rp0,bar0=host \ --vfio host=0000:02:00.0,port=rp1,bar0=host \ ... ``` For a BAR synthesized by a VFIO variant driver, the host physical address may not appear in the device's sysfs resource table. In that case, provide the address directly with `bar=0x`. For example, the `nvgrace-gpu` driver exposes CPU-coherent GPU memory as a synthetic 64-bit BAR4 whose physical base comes from the `nvidia,gpu-mem-base-pa` ACPI `_DSD` property: ```bash --vfio host=0008:06:00.0,port=rp0,bar0=host,bar2=host,bar4=0x110000000000 ``` The `low_mmio_base=` and `high_mmio_base=` options pin the MMIO apertures to fixed addresses so the allocator can place both pinned and dynamic BARs correctly. ## Optional: use hugetlb-backed guest RAM For large VMs, VFIO DMA setup can be much faster when guest RAM is backed by Linux hugetlb pages. Without hugetlb backing, the kernel may have to pin each 4 KB page individually during `VFIO_IOMMU_MAP_DMA`. With 2 MB or 1 GB hugetlb pages, the same mapping work is performed over far fewer pages. Hugetlb pages must be reserved on the host before starting OpenVMM. For example, reserve enough 2 MB pages for a 64 GB VM: ```bash echo 32768 | sudo tee /proc/sys/vm/nr_hugepages ``` For 1 GB pages, reserve pages through the size-specific sysfs pool: ```bash echo 64 | sudo tee \ /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages ``` Then request hugepage-backed RAM with the `--memory` option: ```bash sudo openvmm \ --pcie-root-complex rc0 \ --pcie-root-port rc0:rp0 \ --vfio host=0000:01:00.0,port=rp0 \ --kernel /path/to/vmlinux \ --initrd /path/to/initrd \ --cmdline "console=ttyS0" \ --com1 console \ --memory size=64GB,hugepages=on,hugepage_size=2MB \ --processors 16 ``` Use `hugepage_size=1GB` to request 1 GB pages. If `hugepage_size` is omitted, OpenVMM uses 2 MB pages. ```admonish note Hugepage-backed RAM is Linux-only. It cannot be combined with file-backed memory, private memory (`shared=off`), or legacy x86 RAM splitting such as PCAT firmware. ``` ## Troubleshooting ### "No such file or directory" for `/dev/vfio/*` Make sure you completed Step 2 (allow unsafe interrupts) and that the device is bound to `vfio-pci`. An IOMMU must be available — NoIommu mode is not supported. ### "No interrupt remapping" / ENODEV on IOMMU setup Run Step 2 to enable `allow_unsafe_interrupts`. This is needed when the platform's IOMMU does not support interrupt remapping. ### "failed to open VFIO device" / permission denied Run OpenVMM with `sudo`, or add your user to the `vfio` group and set appropriate permissions on `/dev/vfio/` devices. ### Device not visible in guest `lspci` - Verify the device is bound to `vfio-pci` (Step 3) - Verify the VFIO group exists in `/dev/vfio/` (Step 4) - Verify the `--vfio` port name matches a `--pcie-root-port` name - Check OpenVMM log output for errors during VFIO device initialization ### Hugepage allocation fails If OpenVMM fails while allocating guest RAM with `hugepages=on`, verify that the host has enough free pages in the requested hugetlb pool: ```bash grep -i Huge /proc/meminfo ``` For 1 GB pages, also check the size-specific pool: ```bash grep . /sys/kernel/mm/hugepages/hugepages-1048576kB/* ``` ## Current Limitations - **No save/restore** — VMs with VFIO devices cannot be saved or migrated. - **No hot-plug** — VFIO devices must be specified at VM creation time and cannot be added or removed while the VM is running. - **Linux only** — the `--vfio` flag is only available when OpenVMM is built and run on Linux. On Windows, use `--device` with WHP for device assignment. --- ### Guide Src User Guide Openvmm Vm Configurations (`Guide/src/user_guide/openvmm/vm_configurations.md`) # VM Configurations: Gen1 vs Gen2 Equivalents If you're familiar with Hyper-V's Gen1 and Gen2 VM concepts, this page maps those to the equivalent OpenVMM CLI flags. ## Background Hyper-V defines two VM "generations" that differ in firmware, device model, and boot mechanism: | | Gen1 | Gen2 | |---|---|---| | Firmware | BIOS (PCAT) | UEFI | | Boot disk | IDE | SCSI (VMBus storvsp) | | Guest OS | Legacy and modern (older Windows, DOS, Linux with BIOS support) | Modern (Windows 10+, most Linux) | | Secure Boot | Not available | Available | OpenVMM doesn't use the "Gen1/Gen2" terminology — you select the components directly via CLI flags. ## Gen2-equivalent (UEFI boot) — the common case Most development and testing uses UEFI boot. This is the default for modern Windows and Linux guests. ```bash cargo run -- \ --uefi \ --vmbus-scsi id=scsi0 \ --disk memdiff:file:path/to/disk.vhdx,on=scsi0 \ -p 4 -m 4GB \ --gfx ``` Key flags: - `--uefi` — boot using `mu_msvm` UEFI firmware (implicitly enables Hyper-V enlightenments and VMBus, so `--hv` is not needed separately) - `--vmbus-scsi` — creates a named SCSI controller - `--disk ...,on=scsi0` — attaches a disk to that controller ## Gen1-equivalent (PCAT BIOS boot) Use PCAT for operating systems that support BIOS boot. This includes legacy systems (DOS, older Windows) as well as modern OSes that still support BIOS boot (most Linux distributions, Windows 10+). ```bash cargo run -- \ --pcat \ --ide memdiff:file:path/to/disk.vhd \ --gfx ``` Key flags: - `--pcat` — boot using the Microsoft Hyper-V PCAT BIOS - `--ide` — expose a disk via emulated IDE controller (the traditional Gen1 storage path, no `--hv` required) See the [PCAT BIOS reference](../../reference/devices/firmware/pcat_bios.md) for more details on PCAT boot, including floppy and optical boot order. ## With OpenHCL (VTL2) To run with OpenHCL, add `--hv --vtl2` and `--igvm firmware=path/to/openhcl.igvm,personality=openhcl`. You don't need to separately specify `--uefi` or `--pcat` — the IGVM file contains the OpenHCL paravisor, and most IGVM builds bundle the [mu_msvm UEFI firmware](../../reference/devices/firmware/mu_msvm_uefi.md) for VTL0 guest boot. The build recipe controls whether UEFI is included (see [IGVM architecture](../../reference/architecture/openhcl/igvm.md)). Note: `--vtl2` requires `--hv` to be passed explicitly on the command line, even though other flags like `--uefi` imply it internally. ```bash cargo run -- \ --hv --vtl2 \ --igvm firmware=path/to/openhcl.igvm,personality=openhcl \ --vmbus-scsi id=scsi0 \ --disk memdiff:file:path/to/disk.vhdx,on=scsi0 \ -p 4 -m 4GB ``` ```admonish note OpenVMM does not currently support OpenHCL with PCAT (Gen1-style) boot. OpenHCL + PCAT is supported on Hyper-V, where the host provides the PCAT firmware. If you need Gen1-style boot with OpenHCL, use [Hyper-V](../openhcl/run/hyperv.md) rather than openvmm standalone. ``` See [Running OpenHCL with OpenVMM](../openhcl/run/openvmm.md) for full setup instructions. ## Quick reference | Scenario | Flags | Notes | |----------|-------|-------| | Modern Windows/Linux guest | `--uefi --vmbus-scsi id=scsi0 --disk memdiff:file:disk.vhdx,on=scsi0` | Most common | | With graphical console | add `--gfx` | VNC-based, see [Graphical Console](../../reference/openvmm/graphical_console.md) | | With networking | add `--nic` | Consomme user-mode NAT | | With OpenHCL | `--hv --vtl2 --igvm firmware=path/to/openhcl.igvm,personality=openhcl --vmbus-scsi id=scsi0 --disk memdiff:file:disk.vhdx,on=scsi0` | IGVM carries the paravisor; no `--uefi`/`--pcat` needed | | Legacy OS (DOS, old Windows) | `--pcat --ide memdiff:file:disk.vhd --gfx` | IDE storage, BIOS boot | | Linux direct boot (no firmware) | `--kernel vmlinux --initrd initrd` | Skips UEFI/PCAT entirely | --- ### Guide Src User Guide Openvmm Vm Memory Dumps (`Guide/src/user_guide/openvmm/vm_memory_dumps.md`) # VM Memory Dumps OpenVMM can dump a running VM's processor state and memory to a `.vmrs` file compatible with WinDbg. This is useful for offline debugging of guest crashes without needing a live debugger attached. ## Overview The `dump-state` command captures: - **VP registers** — general-purpose, control, segment, table, debug, and FP/XSAVE registers for every virtual processor - **Guest RAM** — the full contents of guest physical memory, streamed in 1 MiB blocks The output is a `.vmrs` file that can be opened in WinDbg via the `VmSavedStateDumpProvider.dll` from the Windows SDK. ## Usage While the VM is running, open the interactive console and run: ```text dump-state path/to/dump.vmrs ``` The shorter alias `dump` also works: ```text dump path/to/dump.vmrs ``` OpenVMM pauses the VM, collects state, streams guest memory to disk, and then resumes the VM. If the VM was already paused, it remains paused after the dump completes. The file is written atomically — a temporary file is created first and renamed into place on success, so readers never see a partially-written dump. ## Dumping automatically on a guest crash To capture a dump automatically whenever the guest triple-faults, pass `--crash-dump-path` on the command line: ```text openvmm --crash-dump-path path/to/crash.vmrs ... ``` When the guest crashes, OpenVMM writes the `.vmrs` dump to that path *before* applying the configured `--guest-crash-action` (halt, reset, or exit). If the dump fails, the error is logged and the crash action still runs. Each crash overwrites the file at the given path. ## Opening in WinDbg 1. Install the [Windows SDK](https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/) (the "Debugging Tools for Windows" component includes `VmSavedStateDumpProvider.dll`). 2. Open the `.vmrs` file in WinDbg: ```text windbg -z path/to/dump.vmrs ``` WinDbg will load the saved processor state and allow you to inspect registers, stack traces, and memory contents as if attached to a live VM. ```admonish note Unlike snapshots, dump files do **not** capture device state and cannot be used to restore a VM. They are strictly for offline debugging. ``` --- ### Guide Src Dev Guide Dev Tools Flowey Artifacts (`Guide/src/dev_guide/dev_tools/flowey/artifacts.md`) # Artifacts Artifacts enable typed data transfer between jobs with automatic dependency management, abstracting away CI system complexities like name collisions and manual job ordering. ## Typed vs Untyped Artifacts **Typed artifacts (recommended)** provide type-safe artifact handling by defining a custom type that implements the `Artifact` trait: ```rust,ignore #[derive(Serialize, Deserialize)] struct MyArtifact { #[serde(rename = "output.bin")] binary: PathBuf, #[serde(rename = "metadata.json")] metadata: PathBuf, } impl Artifact for MyArtifact {} let (pub_artifact, use_artifact) = pipeline.new_typed_artifact("my-files"); ``` **Untyped artifacts** provide simple directory-based artifacts for simpler cases: ```rust,ignore let (pub_artifact, use_artifact) = pipeline.new_artifact("my-files"); ``` For detailed examples of defining and using artifacts, see the [Artifact trait documentation](https://openvmm.dev/rustdoc/linux/flowey_core/pipeline/trait.Artifact.html). Both `pipeline.new_typed_artifact("name")` and `pipeline.new_artifact("name")` return a tuple of handles: `(pub_artifact, use_artifact)`. When defining a job you convert them with the job context: ```rust,ignore // In a producing job: let artifact_out = ctx.publish_artifact(pub_artifact); // artifact_out : WriteVar (typed) // or WriteVar for untyped // In a consuming job: let artifact_in = ctx.use_artifact(use_artifact); // artifact_in : ReadVar (typed) // or ReadVar for untyped ``` After conversion, you treat the returned `WriteVar` / `ReadVar` like any other flowey variable (claim them in steps, write/read values). Key concepts: - The `Artifact` trait works by serializing your type to JSON in a format that reflects a directory structure - Use `#[serde(rename = "file.exe")]` to specify exact file names - Typed artifacts ensure compile-time type safety when passing data between jobs - Untyped artifacts are simpler but don't provide type guarantees - Tuple handles must be lifted with `ctx.publish_artifact(...)` / `ctx.use_artifact(...)` to become flowey variables ## How Flowey Manages Artifacts Under the Hood During the **pipeline resolution phase** (build-time), flowey: 1. **Identifies artifact producers and consumers** by analyzing which jobs write to vs read from each artifact's `WriteVar`/`ReadVar` 2. **Constructs the job dependency graph** ensuring producers run before consumers 3. **Generates backend-specific upload/download steps** in the appropriate places: - For ADO: Uses `PublishPipelineArtifact` and `DownloadPipelineArtifact` tasks - For GitHub Actions: Uses `actions/upload-artifact` and `actions/download-artifact` - For local execution: Uses filesystem copying At **runtime**, the artifact `ReadVar` and `WriteVar` work just like any other flowey variable: - Producing jobs write artifact files to the path from `WriteVar` - Flowey automatically uploads those files as an artifact - Consuming jobs read the path from `ReadVar` where flowey has downloaded the artifact --- ### Guide Src Dev Guide Dev Tools Flowey Flowey Fundamentals (`Guide/src/dev_guide/dev_tools/flowey/flowey_fundamentals.md`) # Flowey Fundamentals Before diving into how flowey works, let's establish the key building blocks that form the foundation of flowey's automation model. These concepts are flowey's Rust-based abstractions for common CI/CD workflow primitives. ## The Automation Workflow Model In traditional CI/CD systems, workflows are defined using YAML with implicit dependencies and global state. Flowey takes a fundamentally different approach: **automation workflows are modeled as a directed acyclic graph (DAG) of typed, composable Rust components**. Each component has explicit inputs and outputs, and dependencies are tracked through the type system. ### Core Building Blocks Flowey's model consists of a hierarchy of components: **[Pipelines](https://openvmm.dev/rustdoc/linux/flowey_core/pipeline/trait.IntoPipeline.html)** are the top-level construct that defines a complete automation workflow. A pipeline specifies what work needs to be done and how it should be organized. Pipelines can target different execution backends (local machine, Azure DevOps, GitHub Actions) and generate appropriate configuration for each. **[Jobs](https://openvmm.dev/rustdoc/linux/flowey_core/pipeline/struct.PipelineJob.html)** represent units of work that run on a specific platform (Windows, Linux, macOS) and architecture (x86_64, Aarch64). Jobs can run in parallel when they don't depend on each other, or sequentially when one job's output is needed by another. Each job is isolated and runs in its own environment. **[Nodes](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.FlowNode.html)** are reusable units of automation logic that perform specific tasks (e.g., "install Rust toolchain", "run cargo build", "publish test results"). Nodes are invoked by jobs and emit one or more steps to accomplish their purpose. Nodes can depend on other nodes, forming a composable ecosystem of automation building blocks. **Steps** are the individual units of work that execute at runtime. A step might run a shell command, execute Rust code, or interact with the CI backend. Steps are emitted by nodes during the build-time phase and executed in dependency order during runtime. ### Connecting the Pieces These building blocks are connected through three key mechanisms: **[Variables (`ReadVar`/`WriteVar`)](https://openvmm.dev/rustdoc/linux/flowey/node/prelude/struct.ReadVar.html)** enable data flow between steps. A `WriteVar` represents a promise to produce a value of type `T` at runtime, while a `ReadVar` represents a dependency on that value. Variables enforce write-once semantics (each value has exactly one producer) and create explicit dependencies in the DAG. For example, a "build" step might write a binary path to a `WriteVar`, and a "test" step would read from the corresponding `ReadVar`. This echoes Rust's "shared XOR mutable" ownership rule: a value has either one writer or multiple readers, never both concurrently. **[Artifacts](https://openvmm.dev/rustdoc/linux/flowey_core/pipeline/trait.Artifact.html)** enable data transfer between jobs. Since jobs may run on different machines or at different times, artifacts package up files (like compiled binaries, test results, or build outputs) for transfer. Flowey automatically handles uploading artifacts at the end of producing jobs and downloading them at the start of consuming jobs, abstracting away backend-specific artifact APIs. **[Side Effects](https://openvmm.dev/rustdoc/linux/flowey/node/prelude/type.SideEffect.html)** represent dependencies without data. Sometimes step B needs to run after step A, but A doesn't produce any data that B consumes (e.g., "install dependencies" must happen before "run tests", even though the test step doesn't directly use the installation output). Side effects are represented as `ReadVar` and establish ordering constraints in the DAG without transferring actual values. ### Putting It Together Here's an example of how these pieces relate: ```txt Pipeline ├─ Job 1 (Linux x86_64) │ ├─ Node A (install Rust) │ │ └─ Step: Run rustup install │ │ └─ Produces: WriteVar (installation complete) │ └─ Node B (build project) │ └─ Step: Run cargo build │ └─ Consumes: ReadVar (installation complete) │ └─ Produces: WriteVar (binary path) → Artifact │ └─ Job 2 (Windows x86_64) └─ Node C (run tests) └─ Step: Run binary with test inputs └─ Consumes: ReadVar (binary path) ← Artifact └─ Produces: WriteVar (test results) ``` In this example: - The **Pipeline** defines two jobs that run on different platforms - **Job 1** installs Rust and builds the project, with step dependencies expressed through variables - **Job 2** runs tests using the binary from Job 1, with the binary transferred via an artifact - **Variables** create dependencies within a job (build depends on install) - **Artifacts** create dependencies between jobs (Job 2 depends on Job 1's output) - **Side Effects** represent the "Rust is installed" state without carrying data ## Two-Phase Execution Model Flowey operates in two distinct phases: 1. **Build-Time (Resolution Phase)**: When you run `cargo xflowey regen`, flowey: - Reads `.flowey.toml` to determine which pipelines to regenerate - Builds the flowey binary (e.g., `flowey-hvlite`) via `cargo build` - Runs the flowey binary with `pipeline --out ` for each pipeline definition - During this invocation, flowey constructs a **directed acyclic graph (DAG)** by: - Instantiating all nodes (reusable units of automation logic) defined in the pipeline - Processing their requests - Resolving dependencies between nodes via variables and artifacts - Determining the execution order - Performing flowey-specific validations (dependency resolution, type checking, etc.) - Generates YAML files for CI systems (ADO, GitHub Actions) at the paths specified in `.flowey.toml` 2. **Runtime (Execution Phase)**: The generated YAML is executed by the CI system (or locally via `cargo xflowey `). Steps (units of work) run in the order determined at build-time: - Variables are read and written with actual values - Commands are executed - Artifacts (data packages passed between jobs) are published/consumed - Side effects (dependencies) are resolved The `.flowey.toml` file at the repo root defines which pipelines to generate and where. For example: ```toml [[pipeline.flowey_hvlite.github]] file = ".github/workflows/openvmm-pr.yaml" cmd = ["ci", "checkin-gates", "--config=pr"] ``` When you run `cargo xflowey regen`: 1. It reads `.flowey.toml` 2. Builds the `flowey-hvlite` binary 3. Runs `flowey-hvlite pipeline github --out .github/workflows/openvmm-pr.yaml ci checkin-gates --config=pr` 4. This generates/updates the YAML file with the resolved pipeline **Key Distinction:** - `cargo build -p flowey-hvlite` - Only compiles the flowey code to verify it builds successfully. **Does not** construct the DAG or generate YAML files. - `cargo xflowey regen` - Compiles the code **and** runs the full build-time resolution to construct the DAG, validate the pipeline, and regenerate all YAML files defined in `.flowey.toml`. Always run `cargo xflowey regen` after modifying pipeline definitions to ensure the generated YAML files reflect your changes. ### Backend Abstraction Flowey supports multiple execution backends: - **Local**: Runs directly on your development machine - **ADO (Azure DevOps)**: Generates ADO Pipeline YAML - **GitHub Actions**: Generates GitHub Actions workflow YAML ```admonish warning Nodes should be written to work across ALL backends whenever possible. Relying on `ctx.backend()` to query the backend or manually emitting backend-specific steps (via `emit_ado_step` or `emit_gh_step`) should be avoided unless absolutely necessary. Most automation logic should be backend-agnostic, using `emit_rust_step` for cross-platform Rust code that works everywhere. Writing cross-platform flowey code enables locally testing pipelines which can be invaluable when iterating over CI changes. ``` If a node only supports certain backends, it should immediately fast‑fail with a clear error ("`` not supported on ``") instead of silently proceeding. That failure signals it's time either to add the missing backend support or introduce a multi‑platform abstraction/meta‑node that delegates to platform‑specific nodes. --- ### Guide Src Dev Guide Dev Tools Flowey Nix (`Guide/src/dev_guide/dev_tools/flowey/nix.md`) # Nix In order to enable reproducible builds, experimental flowey features are being built to utilize the [Nix package manager](https://nixos.org/) for dependency management and environment isolation. The root of the nix configuration lives in `shell.nix`. If you're unsure where the Nix definition is for a dependency, you should be able to track it down from there. ## Updating Nix Packages Nix dependencies require a hash of their contents to ensure integrity and reproducibility. When updating a dependency, you'll need to update the release that's being pulled and its corresponding hash. For instance, let's say we have a new release of the OpenHCL Kernel and we want to update it in our Nix configuration: 1. Go to the corresponding `.nix` file (in this case, `openhcl_kernel.nix`) 2. Clear the hash to an empty string 3. Update the version 4. Run `nix-shell --pure` and use the printed error to get the new hash > **Warning:** Because Nix caches dependencies based on the hash, if you don't clear the hash to an empty string before updating the version, `nix-shell --pure` will run without error even though the dependency hasn't actually been updated. Here's an example of what the error will look like when done correctly: ```bash error: hash mismatch in fixed-output derivation '/nix/store/cc7hhyslx1dnw01nmjx11zqim2l50awp-source.drv': specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= got: sha256-wUDWFazJM80oztKqpuRwj8Wvto2Uo/OuVGvhpszIw+A= error: Cannot build '/nix/store/spg5vbm6mzmsxpg5v2ibg97qrz8khc70-openhcl-kernel-x64-6.12.52.4.drv'. Reason: 1 dependency failed. Output paths: /nix/store/kv8s7pld70yvzxzd77swz9hb3pygkrhl-openhcl-kernel-x64-6.12.52.4 ``` Given this error, you would update the corresponding hash to `sha256-An1N76i1MPb+rrQ1nBpoiuxnNeD0E+VuwqXdkPzaZn0=` in the `openhcl_kernel.nix` file. ### Updating `mu_msvm` UEFI (`nix/uefi_mu_msvm.nix`) When bumping `mu_msvm`, update both the Nix fetch and the Flowey version constant in the same PR. 1. Update `nix/uefi_mu_msvm.nix`: - Set `version` to the new release. - Ensure the `fetchzip` URL uses that version. - Set both architecture hashes to empty strings. 2. Update `flowey/flowey_lib_hvlite/src/_jobs/cfg_versions.rs`: - Bump `MU_MSVM` to the same version. 3. From the repo root, run `sudo nix-shell` (or `nix-shell` if sudo is not required in your environment). 4. Copy the first `got:` hash from the mismatch error into the matching entry in `nix/uefi_mu_msvm.nix`. 5. Run `sudo nix-shell` again to fetch the other architecture and copy its `got:` hash. 6. Re-run `sudo nix-shell` a final time to verify both hashes are correct. Expected mismatch output: ```bash error: hash mismatch in fixed-output derivation '/nix/store/...-source.drv': specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= got: sha256- ``` Use the `got:` value in the `.nix` file. ### OpenHCL dev kernel packages The Nix shell does not evaluate or export OpenHCL dev kernel packages by default. The main and dev kernels often contain the same promoted changes, so building both variants adds unnecessary download and build work. To restore the dev kernel packages when the branches diverge: 1. Set `enableDevKernel` to `true` in `shell.nix`. 2. Update the dev version in `nix/openhcl_kernel.nix`. 3. Clear and replace the dev hashes using the procedure above. This restores the `NIX_KERNEL_*_DEV` paths and `CARGO_BUILD_ARGS_*_DEVKERN` arguments in the Nix shell. ## Updating the Rust Version The Nix shell derives its Rust toolchain version from `rust-version` in the root `Cargo.toml` and resolves it against a pinned [rust-overlay](https://github.com/oxalica/rust-overlay) commit in `shell.nix`. When `rust-version` is bumped in `Cargo.toml`, the pinned rust-overlay may not yet include the new version, causing an error like: ```bash error: No rust version matching 1.XX.* found in rust-overlay ``` To fix this, update the rust-overlay pin in `shell.nix` to a commit that includes the new Rust version: 1. Go to the [rust-overlay stable branch commits](https://github.com/oxalica/rust-overlay/commits/stable) and copy the latest commit SHA 2. In `shell.nix`, find the `rust_overlay` `fetchTarball` block and replace the commit SHA in the `url` with the new one (the long hex string in the URL path, e.g., `https://github.com/oxalica/rust-overlay/archive/.tar.gz`) 3. Clear the `sha256` field to an empty string — this is a _content_ hash of the tarball (not the commit SHA), and Nix will compute the correct value for you 4. Run `nix-shell --pure` and use the printed error to get the new `sha256` 5. Update the `sha256` with the correct hash from the error --- ### Guide Src Dev Guide Dev Tools Flowey Nodes (`Guide/src/dev_guide/dev_tools/flowey/nodes.md`) # Nodes At a conceptual level, a Flowey node is analogous to a strongly typed function: you "invoke" it by submitting one or more Request values (its parameters), and it responds by emitting steps that perform work and produce outputs (values written to `WriteVar`s, published artifacts, or side-effect dependencies). ## The Node/Request Pattern Every node has an associated **Request** type that defines what operations the node can perform. Requests are defined using the [`flowey_request!`](https://openvmm.dev/rustdoc/linux/flowey_core/macro.flowey_request.html) macro and registered with [`new_flow_node!`](https://openvmm.dev/rustdoc/linux/flowey_core/macro.new_flow_node.html) or [`new_simple_flow_node!`](https://openvmm.dev/rustdoc/linux/flowey_core/macro.new_simple_flow_node.html) macros. For complete examples, see the [`FlowNode` trait documentation](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.FlowNode.html). ## FlowNode vs SimpleFlowNode Flowey provides two node implementation patterns with a fundamental difference in their Request structure and complexity: [**`SimpleFlowNode`**](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.SimpleFlowNode.html) - for straightforward, function-like operations: - Uses a **single struct Request** type - Processes one request at a time independently - Behaves like a "plain old function" that resolves its single request type - Each invocation is isolated - no shared state or coordination between requests - Simpler implementation with less boilerplate - Ideal for straightforward operations like running a command or transforming data **Example use case**: A node that runs `cargo build` - each request is independent and just needs to know what to build. [**`FlowNode`**](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.FlowNode.html) - for complex nodes requiring coordination and non-local configuration: - Often uses an **enum Request** with multiple variants - Receives all requests as a `Vec` and processes them together - Can aggregate, optimize, and consolidate multiple requests into fewer steps - Enables **non-local configuration** - critical for simplifying complex pipelines ### The Non-Local Configuration Pattern The key advantage of `FlowNode` is its ability to accept configuration from different parts of the node graph without forcing intermediate nodes to be aware of that configuration. This is the "non-local" aspect. For nodes that accept **config values** — versions, feature flags, local paths — use `FlowNodeWithConfig` with a typed config struct. Config is declared with the `flowey_config!` macro, set via `ctx.config(...)` by any caller, and automatically merged across all callers before being delivered to `emit()` separately from action requests. Consider an "install Rust toolchain" node: ```rust,ignore flowey_config! { pub struct Config { pub version: Option, pub auto_install: Option, } } flowey_request! { pub enum Request { GetToolchain(WriteVar), } } new_flow_node_with_config!(struct Node); impl FlowNodeWithConfig for Node { type Request = Request; type Config = Config; fn imports(ctx: &mut ImportCtx<'_>) { /* ... */ } fn emit( config: Config, requests: Vec, ctx: &mut NodeCtx<'_>, ) -> anyhow::Result<()> { let version = config.version .expect("version must be set by cfg_versions"); // ... use version to install, then fulfill GetToolchain requests Ok(()) } } ``` Callers set config and submit requests independently: ```rust,ignore // A top-level job configuration node sets the version once: ctx.config(install_rust::Config { version: Some("1.75".into()), ..Default::default() }); // Any node that needs the Rust toolchain just requests it: let toolchain = ctx.reqv(|v| install_rust::Request::GetToolchain(v)); ``` ```txt Root Node → config(version: "1.75") → Node A → Node B → Node C → req(GetToolchain) ``` Flowey merges all config partials automatically. If two callers set the same field, the values must agree or the build fails. The intermediate nodes (A, B, C) never need to know about or pass through the version. This pattern: - **Eliminates plumbing complexity** in large pipelines - **Allows global configuration** to be set once at the top level - **Keeps unrelated nodes decoupled** from configuration they don't need - **Validates consistency** — conflicting config values are caught at build time - **Separates concerns** — config (what version?) is distinct from requests (give me the toolchain path) **Additional Benefits of FlowNode:** - Optimize and consolidate multiple similar requests into fewer steps (e.g., installing a tool once for many consumers) - Resolve conflicts or enforce consistency across requests For detailed comparisons and examples, see the [`FlowNode`](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.FlowNode.html) and [`SimpleFlowNode`](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.SimpleFlowNode.html) documentation. ## Node Registration Nodes are automatically registered using macros that handle most of the boilerplate: - [`new_flow_node!(struct Node)`](https://openvmm.dev/rustdoc/linux/flowey_core/macro.new_flow_node.html) - registers a FlowNode - [`new_flow_node_with_config!(struct Node)`](https://openvmm.dev/rustdoc/linux/flowey_core/macro.new_flow_node_with_config.html) - registers a FlowNodeWithConfig (a FlowNode that also accepts typed config) - [`new_simple_flow_node!(struct Node)`](https://openvmm.dev/rustdoc/linux/flowey_core/macro.new_simple_flow_node.html) - registers a SimpleFlowNode - [`flowey_request!`](https://openvmm.dev/rustdoc/linux/flowey_core/macro.flowey_request.html) - defines the Request type and implements [`IntoRequest`](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.IntoRequest.html) - [`flowey_config!`](https://openvmm.dev/rustdoc/linux/flowey_core/macro.flowey_config.html) - defines a Config struct with automatic merge support and implements [`IntoConfig`](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.IntoConfig.html) ## The imports() Method The `imports()` method declares which other nodes this node might depend on. This enables flowey to: - Validate that all dependencies are available - Build the complete dependency graph - Catch missing dependencies at build-time ```admonish warning Flowey does not catch unused imports today as part of its build-time validation step. ``` **Why declare imports?** Flowey needs to know the full set of potentially-used nodes at compilation time to properly resolve the dependency graph. For more on node imports, see the [`FlowNode::imports` documentation](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.FlowNode.html#tymethod.imports). ## The emit() Method The [`emit()`](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.FlowNode.html#tymethod.emit) method is where a node's actual logic lives. For [`FlowNode`](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.FlowNode.html), it receives all requests together and must: 1. Aggregate and validate requests (ensuring consistency where needed) 2. Emit steps to perform the work 3. Wire up dependencies between steps via variables For [`SimpleFlowNode`](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.SimpleFlowNode.html), the equivalent [`process_request()`](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.SimpleFlowNode.html#tymethod.process_request) method processes one request at a time. For complete implementation examples, see the [`FlowNode::emit` documentation](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.FlowNode.html#tymethod.emit). ## Node Design Philosophy Flowey nodes are designed around several key principles: ### 1. Composability Nodes should be reusable building blocks that can be combined to build complex workflows. Each node should have a single, well-defined responsibility. ❌ **Bad**: A node that "builds and tests the project" ✅ **Good**: Separate nodes for "build project" and "run tests" ### 2. Explicit Dependencies Dependencies between steps should be explicit through variables, not implicit through side effects. ❌ **Bad**: Assuming a tool is already installed ✅ **Good**: Taking a `ReadVar` that proves installation happened ### 3. Backend Abstraction Nodes should work across all backends when possible. Backend-specific behavior should be isolated and documented. ### 4. Separation of Concerns Keep node definition (request types, dependencies) separate from step implementation (runtime logic): - **Node definition**: What the node does, what it depends on - **Step implementation**: How it does it ## Common Patterns ### Node Config vs Request When designing a `FlowNode`, separate **config** (values that must be consistent across all callers) from **requests** (actions the node performs): - **Config** (`flowey_config!`): version strings, feature flags, local override paths, auto-install toggles. Use `FlowNodeWithConfig` and `ctx.config(...)`. Config fields are `Option` or `BTreeMap` — flowey merges them automatically and errors on conflicts. - **Requests** (`flowey_request!`): actions that produce outputs, e.g. `GetBinary(WriteVar)`. Multiple callers can each submit requests, and the node fulfills all of them. ```admonish tip If a value needs `same_across_all_reqs` validation, it should probably be a config field instead. The `flowey_config!` macro provides this validation automatically during merge. ``` ### Request Aggregation and Validation When a FlowNode receives multiple requests, it may need to aggregate or validate them. Common techniques: - Iterate through all requests and separate them by variant - Collect output variables that can have multiple instances - Validate that required values were provided For values that must be consistent across all callers (versions, flags), prefer using `flowey_config!` with `FlowNodeWithConfig` instead of manual validation — see [The Non-Local Configuration Pattern](#the-non-local-configuration-pattern) above. ### Conditional Execution Based on Backend/Platform Nodes can query the current backend and platform to emit platform-specific or backend-specific steps. This allows nodes to adapt their behavior based on the execution environment. **Key concepts:** - Use `ctx.backend()` to check if running locally, on ADO, or on GitHub Actions - Use `ctx.platform()` to check the operating system (Windows, Linux, macOS) - Use `ctx.arch()` to check the architecture (x86_64, Aarch64) - Emit different steps or use different tool configurations based on these values **When to use:** - Installing platform-specific tools or dependencies - Using different commands on Windows vs Unix systems - Optimizing for local development vs CI environments For more on backend and platform APIs, see the [`NodeCtx` documentation](https://openvmm.dev/rustdoc/linux/flowey_core/node/struct.NodeCtx.html). --- ### Guide Src Dev Guide Dev Tools Flowey Pipelines (`Guide/src/dev_guide/dev_tools/flowey/pipelines.md`) # Pipelines Pipelines define complete automation workflows consisting of jobs that run nodes. See the [IntoPipeline trait documentation](https://openvmm.dev/rustdoc/linux/flowey_core/pipeline/trait.IntoPipeline.html) for detailed examples. ## Pipeline Jobs [`PipelineJob`](https://openvmm.dev/rustdoc/linux/flowey_core/pipeline/struct.PipelineJob.html) instances are configured using a builder pattern: ```rust,ignore let job = pipeline .new_job(platform, arch, "my-job") .with_timeout_in_minutes(60) .with_condition(some_param) .ado_set_pool("my-pool") .gh_set_pool(GhRunner::UbuntuLatest) .dep_on(|ctx| { // Define what nodes this job depends on some_node::Request { /* ... */ } }) .finish(); ``` ### Pipeline Parameters Parameters allow runtime configuration of pipelines. In Azure DevOps, parameters appear as editable fields in the Run pipeline UI (name, description, default). ```rust,ignore // Define a boolean parameter let verbose = pipeline.new_parameter_bool( "verbose", "Run with verbose output", ParameterKind::Stable, Some(false) // default value ); // Use the parameter in a job let job = pipeline.new_job(...) .dep_on(|ctx| { let verbose = ctx.use_parameter(verbose); // verbose is now a ReadVar }) .finish(); ``` #### Stable vs Unstable Parameters Every parameter in flowey must be declared as either **Stable** or **Unstable** using [`ParameterKind`](https://openvmm.dev/rustdoc/linux/flowey_core/pipeline/enum.ParameterKind.html). This classification determines the parameter's visibility and API stability: **Stable Parameters ([`ParameterKind::Stable`](https://openvmm.dev/rustdoc/linux/flowey_core/pipeline/enum.ParameterKind.html#variant.Stable))** Stable parameters represent a **public, stable API** for the pipeline: - **External Visibility**: The parameter name is exposed as-is in the generated CI YAML, making it callable by external pipelines and users. - **API Contract**: Once a parameter is marked stable, its name and behavior should be maintained for backward compatibility. Removing or renaming a stable parameter is a breaking change. - **Use Cases**: - Parameters that control major pipeline behavior (e.g., `enable_tests`, `build_configuration`) - Parameters intended for use by other teams or external automation - Parameters documented as part of the pipeline's public interface **Unstable Parameters ([`ParameterKind::Unstable`](https://openvmm.dev/rustdoc/linux/flowey_core/pipeline/enum.ParameterKind.html#variant.Unstable))** Unstable parameters are for **internal use** and experimentation: - **Internal Only**: The parameter name is prefixed with `__unstable_` in the generated YAML (e.g., `__unstable_debug_mode`), signaling that it's not part of the stable API. - **No Stability Guarantee**: Unstable parameters can be renamed, removed, or have their behavior changed without notice. External consumers should not depend on them. - **Use Cases**: - Experimental features or debugging flags - Internal pipeline configuration that may change frequently - Parameters for development/testing that shouldn't be used in production --- ### Guide Src Dev Guide Dev Tools Flowey Steps (`Guide/src/dev_guide/dev_tools/flowey/steps.md`) # Steps **Steps** are units of work that will be executed at runtime. Different step types exist for different purposes. ## Types of Steps ### Rust Steps Rust steps execute Rust code at runtime and are the most common step type in flowey. **`emit_rust_step`**: The primary method for emitting steps that run Rust code. Steps can claim variables, read inputs, perform work, and write outputs. Returns an optional `ReadVar` that other steps can use as a dependency. **`emit_minor_rust_step`**: Similar to `emit_rust_step` but for steps that cannot fail (no `Result` return) and don't need visibility in CI logs. Used for simple transformations and glue logic. Using minor steps also improve performance, since there is a slight cost to starting and ending a 'step' in GitHub and ADO. During the build stage, minor steps that are adjacent to each other will get merged into one giant CI step. **`emit_rust_stepv`**: Convenience method that combines creating a new variable and emitting a step in one call. The step's return value is automatically written to the new variable. For detailed examples of Rust steps, see the [`NodeCtx` emit methods documentation](https://openvmm.dev/rustdoc/linux/flowey/node/prelude/struct.NodeCtx.html). ### ADO Steps **`emit_ado_step`**: Emits a step that generates Azure DevOps Pipeline YAML. Takes a closure that returns a YAML string snippet which is interpolated into the generated pipeline. For ADO step examples, see the [`NodeCtx::emit_ado_step` documentation](https://openvmm.dev/rustdoc/linux/flowey_core/node/struct.NodeCtx.html#method.emit_ado_step). ### GitHub Steps **`emit_gh_step`**: Creates a GitHub Actions step using the fluent `GhStepBuilder` API. Supports specifying the action, parameters, outputs, dependencies, and permissions. Returns a builder that must be finalized with `.finish(ctx)`. For GitHub step examples, see the [`GhStepBuilder` documentation](https://openvmm.dev/rustdoc/linux/flowey_core/node/steps/github/struct.GhStepBuilder.html). ### Side Effect Steps **`emit_side_effect_step`**: Creates a dependency relationship without executing code. Useful for aggregating multiple side effect dependencies into a single side effect. More efficient than emitting an empty Rust step. For side effect step examples, see the [`NodeCtx::emit_side_effect_step` documentation](https://openvmm.dev/rustdoc/linux/flowey_core/node/struct.NodeCtx.html#method.emit_side_effect_step). ### Isolated Working Directories and Path Immutability ```admonish warning title="Critical Constraint" **Each step gets its own fresh local working directory.** This avoids the "single global working directory dumping ground" common in bash + YAML systems. However, while flowey variables enforce sharing XOR mutability at the type-system level, **developers must manually enforce this at the filesystem level**: **Steps must NEVER modify the contents of paths referenced by `ReadVar`.** ``` When you write a path to `WriteVar`, you're creating an immutable contract. Other steps reading that path must treat it as read-only. If you need to modify files from a `ReadVar`, copy them to your step's working directory. ## Runtime Services Runtime services provide the API available during step execution (inside the closures passed to `emit_rust_step`, etc.). ### RustRuntimeServices [`RustRuntimeServices`](https://openvmm.dev/rustdoc/linux/flowey_core/node/steps/rust/struct.RustRuntimeServices.html) is the primary runtime service available in Rust steps. It provides: #### Variable Operations - Reading and writing flowey variables - Secret handling (automatic secret propagation for safety) - Support for reading values of any type that implements [`ReadVarValue`](https://openvmm.dev/rustdoc/linux/flowey_core/node/trait.ReadVarValue.html) #### Environment Queries - Backend identification (Local, ADO, or GitHub) - Platform detection (Windows, Linux, macOS) - Architecture information (x86_64, Aarch64) ### AdoStepServices [`AdoStepServices`](https://openvmm.dev/rustdoc/linux/flowey_core/node/steps/ado/struct.AdoStepServices.html) provides integration with Azure DevOps-specific features when emitting ADO YAML steps: **ADO Variable Bridge:** - Convert ADO runtime variables (like `BUILD.SOURCEBRANCH`) into flowey vars - Convert flowey vars back into ADO variables for use in YAML - Handle secret variables appropriately **Repository Resources:** - Resolve repository IDs declared as pipeline resources - Access repository information in ADO-specific steps ### GhStepBuilder [`GhStepBuilder`](https://openvmm.dev/rustdoc/linux/flowey_core/node/steps/github/struct.GhStepBuilder.html) is a fluent builder for constructing GitHub Actions steps with: **Step Configuration:** - Specifying the action to use (e.g., `actions/checkout@v4`) - Adding input parameters via `.with()` - Capturing step outputs into flowey variables - Setting conditional execution based on variables **Dependency Management:** - Declaring side-effect dependencies via `.run_after()` - Ensuring steps run in the correct order **Permissions:** - Declaring required GITHUB_TOKEN permissions - Automatic permission aggregation at the job level ## Secret Variables and CI Backend Integration Flowey provides built-in support for handling sensitive data like API keys, tokens, and credentials through **secret variables**. Secret variables are treated specially to prevent accidental exposure in logs and CI outputs. ### How Secret Handling Works When a variable is marked as secret, flowey ensures: - The value is not logged or printed in step output - CI backends (ADO, GitHub Actions) are instructed to mask the value in their logs - Secret status is automatically propagated to prevent leaks ### Automatic Secret Propagation To prevent accidental leaks, flowey uses conservative automatic secret propagation: ```admonish warning If a step reads a secret value, **all subsequent writes from that step are automatically marked as secret** by default. This prevents accidentally leaking secrets through derived values. ``` For example: ```rust,ignore ctx.emit_rust_step("process token", |ctx| { let secret_token = secret_token.claim(ctx); let output_var = output_var.claim(ctx); |rt| { let token = rt.read(secret_token); // Reading a secret // This write is AUTOMATICALLY marked as secret // (even though we're just writing "done") rt.write(output_var, &"done".to_string()); Ok(()) } }); ``` If you need to write non-secret data after reading a secret, use `write_not_secret()`: ```rust,ignore rt.write_not_secret(output_var, &"done".to_string()); ``` ### Best Practices for Secrets 1. **Never use `ReadVar::from_static()` for secrets** - static values are encoded in plain text in the generated YAML 2. **Always use `write_secret()`** when writing sensitive data like tokens, passwords, or keys 3. **Minimize secret lifetime** - read secrets as late as possible and don't pass them through more variables than necessary --- ### Guide Src Dev Guide Dev Tools Flowey Variables (`Guide/src/dev_guide/dev_tools/flowey/variables.md`) # Variables Variables are flowey's mechanism for creating typed data dependencies between steps. When a node emits steps, it uses `ReadVar` and `WriteVar` to declare what data each step consumes and produces. This creates explicit edges in the dependency graph: if step B reads from a variable that step A writes to, flowey ensures step A executes before step B. ## Claiming Variables Before a step can use a [`ReadVar`](https://openvmm.dev/rustdoc/linux/flowey/node/prelude/struct.ReadVar.html) or [`WriteVar`](https://openvmm.dev/rustdoc/linux/flowey/node/prelude/struct.WriteVar.html), it must **claim** it. Claiming serves several purposes: 1. Registers that this step depends on (or produces) this variable 2. Converts `ReadVar` to `ReadVar` 3. Allows flowey to track variable usage for graph construction Variables can only be claimed inside step closures using the `claim()` method. **Nested closure pattern and related contexts:** ```rust,ignore // Inside a SimpleFlowNode's process_request() method fn process_request(&self, request: Self::Request, ctx: &mut NodeCtx<'_>) { // Assume a single Request provided an input ReadVar and output WriteVar let input_var: ReadVar = /* from one of the requests */; let output_var: WriteVar = /* from one of the requests */; // Declare a step (still build-time). This adds a node to the DAG. ctx.emit_rust_step("compute length", |step| { // step : StepCtx (outer closure, build-time) // Claim dependencies so the graph knows: this step READS input_var, WRITES output_var. let input_var = input_var.claim(step); let output_var = output_var.claim(step); // Return the runtime closure. move |rt| { // rt : RustRuntimeServices (runtime phase) let input = rt.read(input_var); // consume value let len = input.len() as i32; rt.write(output_var, &len); // fulfill promise Ok(()) } }); } ``` **Why the nested closure dance?** The nested closure pattern is fundamental to flowey's two-phase execution model: 1. **Build-Time (Outer Closure)**: When flowey constructs the DAG, the outer closure runs to: - Claim variables, which registers dependencies in the graph - Determine what this step depends on (reads) and produces (writes) - Allow flowey determine execution order - Returns an inner closure that gets invoked during the job's runtime 2. **Runtime (Inner Closure)**: When the pipeline actually executes, the inner closure runs to: - Read actual values from claimed `ReadVar`s - Perform the real work (computations, running commands, etc.) - Write actual values to claimed `WriteVar`s - [**`NodeCtx`**](https://openvmm.dev/rustdoc/linux/flowey/node/prelude/struct.NodeCtx.html): Used when emitting steps (during the build-time phase). Provides `emit_*` methods, `new_var()`, `req()`, etc. - [**`StepCtx`**](https://openvmm.dev/rustdoc/linux/flowey/node/prelude/struct.StepCtx.html): Used inside step closures (during runtime execution). Provides access to `claim()` for variables, and basic environment info (`backend()`, `platform()`). The type system enforces this separation: `claim()` requires `StepCtx` (only available in the outer closure), while `read()`/`write()` require `RustRuntimeServices` (only available in the inner closure). ## ClaimedReadVar and ClaimedWriteVar These are type aliases for claimed variables: - [`ClaimedReadVar`](https://openvmm.dev/rustdoc/linux/flowey/node/prelude/type.ClaimedReadVar.html) = `ReadVar` - [`ClaimedWriteVar`](https://openvmm.dev/rustdoc/linux/flowey/node/prelude/type.ClaimedWriteVar.html) = `WriteVar` Only claimed variables can be read/written at runtime. ### Implementation Detail: Zero-Sized Types (ZSTs) The claim state markers [`VarClaimed`](https://openvmm.dev/rustdoc/linux/flowey/node/prelude/enum.VarClaimed.html) and [`VarNotClaimed`](https://openvmm.dev/rustdoc/linux/flowey/node/prelude/enum.VarNotClaimed.html) are zero-sized types (ZSTs) - they exist purely at the type level. It allows Rust to statically verify that all variables used in a runtime block have been claimed by that block. The type system ensures that `claim()` is the only way to convert from `VarNotClaimed` to `VarClaimed`, and this conversion can only happen within the outer closure where `StepCtx` is available. ## Static Values vs Runtime Values Sometimes you know a value at build-time: ```rust,ignore // Create a ReadVar with a static value let version = ReadVar::from_static("1.2.3".to_string()); // This is encoded directly in the pipeline, not computed at runtime // WARNING: Never use this for secrets! ``` This can be used as an escape hatch when you have a Request (that expects a value to be determined at runtime), but in a given instance you know the value at build-time. ## Variable Operations `ReadVar` provides operations for transforming and combining variables: - **`map()`**: Transform a `ReadVar` into a `ReadVar` - **`zip()`**: Combine two ReadVars into `ReadVar<(T, U)>` - **`into_side_effect()`**: Convert `ReadVar` to `ReadVar` when you only care about ordering, not the value - **`depending_on()`**: Create a new ReadVar with an explicit dependency For detailed examples, see the [`ReadVar` documentation](https://openvmm.dev/rustdoc/linux/flowey/node/prelude/struct.ReadVar.html). --- ### Guide Src Dev Guide Tests Fuzzing Running (`Guide/src/dev_guide/tests/fuzzing/running.md`) # Running Fuzzers Locally ## Installing Dependencies To begin fuzzing in OpenVMM, you'll need to install `cargo-fuzz`. Installation should be as simple as: ```bash cargo install cargo-fuzz ``` For debugging crashes, a debugger such as `lldb` is useful: ```bash sudo apt-get install -y lldb ``` For coverage reports, install `llvm-tools` and `lcov`: ```bash rustup component add llvm-tools sudo apt-get install -y lcov ``` ```admonish info `cargo-fuzz` requires a nightly toolchain as it compiles targets with [ASAN](https://learn.microsoft.com/en-us/cpp/sanitizers/asan?view=msvc-170) to improve the likelihood of finding bugs and the reproducibility of testcases. ``` ```admonish warning Fuzzing only works on **Linux**. libfuzzer-sys doesn't support Windows. On **aarch64**, you must set `RUSTFLAGS="-Ctarget-feature=+lse,+neon"` before any cargo-fuzz command (build, run, coverage), or builds will fail with atomics errors. This is not needed on x86_64. ``` ## Running While it's entirely possible to run the various fuzzers in the OpenVMM repo using `cargo fuzz` directly, the OpenVMM repo includes additional tooling to streamline working with fuzzers at "OpenVMM scale": `cargo xtask fuzz` `cargo xtask fuzz` bridges the gap between `cargo fuzz`'s "crate-oriented" tooling, and OpenVMM's "repo-oriented" tooling. e.g: instead of manually navigating to each individual `crate/fuzz` directory in order to use `cargo fuzz`, with `cargo xtask fuzz`, you can list/run/build _any_ fuzzer in the OpenVMM repo, regardless where it happens to be in the repo! Before you can run a fuzzer, you need to know its name. To see a list of all fuzzers currently in the OpenVMM tree, you can run: ```bash cargo xtask fuzz list ``` The output will be a list of available "fuzz targets": ```bash $ cargo xtask fuzz list fuzz_chipset_battery fuzz_ide fuzz_scsi_buffers ``` Once you've got a fuzzer you're interested in running (e.g: `fuzz_ide`), starting a fuzzing session is as easy as running: ```bash cargo xtask fuzz run fuzz_ide ``` And you're off! If you see a whole bunch of terminal spew, congrats, you're fuzzing! When run locally using the above command, the fuzzer will run indefinitely until a crash is discovered. ### Building without running To just build a fuzzer without starting it: ```bash cargo xtask fuzz build fuzz_ide ``` The binary lands at `target//release/fuzz_ide`. ### Reproducing a crash When LibFuzzer finds a crash, it saves the input as an artifact. Reproduce it: ```bash # Through xtask (sets XTASK_FUZZ_REPRO=1 automatically for tracing) cargo xtask fuzz run fuzz_ide path/to/crash-artifact # Or run the binary directly (faster for iteration) ./target//release/fuzz_ide path/to/crash-artifact # With tracing enabled (verbose — shows device state at each poll) XTASK_FUZZ_REPRO=1 ./target//release/fuzz_ide path/to/crash-artifact ``` ### Minimizing crash inputs ```bash cargo xtask fuzz tmin fuzz_ide path/to/crash-artifact ``` ### Corpus management Corpus files live in `/fuzz/corpus//`. Crash artifacts land in `/fuzz/artifacts//`. To minimize the corpus (remove redundant inputs): ```bash cargo xtask fuzz cmin fuzz_ide ``` ### Parallel fuzzing Use `-fork=N` to run N fuzzer processes in parallel across CPUs: ```bash cargo xtask fuzz run fuzz_ide -- -- -fork=20 -max_total_time=3600 -print_final_stats=1 ``` To survive crashes/timeouts/OOMs during long campaigns: ```bash cargo xtask fuzz run fuzz_ide -- -- \ -fork=20 -max_total_time=21600 \ -ignore_crashes=1 -ignore_timeouts=1 -ignore_ooms=1 \ -print_final_stats=1 ``` If you need to tweak the runtime behavior of the command, all of libFuzzer's [commandline options][cli-opts] are at your disposal. Alternatively you can print the help of the fuzzer like so: ```bash # NOTE: The "-- --" is required to differentiate between `xtask fuzz`'s # extra-args, and `cargo fuzz`'s extra-args cargo xtask fuzz run fuzz_ide -- -- -help=1 ``` [cli-opts]: https://www.llvm.org/docs/LibFuzzer.html#options ## Other Fuzzing Commands The `cargo xtask fuzz` CLI includes plenty of docs via `--help` text. Don't be afraid to dig into all the tools available via `cargo xtask fuzz` by using `--help` at both the top-level, and for more details regarding the various subcommands. Note that most `cargo xtask fuzz` commands mirror those from `cargo fuzz`, so for additional information on how certain commands work, check out the [cargo-fuzz book](https://rust-fuzz.github.io/book/cargo-fuzz.html). ## Coverage The effectiveness of fuzzing can be measured with code coverage. Code coverage can be analyzed to determine which branches in the target were exercised and which were missed by the fuzzer. This can be used to determine if the fuzzer needs improvements or is doing an adequate job. Before you begin you'll need some additional dependencies to generate an html report: ```bash rustup component add llvm-tools apt install lcov ``` To generate a report with "sane defaults", you can simply run: ```bash cargo xtask fuzz coverage fuzz_ide --with-html-report ``` Simply navigate to the `html/report/dir/index.html` on your machine and inspect the coverage! ```admonish tip The overall coverage percentage in the HTML report covers *all* compiled code including dependencies — it's typically 4–6% and meaningless. Focus on coverage of the **target crate itself** (e.g., `ide/src/lib.rs`, `storvsp/src/lib.rs`). ``` ```admonish note `--with-html-report` offers a quick-and-easy way for an individual user generate a coverage report locally, but it may not be entirely appropriate for more "industrial scale" fuzzing pipelines. ``` ### Manual Coverage Generation (Advanced) The basic way this is done is by running all the discovered input testcases through the fuzzer and merging all the coverage events together (remember, the fuzzers only save testcases which generate new coverage). Cargo-fuzz provides a way to do this with the `coverage` subcommand. This step generates a `coverage.profdata` file which can be turned into a human-readable HTML report: ```bash # cargo xtask fuzz coverage cargo xtask fuzz coverage fuzz_ide # confirm coverage.profdata was created ls -l coverage.profdata ``` OR if you have a large number of inputs (5k+) the below will collect and merge coverage significantly faster: ```bash # rebuild the fuzzer with coverage instrumentation RUSTFLAGS="-C instrument-coverage" cargo fuzz build # set env var to rustup's llvm-tools LLVM_TOOLS_PATH=$(dirname $(find $(rustc --print sysroot) -name 'llvm-profdata')) # make an output directory for corpus minimation mkdir min_corp # run the minimizer putting the raw cov data into coverage.profraw LLVM_PROFILE_FILE="coverage.profraw" ./fuzz/targets//release/fuzz_ide min_corp -merge=1 # merge the raw data into coverage.profdata $LLVM_TOOLS_PATH/llvm-profdata merge -sparse coverage.profraw -o coverage.profdata ``` Next find the location of the llvm-tools you installed with rustup (NOTE: rustup is used to install the LLVM tools to ensure that rust's llvm version and the tool version are in sync), and convert the coverage data into a report: ```bash # set env var to rustup's llvm-tools LLVM_TOOLS_PATH=$(dirname $(find $(rustc --print sysroot) -name 'llvm-profdata')) # convert the coverage data into an lcov format $LLVM_TOOLS_PATH/llvm-cov export -instr-profile=coverage.profdata \ -format=lcov \ -object ./fuzz/targets//coverage//release/fuzz_ide \ --ignore-filename-regex "rustc" > coverage.lcov # summarize the coverage information lcov --summary ./coverage.lcov # make an output directory for the html report mkdir -p lcov_html # generate the html report genhtml -o lcov_html --legend --highlight ./coverage.lcov ``` --- ### Guide Src Dev Guide Tests Fuzzing Writing (`Guide/src/dev_guide/tests/fuzzing/writing.md`) # Writing Fuzzers ## Writing a new fuzzer in OpenVMM The easiest way to get up and running is to look at the existing in-tree fuzzers (which you can list using `cargo xtask fuzz list`), along with reading through the [cargo-fuzz book](https://rust-fuzz.github.io/book/cargo-fuzz.html) (the book is fairly brief and shouldn't take more than 20 minutes to read through) Some examples of in-tree fuzzers: * Simple device example: [chipset/fuzz/battery.rs][fuzz_battery_url] * More complex device example: [ide/fuzz/fuzz_ide.rs][fuzz_ide_url] * Abstraction over unsafe example: [ide/fuzz/fuzz_scsi_buffer.rs][fuzz_scsi_buffer_url] [fuzz_battery_url]: https://github.com/microsoft/openvmm/blob/main/vm/devices/chipset/fuzz/fuzz_battery.rs [fuzz_ide_url]: https://github.com/microsoft/openvmm/blob/main/vm/devices/storage/ide/fuzz/fuzz_ide.rs [fuzz_scsi_buffer_url]: https://github.com/microsoft/openvmm/blob/main/vm/devices/storage/scsi_buffers/fuzz/fuzz_scsi_buffers.rs Once you're ready to take a stab at writing your own fuzzer, spinning up a new fuzzer is as easy as running: ```bash cargo xtask fuzz init openvmm_crate_to_fuzz TEMPLATE ``` Use `--help` for more details on the available TEMPLATE types. ```admonish caution We don't suggest using `cargo fuzz init` (i.e: without `xtask`), as it emits a template that isn't compatible with the OpenVMM repo style, and also doesn't properly update the root Cargo.toml's `workspace.members` array. ``` ## Fuzzing an abstraction over unsafe code Unsafe code is a prioritized target for fuzzing given its self-evident risks. However, it can be hard to reason about how best to exercise unsafe code in OpenVMM via fuzzing as often (and correctly) the unsafe code is not directly interacting with guest-controlled data. The approach taken with OpenVMM then is to target abstractions over unsafe code. Such as interfaces and data structures like `BounceBuffers`, `guest_memory`, or `ucs2`. A fuzzer for one of these will attempt to be a regular consumer of the abstraction, calling those APIs declared safe and using any data structure in a rust-safe way. This attempts to check that the safety guarantees the abstraction is making are being upheld via the API. For example, let's say we want to fuzz `BounceBuffers`, here's what we may want to fuzz: Fuzz logic might then allocate a `BounceBuffer` using `new` and call methods on it such as `as_mut_bytes` and `io_vecs`. Then it could access the return result of both those calls: ```rust,ignore use scsi_buffers::BounceBuffer; #[derive(Arbitrary)] enum BouneBufferAccess { AsMutBytes, IoVecs } #[derive(Arbitrary)] struct FuzzCase { #[arbitrary(with = |u: &mut Unstructured| u.int_in_range(0..=0x40000))] size: usize, accesses: Vec } fn access_mut_bytes(buf: &mut [u8]) { buf.fill(b'A'); // access buf in other ways to test validity of underlying memory and slice } fn access_io_vecs(io_vecs: &[IoBuffer]) { // access each IoBuffer to test validity of ptr and len } fuzz_target!(|fuzz_case: FuzzCase| { do_fuzz(fuzz_case) }); fn do_fuzz(fuzz_case: FuzzCase) { let mut bb = BounceBuffer::new(fuzz_case.size); for access in fuzz_case.accesses { match access { AsMutBytes => { let buf = bb.as_mut_bytes(); access_mut_bytes(buf); }, IoVecs => { let io_vecs = bb.io_vecs(); access_io_vecs(io_vecs) } } } } ``` The fuzzer should work to ensure the safe members of the API cannot be misused in any way that may result in memory corruption or unsoundness. ## Fuzzing a chipset device Writing a fuzzer for a chipset device (e.g: battery, ide, serial, pic, etc...) involves targeting the API that is roughly exposed to guests: the device's port IO, PCI config, and MMIO interfaces. While it's entirely possible to hand-roll a fuzzer that is tailored to the specific register configuration of a particular device, the in-repo `chipset_device_fuzz` crate exports a `FuzzChipset` type that offers a "plug-and-play" way to hook a chipset device up to a fuzzer: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Fuzzing a vmbus device VMBus device fuzzers exercise the guest-facing VMBus channel protocol. Unlike chipset device fuzzers (which use `FuzzChipset` for PIO/MMIO dispatch), VMBus fuzzers create a pair of connected async channels and send packets directly. See [storvsp/fuzz/fuzz_storvsp.rs][fuzz_storvsp_url] for a complete example. The general pattern: 1. **Create connected channels** using `vmbus_channel::connected_async_channels` 2. **Set up the device** (controller, disks, etc.) and start a `TestWorker` 3. **Negotiate the protocol** — most VMBus devices reject all packets until initialization completes, so always negotiate upfront 4. **Loop over arbitrary actions** — send SCSI commands, reset packets, raw packets, and read completions 5. **Race teardown against the fuzz loop** using `futures::select!` ```rust,ignore use futures::{select, FutureExt}; use std::pin::pin; fn do_fuzz(u: &mut Unstructured<'_>) -> Result<(), anyhow::Error> { DefaultPool::run_with(|driver| async move { let (host, guest_channel) = connected_async_channels(4 * 1024); let guest_queue = Queue::new(guest_channel)?; // Set up the device under test let controller = ScsiController::new(); // ... attach disks ... let test_worker = TestWorker::start(controller, driver.clone(), mem, host, None); let mut guest = TestGuest { queue: guest_queue, transaction_id: 0 }; // Always negotiate — without init, the device rejects everything guest.perform_protocol_negotiation().await; // Fuzz loop let mut fuzz_loop = pin!(async { while !u.is_empty() { let action = u.arbitrary::()?; // ... execute action ... } Ok::<(), anyhow::Error>(()) }.fuse()); let mut teardown = pin!(test_worker.teardown_ignore().fuse()); select! { _ = fuzz_loop => {}, _ = teardown => {}, } Ok(()) }) } ``` [fuzz_storvsp_url]: https://github.com/microsoft/openvmm/blob/main/vm/devices/storage/storvsp/fuzz/fuzz_storvsp.rs ## Fuzzing `async` code Depending on the nature of the 'async' code in question, there are two main recommended approaches to fuzzing it: 1. **now_or_never**: The recommended approach for individual asynchronous calls is to use the `now_or_never` method from the `futures` crate. This method will poll the future to completion, but will not block if the future is not ready to complete. This allows you to fuzz the future without needing to run it to completion, which can be useful for testing the behavior of the future in various states. 2. **DefaultPool::run_with**: The recommended approach for more intricate asynchronous requirements is to use the `DefaultPool::run_with` method from our `pal_async` crate. This method takes a custom async function and runs it to completion. This allows you to write custom code using regular async/await syntax, combinators, `join`s, `select`s, or whatever you wish. --- ### Guide Src Dev Guide Tests Vmm Azure Images (`Guide/src/dev_guide/tests/vmm/azure_images.md`) # Azure-hosted Test Images OpenVMM utilizes pre-made VHDs in order to run tests with multiple guest operating systems. These images are as close to a "stock" installation as possible, created from the Azure Marketplace or downloaded directly from a trusted upstream source. These VHDs are stored in Azure Blob Storage, and are downloaded when running VMM tests in CI. ## Lazy Fetching (Remote Artifacts) When running tests with the OpenVMM backend, disk artifacts that are hosted on Azure Blob Storage can be fetched lazily over HTTP instead of being pre-downloaded. The VMM reads disk sectors on demand via HTTP Range requests, with a local SQLite cache so that sectors are only fetched once. This is controlled by two environment variables: - **`PETRI_REMOTE_ARTIFACTS`** — Set to `0` or `false` to force all artifacts to be resolved locally, disabling lazy fetching. By default, remote access is allowed for artifacts that opt in. - **`PETRI_CACHE_DIR`** — Override the directory used for the SQLite read cache. Defaults to a platform-appropriate cache directory (e.g. `~/.cache/petri` on Linux, `~/Library/Caches/petri` on macOS, `%LOCALAPPDATA%\petri\cache` on Windows). ```admonish note VHDX artifacts (e.g. `WINDOWS_11_ENTERPRISE_AARCH64`) do **not** support lazy fetching and must be downloaded locally. The Hyper-V backend also requires all artifacts to be local. ``` ## Downloading VHDs The `cargo xtask guest-test download-image` command can be used to download vhds to your machine. By default it will download all available VHDs, however the `--vhd` option can be used to only download select guests. After running it the tests can be run just like any other. This command requires having [AzCopy](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10) installed. --- ### Guide Src Reference Architecture Devices Storage (`Guide/src/reference/architecture/devices/storage.md`) # Storage pipeline The storage stack carries guest I/O requests from a guest-visible controller to a backing store and back. It's shared between OpenVMM and OpenHCL. Every disk backend implements the [`DiskIo`](https://openvmm.dev/rustdoc/linux/disk_backend/trait.DiskIo.html) trait, and frontends hold a [`Disk`](https://openvmm.dev/rustdoc/linux/disk_backend/struct.Disk.html) wrapper — a cheap, cloneable handle to any backend. For the `DiskIo` trait surface, method contracts, and error model, see the [`disk_backend` rustdoc](https://openvmm.dev/rustdoc/linux/disk_backend/index.html). ## The pipeline Every storage I/O flows through the same layered pipeline: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Key vocabulary: - **Frontend.** Speaks a guest-visible storage protocol and translates requests into `DiskIo` calls. - **SCSI adapter.** For the SCSI and IDE paths, an intermediate layer ([`SimpleScsiDisk`](https://openvmm.dev/rustdoc/linux/scsidisk/struct.SimpleScsiDisk.html) or [`SimpleScsiDvd`](https://openvmm.dev/rustdoc/linux/scsidisk/scsidvd/struct.SimpleScsiDvd.html)) that parses SCSI CDB opcodes before calling `DiskIo`. - **Backend.** A `DiskIo` implementation that reads and writes to a specific backing store. - **Decorator.** A `DiskIo` implementation that wraps another `Disk` and transforms I/O in transit (encryption, delay, persistent reservations). - **Layered disk.** A `DiskIo` implementation composed of ordered layers with per-sector presence tracking. ## Frontends Three frontends exist. Each speaks a different guest-visible protocol but they all produce `DiskIo` calls on the backend side. | Frontend | Protocol | Transport | Crate | |----------|----------|-----------|-------| | [NVMe](../../emulated/NVMe/overview.md) | NVMe 2.0 | PCI MMIO + MSI-X | [`nvme`](https://openvmm.dev/rustdoc/linux/nvme/index.html) | | [StorVSP](../../devices/vmbus/storvsp.md) | SCSI CDB over VMBus | VMBus ring buffers | [`storvsp`](https://openvmm.dev/rustdoc/linux/storvsp/index.html) | | [IDE](../../emulated/legacy_x86/ide.md) | ATA / ATAPI | PCI/ISA I/O ports + DMA | [`ide`](https://openvmm.dev/rustdoc/linux/ide/index.html) | **NVMe** is the simplest path. The NVMe controller's namespace directly holds a `Disk`. NVM opcodes (READ, WRITE, FLUSH, DSM) map nearly 1:1 to `DiskIo` methods. The FUA bit from the NVMe write command is forwarded directly. **StorVSP / SCSI** has a two-layer design. StorVSP handles the VMBus transport — negotiation, ring buffer management, [sub-channel allocation](../../devices/vmbus/storvsp_channels.md). It dispatches each SCSI request to an [`AsyncScsiDisk`](https://openvmm.dev/rustdoc/linux/scsi_core/trait.AsyncScsiDisk.html) implementation. For hard drives, that's [`SimpleScsiDisk`](https://openvmm.dev/rustdoc/linux/scsidisk/struct.SimpleScsiDisk.html), which parses the SCSI CDB and translates it to `DiskIo` calls. For optical drives, it's [`SimpleScsiDvd`](https://openvmm.dev/rustdoc/linux/scsidisk/scsidvd/struct.SimpleScsiDvd.html). **IDE** is the legacy path. ATA commands for hard drives call `DiskIo` directly. ATAPI commands for optical drives delegate to `SimpleScsiDvd` through an ATAPI-to-SCSI translation layer — the same DVD implementation that StorVSP uses. IDE also supports [enlightened INT13 commands](../../emulated/legacy_x86/ide.md#enlightened-io), a Microsoft-specific optimization that collapses the multi-exit register-programming sequence into a single VM exit. ## Backends A backend is a `DiskIo` implementation that reads and writes to a specific backing store. Backends are interchangeable — swap one for another without changing the frontend. The frontend holds a `Disk` and doesn't know what's behind it. See the [storage backends](../../backends/storage.md) page for the full catalog and platform details. ## Decorators A decorator is a `DiskIo` implementation that wraps another `Disk` and transforms I/O in transit. Features compose by stacking decorators without modifying backends: ```text CryptDisk └── BlockDeviceDisk ``` Three decorators exist: [`CryptDisk`](https://openvmm.dev/rustdoc/linux/disk_crypt/struct.CryptDisk.html) (XTS-AES-256 encryption), [`DelayDisk`](https://openvmm.dev/rustdoc/linux/disk_delay/struct.DelayDisk.html) (injected latency), and [`DiskWithReservations`](https://openvmm.dev/rustdoc/linux/disk_prwrap/struct.DiskWithReservations.html) (in-memory persistent reservation emulation). All three forward metadata (sector count, sector size, disk ID, `wait_resize`) to the inner disk unchanged. See the [storage backends](../../backends/storage.md) page for the decorator catalog. ## The layered disk model A [`LayeredDisk`](https://openvmm.dev/rustdoc/linux/disk_layered/struct.LayeredDisk.html) is a `DiskIo` implementation composed of multiple layers, ordered from top to bottom. Each layer is a block device with per-sector *presence* tracking. This model powers diff disks, RAM overlays, and caching. ### Reads fall through When a read arrives, the layered disk checks layers top-to-bottom. The first layer that has the requested sectors provides the data. Sectors not present in any layer are zeroed. ### Writes go to the top Writes always go to the topmost layer. If that layer is configured with *write-through*, the write also propagates to the next layer. ### Read caching A layer can be configured to cache read misses: when sectors are fetched from a lower layer, they're written back to the cache layer. This uses a `write_no_overwrite` operation to avoid overwriting sectors that were written between the read and the cache population. ### Layer implementations Two concrete layers exist today: - **RamDiskLayer** ([`disklayer_ram`](https://openvmm.dev/rustdoc/linux/disklayer_ram/index.html)) — ephemeral, in-memory. Data is stored in a `BTreeMap` keyed by sector number. Fast, but lost when the VM stops. - **SqliteDiskLayer** ([`disklayer_sqlite`](https://openvmm.dev/rustdoc/linux/disklayer_sqlite/index.html)) — persistent, backed by a SQLite database (`.dbhd` file). Designed for dev/test scenarios — no stability guarantees on the on-disk format. A full `Disk` can appear at the bottom of the stack as a fully-present layer (`DiskAsLayer`). This is the typical case: a RAM or sqlite layer on top of a file or block device. ### Worked example: `memdiff:file:disk.vhdx` ```text Layer 0: RamDiskLayer (empty, writable) Layer 1: DiskAsLayer wrapping FileDisk (fully present, read-only from the layered disk's perspective) ``` - Guest write → sector goes to the RAM layer. - Guest read → check RAM; if the sector is present, return it. If absent, fall through to the file. - Sectors absent from both layers → zero-filled. Changes are ephemeral — they live in the RAM layer and are lost when the VM stops. The [Running OpenVMM](../../../user_guide/openvmm/run.md) page shows concrete `memdiff:` examples. ## How configuration becomes a concrete stack The resource resolver connects configuration (CLI flags, VTL2 settings) to concrete backends. A resource *handle* describes what backend to use; a *resolver* creates it. The storage resolver chain is recursive. An NVMe controller resolves each namespace's disk, which may be a layered disk, which resolves each layer in parallel, which may itself be a disk that needs resolving. **Example:** `--vmbus-scsi id=scsi0 --disk memdiff:file:path/to/disk.vhdx,on=scsi0` 1. CLI parses this into a `LayeredDiskHandle` with two layers: - Layer 0: `RamDiskLayerHandle { len: None, sector_size: None }` (RAM diff, inherits size and sector size from backing disk) - Layer 1: `DiskLayerHandle(FileDiskHandle(...))` (the file) 2. The layered disk resolver resolves both layers in parallel. 3. The RAM layer attaches on top of the file layer, inheriting its sector size and capacity. 4. The resulting `LayeredDisk` is wrapped in a `Disk` and handed to the NVMe namespace or SCSI controller. For the OpenHCL settings model (`StorageController`, `Lun`, `PhysicalDevice`), see [Storage Translation](../openhcl/storage_translation.md) and [Storage Configuration Model](../openhcl/storage_configuration.md). ## Backend catalog | Backend | Crate | Wraps | Platform | Note | |---------|-------|-------|----------|------| | FileDisk | [`disk_file`](https://openvmm.dev/rustdoc/linux/disk_file/index.html) | Host file | Cross-platform | Simplest backend | | Vhd1Disk | [`disk_vhd1`](https://openvmm.dev/rustdoc/linux/disk_vhd1/index.html) | VHD1 fixed file | Cross-platform | Parses VHD footer | | VhdmpDisk | `disk_vhdmp` | Windows vhdmp driver | Windows | Dynamic/differencing VHD/VHDX | | VhdxDisk | [`vhdx`](../../backends/vhdx.md) | VHDX file | Cross-platform | Pure-Rust VHDX parser | | BlobDisk | [`disk_blob`](https://openvmm.dev/rustdoc/linux/disk_blob/index.html) | HTTP / Azure Blob | Cross-platform | Read-only, HTTP range requests | | BlockDeviceDisk | [`disk_blockdevice`](https://openvmm.dev/rustdoc/linux/disk_blockdevice/index.html) | Linux block device | Linux | io_uring, resize via uevent, PR passthrough | | NvmeDisk | [`disk_nvme`](https://openvmm.dev/rustdoc/linux/disk_nvme/index.html) | Physical NVMe (VFIO) | Linux/Windows | User-mode NVMe driver, resize via AEN | | StripedDisk | [`disk_striped`](https://openvmm.dev/rustdoc/linux/disk_striped/index.html) | Multiple Disks | Cross-platform | Data striping | ## Online disk resize Disk resize is a cross-cutting concern that spans backends and frontends. ### Backend detection Only two backends detect capacity changes at runtime: - **BlockDeviceDisk** — listens for Linux uevent notifications on the block device. When the host resizes the device, a uevent fires, the backend re-queries the size via ioctl, and `wait_resize` completes. - **NvmeDisk** — the user-mode NVMe driver monitors Async Event Notifications (AEN) from the physical controller and rescans namespace capacity. All other backends default to never signaling (`wait_resize` returns `pending()`). Decorators and layered disks delegate `wait_resize` to the inner backend. ```admonish warning `FileDisk` never signals resize. If you attach a file backend and resize the file at runtime, nothing happens — the guest won't be notified. Use `BlockDeviceDisk` or `NvmeDisk` for runtime resize. ``` ### Frontend notification Once a backend detects a resize, the frontend notifies the guest: | Frontend | Mechanism | How it works | |----------|-----------|-------------| | NVMe | Async Event Notification | Background task per namespace calls `wait_resize`. On change, completes a queued AER command with a changed-namespace-list log page. Guest re-identifies the namespace. | | StorVSP / SCSI | UNIT_ATTENTION | On the next SCSI command after a resize, `SimpleScsiDisk` detects the capacity change and returns CHECK_CONDITION with UNIT_ATTENTION / CAPACITY_DATA_CHANGED. Guest retries and re-reads capacity. | | IDE | Not supported | IDE has no capacity-change notification mechanism. | The resize path is the same in OpenHCL and standalone — `BlockDeviceDisk` detects the uevent from the host, `wait_resize` completes, and the frontend notifies the guest through the standard mechanism. No special paravisor-level interception. ## Virtual optical / DVD DVD and CD-ROM drives use a different model from disk devices. [`SimpleScsiDvd`](https://openvmm.dev/rustdoc/linux/scsidisk/scsidvd/struct.SimpleScsiDvd.html) implements `AsyncScsiDisk` and manages media state: a disk can be `Loaded` or `Unloaded`. Optical media always uses a 2048-byte sector size. The implementation handles optical-specific SCSI commands: `GET_EVENT_STATUS_NOTIFICATION`, `GET_CONFIGURATION`, `START_STOP_UNIT` (eject), and media change events. ### Eject Two eject paths exist: - **Guest-initiated** (SCSI `START_STOP_UNIT` with the load/eject flag): the DVD handler checks the prevent flag, replaces media with `Unloaded`, and calls `disk.eject()`. Once ejected via SCSI, the media is **permanently removed** for the VM lifetime. - **Host-initiated** (`change_media` via the resolver's background task): can insert new media or remove existing media dynamically. ### Frontend support | Frontend | DVD support | How | |----------|-------------|-----| | StorVSP / SCSI | Yes | `SimpleScsiDvd` implements `AsyncScsiDisk` directly. | | IDE | Yes | ATAPI wraps `SimpleScsiDvd` through the ATAPI-to-SCSI layer. | | NVMe | No | NVMe has no removable media concept. Explicitly rejected. | ### CLI - `--disk file:my.iso,dvd,on=scsi0` → SCSI optical drive. - `--ide file:my.iso,dvd` → IDE optical drive (ATAPI). The `dvd` flag implicitly sets `read_only = true`. ## `mem:` and `memdiff:` CLI mapping Both CLI options map to the layered disk model: - **`mem:1G`** creates a single-layer `LayeredDisk` with a `RamDiskLayer` sized to 1 GB. No backing disk — the RAM layer is the entire disk. - **`memdiff:file:disk.vhdx`** creates a two-layer `LayeredDisk`: a `RamDiskLayer` (inheriting size from the backing disk) on top of the file. Writes go to the RAM layer; reads fall through to the file for sectors not yet written. Both use `RamDiskLayerHandle` under the hood. The difference is `len: Some(size)` for `mem:` (standalone RAM disk with explicit size) vs. `len: None` for `memdiff:` (inherits from backing disk). The optional `sector_size` field (default `None`) lets you override the sector size; when `None`, it inherits from the lower layer or defaults to 512 bytes. The [Running OpenVMM](../../../user_guide/openvmm/run.md) page shows concrete examples. ## Controller identity and Azure disk classification In Azure, which controller a disk sits on is a de facto compatibility boundary. Azure VMs present four SCSI controllers (this may change), each with a distinct instance ID. One controller carries the OS disk, resource (temporary) disk, and related infrastructure disks; a separate controller carries remote data disks. For Gen1 VMs, the IDE controllers logically replace that first SCSI controller, while data disks remain on SCSI. Guest agents use controller identity to classify disks. The [azure-vm-utils udev rules](https://github.com/Azure/azure-vm-utils/blob/main/udev/80-azure-disk.rules) match on SCSI controller instance IDs to create stable symlinks under `/dev/disk/azure/`. Moving a disk from one StorVSP controller instance to another changes its classification and can break guest-side automation. For SCSI disk mapping details, see the [Azure disk mapping docs](https://learn.microsoft.com/en-us/azure/virtual-machines/windows/azure-to-guest-disk-mapping). For NVMe, the mapping uses namespace IDs: NSID 1 is the OS disk, NSID 2+ are data disks (portal LUN = NSID − 2). On newer VM sizes (v7+), disks are split across multiple NVMe controllers by caching policy. NVMe is Gen2-only. See the [NVMe overview](https://learn.microsoft.com/en-us/azure/virtual-machines/nvme-overview) and [NVMe disk identification FAQ](https://learn.microsoft.com/en-us/azure/virtual-machines/enable-nvme-remote-faqs) for the full Azure perspective. ## Implementation map | Component | Why read it | Source | Rustdoc | |-----------|-------------|--------|---------| | `disk_backend` | `DiskIo` trait, `Disk` wrapper, error model | [source](https://github.com/microsoft/openvmm/blob/main/vm/devices/storage/disk_backend/src/lib.rs) | [rustdoc](https://openvmm.dev/rustdoc/linux/disk_backend/index.html) | | `disk_layered` | Layered disk, `LayerIo` trait, bitmap tracking | [source](https://github.com/microsoft/openvmm/blob/main/vm/devices/storage/disk_layered/src/lib.rs) | [rustdoc](https://openvmm.dev/rustdoc/linux/disk_layered/index.html) | | `nvme` | NVMe controller emulator | [source](https://github.com/microsoft/openvmm/blob/main/vm/devices/storage/nvme/src/lib.rs) | [rustdoc](https://openvmm.dev/rustdoc/linux/nvme/index.html) | | `storvsp` | VMBus SCSI controller | [source](https://github.com/microsoft/openvmm/blob/main/vm/devices/storage/storvsp/src/lib.rs) | [rustdoc](https://openvmm.dev/rustdoc/linux/storvsp/index.html) | | `scsidisk` | SCSI CDB parser (`SimpleScsiDisk`, `SimpleScsiDvd`) | [source](https://github.com/microsoft/openvmm/blob/main/vm/devices/storage/scsidisk/src/lib.rs) | [rustdoc](https://openvmm.dev/rustdoc/linux/scsidisk/index.html) | | `ide` | IDE controller emulator | [source](https://github.com/microsoft/openvmm/blob/main/vm/devices/storage/ide/src/lib.rs) | [rustdoc](https://openvmm.dev/rustdoc/linux/ide/index.html) | | `scsi_core` | `AsyncScsiDisk` trait, `Request`, `ScsiResult` | [source](https://github.com/microsoft/openvmm/blob/main/vm/devices/storage/scsi_core/src/lib.rs) | [rustdoc](https://openvmm.dev/rustdoc/linux/scsi_core/index.html) | --- ### Guide Src Reference Architecture Openhcl Ak Cert (`Guide/src/reference/architecture/openhcl/ak_cert.md`) # AK Certificate Renewal This page describes how OpenHCL-managed virtual TPM attestation-key (AK) certificates are requested, retained, and replaced. AK certification is separate from VMGS key release. See [Attestation and VMGS Protection](attestation.md) for the surrounding flows. The [IGVM Attestation Protocol](attestation_protocol.md) defines the exact metadata encoding and response contract. ## Request lifecycle When OpenHCL is configured to manage AK certificate renewal and the TPM permits it, the TPM device requests a certificate at initialization and can request renewal later. Only one certificate request is outstanding at a time. For each request: 1. Read the current `TPM_NV_INDEX_AIK_CERT` contents, using the index's declared size rather than a fixed legacy-size buffer. 2. Decode the leading DER certificate and extract renewal metadata. Trailing NV padding or bytes left by an earlier, longer certificate are not part of that certificate. 3. Include successfully decoded metadata in the runtime claims before hashing them and obtaining the attestation report. Record whether metadata was included in this particular request. 4. Send the request through the host IGVM agent and process its response asynchronously. An absent or uninitialized NV index supplies no metadata. Read or decode failures are logged and omit the metadata, allowing normal issuance to proceed. The decoder requires issuer and subject common names; the authority key identifier is optional. If including metadata would exceed GET's 4 KiB request limit, OpenHCL omits it before hashing the claims and generating the attestation report. This is unlikely with ordinary production certificates. A metadata-free request that still exceeds the transport limit fails rather than being sent. Metadata is read for each request, not cached separately. Certificate replacement, TPM key refresh, and state restoration therefore do not require maintaining a second copy of certificate metadata. ## Renewal outcomes A nonempty certificate response follows the existing NV replacement path. OpenHCL writes the returned certificate to the certificate index and advances the renewal timer only after the write succeeds. NV write failures are logged and emit a certificate-renewal-failed event without advancing the timer, so the next eligible guest NV read can retry. NV writes are not atomic: a failure after some chunks have been written can leave a partially replaced certificate. A valid no-renewal response is accepted only if this request included successfully decoded existing-certificate metadata. OpenHCL keeps the NV contents without a write and advances the renewal timer. Without that request-specific metadata, the same response is logged as a renewal failure and does not advance the timer. An empty transport response or an empty certificate payload indicates service unavailability, not approval of the existing certificate. OpenHCL logs it, leaves NV unchanged, and advances the timer to avoid retrying on every read. Transport or response-parsing errors log a renewal failure without advancing the timer. Subsequent renewal is driven by guest NV reads and the existing 24-hour renewal interval, rather than by an expiry-time alarm. The interval uses OpenHCL system time, which can diverge from wall-clock elapsed time across suspension and resumption. ## Trust and guest compatibility Certificate metadata is a renewal hint. Decoding it does not validate the certificate signature, chain, validity period, or relationship to the TPM AK. The no-renewal signal is a host response, not a signed certificate-validation result. Guest-requested attestation reports remain separate from renewal requests. They retain their V1 layout and do not read or include certificate renewal metadata, preserving the existing guest-visible NV format. ## Implementation references - [`tpm_device`][tpm-rustdoc] owns NV access, pending requests, and renewal scheduling. - [`tpm_resources`][resources-rustdoc] defines the request helper interface and typed certificate/no-renewal outcomes. - `openhcl/underhill_core/src/emuplat/tpm.rs` connects the TPM helper to metadata parsing, report generation, and GET. - [`underhill_attestation`][attestation-rustdoc] extracts certificate metadata and builds and parses attestation messages. [tpm-rustdoc]: https://openvmm.dev/rustdoc/tpm_device/index.html [resources-rustdoc]: https://openvmm.dev/rustdoc/tpm_resources/index.html [attestation-rustdoc]: https://openvmm.dev/rustdoc/underhill_attestation/index.html --- ### Guide Src Reference Architecture Openhcl Attestation (`Guide/src/reference/architecture/openhcl/attestation.md`) # Attestation and VMGS Protection This page introduces OpenHCL's attestation flows and directs you to their lifecycle and protocol references. ## Attestation flows OpenHCL uses attestation for two distinct purposes: - **VM Guest State (VMGS) protection:** secure key release obtains key material used to recover access to encrypted VMGS. Hardware sealing provides local key protection where supported and enabled by the VM configuration. - **TPM attestation-key (AK) certification:** the virtual TPM requests a certificate for its AK and can later request renewal. Issuing or retaining this certificate is separate from releasing keys to unlock VMGS. The host IGVM agent is reached through the Guest Emulation Transport (GET). For hardware-attested requests, OpenHCL obtains a TEE report that binds a hash of the serialized runtime claims. Binding the claims does not by itself authenticate host-supplied metadata or establish certificate trust. ## Reference pages - [AK Certificate Renewal](ak_cert.md) owns certificate storage, renewal decisions, scheduling, and the guest-report boundary. - [IGVM Attestation Protocol](attestation_protocol.md) owns the AK request claims, response framing, signal definitions, and compatibility rules. ## Implementation references - [`underhill_attestation`][attestation-rustdoc] prepares attestation requests, parses responses, and implements VMGS key protection. - [`tee_call`][tee-rustdoc] provides access to hardware attestation and supported key derivation. [attestation-rustdoc]: https://openvmm.dev/rustdoc/underhill_attestation/index.html [tee-rustdoc]: https://openvmm.dev/rustdoc/tee_call/index.html ## Boot and key protection Depending on the VM's configuration, platform-security initialization obtains key material through secure key release (SKR), hardware unsealing, and guest state protection services. Hardware sealing can provide either the sole means of recovering the VMGS datastore key (DEK) or a recovery path alongside other key protectors. The hardware protector contains the sealed DEK and the security version numbers (SVNs) needed to reproduce its hardware-derived key. It is stored in `HW_KEY_PROTECTOR` without VMGS-level encryption so it can be read before VMGS is unlocked. Its header, including SVN metadata, and encrypted DEK are covered by a hardware-keyed HMAC. Boot unsealing uses the protector's recorded SVN, which can be lower than the current hardware SVN. Recovery requires hardware support for that derivation and successful protector authentication. This differs from the minimum TCB policy used for runtime resealing below. Ordinary stateless mode bypasses VMGS encryption when hardware sealing is not requested. Stateless mode with the `HardwareSealing` encryption policy instead requires hardware sealing: it seals a new DEK on provisioning and unseals then rotates it on later boots. Unsupported required sealing or failure to derive, unseal, write, or finalize the required protector prevents boot from completing. The main implementation is in [`underhill_attestation`](https://openvmm.dev/rustdoc/linux/underhill_attestation/index.html), with local hardware access provided by [`tee_call`](https://openvmm.dev/rustdoc/linux/tee_call/index.html). ## Resiliency ### Recovery after live migration A hardware-derived sealing key may change when a VM migrates to another host. OpenHCL retains the active DEK in protected memory, allowing it to create a destination-compatible protector without recovering the DEK from the source protector. The GET `NOTIFY_POST_LIVE_MIGRATION` notification triggers this recovery: 1. Obtain a local report and check it against the runtime TCB floor. 2. Derive destination hardware keys and seal the **unchanged** active DEK. 3. Verify the candidate protector, then write and flush it through the VMGS broker. A defensive active-key check prevents publishing a stale DEK if concurrent key rotation is introduced. 4. Verify again after persistence to detect hardware or SVN changes during the operation. Recovery is enabled only when the successful boot-unlock attempt has sealed, written, and flushed a hardware protector for the active DEK and established a trusted runtime floor under a supported policy. This eligibility comes from trusted operation results, not the presence of an entry on disk. An LM event maintains an established recovery path; it does not perform first-time sealing. Once enabled, recovery can repair a subsequently missing or corrupt protector. It neither changes the policy nor rotates the DEK. Hardware calls run off the VP and GET executors. GET coalesces notifications received before callback registration into one pending event and delivers it when registration completes. The worker also latches events received before startup or during recovery. Failed attempts retry with bounded backoff. There is no periodic verification; after success, the worker waits for another event. Restoring serialized resealer state also schedules a durable rewrite, as described below. ### Runtime TCB floor The trusted computing base (TCB) floor prevents runtime resealing at a lower accepted security version. Boot establishes it from local reports already obtained during platform-security initialization, including reports obtained before failed SKR callouts. It is held in protected OpenHCL memory, not loaded from host-controlled VMGS metadata. Compatible higher observations advance the floor. Retries, persistence failures, reset, and normal stop/start cannot lower it. If boot cannot establish a usable floor, runtime resealing stays disabled with a warning; existing boot recovery behavior is unchanged. If optional boot sealing does not succeed, no floor is exported to the runtime worker. When hardware sealing is only a backup, finalization failure disables runtime recovery without failing boot. When sealing is required, or boot used hardware unsealing to recover the DEK, failure to confirm the active sealed key or flush completed writes fails boot. The full unlock/key-rotation sequence is not automatically retried after this failure because it may already have changed the active DEK. Existing protector write/unlock errors retain their handling. The comparison is a partial order, not a packed-integer or lexicographic order: - **SNP:** known TCB components must be non-decreasing within the same supported report version and CPU family/model; reserved bytes must remain equal. For report versions 3–5, Milan/Genoa (family `19h`, models `00h–1Fh`) and Turin (family `1Ah`, models `90h–AFh` and `C0h–CFh`) use distinct typed layouts, as defined by AMD 56860 section 2.3. Report version alone does not select the layout. Version 2 and unknown domains require exact SVN equality; cross-domain transitions are rejected. The floor tracks the reported SVN used for key derivation; it does not enforce the extended TCB added in report v6. - **TDX:** CPU SVN components must be non-decreasing. TEE SVN is parsed using the typed `TeeTcbSvn` ABI layout: major SVN (byte 1) identifies the module and must match, including legacy identity 0. Minor SVN (byte 0) and SE_SVN (byte 2) must be non-decreasing; reserved bytes 3–15 must remain equal. Reserved metadata changes are incompatible, not upgrades. This minimum-TCB comparison does not determine Intel security status, which requires authenticated Intel TCB Info. Candidate verification also checks a fresh report. Its SVN must match the candidate exactly, even if the hardware could still derive an older SVN's key. CPU SVN comparison uses all 16 unsigned bytes in their original positions: every destination byte must be at least its resident floor byte. There is no integer conversion or lexicographic ordering; an increase in one byte cannot compensate for a decrease in another. Intel PCS [Get TDX TCB Info V4][intel-tcb], step 3.a, uses component-wise minima for the 16 PCK certificate TCB components; [Appendix A][intel-tcb-model] describes the components and comparison metadata. OpenHCL's raw-report comparison is a local no-decrease policy, not that complete appraisal algorithm: it neither maps raw CPU SVN bytes to PCK component identities nor establishes cross-platform equivalence or Intel security status. For SNP, `SnpDomain` retains report version and CPU family/model because the raw TCB bytes do not identify their own layout. CPU family/model selects the layout; version gates supported report semantics. Exact domain equality is required even for identical raw SVNs or CPUs sharing a layout. This conservative restriction avoids interpreting one CPU's component bytes using another CPU's meanings. See [AMD 56860][amd-snp], revision 1.59, section 2.3, tables 4 and 5. [intel-tcb]: https://api.portal.trustedservices.intel.com/content/documentation.html#pcs-tcb-info-tdx-v4 [intel-tcb-model]: https://api.portal.trustedservices.intel.com/content/documentation.html#pcs-tcb-info-model-v3 [amd-snp]: https://docs.amd.com/v/u/en-US/56860_PUB_SEV_SNP ### Lifecycle and saved state Memory-preserving migration retains the worker and its floor. Stopping the worker drains an in-flight recovery attempt. Reset does not request resealing; it preserves the floor, existing retries and backoff, and latched notifications. The stopped worker saves a versioned protobuf state unit named `hardware_reseal`, containing: - The accepted TCB floor: raw SNP TCB and report version/CPU comparison domain, or both raw TDX SVN arrays, including identity and reserved bytes. - The serialized VM configuration used by the sealing KDF, checked against the destination configuration rather than used to replace it. - Pending recovery, the failure count, and the latched migration notification. - Remaining retry and minimum-interval delays, bounded to 60 seconds. These are relative durations, not process-specific clock values. No DEK, hardware secret, derived key, or attestation report is saved by this worker. Saving does not consume a notification or clear pending work: if another unit fails to save, the original worker can resume with its obligations intact. Restore constructs the worker without a floor, then installs the validated saved floor before starting. It does not obtain a fresh report to initialize a new floor or trust a VMGS protector as its source. Malformed or unsupported state, an incompatible TEE or sealing configuration, or a decrease relative to an existing resident floor fails restore without replacing that floor. A saved worker also requires compatible hardware sealing support and encrypted VMGS. Successful restore preserves retry backoff and merges saved notifications with notifications received by the new worker. It always schedules a durable rewrite when the worker starts, even when the saved worker was idle: hardware can change while the worker is absent, including at the same SVN. Fresh reports must satisfy the restored floor before any new protector is created. A cached matching protector does not discharge a failed-flush obligation. If the saved VM has no `hardware_reseal` unit, runtime resealing stays disabled after reconstruction. This accommodates state from runtimes that were not enrolled or did not support resealer state, without inventing a replacement floor. Binaries that cannot interpret this state unit cannot restore an enrolled worker's snapshot. ```admonish warning title="Trusted state transfer" Protobuf encoding does not authenticate the saved floor. The enclosing transfer must preserve integrity, bind the state to the VM, and prevent rollback to an older snapshot. The resealer must not load this state from VMGS or unauthenticated host metadata. Host-returned servicing state cannot initialize this worker with the current transport and is rejected. Resealer serialization does not enable isolated-VM servicing by itself. The host-servicing isolation guard and unsupported SNP/TDX VP save/restore paths remain in place. A trusted isolated-servicing handoff must satisfy the above requirements before enabling that path. ``` ### Recovery limits Cold boot establishes a new runtime lifetime. The floor is not a persistent anti-rollback counter and does not prevent replay of an entire VMGS snapshot. Rejecting resealing on a lower-TCB destination also cannot undo delivery of the resident DEK by migration; migration admission is a separate security boundary. ```admonish warning Recovery is best effort. A missed notification can leave the protector stale, and a crash before durable destination resealing can leave hardware-only VMGS unrecoverable. Report, derivation, and persistence operations are not atomic with migration. Durability also depends on backing storage honoring flushes. ``` --- ### Guide Src Reference Architecture Openhcl Attestation Protocol (`Guide/src/reference/architecture/openhcl/attestation_protocol.md`) # IGVM Attestation Protocol This page specifies the AK certificate request claims and response contract used by OpenHCL over the Guest Emulation Transport (GET). ```admonish note title="See also" [Attestation and VMGS Protection](attestation.md) introduces the attestation flows. [AK Certificate Renewal](ak_cert.md) describes NV storage, renewal scheduling, and the trust boundary. ``` ## AK request and response versions AK certificate requests use request-data version V2 and accept V1 or V2 responses. The parser selects the layout from the received response version; other response versions are rejected. Guest-requested attestation reports retain the V1 request layout exposed through TPM NV storage. They do not include the renewal metadata below. ## Existing-certificate claims An AK request with successfully decoded existing-certificate metadata includes an optional top-level `akcert` object in its runtime claims. If metadata is unavailable, the entire object is omitted, not serialized as `null`. Key-release requests and guest-requested reports omit it. | Field | Encoding | | --- | --- | | `issuer` | Certificate issuer common name, not its full distinguished name. | | `subject` | Certificate subject common name. | | `not_before` | ASN.1 time content: `YYMMDDhhmmssZ` for UTCTime or `YYYYMMDDhhmmssZ` for GeneralizedTime. | | `not_after` | The same encoding rule, selected by this field's ASN.1 time type. | | `authority` | Uppercase hex of the Authority Key Identifier's `keyIdentifier` bytes; empty string if absent. | The field names use underscores exactly as shown. The time strings are not Unix timestamps or RFC 3339 strings. The authority identifier has no prefix or separators and does not encode the complete extension. For example, this is an illustrative claim object, not a usable certificate: ```json { "akcert": { "issuer": "Example AK issuer", "subject": "Example AK", "not_before": "260101000000Z", "not_after": "270101000000Z", "authority": "0123456789ABCDEF" } } ``` Claims are serialized with JSON escaping before hashing. The SHA-256 digest occupies the first 32 bytes of the 64-byte report-data field; the remaining bytes are zero. Adding `akcert` therefore changes the report binding, not just an unmeasured transport field. ## AK response framing Header integers are 32-bit little-endian values. `data_size` includes the header and any payload. | Offset | Size | Field | | --- | --- | --- | | 0 | 4 | `data_size` | | 4 | 4 | Response `version` | | 8 | 4 | V2 only: `IgvmErrorInfo.error_code` | | 12 | 4 | V2 only: `IgvmErrorInfo.http_status_code` | | 16 | 4 | V2 only: `IgvmErrorInfo.igvm_signal` | | 20 | 12 | V2 only: reserved | V1 has an 8-byte header; V2 has a 32-byte header. Certificate bytes follow the version-specific header directly, without a JSON response envelope. The AK response buffer is 4 KiB, including the header. The parser requires a complete version-specific header within `data_size`, rejects a declared size larger than the supplied buffer, and ignores bytes after the declared size. A nonzero V2 `error_code` is an error, even when the no-renewal signal is also set. ### No-renewal signal V2 signal **bit 5** (`IGVM_SIGNAL_NO_RENEW_CERT_BIT`, mask `0x20`) means that the service recommends retaining the existing AK certificate described in the request. Bit 4 remains reserved. A successful, validly framed response with bit 5 set returns a no-renewal outcome instead of certificate bytes. The signal takes precedence over any payload in that response. Other signal bits alone do not indicate no-renewal. V1 has no signal field and cannot express this outcome. The TPM caller additionally requires that the request included successfully decoded certificate metadata before accepting no-renewal. See [renewal outcomes](ak_cert.md#renewal-outcomes) for timer, NV, and failure handling. An empty response or certificate payload is distinct from this explicit signal. ## Implementation references - [`openhcl_attestation_protocol::igvm_attest::get`][get-rustdoc] defines the request and response structures, signal bitmap, and runtime claims. - [`underhill_attestation`][attestation-rustdoc] implements claim hashing, certificate metadata extraction, and response validation. [get-rustdoc]: https://openvmm.dev/rustdoc/openhcl_attestation_protocol/igvm_attest/get/ [attestation-rustdoc]: https://openvmm.dev/rustdoc/underhill_attestation/index.html --- ### Guide Src Reference Architecture Openhcl Boot (`Guide/src/reference/architecture/openhcl/boot.md`) # OpenHCL Boot Flow This page describes how the host loads OpenHCL, how `openhcl_boot` starts Linux, and how the OpenHCL user-mode processes begin running. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## 1. IGVM Loading The host VMM follows the directives in the OpenHCL IGVM file to populate VTL2 memory and start the boot shim. The file contains the boot shim, Linux kernel, initial ramdisk, and other components needed by the paravisor. The IGVM file tells the host where to load each component and which initial state contributes to the launch measurement. The host also supplies runtime values such as processor topology, VTL2 memory, serial configuration, and device settings. ## 2. Boot Shim Execution (`openhcl_boot`) The host transfers control to `openhcl_boot`, which performs these steps: 1. **Hardware initialization:** Initialize CPU state and the memory management unit (MMU). 2. **Configuration validation:** Validate imported regions and combine measured build-time parameters with the runtime data permitted by the image policy. 3. **Device tree construction:** Build the hardware topology and the set of devices passed to Linux. 4. **Sidecar setup (x86_64):** Assign processors to Linux or sidecar, initialize their control structures, and start the sidecar processors. 5. **Kernel handoff:** Start Linux with the final device tree, command line, and architecture-specific boot data. ### Measured and host-provided inputs The measured IGVM fixes component addresses, imported regions, initrd metadata, and static command-line options. At launch, the host contributes topology and resource data. For a hardware-isolated VM, the shim filters that host data according to the policy in the measured image before exposing it to Linux. ## 3. Linux Kernel Boot Linux initializes memory management, scheduling, and the drivers used by the paravisor. It mounts the initial ramdisk as its root filesystem and starts `underhill_init` as PID 1. Sidecar CPUs remain in their dispatch loop until Linux hot-plugs them. ## 4. Userspace Initialization (`underhill_init`) `underhill_init` mounts the required pseudo-filesystems, configures the process environment and system limits, and then replaces itself with `/bin/openvmm_hcl`. ## 5. Paravisor Startup (`openvmm_hcl`) `openvmm_hcl` reads topology and configuration from `/proc/device-tree` and other kernel interfaces. It initializes host communication and VTL0 management, then starts the `underhill_vm` worker. `openvmm_hcl` remains the policy and control-plane process after spawning the worker. See the dedicated [`openvmm_hcl`](./openvmm_hcl.md) page for its resource, worker, servicing, and diagnostic responsibilities. ## 6. VM Execution The `underhill_vm` process runs the VTL0 virtual processors, handles exits, and coordinates device emulation. Devices such as the virtual TPM can run in dedicated worker processes, with `underhill_vm` proxying their I/O and guest memory access. `openvmm_hcl` remains responsible for policy, lifecycle, and host communication. --- ### Guide Src Reference Architecture Openhcl Cpu Scheduling (`Guide/src/reference/architecture/openhcl/cpu_scheduling.md`) # CPU Scheduling OpenHCL runs a cooperative async executor on each VP thread. This page explains how VP threads split time between guest execution and device work, what happens when things block, and how the [sidecar kernel](sidecar.md) changes the picture. ```admonish tip This document uses "lower VTL" and "VTL0" to refer to what are similar things. VTLs increase in privilege as the number gets higher. VTL2 is a higher privilege level than VTL1, which is yet higher than VTL0. Engineers focused on IO often think about the "guest" as "VTL0", since that's the VTL that issues IO to storage and networking devices. When this document discusses entering VTL0, though, it's more precise to say that control returns to any VTL that is less privileged than VTL2. It might be VTL0 or it might be VTL1. ``` ## Scope This page covers the **VM worker process** — the main OpenHCL process that runs device emulation and VP dispatch. OpenHCL also runs other processes (see [Processes and Components](processes.md)), but the cooperative executor model described here applies specifically to the worker process and its per-VP threadpool. For background on Rust async executors, see the [Asynchronous Programming in Rust](https://rust-lang.github.io/async-book/) book (especially this [Under the Hood](https://rust-lang.github.io/async-book/02_execution/01_chapter.html) section). ## Thread model OpenHCL's worker process runs one thread per VP in its [threadpool](https://openvmm.dev/rustdoc/linux/underhill_threadpool/index.html). Each thread is CPU-affinitized — thread N is pinned to Linux CPU N, which maps 1:1 to VP index N in current configurations. ```text VP 0 thread (CPU 0) VP 1 thread (CPU 1) VP 2 thread (CPU 2) ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ cooperative │ │ cooperative │ │ cooperative │ │ async executor │ │ async executor │ │ async executor │ │ │ │ │ │ │ │ • async workers │ │ • async workers │ │ • async workers │ │ e.g. devices │ │ e.g. devices │ │ e.g. devices │ │ • when idle: │ │ • when idle: │ │ • when idle: │ │ enter lower VTL │ │ enter lower VTL │ │ enter lower VTL │ └──────────────────┘ └──────────────────┘ └──────────────────┘ ``` The code calls VTL0 execution the thread's "idle task" — meaning the VP thread enters VTL0 when there is no pending VTL2 async work. The thread itself is not idle (the physical CPU is running guest code), but the VTL2 executor has nothing to do. Alongside the VP threads, the worker process runs a few additional threads: - **GET (Guest Emulation Transport) worker** — on a dedicated thread because it issues blocking syscalls that would stall the VP executor. When a GET message arrives, it is processed on this dedicated thread, not on a VP thread. Results are dispatched back to VP threads via async channels. - **Tracing thread** — log collection. - **CPU-online helper threads** — temporary, used when bringing sidecar VPs into Linux. This list may not be exhaustive at the point that you're reading these docs, but the point remains: _most_ work happens on the same thread as the lower VTL VPs and work that occurs on their behalf in OpenHCL. ## Cooperative scheduling Each VP thread runs an async executor that multiplexes all tasks targeted at that VP. Tasks only yield to each other at `.await` points. If you're not familiar with the Rust execution model, it may be tempting to think that the system will time slice execution or blocking requests with other async tasks running on the same VP thread. It won't. ### What runs on a VP thread All tasks with `target_vp = N` and `run_on_target = true` run on VP N's thread, once the target VP is ready (i.e., the CPU is online and affinity is set). All tasks with `target_vp = N` and `run_on_target = false` can be spawned on any VP's thread. However, `target_vp` still matters for IO affinity: IOs issued by the task use VP N's io-uring. When the IO completes, the task is woken on VP N and will likely continue running there. In practice, these tasks gravitate toward their target VP despite not requiring it for scheduling. This keeps per-VP I/O flows localized: submission, completion, and follow-up processing tend to happen on the same VP. All tasks without a specific `target_vp` will fall into the thread pool's untargeted path. These could run on any arbitrary VP. IO operations from untargeted tasks currently use VP 0's io-uring. This is a compatibility default and may change — see the [backend implementation](https://github.com/microsoft/openvmm/blob/main/openhcl/underhill_core/src/threadpool_vm_task_backend.rs) for details. ## Blocking scenarios Because the executor is cooperative and single-threaded per VP, several situations can stall all tasks on a VP. ```admonish note In OpenVMM (not OpenHCL), device workers and VP execution run on separate threads. Code authors *should not* issue a blocking wait inside an async task. This is a general Rust rule, that emphatically applies in OpenVMM and OpenHCL. ``` ### VTL0 guest execution When there are no VTL2 tasks that can make forward progress, the VP thread enters a lower VTL via an ioctl (`hcl_return_to_lower_vtl`). The thread is in the kernel until a VM exit returns control to VTL2. **IO completions still wake VTL2.** OpenHCL registers the io_uring fd with the HCL kernel module via `set_poll_file`. When an io_uring completion fires (e.g., a disk I/O completes via `disk_blockdevice`), the kernel cancels the VM run, returning the thread to VTL2. This applies to any async work that completes through io_uring — not just disk I/O. For device interrupts that don't go through io_uring (e.g., the physical NVMe driver in `disk_nvme` receives interrupts via an eventfd), the eventfd is registered with io_uring as a poll operation, so it also triggers the cancel path. If the VTL0 guest traps into the hypervisor (e.g., for a hypercall or MMIO (Memory-Mapped I/O) access that the hypervisor handles on behalf of the root), the VP is in the hypervisor — not in VTL0 usermode — and the io_uring cancel mechanism does not apply. The VP remains in the hypervisor until the intercept completes. I/O completions may still occur but do not preempt the intercept. ### Kernel syscall blocking If a device worker issues a blocking syscall (e.g., a disk backend falls back to synchronous I/O), the thread is in the kernel. No `.await` yield is possible because the thread itself is blocked. VTL0 cannot execute either. New device backends should use the built-in io_uring primitives in OpenHCL to create workers for those blocking tasks. If that is impossible, that device will need to spawn a new thread. This should be rare, and you should discuss your rationale with the community before implementing something that way. ### Hypervisor intercepts When VTL2 triggers an operation that requires root partition handling — for example, an MMIO (Memory-Mapped I/O) write that traps to the hypervisor — the VP can be stopped in the hypervisor while the root processes the intercept. Both VTL2 and VTL0 are stalled on that VP. This is not a software-level problem in OpenHCL — it's an artifact of the hypervisor/root architecture. The VP physically cannot execute until the root completes the intercept. ### VTL2 blocking VTL0 The reverse of VTL0 blocking: while the VP thread is running VTL2 tasks, VTL0 cannot execute on that VP. A long burst of VTL2 device work (e.g., processing a large batch of StorVSP completions) delays guest execution. During large batches, ensure the code contains `.await` points so other VTL2 tasks (and ultimately VTL0 execution) can make progress; there is no automatic preemption. ## Timeline A VP thread's execution over time: ```text RUNNING STALLED RUNNING BLOCKED ┌──────────────┬─────────────────┬──────────────┬──────────────┐ │▓▓▓▓▓▓▓▓▓▓▓▓▓▓│░░░░░░░░░░░░░░░░░│▓▓▓▓▓▓▓▓▓▓▓▓▓▓│██████████████│ │ VTL2 tasks │ VTL0 guest │ VTL2 tasks │ kernel │ │ │ │ │ syscall │ │ storvsp, │ all VTL2 │ storvsp, │ ALL VTL2 │ │ netvsp, │ futures pending│ netvsp, │ tasks wait │ │ relay │ (exit → wake) │ relay │ │ └──────────────┴─────────────────┴──────────────┴──────────────┘ ▓ = VTL2 work active ░ = VTL0 running █ = kernel blocked ``` Each segment is mutually exclusive — only one of VTL2 tasks, VTL0 guest, or kernel work can run at any instant on a given VP thread. ## No work stealing The OpenHCL threadpool does not implement work stealing. Targeted tasks always run on their target VP's thread. For example: If VP 2's thread is blocked in VTL0, a StorVSP worker targeted at VP 2 cannot be picked up by VP 3's thread. Tasks without a specific `target_vp` run on the thread that wakes them — which is not the same as work stealing. Similarly, tasks with `run_on_target = false` may run on a thread other than their target VP's, but their IOs are still directed to the target VP's io-uring. ## Sidecar changes On x64 non-isolated VMs, the [sidecar kernel](sidecar.md) splits VP execution from device work. Most VPs run in the sidecar — a minimal kernel that handles VTL0 entry/exit without Linux. Only a few CPUs (typically one per NUMA node) boot into Linux. This is to amortize the CPU startup cost until it becomes necessary. Device workers run only on the CPUs that are onlined in the OpenHCL Linux kernel. Device workers that are CPU agnostic can run in this more limited set of Linux CPUs. When a sidecar VP hits an intercept that requires VTL2 processing (the first handled VM exit), the sidecar CPU is hot-plugged into Linux. From that point, the VP's device workers can run on its own thread instead. ## Impact on device design When writing device backends, keep these rules in mind: 1. **Never block synchronously** in a device worker on a VP thread. Use async I/O (io_uring) or spawn a helper thread for blocking work. No VMBus devices used in OpenHCL currently spawn helper threads. Instead, subsystems that need blocking (GET, VMGS) run on their own dedicated threads outside the VP threadpool. 2. **Sidecar VPs run remotely first.** Beware doing work on a targeted VP early in boot. If a device worker is targeted at a sidecar VP, it initially runs on the base CPU, not the target CPU. This can cause contention, but more importantly: work that must occur on certain VP will cause that VP to exit the sidecar and enter Linux. This hot-plug has non-trivial latency; sidecar exists specifically to defer this cost until the VP actually needs VTL2 processing. 3. **Use `TaskControl` for worker lifecycle.** Device workers should implement [`AsyncRun`](https://openvmm.dev/rustdoc/linux/task_control/trait.AsyncRun.html) and be managed via [`TaskControl`](https://openvmm.dev/rustdoc/linux/task_control/struct.TaskControl.html), which provides start/stop/inspect integration. (This doesn't really apply to the CPU scheduling model, but is general good advice for writing device backends). --- ### Guide Src Reference Architecture Openhcl Igvm (`Guide/src/reference/architecture/openhcl/igvm.md`) # IGVM Image The Independent Guest Virtual Machine (IGVM) format describes an isolated VM's initial state. OpenHCL is delivered as an IGVM image. > **Note:** For more details on the IGVM specification, see the [IGVM repository](https://github.com/microsoft/igvm). ## Purpose The IGVM file serves as the firmware image for the OpenHCL paravisor. It instructs the host VMM on how to: 1. Load the OpenHCL components into VTL2 memory. 2. Place them at required physical addresses in a defined order so launch measurements are reproducible. 3. Pass initial configuration data to the paravisor. ## IGVM Image Contents An OpenHCL IGVM image bundles the following artifacts: - **Boot Shim (`openhcl_boot`):** The entry point for VTL2 execution. - **Linux Kernel:** The operating system kernel. - **Sidecar Kernel (x86_64):** The lightweight kernel for APs. - **Initial Ramdisk (initrd):** The root filesystem containing userspace binaries such as `underhill_init` and `openvmm_hcl`. - **Memory Layout:** Directives specifying where each component should be loaded in memory. - **Measurements:** Data used by the platform to confirm that the expected image was loaded. - **Configuration:** Measured build-time parameters and permitted launch-time data such as CPU topology and device settings. See [`ParavisorMeasuredVtl0Config`](https://openvmm.dev/rustdoc/linux/loader_defs/paravisor/struct.ParavisorMeasuredVtl0Config.html) and [`ParavisorMeasuredVtl2Config`](https://openvmm.dev/rustdoc/linux/loader_defs/paravisor/struct.ParavisorMeasuredVtl2Config.html) for examples of measured configuration. ## Build Process The IGVM artifact is generated by the OpenHCL build system. See [Building OpenHCL](../../../dev_guide/getting_started/build_openhcl.md) for build instructions. The [`igvmfilegen`](../../../dev_guide/dev_tools/igvmfilegen.md) page documents manifest generation, image inspection, launch identity, and CoRIM signing. --- ### Guide Src Reference Architecture Openhcl Openvmm Hcl (`Guide/src/reference/architecture/openhcl/openvmm_hcl.md`) # openvmm_hcl `openvmm_hcl` is the main user-mode VMM and management process inside the OpenHCL Linux VTL2 environment. ## Position in the boot flow OpenHCL reaches `openvmm_hcl` after several earlier stages: ```text host loads OpenHCL IGVM into VTL2 `- openhcl_boot prepares memory, CPUs, and Linux boot data `- Linux kernel boots the initrd `- underhill_init prepares userspace `- exec /bin/openvmm_hcl ``` ## Startup responsibilities After Linux userspace starts, `openvmm_hcl` performs these broad tasks: 1. Read VTL2 configuration and topology from the device tree and Linux interfaces established by the boot loader. 2. Initialize host communication, diagnostics, tracing, and servicing control. 3. Resolve the device and backend resources compiled into the image. 4. Start the VM worker that owns the VTL0 processor and device data path. 5. Start separate workers for devices whose isolation or lifecycle requires a process boundary. 6. Coordinate pause, resume, save, restore, servicing, restart, and shutdown. The process remains the policy and control-plane owner while worker processes perform high-volume VM and device work. ## Worker model The VM worker runs VTL0 virtual processors, handles exits, and coordinates the guest device model. Selected devices, including supported TPM configurations, can run in separate worker processes. OpenHCL communicates with those workers through typed channels rather than sharing arbitrary process state. This process model allows a worker to be restarted or serviced while the main paravisor process retains policy and host communication state. It also limits which process directly handles sensitive device operations. ## Trust boundaries OpenHCL does not trust the VTL0 guest. In a confidential VM, it also cannot blindly trust host-provided configuration. The measured IGVM and boot loader establish which runtime inputs are acceptable before `openvmm_hcl` starts. At runtime: - Guest protocol and device requests are parsed as untrusted input. - Confidential diagnostics omit or reject operations that could expose guest secrets. - Host-directed actions are constrained by the image's measured policy and isolation mode. - Repeated guest-triggerable failures use bounded or rate-limited diagnostics. Debug and confidential-debug images intentionally expose more diagnostic surface than production confidential images. Do not treat them as equivalent security configurations. ## Build and package The supported path builds the whole firmware image: ```bash cargo xflowey build-igvm x64 ``` The recipe determines the target, features, kernel, initrd contents, boot loader, sidecar, and manifest. See [Building OpenHCL][] for available recipes, artifact locations, and instructions for substituting a custom `openvmm_hcl` binary. [Building OpenHCL]: ../../../dev_guide/getting_started/build_openhcl.md ```admonish warning A standalone `openvmm_hcl` file is not a bootable product. It expects the OpenHCL kernel, initrd, device tree, memory layout, and measured configuration provided by a matching IGVM recipe. ``` The crate only runs on Linux. Non-Linux workspace builds contain an unsupported-platform stub. ## Diagnostics The OpenHCL diagnostics server exposes information and development operations owned by `openvmm_hcl` and its workers. Common sources include: - Kernel and OpenHCL logs through `kmsg` or configured serial output. - The inspect tree for build identity, control state, workers, devices, and tracing. - Core dumps and saved-state dumps where isolation policy permits them. - Performance, packet-capture, and memory-profile traces in supported builds. Use `ohcldiag-dev` interactively. Its CLI and inspect paths are explicitly unstable and confidential images restrict the available data. ## Failure and restart behavior A fatal failure in the main process ends the paravisor control plane. A worker failure can sometimes be diagnosed or restarted through the main process, depending on the worker and VM state. The `ohcldiag-dev restart` command targets the VM worker while keeping VTL0 running where the runtime supports that transition. For startup failures: 1. Capture the VTL2 kernel and OpenHCL log from the beginning of boot. 2. Verify that the IGVM recipe contains the expected custom binary and initrd. 3. Inspect `/etc/underhill-build-info.json` through a working diagnostic shell to confirm the running revision. 4. Check the device tree and command line produced by `openhcl_boot`. 5. Reproduce with an unmodified recipe before attributing the failure to a worker or host interface. ```admonish note title="See also" [OpenHCL boot flow](boot.md) describes the stages before this process starts. [Processes and components](processes.md) shows the complete process model. [ohcldiag-dev](../../openhcl/diag/ohcldiag_dev.md) documents the development diagnostic client. ``` --- --- METRICS --- - Files Extracted: 117 - Estimated Token Budget: ~125703 tokens - Canonical Reference: https://codewiki.google/github.com/microsoft/openvmm