hyperlight Documentation

Hyperlight is a lightweight Virtual Machine Manager (VMM) designed to be embedded within applications. It enables safe execution of untrusted code within...

4,247 stars Rust Rust 1 file · ~18,271 tokens
RAW Doc

Technical Documentation: hyperlight-dev/hyperlight

ℹ️ Provenance: Hybrid Fusion: hyperlight-dev/hyperlight (README + 10 In-Tree Chapters) · CodeWiki Reference · Recency: Active (< 180 days)

1. Project Overview & Quickstart (hyperlight-dev/hyperlight)

Status: Hyperlight is pre-1.0. The API may change between releases, and upgrading will sometimes require code changes.

Hyperlight lets you safely run untrusted code inside hypervisor-isolated micro VMs that spin up in milliseconds, with guest function calls completing in microseconds. You embed it as a library in your Rust application, hand it a guest binary, and call functions across the VM boundary as naturally as calling a local function. To minimize startup time and memory footprint, there's no guest kernel or OS. Guests are purpose-built using the Hyperlight guest library.

  • Supports KVM, MSHV, and Windows Hypervisor Platform
  • No kernel or OS in the VM. Guests are regular ELF binaries written in no_std Rust or C
  • Host and guest communicate through typed function calls
  • Guests are sandboxed by default with no access to the host filesystem, network, etc.

Example

Host - create a sandbox, register a host function, and call into the guest:

rust
// Build a sandbox from a guest binary, registering a host function the guest
// can call. In a real app that function might query a database, read a config,
// or call an external API. By default, guests can only print to the host.
let mut sandbox = SandboxBuilder::from_file(guest_path)
    .host_function("GetWeekday", || Ok("Monday".to_string()))
    .build()?;

// Call a function inside the VM
let greeting: String = sandbox.call("SayHello", "World".to_string())?;
println!("{greeting}"); // "Hello, World! Today is Monday."

Guest state persists across calls. Use snapshot() and restore() to save and reset VM memory. This avoids recreating the VM while ensuring each call starts from a clean state.

Guest (Rust) - declare host functions and expose guest functions with simple macros. Guests can also be written in C.

rust
#[host_function("GetWeekday")]
fn get_weekday() -> Result<String>;

#[guest_function("SayHello")]
fn say_hello(name: String) -> Result<String> {
    let weekday = get_weekday()?;
    Ok(format!("Hello, {name}! Today is {weekday}."))
}

To get started, see the Getting Started guide. For more details on writing guests, see How to build a Hyperlight guest binary.

When to use Hyperlight

Hyperlight is a good fit when you need to:

  • Run untrusted or third-party code with hypervisor-level isolation
  • Create and tear down sandboxes in milliseconds
  • Make guest function calls in microseconds
  • Embed sandboxed execution directly in your application
  • Build functions-as-a-service with hypervisor-level isolation
  • Reuse sandboxes efficiently with snapshot and restore

Hyperlight is not designed for:

  • General-purpose virtualization (use a full VMM instead)
  • Running full-blown Linux guest workloads that need syscalls, networking, or filesystem access

Getting started

See docs/getting-started.md for detailed prerequisites and platform-specific setup for:

  • Running Hyperlight
  • Building guests

Or skip setup entirely with a codespace:

Repository Structure

Directory Description
src/hyperlight_host Host library - creates and manages micro VMs
src/hyperlight_guest Core guest library - minimal building blocks for guest-host interaction
src/hyperlight_guest_bin Extended guest library - entry point, panic handler, heap, logging, exceptions
src/hyperlight_guest_capi C API wrapper around hyperlight_guest_bin for use via FFI
src/hyperlight_libc C standard library for guests, built from picolibc
src/hyperlight_guest_macro Macros for registering guest and host functions
src/hyperlight_guest_tracing Tracing support for guests
src/hyperlight_common Shared code used by both host and guest
src/hyperlight_component_macro Proc macros for WIT-based host/guest bindings
src/hyperlight_component_util Shared implementation for WIT binding generation
src/hyperlight_testing Shared test utilities
src/schema FlatBuffer schema definitions
src/trace_dump Tool for dumping and visualizing trace data
src/tests Test guest programs (Rust and C)
  • cargo-hyperlight - Cargo subcommand for building and scaffolding Hyperlight guests
  • hyperlight-wasm - Run WebAssembly modules inside Hyperlight micro VMs
  • hyperlight-js - Run JavaScript inside Hyperlight micro VMs
  • hyperlight-sandbox - Multi-backend sandboxing framework for running untrusted code with controlled host capabilities, with Python, .NET, and Rust SDKs
  • hyperlight-unikraft - Run Linux applications (Python, Node.js, Go, Rust, C/C++) on Hyperlight micro VMs using Unikraft as the guest kernel

Contributing

See CONTRIBUTING.md.

Community

2. In-Tree Documentation Chapters (hyperlight-dev/hyperlight)

File: README.md

Status: Hyperlight is pre-1.0. The API may change between releases, and upgrading will sometimes require code changes.

Hyperlight lets you safely run untrusted code inside hypervisor-isolated micro VMs that spin up in milliseconds, with guest function calls completing in microseconds. You embed it as a library in your Rust application, hand it a guest binary, and call functions across the VM boundary as naturally as calling a local function. To minimize startup time and memory footprint, there's no guest kernel or OS. Guests are purpose-built using the Hyperlight guest library.

  • Supports KVM, MSHV, and Windows Hypervisor Platform
  • No kernel or OS in the VM. Guests are regular ELF binaries written in no_std Rust or C
  • Host and guest communicate through typed function calls
  • Guests are sandboxed by default with no access to the host filesystem, network, etc.

Example

Host - create a sandbox, register a host function, and call into the guest:

rust
// Build a sandbox from a guest binary, registering a host function the guest
// can call. In a real app that function might query a database, read a config,
// or call an external API. By default, guests can only print to the host.
let mut sandbox = SandboxBuilder::from_file(guest_path)
    .host_function("GetWeekday", || Ok("Monday".to_string()))
    .build()?;

// Call a function inside the VM
let greeting: String = sandbox.call("SayHello", "World".to_string())?;
println!("{greeting}"); // "Hello, World! Today is Monday."

Guest state persists across calls. Use snapshot() and restore() to save and reset VM memory. This avoids recreating the VM while ensuring each call starts from a clean state.

Guest (Rust) - declare host functions and expose guest functions with simple macros. Guests can also be written in C.

rust
#[host_function("GetWeekday")]
fn get_weekday() -> Result<String>;

#[guest_function("SayHello")]
fn say_hello(name: String) -> Result<String> {
    let weekday = get_weekday()?;
    Ok(format!("Hello, {name}! Today is {weekday}."))
}

To get started, see the Getting Started guide. For more details on writing guests, see How to build a Hyperlight guest binary.

When to use Hyperlight

Hyperlight is a good fit when you need to:

  • Run untrusted or third-party code with hypervisor-level isolation
  • Create and tear down sandboxes in milliseconds
  • Make guest function calls in microseconds
  • Embed sandboxed execution directly in your application
  • Build functions-as-a-service with hypervisor-level isolation
  • Reuse sandboxes efficiently with snapshot and restore

Hyperlight is not designed for:

  • General-purpose virtualization (use a full VMM instead)
  • Running full-blown Linux guest workloads that need syscalls, networking, or filesystem access

Getting started

See docs/getting-started.md for detailed prerequisites and platform-specific setup for:

  • Running Hyperlight
  • Building guests

Or skip setup entirely with a codespace:

Repository Structure

Directory Description
src/hyperlight_host Host library - creates and manages micro VMs
src/hyperlight_guest Core guest library - minimal building blocks for guest-host interaction
src/hyperlight_guest_bin Extended guest library - entry point, panic handler, heap, logging, exceptions
src/hyperlight_guest_capi C API wrapper around hyperlight_guest_bin for use via FFI
src/hyperlight_libc C standard library for guests, built from picolibc
src/hyperlight_guest_macro Macros for registering guest and host functions
src/hyperlight_guest_tracing Tracing support for guests
src/hyperlight_common Shared code used by both host and guest
src/hyperlight_component_macro Proc macros for WIT-based host/guest bindings
src/hyperlight_component_util Shared implementation for WIT binding generation
src/hyperlight_testing Shared test utilities
src/schema FlatBuffer schema definitions
src/trace_dump Tool for dumping and visualizing trace data
src/tests Test guest programs (Rust and C)
  • cargo-hyperlight - Cargo subcommand for building and scaffolding Hyperlight guests
  • hyperlight-wasm - Run WebAssembly modules inside Hyperlight micro VMs
  • hyperlight-js - Run JavaScript inside Hyperlight micro VMs
  • hyperlight-sandbox - Multi-backend sandboxing framework for running untrusted code with controlled host capabilities, with Python, .NET, and Rust SDKs
  • hyperlight-unikraft - Run Linux applications (Python, Node.js, Go, Rust, C/C++) on Hyperlight micro VMs using Unikraft as the guest kernel

Contributing

See CONTRIBUTING.md.

Community

File: docs/benchmarking-hyperlight.md

Benchmark Notes

Hyperlight uses the Criterion framework to run and analyze benchmarks. A benefit to this framework is that it doesn't require the nightly toolchain.

When Benchmarks are run

  1. Daily (scheduled)

    • Benchmarks run daily via DailyBenchmarks.yml, comparing results against the previous day's run. Results are stored as workflow artifacts with 90-day retention.
    text
    sandboxes/create_sandbox
                    time:   [33.803 ms 34.740 ms 35.763 ms]
                    change: [+0.7173% +3.7017% +7.1346%] (p = 0.03 < 0.05)
                    Change within noise threshold.*

  2. For each release

    • For each release, benchmarks are run as part of the release pipeline in CreateRelease.yml, which invokes dep_benchmarks.yml. These benchmark results are compared to the previous release, and are uploaded as part of the "Release assets" on the GitHub release page.

Currently, benchmarks are run on windows, linux-kvm (ubuntu), and linux-hyperv (mariner). Only release builds are benchmarked, not debug.

Criterion artifacts

When running cargo bench -- --save-baseline my_baseline, criterion runs all benchmarks defined in src/hyperlight_host/benches/, prints the results to the stdout, as well as produces several artifacts. All artifacts can be found in target/criterion/. For each benchmarking group, for each benchmark, a subfolder with the name of the benchmark is created. This folder in turn contains folders my_baseline, new and report. When running cargo bench, criterion always creates new and report, which always contains the most recent benchmark result and html report, but because we provided the --save-baseline flag, we also have a my_baseline folder, which is an exact copy of new. Moreover, if this my_baseline folder already existed before we ran cargo bench -- --save-baseline my_baseline, criterion would also compare the benchmark results with the old my_baseline folder, and then overwrite the folder.

The first time we run cargo bench -- --save-baseline my_baseline (starting with a clean project), we get the following structure.

text
target/criterion/
|-- report
`-- sandboxes
    |-- create_sandbox
    |   |-- my_baseline
    |   |-- new
    |   `-- report
    |-- create_sandbox_and_call_context
    |   |-- my_baseline
    |   |-- new
    |   `-- report
    `-- report

If we run the exact same command again, we get

text
target/criterion/
|-- report
`-- sandboxes
    |-- create_sandbox
    |   |-- change
    |   |-- my_baseline
    |   |-- new
    |   `-- report
    |-- create_sandbox_and_call_context
    |   |-- change
    |   |-- my_baseline
    |   |-- new
    |   `-- report
    `-- report

Note that it overwrote the previous my_baseline with the new result. But notably, there is a new change folder, which contains the benchmarking difference between the two runs. In addition, on stdout you'll also find a comparison to our previous my_baseline run.

text
time:   [40.434 ms 40.777 ms 41.166 ms]
                        change: [+0.0506% +1.1399% +2.2775%] (p = 0.06 > 0.05)
                        No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)

Note that Criterion does not differ between release and debug/dev benchmark results, so it's up to the developer to make sure baselines of the same config are compared.

Running benchmarks locally

Use just bench to run benchmarks with release builds (the only supported configuration). Comparing local benchmark results to GitHub-saved benchmarks doesn't make much sense, since you'd be using different hardware, but you can use just bench-download os hypervisor cpu_vendor [tag] to download and extract the GitHub release benchmarks to the correct folder. You can then run just bench-ci main to compare to (and overwrite) the previous release benchmarks. Note that main is the name of the baselines stored in GitHub.

Important: The just bench command uses release builds by default to ensure meaningful performance measurements. For profiling purposes, you can compile benchmarks with debug symbols by running cargo bench directly.

File: docs/github-labels.md

**Pull Requests (PRs)**

We use GitHub labels to categorize PRs. Before a PR can be merged, it must be assigned one of the following kind/ labels:

  • kind/bugfix - For PRs that fix bugs.
  • kind/dependencies - For PRs that update dependencies or related components.
  • kind/enhancement - For PRs that introduce new features or improve existing functionality. This label also applies to improvements in documentation, testing, and similar areas. Any changes must be backward-compatible.
  • kind/refactor - For PRs that restructure or remove code without adding new functionality. This label also applies to changes that affect user-facing APIs.

Review readiness

We use the ready-for-review label to signal that a PR is waiting for a (re-)review:

  • Add ready-for-review when you open a PR that is ready for review, or when a PR is ready for re-review (for example, once you have addressed requested changes and re-requested review).
  • The label is removed automatically by the ready-for-review label automation (the collect and manage workflows) once the PR is no longer awaiting that review, specifically when any of the following become true:
    • the PR is closed or merged,
    • the PR is converted to a draft,
    • the PR has two or more approvals, or
    • the PR has two or more change requests.

You only ever need to add the label; removal is fully automated.

**Issues**

Issues are categorized using the following three GitHub types (not GitHub labels):

  • bug - Reports an unexpected problem or incorrect behavior.
  • design - Relates to design considerations or decisions.
  • enhancement - Suggests a new feature, improvement, or idea.

To track the lifecycle of issues, we also use GitHub labels:

  • lifecycle/needs-review - A temporary label indicating that the issue has not yet been reviewed.
  • lifecycle/confirmed - Confirms the issue’s validity:
    • If the issue type is bug, the bug has been verified.
    • If the issue type is enhancement, the proposal is considered reasonable but does not guarantee implementation.
    • This label does not indicate when or if the fix or enhancement will be implemented.
  • lifecycle/needs-info - The issue requires additional information from the original poster (OP).
  • lifecycle/blocked - The issue is blocked by another issue or external factor.

The following labels should be applied to issues prior to closing, indicating the resolution status of the issue:

  • lifecycle/duplicate - The issue is a duplicate of another issue.
  • lifecycle/fixed - The issue has been resolved.
  • lifecycle/not-a-bug - The issue is not considered a bug, and no further action is needed.
  • lifecycle/wont-fix - The issue will not be fixed.

In addition to lifecycle labels, we use the following labels to further categorize issues:

  • good-first-issue - The issue is suitable for new contributors or those looking for a simple task to start with.
  • help-wanted - The issue is a request for help or assistance.
  • question - The issue is a question or request for information.
  • release-blocker - Critical issues that must be resolved before the next release can be made. The presence of this label on any open issue will prevent releases being created by the release workflow

**Issues & PRs**

In addition to kind/* labels, we use optional area/* labels to specify the focus of a PR or issue. These labels are purely for categorization, and are not mandatory.

  • area/API - Related to the API or public interface.
  • area/ci-periodics - Applied to issues that track failures from scheduled (periodic) CI jobs. These issues are opened automatically by the CI failure notifier (dev/notify-ci-failure.sh).
  • area/ci-periodics-aarch64 - De-duplication label for failures from the daily aarch64 workflow (.github/workflows/DailyArm64.yml). It keeps those failures on their own dedicated, release-blocking issue instead of sharing the general area/ci-periodics issue.
  • area/dependencies - Concerns dependencies or related components. This label is different from kind/dependencies, which should only used for PRs.
  • area/documentation - Related to documentation updates or improvements.
  • area/infrastructure - Concerns infrastructure rather than core functionality.
  • area/performance - Addresses performance.
  • area/security - Involves security-related changes or fixes.
  • area/testing - Related to tests or testing infrastructure.

Workflow labels

Some labels change CI behaviour on a PR rather than categorizing it:

  • regen-goldens - Switches the snapshot golden verify job into regenerate mode. A PR that intentionally changes the snapshot format and bumps GOLDENS_VERSION carries this label so the verify job generates the goldens from the branch and runs them back through the branch loader, rather than pulling a published tag set that does not exist yet. See snapshot-versioning.md.

Notes

This document is a work in progress and may be updated as needed. The labels and categories are subject to change based on the evolving needs of the project and community feedback.

File: docs/glossary.md

Glossary

Hyperlight

Hyperlight refers to the Hyperlight Project and not a specific component. Hyperlight is intended to be used as a library to embed hypervisor-isolated execution support inside a host application.

Host Application

This is an application that consumes the Hyperlight library, in order to execute code in an hypervisor-isolated environment.

Host

Host is the machine on which the host application is running. A host could be a bare metal or virtual machine, when the host is a virtual machine, the nested virtualization is required to run Hyperlight.

Hypervisor

Hypervisor is the software responsible for creating isolated micro virtual machines, as well as executing guests inside of those micro virtual machines. Hyperlight has drivers for the following hypervisors: Hyper-V on Windows, Hyper-V on Linux, and KVM.

Driver

Hyperlight supports executing workloads on particular hypervisors through drivers. Each supported hypervisor has its own driver to manage interacting with that hypervisor.

Hyper-V

Hyper-V is a hypervisor capable of creating and executing isolated micro virtual machines on both Windows and Linux. On Linux, Hyper-V is sometimes referred to as MSHV (Microsoft Hypervisor).

KVM

Kernel-based Virtual Machine (KVM) is a hypervisor capable of creating and executing isolated micro virtual machines on Linux.

MSHV

MSHV stands for Microsoft Hypervisor and is the name commonly used for Hyper-V when the hypervisor is running Linux dom0 (as opposed to Windows dom0).

Guest

A guest is a standalone executable binary that is executed inside a hypervisor micro virtual machine. By having purpose-fit guests binaries, as opposed to running a full operating system, Hyperlight achieves low-latency startup times of workloads, since it doesn't need to first boot an entire operating system before executing the workload.

The interface that a guest must implement is specific to the associated host and the type of workloads that it may be specialized for executing, such as WebAssembly Modules (Wasm), or a specific language.

Micro Virtual Machine

A micro virtual machine is an execution environment managed by a hypervisor that isolates a guest from the host. A hypervisor prevents the guest from directly accessing the host's resources, such as memory, filesystem, devices, memory or CPU.

We use the term Micro Virtual Machine as the VMs are very lightweight compared to traditional VMs, containing no operating system or other unnecessary components. The goal is to provide a minimal environment for executing workloads with low latency and high density. However the isolation provided by the hypervisor is the same as that of a traditional VM.

Workload

A workload is the code that the host application wants to execute in an isolated micro virtual machine.

Sandbox

A Sandbox is the abstraction used in Hyperlight to represent the isolated environment in which a workload is executed. A sandbox is used to create, configure, execute and destroy a micro virtual machine that runs a guest workload. Sandboxes are created by the host application using the Hyperlight host library.

File: docs/how-to-build-a-hyperlight-guest-binary.md

Building a Hyperlight guest binary

This document explains how to build a binary to be used as a Hyperlight guest.

A Hyperlight guest is a regular ELF binary built for a custom target. It runs
without an operating system, exposes functions that the host can call, and may
call functions registered by the host. Builds use cargo hyperlight build,
which sets up the target, sysroot, and environment variables.

Install cargo-hyperlight
with:

sh
cargo install --locked cargo-hyperlight

To scaffold a working host plus guest project, run:

sh
cargo hyperlight new my-project

Pass --no-host to generate only a guest crate, or --no-guest to generate
only a host crate.

The rest of this document explains what the generated guest contains and how to
build one by hand.

Rust guest binary

Minimal guest

A minimal Rust guest depends only on hyperlight-guest-bin and uses its
attribute macros to register guest functions and declare host functions.

Cargo.toml:

toml
[package]
name = "my-guest"
version = "0.1.0"
edition = "2024"

[dependencies]
hyperlight-guest-bin = "*"  # use the latest version from crates.io

src/main.rs:

rust
#![no_std]
#![no_main]
extern crate alloc;

use alloc::string::String;

use hyperlight_guest_bin::error::Result;
use hyperlight_guest_bin::{guest_function, host_function};

// Declare a host function the guest can call. The string is the name the
// host registered it under. If omitted, the Rust function name is used.
#[host_function("GetWeekday")]
fn get_weekday() -> Result<String>;

// Register a guest function the host can call.
#[guest_function("SayHello")]
fn say_hello(name: String) -> Result<String> {
    let weekday = get_weekday()?;
    Ok(alloc::format!("Hello, {name}! Today is {weekday}."))
}

Build with:

sh
cargo hyperlight build

The resulting binary in target/x86_64-hyperlight-none//my-guest is
what the host loads with GuestBinary::FilePath(...).

What the macros generate

  • #[guest_function] registers the function with the guest runtime so the host
    can call it by name. Argument and return types must be supported parameter
    and return types, or Result<T, HyperlightGuestError> wrapping one.
  • #[host_function] turns an extern-style signature into a stub that
    marshals arguments to the host and returns the host's reply.
  • The runtime provides a default hyperlight_main entry point and a default
    dispatch function. You do not need to write either one for a typical guest.

Required attributes

  • #![no_std] because there is no operating system in the guest.
  • #![no_main] because the entry point is hyperlight_main, not main.
  • extern crate alloc to use Vec, String, and other heap types.

Advanced: manual entry point and dispatch

The macros are optional. A guest can define the underlying symbols directly,
which is useful for advanced setup work or custom dispatch logic (for example
the WIT-based guests in src/tests/rust_guests/witguest).

The host expects two guest symbols:

  • pub extern "C" fn hyperlight_main() runs once at guest startup. Use it to
    register functions or initialize global state.
  • pub extern "Rust" fn guest_dispatch_function(function_call: FunctionCall) -> Result<Vec, HyperlightGuestError>
    is invoked when the host calls a function name that is not registered.

hyperlight-guest-bin exposes #[main] and #[dispatch] macros that generate
these symbols from a regular Rust function, so most "advanced" guests still use
the macros rather than writing the raw extern "C" items themselves.

If you mix the manual form with #[guest_function], registrations from the
macro still happen automatically. Your hyperlight_main only needs to do
whatever extra setup the macros do not cover.

Troubleshooting

"duplicate lang item `panic_impl`" error

This error means the standard library's panic handler is being linked
alongside hyperlight_guest_bin's. To fix:

  1. Ensure your crate has #![no_std] at the top of main.rs.
  2. Confirm any transitive dependency on hyperlight-common uses
    default-features = false.
  3. Run cargo clean to clear stale artifacts.
  4. Build with cargo hyperlight build, not cargo build.

Build errors with dependencies

If you see errors building dependencies (such as serde), make sure you are
using cargo hyperlight build. It sets up the environment variables and
sysroot needed for the custom Hyperlight target.

C guest binary

For the binary written in C, the generated C bindings can be downloaded from the
latest release page that contain: the hyperlight_guest.h header and the
C API library.
The hyperlight_guest.h header contains the corresponding APIs to register
guest functions and call host functions from within the guest.

See src/tests/c_guests/c_simpleguest/main.c
for a complete example.

Version compatibility

Guest binaries built with hyperlight-guest-bin automatically embed the crate
version in an ELF note section (.note.hyperlight-version). When the host
loads a guest binary, it checks this version and rejects the binary if it does
not match the host's version of hyperlight-host.

Hyperlight currently provides no backwards compatibility guarantees for guest
binaries — the guest and host crate versions must match exactly. If you see a
GuestBinVersionMismatch error, rebuild the guest binary with a matching
version of hyperlight-guest-bin.

File: docs/how-to-debug-a-hyperlight-guest.md

How to debug a Hyperlight guest using gdb or lldb

Hyperlight supports gdb debugging of a guest running inside a Hyperlight sandbox on Linux or Windows.
When Hyperlight is compiled with the gdb feature enabled, a Hyperlight sandbox can be configured
to start listening for a gdb connection.

Supported features

The Hyperlight gdb feature enables guest debugging to:

  • stop at an entry point breakpoint which is automatically set by Hyperlight
  • add and remove HW breakpoints (maximum 4 set breakpoints at a time)
  • add and remove SW breakpoints
  • read and write registers
  • read and write addresses
  • step/continue
  • get code offset from target
  • stop when a crash occurs and only allow read access to the guest memory and registers

Expected behavior

Below is a list describing some cases of expected behavior from a gdb debug
session of a guest binary running inside a Hyperlight sandbox.

  • when the gdb feature is enabled and the sandbox builder is given a debug
    port, the created sandbox will wait for a gdb client to connect on the
    configured port
  • when the gdb client attaches, the guest vCPU is expected to be stopped at the
    entry point
  • if a gdb client disconnects unexpectedly, the debug session will be closed and
    the guest will continue executing disregarding any prior breakpoints
  • if multiple sandbox instances are created, each instance will have its own
    gdb thread listening on the configured port
  • if two sandbox instances are created with the same debug port, the second
    instance logs an error and the gdb thread will not be created, but the sandbox
    will continue to run without gdb debugging
  • when a crash happens, the debugger session remains active, and the guest
    vCPU is stopped, allowing the gdb client to inspect the state of the guest.
    The debug target will refuse any resume, step actions and write operations to
    the guest memory and registers until the gdb client disconnects or the sandbox is stopped.

Example

Sandbox configuration

The guest-debugging example in Hyperlight demonstrates how to configure a Hyperlight
sandbox to listen for a gdb client on a specific port.

CLI Gdb configuration

One can use a gdb config file to provide the symbols and desired configuration.

The below contents of the .gdbinit file can be used to provide a basic configuration
to gdb startup.

gdb
# Path to symbols
file path/to/symbols.elf
# The port on which Hyperlight listens for a connection
target remote :8080
set disassembly-flavor intel
set disassemble-next-line on
enable pretty-printer
layout src

One can find more information about the .gdbinit file at gdbinit(5).

End to end example

Using the example mentioned at Sandbox configuration
one can run the below commands to debug the guest binary:

bash
# Terminal 1
$ cargo run --example guest-debugging --features gdb
bash
# Terminal 2
$ cat .gdbinit
file src/tests/rust_guests/bin/debug/simpleguest
target remote :8080
set disassembly-flavor intel
set disassemble-next-line on
enable pretty-printer
layout src

$ gdb

Using VSCode to debug a Hyperlight guest

To replicate the above behavior using VSCode follow the below steps:

  • To use gdb:
    1. install the gdb package on the host machine
    2. install the C/C++ Extension Pack extension in VSCode to add debugging capabilities
  • To use lldb:
    1. install lldb on the host machine
    2. install the CodeLLDB extension in VSCode to add debugging capabilities
  • create a .vscode/launch.json file in the project directory with the below content:
    json
    {
    "version": "0.2.0",
    "configurations": [
        {
            "name": "LLDB",
            "type": "lldb",
            "request": "launch",
            "targetCreateCommands": ["target create ${workspaceFolder}/src/tests/rust_guests/bin/debug/simpleguest"],
            "processCreateCommands": ["gdb-remote localhost:8080"]
        },
        {
            "name": "GDB",
            "type": "cppdbg",
            "request": "launch",
            "program": "${workspaceFolder}/src/tests/rust_guests/bin/debug/simpleguest",
            "args": [],
            "stopAtEntry": true,
            "hardwareBreakpoints": {"require": false, "limit": 4},
            "cwd": "${workspaceFolder}",
            "environment": [],
            "externalConsole": false,
            "MIMode": "gdb",
            "miDebuggerPath": "/usr/bin/gdb",
            "miDebuggerServerAddress": "localhost:8080",
            "setupCommands": [
                {
                    "description": "Enable pretty-printing for gdb",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                },
                {
                    "description": "Set Disassembly Flavor to Intel",
                    "text": "-gdb-set disassembly-flavor intel",
                    "ignoreFailures": true
                }
            ]
        }
    ]
}
  • in Run and Debug tab, select either GDB or LLDB configuration and click on the Run
    button to start the debugging session.
    The debugger will connect to the Hyperlight sandbox and the guest vCPU will
    stop at the entry point.

How it works

The gdb feature is designed to work like a Request - Response protocol between
a thread that accepts commands from a gdb client and main thread of the sandbox.

All the functionality is implemented on the hypervisor side so it has access to
the shared memory and the vCPU.

The gdb thread uses the gdbstub crate to handle the communication with the gdb client.
When the gdb client requests one of the supported features mentioned above, a request
is sent over the communication channel to the main thread for the sandbox
to resolve.

Below is a sequence diagram that shows the interaction between the entities
involved in the gdb debugging of a Hyperlight guest running inside a KVM or MSHV sandbox.

text
/* Detailed source-code truncated for AI context efficiency. */

Dumping the guest state to an ELF core dump

When a guest crashes because of an unknown VmExit or unhandled exception, the vCPU state can be optionally dumped to an ELF core dump file.
This can be used to inspect the state of the guest at the time of the crash.

To make Hyperlight dump the state of the vCPU (general purpose registers, registers) to an ELF core dump file, enable the crashdump feature and run.
The feature enables the creation of core dump files for both debug and release builds of Hyperlight hosts.
By default, Hyperlight places the core dumps in the temporary directory (platform specific).
To change this, use the HYPERLIGHT_CORE_DUMP_DIR environment variable to specify a directory.
The name and location of the dump file will be printed to the console and logged as an error message.

NOTE: If the directory provided by HYPERLIGHT_CORE_DUMP_DIR does not exist, Hyperlight places the file in the temporary directory.
NOTE: By enabling the crashdump feature, you instruct Hyperlight to create core dump files for all sandboxes when an unhandled crash occurs.
To selectively disable this feature for a specific sandbox, call guest_core_dump(false) on the SandboxBuilder.

rust
let sandbox = SandboxBuilder::from_file(guest_path)
        .guest_core_dump(false) // Disable core dump for this sandbox
        .build()?;

Creating a dump on demand

You can also create a core dump of the current state of the guest on demand by calling the generate_crashdump method on the InitializedMultiUseSandbox instance. This can be useful for debugging issues in the guest that do not cause crashes (e.g., a guest function that does not return).

This is only available when the crashdump feature is enabled and then only if the sandbox
is also configured to allow core dumps (which is the default behavior).

Example

Attach to your running process with gdb and call this function:

shell
sudo gdb -p 
(gdb) info threads
# find the thread that is running the guest function you want to debug
(gdb) thread 
# switch to the frame where you have access to your MultiUseSandbox instance
(gdb) backtrace
(gdb) frame <frame_number>
# get the pointer to your MultiUseSandbox instance
# Get the sandbox pointer
(gdb) print sandbox
# Call the crashdump function with the pointer
    # Call the crashdump function 
call sandbox.generate_crashdump()

The crashdump should be available /tmp or in the crash dump directory (see HYPERLIGHT_CORE_DUMP_DIR env var). To make this process easier, you can also create a gdb script that automates these steps. You can find an example script here. This script will try and generate a crashdump for every active thread except thread 1 , it assumes that the variable sandbox exists in frame 15 on every thread. You can edit it to fit your needs. Then use it like this:

shell
(gdb) source scripts/dump_all_sandboxes.gdb
(gdb) dump_all_sandboxes

Inspecting the core dump

After the core dump has been created, to inspect the state of the guest, load the core dump file using gdb or lldb.
NOTE: This feature has been tested with version 15.0 of gdb and version 17 of lldb, earlier versions may not work, it is recommended to use these versions or later.

Using gdb

Load the core dump alongside the guest binary that was running when the crash occurred:

bash
gdb <guest-binary> -c <core-dump-file>

For example:

bash
gdb src/tests/rust_guests/bin/debug/simpleguest -c /tmp/hl_dumps/hl_core_20260225_T165358.517.elf

Common commands for inspecting the dump:

gdb
# View all general-purpose registers (rip, rsp, rflags, etc.)
(gdb) info registers

# Disassemble around the crash site
(gdb) x/10i $rip

# View the stack
(gdb) x/16xg $rsp

# Backtrace (requires debug info in guest binary)
(gdb) bt

# List all memory regions in the dump (snapshot, scratch, mapped regions)
(gdb) info files

# Read memory at a specific address
(gdb) x/s <address>        # null-terminated string
(gdb) x/32xb <address>     # 32 bytes in hex

See the crashdump example (cargo run --example crashdump --features crashdump)
for a runnable demonstration of both automatic and on-demand crash dumps.

Using VSCode

To do this in vscode, the following configuration can be used to add debug configurations:

vscode
{
    "version": "0.2.0",
    "inputs": [
        {
            "id": "core_dump",
            "type": "promptString",
            "description": "Path to the core dump file",
        },
        {
            "id": "program",
            "type": "promptString",
            "description": "Path to the program to debug",
        }
    ],
    "configurations": [
        {
            "name": "[GDB] Load core dump file",
            "type": "cppdbg",
            "request": "launch",
            "program": "${input:program}",
            "coreDumpPath": "${input:core_dump}",
            "cwd": "${workspaceFolder}",
            "MIMode": "gdb",
            "externalConsole": false,
            "miDebuggerPath": "/usr/bin/gdb",
            "setupCommands": [
            {
                "description": "Enable pretty-printing for gdb",
                "text": "-enable-pretty-printing",
                "ignoreFailures": true
            },
            {
                "description": "Set Disassembly Flavor to Intel",
                "text": "-gdb-set disassembly-flavor intel",
                "ignoreFailures": true
            }
            ]
        },
        {
        "name": "[LLDB] Load core dump file",
        "type": "lldb",
        "request": "launch",
        "stopOnEntry": true,
        "processCreateCommands": [],
        "targetCreateCommands": [
            "target create -c ${input:core_dump} ${input:program}",
        ],
        },
    ]
}

NOTE: The CodeLldb debug session does not stop after launching. To see the code, stack frames and registers you need to
press the pause button. This is a known issue with the CodeLldb extension #1245.
The cppdbg extension works as expected and stops at the entry point of the program.

Compiling guests with debug information for release builds

This section explains how to compile a guest with debugging information but still have optimized code, and how to separate the debug information from the binary.

Creating a release build with debug information

To create a release build with debug information, you can add a custom profile to your Cargo.toml file:

toml
[profile.release-with-debug]
inherits = "release"
debug = true

This creates a new profile called release-with-debug that inherits all settings from the release profile but adds debug information.

Splitting debug information from the binary

To reduce the binary size while still having debug information available, you can split the debug information into a separate file.
This is useful for production environments where you want smaller binaries but still want to be able to debug crashes.

Here's a step-by-step guide:

  1. Build your guest with the release-with-debug profile:

    bash
    cargo build --profile release-with-debug

  2. Locate your binary in the target directory:

    bash
    TARGET_DIR="target"
PROFILE="release-with-debug"
ARCH="x86_64-unknown-none" # Your target architecture
BUILD_DIR="${TARGET_DIR}/${ARCH}/${PROFILE}"
BINARY=$(find "${BUILD_DIR}" -type f -executable -name "guest-binary" | head -1)

  3. Extract debug information into a full debug file:

    bash
    DEBUG_FILE_FULL="${BINARY}.debug.full"
objcopy --only-keep-debug "${BINARY}" "${DEBUG_FILE_FULL}"

  4. Create a symbols-only debug file (smaller, but still useful for stack traces):

    bash
    DEBUG_FILE="${BINARY}.debug"
objcopy --keep-file-symbols "${DEBUG_FILE_FULL}" "${DEBUG_FILE}"

  5. Strip debug information from the original binary but keep function names:

    bash
    objcopy --strip-debug "${BINARY}"

  6. Add a debug link to the stripped binary:

    bash
    objcopy --add-gnu-debuglink="${DEBUG_FILE}" "${BINARY}"

After these steps, you'll have:

  • An optimized binary with function names for basic stack traces
  • A symbols-only debug file for stack traces
  • A full debug file for complete source-level debugging

Analyzing core dumps with the debug files

When you have a core dump from a crashed guest, you can analyze it with different levels of detail using either GDB or LLDB.

Using GDB

  1. For basic analysis with function names (stack traces):

    bash
    gdb ${BINARY} -c /path/to/core.dump

  2. For full source-level debugging:

    bash
    gdb -s ${DEBUG_FILE_FULL} ${BINARY} -c /path/to/core.dump

Using LLDB

LLDB provides similar capabilities with slightly different commands:

  1. For basic analysis with function names (stack traces):

    bash
    lldb ${BINARY} -c /path/to/core.dump

  2. For full source-level debugging:

    bash
    lldb -o "target create -c /path/to/core.dump ${BINARY}" -o "add-dsym ${DEBUG_FILE_FULL}"

  3. If your debug symbols are in a separate file:

    bash
    lldb ${BINARY} -c /path/to/core.dump
(lldb) add-dsym ${DEBUG_FILE_FULL}

VSCode Debug Configurations

You can configure VSCode (in .vscode/launch.json) to use these files by modifying the debug configurations:

For GDB

json
{
    "name": "[GDB] Load core dump with full debug symbols",
    "type": "cppdbg",
    "request": "launch",
    "program": "${input:program}",
    "coreDumpPath": "${input:core_dump}",
    "cwd": "${workspaceFolder}",
    "MIMode": "gdb",
    "externalConsole": false,
    "miDebuggerPath": "/usr/bin/gdb",
    "setupCommands": [
        {
            "description": "Enable pretty-printing for gdb",
            "text": "-enable-pretty-printing",
            "ignoreFailures": true
        }
    ]
}

For LLDB

json
{
    "name": "[LLDB] Load core dump with full debug symbols",
    "type": "lldb",
    "request": "launch",
    "program": "${input:program}",
    "cwd": "${workspaceFolder}",
    "processCreateCommands": [],
    "targetCreateCommands": [
        "target create -c ${input:core_dump} ${input:program}"
    ],
    "postRunCommands": [
        // if debug symbols are in a different file
        "add-dsym ${input:debug_file_path}"
    ]
}

File: docs/how-to-make-releases.md

Releasing a new Hyperlight version to Cargo

This document details the process of releasing a new version of Hyperlight to crates.io. It's intended to be used as a checklist for the developer doing the release. The checklist is represented in the below sections.

Update Cargo.toml Versions

Currently, we need to manually update the workspace Cargo.toml version number to match to whatever release we are making. This will affect the version of all the crates in the workspace.

Note: we'll use v0.4.0 as the version for the above and all subsequent instructions. You should replace this with the version you're releasing. Make sure your version follows SemVer conventions as closely as possible, and is prefixed with a v character. In particular do not use a patch version unless you are patching an issue in a release branch, releases from main should always be minor or major versions.

Create a PR with this change and merge it into the main branch.

Update `CHANGELOG.md`

The CHANGELOG.md file is a critical document used to track changes made to Hyperlight. It serves as the foundation for generating release notes, so it's essential to keep it up to date with each release. While not every change needs to be added to this file (since a complete changelog of all PRs will be automatically generated), it's crucial to include all significant and user-facing updates. The changelog entries must follow Keep a Changelog format.

Steps to Update `CHANGELOG.md`:

  • Manually update the CHANGELOG.md with important changes since the latest release. Ideally, contributors should update this file as part of their PR, but this may not always happen.

  • Rename the [Prerelease] - Unreleased section to reflect the new version number (if not already done). Ensure that it links to the GitHub comparison between the current and previous versions. For example, v0.2.0 should link to https://github.com/hyperlight-dev/hyperlight/compare/v0.1.0...v0.2.0 (see the footer of CHANGELOG.md).

  • Add a new [Prerelease] section at the top of the file. This section should initially be empty and will track changes for the next release.

  • Preview the automatically generated release notes locally using the command:
    just create-release-notes v0.4.0 > notes.md. Review the notes to ensure everything looks accurate.

  • Create a PR with the updated CHANGELOG.md and merge it into the main branch once all changes are confirmed.

Create a tag

When both above PRs has merged into main branch you should create a tag. Make sure you have pulled the recently updated main branch, and do the following on the main branch:

bash
git tag -a v0.4.0 -m "A brief description of the release"
git push origin v0.4.0 # if you've named your git remote for the hyperlight-dev/hyperlight repo differently, change 'origin' to your remote name

If you are creating a patch release see the instructions here.

Create a release branch (no manual steps)

After you push your new tag in the previous section, the "Create a Release Branch" CI job will automatically run. When this job completes, a new release/v0.4.0 branch will be automatically created for you.

Create a new GitHub release

After the previous CI job runs to create the new release branch, go to the "Create a Release". GitHub actions workflow and do the following:

  1. Click the "Run workflow" button near the top right
  2. In the Use workflow from dropdown, select the release/v0.4.0 branch
  3. Click the green Run workflow button

Note: In case you see a "Create a Release" job already running before starting this step, that is because the "Create a Release" workflow also automatically runs on push to main branch to create a pre-release. You must still do the steps outlined above.

When this job is done, a new GitHub release will be created for you. This job also publishes the following rust packages to the crates.io:

  • hyperlight-common
  • hyperlight-guest
  • hyperlight-guest-macro
  • hyperlight-guest-bin
  • hyperlight-component-util
  • hyperlight-component-macro
  • hyperlight-host
  • hyperlight-guest-tracing

Patching a release

Note: for this example we'll assume v0.4.0 is already released, and you want to release a v0.4.1 patch for it.

  1. Make sure the patch/patches you want include are already merged to main branch.
  2. Make sure CHANGELOG.md is updated with the changes you want to include in the patch release (see instructions above) and is merged to main branch.
  3. Make sure the Cargo.toml versions are updated to the new patch version (e.g. v0.4.1), and is merged to main branch.
  4. Manually create a new branch from the release/v0.4.0 branch and name it release/v0.4.1. Important: do not create the branch from main branch.
  5. Cherry-pick the commits from main that you want to include in the patch (and resolve any conflicts). You must include the commit that updated the CHANGELOG.md and the commit that updated the Cargo.toml versions.
  6. Create a tag for the patch release, e.g. v0.4.1, similar to the steps above. Push the tag. A job will start to try to make a new release branch for you, but it will fail because the release/v0.4.1 branch already exists. This is expected, so don't worry about it.
  7. Follow the steps above to create a new GitHub release, but this time select the release/v0.4.1 branch in the "Use workflow from" dropdown. This will create a new GitHub release for you, and publish the updated packages to crates.io.

File: docs/how-to-use-flatbuffers.md

How to use FlatBuffers

Note: the last generation of the flatbuffer code was with done with flatc version 25.9.23 (i.e., the last version as of Oct 2nd, 2025).

Flatbuffers is used to serialize and deserialize some data structures.

Schema files are used to define the data structures and are used to generate the code to serialize and deserialize the data structures.

Those files are located in the schema directory.

Code generated from the schema files is checked in to the repository, therefore you only need to generate the code if you change an existing schema file or add a new one. You can find details on how to update schema files here.

Generating code

We use flatc to generate rust code.

We recommend building flatc from source. To generate rust code, use

console
just gen-all-fbs-rust-code

File: docs/hyperlight-execution-details.md

How code is run inside a VM

This document details how VMs are very quickly and efficiently created and configured to run arbitrary code.

Background

Hyperlight is a library for creating micro virtual machines (VMs) intended for executing small, short-running functions. This use case is different from that of many other VM platforms, which are aimed at longer-running, more complex workloads.

A very rough contrast between Hyperlight's offerings and other platforms is as follows:

Feature Hyperlight Other platforms
Hardware isolation (vCPU, virtual memory) Yes Yes
Shared memory between host and in-VM process Yes Yes [2]
Lightweight function calls between host and in-VM process (the "guest") Yes No
Bootloader/OS kernel No Yes [1]
Virtual networking No Yes [2]
Virtual filesystem No Yes [1]

As seen in this table, Hyperlight offers little more than a CPU and memory. We've removed every feature we could, while still providing a machine on which arbitrary code can execute, so we can achieve our various use cases and efficiency targets.

How code runs

With this background in mind, it's well worth focusing on the "lifecycle" of a VM -- how, exactly, a VM is created, modified, loaded, executed, and ultimately destroyed.

At the highest level, Hyperlight takes roughly the following steps to create and run arbitrary code inside a VM:

  1. Loads a specially built, statically linked binary (currently, the PE and ELF executable formats are supported) into memory. This is the code that is executed inside a virtual machine.
  2. Allocates additional memory regions, for example stack and heap for the guest, as well as some regions used for communication between the host and the guest.
  3. Creates a Virtual Machine and maps shared memory into it
  4. Create one virtual CPU (vCPU) within the newly created VM
  5. Write appropriate values to the new vCPUs registers.
  6. In a loop, tell previously created vCPU to run until we reach a halt message, one of several known error states, or an unsupported message
    1. In the former case, exit successfully
    2. In any of the latter cases, exit with a failure message

[1] nearly universal support

[2] varied support

File: docs/hyperlight-metrics-logs-and-traces.md

Observability

Hyperlight provides the following observability features:

  • Metrics are provided using the metrics crate, which is a lightweight metrics facade.
  • Logs are provided using the Rust tracing crate with the log feature enabled, allowing logs to be consumed by any Rust logger implementation compatible with the log crate.
  • Tracing is provided using the Rust tracing crate, and can be consumed by any Rust tracing implementation. In addition, the log feature is enabled which means that should a hyperlight host application not want to consume tracing events, you can still consume them as logs.

Metrics

Metrics are provided using the metrics crate, which is a lightweight metrics facade. When an executable installs a recorder, Hyperlight will emit its metrics to that recorder, which allows library authors to seamless emit their own metrics without knowing or caring which exporter implementation is chosen, or even if one is installed. In case where no recorder is installed, the metrics will be emitted to the default recorder, which is a no-op implementation with minimal overhead.

There are many different implementations of recorders. One example is the prometheus exporter which can be used to export metrics to a Prometheus server. An example of how to use this is provided in the examples/metrics directory.

The following metrics are provided and are enabled by default:

  • guest_errors_total - Counter that tracks the number of guest errors by error code.
  • guest_cancellations_total - Counter that tracks the number of guest executions that have been cancelled because the execution time exceeded the time allowed.

The following metrics are provided but are disabled by default:

  • guest_call_duration_seconds - Histogram that tracks the execution time of guest functions in seconds by function name. The histogram also tracks the number of calls to each function.
  • host_call_duration_seconds - Histogram that tracks the execution time of host functions in seconds by function name. The histogram also tracks the number of calls to each function.

The rationale for disabling the function call metrics by default is that:

  • A Hyperlight host may wish to provide its own metrics for function calls.
  • Enabling a trace subscriber will cause the function call metrics to be emitted as trace events, which may be sufficient for some use cases.
  • These 2 metrics require string clones for the function names, which may be too expensive for some use cases.
    We might consider enabling these metrics by default in the future.

Logs

Hyperlight provides logs using the Rust tracing crate with the log feature enabled. This means log events can be consumed by any Rust logger implementation compatible with the log crate. To consume logs, the host application must provide a logger implementation either by using the set_logger function directly or using a logger implementation that is compatible with the log crate.

For an example that uses the env_logger crate, see the examples/logging directory. By default, the env_logger crate will only log messages at the error level or higher. To see all log messages, set the RUST_LOG environment variable to debug.

Hyperlight also provides tracing capabilities (see below for more details), if no trace subscriber is registered, trace records will be emitted as log records, using the log feature of the tracing crate.

Tracing

Tracing spans are created for any call to a public API and the parent span will be set to the current span in the host if one exists, the level of the span is set to info. The span will be closed when the call returns. Any Result that contains an error variant will be logged as an error event. In addition to the public APIs, all internal functions are instrumented with trace spans at the trace level, therefore in order to see full trace information, the trace level should be enabled.

Hyperlight provides tracing using the Rust tracing crate, and can be consumed by any Rust trace subscriber implementation(seehere for some examples). When no tracing subscriber is set, trace events are automatically forwarded to the log facade via the log feature of the tracing crate, so consumers using only a log logger will still receive these events.

There are two examples that show how to consume both tracing events and log records as tracing events.

Using tracing_forest

In the examples/tracing directory, there is an example that shows how to capture and output trace and log information using the tracing_forest crate. With this example the following commands can be used to set the verbosity of the trace output to INFO and run the example:

Linux

bash
RUST_LOG='none,hyperlight_host=info,tracing=info' cargo run --example tracing

Windows

powershell
$env:RUST_LOG='none,hyperlight_host=info,tracing=info'; cargo run --example tracing

Using OTLP exporter and Jaeger

In the examples/tracing-otlp directory, there is an example that shows how to capture and send trace and log information to an otlp_collector using the opentelemetry_otlp crate. With this example the following commands can be used to set the verbosity of the trace output to INFO and run the example to generate trace data:

Linux

bash
RUST_LOG='none,hyperlight_host=info,tracing=info' cargo run --example tracing-otlp

Windows

powershell
$env:RUST_LOG='none,hyperlight_host=info,tracing=info';cargo run --example tracing-otlp

The sample will run and generate trace data until any key is pressed.

To view the trace data, leave the example running and use the jaegertracing/all-in-one container image with the following command:

console
docker run -p 16686:16686 -p 4317:4317 -p 4318:4318 -e COLLECTOR_OTLP_ENABLED=true jaegertracing/all-in-one:latest

NOTE: when running this on windows that this is a linux container, so you will need to ensure that docker is configured to run linux containers using WSL2. Alternatively, you can download the Jaeger binaries from here. Extract the archive and run the jaeger-all-in-one executable as follows:

powershell
.\jaeger-all-in-one.exe

Once the container or the exe is running, the trace output can be viewed in the jaeger UI at http://localhost:16686/search.

Guest Tracing, Unwinding, and Memory Profiling

Hyperlight provides advanced observability features for guest code running inside micro virtual machines. You can enable guest-side tracing, stack unwinding, and memory profiling using the trace_guest and mem_profile features. This section explains how to build, run, and inspect guest traces.

The following features are available for guest tracing:

  • trace_guest: Enables tracing for guest code, capturing function calls and execution time.
  • mem_profile: Enables memory profiling for guest code with stack unwinding, capturing memory allocations and usage.

Building a Guest with Tracing Support

To build a guest with tracing enabled, use the following commands:

bash
just build-rust-guests debug trace_guest
just move-rust-guests debug

This builds the guest binaries with the trace_guest feature enabled and move them to the appropriate location for use by the host.

NOTE: To enable the tracing in your application you need to use the trace_guest feature on the hyperlight-guest-bin and hyperlight-guest crates.

Running a Hyperlight Example with Guest Tracing

Once the guest is built, you can run a Hyperlight example with guest tracing enabled. For example:

bash
RUST_LOG="info,hyperlight_host::sandbox=info,hyperlight_guest=trace,hyperlight_guest_bin=trace" cargo run --example tracing-otlp --features trace_guest

This will execute the tracing-otlp example, loading the guest with tracing enabled.
During execution, trace data will be collected on the host and exported as opentelemetry spans/events.

You can set up a collector to gather all the traces and inspect the traces from both host and guests.

Due to the nature of execution inside a Sandbox, on a call basis, the guest tracing sets up a stack of spans to keep track of the correct parents for the incoming
guest spans.
We start with call-to-guest which contains all the spans coming from a guest. Additionally, for each exit into the host, we add another layer marking it with
a call-to-host span to follow the execution in the host and correctly set it as a child of the active span in the guest.
This logic simulates the propagation of Opentelemetry context that is usually done between two services, but cannot be done here seamlessly because the guest side
runs no_std which opentelemetry doesn't know.

How it works

Guest

When the guest starts executing the entrypoint function, it receives a max_log_level parameter that tells the guest what kind of logging level is expected from it.

The trace_guest logic takes advantage of this parameter and when the max_log_level is trace, it allocates a custom made GuestSubscriber that implements the Subscriber
trait from tracing_core that allows defining a subscriber for the tracing crate to handle new spans and events.

This custom subscriber stores the spans and events in a buffer initialized only when tracing is enabled. For each new span and event, a method is called on the custom subscriber which not only stores the data, but also keeps track of the hierarchy and dependencies between the other spans/events.
NOTE: The spans/events attributes are truncated to fit in the allocated buffer.

When the storage space is filled, the guest triggers a VM Exit that sends the guest pointers to the host. The host can access the guest memory, get the data and parse it to create the spans and events using the opentelemetry crate which allows specifying the starting and ending timestamps
which are captured in the guest using the TSC.

To improve performance, for each VMExit, the guest adds metadata for the host to be able to report the tracing data and free space.

Host

When a guest exits, the host checks for metadata from the guest reporting tracing data.
If tracing data is found, the host starts parsing it and reconstructing a tree which represents the spans hierarchy.

Additionally, the host also adds new children spans to the guest's reported active span, emphasizing the spans created on the host as a result of a temporary VM Exit. This helps visualize a call into the guest with context propagated across the VM boundary.

The host creates opentelemetry spans and events for each guest span and event reported.

Inspecting Guest memory Trace Files (for mem_profile)

To inspect the trace file generated by the guest, use the trace_dump crate. You will need the path to the guest symbols and the trace file. Run the following command:

bash
cargo run -p trace_dump   list_frames

Replace with the path to the guest binary or symbol file, and with the path to the trace file in the trace directory.

This command will list the stack frames and tracing information captured during guest execution, allowing you to analyze guest behavior, stack traces, and memory usage.

Example

bash
cargo run -p trace_dump ./src/tests/rust_guests/bin/debug/simpleguest ./trace/.trace list_frames

You can use the mem_profile additional feature by enabling them during the build and run steps.

Note: Make sure to follow the build and run steps in order, and ensure that the guest binaries are up to date before running the host example.

System Prerequisites for `trace_dump`

To build and use the trace_dump crate and related guest tracing features, you must have the following system libraries and development tools installed on your system:

  • glib-2.0 development files
    • Fedora/RHEL/CentOS:
      bash
      sudo dnf install glib2-devel pkgconf-pkg-config
  • cairo and cairo-gobject development files
    • Fedora/RHEL/CentOS:
      bash
      sudo dnf install cairo-devel cairo-gobject-devel
  • pango development files
    • Fedora/RHEL/CentOS:
      bash
      sudo dnf install pango-devel

These libraries are required by Rust crates such as glib-sys, cairo-sys-rs, and pango-sys, which are dependencies of the tracing and visualization tools. If you encounter errors about missing .pc files (e.g., glib-2.0.pc, cairo.pc, pango.pc), ensure the corresponding -devel packages are installed.

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →