# Technical Documentation: hyperlight-dev/hyperlight

> ℹ️ **Provenance:** Hybrid Fusion: `hyperlight-dev/hyperlight` (README + 10 In-Tree Chapters) · [CodeWiki Reference](https://codewiki.google/github.com/hyperlight-dev/hyperlight) · Recency: Active (< 180 days)

## 1. Project Overview & Quickstart (hyperlight-dev/hyperlight)

> **Status:** Hyperlight is pre-1.0. The API may change between releases, and upgrading will sometimes require code changes.

Hyperlight lets you safely run untrusted code inside hypervisor-isolated micro VMs that spin up in milliseconds, with guest function calls completing in microseconds. You embed it as a library in your Rust application, hand it a guest binary, and call functions across the VM boundary as naturally as calling a local function. To minimize startup time and memory footprint, there's no guest kernel or OS. Guests are purpose-built using the Hyperlight guest library.

- Supports [KVM](https://linux-kvm.org/page/Main_Page), [MSHV](https://github.com/rust-vmm/mshv), and [Windows Hypervisor Platform](https://docs.microsoft.com/en-us/virtualization/api/#windows-hypervisor-platform)
- No kernel or OS in the VM. Guests are regular ELF binaries written in `no_std` Rust or C
- Host and guest communicate through typed function calls
- Guests are sandboxed by default with no access to the host filesystem, network, etc.

## Example

**Host** - create a sandbox, register a host function, and call into the guest:

```rust
// Build a sandbox from a guest binary, registering a host function the guest
// can call. In a real app that function might query a database, read a config,
// or call an external API. By default, guests can only print to the host.
let mut sandbox = SandboxBuilder::from_file(guest_path)
    .host_function("GetWeekday", || Ok("Monday".to_string()))
    .build()?;

// Call a function inside the VM
let greeting: String = sandbox.call("SayHello", "World".to_string())?;
println!("{greeting}"); // "Hello, World! Today is Monday."
```

Guest state persists across calls. Use `snapshot()` and `restore()` to save and reset VM memory. This avoids recreating the VM while ensuring each call starts from a clean state.

**Guest** (Rust) - declare host functions and expose guest functions with simple macros. Guests can also be [written in C](./src/hyperlight_guest_capi).

```rust
#[host_function("GetWeekday")]
fn get_weekday() -> Result<String>;

#[guest_function("SayHello")]
fn say_hello(name: String) -> Result<String> {
    let weekday = get_weekday()?;
    Ok(format!("Hello, {name}! Today is {weekday}."))
}
```

To get started, see the [Getting Started](./docs/getting-started.md) guide. For more details on writing guests, see [How to build a Hyperlight guest binary](./docs/how-to-build-a-hyperlight-guest-binary.md).

## When to use Hyperlight

Hyperlight is a good fit when you need to:

- Run untrusted or third-party code with hypervisor-level isolation
- Create and tear down sandboxes in milliseconds
- Make guest function calls in microseconds
- Embed sandboxed execution directly in your application
- Build functions-as-a-service with hypervisor-level isolation
- Reuse sandboxes efficiently with snapshot and restore

Hyperlight is *not* designed for:

- General-purpose virtualization (use a full VMM instead)
- Running full-blown Linux guest workloads that need syscalls, networking, or filesystem access

## Getting started

See [docs/getting-started.md](./docs/getting-started.md) for detailed prerequisites and platform-specific setup for:
- **Running** Hyperlight
- **Building guests**

Or skip setup entirely with a codespace:

[](https://codespaces.new/hyperlight-dev/hyperlight)

## Repository Structure

| Directory | Description |
|---|---|
| [src/hyperlight_host](./src/hyperlight_host) | Host library - creates and manages micro VMs |
| [src/hyperlight_guest](./src/hyperlight_guest) | Core guest library - minimal building blocks for guest-host interaction |
| [src/hyperlight_guest_bin](./src/hyperlight_guest_bin) | Extended guest library - entry point, panic handler, heap, logging, exceptions |
| [src/hyperlight_guest_capi](./src/hyperlight_guest_capi) | C API wrapper around `hyperlight_guest_bin` for use via FFI |
| [src/hyperlight_libc](./src/hyperlight_libc) | C standard library for guests, built from picolibc |
| [src/hyperlight_guest_macro](./src/hyperlight_guest_macro) | Macros for registering guest and host functions |
| [src/hyperlight_guest_tracing](./src/hyperlight_guest_tracing) | Tracing support for guests |
| [src/hyperlight_common](./src/hyperlight_common) | Shared code used by both host and guest |
| [src/hyperlight_component_macro](./src/hyperlight_component_macro) | Proc macros for WIT-based host/guest bindings |
| [src/hyperlight_component_util](./src/hyperlight_component_util) | Shared implementation for WIT binding generation |
| [src/hyperlight_testing](./src/hyperlight_testing) | Shared test utilities |
| [src/schema](./src/schema) | FlatBuffer schema definitions |
| [src/trace_dump](./src/trace_dump) | Tool for dumping and visualizing trace data |
| [src/tests](./src/tests) | Test guest programs (Rust and C) |

## Related Projects

- [cargo-hyperlight](https://github.com/hyperlight-dev/cargo-hyperlight) - Cargo subcommand for building and scaffolding Hyperlight guests
- [hyperlight-wasm](https://github.com/hyperlight-dev/hyperlight-wasm) - Run WebAssembly modules inside Hyperlight micro VMs
- [hyperlight-js](https://github.com/hyperlight-dev/hyperlight-js) - Run JavaScript inside Hyperlight micro VMs
- [hyperlight-sandbox](https://github.com/hyperlight-dev/hyperlight-sandbox) - Multi-backend sandboxing framework for running untrusted code with controlled host capabilities, with Python, .NET, and Rust SDKs
- [hyperlight-unikraft](https://github.com/hyperlight-dev/hyperlight-unikraft) - Run Linux applications (Python, Node.js, Go, Rust, C/C++) on Hyperlight micro VMs using Unikraft as the guest kernel

## Contributing

See [CONTRIBUTING.md](./CONTRIBUTING.md).

## Community

- **Meetings**: Every other Wednesday 09:00 PST/PDT ([convert to your time](https://dateful.com/convert/pst-pdt-pacific-time?t=09)). Agenda and join info in the [Community Meeting Notes](https://hackmd.io/blCrncfOSEuqSbRVT9KYkg#Agenda).
- **Slack**: [#hyperlight](https://cloud-native.slack.com/archives/hyperlight) on CNCF Slack ([join here](https://www.cncf.io/membership-faq/#how-do-i-join-cncfs-slack)).
- **Docs**: [`docs/` directory](./docs/README.md)
- **Code of Conduct**: [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md)

---

[](https://app.fossa.com/projects/git%2Bgithub.com%2Fhyperlight-dev%2Fhyperlight?ref=badge_large)

## 2. In-Tree Documentation Chapters (hyperlight-dev/hyperlight)

## File: README.md

> **Status:** Hyperlight is pre-1.0. The API may change between releases, and upgrading will sometimes require code changes.

Hyperlight lets you safely run untrusted code inside hypervisor-isolated micro VMs that spin up in milliseconds, with guest function calls completing in microseconds. You embed it as a library in your Rust application, hand it a guest binary, and call functions across the VM boundary as naturally as calling a local function. To minimize startup time and memory footprint, there's no guest kernel or OS. Guests are purpose-built using the Hyperlight guest library.

- Supports [KVM](https://linux-kvm.org/page/Main_Page), [MSHV](https://github.com/rust-vmm/mshv), and [Windows Hypervisor Platform](https://docs.microsoft.com/en-us/virtualization/api/#windows-hypervisor-platform)
- No kernel or OS in the VM. Guests are regular ELF binaries written in `no_std` Rust or C
- Host and guest communicate through typed function calls
- Guests are sandboxed by default with no access to the host filesystem, network, etc.

## Example

**Host** - create a sandbox, register a host function, and call into the guest:

```rust
// Build a sandbox from a guest binary, registering a host function the guest
// can call. In a real app that function might query a database, read a config,
// or call an external API. By default, guests can only print to the host.
let mut sandbox = SandboxBuilder::from_file(guest_path)
    .host_function("GetWeekday", || Ok("Monday".to_string()))
    .build()?;

// Call a function inside the VM
let greeting: String = sandbox.call("SayHello", "World".to_string())?;
println!("{greeting}"); // "Hello, World! Today is Monday."
```

Guest state persists across calls. Use `snapshot()` and `restore()` to save and reset VM memory. This avoids recreating the VM while ensuring each call starts from a clean state.

**Guest** (Rust) - declare host functions and expose guest functions with simple macros. Guests can also be [written in C](./src/hyperlight_guest_capi).

```rust
#[host_function("GetWeekday")]
fn get_weekday() -> Result<String>;

#[guest_function("SayHello")]
fn say_hello(name: String) -> Result<String> {
    let weekday = get_weekday()?;
    Ok(format!("Hello, {name}! Today is {weekday}."))
}
```

To get started, see the [Getting Started](./docs/getting-started.md) guide. For more details on writing guests, see [How to build a Hyperlight guest binary](./docs/how-to-build-a-hyperlight-guest-binary.md).

## When to use Hyperlight

Hyperlight is a good fit when you need to:

- Run untrusted or third-party code with hypervisor-level isolation
- Create and tear down sandboxes in milliseconds
- Make guest function calls in microseconds
- Embed sandboxed execution directly in your application
- Build functions-as-a-service with hypervisor-level isolation
- Reuse sandboxes efficiently with snapshot and restore

Hyperlight is *not* designed for:

- General-purpose virtualization (use a full VMM instead)
- Running full-blown Linux guest workloads that need syscalls, networking, or filesystem access

## Getting started

See [docs/getting-started.md](./docs/getting-started.md) for detailed prerequisites and platform-specific setup for:
- **Running** Hyperlight
- **Building guests**

Or skip setup entirely with a codespace:

[](https://codespaces.new/hyperlight-dev/hyperlight)

## Repository Structure

| Directory | Description |
|---|---|
| [src/hyperlight_host](./src/hyperlight_host) | Host library - creates and manages micro VMs |
| [src/hyperlight_guest](./src/hyperlight_guest) | Core guest library - minimal building blocks for guest-host interaction |
| [src/hyperlight_guest_bin](./src/hyperlight_guest_bin) | Extended guest library - entry point, panic handler, heap, logging, exceptions |
| [src/hyperlight_guest_capi](./src/hyperlight_guest_capi) | C API wrapper around `hyperlight_guest_bin` for use via FFI |
| [src/hyperlight_libc](./src/hyperlight_libc) | C standard library for guests, built from picolibc |
| [src/hyperlight_guest_macro](./src/hyperlight_guest_macro) | Macros for registering guest and host functions |
| [src/hyperlight_guest_tracing](./src/hyperlight_guest_tracing) | Tracing support for guests |
| [src/hyperlight_common](./src/hyperlight_common) | Shared code used by both host and guest |
| [src/hyperlight_component_macro](./src/hyperlight_component_macro) | Proc macros for WIT-based host/guest bindings |
| [src/hyperlight_component_util](./src/hyperlight_component_util) | Shared implementation for WIT binding generation |
| [src/hyperlight_testing](./src/hyperlight_testing) | Shared test utilities |
| [src/schema](./src/schema) | FlatBuffer schema definitions |
| [src/trace_dump](./src/trace_dump) | Tool for dumping and visualizing trace data |
| [src/tests](./src/tests) | Test guest programs (Rust and C) |

## Related Projects

- [cargo-hyperlight](https://github.com/hyperlight-dev/cargo-hyperlight) - Cargo subcommand for building and scaffolding Hyperlight guests
- [hyperlight-wasm](https://github.com/hyperlight-dev/hyperlight-wasm) - Run WebAssembly modules inside Hyperlight micro VMs
- [hyperlight-js](https://github.com/hyperlight-dev/hyperlight-js) - Run JavaScript inside Hyperlight micro VMs
- [hyperlight-sandbox](https://github.com/hyperlight-dev/hyperlight-sandbox) - Multi-backend sandboxing framework for running untrusted code with controlled host capabilities, with Python, .NET, and Rust SDKs
- [hyperlight-unikraft](https://github.com/hyperlight-dev/hyperlight-unikraft) - Run Linux applications (Python, Node.js, Go, Rust, C/C++) on Hyperlight micro VMs using Unikraft as the guest kernel

## Contributing

See [CONTRIBUTING.md](./CONTRIBUTING.md).

## Community

- **Meetings**: Every other Wednesday 09:00 PST/PDT ([convert to your time](https://dateful.com/convert/pst-pdt-pacific-time?t=09)). Agenda and join info in the [Community Meeting Notes](https://hackmd.io/blCrncfOSEuqSbRVT9KYkg#Agenda).
- **Slack**: [#hyperlight](https://cloud-native.slack.com/archives/hyperlight) on CNCF Slack ([join here](https://www.cncf.io/membership-faq/#how-do-i-join-cncfs-slack)).
- **Docs**: [`docs/` directory](./docs/README.md)
- **Code of Conduct**: [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md)

---

[](https://app.fossa.com/projects/git%2Bgithub.com%2Fhyperlight-dev%2Fhyperlight?ref=badge_large)

---

## File: docs/benchmarking-hyperlight.md

# Benchmark Notes

Hyperlight uses the [Criterion](https://bheisler.github.io/criterion.rs/book/index.html) framework to run and analyze benchmarks. A benefit to this framework is that it doesn't require the nightly toolchain.

## When Benchmarks are run

1. Daily (scheduled)
    - Benchmarks run daily via `DailyBenchmarks.yml`, comparing results against the previous day's run. Results are stored as workflow artifacts with 90-day retention.

    ```
    sandboxes/create_sandbox
                        time:   [33.803 ms 34.740 ms 35.763 ms]
                        change: [+0.7173% +3.7017% +7.1346%] (p = 0.03 < 0.05)
                        Change within noise threshold.*
    ```
   
2. For each release
    - For each release, benchmarks are run as part of the release pipeline in `CreateRelease.yml`, which invokes `dep_benchmarks.yml`. These benchmark results are compared to the previous release, and are uploaded as part of the "Release assets" on the GitHub release page.

Currently, benchmarks are run on windows, linux-kvm (ubuntu), and linux-hyperv (mariner). Only release builds are benchmarked, not debug.

## Criterion artifacts

When running `cargo bench -- --save-baseline my_baseline`, criterion runs all benchmarks defined in `src/hyperlight_host/benches/`, prints the results to the stdout, as well as produces several artifacts. All artifacts can be found in `target/criterion/`. For each benchmarking group, for each benchmark, a subfolder with the name of the benchmark is created. This folder in turn contains folders `my_baseline`, `new`  and `report`. When running `cargo bench`, criterion always creates `new` and `report`, which always contains the most recent benchmark result and html report, but because we provided the `--save-baseline` flag, we also have a `my_baseline` folder, which is an exact copy of `new`. Moreover, if this `my_baseline` folder already existed before we ran `cargo bench -- --save-baseline my_baseline`, criterion would also compare the benchmark results with the old `my_baseline` folder, and then overwrite the folder.

The first time we run `cargo bench -- --save-baseline my_baseline` (starting with a clean project), we get the following structure. 

```
target/criterion/
|-- report
`-- sandboxes
    |-- create_sandbox
    |   |-- my_baseline
    |   |-- new
    |   `-- report
    |-- create_sandbox_and_call_context
    |   |-- my_baseline
    |   |-- new
    |   `-- report
    `-- report
```

If we run the exact same command again, we get 

```
target/criterion/
|-- report
`-- sandboxes
    |-- create_sandbox
    |   |-- change
    |   |-- my_baseline
    |   |-- new
    |   `-- report
    |-- create_sandbox_and_call_context
    |   |-- change
    |   |-- my_baseline
    |   |-- new
    |   `-- report
    `-- report
```

Note that it overwrote the previous `my_baseline` with the new result. But notably, there is a new `change` folder, which contains the benchmarking difference between the two runs. In addition, on stdout you'll also find a comparison to our previous `my_baseline` run.

```
                        time:   [40.434 ms 40.777 ms 41.166 ms]
                        change: [+0.0506% +1.1399% +2.2775%] (p = 0.06 > 0.05)
                        No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
```

**Note** that Criterion does not differ between release and debug/dev benchmark results, so it's up to the developer to make sure baselines of the same config are compared.

## Running benchmarks locally

Use `just bench` to run benchmarks with release builds (the only supported configuration). Comparing local benchmark results to GitHub-saved benchmarks doesn't make much sense, since you'd be using different hardware, but you can use `just bench-download os hypervisor cpu_vendor [tag] ` to download and extract the GitHub release benchmarks to the correct folder. You can then run `just bench-ci main` to compare to (and overwrite) the previous release benchmarks. Note that `main` is the name of the baselines stored in GitHub.

**Important**: The `just bench` command uses release builds by default to ensure meaningful performance measurements. For profiling purposes, you can compile benchmarks with debug symbols by running `cargo bench` directly.

---

## File: docs/github-labels.md

# **Pull Requests (PRs)**

We use GitHub labels to categorize PRs. Before a PR can be merged, it must be assigned one of the following **kind/** labels:

- **kind/bugfix** - For PRs that fix bugs.
- **kind/dependencies** - For PRs that update dependencies or related components.
- **kind/enhancement** - For PRs that introduce new features or improve existing functionality. This label also applies to improvements in documentation, testing, and similar areas. Any changes must be backward-compatible.
- **kind/refactor** - For PRs that restructure or remove code without adding new functionality. This label also applies to changes that affect user-facing APIs.

## Review readiness

We use the **ready-for-review** label to signal that a PR is waiting for a (re-)review:

- **Add** `ready-for-review` when you open a PR that is ready for review, or when a PR is ready for re-review (for example, once you have addressed requested changes and re-requested review).
- The label is **removed automatically** by the ready-for-review label automation (the [`collect`](../.github/workflows/ready-for-review-label.yml) and [`manage`](../.github/workflows/ready-for-review-manage.yml) workflows) once the PR is no longer awaiting that review, specifically when any of the following become true:
  - the PR is closed or merged,
  - the PR is converted to a draft,
  - the PR has two or more approvals, or
  - the PR has two or more change requests.

You only ever need to add the label; removal is fully automated.

---

# **Issues**

Issues are categorized using the following three **GitHub types** (not GitHub labels):

- **bug** - Reports an unexpected problem or incorrect behavior.
- **design** - Relates to design considerations or decisions.
- **enhancement** - Suggests a new feature, improvement, or idea.

To track the lifecycle of issues, we also use GitHub labels:

- **lifecycle/needs-review** - A temporary label indicating that the issue has not yet been reviewed.
- **lifecycle/confirmed** - Confirms the issue’s validity:
  - If the issue type is **bug**, the bug has been verified.
  - If the issue type is **enhancement**, the proposal is considered reasonable but does not guarantee implementation.
  - This label does not indicate when or if the fix or enhancement will be implemented.
- **lifecycle/needs-info** - The issue requires additional information from the original poster (OP).
- **lifecycle/blocked** - The issue is blocked by another issue or external factor.

The following labels should be applied to issues prior to closing, indicating the resolution status of the issue:

- **lifecycle/duplicate** - The issue is a duplicate of another issue.
- **lifecycle/fixed** - The issue has been resolved.
- **lifecycle/not-a-bug** - The issue is not considered a bug, and no further action is needed.
- **lifecycle/wont-fix** - The issue will not be fixed.

In addition to lifecycle labels, we use the following labels to further categorize issues:

- **good-first-issue** - The issue is suitable for new contributors or those looking for a simple task to start with.
- **help-wanted** - The issue is a request for help or assistance.
- **question** - The issue is a question or request for information.
- **release-blocker** - Critical issues that must be resolved before the next release can be made. The presence of this label on any open issue will prevent releases being created by the release workflow

---

# **Issues & PRs**

In addition to **kind/*** labels, we use optional **area/*** labels to specify the focus of a PR or issue. These labels are purely for categorization, and are not mandatory.

- **area/API** - Related to the API or public interface.
- **area/ci-periodics** - Applied to issues that track failures from scheduled (periodic) CI jobs. These issues are opened automatically by the CI failure notifier (`dev/notify-ci-failure.sh`).
- **area/ci-periodics-aarch64** - De-duplication label for failures from the daily aarch64 workflow (`.github/workflows/DailyArm64.yml`). It keeps those failures on their own dedicated, release-blocking issue instead of sharing the general `area/ci-periodics` issue.
- **area/dependencies** - Concerns dependencies or related components. This label is different from **kind/dependencies**, which should only used for PRs.
- **area/documentation** - Related to documentation updates or improvements.
- **area/infrastructure** - Concerns infrastructure rather than core functionality.
- **area/performance** - Addresses performance.
- **area/security** - Involves security-related changes or fixes.
- **area/testing** - Related to tests or testing infrastructure.

## Workflow labels

Some labels change CI behaviour on a PR rather than categorizing it:

- **regen-goldens** - Switches the snapshot golden verify job into regenerate mode. A PR that intentionally changes the snapshot format and bumps `GOLDENS_VERSION` carries this label so the verify job generates the goldens from the branch and runs them back through the branch loader, rather than pulling a published tag set that does not exist yet. See [snapshot-versioning.md](snapshot-versioning.md).


## Notes
This document is a work in progress and may be updated as needed. The labels and categories are subject to change based on the evolving needs of the project and community feedback.

---

## File: docs/glossary.md

# Glossary

* [Hyperlight](#hyperlight)
* [Host Application](#host-application)
* [Host](#host)
* [Hypervisor](#hypervisor)
* [Driver](#driver)
* [Hyper-V](#hyper-v)
* [KVM](#kvm)
* [Guest](#guest)
* [Micro Virtual Machine](#micro-virtual-machine)
* [Workload](#workload)
* [Sandbox](#sandbox)

## Hyperlight

Hyperlight refers to the Hyperlight Project and not a specific component. Hyperlight is intended to be used as a library to embed hypervisor-isolated execution support inside a [host application](#host-application).

## Host Application

This is an application that consumes the Hyperlight library, in order to execute code in an hypervisor-isolated environment.

## Host

Host is the machine on which the [host application](#host-application) is running. A host could be a bare metal or virtual machine, when the host is a virtual machine, the nested virtualization is required to run Hyperlight.

## Hypervisor

Hypervisor is the software responsible for creating isolated [micro virtual machines](#micro-virtual-machine), as well as executing [guests](#guest) inside of those micro virtual machines. Hyperlight has [drivers](#driver) for the following hypervisors: [Hyper-V](#hyper-v) on Windows, [Hyper-V](#hyper-v) on Linux, and [KVM](#kvm).

## Driver

Hyperlight supports executing workloads on particular [hypervisors](#hypervisor) through drivers. Each supported hypervisor has its own driver to manage interacting with that hypervisor.

## Hyper-V

Hyper-V is a [hypervisor](#hypervisor) capable of creating and executing isolated [micro virtual machines](#micro-virtual-machine) on both Windows and Linux. On Linux, Hyper-V is sometimes referred to as MSHV (Microsoft Hypervisor).

## KVM

Kernel-based Virtual Machine (KVM) is a [hypervisor](#hypervisor) capable of creating and executing isolated [micro virtual machines](#micro-virtual-machine) on Linux.

## MSHV

MSHV stands for Microsoft Hypervisor and is the name commonly used for Hyper-V when the hypervisor is running Linux dom0 (as opposed to Windows dom0).

## Guest

A guest is a standalone executable binary that is executed inside a hypervisor [micro virtual machine](#micro-virtual-machine). By having purpose-fit guests binaries, as opposed to running a full operating system, Hyperlight achieves low-latency startup times of workloads, since it doesn't need to first boot an entire operating system before executing the workload.

The interface that a guest must implement is specific to the associated [host](#host) and the type of workloads that it may be specialized for executing, such as WebAssembly Modules (Wasm), or a specific language.

## Micro Virtual Machine

A micro virtual machine is an execution environment managed by a hypervisor that isolates a [guest](#guest) from the [host](#host). A hypervisor prevents the guest from directly accessing the host's resources, such as memory, filesystem, devices, memory or CPU.

We use the term Micro Virtual Machine as the VMs are very lightweight compared to traditional VMs, containing no operating system or other unnecessary components. The goal is to provide a minimal environment for executing workloads with low latency and high density. However the isolation provided by the hypervisor is the same as that of a traditional VM.

## Workload

A workload is the code that the [host application](#host-application) wants to execute in an isolated [micro virtual machine](#micro-virtual-machine).

## Sandbox

A Sandbox is the abstraction used in Hyperlight to represent the isolated environment in which a workload is executed. A sandbox is used to create, configure, execute and destroy a [micro virtual machine](#micro-virtual-machine) that runs a [guest](#guest) workload. Sandboxes are created by the [host application](#host-application) using the Hyperlight host library.

---

## File: docs/how-to-build-a-hyperlight-guest-binary.md

# Building a Hyperlight guest binary

This document explains how to build a binary to be used as a Hyperlight guest.

A Hyperlight guest is a regular ELF binary built for a custom target. It runs
without an operating system, exposes functions that the host can call, and may
call functions registered by the host. Builds use `cargo hyperlight build`,
which sets up the target, sysroot, and environment variables.

Install [`cargo-hyperlight`](https://github.com/hyperlight-dev/cargo-hyperlight)
with:

```sh
cargo install --locked cargo-hyperlight
```

To scaffold a working host plus guest project, run:

```sh
cargo hyperlight new my-project
```

Pass `--no-host` to generate only a guest crate, or `--no-guest` to generate
only a host crate.

The rest of this document explains what the generated guest contains and how to
build one by hand.

## Rust guest binary

### Minimal guest

A minimal Rust guest depends only on `hyperlight-guest-bin` and uses its
attribute macros to register guest functions and declare host functions.

`Cargo.toml`:

```toml
[package]
name = "my-guest"
version = "0.1.0"
edition = "2024"

[dependencies]
hyperlight-guest-bin = "*"  # use the latest version from crates.io
```

`src/main.rs`:

```rust
#![no_std]
#![no_main]
extern crate alloc;

use alloc::string::String;

use hyperlight_guest_bin::error::Result;
use hyperlight_guest_bin::{guest_function, host_function};

// Declare a host function the guest can call. The string is the name the
// host registered it under. If omitted, the Rust function name is used.
#[host_function("GetWeekday")]
fn get_weekday() -> Result<String>;

// Register a guest function the host can call.
#[guest_function("SayHello")]
fn say_hello(name: String) -> Result<String> {
    let weekday = get_weekday()?;
    Ok(alloc::format!("Hello, {name}! Today is {weekday}."))
}
```

Build with:

```sh
cargo hyperlight build
```

The resulting binary in `target/x86_64-hyperlight-none//my-guest` is
what the host loads with `GuestBinary::FilePath(...)`.

#### What the macros generate

* `#[guest_function]` registers the function with the guest runtime so the host
  can call it by name. Argument and return types must be supported parameter
  and return types, or `Result<T, HyperlightGuestError>` wrapping one.
* `#[host_function]` turns an `extern`-style signature into a stub that
  marshals arguments to the host and returns the host's reply.
* The runtime provides a default `hyperlight_main` entry point and a default
  dispatch function. You do not need to write either one for a typical guest.

#### Required attributes

* `#![no_std]` because there is no operating system in the guest.
* `#![no_main]` because the entry point is `hyperlight_main`, not `main`.
* `extern crate alloc` to use `Vec`, `String`, and other heap types.

### Advanced: manual entry point and dispatch

The macros are optional. A guest can define the underlying symbols directly,
which is useful for advanced setup work or custom dispatch logic (for example
the WIT-based guests in `src/tests/rust_guests/witguest`).

The host expects two guest symbols:

* `pub extern "C" fn hyperlight_main()` runs once at guest startup. Use it to
  register functions or initialize global state.
* `pub extern "Rust" fn guest_dispatch_function(function_call: FunctionCall) -> Result<Vec, HyperlightGuestError>`
  is invoked when the host calls a function name that is not registered.

`hyperlight-guest-bin` exposes `#[main]` and `#[dispatch]` macros that generate
these symbols from a regular Rust function, so most "advanced" guests still use
the macros rather than writing the raw `extern "C"` items themselves.

If you mix the manual form with `#[guest_function]`, registrations from the
macro still happen automatically. Your `hyperlight_main` only needs to do
whatever extra setup the macros do not cover.

### Troubleshooting

#### "duplicate lang item `panic_impl`" error

This error means the standard library's panic handler is being linked
alongside `hyperlight_guest_bin`'s. To fix:

1. Ensure your crate has `#![no_std]` at the top of `main.rs`.
2. Confirm any transitive dependency on `hyperlight-common` uses
   `default-features = false`.
3. Run `cargo clean` to clear stale artifacts.
4. Build with `cargo hyperlight build`, not `cargo build`.

#### Build errors with dependencies

If you see errors building dependencies (such as `serde`), make sure you are
using `cargo hyperlight build`. It sets up the environment variables and
sysroot needed for the custom Hyperlight target.

## C guest binary

For the binary written in C, the generated C bindings can be downloaded from the
latest release page that contain: the `hyperlight_guest.h` header and the
C API library.
The `hyperlight_guest.h` header contains the corresponding APIs to register
guest functions and call host functions from within the guest.

See [src/tests/c_guests/c_simpleguest/main.c](../src/tests/c_guests/c_simpleguest/main.c)
for a complete example.

## Version compatibility

Guest binaries built with `hyperlight-guest-bin` automatically embed the crate
version in an ELF note section (`.note.hyperlight-version`). When the host
loads a guest binary, it checks this version and rejects the binary if it does
not match the host's version of `hyperlight-host`.

Hyperlight currently provides no backwards compatibility guarantees for guest
binaries — the guest and host crate versions must match exactly. If you see a
`GuestBinVersionMismatch` error, rebuild the guest binary with a matching
version of `hyperlight-guest-bin`.

---

## File: docs/how-to-debug-a-hyperlight-guest.md

# How to debug a Hyperlight guest using gdb or lldb

Hyperlight supports gdb debugging of a guest running inside a Hyperlight sandbox on Linux or Windows.
When Hyperlight is compiled with the `gdb` feature enabled, a Hyperlight sandbox can be configured
to start listening for a gdb connection.

## Supported features

The Hyperlight `gdb` feature enables guest debugging to:
   - stop at an entry point breakpoint which is automatically set by Hyperlight
   - add and remove HW breakpoints (maximum 4 set breakpoints at a time)
   - add and remove SW breakpoints
   - read and write registers
   - read and write addresses
   - step/continue
   - get code offset from target
   - stop when a crash occurs and only allow read access to the guest memory and registers

## Expected behavior

Below is a list describing some cases of expected behavior from a gdb debug 
session of a guest binary running inside a Hyperlight sandbox.

- when the `gdb` feature is enabled and the sandbox builder is given a debug
  port, the created sandbox will wait for a gdb client to connect on the
  configured port
- when the gdb client attaches, the guest vCPU is expected to be stopped at the
  entry point
- if a gdb client disconnects unexpectedly, the debug session will be closed and
  the guest will continue executing disregarding any prior breakpoints
- if multiple sandbox instances are created, each instance will have its own
  gdb thread listening on the configured port
- if two sandbox instances are created with the same debug port, the second
  instance logs an error and the gdb thread will not be created, but the sandbox
  will continue to run without gdb debugging
- when a crash happens, the debugger session remains active, and the guest
  vCPU is stopped, allowing the gdb client to inspect the state of the guest.
  The debug target will refuse any resume, step actions and write operations to
  the guest memory and registers until the gdb client disconnects or the sandbox is stopped.

## Example

### Sandbox configuration

The `guest-debugging` example in Hyperlight demonstrates how to configure a Hyperlight
sandbox to listen for a gdb client on a specific port.

### CLI Gdb configuration

One can use a gdb config file to provide the symbols and desired configuration.

The below contents of the `.gdbinit` file can be used to provide a basic configuration
to gdb startup.

```gdb
# Path to symbols
file path/to/symbols.elf
# The port on which Hyperlight listens for a connection
target remote :8080
set disassembly-flavor intel
set disassemble-next-line on
enable pretty-printer
layout src
```
One can find more information about the `.gdbinit` file at [gdbinit(5)](https://www.man7.org/linux/man-pages/man5/gdbinit.5.html).

### End to end example

Using the example mentioned at [Sandbox configuration](#sandbox-configuration) 
one can run the below commands to debug the guest binary:

```bash
# Terminal 1
$ cargo run --example guest-debugging --features gdb
```

```bash
# Terminal 2
$ cat .gdbinit
file src/tests/rust_guests/bin/debug/simpleguest
target remote :8080
set disassembly-flavor intel
set disassemble-next-line on
enable pretty-printer
layout src

$ gdb
```

### Using VSCode to debug a Hyperlight guest

To replicate the above behavior using VSCode follow the below steps:
- To use gdb: 
    1. install the `gdb` package on the host machine
    2. install the [C/C++ Extension Pack](https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools-extension-pack) extension in VSCode to add debugging capabilities
- To use lldb:
    1. install `lldb` on the host machine
    2. install the [CodeLLDB](https://marketplace.visualstudio.com/items?itemName=vadimcn.vscode-lldb) extension in VSCode to add debugging capabilities
- create a `.vscode/launch.json` file in the project directory with the below content:
    ```json
    {
        "version": "0.2.0",
        "configurations": [
            {
                "name": "LLDB",
                "type": "lldb",
                "request": "launch",
                "targetCreateCommands": ["target create ${workspaceFolder}/src/tests/rust_guests/bin/debug/simpleguest"],
                "processCreateCommands": ["gdb-remote localhost:8080"]
            },
            {
                "name": "GDB",
                "type": "cppdbg",
                "request": "launch",
                "program": "${workspaceFolder}/src/tests/rust_guests/bin/debug/simpleguest",
                "args": [],
                "stopAtEntry": true,
                "hardwareBreakpoints": {"require": false, "limit": 4},
                "cwd": "${workspaceFolder}",
                "environment": [],
                "externalConsole": false,
                "MIMode": "gdb",
                "miDebuggerPath": "/usr/bin/gdb",
                "miDebuggerServerAddress": "localhost:8080",
                "setupCommands": [
                    {
                        "description": "Enable pretty-printing for gdb",
                        "text": "-enable-pretty-printing",
                        "ignoreFailures": true
                    },
                    {
                        "description": "Set Disassembly Flavor to Intel",
                        "text": "-gdb-set disassembly-flavor intel",
                        "ignoreFailures": true
                    }
                ]
            }
        ]
    }
    ```
- in `Run and Debug` tab, select either `GDB` or `LLDB` configuration and click on the `Run`
  button to start the debugging session.
  The debugger will connect to the Hyperlight sandbox and the guest vCPU will
  stop at the entry point.


## How it works

The gdb feature is designed to work like a Request - Response protocol between
a thread that accepts commands from a gdb client and main thread of the sandbox.

All the functionality is implemented on the hypervisor side so it has access to
the shared memory and the vCPU.

The gdb thread uses the `gdbstub` crate to handle the communication with the gdb client.
When the gdb client requests one of the supported features mentioned above, a request
is sent over the communication channel to the main thread for the sandbox
to resolve.

Below is a sequence diagram that shows the interaction between the entities
involved in the gdb debugging of a Hyperlight guest running inside a **KVM** or **MSHV** sandbox.

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Dumping the guest state to an ELF core dump 

When a guest crashes because of an unknown VmExit or unhandled exception, the vCPU state can be optionally dumped to an `ELF` core dump file.
This can be used to inspect the state of the guest at the time of the crash.

To make Hyperlight dump the state of the vCPU (general purpose registers, registers) to an `ELF` core dump file, enable the `crashdump` feature and run.
The feature enables the creation of core dump files for both debug and release builds of Hyperlight hosts.
By default, Hyperlight places the core dumps in the temporary directory (platform specific).
To change this, use the `HYPERLIGHT_CORE_DUMP_DIR` environment variable to specify a directory.
The name and location of the dump file will be printed to the console and logged as an error message.

**NOTE**: If the directory provided by `HYPERLIGHT_CORE_DUMP_DIR` does not exist, Hyperlight places the file in the temporary directory.
**NOTE**: By enabling the `crashdump` feature, you instruct Hyperlight to create core dump files for all sandboxes when an unhandled crash occurs.
To selectively disable this feature for a specific sandbox, call `guest_core_dump(false)` on the `SandboxBuilder`.
```rust
    let sandbox = SandboxBuilder::from_file(guest_path)
        .guest_core_dump(false) // Disable core dump for this sandbox
        .build()?;
```

## Creating a dump on demand

You can also create a core dump of the current state of the guest on demand by calling the `generate_crashdump` method on the `InitializedMultiUseSandbox` instance. This can be useful for debugging issues in the guest that do not cause crashes (e.g., a guest function that does not return).

This is only available when the `crashdump` feature is enabled and then only if the sandbox
is also configured to allow core dumps (which is the default behavior).

### Example

Attach to your running process with gdb and call this function:

```shell
sudo gdb -p 
(gdb) info threads
# find the thread that is running the guest function you want to debug
(gdb) thread 
# switch to the frame where you have access to your MultiUseSandbox instance
(gdb) backtrace
(gdb) frame <frame_number>
# get the pointer to your MultiUseSandbox instance
# Get the sandbox pointer
(gdb) print sandbox
# Call the crashdump function with the pointer
    # Call the crashdump function 
call sandbox.generate_crashdump()
```
The crashdump should be available `/tmp` or in the crash dump directory (see `HYPERLIGHT_CORE_DUMP_DIR` env var). To make this process easier, you can also create a gdb script that automates these steps. You can find an example script [here](../scripts/dump_all_sandboxes.gdb). This script will try and generate a crashdump for every active thread except thread 1 , it assumes that the variable sandbox exists in frame 15 on every thread. You can edit it to fit your needs. Then use it like this:

```shell
(gdb) source scripts/dump_all_sandboxes.gdb
(gdb) dump_all_sandboxes
```

### Inspecting the core dump

After the core dump has been created, to inspect the state of the guest, load the core dump file using `gdb` or `lldb`.
**NOTE: This feature has been tested with version `15.0` of `gdb` and version `17` of `lldb`, earlier versions may not work, it is recommended to use these versions or later.**

#### Using gdb

Load the core dump alongside the guest binary that was running when the crash occurred:

```bash
gdb <guest-binary> -c <core-dump-file>
```

For example:

```bash
gdb src/tests/rust_guests/bin/debug/simpleguest -c /tmp/hl_dumps/hl_core_20260225_T165358.517.elf
```

Common commands for inspecting the dump:

```gdb
# View all general-purpose registers (rip, rsp, rflags, etc.)
(gdb) info registers

# Disassemble around the crash site
(gdb) x/10i $rip

# View the stack
(gdb) x/16xg $rsp

# Backtrace (requires debug info in guest binary)
(gdb) bt

# List all memory regions in the dump (snapshot, scratch, mapped regions)
(gdb) info files

# Read memory at a specific address
(gdb) x/s <address>        # null-terminated string
(gdb) x/32xb <address>     # 32 bytes in hex
```

See the `crashdump` example (`cargo run --example crashdump --features crashdump`)
for a runnable demonstration of both automatic and on-demand crash dumps.

#### Using VSCode

To do this in vscode, the following configuration can be used to add debug configurations:

```vscode
{
    "version": "0.2.0",
    "inputs": [
        {
            "id": "core_dump",
            "type": "promptString",
            "description": "Path to the core dump file",
        },
        {
            "id": "program",
            "type": "promptString",
            "description": "Path to the program to debug",
        }
    ],
    "configurations": [
        {
            "name": "[GDB] Load core dump file",
            "type": "cppdbg",
            "request": "launch",
            "program": "${input:program}",
            "coreDumpPath": "${input:core_dump}",
            "cwd": "${workspaceFolder}",
            "MIMode": "gdb",
            "externalConsole": false,
            "miDebuggerPath": "/usr/bin/gdb",
            "setupCommands": [
            {
                "description": "Enable pretty-printing for gdb",
                "text": "-enable-pretty-printing",
                "ignoreFailures": true
            },
            {
                "description": "Set Disassembly Flavor to Intel",
                "text": "-gdb-set disassembly-flavor intel",
                "ignoreFailures": true
            }
            ]
        },
        {
        "name": "[LLDB] Load core dump file",
        "type": "lldb",
        "request": "launch",
        "stopOnEntry": true,
        "processCreateCommands": [],
        "targetCreateCommands": [
            "target create -c ${input:core_dump} ${input:program}",
        ],
        },
    ]
}
```
**NOTE: The `CodeLldb` debug session does not stop after launching. To see the code, stack frames and registers you need to
press the `pause` button. This is a known issue with the `CodeLldb` extension [#1245](https://github.com/vadimcn/codelldb/issues/1245).
The `cppdbg` extension works as expected and stops at the entry point of the program.**

## Compiling guests with debug information for release builds

This section explains how to compile a guest with debugging information but still have optimized code, and how to separate the debug information from the binary.

### Creating a release build with debug information

To create a release build with debug information, you can add a custom profile to your `Cargo.toml` file:

```toml
[profile.release-with-debug]
inherits = "release"
debug = true
```

This creates a new profile called `release-with-debug` that inherits all settings from the release profile but adds debug information.

### Splitting debug information from the binary

To reduce the binary size while still having debug information available, you can split the debug information into a separate file.
This is useful for production environments where you want smaller binaries but still want to be able to debug crashes.

Here's a step-by-step guide:

1. Build your guest with the release-with-debug profile:
   ```bash
   cargo build --profile release-with-debug
   ```

2. Locate your binary in the target directory:
   ```bash
   TARGET_DIR="target"
   PROFILE="release-with-debug"
   ARCH="x86_64-unknown-none" # Your target architecture
   BUILD_DIR="${TARGET_DIR}/${ARCH}/${PROFILE}"
   BINARY=$(find "${BUILD_DIR}" -type f -executable -name "guest-binary" | head -1)
   ```

3. Extract debug information into a full debug file:
   ```bash
   DEBUG_FILE_FULL="${BINARY}.debug.full"
   objcopy --only-keep-debug "${BINARY}" "${DEBUG_FILE_FULL}"
   ```

4. Create a symbols-only debug file (smaller, but still useful for stack traces):
   ```bash
   DEBUG_FILE="${BINARY}.debug"
   objcopy --keep-file-symbols "${DEBUG_FILE_FULL}" "${DEBUG_FILE}"
   ```

5. Strip debug information from the original binary but keep function names:
   ```bash
   objcopy --strip-debug "${BINARY}"
   ```

6. Add a debug link to the stripped binary:
   ```bash
   objcopy --add-gnu-debuglink="${DEBUG_FILE}" "${BINARY}"
   ```

After these steps, you'll have:
- An optimized binary with function names for basic stack traces
- A symbols-only debug file for stack traces
- A full debug file for complete source-level debugging

### Analyzing core dumps with the debug files

When you have a core dump from a crashed guest, you can analyze it with different levels of detail using either GDB or LLDB.

#### Using GDB

1. For basic analysis with function names (stack traces):
   ```bash
   gdb ${BINARY} -c /path/to/core.dump
   ```

2. For full source-level debugging:
   ```bash
   gdb -s ${DEBUG_FILE_FULL} ${BINARY} -c /path/to/core.dump
   ```

#### Using LLDB

LLDB provides similar capabilities with slightly different commands:

1. For basic analysis with function names (stack traces):
   ```bash
   lldb ${BINARY} -c /path/to/core.dump
   ```

2. For full source-level debugging:
   ```bash
   lldb -o "target create -c /path/to/core.dump ${BINARY}" -o "add-dsym ${DEBUG_FILE_FULL}"
   ```

3. If your debug symbols are in a separate file:
   ```bash
   lldb ${BINARY} -c /path/to/core.dump
   (lldb) add-dsym ${DEBUG_FILE_FULL}
   ```

### VSCode Debug Configurations

You can configure VSCode (in `.vscode/launch.json`) to use these files by modifying the debug configurations:

#### For GDB

```json
{
    "name": "[GDB] Load core dump with full debug symbols",
    "type": "cppdbg",
    "request": "launch",
    "program": "${input:program}",
    "coreDumpPath": "${input:core_dump}",
    "cwd": "${workspaceFolder}",
    "MIMode": "gdb",
    "externalConsole": false,
    "miDebuggerPath": "/usr/bin/gdb",
    "setupCommands": [
        {
            "description": "Enable pretty-printing for gdb",
            "text": "-enable-pretty-printing",
            "ignoreFailures": true
        }
    ]
}
```

#### For LLDB

```json
{
    "name": "[LLDB] Load core dump with full debug symbols",
    "type": "lldb",
    "request": "launch",
    "program": "${input:program}",
    "cwd": "${workspaceFolder}",
    "processCreateCommands": [],
    "targetCreateCommands": [
        "target create -c ${input:core_dump} ${input:program}"
    ],
    "postRunCommands": [
        // if debug symbols are in a different file
        "add-dsym ${input:debug_file_path}"
    ]
}
```

---

## File: docs/how-to-make-releases.md

# Releasing a new Hyperlight version to Cargo

This document details the process of releasing a new version of Hyperlight to [crates.io](https://crates.io). It's intended to be used as a checklist for the developer doing the release. The checklist is represented in the below sections.

## Update Cargo.toml Versions

Currently, we need to manually update the workspace `Cargo.toml` version number to match to whatever release we are making. This will affect the version of all the crates in the workspace.

> Note: we'll use `v0.4.0` as the version for the above and all subsequent instructions. You should replace this with the version you're releasing. Make sure your version follows [SemVer](https://semver.org) conventions as closely as possible, and is prefixed with a `v` character. *In particular do not use a patch version unless you are patching an issue in a release branch, releases from main should always be minor or major versions*.

Create a PR with this change and merge it into the main branch.

## Update `CHANGELOG.md`

The `CHANGELOG.md` file is a critical document used to track changes made to Hyperlight. It serves as the foundation for generating release notes, so it's essential to keep it up to date with each release. While not every change needs to be added to this file (since a complete changelog of all PRs will be automatically generated), it's crucial to include all significant and user-facing updates. The changelog entries must follow [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) format.

### Steps to Update `CHANGELOG.md`:

- **Manually update the `CHANGELOG.md`** with important changes since the latest release. Ideally, contributors should update this file as part of their PR, but this may not always happen.
  
- **Rename the `[Prerelease] - Unreleased` section** to reflect the new version number (if not already done). Ensure that it links to the GitHub comparison between the current and previous versions. For example, `v0.2.0` should link to `https://github.com/hyperlight-dev/hyperlight/compare/v0.1.0...v0.2.0` (see the footer of `CHANGELOG.md`).

- **Add a new `[Prerelease]` section** at the top of the file. This section should initially be empty and will track changes for the next release.

- **Preview the automatically generated release notes** locally using the command:  
  `just create-release-notes v0.4.0 > notes.md`. Review the notes to ensure everything looks accurate.

- **Create a PR** with the updated `CHANGELOG.md` and merge it into the main branch once all changes are confirmed.

## Create a tag

When both above PRs has merged into `main` branch you should create a tag. ***Make sure you have pulled the recently updated `main` branch***, and do the following on the `main` branch:

```bash
git tag -a v0.4.0 -m "A brief description of the release"
git push origin v0.4.0 # if you've named your git remote for the hyperlight-dev/hyperlight repo differently, change 'origin' to your remote name
```

If you are creating a patch release see the instructions [here](#patching-a-release).

## Create a release branch (no manual steps)

After you push your new tag in the previous section, the ["Create a Release Branch"](https://github.com/hyperlight-dev/hyperlight/actions/workflows/CreateReleaseBranch.yml) CI job will automatically run. When this job completes, a new `release/v0.4.0` branch will be automatically created for you.

## Create a new GitHub release

After the previous CI job runs to create the new release branch, go to the ["Create a Release"](https://github.com/hyperlight-dev/hyperlight/actions/workflows/CreateRelease.yml). GitHub actions workflow and do the following:

1. Click the "Run workflow" button near the top right
2. In the Use workflow from dropdown, select the `release/v0.4.0` branch
3. Click the green **Run workflow** button

> Note: In case you see a "Create a Release" job already running before starting this step, that is because the "Create a Release" workflow also automatically runs on push to `main` branch to create a pre-release. You must still do the steps outlined above.

When this job is done, a new [GitHub release](https://github.com/hyperlight-dev/hyperlight/releases) will be created for you. This job also publishes the following rust packages to the crates.io:
- `hyperlight-common`
- `hyperlight-guest`
- `hyperlight-guest-macro`
- `hyperlight-guest-bin`
- `hyperlight-component-util`
- `hyperlight-component-macro`
- `hyperlight-host`
- `hyperlight-guest-tracing`

## Patching a release
> Note: for this example we'll assume `v0.4.0` is already released, and you want to release a `v0.4.1` patch for it.

1. Make sure the patch/patches you want include are already merged to `main` branch.
2. Make sure `CHANGELOG.md` is updated with the changes you want to include in the patch release (see instructions above) and is merged to `main` branch.
3. Make sure the `Cargo.toml` versions are updated to the new patch version (e.g. `v0.4.1`), and is merged to `main` branch.
4. Manually create a new branch **from the `release/v0.4.0` branch** and name it `release/v0.4.1`. Important: do not create the branch from `main` branch.
5. Cherry-pick the commits from `main` that you want to include in the patch (and resolve any conflicts). You must include the commit that updated the `CHANGELOG.md` and the commit that updated the `Cargo.toml` versions.
6. Create a tag for the patch release, e.g. `v0.4.1`, similar to the steps above. Push the tag. A job will start to try to make a new release branch for you, but it will fail because the `release/v0.4.1` branch already exists. This is expected, so don't worry about it.
7. Follow the steps above to create a new GitHub release, but this time select the `release/v0.4.1` branch in the "Use workflow from" dropdown. This will create a new GitHub release for you, and publish the updated packages to crates.io.

---

## File: docs/how-to-use-flatbuffers.md

# How to use FlatBuffers

> Note: the last generation of the flatbuffer code was with done with flatc version 25.9.23 (i.e., the last version as of Oct 2nd, 2025).

Flatbuffers is used to serialize and deserialize some data structures.

Schema files are used to define the data structures and are used to generate the code to serialize and deserialize the data structures.

Those files are located in the [`schema`](../src/schema) directory.

Code generated from the schema files is checked in to the repository, therefore you only need to generate the code if you change an existing schema file or add a new one. You can find details on how to update schema files [here](https://google.github.io/flatbuffers/flatbuffers_guide_writing_schema.html).

## Generating code

We use [flatc](https://google.github.io/flatbuffers/flatbuffers_guide_using_schema_compiler.html) to generate rust code.

We recommend building `flatc` from source. To generate rust code, use

```console
just gen-all-fbs-rust-code
```

---

## File: docs/hyperlight-execution-details.md

# How code is run inside a VM

This document details how VMs are very quickly and efficiently created and configured to run arbitrary code.

## Background

Hyperlight is a library for creating micro virtual machines (VMs) intended for executing small, short-running functions. This use case is different from that of many other VM platforms, which are aimed at longer-running, more complex workloads.

A very rough contrast between Hyperlight's offerings and other platforms is as follows:

| Feature                                                                 | Hyperlight | Other platforms    |
|-------------------------------------------------------------------------|------------|--------------------|
| Hardware isolation (vCPU, virtual memory)                               | Yes        | Yes                |
| Shared memory between host and in-VM process                            | Yes        | Yes <sup>[2]</sup> |
| Lightweight function calls between host and in-VM process (the "guest") | Yes        | No                 |
| Bootloader/OS kernel                                                    | No         | Yes <sup>[1]</sup> |
| Virtual networking                                                      | No         | Yes <sup>[2]</sup> |
| Virtual filesystem                                                      | No         | Yes <sup>[1]</sup> |


As seen in this table, Hyperlight offers little more than a CPU and memory. We've removed every feature we could, while still providing a machine on which arbitrary code can execute, so we can achieve our various use cases and efficiency targets.

## How code runs

With this background in mind, it's well worth focusing on the "lifecycle" of a VM -- how, exactly, a VM is created, modified, loaded, executed, and ultimately destroyed.

At the highest level, Hyperlight takes roughly the following steps to create and run arbitrary code inside a VM:

1. Loads a specially built, statically linked binary (currently, the [PE](https://en.wikipedia.org/wiki/Portable_Executable) and [ELF](https://en.wikipedia.org/wiki/Executable_and_Linkable_Format) executable formats are supported) into memory. This is the code that is executed inside a virtual machine.
2. Allocates additional memory regions, for example stack and heap for the guest, as well as some regions used for communication between the host and the guest.
3. Creates a Virtual Machine and maps shared memory into it
5. Create one virtual CPU (vCPU) within the newly created VM
6. Write appropriate values to the new vCPUs registers.
7. In a loop, tell previously created vCPU to run until we reach a halt message, one of several known error states, or an unsupported message
   1. In the former case, exit successfully
   2. In any of the latter cases, exit with a failure message

---

_<sup>[1]</sup> nearly universal support_

_<sup>[2]</sup> varied support_

---

## File: docs/hyperlight-metrics-logs-and-traces.md

# Observability

Hyperlight provides the following observability features:

* [Metrics](#metrics) are provided using the [metrics](https://docs.rs/metrics/latest/metrics/index.html) crate, which is a lightweight metrics facade.
* [Logs](#logs) are provided using the Rust [tracing crate](https://docs.rs/tracing/latest/tracing/) with the `log` feature enabled, allowing logs to be consumed by any Rust logger implementation compatible with the [log crate](https://docs.rs/log/latest/log/).
* [Tracing](#tracing) is provided using the Rust [tracing crate](https://docs.rs/tracing/0.1.37/tracing/), and can be consumed by any Rust tracing implementation. In addition, the [log feature](https://docs.rs/tracing/latest/tracing/#crate-feature-flags) is enabled which means that should a hyperlight host application not want to consume tracing events, you can still consume them as logs.

## Metrics

Metrics are provided using the [metrics](https://docs.rs/metrics/latest/metrics/index.html) crate, which is a lightweight metrics facade. When an executable installs a [recorder](https://docs.rs/metrics/latest/metrics/trait.Recorder.html), Hyperlight will emit its metrics to that recorder, which allows library authors to seamless emit their own metrics without knowing or caring which exporter implementation is chosen, or even if one is installed. In case where no recorder is installed, the metrics will be emitted to the default recorder, which is a no-op implementation with minimal overhead.

There are many different implementations of recorders. One example is the [prometheus exporter](https://docs.rs/metrics-exporter-prometheus/latest/metrics_exporter_prometheus/) which can be used to export metrics to a Prometheus server. An example of how to use this is provided in the [examples/metrics](../src/hyperlight_host/examples/metrics) directory.

The following metrics are provided and are enabled by default:

* `guest_errors_total` - Counter that tracks the number of guest errors by error code.
* `guest_cancellations_total` - Counter that tracks the number of guest executions that have been cancelled because the execution time exceeded the time allowed.

The following metrics are provided but are disabled by default:

* `guest_call_duration_seconds` - Histogram that tracks the execution time of guest functions in seconds by function name. The histogram also tracks the number of calls to each function.
* `host_call_duration_seconds` - Histogram that tracks the execution time of host functions in seconds by function name. The histogram also tracks the number of calls to each function.

The rationale for disabling the function call metrics by default is that:
* A Hyperlight host may wish to provide its own metrics for function calls.
* Enabling a trace subscriber will cause the function call metrics to be emitted as trace events, which may be sufficient for some use cases.
* These 2 metrics require string clones for the function names, which may be too expensive for some use cases.
We might consider enabling these metrics by default in the future.

## Logs

Hyperlight provides logs using the Rust [tracing crate](https://docs.rs/tracing/latest/tracing/) with the [`log` feature](https://docs.rs/tracing/latest/tracing/#crate-feature-flags) enabled. This means log events can be consumed by any Rust logger implementation compatible with the [log crate](https://docs.rs/log/latest/log/). To consume logs, the host application must provide a logger implementation either by using the `set_logger` function directly or using a logger implementation that is compatible with the log crate.

For an example that uses the `env_logger` crate, see the [examples/logging](../src/hyperlight_host/examples/logging) directory. By default, the `env_logger` crate will only log messages at the `error` level or higher. To see all log messages, set the `RUST_LOG` environment variable to `debug`.

Hyperlight also provides tracing capabilities (see below for more details), if no trace subscriber is registered, trace records will be emitted as log records, using the `log` feature of the [tracing crate](https://docs.rs/tracing/latest/tracing/#crate-feature-flags).

## Tracing

Tracing spans are created for any call to a public API and the parent span will be set to the current span in the host if one exists, the level of the span is set to `info`. The span will be closed when the call returns. Any Result that contains an error variant will be logged as an error event. In addition to the public APIs, all internal functions are instrumented with trace spans at the `trace` level, therefore in order to see full trace information, the trace level should be enabled.

Hyperlight provides tracing using the Rust [tracing crate](https://docs.rs/tracing/0.1.37/tracing/), and can be consumed by any Rust trace subscriber implementation(see[here](https://docs.rs/tracing/latest/tracing/index.html#related-crates) for some examples). When no tracing subscriber is set, trace events are automatically forwarded to the `log` facade via the `log` feature of the `tracing` crate, so consumers using only a `log` logger will still receive these events.

There are two examples that show how to consume both tracing events and log records as tracing events.

### Using tracing_forest

In the [examples/tracing](../src/hyperlight_host/examples/tracing) directory, there is an example that shows how to capture and output trace and log information using the tracing_forest crate. With this example the following commands can be used to set the verbosity of the trace output to `INFO` and run the example:

#### Linux

```bash
RUST_LOG='none,hyperlight_host=info,tracing=info' cargo run --example tracing
```

#### Windows

```powershell
$env:RUST_LOG='none,hyperlight_host=info,tracing=info'; cargo run --example tracing
```

### Using OTLP exporter and Jaeger

In the [examples/tracing-otlp](../src/hyperlight_host/examples/tracing-otlp) directory, there is an example that shows how to capture and send trace and log information to an otlp_collector using the opentelemetry_otlp crate. With this example the following commands can be used to set the verbosity of the trace output to `INFO` and run the example to generate trace data:

#### Linux

```bash
RUST_LOG='none,hyperlight_host=info,tracing=info' cargo run --example tracing-otlp
```

#### Windows

```powershell
$env:RUST_LOG='none,hyperlight_host=info,tracing=info';cargo run --example tracing-otlp
```

The sample will run and generate trace data until any key is pressed.

To view the trace data, leave the example running and use the jaegertracing/all-in-one container image with the following command:

```console
 docker run -p 16686:16686 -p 4317:4317 -p 4318:4318 -e COLLECTOR_OTLP_ENABLED=true jaegertracing/all-in-one:latest
```

NOTE: when running this on windows that this is a linux container, so you will need to ensure that docker is configured to run linux containers using WSL2. Alternatively, you can download the Jaeger binaries from [here](https://www.jaegertracing.io/download/). Extract the archive and run the `jaeger-all-in-one` executable as follows:

```powershell
.\jaeger-all-in-one.exe
```

Once the container or the exe is running, the trace output can be viewed in the jaeger UI at [http://localhost:16686/search](http://localhost:16686/search).

## Guest Tracing, Unwinding, and Memory Profiling

Hyperlight provides advanced observability features for guest code running inside micro virtual machines. You can enable guest-side tracing, stack unwinding, and memory profiling using the `trace_guest` and `mem_profile` features. This section explains how to build, run, and inspect guest traces.

The following features are available for guest tracing:
- `trace_guest`: Enables tracing for guest code, capturing function calls and execution time.
- `mem_profile`: Enables memory profiling for guest code with stack unwinding, capturing memory allocations and usage.

### Building a Guest with Tracing Support

To build a guest with tracing enabled, use the following commands:

```bash
just build-rust-guests debug trace_guest
just move-rust-guests debug
```

This builds the guest binaries with the `trace_guest` feature enabled and move them to the appropriate location for use by the host.

**NOTE**: To enable the tracing in your application you need to use the `trace_guest` feature on the `hyperlight-guest-bin` and `hyperlight-guest` crates.

### Running a Hyperlight Example with Guest Tracing

Once the guest is built, you can run a Hyperlight example with guest tracing enabled. For example:

```bash
RUST_LOG="info,hyperlight_host::sandbox=info,hyperlight_guest=trace,hyperlight_guest_bin=trace" cargo run --example tracing-otlp --features trace_guest
```

This will execute the `tracing-otlp` example, loading the guest with tracing enabled.
During execution, trace data will be collected on the host and exported as `opentelemetry` spans/events.

You can set up a collector to gather all the traces and inspect the traces from both host and guests.

Due to the nature of execution inside a Sandbox, on a call basis, the guest tracing sets up a stack of spans to keep track of the correct parents for the incoming
guest spans.
We start with `call-to-guest` which contains all the spans coming from a guest. Additionally, for each exit into the host, we add another layer marking it with
a `call-to-host` span to follow the execution in the host and correctly set it as a child of the active span in the guest.
This logic simulates the propagation of `Opentelemetry` context that is usually done between two services, but cannot be done here seamlessly because the guest side
runs `no_std` which `opentelemetry` doesn't know.

#### How it works

##### Guest

When the guest starts executing the `entrypoint` function, it receives a `max_log_level` parameter that tells the guest what kind of logging level is expected from it.

The `trace_guest` logic takes advantage of this parameter and when the `max_log_level` is `trace`, it allocates a custom made `GuestSubscriber` that implements the `Subscriber`
trait from `tracing_core` that allows defining a subscriber for the `tracing` crate to handle new spans and events.

This custom subscriber stores the spans and events in a buffer initialized only when tracing is enabled. For each new span and event, a method is called on the custom subscriber which not only stores the data, but also keeps track of the hierarchy and dependencies between the other spans/events.
**NOTE**: The spans/events attributes are truncated to fit in the allocated buffer.

When the storage space is filled, the guest triggers a VM Exit that sends the guest pointers to the host. The host can access the guest memory, get the data and parse it to create the `spans` and `events` using the `opentelemetry` crate which allows specifying the starting and ending timestamps
which are captured in the guest using the `TSC`.

To improve performance, for each VMExit, the guest adds metadata for the host to be able to report the tracing data and free space.

##### Host

When a guest exits, the host checks for metadata from the guest reporting tracing data.
If tracing data is found, the host starts parsing it and reconstructing a tree which represents the spans hierarchy.

Additionally, the host also adds new children `span`s to the guest's reported active span, emphasizing the spans created on the host as a result of a temporary VM Exit. This helps visualize a call into the guest with context propagated across the VM boundary.

The host creates `opentelemetry` spans and events for each guest span and event reported.

### Inspecting Guest memory Trace Files (for mem_profile)

To inspect the trace file generated by the guest, use the `trace_dump` crate. You will need the path to the guest symbols and the trace file. Run the following command:

```bash
cargo run -p trace_dump   list_frames
```

Replace `` with the path to the guest binary or symbol file, and `` with the path to the trace file in the `trace` directory.

This command will list the stack frames and tracing information captured during guest execution, allowing you to analyze guest behavior, stack traces, and memory usage.

#### Example

```bash
cargo run -p trace_dump ./src/tests/rust_guests/bin/debug/simpleguest ./trace/.trace list_frames
```

You can use the `mem_profile` additional feature by enabling them during the build and run steps.

> **Note:** Make sure to follow the build and run steps in order, and ensure that the guest binaries are up to date before running the host example.

## System Prerequisites for `trace_dump`

To build and use the `trace_dump` crate and related guest tracing features, you must have the following system libraries and development tools installed on your system:

- **glib-2.0** development files  
  - Fedora/RHEL/CentOS:  
    ```bash
    sudo dnf install glib2-devel pkgconf-pkg-config
    ```
- **cairo** and **cairo-gobject** development files  
  - Fedora/RHEL/CentOS:  
    ```bash
    sudo dnf install cairo-devel cairo-gobject-devel
    ```
- **pango** development files  
  - Fedora/RHEL/CentOS:  
    ```bash
    sudo dnf install pango-devel
    ```

These libraries are required by Rust crates such as `glib-sys`, `cairo-sys-rs`, and `pango-sys`, which are dependencies of the tracing and visualization tools. If you encounter errors about missing `.pc` files (e.g., `glib-2.0.pc`, `cairo.pc`, `pango.pc`), ensure the corresponding `-devel` packages are installed.

--- METRICS ---
- Files Extracted: 11
- Estimated Token Budget: ~16809 tokens
- Recency Window: Active (< 180 days)
- Canonical Reference: https://codewiki.google/github.com/hyperlight-dev/hyperlight