incus Documentation

Technical Documentation: lxc/incus

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Incus

Incus is a modern, secure and powerful system container and virtual machine manager.

It provides a unified experience for running and managing full Linux systems inside containers or virtual machines. Incus supports images for a large number of Linux distributions (official Ubuntu images and images provided by the community) and is built around a very powerful, yet pretty simple, REST API. Incus scales from one instance on a single machine to a cluster in a full data center rack, making it suitable for running workloads both for development and in production.

Incus allows you to easily set up a system that feels like a small private cloud. You can run any type of workload in an efficient way while keeping your resources optimized.

You should consider using Incus if you want to containerize different environments or run virtual machines, or in general run and manage your infrastructure in a cost-effective way.

You can try Incus online at: https://linuxcontainers.org/incus/try-it/

Project history Copy Chapter

Incus, which is named after the Cumulonimbus incus or anvil cloud
started as a community fork of Canonical's LXD following Canonical's takeover of the LXD project from the
Linux Containers community.

The project was then adopted by the Linux Containers community, taking back the spot left empty by LXD's departure.

Incus is a true open source community project, free of any CLA and
remains released under the Apache 2.0 license.
It's maintained by the same team of developers that first created LXD.

Get started Copy Chapter

See Getting started in the Incus documentation for installation instructions and first steps.

• Release announcements: https://discuss.linuxcontainers.org/c/news/
• Release tarballs: https://github.com/lxc/incus/releases/
• Documentation: https://linuxcontainers.org/incus/docs/main/

Status Copy Chapter

Security Copy Chapter

Consider the following aspects to ensure that your Incus installation is secure:

• Keep your operating system up-to-date and install all available security patches.
• Use only supported Incus versions.
• Restrict access to the Incus daemon and the remote API.
• Do not use privileged containers unless required. If you use privileged containers, put appropriate security measures in place. See the LXC security page for more information.
• Configure your network interfaces to be secure.

See Security for detailed information.

IMPORTANT:

Local access to Incus through the Unix socket always grants full access to Incus.
This includes the ability to attach file system paths or devices to any instance as well as tweak the security features on any instance.

Therefore, you should only give such access to users who you'd trust with root access to your system.

Support and community Copy Chapter

The following channels are available for you to interact with the Incus community.

Bug reports Copy Section

You can file bug reports and feature requests at: https://github.com/lxc/incus/issues/new

Community support Copy Section

Community support is handled at: https://discuss.linuxcontainers.org

Commercial support Copy Section

Commercial support is currently available from Zabbly for users of their Debian or Ubuntu packages.

Documentation Copy Chapter

The official documentation is available at: https://github.com/lxc/incus/tree/main/doc

Contributing Copy Chapter

Fixes and new features are greatly appreciated. Make sure to read our contributing guidelines first!

Overview (`doc/README.md`) Copy Section

Incus documentation

The Incus documentation is available at: https://linuxcontainers.org/incus/docs/main/

GitHub provides a basic rendering of the documentation as well, but important features like includes and clickable links are missing. Therefore, we recommend reading the published documentation.

Documentation framework Copy Chapter

Incus' documentation is built with Sphinx.

It is written in Markdown with MyST extensions.
For syntax help and guidelines, see the documentation cheat sheet (source).

For structuring, the documentation uses the Diátaxis approach.

Build the documentation Copy Chapter

To build the documentation, run make doc from the root directory of the repository.
This command installs the required tools and renders the output to the doc/html/ directory.
To update the documentation for changed files only (without re-installing the tools), run make doc-incremental.

Before opening a pull request, make sure that the documentation builds without any warnings (warnings are treated as errors).
To preview the documentation locally, run make doc-serve and go to http://localhost:8001 to view the rendered documentation.

Overview (`test/README.md`) Copy Section

How to run

To run all tests, including the Go tests, run from repository root:

sudo -E make check

To run only the integration tests, run from the test directory:

sudo -E ./main.sh

Environment variables

Overview (`cmd/generate-config/README.md`) Copy Section

generate-config

A small CLI to parse comments in a Golang codebase meant to be used for a documentation tool (like Sphinx for example).

It parses the comments from the AST and extracts their documentation.

Disclaimer Copy Chapter

generate-config is intended for internal use within the
Incus code base. There are no guarantees regarding
backwards compatibility, API stability, or long-term availability. It may change
or be removed at any time without prior notice. Use at your own discretion.

Usage Copy Chapter

$ generate-config -h
Usage of generate-config:
-e value
    Path that will be excluded from the process

Formatting Copy Chapter

A comment is formatted this way:

/* Detailed source-code truncated for AI context efficiency. */

The go-swagger spec from source generator can only handles swagger:meta (global file/package level documentation), swagger:route (API endpoints), swagger:params (function parameters), swagger:operation (method documentation), swagger:response (API response content documentation), swagger:model (struct documentation) generation. In our use case, we would want a config variable spec generator that can bundle any key-value data pairs alongside metadata to build a sense of hierarchy and identity (we want to associate a unique key to each gendoc comment group that will also be displayed in the generated documentation)

In a swagger fashion, generate-config can associate metadata key-value pairs (here for example, group and key) to data key-value pairs. As a result, it can generate a YAML tree out of the code documentation and also a Markdown document.

Output Copy Section

Here is the JSON output of the example shown above:

/* Detailed source-code truncated for AI context efficiency. */

Here is the .txt output of the example shown above:

\`\`\`{config:option} user.* cluster
:type: string
:liveupdate: `yes`
:shortdesc: Free form user key/value storage (can be used in search).
:condition: container
:default: -

This is the real long desc.

With two paragraphs.

And a list:

- Item
- Item
- Item

example of a table:

Key                                 | Type      | Scope     | Default                                          | Description
:--                                 | :---      | :----     | :------                                          | :----------
`acme.agree_tos`                    | bool      | global    | `false`                                          | Agree to ACME terms of service
`acme.ca_url`                       | string    | global    | `https://acme-v02.api.letsencrypt.org/directory` | URL to the directory resource of the ACME service
`acme.domain`                       | string    | global    | -                                                | Domain for which the certificate is issued
`acme.email`                        | string    | global    | -                                                | Email address used for the account registration


\`\`\`

\`\`\`{config:option} scheduler.instance cluster
:liveupdate: `yes`
:shortdesc: Possible values are all, manual and group. See Automatic placement of instances for more information.
:condition: container
:default: `all`
:type: integer


\`\`\`

Overview (`cmd/generate-database/README.md`) Copy Section

`generate-database`

Introduction Copy Chapter

generate-database is a database statement and associated go function generator
for Incus and related projects. generate-database utilizes go's code generation
directives (//go:generate ...) alongside go's ast
and types packages for parsing the syntax tree for
go structs and variables. We use generate-database for the majority of our
SQL statements and database interactions on the go side for consistency and
predictability.

Disclaimer Copy Chapter

generate-database is intended for internal use within the
Incus code base. There are no guarantees regarding
backwards compatibility, API stability, or long-term availability. It may change
or be removed at any time without prior notice. Use at your own discretion.

Usage Copy Chapter

Initialization Copy Section

Package global

Once per package, that uses generate-database for generation of database
statements and associated go functions, generate-database needs to be invoked
using the following go:generate instruction:

//go:generate generate-database db mapper generate

This will initiate a call to generate-database db mapper generate,
which will then search for //generate-database:mapper directives in the same file
and process those.

The following flags are available:

• --package / -p: Package import paths to search for structs to parse. Defaults to the caller package. Can be used more than once.

File

Generally the first thing we will want to do for any newly generated file is to
ensure the file has been cleared of content:

//generate-database:mapper target instances.mapper.go
//generate-database:mapper reset -i -b "//go:build linux && cgo && !agent"

Generation Directive Arguments Copy Section

The generation directive arguments have the following form:

//generate-database:mapper <command> flags <kind> <args...>

The following flags are available:

• --build / -b: build comment to include (commands: reset)
• --interface / -i: create interface files (commands: reset, method)
• --entity / -e: database entity to generate the method or statement for (commands: stmt, method)

Example:

• //generate-database:mapper stmt -e instance objects table=table_name

The table key can be used to override the generated table name for a specified one.

• //generate-database:mapper method -i -e instance Create references=Config,Device

For some tables (defined below under Additional Information as EntityTable, the references=<ReferenceEntity> key can be provided with the name of
a ReferenceTable or MapTable struct. This directive would produce CreateInstance in addition to CreateInstanceConfig and CreateInstanceDevices:

• //generate-database:mapper method -i -e instance_profile Create struct=Instance
• //generate-database:mapper method -i -e instance_profile Create struct=Profile

For some tables (defined below under Additional Information as AssociationTable, method declarations must
include a struct=<Entity> to indicate the directionality of the function. An invocation can be called for each direction.
This would produce CreateInstanceProfiles and CreateProfileInstances respectively.

SQL Statement Generation Copy Section

SQL generation supports the following SQL statement types:

Examples

//generate-database:mapper stmt -e instance objects
//generate-database:mapper stmt -e instance objects-by-Name-and-Project
//generate-database:mapper stmt -e instance create
//generate-database:mapper stmt -e instance update
//generate-database:mapper stmt -e instance delete-by-Name-and-Project

Statement Related Go Tags

There are several tags that can be added to fields of a struct that will be parsed by the ast package.

Go Function Generation Copy Section

Go function generation supports the following types:

//generate-database:mapper method -i -e instance GetMany
//generate-database:mapper method -i -e instance GetOne
//generate-database:mapper method -i -e instance ID
//generate-database:mapper method -i -e instance Exist
//generate-database:mapper method -i -e instance Create
//generate-database:mapper method -i -e instance Update
//generate-database:mapper method -i -e instance DeleteOne-by-Project-and-Name
//generate-database:mapper method -i -e instance DeleteMany-by-Name

Additional Information Copy Section

All structs should have an ID field, as well as an additional Filter struct prefixed with the original struct name.
This should include any fields that should be considered for filtering in WHERE clauses.
These fields should be pointers to facilitate omission and inclusion without setting default values.

Example:

type Instance struct {
  ID int
  Name string
  Project string
}

type InstanceFilter struct {
  Name *string
  Project *string
}

generate-database will handle parsing of structs differently based on the composition of the struct in four different ways.

Non-EntityType structs will only support GetMany, Create, Update, and Delete functions.

EntityTable Copy Section

Most structs will get treated this way, and represent a normal table.

• If a table has an associated table for which a ReferenceTable or MapTable as defined below is applicable, functions specific to this entity can be generated by
  including a comma separated list to references=<OtherEntity> in the code generation directive for GetMany, Create, or Update directives.

• The Create method directive for EntityTable will expect on the ID and Exist method directives to be present.

• All CREATE, UPDATE, and DELETE statements that include a joined table will expect a var <entity>ID = RegisterStmt('SQL String') to exist for the joining table.

ReferenceTable Copy Section

A struct that contains a field named ReferenceID will be parsed this way.
generate-database will use this struct to generate more abstract SQL statements and functions of the form _.

The associated Filter struct may include a ReferenceID []int field.
This generates an IN clause matching on the parent column, with the integer values inlined into the query to avoid query parameter count limits.
A nil slice leaves the filter unset while an empty (non-nil) slice matches nothing.
When the field is present, the generated per-parent helpers and nested reference fetches are automatically scoped to the relevant parent IDs rather than fetching the whole table.

Real world invocation of these statements and functions should be done through an EntityTable method call with the tag references=. This EntityTable will replace the `` above.

Example:

//generate-database:mapper stmt -e device create
//generate-database:mapper method -e device Create

type Device struct {
  ID int
  ReferenceID int
  Name string
  Type string
}

//...
//generate-database:mapper method -e instance Create references=Device
// This will produce a function called `CreateInstanceDevices`.

MapTable Copy Section

This is a special type of ReferenceTable with fields named Key and Value.
On the SQL side, this is treated exactly like a ReferenceTable, but on the go side, the return values will be a map.

Example:

//generate-database:mapper stmt -e config create
//generate-database:mapper method -e config Create

type Config struct {
  ID int
  ReferenceID int
  Key string
  Value string
}

//...
//generate-database:mapper method -e instance Create references=Config
// This will produce a function called `CreateInstanceConfig`, which will return a `map[string]string`.

AssociationTable Copy Section

This is a special type of table that contains two fields of the form <Entity>ID, where <Entity> corresponds to two other structs present in the same package.
This will generate code for compound tables of the form <entity1>_<entity2> that are generally used to associate two tables together by their IDs.

method generation declarations for these statements should include a struct=<Entity> to indicate the directionality of the function.
An invocation can be called for each direction.

Example:

//generate-database:mapper method -i -e instance_profile Create struct=Instance
//generate-database:mapper method -i -e instance_profile Create struct=Profile

type InstanceProfile struct {
  InstanceID int
  ProfileID int
}

Overview (`test/mini-oidc/README.md`) Copy Section

mini-oidc

mini-oidc is an extremely basic OIDC provider which can be used with the incus command line.
It doesn't use web authentication and instead just automatically approves any authentication request.

Usage:

mini-oidc <port> [<user-file>]

By default, it will authenticate everyone as unknown, but this can be overridden by writing the username to be returned in the file named in the 2nd
(optional) argument. This effectively allows scripting a variety of users without having to deal with actual login.

The storage sub-package is a based on https://github.com/zitadel/oidc/tree/main/example/server/storage.
The following changes were made:

• Added IncusDeviceClient
• Added option to configure access token and refresh token expiry

Overview (`shared/resources/usbid/README.md`) Copy Section

This Go package is a variant of usbid from https://github.com/google/gousb.
It was written by the gousb maintainers and contributors and then adapted for our use.

Main changes:

• Doesn't load on import (reduced memory footprint).
• Uses system-local USB database (to always match lsusb).
• Doesn't import anything outside of built-in Go packages.
• Doesn't use or indirectly rely on CGO as this code is often used in cross-built static binaries.

Most users will want to stick to the upstream google/gousb version instead.
This fork is really meant for our special use case and may be further trimmed down in the future.

Doc Index (`doc/index.md`) Copy Section

Incus

Incus is a modern, secure and powerful system container and virtual machine manager.

% Include content from ../README.md

:start-after: 
    :end-before:

Security Copy Chapter

% Include content from ../README.md

:start-after: 
    :end-before:

See Security for detailed information.

% Include content from [../README.md](../README.md)
```{include} ../README.md
    :start-after: 
    :end-before: 
```

Project and community Copy Chapter

Incus is free software and developed under the Apache 2 license.
It’s an open source project that warmly welcomes community projects, contributions, suggestions, fixes and constructive feedback.

• Code of Conduct
• Contribute to the project
• Release announcements
• Release tarballs
• Get support
• Watch tutorials and announcements on YouTube
• Ask and answer questions on the forum

:hidden:
:titlesonly:

self
Getting Started </tutorial/first_steps>
General </general>
Client </client>
Server </server>
Instances </instances>
Storage </storage>
Networks </networks>
Images </images>
Projects </projects>
Clustering </clustering>
API </api>
Security </security>
Internals </internals>
Contributing </contributing>
External resources </external_resources>

Doc .Wordlist (`doc/.wordlist.txt`) Copy Section

AAAA
AAVMF
ABI
ACL
ACLs
AI
AIO
allocator
AMD
Ansible
Ansible's
API
APIs
AppArmor
ARMv
ARP
ASN
AXFR
backend
backends
backported
Backports
balancer
balancers
benchmarking
BFD
BGP
bibi
BitLocker
BMC
bool
bootable
BPF
Btrfs
bugfix
bugfixes
Centos
Ceph
CephFS
Ceph's
CFS
cgroup
cgroupfs
cgroups
checksum
checksums
Chocolatey
CIDR
CLI
Colima
COPR
Cowsql
CPUs
CRIU
CRL
cron
CSV
CUDA
customizable
dataset
DCO
dereferenced
devtmpfs
DHCP
DHCPv
Diátaxis
Diffie
diskful
diskless
Diskless
Distrobuilder
DMA
DNAT
DNS
dnsmasq
DNSSEC
DoS
DRBD
DRM
EB
Ebit
eBPF
ECDHE
EAB
ECDSA
ECMP
EDK
EiB
Eibit
endian
EPEL
ES
ESA
ETag
failover
formatters
FQDNs
FreeBSD
frontend
Furo
gapped
GARM
GARP
GbE
Gbit
Geneve
GiB
Gibit
GID
GIDs
Github
Golang
goroutines
GPG
GPUDirect
GPUs
Grafana
GUID
HAProxy
hardcoded
HDDs
Hellman
HMAC
Homebrew
hostname
hotplug
hotplugged
hotplugging
HTTPS
hwdata
ICMP
idmap
idmapped
idmaps
Incant
incrementing
Incus
Incus'
InfiniBand
InfluxDB
init
initramfs
integrations
IOMMU
IOPS
IOV
IPAM
IPs
IPv
IPVLAN
iSCSI
JIT
jq
JSON
JWT
kB
kbit
KiB
kibi
Kibit
Kubernetes
KVM
LINBIT
LINSTOR
LINSTOR's
LLM
LLMs
lookups
Loongarch
LRU
LTS
LV
LVM
LXC
LXCFS
LXC's
LXD
LXD's
macOS
macvlan
Makefile
manpages
Mbit
mDNS
MiB
Mibit
MicroCeph
MicroCloud
misconfigured
MCS
MII
MITM
MTU
Mullvad
multicast
multiport
MyST
namespace
namespaced
namespaces
NATed
natively
NDP
NetBSD
netmask
NFS
NIC
NIC's
NICs
NixOS
NTP
NUMA
NVMe
NVRAM
OCI
OData
OIDC
OpenFGA
OpenID
OpenMetrics
OpenSSL
openSUSE
OpenSUSE
OpenTofu
OSD
overcommit
overcommitting
overlayfs
OVMF
OVN
OVS
PackageHub
passthrough
Pbit
PEM
PCI
PCIe
PDU
peerings
Permalink
PFs
PiB
Pibit
PID
PKI
PNG
Podman
Pongo
POSIX
PPA
pre
preselects
preseed
proxied
Proxmox
proxying
PTS
PXE
qdisc
QEMU
qgroup
qgroups
QMP
RADOS
RBAC
RBD
RDNSS
README
reconfiguring
requestor
resolvers
RESTful
RHEL
rootfs
RSA
rST
RTC
runtime
SATA
scalable
scriptlet
SDN
SeaBIOS
Seccomp
SELinux
SEV
SFTP
SHA
shiftfs
SIGHUP
SIGTERM
simplestreams
SLAAC
SMTP
Snapcraft
snapshotted
SNAT
SOA
Solaris
SPAs
SPL
SquashFS
SSDs
SSL
Starlark
stateful
stderr
stdin
stdout
STP
struct
structs
subcommand
subcommands
subitem
subnet
subnets
subpage
substep
subtree
subtrees
subvolume
subvolumes
superset
SVG
symlink
symlinks
syscall
syscalls
sysfs
syslog
systemd
Tbit
TCP
Telegraf
Terraform
TiB
Tibit
TLS
tmpfs
toolchain
topologies
TPM
TrueNAS
TSIG
TTL
UDP
UEFI
UFW
UI
UID
UIDs
uncomment
unconfigured
unevictable
unixgram
unmanaged
unmount
unmounting
uplink
uptime
URI
URIs
userspace
UUID
vCPU
vCPUs
VE
VDPA
VFs
VFS
VirtIO
virtualize
virtualized
VLAN
VLANs
VM
VMs
VPD
VPN
VPS
VRF
vSwitch
VXLAN
webhook
WebSocket
WebSockets
Winget
XFS
XHR
YAML
YAML's
Zabbly
Zettabyte
ZFS
zpool
zpools

Doc Api Extensions (`doc/api-extensions.md`) Copy Section

API extensions

The changes below were introduced to the Incus API after the 1.0 API was finalized.

They are all backward compatible and can be detected by client tools by
looking at the api_extensions field in GET /1.0.

`storage_zfs_remove_snapshots` Copy Chapter

A storage.zfs_remove_snapshots daemon configuration key was introduced.

It's a Boolean that defaults to false and that when set to true instructs Incus
to remove any needed snapshot when attempting to restore another.

This is needed as ZFS will only let you restore the latest snapshot.

`container_host_shutdown_timeout` Copy Chapter

A boot.host_shutdown_timeout container configuration key was introduced.

It's an integer which indicates how long Incus should wait for the container
to stop before killing it.

Its value is only used on clean Incus daemon shutdown. It defaults to 30s.

`container_stop_priority` Copy Chapter

A boot.stop.priority container configuration key was introduced.

It's an integer which indicates the priority of a container during shutdown.

Containers will shutdown starting with the highest priority level.

Containers with the same priority will shutdown in parallel. It defaults to 0.

`container_syscall_filtering` Copy Chapter

A number of new syscalls related container configuration keys were introduced.

• security.syscalls.blacklist_default
• security.syscalls.blacklist_compat
• security.syscalls.blacklist
• security.syscalls.whitelist

See Instance configuration for how to use them.

`auth_pki` Copy Chapter

This indicates support for PKI authentication mode.

In this mode, the client and server both must use certificates issued by the same PKI.

See Security for details.

`container_last_used_at` Copy Chapter

A last_used_at field was added to the GET /1.0/containers/<name> endpoint.

It is a timestamp of the last time the container was started.

If a container has been created but not started yet, last_used_at field
will be 1970-01-01T00:00:00Z

`etag` Copy Chapter

Add support for the ETag header on all relevant endpoints.

This adds the following HTTP header on answers to GET:

• ETag (SHA-256 of user modifiable content)

And adds support for the following HTTP header on PUT requests:

• If-Match (ETag value retrieved through previous GET)

This makes it possible to GET an Incus object, modify it and PUT it without
risking to hit a race condition where Incus or another client modified the
object in the meantime.

`patch` Copy Chapter

Add support for the HTTP PATCH method.

PATCH allows for partial update of an object in place of PUT.

`usb_devices` Copy Chapter

Add support for USB hotplug.

`https_allowed_credentials` Copy Chapter

To use Incus API with all Web Browsers (via SPAs) you must send credentials
(certificate) with each XHR (in order for this to happen, you should set
withCredentials=true
flag to each XHR Request).

Some browsers like Firefox and Safari can't accept server response without
Access-Control-Allow-Credentials: true header. To ensure that the server will
return a response with that header, set core.https_allowed_credentials=true.

`image_compression_algorithm` Copy Chapter

This adds support for a compression_algorithm property when creating an image (POST /1.0/images).

Setting this property overrides the server default value (images.compression_algorithm).

`directory_manipulation` Copy Chapter

This allows for creating and listing directories via the Incus API, and exports
the file type via the X-Incus-type header, which can be either file or
directory right now.

`container_cpu_time` Copy Chapter

This adds support for retrieving CPU time for a running container.

`storage_zfs_use_refquota` Copy Chapter

Introduces a new server property storage.zfs_use_refquota which instructs Incus
to set the refquota property instead of quota when setting a size limit
on a container. Incus will also then use usedbydataset in place of used
when being queried about disk utilization.

This effectively controls whether disk usage by snapshots should be
considered as part of the container's disk space usage.

`storage_lvm_mount_options` Copy Chapter

Adds a new storage.lvm_mount_options daemon configuration option
which defaults to discard and allows the user to set addition mount
options for the file system used by the LVM LV.

`network` Copy Chapter

Network management API for Incus.

This includes:

• Addition of the managed property on /1.0/networks entries
• All the network configuration options (see Network configuration for details)
• POST /1.0/networks (see RESTful API for details)
• PUT /1.0/networks/<entry> (see RESTful API for details)
• PATCH /1.0/networks/<entry> (see RESTful API for details)
• DELETE /1.0/networks/<entry> (see RESTful API for details)
• ipv4.address property on nic type devices (when nictype is bridged)
• ipv6.address property on nic type devices (when nictype is bridged)
• security.mac_filtering property on nic type devices (when nictype is bridged)

`profile_usedby` Copy Chapter

Adds a new used_by field to profile entries listing the containers that are using it.

`container_push` Copy Chapter

When a container is created in push mode, the client serves as a proxy between
the source and target server. This is useful in cases where the target server
is behind a NAT or firewall and cannot directly communicate with the source
server and operate in pull mode.

`container_exec_recording` Copy Chapter

Introduces a new Boolean record-output, parameter to
/1.0/containers/<name>/exec which when set to true and combined with
with wait-for-websocket set to false, will record stdout and stderr to
disk and make them available through the logs interface.

The URL to the recorded output is included in the operation metadata
once the command is done running.

That output will expire similarly to other log files, typically after 48 hours.

`certificate_update` Copy Chapter

Adds the following to the REST API:

• ETag header on GET of a certificate
• PUT of certificate entries
• PATCH of certificate entries

`container_exec_signal_handling` Copy Chapter

Adds support /1.0/containers/<name>/exec for forwarding signals sent to the
client to the processes executing in the container. Currently SIGTERM and
SIGHUP are forwarded. Further signals that can be forwarded might be added
later.

`gpu_devices` Copy Chapter

Enables adding GPUs to a container.

`container_image_properties` Copy Chapter

Introduces a new image configuration key space. Read-only, includes the properties of the parent image.

`migration_progress` Copy Chapter

Transfer progress is now exported as part of the operation, on both sending and receiving ends.
This shows up as a fs_progress attribute in the operation metadata.

`id_map` Copy Chapter

Enables setting the security.idmap.isolated and security.idmap.isolated,
security.idmap.size, and raw.id_map fields.

`network_firewall_filtering` Copy Chapter

Add two new keys, ipv4.firewall and ipv6.firewall which if set to
false will turn off the generation of iptables FORWARDING rules. NAT
rules will still be added so long as the matching ipv4.nat or
ipv6.nat key is set to true.

Rules necessary for dnsmasq to work (DHCP/DNS) will always be applied if
dnsmasq is enabled on the bridge.

`network_routes` Copy Chapter

Introduces ipv4.routes and ipv6.routes which allow routing additional subnets to an Incus bridge.

`storage` Copy Chapter

Storage management API for Incus.

This includes:

• GET /1.0/storage-pools

• POST /1.0/storage-pools (see RESTful API for details)

• GET /1.0/storage-pools/<name> (see RESTful API for details)

• POST /1.0/storage-pools/<name> (see RESTful API for details)

• PUT /1.0/storage-pools/<name> (see RESTful API for details)

• PATCH /1.0/storage-pools/<name> (see RESTful API for details)

• DELETE /1.0/storage-pools/<name> (see RESTful API for details)

• GET /1.0/storage-pools/<name>/volumes (see RESTful API for details)

• GET /1.0/storage-pools/<name>/volumes/<volume_type> (see RESTful API for details)

• POST /1.0/storage-pools/<name>/volumes/<volume_type> (see RESTful API for details)

• GET /1.0/storage-pools//volumes/<volume_type>/<name> (see RESTful API for details)

• POST /1.0/storage-pools//volumes/<volume_type>/<name> (see RESTful API for details)

• PUT /1.0/storage-pools//volumes/<volume_type>/<name> (see RESTful API for details)

• PATCH /1.0/storage-pools//volumes/<volume_type>/<name> (see RESTful API for details)

• DELETE /1.0/storage-pools//volumes/<volume_type>/<name> (see RESTful API for details)

• All storage configuration options (see Storage configuration for details)

`file_delete` Copy Chapter

Implements DELETE in /1.0/containers/<name>/files

`file_append` Copy Chapter

Implements the X-Incus-write header which can be one of overwrite or append.

`network_dhcp_expiry` Copy Chapter

Introduces ipv4.dhcp.expiry and ipv6.dhcp.expiry allowing to set the DHCP lease expiry time.

`storage_lvm_vg_rename` Copy Chapter

Introduces the ability to rename a volume group by setting storage.lvm.vg_name.

`storage_lvm_thinpool_rename` Copy Chapter

Introduces the ability to rename a thin pool name by setting storage.thinpool_name.

`network_vlan` Copy Chapter

This adds a new vlan property to macvlan network devices.

When set, this will instruct Incus to attach to the specified VLAN. Incus
will look for an existing interface for that VLAN on the host. If one
can't be found it will create one itself and then use that as the
macvlan parent.

`image_create_aliases` Copy Chapter

Adds a new aliases field to POST /1.0/images allowing for aliases to
be set at image creation/import time.

`container_stateless_copy` Copy Chapter

This introduces a new live attribute in POST /1.0/containers/<name>.
Setting it to false tells Incus not to attempt running state transfer.

`container_only_migration` Copy Chapter

Introduces a new Boolean container_only attribute. When set to true only the
container will be copied or moved.

`storage_zfs_clone_copy` Copy Chapter

Introduces a new Boolean storage_zfs_clone_copy property for ZFS storage
pools. When set to false copying a container will be done through zfs send and
receive. This will make the target container independent of its source
container thus avoiding the need to keep dependent snapshots in the ZFS pool
around. However, this also entails less efficient storage usage for the
affected pool.
The default value for this property is true, i.e. space-efficient snapshots
will be used unless explicitly set to false.

`unix_device_rename` Copy Chapter

Introduces the ability to rename the unix-block/unix-char device inside container by setting path,
and the source attribute is added to specify the device on host.
If source is set without a path, we should assume that path will be the same as source.
If path is set without source and major/minor isn't set,
we should assume that source will be the same as path.
So at least one of them must be set.

`storage_rsync_bwlimit` Copy Chapter

When rsync has to be invoked to transfer storage entities setting rsync.bwlimit
places an upper limit on the amount of socket I/O allowed.

`network_vxlan_interface` Copy Chapter

This introduces a new tunnel.NAME.interface option for networks.

This key control what host network interface is used for a VXLAN tunnel.

`storage_btrfs_mount_options` Copy Chapter

This introduces the btrfs.mount_options property for Btrfs storage pools.

This key controls what mount options will be used for the Btrfs storage pool.

`entity_description` Copy Chapter

This adds descriptions to entities like containers, snapshots, networks, storage pools and volumes.

`image_force_refresh` Copy Chapter

This allows forcing a refresh for an existing image.

`storage_lvm_lv_resizing` Copy Chapter

This introduces the ability to resize logical volumes by setting the size
property in the containers root disk device.

`id_map_base` Copy Chapter

This introduces a new security.idmap.base allowing the user to skip the
map auto-selection process for isolated containers and specify what host
UID/GID to use as the base.

`file_symlinks` Copy Chapter

This adds support for transferring symlinks through the file API.
X-Incus-type can now be symlink with the request content being the target path.

`container_push_target` Copy Chapter

This adds the target field to POST /1.0/containers/<name> which can be
used to have the source Incus host connect to the target during migration.

`network_vlan_physical` Copy Chapter

Allows use of vlan property with physical network devices.

When set, this will instruct Incus to attach to the specified VLAN on the parent interface.
Incus will look for an existing interface for that parent and VLAN on the host.
If one can't be found it will create one itself.
Then, Incus will directly attach this interface to the container.

`storage_images_delete` Copy Chapter

This enabled the storage API to delete storage volumes for images from
a specific storage pool.

`container_edit_metadata` Copy Chapter

This adds support for editing a container metadata.yaml and related templates
via API, by accessing URLs under /1.0/containers/<name>/metadata. It can be used
to edit a container before publishing an image from it.

`container_snapshot_stateful_migration` Copy Chapter

This enables migrating stateful container snapshots to new containers.

`storage_driver_ceph` Copy Chapter

This adds a Ceph storage driver.

`storage_ceph_user_name` Copy Chapter

This adds the ability to specify the Ceph user.

`instance_types` Copy Chapter

This adds the instance_type field to the container creation request.
Its value is expanded to Incus resource limits.

`storage_volatile_initial_source` Copy Chapter

This records the actual source passed to Incus during storage pool creation.

`storage_ceph_force_osd_reuse` Copy Chapter

This introduces the ceph.osd.force_reuse property for the Ceph storage
driver. When set to true Incus will reuse an OSD storage pool that is already in
use by another Incus instance.

`storage_block_filesystem_btrfs` Copy Chapter

This adds support for Btrfs as a storage volume file system, in addition to ext4
and xfs.

`resources` Copy Chapter

This adds support for querying an Incus daemon for the system resources it has
available.

`kernel_limits` Copy Chapter

This adds support for setting process limits such as maximum number of open
files for the container via nofile. The format is limits.kernel.[limit name].

`storage_api_volume_rename` Copy Chapter

This adds support for renaming custom storage volumes.

`network_sriov` Copy Chapter

This adds support for SR-IOV enabled network devices.

`console` Copy Chapter

This adds support to interact with the container console device and console log.

`restrict_dev_incus` Copy Chapter

A new security.guestapi container configuration key was introduced.
The key controls whether the /dev/incus interface is made available to the container.
If set to false, this effectively prevents the container from interacting with the Incus daemon.

`migration_pre_copy` Copy Chapter

This adds support for optimized memory transfer during live migration.

`infiniband` Copy Chapter

This adds support to use InfiniBand network devices.

`dev_incus_events` Copy Chapter

This adds a WebSocket API to the /dev/incus socket.

When connecting to /1.0/events over the /dev/incus socket, you will now be
getting a stream of events over WebSocket.

`proxy` Copy Chapter

This adds a new proxy device type to containers, allowing forwarding
of connections between the host and container.

`network_dhcp_gateway` Copy Chapter

Introduces a new ipv4.dhcp.gateway network configuration key to set an alternate gateway.

`file_get_symlink` Copy Chapter

This makes it possible to retrieve symlinks using the file API.

`network_leases` Copy Chapter

Adds a new /1.0/networks/NAME/leases API endpoint to query the lease database on
bridges which run an Incus-managed DHCP server.

`unix_device_hotplug` Copy Chapter

This adds support for the required property for Unix devices.

`storage_api_local_volume_handling` Copy Chapter

This add the ability to copy and move custom storage volumes locally in the
same and between storage pools.

`operation_description` Copy Chapter

Adds a description field to all operations.

`clustering` Copy Chapter

Clustering API for Incus.

This includes the following new endpoints (see RESTful API for details):

• GET /1.0/cluster

• UPDATE /1.0/cluster

• GET /1.0/cluster/members

• GET /1.0/cluster/members/<name>

• POST /1.0/cluster/members/<name>

• DELETE /1.0/cluster/members/<name>

The following existing endpoints have been modified:

• POST /1.0/containers accepts a new target query parameter
• POST /1.0/storage-pools accepts a new target query parameter
• GET /1.0/storage-pool/<name> accepts a new target query parameter
• POST /1.0/storage-pool//volumes/<type> accepts a new target query parameter
• GET /1.0/storage-pool//volumes/<type>/<name> accepts a new target query parameter
• POST /1.0/storage-pool//volumes/<type>/<name> accepts a new target query parameter
• PUT /1.0/storage-pool//volumes/<type>/<name> accepts a new target query parameter
• PATCH /1.0/storage-pool//volumes/<type>/<name> accepts a new target query parameter
• DELETE /1.0/storage-pool//volumes/<type>/<name> accepts a new target query parameter
• POST /1.0/networks accepts a new target query parameter
• GET /1.0/networks/<name> accepts a new target query parameter

`event_lifecycle` Copy Chapter

This adds a new lifecycle message type to the events API.

`storage_api_remote_volume_handling` Copy Chapter

This adds the ability to copy and move custom storage volumes between remote.

`nvidia_runtime` Copy Chapter

Adds a nvidia_runtime configuration option for containers, setting this to
true will have the NVIDIA runtime and CUDA libraries passed to the
container.

`container_mount_propagation` Copy Chapter

This adds a new propagation option to the disk device type, allowing
the configuration of kernel mount propagation.

`container_backup` Copy Chapter

Add container backup support.

This includes the following new endpoints (see RESTful API for details):

• GET /1.0/containers/<name>/backups

• POST /1.0/containers/<name>/backups

• GET /1.0/containers/<name>/backups/<name>

• POST /1.0/containers/<name>/backups/<name>

• DELETE /1.0/containers/<name>/backups/<name>

• GET /1.0/containers/<name>/backups/<name>/export

The following existing endpoint has been modified:

• POST /1.0/containers accepts the new source type backup

`dev_incus_images` Copy Chapter

Adds a security.guestapi.images configuration option for containers which
controls the availability of a /1.0/images/FINGERPRINT/export API over
/dev/incus. This can be used by a container running nested Incus to access raw
images from the host.

`container_local_cross_pool_handling` Copy Chapter

This enables copying or moving containers between storage pools on the same Incus
instance.

`proxy_unix` Copy Chapter

Add support for both Unix sockets and abstract Unix sockets in proxy devices.
They can be used by specifying the address as unix:/path/to/unix.sock (normal
socket) or unix:@/tmp/unix.sock (abstract socket).

Supported connections are now:

• TCP <-> TCP
• UNIX <-> UNIX
• TCP <-> UNIX
• UNIX <-> TCP

`proxy_udp` Copy Chapter

Add support for UDP in proxy devices.

Supported connections are now:

• TCP <-> TCP
• UNIX <-> UNIX
• TCP <-> UNIX
• UNIX <-> TCP
• UDP <-> UDP
• TCP <-> UDP
• UNIX <-> UDP

`clustering_join` Copy Chapter

This makes GET /1.0/cluster return information about which storage pools and
networks are required to be created by joining nodes and which node-specific
configuration keys they are required to use when creating them. Likewise the PUT /1.0/cluster endpoint now accepts the same format to pass information about
storage pools and networks to be automatically created before attempting to join
a cluster.

`proxy_tcp_udp_multi_port_handling` Copy Chapter

Adds support for forwarding traffic for multiple ports. Forwarding is allowed
between a range of ports if the port range is equal for source and target
(for example 1.2.3.4 0-1000 -> 5.6.7.8 1000-2000) and between a range of source
ports and a single target port (for example 1.2.3.4 0-1000 -> 5.6.7.8 1000).

`network_state` Copy Chapter

Adds support for retrieving a network's state.

This adds the following new endpoint (see RESTful API for details):

• GET /1.0/networks/<name>/state

`proxy_unix_dac_properties` Copy Chapter

This adds support for GID, UID, and mode properties for non-abstract Unix
sockets.

`container_protection_delete` Copy Chapter

Enables setting the security.protection.delete field which prevents containers
from being deleted if set to true. Snapshots are not affected by this setting.

`proxy_priv_drop` Copy Chapter

Adds security.uid and security.gid for the proxy devices, allowing
privilege dropping and effectively changing the UID/GID used for
connections to Unix sockets too.

`pprof_http` Copy Chapter

This adds a new core.debug_address configuration option to start a debugging HTTP server.

That server currently includes a pprof API and replaces the old
cpu-profile, memory-profile and print-goroutines debug options.

`proxy_haproxy_protocol` Copy Chapter

Adds a proxy_protocol key to the proxy device which controls the use of the HAProxy PROXY protocol header.

`network_hwaddr` Copy Chapter

Adds a bridge.hwaddr key to control the MAC address of the bridge.

`proxy_nat` Copy Chapter

This adds optimized UDP/TCP proxying. If the configuration allows, proxying
will be done via iptables instead of proxy devices.

`network_nat_order` Copy Chapter

This introduces the ipv4.nat.order and ipv6.nat.order configuration keys for Incus bridges.
Those keys control whether to put the Incus rules before or after any pre-existing rules in the chain.

`container_full` Copy Chapter

This introduces a new recursion=2 mode for GET /1.0/containers which allows for the retrieval of
all container structs, including the state, snapshots and backup structs.

This effectively allows for incus list to get all it needs in one query.

`backup_compression` Copy Chapter

This introduces a new backups.compression_algorithm configuration key which
allows configuration of backup compression.

`nvidia_runtime_config` Copy Chapter

This introduces a few extra configuration keys when using nvidia.runtime and the libnvidia-container library.
Those keys translate pretty much directly to the matching NVIDIA container environment variables:

• nvidia.driver.capabilities => NVIDIA_DRIVER_CAPABILITIES
• nvidia.require.cuda => NVIDIA_REQUIRE_CUDA
• nvidia.require.driver => NVIDIA_REQUIRE_DRIVER

`storage_api_volume_snapshots` Copy Chapter

Add support for storage volume snapshots. They work like container snapshots,
only for volumes.

This adds the following new endpoint (see RESTful API for details):

• GET /1.0/storage-pools//volumes/<type>/<name>/snapshots

• POST /1.0/storage-pools//volumes/<type>/<name>/snapshots

• GET /1.0/storage-pools//volumes/<type>/<volume>/snapshots/<name>

• PUT /1.0/storage-pools//volumes/<type>/<volume>/snapshots/<name>

• POST /1.0/storage-pools//volumes/<type>/<volume>/snapshots/<name>

• DELETE /1.0/storage-pools//volumes/<type>/<volume>/snapshots/<name>

`storage_unmapped` Copy Chapter

Introduces a new security.unmapped Boolean on storage volumes.

Setting it to true will flush the current map on the volume and prevent
any further idmap tracking and remapping on the volume.

This can be used to share data between isolated containers after
attaching it to the container which requires write access.

`projects` Copy Chapter

Add a new project API, supporting creation, update and deletion of projects.

Projects can hold containers, profiles or images at this point and let
you get a separate view of your Incus resources by switching to it.

`network_vxlan_ttl` Copy Chapter

This adds a new tunnel.NAME.ttl network configuration option which
makes it possible to raise the TTL on VXLAN tunnels.

`container_incremental_copy` Copy Chapter

This adds support for incremental container copy. When copying a container
using the --refresh flag, only the missing or outdated files will be
copied over. Should the target container not exist yet, a normal copy operation
is performed.

`usb_optional_vendorid` Copy Chapter

As the name implies, the vendorid field on USB devices attached to
containers has now been made optional, allowing for all USB devices to
be passed to a container (similar to what's done for GPUs).

`snapshot_scheduling` Copy Chapter

This adds support for snapshot scheduling. It introduces three new
configuration keys: snapshots.schedule, snapshots.schedule.stopped, and
snapshots.pattern. Snapshots can be created automatically up to every minute.

`snapshots_schedule_aliases` Copy Chapter

Snapshot schedule can be configured by a comma-separated list of schedule aliases.
Available aliases are <@hourly> <@daily> <@midnight> <@weekly> <@monthly> <@annually> <@yearly> <@startup> for instances,
and <@hourly> <@daily> <@midnight> <@weekly> <@monthly> <@annually> <@yearly> for storage volumes.

`container_copy_project` Copy Chapter

Introduces a project field to the container source JSON object, allowing for
copy/move of containers between projects.

`clustering_server_address` Copy Chapter

This adds support for configuring a server network address which differs from
the REST API client network address. When bootstrapping a new cluster, clients
can set the new cluster.https_address configuration key to specify the address of
the initial server. When joining a new server, clients can set the
core.https_address configuration key of the joining server to the REST API
address the joining server should listen at, and set the server_address
key in the PUT /1.0/cluster API to the address the joining server should
use for clustering traffic (the value of server_address will be
automatically copied to the cluster.https_address configuration key of the
joining server).

`clustering_image_replication` Copy Chapter

Enable image replication across the nodes in the cluster.
A new cluster.images_minimal_replica configuration key was introduced can be used
to specify to the minimal numbers of nodes for image replication.

`container_protection_shift` Copy Chapter

Enables setting the security.protection.shift option which prevents containers
from having their file system shifted.

`snapshot_expiry` Copy Chapter

This adds support for snapshot expiration. The task is run minutely. The configuration
option snapshots.expiry takes an expression in the form of 1M 2H 3d 4w 5m 6y (1 minute, 2 hours, 3 days, 4 weeks, 5 months, 6 years), however not all
parts have to be used.

Snapshots which are then created will be given an expiry date based on the
expression. This expiry date, defined by expires_at, can be manually edited
using the API or incus config edit. Snapshots with a valid expiry date will be
removed when the task in run. Expiry can be disabled by setting expires_at to
an empty string or 0001-01-01T00:00:00Z (zero time). This is the default if
snapshots.expiry is not set.

This adds the following new endpoint (see RESTful API for details):

• PUT /1.0/containers/<name>/snapshots/<name>

`snapshot_expiry_creation` Copy Chapter

Adds expires_at to container creation, allowing for override of a
snapshot's expiry at creation time.

`network_leases_location` Copy Chapter

Introduces a Location field in the leases list.
This is used when querying a cluster to show what node a particular
lease was found on.

`resources_cpu_socket` Copy Chapter

Add Socket field to CPU resources in case we get out of order socket information.

`resources_gpu` Copy Chapter

Add a new GPU struct to the server resources, listing all usable GPUs on the system.

`resources_numa` Copy Chapter

Shows the NUMA node for all CPUs and GPUs.

`kernel_features` Copy Chapter

Exposes the state of optional kernel features through the server environment.

`id_map_current` Copy Chapter

This introduces a new internal volatile.idmap.current key which is
used to track the current mapping for the container.

This effectively gives us:

• volatile.last_state.idmap => On-disk idmap
• volatile.idmap.current => Current kernel map
• volatile.idmap.next => Next on-disk idmap

This is required to implement environments where the on-disk map isn't
changed but the kernel map is (e.g. idmapped mounts).

`event_location` Copy Chapter

Expose the location of the generation of API events.

`storage_api_remote_volume_snapshots` Copy Chapter

This allows migrating storage volumes including their snapshots.

`network_nat_address` Copy Chapter

This introduces the ipv4.nat.address and ipv6.nat.address configuration keys for Incus bridges.
Those keys control the source address used for outbound traffic from the bridge.

`container_nic_routes` Copy Chapter

This introduces the ipv4.routes and ipv6.routes properties on nic type devices.
This allows adding static routes on host to container's NIC.

`cluster_internal_copy` Copy Chapter

This makes it possible to do a normal POST /1.0/containers to copy a
container between cluster nodes with Incus internally detecting whether a
migration is required.

`seccomp_notify` Copy Chapter

If the kernel supports seccomp-based syscall interception Incus can be notified
by a container that a registered syscall has been performed. Incus can then
decide to trigger various actions.

`lxc_features` Copy Chapter

This introduces the lxc_features section output from the incus info command
via the GET /1.0 route. It outputs the result of checks for key features being present in the
underlying LXC library.

`container_nic_ipvlan` Copy Chapter

This introduces the ipvlan nic device type.

`network_vlan_sriov` Copy Chapter

This introduces VLAN (vlan) and MAC filtering (security.mac_filtering) support for SR-IOV devices.

`storage_cephfs` Copy Chapter

Add support for CephFS as a storage pool driver. This can only be used
for custom volumes, images and containers should be on Ceph (RBD)
instead.

`container_nic_ipfilter` Copy Chapter

This introduces container IP filtering (security.ipv4_filtering and security.ipv6_filtering) support for bridged NIC devices.

`resources_v2` Copy Chapter

Rework the resources API at /1.0/resources, especially:

• CPU
  • Fix reporting to track sockets, cores and threads
  • Track NUMA node per core
  • Track base and turbo frequency per socket
  • Track current frequency per core
  • Add CPU cache information
  • Export the CPU architecture
  • Show online/offline status of threads
• Memory
  • Add huge-pages tracking
  • Track memory consumption per NUMA node too
• GPU
  • Split DRM information to separate struct
  • Export device names and nodes in DRM struct
  • Export device name and node in NVIDIA struct
  • Add SR-IOV VF tracking

`container_exec_user_group_cwd` Copy Chapter

Adds support for specifying User, Group and Cwd during POST /1.0/containers/NAME/exec.

`container_syscall_intercept` Copy Chapter

Adds the security.syscalls.intercept.* configuration keys to control
what system calls will be intercepted by Incus and processed with
elevated permissions.

`container_disk_shift` Copy Chapter

Adds the shift property on disk devices which controls the use of the idmapped mounts overlay.

`storage_shifted` Copy Chapter

Introduces a new security.shifted Boolean on storage volumes.

Setting it to true will allow multiple isolated containers to attach the
same storage volume while keeping the file system writable from all of
them.

This makes use of idmapped mounts as an overlay file system.

`resources_infiniband` Copy Chapter

Export InfiniBand character device information (issm, umad, uverb) as part of the resources API.

`daemon_storage` Copy Chapter

This introduces two new configuration keys storage.images_volume and
storage.backups_volume to allow for a storage volume on an existing
pool be used for storing the daemon-wide images and backups artifacts.

`instances` Copy Chapter

This introduces the concept of instances, of which currently the only type is container.

`image_types` Copy Chapter

This introduces support for a new Type field on images, indicating what type of images they are.

`resources_disk_sata` Copy Chapter

Extends the disk resource API struct to include:

• Proper detection of SATA devices (type)
• Device path
• Drive RPM
• Block size
• Firmware version
• Serial number

`clustering_roles` Copy Chapter

This adds a new roles attribute to cluster entries, exposing a list of
roles that the member serves in the cluster.

`images_expiry` Copy Chapter

This allows for editing of the expiry date on images.

`resources_network_firmware` Copy Chapter

Adds a FirmwareVersion field to network card entries.

`backup_compression_algorithm` Copy Chapter

This adds support for a compression_algorithm property when creating a backup (POST /1.0/containers/<name>/backups).

Setting this property overrides the server default value (backups.compression_algorithm).

`ceph_data_pool_name` Copy Chapter

This adds support for an optional argument (ceph.osd.data_pool_name) when creating
storage pools using Ceph RBD, when this argument is used the pool will store it's
actual data in the pool specified with data_pool_name while keeping the metadata
in the pool specified by pool_name.

`container_syscall_intercept_mount` Copy Chapter

Adds the security.syscalls.intercept.mount,
security.syscalls.intercept.mount.allowed, and
security.syscalls.intercept.mount.shift configuration keys to control whether
and how the mount system call will be intercepted by Incus and processed with
elevated permissions.

`compression_squashfs` Copy Chapter

Adds support for importing/exporting of images/backups using SquashFS file system format.

`container_raw_mount` Copy Chapter

This adds support for passing in raw mount options for disk devices.

`container_nic_routed` Copy Chapter

This introduces the routed nic device type.

`container_syscall_intercept_mount_fuse` Copy Chapter

Adds the security.syscalls.intercept.mount.fuse key. It can be used to
redirect file-system mounts to their fuse implementation. To this end, set e.g.
security.syscalls.intercept.mount.fuse=ext4=fuse2fs.

`container_disk_ceph` Copy Chapter

This allows for existing a Ceph RBD or CephFS to be directly connected to an Incus container.

`virtual-machines` Copy Chapter

Add virtual machine support.

`image_profiles` Copy Chapter

Allows a list of profiles to be applied to an image when launching a new container.

`clustering_architecture` Copy Chapter

This adds a new architecture attribute to cluster members which indicates a cluster
member's architecture.

`resources_disk_id` Copy Chapter

Add a new device_id field in the disk entries on the resources API.

`storage_lvm_stripes` Copy Chapter

This adds the ability to use LVM stripes on normal volumes and thin pool volumes.

`vm_boot_priority` Copy Chapter

Adds a boot.priority property on NIC and disk devices to control the boot order.

`unix_hotplug_devices` Copy Chapter

Adds support for Unix char and block device hotplugging.

`api_filtering` Copy Chapter

Adds support for filtering the result of a GET request for instances and images.

`instance_nic_network` Copy Chapter

Adds support for the network property on a NIC device to allow a NIC to be linked to a managed network.
This allows it to inherit some of the network's settings and allows better validation of IP settings.

`clustering_sizing` Copy Chapter

Support specifying a custom values for database voters and standbys.
The new cluster.max_voters and cluster.max_standby configuration keys were introduced
to specify to the ideal number of database voter and standbys.

`firewall_driver` Copy Chapter

Adds the Firewall property to the ServerEnvironment struct indicating the firewall driver being used.

`storage_lvm_vg_force_reuse` Copy Chapter

Introduces the ability to create a storage pool from an existing non-empty volume group.
This option should be used with care, as Incus can then not guarantee that volume name conflicts won't occur
with non-Incus created volumes in the same volume group.
This could also potentially lead to Incus deleting a non-Incus volume should name conflicts occur.

`container_syscall_intercept_hugetlbfs` Copy Chapter

When mount syscall interception is enabled and hugetlbfs is specified as an
allowed file system type Incus will mount a separate hugetlbfs instance for the
container with the UID and GID mount options set to the container's root UID
and GID. This ensures that processes in the container can use huge pages.

`limits_hugepages` Copy Chapter

This allows to limit the number of huge pages a container can use through the
hugetlb cgroup. This means the hugetlb cgroup needs to be available. Note, that
limiting huge pages is recommended when intercepting the mount syscall for the
hugetlbfs file system to avoid allowing the container to exhaust the host's
huge pages resources.

`container_nic_routed_gateway` Copy Chapter

This introduces the ipv4.gateway and ipv6.gateway NIC configuration keys that can take a value of either auto or
none. The default value for the key if unspecified is auto. This will cause the current behavior of a default
gateway being added inside the container and the same gateway address being added to the host-side interface.
If the value is set to none then no default gateway nor will the address be added to the host-side interface.
This allows multiple routed NIC devices to be added to a container.

`projects_restrictions` Copy Chapter

This introduces support for the restricted configuration key on project, which
can prevent the use of security-sensitive features in a project.

`custom_volume_snapshot_expiry` Copy Chapter

This allows custom volume snapshots to expiry.
Expiry dates can be set individually, or by setting the snapshots.expiry configuration key on the parent custom volume which then automatically applies to all created snapshots.

`volume_snapshot_scheduling` Copy Chapter

This adds support for custom volume snapshot scheduling. It introduces two new
configuration keys: snapshots.schedule and
snapshots.pattern. Snapshots can be created automatically up to every minute.

`trust_ca_certificates` Copy Chapter

This allows for checking client certificates trusted by the provided CA (server.ca).
It can be enabled by setting core.trust_ca_certificates to true.
If enabled, it will perform the check, and bypass the trusted password if true.
An exception will be made if the connecting client certificate is in the provided CRL (ca.crl).
In this case, it will ask for the password.

`snapshot_disk_usage` Copy Chapter

This adds a new size field to the output of /1.0/instances/<name>/snapshots/<snapshot> which represents the disk usage of the snapshot.

`clustering_edit_roles` Copy Chapter

This adds a writable endpoint for cluster members, allowing the editing of their roles.

`container_nic_routed_host_address` Copy Chapter

This introduces the ipv4.host_address and ipv6.host_address NIC configuration keys that can be used to control the
host-side veth interface's IP addresses. This can be useful when using multiple routed NICs at the same time and
needing a predictable next-hop address to use.

This also alters the behavior of ipv4.gateway and ipv6.gateway NIC configuration keys. When they are set to auto
the container will have its default gateway set to the value of ipv4.host_address or ipv6.host_address respectively.

The default values are:

ipv4.host_address: 169.254.0.1
ipv6.host_address: fe80::1

This is backward compatible with the previous default behavior.

`container_nic_ipvlan_gateway` Copy Chapter

This introduces the ipv4.gateway and ipv6.gateway NIC configuration keys that can take a value of either auto or
none. The default value for the key if unspecified is auto. This will cause the current behavior of a default
gateway being added inside the container and the same gateway address being added to the host-side interface.
If the value is set to none then no default gateway nor will the address be added to the host-side interface.
This allows multiple IPVLAN NIC devices to be added to a container.

`resources_usb_pci` Copy Chapter

This adds USB and PCI devices to the output of /1.0/resources.

`resources_cpu_threads_numa` Copy Chapter

This indicates that the numa_node field is now recorded per-thread
rather than per core as some hardware apparently puts threads in
different NUMA domains.

`resources_cpu_core_die` Copy Chapter

Exposes the die_id information on each core.

`api_os` Copy Chapter

This introduces two new fields in /1.0, os and os_version.

Those are taken from the OS-release data on the system.

`container_nic_routed_host_table` Copy Chapter

This introduces the ipv4.host_table and ipv6.host_table NIC configuration keys that can be used to add static routes
for the instance's IPs to a custom policy routing table by ID.

`container_nic_ipvlan_host_table` Copy Chapter

This introduces the ipv4.host_table and ipv6.host_table NIC configuration keys that can be used to add static routes
for the instance's IPs to a custom policy routing table by ID.

`container_nic_ipvlan_mode` Copy Chapter

This introduces the mode NIC configuration key that can be used to switch the ipvlan mode into either l2 or l3s.
If not specified, the default value is l3s (which is the old behavior).

In l2 mode the ipv4.address and ipv6.address keys will accept addresses in either CIDR or singular formats.
If singular format is used, the default subnet size is taken to be /24 and /64 for IPv4 and IPv6 respectively.

In l2 mode the ipv4.gateway and ipv6.gateway keys accept only a singular IP address.

`resources_system` Copy Chapter

This adds system information to the output of /1.0/resources.

`images_push_relay` Copy Chapter

This adds the push and relay modes to image copy.
It also introduces the following new endpoint:

• POST 1.0/images/<fingerprint>/export

`network_dns_search` Copy Chapter

This introduces the dns.search configuration option on networks.

`container_nic_routed_limits` Copy Chapter

This introduces limits.ingress, limits.egress and limits.max for routed NICs.

`instance_nic_bridged_vlan` Copy Chapter

This introduces the vlan and vlan.tagged settings for bridged NICs.

vlan specifies the non-tagged VLAN to join, and vlan.tagged is a comma-delimited list of tagged VLANs to join.

`network_state_bond_bridge` Copy Chapter

This adds a bridge and bond section to the /1.0/networks/NAME/state API.

Those contain additional state information relevant to those particular types.

Bond:

• Mode
• Transmit hash
• Up delay
• Down delay
• MII frequency
• MII state
• Lower devices

Bridge:

• ID
• Forward delay
• STP mode
• Default VLAN
• VLAN filtering
• Upper devices

`resources_cpu_isolated` Copy Chapter

Add an Isolated property on CPU threads to indicate if the thread is
physically Online but is configured not to accept tasks.

`usedby_consistency` Copy Chapter

This extension indicates that UsedBy should now be consistent with
suitable ?project= and ?target= when appropriate.

The 5 entities that have UsedBy are:

• Profiles
• Projects
• Networks
• Storage pools
• Storage volumes

`custom_block_volumes` Copy Chapter

This adds support for creating and attaching custom block volumes to instances.
It introduces the new --type flag when creating custom storage volumes, and accepts the values fs and block.

`clustering_failure_domains` Copy Chapter

This extension adds a new failure_domain field to the PUT /1.0/cluster/<node> API,
which can be used to set the failure domain of a node.

`container_syscall_filtering_allow_deny_syntax` Copy Chapter

A number of new syscalls related container configuration keys were updated.

• security.syscalls.deny_default
• security.syscalls.deny_compat
• security.syscalls.deny
• security.syscalls.allow

`resources_gpu_mdev` Copy Chapter

Expose available mediated device profiles and devices in /1.0/resources.

`console_vga_type` Copy Chapter

This extends the /1.0/console endpoint to take a ?type= argument, which can
be set to console (default) or vga (the new type added by this extension).

When doing a POST to /1.0//console?type=vga the data WebSocket
returned by the operation in the metadata field will be a bidirectional proxy
attached to a SPICE Unix socket of the target virtual machine.

`projects_limits_disk` Copy Chapter

Add limits.disk to the available project configuration keys. If set, it limits
the total amount of disk space that instances volumes, custom volumes and images
volumes can use in the project.

`network_type_macvlan` Copy Chapter

Adds support for additional network type macvlan and adds parent configuration key for this network type to
specify which parent interface should be used for creating NIC device interfaces on top of.

Also adds network configuration key support for macvlan NICs to allow them to specify the associated network of
the same type that they should use as the basis for the NIC device.

`network_type_sriov` Copy Chapter

Adds support for additional network type sriov and adds parent configuration key for this network type to
specify which parent interface should be used for creating NIC device interfaces on top of.

Also adds network configuration key support for sriov NICs to allow them to specify the associated network of
the same type that they should use as the basis for the NIC device.

`container_syscall_intercept_bpf_devices` Copy Chapter

This adds support to intercept the bpf syscall in containers. Specifically, it allows to manage device cgroup bpf programs.

`network_type_ovn` Copy Chapter

Adds support for additional network type ovn with the ability to specify a bridge type network as the parent.

Introduces a new NIC device type of ovn which allows the network configuration key to specify which ovn
type network they should connect to.

Also introduces two new global configuration keys that apply to all ovn networks and NIC devices:

• network.ovn.integration_bridge - the OVS integration bridge to use.
• network.ovn.northbound_connection - the OVN northbound database connection string.

`projects_networks` Copy Chapter

Adds the features.networks configuration key to projects and the ability for a project to hold networks.

`projects_networks_restricted_uplinks` Copy Chapter

Adds the restricted.networks.uplinks project configuration key to indicate (as a comma-delimited list) which networks
the networks created inside the project can use as their uplink network.

`custom_volume_backup` Copy Chapter

Add custom volume backup support.

This includes the following new endpoints (see RESTful API for details):

• GET /1.0/storage-pools//<type>/<volume>/backups

• POST /1.0/storage-pools//<type>/<volume>/backups

• GET /1.0/storage-pools//<type>/<volume>/backups/<name>

• POST /1.0/storage-pools//<type>/<volume>/backups/<name>

• DELETE /1.0/storage-pools//<type>/<volume>/backups/<name>

• GET /1.0/storage-pools//<type>/<volume>/backups/<name>/export

The following existing endpoint has been modified:

• POST /1.0/storage-pools//<type>/<volume> accepts the new source type backup

`backup_override_name` Copy Chapter

Adds Name field to InstanceBackupArgs to allow specifying a different instance name when restoring a backup.

Adds Name and PoolName fields to StoragePoolVolumeBackupArgs to allow specifying a different volume name
when restoring a custom volume backup.

`storage_rsync_compression` Copy Chapter

Adds rsync.compression configuration key to storage pools. This key can be used
to disable compression in rsync while migrating storage pools.

`network_type_physical` Copy Chapter

Adds support for additional network type physical that can be used as an uplink for ovn networks.

The interface specified by parent on the physical network will be connected to the ovn network's gateway.

`network_ovn_external_subnets` Copy Chapter

Adds support for ovn networks to use external subnets from uplink networks.

Introduces the ipv4.routes and ipv6.routes setting on physical networks that defines the external routes
allowed to be used in child OVN networks in their ipv4.routes.external and ipv6.routes.external settings.

Introduces the restricted.networks.subnets project setting that specifies which external subnets are allowed to
be used by OVN networks inside the project (if not set then all routes defined on the uplink network are allowed).

`network_ovn_nat` Copy Chapter

Adds support for ipv4.nat and ipv6.nat settings on ovn networks.

When creating the network if these settings are unspecified, and an equivalent IP address is being generated for
the subnet, then the appropriate NAT setting will added set to true.

If the setting is missing then the value is taken as false.

`network_ovn_external_routes_remove` Copy Chapter

Removes the settings ipv4.routes.external and ipv6.routes.external from ovn networks.

The equivalent settings on the ovn NIC type can be used instead for this, rather than having to specify them
both at the network and NIC level.

`tpm_device_type` Copy Chapter

This introduces the tpm device type.

`storage_zfs_clone_copy_rebase` Copy Chapter

This introduces rebase as a value for zfs.clone_copy causing Incus to
track down any image dataset in the ancestry line and then perform
send/receive on top of that.

`gpu_mdev` Copy Chapter

This adds support for virtual GPUs. It introduces the mdev configuration key for GPU devices which takes
a supported mdev type, e.g. i915-GVTg_V5_4.

`resources_pci_iommu` Copy Chapter

This adds the IOMMUGroup field for PCI entries in the resources API.

`resources_network_usb` Copy Chapter

Adds the usb_address field to the network card entries in the resources API.

`resources_disk_address` Copy Chapter

Adds the usb_address and pci_address fields to the disk entries in the resources API.

`network_physical_ovn_ingress_mode` Copy Chapter

Adds ovn.ingress_mode setting for physical networks.

Sets the method that OVN NIC external IPs will be advertised on uplink network.

Either l2proxy (proxy ARP/NDP) or routed.

`network_ovn_dhcp` Copy Chapter

Adds ipv4.dhcp and ipv6.dhcp settings for ovn networks.

Allows DHCP (and RA for IPv6) to be disabled. Defaults to on.

`network_physical_routes_anycast` Copy Chapter

Adds ipv4.routes.anycast and ipv6.routes.anycast Boolean settings for physical networks. Defaults to false.

Allows OVN networks using physical network as uplink to relax external subnet/route overlap detection when used
with ovn.ingress_mode=routed.

`projects_limits_instances` Copy Chapter

Adds limits.instances to the available project configuration keys. If set, it
limits the total number of instances (VMs and containers) that can be used in the project.

`network_state_vlan` Copy Chapter

This adds a vlan section to the /1.0/networks/NAME/state API.

Those contain additional state information relevant to VLAN interfaces:

• lower_device
• vid

`instance_nic_bridged_port_isolation` Copy Chapter

This adds the security.port_isolation field for bridged NIC instances.

`instance_bulk_state_change` Copy Chapter

Adds the following endpoint for bulk state change (see RESTful API for details):

• PUT /1.0/instances

`network_gvrp` Copy Chapter

This adds an optional gvrp property to macvlan and physical networks,
and to ipvlan, macvlan, routed and physical NIC devices.

When set, this specifies whether the VLAN should be registered using GARP VLAN
Registration Protocol. Defaults to false.

`instance_pool_move` Copy Chapter

This adds a pool field to the POST /1.0/instances/NAME API,
allowing for easy move of an instance root disk between pools.

`gpu_sriov` Copy Chapter

This adds support for SR-IOV enabled GPUs.
It introduces the sriov GPU type property.

`pci_device_type` Copy Chapter

This introduces the pci device type.

`storage_volume_state` Copy Chapter

Add new /1.0/storage-pools/POOL/volumes/VOLUME/state API endpoint to get usage data on a volume.

`network_acl` Copy Chapter

This adds the concept of network ACLs to API under the API endpoint prefix /1.0/network-acls.

`migration_stateful` Copy Chapter

Add a new migration.stateful configuration key.

`disk_state_quota` Copy Chapter

This introduces the size.state device configuration key on disk devices.

`storage_ceph_features` Copy Chapter

Adds a new ceph.rbd.features configuration key on storage pools to control the RBD features used for new volumes.

`projects_compression` Copy Chapter

Adds new backups.compression_algorithm and images.compression_algorithm configuration keys which
allows configuration of backup and image compression per-project.

`projects_images_remote_cache_expiry` Copy Chapter

Add new images.remote_cache_expiry configuration key to projects,
allowing for set number of days after which an unused cached remote image will be flushed.

`certificate_project` Copy Chapter

Adds a new restricted property to certificates in the API as well as
projects holding a list of project names that the certificate has
access to.

`network_ovn_acl` Copy Chapter

Adds a new security.acls property to OVN networks and OVN NICs, allowing Network ACLs to be applied.

`projects_images_auto_update` Copy Chapter

Adds new images.auto_update_cached and images.auto_update_interval configuration keys which
allows configuration of images auto update in projects

`projects_restricted_cluster_target` Copy Chapter

Adds new restricted.cluster.target configuration key to project which prevent the user from using --target
to specify what cluster member to place a workload on or the ability to move a workload between members.

`images_default_architecture` Copy Chapter

Adds new images.default_architecture global configuration key and matching per-project key which lets user tell Incus
what architecture to go with when no specific one is specified as part of the image request.

`network_ovn_acl_defaults` Copy Chapter

Adds new security.acls.default.{in,e}gress.action and security.acls.default.{in,e}gress.logged configuration keys for
OVN networks and NICs. This replaces the removed ACL default.action and default.logged keys.

`gpu_mig` Copy Chapter

This adds support for NVIDIA MIG. It introduces the mig GPU type and associated configuration keys.

`project_usage` Copy Chapter

Adds an API endpoint to get current resource allocations in a project.
Accessible at API GET /1.0/projects/<name>/state.

`network_bridge_acl` Copy Chapter

Adds a new security.acls configuration key to bridge networks, allowing Network ACLs to be applied.

Also adds security.acls.default.{in,e}gress.action and security.acls.default.{in,e}gress.logged configuration keys for
specifying the default behavior for unmatched traffic.

`warnings` Copy Chapter

Warning API for Incus.

This includes the following endpoints (see Restful API for details):

• GET /1.0/warnings

• GET /1.0/warnings/

• PUT /1.0/warnings/

• DELETE /1.0/warnings/

`projects_restricted_backups_and_snapshots` Copy Chapter

Adds new restricted.backups and restricted.snapshots configuration keys to project which
prevents the user from creation of backups and snapshots.

`clustering_join_token` Copy Chapter

Adds POST /1.0/cluster/members API endpoint for requesting a join token used when adding new cluster members
without using the trust password.

`clustering_description` Copy Chapter

Adds an editable description to the cluster members.

`server_trusted_proxy` Copy Chapter

This introduces support for core.https_trusted_proxy which has Incus
parse a HAProxy style connection header on such connections and if
present, will rewrite the request's source address to that provided by
the proxy server.

`clustering_update_cert` Copy Chapter

Adds PUT /1.0/cluster/certificate endpoint for updating the cluster
certificate across the whole cluster

`storage_api_project` Copy Chapter

This adds support for copy/move custom storage volumes between projects.

`server_instance_driver_operational` Copy Chapter

This modifies the driver output for the /1.0 endpoint to only include drivers which are actually supported and
operational on the server (as opposed to being included in Incus but not operational on the server).

`server_supported_storage_drivers` Copy Chapter

This adds supported storage driver info to server environment info.

`event_lifecycle_requestor_address` Copy Chapter

Adds a new address field to lifecycle requestor.

`resources_gpu_usb` Copy Chapter

Add a new USBAddress (usb_address) field to ResourcesGPUCard (GPU entries) in the resources API.

`clustering_evacuation` Copy Chapter

Adds POST /1.0/cluster/members/<name>/state endpoint for evacuating and restoring cluster members.
It also adds the configuration keys cluster.evacuate and volatile.evacuate.origin for setting the evacuation method (auto, stop or migrate) and the origin of any migrated instance respectively.

`network_ovn_nat_address` Copy Chapter

This introduces the ipv4.nat.address and ipv6.nat.address configuration keys for Incus ovn networks.
Those keys control the source address used for outbound traffic from the OVN virtual network.
These keys can only be specified when the OVN network's uplink network has ovn.ingress_mode=routed.

`network_bgp` Copy Chapter

This introduces support for Incus acting as a BGP router to advertise
routes to bridge and ovn networks.

This comes with the addition to global configuration of:

• core.bgp_address
• core.bgp_asn
• core.bgp_routerid

The following network configurations keys (bridge and physical):

• bgp.peers.<name>.address
• bgp.peers.<name>.asn
• bgp.peers.<name>.password

The nexthop configuration keys (bridge):

• bgp.ipv4.nexthop
• bgp.ipv6.nexthop

And the following NIC-specific configuration keys (bridged NIC type):

• ipv4.routes.external
• ipv6.routes.external

`network_forward` Copy Chapter

This introduces the networking address forward functionality. Allowing for bridge and ovn networks to define
external IP addresses that can be forwarded to internal IP(s) inside their respective networks.

`custom_volume_refresh` Copy Chapter

Adds support for refresh during volume migration.

`network_counters_errors_dropped` Copy Chapter

This adds the received and sent errors as well as inbound and outbound dropped packets to the network counters.

`metrics` Copy Chapter

This adds metrics to Incus. It returns metrics of running instances using the OpenMetrics format.

This includes the following endpoints:

• GET /1.0/metrics

`image_source_project` Copy Chapter

Adds a new project field to POST /1.0/images allowing for the source project
to be set at image copy time.

`clustering_config` Copy Chapter

Adds new config property to cluster members with configurable key/value pairs.

`network_peer` Copy Chapter

This adds network peering to allow traffic to flow between OVN networks without leaving the OVN subsystem.

`linux_sysctl` Copy Chapter

Adds new linux.sysctl.* configuration keys allowing users to modify certain kernel parameters
within containers.

`network_dns` Copy Chapter

Introduces a built-in DNS server and zones API to provide DNS records for Incus instances.

This introduces the following server configuration key:

• core.dns_address

The following network configuration key:

• dns.zone.forward
• dns.zone.reverse.ipv4
• dns.zone.reverse.ipv6

And the following project configuration key:

• restricted.networks.zones

A new REST API is also introduced to manage DNS zones:

• /1.0/network-zones (GET, POST)
• /1.0/network-zones/<name> (GET, PUT, PATCH, DELETE)

`ovn_nic_acceleration` Copy Chapter

Adds new acceleration configuration key to OVN NICs which can be used for enabling hardware offloading.
It takes the values none or sriov.

`certificate_self_renewal` Copy Chapter

This adds support for renewing a client's own trust certificate.

`instance_project_move` Copy Chapter

This adds a project field to the POST /1.0/instances/NAME API,
allowing for easy move of an instance between projects.

`storage_volume_project_move` Copy Chapter

This adds support for moving storage volume between projects.

`cloud_init` Copy Chapter

This adds a new cloud-init configuration key namespace which contains the following keys:

• cloud-init.vendor-data
• cloud-init.user-data
• cloud-init.network-config

It also adds a new endpoint /1.0/devices to /dev/incus which shows an instance's devices.

`network_dns_nat` Copy Chapter

This introduces network.nat as a configuration option on network zones (DNS).

It defaults to the current behavior of generating records for all
instances NICs but if set to false, it will instruct Incus to only
generate records for externally reachable addresses.

`database_leader` Copy Chapter

Adds new database-leader role which is assigned to cluster leader.

`instance_all_projects` Copy Chapter

This adds support for displaying instances from all projects.

`clustering_groups` Copy Chapter

Add support for grouping cluster members.

This introduces the following new endpoints:

• /1.0/cluster/groups (GET, POST)
• /1.0/cluster/groups/<name> (GET, POST, PUT, PATCH, DELETE)

The following project restriction is added:

• restricted.cluster.groups

`ceph_rbd_du` Copy Chapter

Adds a new ceph.rbd.du Boolean on Ceph storage pools which allows
disabling the use of the potentially slow rbd du calls.

`instance_get_full` Copy Chapter

This introduces a new recursion=1 mode for GET /1.0/instances/{name} which allows for the retrieval of
all instance structs, including the state, snapshots and backup structs.

`qemu_metrics` Copy Chapter

This adds a new security.agent.metrics Boolean which defaults to true.
When set to false, it doesn't connect to the incus-agent for metrics and other state information, but relies on stats from QEMU.

`gpu_mig_uuid` Copy Chapter

Adds support for the new MIG UUID format used by NVIDIA 470+ drivers (for example, MIG-74c6a31a-fde5-5c61-973b-70e12346c202),
the MIG- prefix can be omitted

This extension supersedes old mig.gi and mig.ci parameters which are kept for compatibility with old drivers and
cannot be set together.

`event_project` Copy Chapter

Expose the project an API event belongs to.

`clustering_evacuation_live` Copy Chapter

This adds live-migrate as a configuration option to cluster.evacuate, which forces live-migration
of instances during cluster evacuation.

`instance_allow_inconsistent_copy` Copy Chapter

Adds allow_inconsistent field to instance source on POST /1.0/instances. If true, rsync will ignore the
Partial transfer due to vanished source files (code 24) error when creating an instance from a copy.

`network_state_ovn` Copy Chapter

This adds an ovn section to the /1.0/networks/NAME/state API which contains additional state information relevant to
OVN networks:

• chassis

`storage_volume_api_filtering` Copy Chapter

Adds support for filtering the result of a GET request for storage volumes.

`image_restrictions` Copy Chapter

This extension adds on to the image properties to include image restrictions/host requirements. These requirements
help determine the compatibility between an instance and the host system.

`storage_zfs_export` Copy Chapter

Introduces the ability to disable zpool export when unmounting pool by setting zfs.export.

`network_dns_records` Copy Chapter

This extends the network zones (DNS) API to add the ability to create and manage custom records.

This adds:

• GET /1.0/network-zones/ZONE/records
• POST /1.0/network-zones/ZONE/records
• GET /1.0/network-zones/ZONE/records/RECORD
• PUT /1.0/network-zones/ZONE/records/RECORD
• PATCH /1.0/network-zones/ZONE/records/RECORD
• DELETE /1.0/network-zones/ZONE/records/RECORD

`network_zones_all_projects` Copy Chapter

This adds support for listing network zones across all projects through the all-projects parameter on the GET /1.0/network-zonesAPI.

`storage_zfs_reserve_space` Copy Chapter

Adds ability to set the reservation/refreservation ZFS property along with quota/refquota.

`network_acl_log` Copy Chapter

Adds a new GET /1.0/networks-acls/NAME/log API to retrieve ACL firewall logs.

`storage_zfs_blocksize` Copy Chapter

Introduces a new zfs.blocksize property for ZFS storage volumes which allows to set volume block size.

`metrics_cpu_seconds` Copy Chapter

This is used to detect whether Incus was fixed to output used CPU time in seconds rather than as milliseconds.

`instance_snapshot_never` Copy Chapter

Adds a @never option to snapshots.schedule which allows disabling inheritance.

`certificate_token` Copy Chapter

This adds token-based certificate addition to the trust store as a safer alternative to a trust password.

It adds the token field to POST /1.0/certificates.

`instance_nic_routed_neighbor_probe` Copy Chapter

This adds the ability to disable the routed NIC IP neighbor probing for availability on the parent network.

Adds the ipv4.neighbor_probe and ipv6.neighbor_probe NIC settings. Defaulting to true if not specified.

`event_hub` Copy Chapter

This adds support for event-hub cluster member role and the ServerEventMode environment field.

`agent_nic_config` Copy Chapter

If set to true, on VM start-up the incus-agent will apply NIC configuration to change the names and MTU of the instance NIC
devices.

`projects_restricted_intercept` Copy Chapter

Adds new restricted.container.intercept configuration key to allow usually safe system call interception options.

`metrics_authentication` Copy Chapter

Introduces a new core.metrics_authentication server configuration option to
allow for the /1.0/metrics endpoint to be generally available without
client authentication.

`images_target_project` Copy Chapter

Adds ability to copy image to a project different from the source.

`images_all_projects` Copy Chapter

This adds support for listing images across all projects through the all-projects parameter on the GET /1.0/imagesAPI.

`cluster_migration_inconsistent_copy` Copy Chapter

Adds allow_inconsistent field to POST /1.0/instances/<name>. Set to true to allow inconsistent copying between cluster
members.

`cluster_ovn_chassis` Copy Chapter

Introduces a new ovn-chassis cluster role which allows for specifying what cluster member should act as an OVN chassis.

`container_syscall_intercept_sched_setscheduler` Copy Chapter

Adds the security.syscalls.intercept.sched_setscheduler to allow advanced process priority management in containers.

`storage_lvm_thinpool_metadata_size` Copy Chapter

Introduces the ability to specify the thin pool metadata volume size via storage.thinpool_metadata_size.

If this is not specified then the default is to let LVM pick an appropriate thin pool metadata volume size.

`storage_volume_state_total` Copy Chapter

This adds total field to the GET /1.0/storage-pools/{name}/volumes/{type}/{volume}/state API.

`instance_file_head` Copy Chapter

Implements HEAD on /1.0/instances/NAME/file.

`instances_nic_host_name` Copy Chapter

This introduces the instances.nic.host_name server configuration key that can take a value of either random or
mac. The default value for the key if unspecified is random. If it is set to random then use the random host interface names.
If it's set to mac, then generate a name in the form inc1122334455.

`image_copy_profile` Copy Chapter

Adds ability to modify the set of profiles when image is copied.

`container_syscall_intercept_sysinfo` Copy Chapter

Adds the security.syscalls.intercept.sysinfo to allow the sysinfo syscall to be populated with cgroup-based resource usage information.

`clustering_evacuation_mode` Copy Chapter

This introduces a mode field to the evacuation request which allows
for overriding the evacuation mode traditionally set through
cluster.evacuate.

`resources_pci_vpd` Copy Chapter

Adds a new VPD struct to the PCI resource entries.
This struct extracts vendor provided data including the full product name and additional key/value configuration pairs.

`qemu_raw_conf` Copy Chapter

Introduces a raw.qemu.conf configuration key to override select sections of the generated qemu.conf.

`storage_cephfs_fscache` Copy Chapter

Add support for fscache/cachefilesd on CephFS pools through a new cephfs.fscache configuration option.

`network_load_balancer` Copy Chapter

This introduces the networking load balancer functionality. Allowing ovn networks to define port(s) on external
IP addresses that can be forwarded to one or more internal IP(s) inside their respective networks.

`vsock_api` Copy Chapter

This introduces a bidirectional vsock interface which allows the incus-agent and the Incus server to communicate better.

`instance_ready_state` Copy Chapter

This introduces a new Ready state for instances which can be set using /dev/incus.

`network_bgp_holdtime` Copy Chapter

This introduces a new bgp.peers.<name>.holdtime configuration key to control the BGP hold time for a particular peer.

`storage_volumes_all_projects` Copy Chapter

This introduces the ability to list storage volumes from all projects.

`metrics_memory_oom_total` Copy Chapter

This introduces a new incus_memory_OOM_kills_total metric to the /1.0/metrics API.
It reports the number of times the out of memory killer (OOM) has been triggered.

`storage_buckets` Copy Chapter

This introduces the storage bucket API. It allows the management of S3 object storage buckets for storage pools.

`storage_buckets_create_credentials` Copy Chapter

This updates the storage bucket API to return initial admin credentials at bucket creation time.

`metrics_cpu_effective_total` Copy Chapter

This introduces a new incus_cpu_effective_total metric to the /1.0/metrics API.
It reports the total number of effective CPUs.

`projects_networks_restricted_access` Copy Chapter

Adds the restricted.networks.access project configuration key to indicate (as a comma-delimited list) which networks can be accessed inside the project.
If not specified, all networks are accessible (assuming it is also allowed by the restricted.devices.nic setting, described below).

This also introduces a change whereby network access is controlled by the project's restricted.devices.nic setting:

• If restricted.devices.nic is set to managed (the default if not specified), only managed networks are accessible.
• If restricted.devices.nic is set to allow, all networks are accessible (dependent on the restricted.networks.access setting).
• If restricted.devices.nic is set to block, no networks are accessible.

`storage_buckets_local` Copy Chapter

This introduces the ability to use storage buckets on local storage pools by setting the new core.storage_buckets_address global configuration setting.

`loki` Copy Chapter

This adds support for sending life cycle and logging events to a Loki server.

It adds the following global configuration keys:

• loki.api.ca_cert: CA certificate which can be used when sending events to the Loki server
• loki.api.url: URL to the Loki server (protocol, name or IP and port)
• loki.auth.username and loki.auth.password: Used if Loki is behind a reverse proxy with basic authentication enabled
• loki.labels: Comma-separated list of values which are to be used as labels for Loki events.
• loki.loglevel: Minimum log level for events sent to the Loki server.
• loki.types: Types of events which are to be sent to the Loki server (lifecycle and/or logging).

`acme` Copy Chapter

This adds ACME support, which allows Let's Encrypt or other ACME services to issue certificates.

It adds the following global configuration keys:

• acme.domain: The domain for which the certificate should be issued.
• acme.email: The email address used for the account of the ACME service.
• acme.ca_url: The directory URL of the ACME service, defaults to https://acme-v02.api.letsencrypt.org/directory.

It also adds the following endpoint, which is required for the HTTP-01 challenge:

• /.well-known/acme-challenge/<token>

`internal_metrics` Copy Chapter

This adds internal metrics to the list of metrics.
These include:

• Total running operations
• Total active warnings
• Daemon uptime in seconds
• Go memory stats
• Number of goroutines

`cluster_join_token_expiry` Copy Chapter

This adds an expiry to cluster join tokens which defaults to 3 hours, but can be changed by setting the cluster.join_token_expiry configuration key.

`remote_token_expiry` Copy Chapter

This adds an expiry to remote add join tokens.
It can be set in the core.remote_token_expiry configuration key, and default to no expiry.

`storage_volumes_created_at` Copy Chapter

This change adds support for storing the creation date and time of storage volumes and their snapshots.

This adds the CreatedAt field to the StorageVolume and StorageVolumeSnapshot API types.

`cpu_hotplug` Copy Chapter

This adds CPU hotplugging for VMs.
Hotplugging is disabled when using CPU pinning, because this would require hotplugging NUMA devices as well, which is not possible.

`projects_networks_zones` Copy Chapter

This adds support for the features.networks.zones project feature, which changes which project network zones are
associated with when they are created. Previously network zones were tied to the value of features.networks,
meaning they were created in the same project as networks were.

Now this has been decoupled from features.networks to allow projects that share a network in the default project
(i.e those with features.networks=false) to have their own project level DNS zones that give a project oriented
"view" of the addresses on that shared network (which only includes addresses from instances in their project).

This also introduces a change to the network dns.zone.forward setting, which now accepts a comma-separated of
DNS zone names (a maximum of one per project) in order to associate a shared network with multiple zones.

No change to the dns.zone.reverse.* settings have been made, they still only allow a single DNS zone to be set.
However the resulting zone content that is generated now includes PTR records covering addresses from all
projects that are referencing that network via one of their forward zones.

Existing projects that have features.networks=true will have features.networks.zones=true set automatically,
but new projects will need to specify this explicitly.

`instance_nic_txqueuelength` Copy Chapter

Adds a txqueuelen key to control the txqueuelen parameter of the NIC device.

`cluster_member_state` Copy Chapter

Adds GET /1.0/cluster/members/<member>/state API endpoint and associated ClusterMemberState API response type.

`instances_placement_scriptlet` Copy Chapter

Adds support for a Starlark scriptlet to be provided to Incus to allow customized logic that controls placement of new instances in a cluster.

The Starlark scriptlet is provided to Incus via the new global configuration option instances.placement.scriptlet.

`storage_pool_source_wipe` Copy Chapter

Adds support for a source.wipe Boolean on the storage pool, indicating
that Incus should wipe partition headers off the requested disk rather
than potentially fail due to pre-existing file systems.

`zfs_block_mode` Copy Chapter

This adds support for using ZFS block {spellexception}filesystem volumes allowing the use of different file systems on top of ZFS.

This adds the following new configuration options for ZFS storage pools:

• volume.zfs.block_mode
• volume.block.mount_options
• volume.block.filesystem

`instance_generation_id` Copy Chapter

Adds support for instance generation ID. The VM or container generation ID will change whenever the instance's place in time moves backwards. As of now, the generation ID is only exposed through to VM type instances. This allows for the VM guest OS to reinitialize any state it needs to avoid duplicating potential state that has already occurred:

• volatile.uuid.generation

`disk_io_cache` Copy Chapter

This introduces a new io.cache property to disk devices which can be used to override the VM caching behavior.

`amd_sev` Copy Chapter

Adds support for AMD SEV (Secure Encrypted Virtualization) that can be used to encrypt the memory of a guest VM.

This adds the following new configuration options for SEV encryption:

• security.sev : (bool) is SEV enabled for this VM
• security.sev.policy.es : (bool) is SEV-ES enabled for this VM
• security.sev.session.dh : (string) guest owner's base64-encoded Diffie-Hellman key
• security.sev.session.data : (string) guest owner's base64-encoded session blob

`storage_pool_loop_resize` Copy Chapter

This allows growing loop file backed storage pools by changing the size setting of the pool.

`migration_vm_live` Copy Chapter

This adds support for performing VM QEMU to QEMU live migration for both shared storage (clustered Ceph) and
non-shared storage pools.

This also adds the CRIUType_VM_QEMU value of 3 for the migration CRIUType protobuf field.

`ovn_nic_nesting` Copy Chapter

This adds support for nesting an ovn NIC inside another ovn NIC on the same instance.
This allows for an OVN logical switch port to be tunneled inside another OVN NIC using VLAN tagging.

This feature is configured by specifying the parent NIC name using the nested property and the VLAN ID to use for tunneling with the vlan property.

`oidc` Copy Chapter

This adds support for OpenID Connect (OIDC) authentication.

This adds the following new configuration keys:

• oidc.issuer
• oidc.client.id
• oidc.audience

`network_ovn_l3only` Copy Chapter

This adds the ability to set an ovn network into "layer 3 only" mode.
This mode can be enabled at IPv4 or IPv6 level using ipv4.l3only and ipv6.l3only configuration options respectively.

With this mode enabled the following changes are made to the network:

• The virtual router's internal port address will be configured with a single host netmask (e.g. /32 for IPv4 or /128 for IPv6).
• Static routes for active instance NIC addresses will be added to the virtual router.
• A discard route for the entire internal subnet will be added to the virtual router to prevent packets destined for inactive addresses from escaping to the uplink network.
• The DHCPv4 server will be configured to indicate that a netmask of 255.255.255.255 be used for instance configuration.

`ovn_nic_acceleration_vdpa` Copy Chapter

This updates the ovn_nic_acceleration API extension. The acceleration configuration key for OVN NICs can now takes the value vdpa to support Virtual Data Path Acceleration (VDPA).

`cluster_healing` Copy Chapter

This adds cluster healing which automatically evacuates offline cluster members.

This adds the following new configuration key:

• cluster.healing_threshold

The configuration key takes an integer, and can be disabled by setting it to 0 (default). If set, the value represents the threshold after which an offline cluster member is to be evacuated. In case the value is lower than cluster.offline_threshold, that value will be used instead.

When the offline cluster member is evacuated, only remote-backed instances will be migrated. Local instances will be ignored as there is no way of migrating them once the cluster member is offline.

`instances_state_total` Copy Chapter

This extension adds a new total field to InstanceStateDisk and InstanceStateMemory, both part of the instance's state API.

`auth_user` Copy Chapter

Add current user details to the main API endpoint.

This introduces:

• auth_user_name
• auth_user_method

`security_csm` Copy Chapter

Introduce a new security.csm configuration key to control the use of
CSM (Compatibility Support Module) to allow legacy operating systems to
be run in Incus VMs.

`instances_rebuild` Copy Chapter

This extension adds the ability to rebuild an instance with the same origin image, alternate image or as empty. A new POST /1.0/instances/<name>/rebuild?project= API endpoint has been added as well as a new CLI command incus rebuild.

`numa_cpu_placement` Copy Chapter

This adds the possibility to place a set of CPUs in a desired set of NUMA nodes.

This adds the following new configuration key:

• limits.cpu.nodes : (string) comma-separated list of NUMA node IDs or NUMA node ID ranges to place the CPUs (chosen with a dynamic value of limits.cpu) in.

`custom_volume_iso` Copy Chapter

This adds the possibility to import ISO images as custom storage volumes.

This adds the --type flag to incus storage volume import.

`network_allocations` Copy Chapter

This adds the possibility to list an Incus deployment's network allocations.

Through the incus network list-allocations command and the --project | --all-projects flags,
you can list all the used IP addresses, hardware addresses (for instances), resource URIs and whether it uses NAT for
each instance, network, network forward and network load-balancer.

`zfs_delegate` Copy Chapter

This implements a new zfs.delegate volume Boolean for volumes on a ZFS storage driver.
When enabled and a suitable system is in use (requires ZFS 2.2 or higher), the ZFS dataset will be delegated to the container, allowing for its use through the zfs command line tool.

`storage_api_remote_volume_snapshot_copy` Copy Chapter

This allows copying storage volume snapshots to and from remotes.

`operations_get_query_all_projects` Copy Chapter

This introduces support for the all-projects query parameter for the GET API calls to both /1.0/operations and /1.0/operations?recursion=1.
This parameter allows bypassing the project name filter.

`metadata_configuration` Copy Chapter

Adds the GET /1.0/metadata/configuration API endpoint to retrieve the generated metadata configuration in a JSON format. The JSON structure adopts the structure "configs" > `ENTITY` > `ENTITY_SECTION` > "keys" > [<CONFIG_OPTION_0>, <CONFIG_OPTION_1>, ...].

`syslog_socket` Copy Chapter

This introduces a syslog socket that can receive syslog formatted log messages. These can be viewed in the events API and incus monitor, and can be forwarded to Loki. To enable this feature, set core.syslog_socket to true.

`event_lifecycle_name_and_project` Copy Chapter

This adds the fields Name and Project to lifecycle events.

`instances_nic_limits_priority` Copy Chapter

This introduces a new per-NIC limits.priority option that works with both cgroup1 and cgroup2 unlike the deprecated limits.network.priority instance setting, which only worked with cgroup1.

`disk_initial_volume_configuration` Copy Chapter

This API extension provides the capability to set initial volume configurations for instance root devices.
Initial volume configurations are prefixed with initial. and can be specified either through profiles or directly
during instance initialization using the --device flag.

Note that these configuration are applied only at the time of instance creation and subsequent modifications have
no effect on existing devices.

`operation_wait` Copy Chapter

This API extension indicates that the /1.0/operations/{id}/wait endpoint exists on the server. This indicates to the client
that the endpoint can be used to wait for an operation to complete rather than waiting for an operation event via the
/1.0/events endpoint.

`image_restriction_privileged` Copy Chapter

This extension adds a new image restriction, requirements.privileged which when false indicates that an image cannot be run in a privileged container.

`cluster_internal_custom_volume_copy` Copy Chapter

This extension adds support for copying and moving custom storage volumes within a cluster with a single API call.
Calling POST /1.0/storage-pools//custom?target=<target> will copy the custom volume specified in the source part of the request.
Calling POST /1.0/storage-pools//custom/<volume>?target=<target> will move the custom volume from the source, specified in the source part of the request, to the target.

`disk_io_bus` Copy Chapter

This introduces a new io.bus property to disk devices which can be used to override the bus the disk is attached to.

`storage_cephfs_create_missing` Copy Chapter

This introduces the configuration keys cephfs.create_missing, cephfs.osd_pg_num, cephfs.meta_pool and cephfs.osd_pool to be used when adding a cephfs storage pool to instruct Incus to create the necessary entities for the storage pool, if they do not exist.

`instance_move_config` Copy Chapter

This API extension provides the ability to use flags --profile, --no-profile, --device, and --config
when moving an instance between projects and/or storage pools.

`ovn_ssl_config` Copy Chapter

This introduces new server configuration keys to provide the SSL CA and client key pair to access the OVN databases.
The new configuration keys are network.ovn.ca_cert, network.ovn.client_cert and network.ovn.client_key.

`certificate_description` Copy Chapter

Adds a description field to certificate.

`disk_io_bus_virtio_blk` Copy Chapter

Adds a new virtio-blk value for io.bus on disk devices which allows
for the attached disk to be connected to the virtio-blk bus.

`loki_config_instance` Copy Chapter

Adds a new loki.instance server configuration key to customize the instance field in Loki events.
This can be used to expose the name of the cluster rather than the individual system name sending
the event as that's usually already covered by the location field.

`instance_create_start` Copy Chapter

Adds a new start field to the POST /1.0/instances API which when set
to true will have the instance automatically start upon creation.

In this scenario, the creation and startup is part of a single background operation.

`clustering_evacuation_stop_options` Copy Chapter

This introduces new options for the cluster.evacuate option:

• stateful-stop has the instance store its state to disk to be resume on restore.
• force-stop has the instance immediately stopped without waiting for it to shut down.

`boot_host_shutdown_action` Copy Chapter

This introduces a new boot.host_shutdown_action instance configuration key which can be used to override the default stop behavior on system shutdown.
It supports the value stop, stateful-stop and force-stop.

`agent_config_drive` Copy Chapter

This introduces a new agent:config disk source which can be used to expose an ISO to the VM guest containing the agent and its configuration.

`network_state_ovn_lr` Copy Chapter

Adds a new LogicalRouter field to the NetworkStateOVN struct which is part of the GET /1.0/networks/NAME/state API.

This is used to get the OVN logical router name.

`image_template_permissions` Copy Chapter

This adds uid, gid and mode fields to the image metadata template entries.

`storage_bucket_backup` Copy Chapter

Add storage bucket backup support.

This includes the following new endpoints (see RESTful API for details):

• GET /1.0/storage-pools//buckets//backups

• POST /1.0/storage-pools//buckets//backups

• GET /1.0/storage-pools//buckets//backups/<name>

• POST /1.0/storage-pools//buckets//backups/<name>

• DELETE /1.0/storage-pools//buckets//backups/<name>

• GET /1.0/storage-pools//buckets//backups/<name>/export

`storage_lvm_cluster` Copy Chapter

This adds a new lvmcluster storage driver which makes use of LVM shared VG through lvmlockd.

With this, it's possible to have a single shared LVM pool across multiple servers so long as they all see the same backing device(s).

`shared_custom_block_volumes` Copy Chapter

This adds a new configuration key security.shared to custom block volumes.
If unset or false, the custom block volume cannot be attached to multiple instances.
This feature was added to prevent data loss which can happen when custom block volumes are attached to multiple instances at once.

`auth_tls_jwt` Copy Chapter

This adds the ability to use a signed JSON Web Token (JWT) instead of using the TLS client certificate directly.

In this scenario, the client derives a JWT from their own TLS client certificate providing it as a bearer token through the Authorization HTTP header.

The JWT must have the certificate's fingerprint as its Subject and must be signed by the client's private key.

`oidc_claim` Copy Chapter

This introduces a new oidc.claim server configuration key which can be used to specify what OpenID Connect claim to use as the username.

`device_usb_serial` Copy Chapter

This adds a new configuration key serial for device type usb.
Feature has been added, to make it possible to distinguish between devices with identical vendorid and productid.

`numa_cpu_balanced` Copy Chapter

This adds balanced as a new value for limits.cpu.nodes.

When set to balanced, Incus will attempt to select the least busy NUMA
node at startup time for the instance, trying to keep the load spread
across NUMA nodes on the system.

`image_restriction_nesting` Copy Chapter

This extension adds a new image restriction, requirements.nesting which when true indicates that an image cannot be run without nesting.

`network_integrations` Copy Chapter

Adds the concept of network integrations and initial support for OVN Interconnection.

New API:

• /1.0/network-integrations (GET, POST)
• /1.0/network-integrations/NAME (GET, PUT, PATCH, DELETE, POST)

Each integration is made of:

• name
• description
• type (only ovn for now)
• configuration
  • ovn.northbound_connection (database connection string for the OVN Interconnection database)
  • ovn.ca_cert (optional, SSL CA certificate for the OVN Interconnection database)
  • ovn.client_cert (optional, SSL client certificate to connect to the OVN Interconnection database)
  • ovn.client_key (optional, SSL client key to connect to the OVN Interconnection database)
  • ovn.transit.pattern (Pongo2 template to generate the transit switch name)

Those integrations attach to network peers through some new fields:

• type (local for current behavior, remote for integrations)
• target_integration (reference to the integration)

`instance_memory_swap_bytes` Copy Chapter

This extends limits.memory.swap to allow for a total limit in bytes.

`network_bridge_external_create` Copy Chapter

This adds the ability for bridge.external_interfaces to create a parent interface using a interface/parent/vlan syntax.

`storage_zfs_vdev` Copy Chapter

This adds support for mirror, raidz1 and raidz2 ZFS vdev types by extending storage source configuration.

`container_migration_stateful` Copy Chapter

A migration.stateful configuration key was introduced.

It's a Boolean flag set to true whenever the container is in a stateful mode
during the start, stop, and snapshot functions.

This makes it less likely for users to run into CRIU errors when copying
containers to another system.

`profiles_all_projects` Copy Chapter

This adds support for listing profiles across all projects through the all-projects parameter on the GET /1.0/profilesAPI.

`instances_scriptlet_get_instances` Copy Chapter

This allows the instance scriptlet to fetch a list of instances given an optional Project or Location filter.

`instances_scriptlet_get_cluster_members` Copy Chapter

This allows the instance scriptlet to fetch a list of cluster members given an optional cluster group.

`instances_scriptlet_get_project` Copy Chapter

This allows the instance scriptlet to fetch a project given name of a project.

`network_acl_stateless` Copy Chapter

This adds support for stateless rules in network ACLs.

`instance_state_started_at` Copy Chapter

This adds a started_at timestamp to the instance state API.

`networks_all_projects` Copy Chapter

This adds support for listing networks across all projects through the all-projects parameter on the GET /1.0/networksAPI.

`network_acls_all_projects` Copy Chapter

This adds support for listing network ACLs across all projects through the all-projects parameter on the GET /1.0/network-aclsAPI.

`storage_buckets_all_projects` Copy Chapter

This adds support for listing storage buckets across all projects through the all-projects parameter on the GET /1.0/storage-pools/POOL/bucketsAPI.

`resources_load` Copy Chapter

Add a new Load section to the resources API.

`instance_access` Copy Chapter

This introduces a new API endpoint at GET /1.0/instances/NAME/access which exposes who can interact with the instance and what role they have.

`project_access` Copy Chapter

This introduces a new API endpoint at GET /1.0/projects/NAME/access which exposes who can interact with the project and what role they have.

`projects_force_delete` Copy Chapter

This extends DELETE /1.0/projects to allow ?force=true which will
delete everything inside of the project along with the project itself.

`resources_cpu_flags` Copy Chapter

This exposes the CPU flags/extensions in our resources API to check the CPU features.

`disk_io_bus_cache_filesystem` Copy Chapter

This adds support for both io.bus and io.cache to disks that are backed by a file system.

`instance_oci` Copy Chapter

Adds initial support for running OCI containers.

`clustering_groups_config` Copy Chapter

This introduces a standard key/value config option to clustering
groups which will allow placing some restrictions or configuration on
those groups.

`instances_lxcfs_per_instance` Copy Chapter

This introduces a new instances.lxcfs.per_instance server
configuration key to control whether to run LXCFS per instance instead
of globally on the system.

`clustering_groups_vm_cpu_definition` Copy Chapter

This introduces a few new configuration options to control the virtual machine CPU definitions through cluster group configuration.

The new configuration keys are:

• instances.vm.cpu.ARCHITECTURE.baseline
• instances.vm.cpu.ARCHITECTURE.flag

`disk_volume_subpath` Copy Chapter

This introduces the ability to access the sub-path of a file system custom volume by using the source=volume/path syntax.

`projects_limits_disk_pool` Copy Chapter

This introduces per-pool project disk limits, introducing a limits.disk.pool.NAME configuration option to the project limits.

`network_ovn_isolated` Copy Chapter

This allows using none as the uplink network for an OVN network, making the network isolated.

`qemu_raw_qmp` Copy Chapter

This adds new configuration options to virtual machines to directly issue QMP commands at various stages of startup:

• raw.qemu.qmp.early
• raw.qemu.qmp.pre-start
• raw.qemu.qmp.post-start

`network_load_balancer_health_check` Copy Chapter

This adds the ability to perform health checks for load balancer backends.

The following new configuration options are introduced:

• healthcheck
• healthcheck.interval
• healthcheck.timeout
• healthcheck.failure_count
• healthcheck.success_count

`oidc_scopes` Copy Chapter

This introduces a new oidc.scopes server configuration key which can take a comma separate list of OIDC scopes to request from the identity provider.

`network_integrations_peer_name` Copy Chapter

This extends ovn.transit.pattern to allow peerName as a template variable.

`qemu_scriptlet` Copy Chapter

This adds the ability to run a scriptlet at various stages of startup: using the raw.qemu.scriptlet configuration key.

`instance_auto_restart` Copy Chapter

This introduces a new boot.autorestart configuration key which when
set to true will have the instance automatically be restarted upon
unexpected exit for up to 10 times over a 1 minute period.

`storage_lvm_metadatasize` Copy Chapter

This introduces a new lvm.metadata_size option for LVM storage pools
which allows overriding the default metadata size when creating a new
LVM physical volume.

`ovn_nic_promiscuous` Copy Chapter

This implements a new security.promiscuous configuration option on OVN NICs.

`ovn_nic_ip_address_none` Copy Chapter

This adds none as a value for ipv4.address and ipv6.address for OVN NICs.

`instances_state_os_info` Copy Chapter

This extension adds a pointer to an InstanceStateOSInfo struct to the instance's state API.

`network_load_balancer_state` Copy Chapter

This adds a new /1.0/networks/NAME/load-balancers/IP/state API endpoint
which returns load-balancer health check information (when configured).

`instance_nic_macvlan_mode` Copy Chapter

This adds a mode configuration key on macvlan network interfaces which allows for configuring the Macvlan mode.

`storage_lvm_cluster_create` Copy Chapter

Allow for creating new LVM cluster pools by setting the source to the shared block device.

`network_ovn_external_interfaces` Copy Chapter

This adds support for bridge.external_interfaces on OVN networks.

`instances_scriptlet_get_instances_count` Copy Chapter

This allows the instance scriptlet to fetch the count instances given an optional Project or Location filter as well as including pending instances.

`cluster_rebalance` Copy Chapter

This adds automatic live-migration to balance load on cluster again.

As part of this, the following configuration options have been added:

• cluster.rebalance.batch
• cluster.rebalance.cooldown
• cluster.rebalance.interval
• cluster.rebalance.threshold

`custom_volume_refresh_exclude_older_snapshots` Copy Chapter

This adds support for excluding source snapshots earlier than latest target snapshot.

`storage_initial_owner` Copy Chapter

This adds ability to set the initial owner of a custom volume.

The following configuration options have been added:

• initial.gid
• initial.mode
• initial.uid

`storage_live_migration` Copy Chapter

This adds support for virtual-machines live-migration between storage pools.

`instance_console_screenshot` Copy Chapter

This adds support to take screenshots of the current VGA console of a VM.

`image_import_alias` Copy Chapter

Adds a new X-Incus-aliases HTTP header to set aliases while uploading an image.

`authorization_scriptlet` Copy Chapter

This adds the ability to define a scriptlet in a new configuration key, authorization.scriptlet, managing authorization on the Incus cluster.

`console_force` Copy Chapter

This adds support for forcing a connection to the console, even if there is already an active session.
It introduces the new --force flag for connecting to the instance console.

`network_ovn_state_addresses` Copy Chapter

This adds extra fields to the OVN network state struct for the IPv4 and IPv6 addresses used on the uplink.

`qemu_scriptlet_config` Copy Chapter

This extends the QEMU scriptlet feature by allowing to modify QEMU configuration before a VM starts, and passing information about the instance to the scriptlet.

`network_bridge_acl_devices` Copy Chapter

This adds support for device ACLs when attached to a bridged network.

`instance_debug_memory` Copy Chapter

Add new memory dump API at /1.0/instances/NAME/debug/memory.

`init_preseed_storage_volumes` Copy Chapter

This API extension provides the ability to configure storage volumes in preseed init.

`init_preseed_profile_project` Copy Chapter

This API extension provides the ability to specify the project as part of profile definitions in preseed init.

`instance_nic_routed_host_address` Copy Chapter

Adds support for specifying the VRF to add the routes to.

`instance_smbios11` Copy Chapter

A new category of configuration options, smbios11.XYZ has been added
which allows passing key/value pairs through SMBIOS Type 11 on systems that
support it.

`api_filtering_extended` Copy Chapter

This extends the API filtering mechanism to all API collections.

`acme_dns01` Copy Chapter

Adds support for DNS-01 challenge to the Incus ACME support for certificate generation.

`security_iommu` Copy Chapter

Introduce a new security.iommu configuration key to control whether to
enable IOMMU emulation. This is done through virtio_iommu on Linux and the emulated Intel IOMMU on Windows.

`network_ipv4_dhcp_routes` Copy Chapter

Introduces a new ipv4.dhcp.routes configuration option on bridged and OVN networks.
This allows specifying pairs of CIDR networks and gateway address to be announced by the DHCP server.

`network_state_ovn_ls` Copy Chapter

Adds a new LogicalSwitch field to the NetworkStateOVN struct which is part of the GET /1.0/networks/NAME/state API.

This is used to get the OVN logical switch name.

`network_dns_nameservers` Copy Chapter

Introduces the dns.nameservers configuration option on bridged and OVN networks.
This allows specifying IPv4 and IPv6 DNS server addresses to be announced by the DHCP server and via Router Advertisements.

`acme_http01_port` Copy Chapter

Adds acme.http.port to control an alternative HTTP port for HTTP-01 validation.

`network_ovn_ipv4_dhcp_expiry` Copy Chapter

Introduces ipv4.dhcp.expiry for OVN networks.

`instance_state_cpu_time` Copy Chapter

This adds an allocated_time field below CPU in the instance state API.

`network_io_bus` Copy Chapter

This introduces a new io.bus property for compatible network devices allowing to choose between virtio (default) and usb.

`disk_io_bus_usb` Copy Chapter

Adds a new usb value for io.bus on disk devices.

`storage_driver_linstor` Copy Chapter

This adds a LINSTOR storage driver.

`instance_oci_entrypoint` Copy Chapter

This introduces a set of new configuration options on the container to configure the OCI entry point:

• oci.entrypoint
• oci.cwd
• oci.uid
• oci.gid

Those are initialized at creation time using the values from the OCI image.

`network_address_set` Copy Chapter

This adds the concept of network address sets to API under the API endpoint prefix /1.0/network-address-sets.

`server_logging` Copy Chapter

This implements a new set of logging configuration keys on the server, allowing for multiple logging targets.
The former loki configuration keys are being transitioned over as part of this.

`network_forward_snat` Copy Chapter

Adds a snat configuration option for network forwards which will cause any DNAT to get a matching SNAT applied.
So new connections from the target will appear as coming from the network forward address.

This is limited to bridged networks as OVN doesn't support flexible enough SNAT for this.

`memory_hotplug` Copy Chapter

This adds memory hotplugging for VMs, allowing them to add memory at runtime without rebooting.

`instance_nic_routed_host_tables` Copy Chapter

This adds support for specifying host-routing tables on nic devices that use the routed mode.

`instance_publish_split` Copy Chapter

This adds support for creating a split format image out of an existing instance.

`init_preseed_certificates` Copy Chapter

This API extension provides the ability to configure certificates in preseed init.

`custom_volume_sftp` Copy Chapter

This adds the SFTP API to custom storage volumes.

`network_ovn_external_nic_address` Copy Chapter

This adds support for configuring a custom external IPv4 or IPv6 address
for a given instance so long as that address is available through a
network forward.

`network_physical_gateway_hwaddr` Copy Chapter

Allows setting the MAC address of the IPv4 and IPv6 gateways when used with OVN.

`backup_s3_upload` Copy Chapter

Adds support for immediately uploading instance or volume backups to an S3 compatible endpoint.

`snapshot_manual_expiry` Copy Chapter

Introduces a snapshots.expiry.manual configuration key to both
instances and storage volumes which allows overriding the default expiry
value for snapshots created directly by the user as opposed to created
on schedule.

`resources_cpu_address_sizes` Copy Chapter

This adds tracking of CPU address sizes in the resources API.
The main use of this is within clusters to calculate a cluster-wide
maximum memory amount for hotplugging into virtual machines.

`disk_attached` Copy Chapter

This introduces a new attached property to disk devices describing whether disks are attached or ejected.

`limits_memory_hotplug` Copy Chapter

The limits.memory.hotplug option controls how memory hotplug is handled for the virtual machine.
It can be set to false to completely disable memory hotplugging.
Alternatively, it can be set to a value that defines the maximum amount of memory the VM can reach through hotplug.
This value must be greater than or equal to limits.memory.

`disk_wwn` Copy Chapter

Add support for setting the disk World Wide Name property through the new wwn disk configuration option.

`server_logging_webhook` Copy Chapter

This adds support for basic webhook as a logging target.

It can be selected through logging.NAME.target.type with the webhook value.

The following target keys are supported:

• logging.NAME.target.address (URL of the target)
• logging.NAME.target.ca_cert (Certificate when using an HTTPS target with a self-signed certificate)
• logging.NAME.target.username (Username for HTTP authentication)
• logging.NAME.target.password (Password for HTTP authentication)
• logging.NAME.target.retry (How many times to retry the transmission)

The webhook data matches what's sent over the existing events API.

`storage_driver_truenas` Copy Chapter

This adds a TrueNAS storage driver.

`container_disk_tmpfs` Copy Chapter

This adds tmpfs support for disk devices.

• source=tmpfs: mounts a tmpfs file system, respecting size, uid, gid and mode options
• source=tmpfs-overlay: same as tmpfs but with additional overlayfs behavior

`instance_limits_oom` Copy Chapter

This adds a new limits.memory.oom_priority configuration option to
configure the Out Of Memory score for the container or virtual-machine.

`backup_override_config` Copy Chapter

This adds support for overriding both configuration and devices during
backup import by using the X-Incus-config and X-Incus-devices HTTP
headers.

`network_ovn_tunnels` Copy Chapter

This adds support for network tunnels to OVN networks.

`init_preseed_cluster_groups` Copy Chapter

This API extension provides the ability to configure cluster groups in preseed init.

`usb_attached` Copy Chapter

This introduces a new attached property to USB devices describing whether they are plugged in or not.

`backup_iso` Copy Chapter

This allows to backup ISO custom volumes by simply copying them. It enables exporting ISO volumes with incus storage volume export and get an ISO back.

`instance_systemd_credentials` Copy Chapter

This adds two categories of configuration options, systemd.credential.* and systemd.credential-binary.*, which allow passing systemd credentials through a bind-mounted directory for containers and SMBIOS Type 11 for virtual machines.

`cluster_group_usedby` Copy Chapter

A used_by field was added to the GET /1.0/cluster/groups/{name} endpoint.
used_by holds the URLs of all instances and projects using the cluster group.

`bpf_token_delegation` Copy Chapter

This adds support for eBPF token delegation.

`file_storage_volume` Copy Chapter

This adds file transfer API on the custom volumes.
Implements DELETE, GET, HEAD, POST operations on the /1.0/storage-pools/{poolName}/volumes/{type}/{volumeName}/files endpoint.

`network_hwaddr_pattern` Copy Chapter

This adds network.hwaddr_pattern global and per-project configuration keys to customize MAC address allocation.

`storage_volume_full` Copy Chapter

This adds ?recursion=1 support to /1.0/storage-pools/POOL/volumes/VOLUME and ?recursion=2 support to /1.0/storage-pools/POOL/volumes.

As well as a matching StorageVolumeFull struct.

`storage_bucket_full` Copy Chapter

This adds ?recursion=1 support to /1.0/storage-pools/POOL/buckets/VOLUME and ?recursion=2 support to /1.0/storage-pools/POOL/buckets.

As well as a matching StorageBucketFull struct.

`device_pci_firmware` Copy Chapter

This extension introduces a new boolean firmware option for PCI devices.
When set to false, Incus tells QEMU to disable the ROM-BAR (rombar=0) for the device.

`resources_serial` Copy Chapter

This adds serial device tracking to the resources API.

`ovn_nic_limits` Copy Chapter

This adds support for limits.egress, limits.ingress, limits.max and limits.priority on ovn type network devices.

`storage_lvmcluster_qcow2` Copy Chapter

This adds support for running virtual machines from QCOW2 images. Support is currently limited to lvmcluster storage driver.

`oidc_allowed_subnets` Copy Chapter

This indicates support by Incus for the custom incus.allowed_subnets OIDC claim.

`file_delete_force` Copy Chapter

This adds a new X-Incus-force HTTP header that when set to true allows for recursive deletion of instance or custom volume files.

`nic_sriov_select_ext` Copy Chapter

Adds support for selecting an SR-IOV network interface by vendor ID, product ID, or PCI address.

`network_zones_dns_contact` Copy Chapter

Adds a dns.contact configuration key to network zones.

`nic_attached_connected` Copy Chapter

This introduces two new properties for NICs:

• attached, behaving like the attached key for disk and USB devices;
• connected, setting the up/down link state for the NIC (when supported).

`nic_sriov_security_trusted` Copy Chapter

This introduces a new property for SR-IOV NICs:

• security.trusted allows setting the trusted flag for the virtual function if the parent NIC supports it.

`direct_backup` Copy Chapter

This allows to perform backups of instances, custom storage volumes and storage buckets without disk buffering when the Accept header is set to application/octet-stream in the respective POST API endpoints. Doing so, said POST endpoints return the data stream instead of creating a backup entry in the database.

`instance_snapshot_disk_only_restore` Copy Chapter

This adds support for only restoring the disk of a snapshotted instance.

`unix_hotplug_pci` Copy Chapter

Adds a pci configuration key on the unix-hotplug device to allow filtering based on the USB controller PCI address.

`cluster_evacuating_restoring` Copy Chapter

Introduces new Evacuating and Restoring cluster member states.

`projects_restricted_image_servers` Copy Chapter

Introduces a new restricted.images.servers project configuration option.
This allows specifying a comma separate list of image server domains from which the user may download images.

`storage_lvmcluster_size` Copy Chapter

Adds support for specifying the size parameter when updating lvmcluster storage pool.

`authorization_scriptlet_cert` Copy Chapter

This adds two fields, Chain and Certificate, to the details
argument of the authorization scriptlet, to give more information about
the client to authorize.

`lvmcluster_remove_snapshots` Copy Chapter

It is a Boolean that defaults to false. When set to true, it instructs Incus
to remove any required snapshots when attempting to restore another snapshot.

This is necessary because lvmcluster only allows restoring the latest snapshot.

`daemon_storage_logs` Copy Chapter

This adds storage.logs_volume alongside the existing storage.backups_volume and storage.images_volume.

`instances_debug_repair` Copy Chapter

This adds a new API at POST /1.0/instances/NAME/debug/repair to trigger low-level repair actions.

`network_io_bus_ovn` Copy Chapter

This ports the io.bus property available on most NIC devices to non-accelerated OVN NICs.

`dependent` Copy Chapter

This introduces a new dependent configuration key for disk devices and custom volumes.
Volumes marked as dependent are tied to the lifecycle of the instance they are attached to.
Snapshot operations on the instance are automatically propagated to the dependent volume.
Creating an instance snapshot creates a corresponding snapshot on the dependent volume,
and deleting the instance snapshot removes it as well.
Direct snapshot creation or deletion on a dependent volume is not allowed.
Exporting and importing an instance also includes dependent volumes.

`metrics_project_resources` Copy Chapter

This adds project-level metrics to the /1.0/metrics endpoint including resource counts, configured limits, and current usage per project.

`storage_volume_nbd` Copy Chapter

This adds a few new APIs for storage volumes and instances:

• POST /1.0/storage-pools/POOL/volumes/TYPE/VOLUME/nbd
• GET /1.0/storage-pools/POOL/volumes/TYPE/VOLUME/bitmaps
• POST /1.0/storage-pools/POOL/volumes/TYPE/VOLUME/bitmaps
• GET /1.0/storage-pools/POOL/volumes/TYPE/VOLUME/bitmaps/NAME
• DELETE /1.0/storage-pools/POOL/volumes/TYPE/VOLUME/bitmaps/NAME
• POST /1.0/instances/NAME/bitmaps

This introduces the ability to get a raw NBD connection to an Incus block storage volume.
It also allows interacting with dirty bitmaps on those volumes which can then be used as the basis for incremental backups.
The instance level endpoint allows for consistent bitmap creation across the VM and its dependent volumes.

`projects_restricted_storage_pool_access` Copy Chapter

Adds the restricted.storage-pools.access project configuration key to
indicate (as a comma-delimited list) which storage pools can be accessed
inside the project. If not specified, all storage pools are accessible.

Storage pools that are not in the list are treated as equivalent to
having a pool size limit of 0 (limits.disk.pool.POOLNAME=0), making
them inaccessible and hidden from the project.

`server_shutdown_action` Copy Chapter

This adds a new core.shutdown_action server configuration option.

Its default value of shutdown keeps the current behavior but
evacuate can be set to have a clustered server attempt an evacuation
on shutdown with any instance left after it getting shutdown.

`instances_placement_scriptlet_rebalance` Copy Chapter

Add a new placement scriptlet trigger for cluster re-balancing.

`network_bridge_multicast_snooping` Copy Chapter

This adds a new bridge.multicast_snooping configuration option for
managed bridge networks. When set to false, multicast snooping is
disabled on the bridge, which can help with IPv6 protocols that rely
on multicast such as NDP.

`storage_create_options` Copy Chapter

This introduces a new block.create_options configuration option to
allow controlling the mkfs arguments when creating volumes on block
devices.

btrfs.create_options is also introduced to similarly control
mkfs.btrfs options when creating a new storage pool.

`instances_tpm_platform_cert` Copy Chapter

This introduces a couple of new configuration options:

• instances.tpm.platform_cert
• instances.tpm.platform_key

Those can be used to provide a TPM platform CA certificate.

When set, new TPM devices will be getting an Endorsement Key that's
signed by that CA. This can be used to validate that the VM is in fact
running in the expected Incus environment, providing a root of trust for
those TPM devices.

`linstor_raw` Copy Chapter

This introduces linstor.raw.* configuration keys for both LINSTOR
storage pools and storage volumes, allowing to set low-level
properties to resource groups and resource definitions.

`network_address_set_ip_ranges` Copy Chapter

This adds support for IP ranges (e.g. 10.0.0.120-10.0.0.130) in network
address sets, matching what's already allowed in network ACL rules.

A single range may expand to at most 256 addresses. Larger sets of
addresses should be expressed using CIDR notation instead.

`storage_volumes_rebuild` Copy Chapter

This extension adds the ability to rebuild a custom storage volume.
The underlying volume is deleted and a new empty one is created with
the same configuration. The rebuild is only allowed when the volume
has no snapshots.

`api_fragments` Copy Chapter

URL fragments in API paths are to be treated by clients as references
to configuration keys for the objects said paths point to.

For example, /1.0?target=foo#storage.logs_volume refers to the
storage.logs_volume configuration key for the cluster member foo.

`instance_limits_cpu_topology` Copy Chapter

This extends the limits.cpu configuration key for virtual machines to
allow specifying an explicit CPU topology of the form
sockets=2,cores=4,threads=2.

`instance_nbd` Copy Chapter

This adds a new GET /1.0/instances/{name}/nbd endpoint which acts
similarly to the equivalent storage volume endpoint but allows concurrent
access to all disks attached to the VM.

`network_bridge_bgp_instances` Copy Chapter

This adds the bgp.ipv4.instances and bgp.ipv6.instances configuration
keys to managed bridge networks.

When enabled, Incus advertises a /32 (IPv4) or /128 (IPv6) route over
BGP for each running instance connected to the network, withdrawing it
again when the instance stops.

`core_https_allowed_websocket_origin` Copy Chapter

This adds a new core.https_allowed_websocket_origin server
configuration key. It can be set to a comma-separate list of allowed
origins or to the * wildcard.

`storage_btrfs_compression` Copy Chapter

This adds a new btrfs.compression storage volume configuration key for
the btrfs driver. It maps to the Btrfs compression property and takes
the same values (for example zstd, lzo, zlib or none).

`oci_network_config` Copy Chapter

This allows for static network configuration of OCI application containers.

The NIC ipv4.address and ipv6.address keys can now be set to a CIDR
value to statically configure the address inside the container and the new
ipv4.gateway and ipv6.gateway keys can be used to set the default
gateway. Setting either address to none prevents any configuration for
that address family and stops the built-in DHCP client from running on it.

For DNS, the oci.dns.nameservers, oci.dns.domain and oci.dns.search
instance configuration keys can be used to set the initial content of the
container's resolv.conf, which is then extended with any value received
over DHCP.

All of those keys are only valid for OCI containers.

`infiniband_sriov_guid` Copy Chapter

This adds two new configuration keys, node_guid and port_guid, to
infiniband devices using the sriov nictype.

When set, the matching GUID of the allocated virtual function is changed
to the provided value when the instance starts and restored to its
original value when the instance stops.

`instance_selinux` Copy Chapter

This adds per-instance SELinux integration for containers and virtual
machines, with automatic MCS level allocation for isolation between
instances on the same host.

The following instance configuration keys are added:

• security.selinux.domain: Override the SELinux process domain.
• security.selinux.type: Override the SELinux file type used for the
  instance storage.
• security.selinux.level: Override the SELinux MCS level.
• security.selinux.label_rootfs: Control rootfs labeling behavior
  (auto, always or never).

The computed context is persisted in the volatile.selinux.context key so
that MCS ranges stay stable across restarts.

`network_bgp_peer_interface` Copy Chapter

This adds a new bgp.peers.NAME.interface configuration key to bridge
and physical networks as an alternative to bgp.peers.NAME.address.

When set, the BGP session is established over the given interface using
BGP unnumbered (the peer's IPv6 link-local address is discovered
automatically and IPv4 routes are exchanged using extended next-hop).

`projects_restricted_virtual_machines_nesting` Copy Chapter

This adds a new restricted.virtual-machines.nesting project configuration key.

When set to block, all virtual machines in the project must have
security.nesting set to false, turning off nested virtualization.

`authorization_config` Copy Chapter

This moves the authorization configuration keys under a general
authorization.* namespace.

The following server configuration keys replace the former openfga.* keys:

• authorization.openfga.api.url: URL of the OpenFGA server
• authorization.openfga.api.token: API token of the OpenFGA server
• authorization.openfga.store.id: ID of the OpenFGA permission store

Existing openfga.* values are automatically migrated to the new keys on
upgrade.

`network_allocations_network` Copy Chapter

Adds the network field to the network allocations API response.

`gpu_native_context` Copy Chapter

This adds a new native-context gputype for gpu devices on virtual machines.
Instead of passing through a PCI device, it gives the VM an accelerated virtio-gpu
that uses DRM native context, so the guest's own driver drives the host GPU
directly, with graphics and video acceleration. One host GPU can be shared by several
VMs and the host keeps using it. The blob.size device option sets the
host-visible blob window (default 2GiB). Requires QEMU 11.0.0 or newer.

`instance_port_forward` Copy Chapter

This adds a new POST /1.0/instances/NAME/port-forward API endpoint which
upgrades the connection to a raw TCP connection to the specified address
and port inside of the instance.

For containers, the connection is established directly by the server from
within the container's network namespace. For virtual machines, the request
is forwarded to the incus-agent which then handles the connection, this
is controlled by the new port-forward agent feature.

A matching incus port-forward command is added to the client, providing
a local TCP listener which forwards every connection to the instance.

`unix_block_limits` Copy Chapter

This adds the limits.read and limits.write configuration keys to
unix-block devices. These behave similarly to their disk device
equivalents, accepting either a byte/s value or an IOPS value.

`authorization_client_routing` Copy Chapter

This allows loading multiple authorization drivers at once and routing each
request to one of them based on the authentication class of the client.

The following server configuration keys are added:

• authorization.client.default: driver for clients without a more specific class route
• authorization.client.unix: driver for local (unix socket) clients
• authorization.client.tls: driver for unrestricted TLS clients
• authorization.client.tls-restricted: driver for restricted (project-scoped) TLS clients
• authorization.client.oidc: driver for OIDC-authenticated clients

Each key accepts one of allow, deny, openfga or scriptlet. A per-class
key falls back to authorization.client.default when unset.

authorization.client.tls-restricted additionally accepts tls, as the TLS
authorization method exists to enforce the per-certificate project restrictions
that only apply to restricted certificates.

The following server configuration key is also added:

• authorization.openfga.tls.identifier: certificate attribute (fingerprint
  or name) used as the OpenFGA user when a TLS client is authorized by
  OpenFGA (defaults to name).

`instance_nvram` Copy Chapter

This adds new endpoints to manage virtual machines’ UEFI variables:

• GET /1.0/instances/{name}/nvram, to get all UEFI variables
• GET /1.0/instances/{name}/nvram/{guid}, to get UEFI variables under the given GUID
• GET /1.0/instances/{name}/nvram/{guid}/{var}, to get specific UEFI variables
• DELETE /1.0/instances/{name}/nvram/{guid}/{var}, to delete specific UEFI variables
• PUT /1.0/instances/{name}/nvram/{guid}/{var}, to set/update specific UEFI variables

It also adds rebuild-nvram as an instance debug action.

`disk_io_limits_combined` Copy Chapter

This makes it possible to set both a byte/s limit and an IOPS limit at the
same time in the I/O limit keys of disk and unix-block devices by using
a comma separated list of values (for example, limits.read=30MiB,1000iops).

`resources_cpu_cluster` Copy Chapter

Adds a cluster field to CPU core entries in the resources API, telling
apart cores with identical identifiers on systems where core identifiers
are only unique within a CPU cluster (ARM big.LITTLE).

`acme_eab` Copy Chapter

Adds support for ACME External Account Binding (EAB) through two new
server configuration keys:

• acme.eab.kid
• acme.eab.hmac

`instance_protection_start` Copy Chapter

Adds a new security.protection.start instance configuration key which
when set to true prevents the instance from being started.

`instance_nvram_bulk_update` Copy Chapter

This adds a new PATCH /1.0/instances/{name}/nvram endpoint to update NVRAM in bulk.

`instance_refresh_migration` Copy Chapter

Adds support for refresh migration, where a running container is moved to another
cluster member through a series of incremental transfers, only being stopped for
the last one.

This adds a refresh field to POST /1.0/instances/{name}. When set to true on
a stateless migration (migration set to true and live set to false), a
running container is migrated this way.

It also adds refresh-migrate as a configuration option to cluster.evacuate
and as an evacuation mode, which forces refresh migration of instances during
cluster evacuation.

`image_locations` Copy Chapter

Adds a new locations field to the image API, listing the names of the
cluster members which have a local copy of the image.

`network_ovn_multicast` Copy Chapter

Adds multicast configuration to OVN networks through two new
configuration keys:

• bridge.multicast_snooping
• bridge.multicast_relay

`device_burst_limits` Copy Chapter

This adds burst support to the I/O limits of disk and nic devices.

For disk devices (virtual machines only), the new limits.read.burst,
limits.write.burst and limits.max.burst keys take the same syntax as their
sustained counterparts and define the rate the device may reach while bursting.

The new limits.read.burst.length, limits.write.burst.length and
limits.max.burst.length keys define how long the burst rate may be
sustained for, defaulting to one second.

For nic devices (bridged, p2p, routed and ovn), the new
limits.ingress.bucket, limits.egress.bucket and limits.max.bucket keys
define the amount of data, in bit, that may be sent in excess of the sustained
limit.

The bridged, p2p and routed types also take limits.ingress.burst,
limits.egress.burst and limits.max.burst, defining the bit/s rate at which
that bucket may be spent. A direction needs both keys to burst.

The ovn type only takes the bucket keys, as OVN has no separate burst rate.

`network_ipv6_ra` Copy Chapter

Adds a new ipv6.ra configuration key to both bridge and ovn networks,
controlling whether IPv6 router advertisements are sent on the network.

`qemu_scriptlet_nvram` Copy Chapter

This extends the QEMU scriptlet feature by adding functions to explore and edit
the NVRAM:

• get_nvram_var
• has_nvram_var
• set_nvram_var
• unset_nvram_var
• get_raw_nvram_var
• set_raw_nvram_var
• list_nvram_vars

`storage_ceph_rbd_backend` Copy Chapter

Adds a new ceph.rbd.backend configuration key to the Ceph RBD storage driver.

When set to librbd, volumes are accessed through librbd rather than the
RBD kernel driver, with qemu-nbd being used whenever a block device is
needed on the host.

`instance_nvram_config` Copy Chapter

This adds three categories of initial configuration options, initial.nvram.*,
initial.nvram-binary.* and initial.secureboot, which allow defining default
NVRAM variables and Secure Boot certificates and keys when rebuilding the NVRAM.

`storage_cephobject_endpoint_cert` Copy Chapter

Adds a new cephobject.radosgw.endpoint_cert configuration key to the Ceph
Object storage driver, holding the TLS certificate (PEM bundle) to trust when
communicating with the radosgw endpoint.

This replaces cephobject.radosgw.endpoint_cert_file, existing values are
converted automatically.

`device_queue_disc` Copy Chapter

Adds new queue.discipline and queue.discipline.attach configuration keys
to nic devices of type bridged, p2p and routed, controlling the queuing
discipline used on the host side of the NIC and where it gets attached.

`network_bridge_dns_include_hosts` Copy Chapter

Adds a new dns.include_hosts configuration key to bridge networks,
controlling whether the network's dnsmasq serves records from the host's
/etc/hosts file. Setting it to false avoids exposing host-only entries
to instances while keeping AppArmor confinement.

`gpu_physical_clique` Copy Chapter

Adds a new nvidia.clique configuration key to gpu devices of type physical
when used with virtual machines. It sets the NVIDIA GPUDirect P2P clique ID
(0 to 15) advertised to the guest driver, allowing peer-to-peer DMA between
GPUs passed through to the same virtual machine when they share a clique ID.

`authorization_scriptlet_claims` Copy Chapter

This adds a Claims field to the details argument of the authorization
scriptlet, holding the validated OIDC token claims of the client as a
dictionary. This allows writing rules based on claims such as groups.

`instance_project_move_live` Copy Chapter

This allows a running instance to change project as part of a live migration
to another cluster member. It also lets instances with dependent disks
attached change project.

`metrics_cluster_members` Copy Chapter

This adds cluster member metrics to the /1.0/metrics endpoint:

• incus_cluster_member with the member's architecture and failure domain
• incus_cluster_member_status with one sample per possible status
• incus_cluster_member_role with one sample per role held
• incus_cluster_member_group with one sample per group the member belongs to

`instance_security_tags` Copy Chapter

Adds a new security.tags configuration key to instances, holding a
comma-separated list of tags. The tags are exposed to the authorization
backend, with OpenFGA maintaining a security_tag object for each tag
along with a tag relation to every instance carrying it.

`network_ovn_parent` Copy Chapter

Adds a new parent configuration key to ovn networks, referencing another
ovn network in the same project.

`disk_initial_copy` Copy Chapter

This adds the initial.copy property to disk devices backed by a custom storage volume.

When set to true on a container, the content already present at the device path inside
the container is copied into the volume the first time the volume is used, if it's empty.

`internal_debug_pprof` Copy Chapter

This adds /internal/debug/pprof/ to the API, serving the same pprof
profiles as the core.debug_address listener but over the regular
listeners, restricted to trusted clients with server administration rights.

`instance_state_disk_counters` Copy Chapter

Adds a counters field to each entry of the disk section of an instance's
state, holding bytes_read, bytes_written, reads_completed and
writes_completed.

These are only populated for virtual machines, using the block I/O
statistics reported by QEMU. Containers do not report them because cgroup
I/O accounting is keyed by host block device rather than by Incus device
name, and a container's root file system and custom volumes routinely share
a single host block device, making per-device attribution impossible.

`instances_state_tpm` Copy Chapter

Adds a tpm section to the instance state, listing every tpm device along
with its endorsement keys, each with a type, size, SHA-256 fingerprint and
PEM encoded public key.

The endorsement keys are only known for TPM devices provisioned by Incus, that is
when instances.tpm.platform_cert was set at the time the device was first started.

`ovn_nic_security_filtering` Copy Chapter

This adds the security.mac_filtering, security.ipv4_filtering and security.ipv6_filtering options to OVN NICs.

They are implemented through OVN port security on the instance's logical switch port,
restricting the source MAC address and, when IP filtering is enabled, the IP addresses
(including any static routes) the instance may use.

`authorization_scriptlet_request` Copy Chapter

This adds the following fields to the details argument of the authorization scriptlet:

• URL, holding the raw client request URL (?<query>)
• Path, holding the API path requested by the client
• Query, holding the dictionary of query parameters (note: the values are lists of strings and not single strings)
• Method, holding the HTTP method used in the request

`vm_boot_priority_skip` Copy Chapter

This adds the ability to define boot.priority to skip on NIC and disk devices, to skip them from the boot sequence. The new default for NICs is to be skipped.
This additionally adds a boot.pxe VM configuration key, defaulting to false, to enable PXE on all the NICs included in the boot sequence. This doesn’t affect HTTP boot, which is always enabled for those NICs.

`network_ovn_nat_address_l2proxy` Copy Chapter

Allows ipv4.nat.address and ipv6.nat.address on an ovn network whose uplink uses
ovn.ingress_mode=l2proxy, previously only supported with ovn.ingress_mode=routed.

`oidc_id_token` Copy Chapter

This adds support for OIDC providers that issue opaque (non-JWT) access tokens.

When the access token isn't a JWT, Incus verifies the ID token instead, checking that it was issued by the configured issuer for the configured client ID.
Clients send the ID token alongside the opaque access token through the X-Incus-OIDC-idtoken header.

`certificate_ca_trust` Copy Chapter

A CA certificate added to the trust store now also trusts the client certificates it issued.
Those clients are identified by the CA entry's fingerprint and inherit its type, restrictions and projects.

`network_bgp_communities` Copy Chapter

This adds a bgp.communities network configuration key for bridge and ovn networks.
It takes a comma-separated list of standard BGP communities (ASN:value)
which are attached to all prefixes advertised by the network.

Doc Api (`doc/api.md`) Copy Section

REST API

:maxdepth: 1

Main API documentation <rest-api>
rest-api-spec
Main API extensions <api-extensions>
Instance API documentation <dev-incus>
Events API documentation <events>
Metrics API documentation <reference/provided_metrics>

Doc Architectures (`doc/architectures.md`) Copy Section

(architectures)=

Architectures

Incus can run on just about any architecture that is supported by the Linux kernel and by Go.

Some entities in Incus are tied to an architecture, for example, the instances, instance snapshots and images.

The following table lists all supported architectures including their unique identifier and the name used to refer to them.
The architecture names are typically aligned with the Linux kernel architecture names.

Incus cares only about the kernel architecture, not the particular userspace flavor as determined by the toolchain.

That means that Incus considers ARMv7 hard-float to be the same as ARMv7 soft-float and refers to both as `armv7l`.
If useful to the user, the exact userspace ABI may be set as an image and container property, allowing easy query.

Virtual-machine support Copy Chapter

Incus only supports running virtual-machines on the following host architectures:

• x86_64
• aarch64
• ppc64le
• s390x
• loongarch64

The virtual machine guest architecture can usually be the 32bit personality of the host architecture,
so long as the virtual machine firmware is capable of booting it.

Doc Authentication (`doc/authentication.md`) Copy Section

(authentication)=

Remote API authentication

Remote communications with the Incus daemon happen using JSON over HTTPS.

To be able to access the remote API, clients must authenticate with the Incus server.
The following authentication methods are supported:

• {ref}authentication-tls-certs
• {ref}authentication-openid

(authentication-tls-certs)=

TLS client certificates Copy Chapter

When using {abbr}TLS (Transport Layer Security) client certificates for authentication, both the client and the server will generate a key pair the first time they're launched.
The server will use that key pair for all HTTPS connections to the Incus socket.
The client will use its certificate as a client certificate for any client-server communication.

To cause certificates to be regenerated, simply remove the old ones.
On the next connection, a new certificate is generated.

Communication protocol Copy Section

The supported protocol must be TLS 1.3 or better.

It's possible to force Incus to accept TLS 1.2 by setting the INCUS_INSECURE_TLS environment variable on both client and server.
However this isn't a supported setup and should only ever be used when forced to use an outdated corporate proxy.

All communications must use perfect forward secrecy, and ciphers must be limited to strong elliptic curve ones (such as ECDHE-RSA or ECDHE-ECDSA).

Any generated key should be at least 4096 bit RSA, preferably 384 bit ECDSA.
When using signatures, only SHA-2 signatures should be trusted.

Since we control both client and server, there is no reason to support
any backward compatibility to broken protocol or ciphers.

(authentication-trusted-clients)=

Trusted TLS clients Copy Section

You can obtain the list of TLS certificates trusted by an Incus server with incus config trust list.

Trusted clients can be added in either of the following ways:

• {ref}authentication-add-certs
• {ref}authentication-token

The workflow to authenticate with the server is similar to that of SSH, where an initial connection to an unknown server triggers a prompt:

1. When the user adds a server with incus remote add, the server is contacted over HTTPS, its certificate is downloaded and the fingerprint is shown to the user.

2. The user is asked to confirm that this is indeed the server's fingerprint, which they can manually check by connecting to the server or by asking someone with access to the server to run the info command and compare the fingerprints.

3. The server attempts to authenticate the client:

   • If the client certificate is in the server's trust store, the connection is granted.
   • If the client certificate is not in the server's trust store, the server prompts the user for a token.
     If the provided token matches, the client certificate is added to the server's trust store and the connection is granted.
     Otherwise, the connection is rejected.

It is possible to restrict a TLS client's access to Incus via {ref}authorization-tls.
To revoke trust to a client, remove its certificate from the server with incus config trust remove <fingerprint>.

(authentication-tls-jwt)=

Using `JSON Web Token` (`JWT`) to perform TLS authentication

As an alternative to directly using the client's TLS certificate for
authentication, Incus also supports the user derive a bearer token and
use it through the HTTP Authorization header.

To do this, the user must generate a signed JWT which has its
Subject field set to the full fingerprint of their client certificate,
it must have valid NotBefore and NotAfter fields and be signed by
the client certificate's private key.

(authentication-add-certs)=

Adding trusted certificates to the server

The preferred way to add trusted clients is to directly add their certificates to the trust store on the server.
To do so, copy the client certificate to the server and register it using incus config trust add-certificate <file>.

(authentication-ca-certs)=

Trusting a certificate authority

Instead of adding every client certificate, you can add a CA certificate to the trust store.
Any client certificate issued by that CA is then trusted as if the CA entry itself had been presented: the client is identified by the fingerprint of the CA entry and inherits its type, restrictions and projects.
The client must present any intermediate certificate between its own certificate and the trusted CA, and its certificate must allow the clientAuth extended key usage if it specifies any.

This makes it possible to use short-lived client certificates issued by an internal PKI without updating the trust store on every renewal.
Revocation lists aren't consulted for such certificates, so remove the CA entry from the trust store to revoke all the certificates it issued.
Certificates that are directly present in the trust store always take precedence over a matching CA entry.

(authentication-token)=

Adding client certificates using tokens

You can also add new clients by using tokens. Tokens expire after a configurable time ({config:option}server-core:core.remote_token_expiry) or once they've been used.

To use this method, generate a token for each client by calling incus config trust add, which will prompt for the client name.
The clients can then add their certificates to the server's trust store by providing the generated token when prompted.

If your Incus server is behind NAT, you must specify its external public address when adding it as a remote for a client:

    incus remote add <name> <IP_address>

When generating the token on the server, Incus includes a list of IP addresses that the client can use to access the server.
However, if the server is behind NAT, these addresses might be local addresses that the client cannot connect to.
In this case, you must specify the external address manually.

Alternatively, the clients can provide the token directly when adding the remote: incus remote add <name> <token>.

Using a PKI system Copy Section

In a {abbr}PKI (Public key infrastructure) setup, a system administrator manages a central PKI that issues client certificates for all the Incus clients and server certificates for all the Incus daemons.

To enable PKI mode, complete the following steps:

1. Add the {abbr}CA (Certificate authority) certificate to all machines:

   • Place the client.ca file in the clients' configuration directories (~/.config/incus).
   • Place the server.ca file in the server's configuration directory (/var/lib/incus).

2. Place the certificates issued by the CA on the clients and the server, replacing the automatically generated ones.

3. Restart the server.

In that mode, any connection to an Incus daemon will be done using the
pre-seeded CA certificate.

If the server certificate isn't signed by the CA, the connection will simply go through the normal authentication mechanism.
If the server certificate is valid and signed by the CA, then the connection continues without prompting the user for the certificate.

Note that the generated certificates are not automatically trusted. You must still add them to the server in one of the ways described in {ref}authentication-trusted-clients.
In that mode, only client certificates issued by the CA can be added to the trust store.

Alternatively, you can set {config:option}server-core:core.trust_ca_certificates to true to have the server automatically trust any valid client certificate issued by the CA.
Such certificates don't need to be added to the trust store and get full access to the server, unless they are added to the trust store as restricted certificates.
Certificates listed in a ca.crl file placed in the server's configuration directory are rejected.

Only enable `core.trust_ca_certificates` with a CA dedicated to Incus, as anyone holding a certificate issued by that CA can then administer the server.

Encrypting local keys Copy Section

The incus client also supports encrypted client keys. Keys generated via the methods above can be encrypted with a password, using:

ssh-keygen -p -o -f .config/incus/client.key

Unless you enable [`keepalive` mode](remote-keepalive), then every single call to Incus will cause the prompt which may get a bit annoying:

    $ incus list remote-host:
    Password for client.key:
    ┌──────┬───────┬──────┬──────┬──────┬───────────┐
    │ NAME │ STATE │ IPV4 │ IPV6 │ TYPE │ SNAPSHOTS │
    └──────┴───────┴──────┴──────┴──────┴───────────┘

While the `incus` command line supports encrypted keys, tools such as [Ansible's connection plugin](https://docs.ansible.com/ansible/latest/collections/community/general/incus_connection.html) do not.

(authentication-openid)=

OpenID Connect authentication Copy Chapter

Incus supports using OpenID Connect to authenticate users through an {abbr}OIDC (OpenID Connect) Identity Provider.

Authentication through OpenID Connect is supported, but there is no user role handling in place so far.
Any user that authenticates through the configured OIDC Identity Provider gets full access to Incus.

To configure Incus to use OIDC authentication, set the oidc.* server configuration options.
Your OIDC provider must be configured to enable the Device Authorization Grant type.

To add a remote pointing to an Incus server configured with OIDC authentication, run incus remote add <remote_name> <remote_address>.
You are then prompted to authenticate through your web browser, where you must confirm the device code that Incus uses.
The Incus client then retrieves and stores the access and refresh tokens and provides those to Incus for all interactions.

Incus verifies JWT access tokens offline using the provider's signing keys.
Providers that issue opaque access tokens are also supported, in which case Incus verifies the ID token instead and checks that it was issued for the configured client ID.
The client ID used by Incus must therefore not be shared with any other application registered with the provider.
Setting oidc.audience disables this fallback as the audience can't be verified on an opaque access token.

Incus supports a custom OIDC claim of incus.allowed_subnets (list of strings), if the claim is set,
the user will only be allowed if connecting from an IP address that's part of one of the CIDR subnets listed in the claim.

Any user that authenticates through the configured OIDC Identity Provider gets full access to Incus.
To restrict user access, you must also configure {ref}`authorization`.
Currently, the only authorization method that is compatible with OIDC is {ref}`authorization-openfga`.

(authentication-server-certificate)=

TLS server certificate Copy Chapter

Incus supports issuing server certificates using {abbr}ACME (Automatic Certificate Management Environment) services, for example, Let's Encrypt.

To enable this feature, set the relevant server configuration options.

Incus supports both HTTP-01 and DNS-01 challenges. The set of configuration option varies between the two.

For DNS-01, the relevant {config:option}server-acme:acme.provider and {config:option}server-acme:acme.provider.environment
values can be found directly in the documentation of lego,
the ACME client that Incus uses behind the scenes.

For HTTP-01, Incus will cause lego to temporarily listen on port 80 so the HTTP challenge can go through.
In a cluster, the challenge runs on the leader while the other members temporarily listen on that same port and forward the challenge requests to the leader.
This allows for a certificate covering the names of all cluster members.
If your Incus server sits behind a reverse proxy, you'll need that reverse proxy to redirect HTTP traffic to HTTPS.

For ACME services that require External Account Binding (EAB), set both
{config:option}server-acme:acme.eab.kid and {config:option}server-acme:acme.eab.hmac
to the values provided by your ACME service.

Failure scenarios Copy Chapter

In the following scenarios, authentication is expected to fail.

Server certificate changed Copy Section

The server certificate might change in the following cases:

• The server was fully reinstalled and therefore got a new certificate.
• The connection is being intercepted ({abbr}MITM (Machine in the middle)).

In such cases, the client will refuse to connect to the server because the certificate fingerprint does not match the fingerprint in the configuration for this remote.

It is then up to the user to contact the server administrator to check if the certificate did in fact change.
If it did, the certificate can be replaced by the new one, or the remote can be removed altogether and re-added.

Server trust relationship revoked Copy Section

The server trust relationship is revoked for a client if another trusted client or the local server administrator removes the trust entry for the client on the server.

In this case, the server still uses the same certificate, but all API calls return a 403 code with an error indicating that the client isn't trusted.

Doc Authorization (`doc/authorization.md`) Copy Section

(authorization)=

Authorization

When interacting with Incus over the Unix socket, members of the incus-admin group will have full access to the Incus API.
Those who are only members of the incus group will instead be restricted to a single project tied to their user.

When interacting with Incus over the network (see {ref}server-expose for instructions), it is possible to further authenticate and restrict user access.
There are three supported authorization methods:

• {ref}authorization-tls
• {ref}authorization-openfga
• {ref}authorization-scriptlet

By default, the method used for a request is determined automatically from the
client's authentication protocol. This can be customized so that different kinds
of clients are handled by different methods; see {ref}authorization-client-routing.

(authorization-tls)=

TLS authorization Copy Chapter

Incus natively supports restricting {ref}authentication-trusted-clients to one or more projects.
When a client certificate is restricted, the client will also be prevented from performing global configuration changes or altering the configuration (limits, restrictions) of the projects it's allowed access to.

To restrict access, use incus config trust edit <fingerprint>.
Set the restricted key to true and specify a list of projects to restrict the client to.
If the list of projects is empty, the client will not be allowed access to any of them.

By default, this authorization method is used for clients authenticating with a restricted TLS certificate.
Clients using an unrestricted certificate are granted full access instead.
Both can be changed, see {ref}authorization-client-routing.

(authorization-openfga)=

Open Fine-Grained Authorization (OpenFGA) Copy Chapter

Incus supports integrating with {abbr}OpenFGA (Open Fine-Grained Authorization).
This authorization method is highly granular.
For example, it can be used to restrict user access to a single instance.

To use OpenFGA for authorization, you must configure and run an OpenFGA server yourself.
Incus will connect to the OpenFGA server, write the {ref}openfga-model, and query this server for authorization for all subsequent requests.

To enable this authorization method in Incus, set the authorization.openfga.* server configuration options.
All options must be set in order to enable OpenFGA. Though, you do not have to create the authorization-model yourself, Incus will generate it including the initial tuple to allow only authenticated users: server:incus#authenticated@user:*.

Setting the `authorization.openfga.*` options only makes OpenFGA available.
It does not on its own cause any request to be authorized by it.

OpenFGA is only consulted for clients whose class is routed to it with an
`authorization.client.*` option, see {ref}`authorization-client-routing`.

(openfga-model)=

OpenFGA model Copy Section

With OpenFGA, access to a particular API resource is determined by the user's relationship to it.
These relationships are determined by an OpenFGA authorization model.
The Incus OpenFGA authorization model describes API resources in terms of their relationship to other resources, and a relationship a user or group might have with that resource.

The full Incus OpenFGA authorization model is defined in internal/server/auth/driver_openfga_model.openfga:

:language: none

(openfga-security-tags)=

Security tags Copy Section

Instances can be tagged through the security.tags configuration key, which holds a comma-separated list of tags.
Incus does not use those tags in its own authorization decisions.
Instead, every tag is exposed in OpenFGA as a security_tag object related to the server and holding a tag relation with every instance carrying it (for example, security_tag:pci tag instance:default/web1).
This makes the tags visible to external tooling working directly with the OpenFGA store.

Users that you do not trust with root access to the host should not be granted the following relations:

- `server -> admin`
- `server -> operator`
- `server -> can_edit`
- `server -> can_create_storage_pools`
- `server -> can_create_projects`
- `server -> can_create_certificates`
- `certificate -> can_edit`
- `storage_pool -> can_edit`
- `project -> admin`

The remaining relations may be granted.
However, you must apply appropriate {ref}`project-restrictions`.

(authorization-scriptlet)=

Scriptlet authorization Copy Chapter

Incus supports defining a scriptlet to manage fine-grained authorization, allowing to write precise authorization rules with no dependency on external tools.

Setting `authorization.scriptlet` only makes the scriptlet available.
It does not on its own cause any request to be authorized by it.

The scriptlet is only run for clients whose class is routed to it with an
`authorization.client.*` option, see {ref}`authorization-client-routing`.

To use scriptlet authorization, you can write a scriptlet in the authorization.scriptlet server configuration option implementing a function authorize, which takes three arguments:

• details, an object with the following attributes:
  • Username: the user name or certificate fingerprint
  • Protocol: the authentication protocol
  • IsAllProjectsRequest: whether the request is made on all projects
  • ProjectName: the project name
  • Chain: the certificate chain as a list of dissected x509 certificates
  • Certificate: the certificate data stored in the database
  • Claims: the validated OIDC token claims as a dictionary (empty for non-OIDC clients)
  • URL: the raw client request URL (?<query>)
  • Path: the API path requested by the client
  • Query: the dictionary of query parameters (note: the values are lists of strings and not single strings)
  • Method: the HTTP method used in the request
• object, the object on which the user requests authorization
• entitlement, the authorization level asked by the user

This function must return a Boolean indicating whether the user has access or not to the given object with the given entitlement.

Additionally, two optional functions can be defined so that users can be listed through the access API:

• get_instance_access, with two arguments (project_name and instance_name), returning a list of users able to access a given instance
• get_project_access, with one argument (project_name), returning a list of users able to access a given project

(authorization-client-routing)=

Client routing Copy Chapter

More than one authorization method can be loaded at the same time, with each
request routed to exactly one of them based on the authentication class of the
client. This is controlled through the authorization.client.* server
configuration options, one per client class:

• authorization.client.unix: local clients connecting over the Unix socket
• authorization.client.tls: clients using an unrestricted client certificate
• authorization.client.tls-restricted: clients using a restricted (project-scoped) client certificate
• authorization.client.oidc: OIDC-authenticated clients
• authorization.client.default: any client class not set above

Each option accepts one of the following values:

• allow: unconditionally grant access
• deny: unconditionally refuse access
• tls: use {ref}authorization-tls, only valid for authorization.client.tls-restricted
• openfga: use {ref}authorization-openfga
• scriptlet: use {ref}authorization-scriptlet

A per-class option falls back to authorization.client.default when unset.
When that is unset too, the following built-in routing applies:

This routing is fixed.

The per-certificate project restrictions described in {ref}`authorization-tls`
are enforced by the TLS authorization method only.

If `authorization.client.tls-restricted` is set to any value other than `tls`,
the project list of the client certificate are no longer consulted, and a restricted
certificate may gain access well beyond the projects it is limited to.
Only set this option to something other than `tls` if the method you route to
reproduces the restrictions you rely on.

Always set `authorization.client.default` last, once you have confirmed that
every explicitly configured client class behaves as expected.

The default applies to every class that is not set explicitly, so a single
mistake affects all of them at once. If it is set to `openfga` or `scriptlet`
while that method is misconfigured or unreachable, or to `deny`, every remote
client is refused.

The `root` user is always allowed to interact with the API over the Unix
socket, regardless of the configured authorization method. This is what allows
a server locked out by a bad routing configuration to be recovered.

For example, to have OpenFGA authorize OIDC clients while restricted TLS clients
keep using TLS authorization:

incus config set authorization.client.oidc=openfga
incus config set authorization.client.tls-restricted=scriptlet

Doc Backup (`doc/backup.md`) Copy Section

(backups)=

How to back up an Incus server

In a production setup, you should always back up the contents of your Incus server.

The Incus server contains a variety of different entities, and when choosing
your backup strategy, you must decide which of these entities you want to
back up and how frequently you want to save them.

What to back up Copy Chapter

The various contents of your Incus server are located on your file system and, in
addition, recorded in the {ref}Incus database <database>.
Therefore, only backing up the database or only backing up the files on disk
does not give you a full functional backup.

Your Incus server contains the following entities:

• Instances (database records and file systems)
• Images (database records, image files, and file systems)
• Networks (database records and state files)
• Profiles (database records)
• Storage volumes (database records and file systems)

Consider which of these you need to back up.
For example, if you don't use custom images, you don't need to back up your
images since they are available on the image server.
If you use only the default profile, or only the standard incusbr0 network bridge,
you might not need to worry about backing them up, because they can easily be re-created.

Full backup Copy Chapter

To create a full backup of all contents of your Incus server,
back up the /var/lib/incus directory.

This directory contains your local storage, the Incus database, and your configuration.
It does not contain separate storage devices, however.
That means that whether the directory also contains the data of your
instances depends on the storage drivers that you use.

If your Incus server uses any external storage
(for example, LVM volume groups, ZFS zpools, or any other resource that isn't
directly self-contained to Incus), you must back this up separately.

To back up your data, create a tarball of /var/lib/incus.
If your system uses /etc/subuid and /etc/subgid file, you
should also back up these files.
Restoring them avoids needless shifting of instance file systems.

To restore your data, complete the following steps:

1. Stop Incus on your server (for example, with
   sudo systemctl stop incus.service incus.socket).
2. Delete the directory (/var/lib/incus/).
3. Restore the directory from the backup.
4. Delete and restore any external storage devices.
5. Restore the /etc/subuid and /etc/subgid files if present.
6. Restart Incus (for example, with
   sudo systemctl start incus.socket incus.service or by restarting your machine).

Partial backup Copy Chapter

If you decide to only back up specific entities, you
have different options for how to do this.
You should consider doing some of these partial backups even
if you are doing full backups in addition.
It can be easier and safer to, for example, restore a single instance
or reconfigure a profile than to restore the full Incus server.

Back up instances and volumes Copy Section

Instances and storage volumes are backed up in a very similar
way (because when backing up an instance, you basically back up
its instance volume, see {ref}storage-volume-types).

See {ref}instances-backup and {ref}howto-storage-backup-volume for detailed information.
The following sections give a brief summary of the options you have
for backing up instances and volumes.

Secondary backup Incus server

Incus supports copying and moving instances and storage volumes between two hosts.
See {ref}move-instances and {ref}howto-storage-move-volume for instructions.

So if you have a spare server, you can regularly copy your
instances and storage volumes to that secondary server to back them up.
If needed, you can either switch over to the secondary server or
copy your instances or storage volumes back from it.

If you use the secondary server as a pure storage server, it doesn't
need to be as powerful as your main Incus server.

Export tarballs

You can use the export command to export instances and volumes to a backup tarball.
By default, those tarballs include all snapshots.

You can use an optimized export option, which is usually quicker
and results in a smaller size of the tarball.
However, you must then use the same storage driver when restoring the backup tarball.

See {ref}instances-backup-export and {ref}storage-backup-export for instructions.

Snapshots

Snapshots save the state of an instance or volume at a specific point in time.
However, they are stored in the same storage pool and are therefore likely
to be lost if the original data is deleted or lost.
This means that while snapshots are very quick and easy to create
and restore, they don't constitute a secure backup.

See {ref}instances-snapshots and {ref}storage-backup-snapshots for more information.

(backup-database)=

Back up the database Copy Section

While there is no trivial method to restore the contents of the
{ref}Incus database <database>, it can still be very convenient
to keep a backup of its content.
Such a backup can make it much easier to re-create, for example,
networks or profiles if the need arises.

Use the following command to dump the content of the local database to a file:

incus admin sql local .dump > <output_file>

Use the following command to dump the content of the global database to a file:

incus admin sql global .dump > <output_file>

You should include these two commands in your regular Incus backup.

Doc Client Config (`doc/client-config.md`) Copy Section

(client-config)=

CLI configuration file

The Incus CLI client uses a configuration file to store environment data and modify its behavior. This documentation describes the file structure and its options.

Changing this configuration file manually may leave the CLI in an inconsistent state, so you should only edit it via CLI commands such as [`incus remote`](incus_remote.md) and [`incus alias`](incus_alias.md).

The configuration file Copy Chapter

By default, the Incus CLI will use the $HOME/.config/incus/config.yml, but it can also use the config.yml in an arbitrary directory provided by the INCUS_CONF environment variable.

If the file or its path don't exist, it will use the default:

default-remote: local
remotes:
  images:
    protocol: simplestreams
    public: true
    addr: https://images.linuxcontainers.org
  local:
    protocol: incus
    public: false
    addr: unix://
aliases: {}
defaults:
  list_format: ""
  console_type: ""
  console_spice_command: ""
  no_color: false

`remotes` Copy Chapter

Incus remote information managed by incus remote commands is stored in the remotes section. Remotes may have the following configuration keys.

`addr` Copy Section

Address of the remote server. Accepts IPs, FQDNs and URLs. Multiple addresses can be provided, separated with commas addr1,addr2,....

`last_working_address` Copy Section

Used by the CLI to register the last working address of a remote, if the remote has multiple addresses.

`auth_type` Copy Section

Method that the client will use to authenticate to the server API. Accepts tls and oidc.

`keepalive` Copy Section

Timeout in seconds that the keepalive feature will maintain the connection opened.

`project` Copy Section

Default project used by the CLI when using that remote. It's defined by the incus project switch command. It's overwritten by the --project flag and the INCUS_PROJECT environment variable.

`protocol` Copy Section

The protocol the CLI will use to communicate with the remote. This defines if the remote will be used as an Incus or image server. Accepts the following options:

• incus: private Incus server accessible over the network
• oci: Open Container Initiative (OCI) server that provides application container images
• public: Incus server accessible over the network to provide images only
• simplestream: Simple Stream server that provide images

`credentials_helper` Copy Section

Defines a helper command that will handle OCI service authentication.

`public` Copy Section

Defines if the remote is an image-only server (public).

`default_remotes` Copy Chapter

Defines the default Incus remote used by the CLI to execute commands. It's defined by the incus remote switch command. This configuration is overwritten by the environment variable INCUS_REMOTE or by the <remote>: prefix.

`aliases` Copy Chapter

Stores the alias managed by the incus alias command. Each alias is a key/value pair that consists of the alias name and the alias command (enclosed by quotes).

`defaults` Copy Chapter

Determines the default CLI behavior for certain CLI commands.

`list_format` Copy Section

Defines how list outputs will be displayed. Equivalent to the --format (-f) flag. Accepts csv, json, table, yaml, compact and markdown formats.

`console_type` Copy Section

Defines the default console type used by the incus console command. Available options are:

• console: text based console
• vga: graphic UI console

`console_spice_command` Copy Section

Defines an alternative SPICE command to provide the VGA console used by incus console.

`no_color` Copy Section

Disables CLI colors.

Doc Client (`doc/client.md`) Copy Section

(incus-client)=

Incus client

:maxdepth: 1

Add remote servers <remotes>
Add command aliases <howto/incus_alias>
CLI configuration file <client-config>
/reference/manpages

Doc Cloud Init (`doc/cloud-init.md`) Copy Section

relatedlinks: https://cloudinit.readthedocs.org/ Copy Chapter

(cloud-init)=

How to use `cloud-init`

cloud-init is a tool for automatically initializing and customizing an instance of a Linux distribution.

By adding cloud-init configuration to your instance, you can instruct cloud-init to execute specific actions at the first start of an instance.
Possible actions include, for example:

• Updating and installing packages
• Applying certain configurations
• Adding users
• Enabling services
• Running commands or scripts
• Automatically growing the file system of a VM to the size of the disk

See the {ref}cloud-init:index for detailed information.

The `cloud-init` actions are run only once on the first start of the instance.
Rebooting the instance does not re-trigger the actions.

`cloud-init` support in images Copy Chapter

To use cloud-init, you must base your instance on an image that has cloud-init installed.

Images from the images remote have cloud-init-enabled variants, which are usually bigger in size than the default variant.
The cloud variants use the /cloud suffix, for example, images:debian/12/cloud.

`cloud-init` and virtual machines Copy Chapter

For cloud-init to work inside of a virtual machine, you need to either have a functional incus-agent in the VM or need to provide the cloud-init data through a special extra disk.
All images coming from the images: remote will have the agent already setup and so are good to go from the start.

For instances that do not have the incus-agent, you can pass in the extra cloud-init disk with:

incus config device add INSTANCE cloud-init disk source=cloud-init:config

When relying on the `incus-agent` to apply the `cloud-init` configuration, the instance needs to have a set of file templates in place for:

* `/var/lib/cloud/seed/nocloud-net/meta-data`
* `/var/lib/cloud/seed/nocloud-net/network-config`
* `/var/lib/cloud/seed/nocloud-net/user-data`
* `/var/lib/cloud/seed/nocloud-net/vendor-data`

When making a new image, it's easiest to replicate those from one of our own `cloud-init` enabled images.

Those file templates get applied by the `incus-agent` on first boot. The
agent then triggers a reboot, causing `cloud-init` to run cleanly and
apply the configuration to the virtual machine.

Configuration options Copy Chapter

Incus supports two different sets of configuration options for configuring cloud-init: cloud-init.* and user.*.
Which of these sets you must use depends on the cloud-init support in the image that you use.
As a rule of thumb, newer images support the cloud-init.* configuration options, while older images support user.*.
However, there might be exceptions to that rule.

The following configuration options are supported:

• cloud-init.vendor-data or user.vendor-data (see {ref}cloud-init:vendor-data)
• cloud-init.user-data or user.user-data (see {ref}cloud-init:user_data_formats)
• cloud-init.network-config or user.network-config (see {ref}cloud-init:network_config)

For more information about the configuration options, see the cloud-init instance options.

Vendor data and user data Copy Section