# Technical Documentation: lxc/incus > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/lxc/incus) ### Overview (`README.md`) # Incus Incus is a modern, secure and powerful system container and virtual machine manager. It provides a unified experience for running and managing full Linux systems inside containers or virtual machines. Incus supports images for a large number of Linux distributions (official Ubuntu images and images provided by the community) and is built around a very powerful, yet pretty simple, REST API. Incus scales from one instance on a single machine to a cluster in a full data center rack, making it suitable for running workloads both for development and in production. Incus allows you to easily set up a system that feels like a small private cloud. You can run any type of workload in an efficient way while keeping your resources optimized. You should consider using Incus if you want to containerize different environments or run virtual machines, or in general run and manage your infrastructure in a cost-effective way. You can try Incus online at: [`https://linuxcontainers.org/incus/try-it/`](https://linuxcontainers.org/incus/try-it/) ## Project history Incus, which is named after the [Cumulonimbus incus](https://en.wikipedia.org/wiki/Cumulonimbus_incus) or anvil cloud started as a community fork of Canonical's LXD following [Canonical's takeover](https://linuxcontainers.org/lxd/) of the LXD project from the Linux Containers community. The project was then adopted by the Linux Containers community, taking back the spot left empty by LXD's departure. Incus is a true open source community project, free of any [CLA](https://en.wikipedia.org/wiki/Contributor_License_Agreement) and remains released under the [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0). It's maintained by the same team of developers that first created LXD. ## Get started See [Getting started](https://linuxcontainers.org/incus/docs/main/tutorial/first_steps/) in the Incus documentation for installation instructions and first steps. - Release announcements: [`https://discuss.linuxcontainers.org/c/news/`](https://discuss.linuxcontainers.org/c/news/) - Release tarballs: [`https://github.com/lxc/incus/releases/`](https://github.com/lxc/incus/releases/) - Documentation: [`https://linuxcontainers.org/incus/docs/main/`](https://linuxcontainers.org/incus/docs/main/) ## Status Type | Service | Status --- | --- | --- Tests | GitHub | [](https://github.com/lxc/incus/actions?query=event%3Apush+branch%3Amain) Go documentation | Godoc | [](https://godoc.org/github.com/lxc/incus/v7/client) Static analysis | GoReport | [](https://goreportcard.com/report/github.com/lxc/incus) Translations | Weblate | [](https://hosted.weblate.org/projects/incus/) ## Security Consider the following aspects to ensure that your Incus installation is secure: - Keep your operating system up-to-date and install all available security patches. - Use only supported Incus versions. - Restrict access to the Incus daemon and the remote API. - Do not use privileged containers unless required. If you use privileged containers, put appropriate security measures in place. See the [LXC security page](https://linuxcontainers.org/lxc/security/) for more information. - Configure your network interfaces to be secure. See [Security](https://github.com/lxc/incus/blob/main/doc/explanation/security.md) for detailed information. **IMPORTANT:** Local access to Incus through the Unix socket always grants full access to Incus. This includes the ability to attach file system paths or devices to any instance as well as tweak the security features on any instance. Therefore, you should only give such access to users who you'd trust with root access to your system. ## Support and community The following channels are available for you to interact with the Incus community. ### Bug reports You can file bug reports and feature requests at: [`https://github.com/lxc/incus/issues/new`](https://github.com/lxc/incus/issues/new) ### Community support Community support is handled at: [`https://discuss.linuxcontainers.org`](https://discuss.linuxcontainers.org) ### Commercial support Commercial support is currently available from [Zabbly](https://zabbly.com) for users of their [Debian or Ubuntu packages](https://github.com/zabbly/incus). ## Documentation The official documentation is available at: [`https://github.com/lxc/incus/tree/main/doc`](https://github.com/lxc/incus/tree/main/doc) ## Contributing Fixes and new features are greatly appreciated. Make sure to read our [contributing guidelines](CONTRIBUTING.md) first! --- ### Overview (`doc/README.md`) # Incus documentation The Incus documentation is available at: GitHub provides a basic rendering of the documentation as well, but important features like includes and clickable links are missing. Therefore, we recommend reading the [published documentation](https://linuxcontainers.org/incus/docs/main/). ## Documentation framework Incus' documentation is built with [Sphinx](https://www.sphinx-doc.org/en/master/index.html). It is written in [Markdown](https://commonmark.org/) with [MyST](https://myst-parser.readthedocs.io/) extensions. For syntax help and guidelines, see the [documentation cheat sheet](https://linuxcontainers.org/incus/docs/main/doc-cheat-sheet/) ([source](https://raw.githubusercontent.com/lxc/incus/main/doc/doc-cheat-sheet.md)). For structuring, the documentation uses the [Diátaxis](https://diataxis.fr/) approach. ## Build the documentation To build the documentation, run `make doc` from the root directory of the repository. This command installs the required tools and renders the output to the `doc/html/` directory. To update the documentation for changed files only (without re-installing the tools), run `make doc-incremental`. Before opening a pull request, make sure that the documentation builds without any warnings (warnings are treated as errors). To preview the documentation locally, run `make doc-serve` and go to [`http://localhost:8001`](http://localhost:8001) to view the rendered documentation. --- ### Overview (`test/README.md`) # How to run To run all tests, including the Go tests, run from repository root: sudo -E make check To run only the integration tests, run from the test directory: sudo -E ./main.sh # Environment variables Name | Default | Description :-- | :--- | :---------- `INCUS_BACKEND` | dir | What backend to test against (btrfs, ceph, dir, lvm, zfs, or random) `INCUS_CEPH_CLUSTER` | ceph | The name of the ceph cluster to create osd pools in `INCUS_CEPH_CEPHFS` | "" | Enables the CephFS tests using the specified cephfs filesystem for `cephfs` pools `INCUS_CEPH_CEPHOBJECT_RADOSGW` | "" | Enables the Ceph Object tests using the specified radosgw HTTP endpoint for `cephobject` pools `INCUS_CONCURRENT` | 0 | Run concurrency tests, very CPU intensive `INCUS_DEBUG` | 0 | Run incusd, incus and the shell in debug mode (very verbose) `INCUS_INSPECT` | 0 | Don't teardown the test environment on failure `INCUS_LOGS ` | "" | Path to a directory to copy all the Incus logs to `INCUS_OFFLINE` | 0 | Skip anything that requires network access `INCUS_SKIP_TESTS` | "" | Space-delimited list of test names to skip `INCUS_TEST_IMAGE` | "" (busybox test image) | Path to an image tarball to use instead of the default busybox image `INCUS_TMPFS` | 0 | Sets up a tmpfs for the whole testsuite to run on (fast but needs memory) `INCUS_TRUENAS_DATASET` | "" | The remote TrueNAS dataset that will contain all the `truenas` pools created whilst running tests and is required, and must exist, when using the `truenas` backend. `INCUS_TRUENAS_API_KEY` | "" | If set, will be applied as `truenas.api_key` on `truenas` pools. `INCUS_TRUENAS_CONFIG` | "" | If set, will be applied as `truenas.config` on `truenas` pools. `INCUS_TRUENAS_HOST` | "" | If set, will be applied as `truenas.host` on `truenas` pools. `INCUS_TRUENAS_ALLOW_INSECURE` | "" | If set, will be applied as `truenas.allow_insecure` on `truenas` pools. `INCUS_NIC_SRIOV_PARENT` | "" | Enables SR-IOV NIC tests using the specified parent device `INCUS_IB_PHYSICAL_PARENT` | "" | Enables Infiniband physical tests using the specified parent device `INCUS_IB_SRIOV_PARENT` | "" | Enables Infiniband SR-IOV tests using the specified parent device `INCUS_NIC_BRIDGED_DRIVER` | "" | Specifies bridged NIC driver for tests (either native or openvswitch, defaults to native) `INCUS_REQUIRED_TESTS` | "" | Space-delimited list of test names that must not be skipped if their prerequisites are not met `INCUS_VERBOSE` | 0 | Run incusd, incus and the shell in verbose mode --- ### Overview (`cmd/generate-config/README.md`) # generate-config A small CLI to parse comments in a Golang codebase meant to be used for a documentation tool (like Sphinx for example). It parses the comments from the AST and extracts their documentation. ## Disclaimer `generate-config` is intended for internal use within the [Incus](https://github.com/lxc/incus) code base. There are no guarantees regarding backwards compatibility, API stability, or long-term availability. It may change or be removed at any time without prior notice. Use at your own discretion. ## Usage ```shell $ generate-config -h Usage of generate-config: -e value Path that will be excluded from the process ``` ## Formatting A comment is formatted this way: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` The go-swagger spec from source generator can only handles `swagger:meta` (global file/package level documentation), `swagger:route` (API endpoints), `swagger:params` (function parameters), `swagger:operation` (method documentation), `swagger:response` (API response content documentation), `swagger:model` (struct documentation) generation. In our use case, we would want a config variable spec generator that can bundle any key-value data pairs alongside metadata to build a sense of hierarchy and identity (we want to associate a unique key to each gendoc comment group that will also be displayed in the generated documentation) In a swagger fashion, `generate-config` can associate metadata key-value pairs (here for example, `group` and `key`) to data key-value pairs. As a result, it can generate a YAML tree out of the code documentation and also a Markdown document. ### Output Here is the JSON output of the example shown above: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Here is the `.txt` output of the example shown above: ```plain \`\`\`{config:option} user.* cluster :type: string :liveupdate: `yes` :shortdesc: Free form user key/value storage (can be used in search). :condition: container :default: - This is the real long desc. With two paragraphs. And a list: - Item - Item - Item example of a table: Key | Type | Scope | Default | Description :-- | :--- | :---- | :------ | :---------- `acme.agree_tos` | bool | global | `false` | Agree to ACME terms of service `acme.ca_url` | string | global | `https://acme-v02.api.letsencrypt.org/directory` | URL to the directory resource of the ACME service `acme.domain` | string | global | - | Domain for which the certificate is issued `acme.email` | string | global | - | Email address used for the account registration \`\`\` \`\`\`{config:option} scheduler.instance cluster :liveupdate: `yes` :shortdesc: Possible values are all, manual and group. See Automatic placement of instances for more information. :condition: container :default: `all` :type: integer \`\`\` ``` --- ### Overview (`cmd/generate-database/README.md`) # `generate-database` ## Introduction `generate-database` is a database statement and associated `go` function generator for Incus and related projects. `generate-database` utilizes `go`'s code generation directives (`//go:generate ...`) alongside go's [ast](https://pkg.go.dev/go/ast) and [types](https://pkg.go.dev/go/types) packages for parsing the syntax tree for go structs and variables. We use `generate-database` for the majority of our SQL statements and database interactions on the `go` side for consistency and predictability. ## Disclaimer `generate-database` is intended for internal use within the [Incus](https://github.com/lxc/incus) code base. There are no guarantees regarding backwards compatibility, API stability, or long-term availability. It may change or be removed at any time without prior notice. Use at your own discretion. ## Usage ### Initialization #### Package global Once per package, that uses `generate-database` for generation of database statements and associated `go` functions, `generate-database` needs to be invoked using the following `go:generate` instruction: ```go //go:generate generate-database db mapper generate ``` This will initiate a call to `generate-database db mapper generate`, which will then search for `//generate-database:mapper` directives in the same file and process those. The following flags are available: * `--package` / `-p`: Package import paths to search for structs to parse. Defaults to the caller package. Can be used more than once. #### File Generally the first thing we will want to do for any newly generated file is to ensure the file has been cleared of content: ```go //generate-database:mapper target instances.mapper.go //generate-database:mapper reset -i -b "//go:build linux && cgo && !agent" ``` ### Generation Directive Arguments The generation directive arguments have the following form: `//generate-database:mapper flags ` The following flags are available: * `--build` / `-b`: build comment to include (commands: `reset`) * `--interface` / `-i`: create interface files (commands: `reset`, `method`) * `--entity` / `-e`: database entity to generate the method or statement for (commands: `stmt`, `method`) Example: * `//generate-database:mapper stmt -e instance objects table=table_name` The `table` key can be used to override the generated table name for a specified one. * `//generate-database:mapper method -i -e instance Create references=Config,Device` For some tables (defined below under [Additional Information](#Additional-Information) as [EntityTable](#EntityTable), the `references=` key can be provided with the name of a [ReferenceTable](#ReferenceTable) or [MapTable](#MapTable) struct. This directive would produce `CreateInstance` in addition to `CreateInstanceConfig` and `CreateInstanceDevices`: * `//generate-database:mapper method -i -e instance_profile Create struct=Instance` * `//generate-database:mapper method -i -e instance_profile Create struct=Profile` For some tables (defined below under [Additional Information](#Additional-Information) as [AssociationTable](#AssociationTable), `method` declarations must include a `struct=` to indicate the directionality of the function. An invocation can be called for each direction. This would produce `CreateInstanceProfiles` and `CreateProfileInstances` respectively. ### SQL Statement Generation SQL generation supports the following SQL statement types: Type | Description :--- | :---- `objects` | Creates a basic SELECT statement of the form `SELECT FROM ORDER BY `. `objects-by--and-...` | Parses a pre-existing SELECT statement variable declaration of the form produced by`objects`, and appends a `WHERE` clause with the given fields located in the associated struct. Specifically looks for a variable declaration of the form `var Objects = RegisterStmt("SQL String")` `names` | Creates a basic SELECT statement of the form `SELECT FROM
ORDER BY `. `names-by--and-...` | Parses a pre-existing SELECT statement variable declaration of the form produced by`names`, and appends a `WHERE` clause with the given fields located in the associated struct. Specifically looks for a variable declaration of the form `var Objects = RegisterStmt("SQL String")` `create` | Creates a basic INSERT statement of the form `INSERT INTO
VALUES`. `create-or-replace` | Creates a basic INSERT statement of the form `INSERT OR REPLACE INTO
VALUES`. `delete-by--and-...` | Creates a DELETE statement of the form `DELETE FROM
WHERE ` where the constraint is based on the given fields of the associated struct. `id` | Creates a basic SELECT statement that returns just the internal ID of the table. `rename` | Creates an UPDATE statement that updates the primary key of a table: `UPDATE
SET WHERE `. `update` | Creates an UPDATE statement of the form `UPDATE
SET WHERE `. #### Examples ```go //generate-database:mapper stmt -e instance objects //generate-database:mapper stmt -e instance objects-by-Name-and-Project //generate-database:mapper stmt -e instance create //generate-database:mapper stmt -e instance update //generate-database:mapper stmt -e instance delete-by-Name-and-Project ``` #### Statement Related Go Tags There are several tags that can be added to fields of a struct that will be parsed by the `ast` package. Tag | Description :-- | :---- `sql=
.` | Supply an explicit table and column name to use for this struct field. `coalesce=` | Generates a SQL coalesce function with the given value `coalesce(, value)`. `order=yes` | Override the default `ORDER BY` columns with all fields specifying this tag. `join=` | Applies a `JOIN` of the form `JOIN ON
. = `. `leftjoin=` | Applies a `LEFT JOIN` of the same form as a `JOIN`. `joinon=
.` | Overrides the default `JOIN ON` clause with the given table and column, replacing `
.` above. `jointo=` | Overrides the default target column `id` with the given column, replacing the `id` in `` above. This is intended for "loose" foreign keys, not using the ID column. Therefore, this is intended to be used in conjunction with `joinon` and `omit=create,update` to get the expected behavior. `joinas=` | Sets an alias for the joined table name, in case it clashes with another table. `primary=yes` | Assigns column associated with the field to be sufficient for returning a row from the table. Will default to `Name` if unspecified. Fields with this key will be included in the default 'ORDER BY' clause. `omit=` | Omits a given field from consideration for the comma separated list of statement types (`create`, `objects-by-Name`, `update`). `ignore` | Outright ignore the struct field as though it does not exist. `ignore` needs to be the only tag value in order to be recognized. `marshal=` | Marshal/Unmarshal data into the field. The column must be a TEXT column. If `marshal=yes`, then the type must implement both `Marshal` and `Unmarshal`. If `marshal=json`, the type is marshaled to JSON using the standard library ([json.Marshal](https://pkg.go.dev/encoding/json#Marshal)). This works for entity tables only, and not for association or mapping tables. `create_timestamp` | Automatically set the value of this column to the current time (UTC) when the respective record is created, namely in `Create` and `CreateOrReplace` (regardless if the record is actually created or updated). `update_timestamp` | Automatically set the value of this column to the current time (UTC) for every operation altering the record, namely `Create`, `CreateOrReplace`, `Rename` and `Update`. `nullable=yes` | Modify filters for this field to handle null types like `sql.Null*` by changing `field = ?` to `(field = ? OR ( field IS NULL AND ? IS NULL ))`. ### Go Function Generation Go function generation supports the following types: Type | Description :--- | :---- `GetNames` | Return a slice of primary keys for all rows in a table matching the filter. Cannot be used with composite keys. `GetMany` | Return a slice of structs for all rows in a table matching the filter. `GetOne` | Return a single struct corresponding to a row with the given primary keys. Depends on `GetMany`. `ID` | Return the ID column from the table corresponding to the given primary keys. `Exists` | Returns whether there is an row in the table with the given primary keys. Depends on `ID.` `Create` | Insert a row from the given struct into the table if not already present. Depends on `Exists` `CreateOrReplace` | Insert a row from the given struct into the table, regardless of if an entry already exists. `Rename` | Update the primary key for a table row. `Update` | Update the columns at a given row, specified by primary key. `DeleteOne` | Delete exactly one row from the table. `DeleteMany` | Delete one or more rows from the table. ```go //generate-database:mapper method -i -e instance GetMany //generate-database:mapper method -i -e instance GetOne //generate-database:mapper method -i -e instance ID //generate-database:mapper method -i -e instance Exist //generate-database:mapper method -i -e instance Create //generate-database:mapper method -i -e instance Update //generate-database:mapper method -i -e instance DeleteOne-by-Project-and-Name //generate-database:mapper method -i -e instance DeleteMany-by-Name ``` ### Additional Information All structs should have an `ID` field, as well as an additional `Filter` struct prefixed with the original struct name. This should include any fields that should be considered for filtering in `WHERE` clauses. These fields should be pointers to facilitate omission and inclusion without setting default values. Example: ```go type Instance struct { ID int Name string Project string } type InstanceFilter struct { Name *string Project *string } ``` `generate-database` will handle parsing of structs differently based on the composition of the struct in four different ways. Non-`EntityType` structs will only support `GetMany`, `Create`, `Update`, and `Delete` functions. ### EntityTable Most structs will get treated this way, and represent a normal table. * If a table has an associated table for which a `ReferenceTable` or `MapTable` as defined below is applicable, functions specific to this entity can be generated by including a comma separated list to `references=` in the code generation directive for `GetMany`, `Create`, or `Update` directives. * The `Create` method directive for `EntityTable` will expect on the `ID` and `Exist` method directives to be present. * All `CREATE`, `UPDATE`, and `DELETE` statements that include a joined table will expect a `var ID = RegisterStmt('SQL String')` to exist for the joining table. ### ReferenceTable A struct that contains a field named `ReferenceID` will be parsed this way. `generate-database` will use this struct to generate more abstract SQL statements and functions of the form `_`. The associated `Filter` struct may include a `ReferenceID []int` field. This generates an `IN` clause matching on the parent column, with the integer values inlined into the query to avoid query parameter count limits. A nil slice leaves the filter unset while an empty (non-nil) slice matches nothing. When the field is present, the generated per-parent helpers and nested reference fetches are automatically scoped to the relevant parent IDs rather than fetching the whole table. Real world invocation of these statements and functions should be done through an `EntityTable` `method` call with the tag `references=`. This `EntityTable` will replace the `` above. Example: ```go //generate-database:mapper stmt -e device create //generate-database:mapper method -e device Create type Device struct { ID int ReferenceID int Name string Type string } //... //generate-database:mapper method -e instance Create references=Device // This will produce a function called `CreateInstanceDevices`. ``` ### MapTable This is a special type of `ReferenceTable` with fields named `Key` and `Value`. On the SQL side, this is treated exactly like a `ReferenceTable`, but on the `go` side, the return values will be a map. Example: ```go //generate-database:mapper stmt -e config create //generate-database:mapper method -e config Create type Config struct { ID int ReferenceID int Key string Value string } //... //generate-database:mapper method -e instance Create references=Config // This will produce a function called `CreateInstanceConfig`, which will return a `map[string]string`. ``` ### AssociationTable This is a special type of table that contains two fields of the form `ID`, where `` corresponds to two other structs present in the same package. This will generate code for compound tables of the form `_` that are generally used to associate two tables together by their IDs. `method` generation declarations for these statements should include a `struct=` to indicate the directionality of the function. An invocation can be called for each direction. Example: ```go //generate-database:mapper method -i -e instance_profile Create struct=Instance //generate-database:mapper method -i -e instance_profile Create struct=Profile type InstanceProfile struct { InstanceID int ProfileID int } ``` --- ### Overview (`test/mini-oidc/README.md`) # mini-oidc `mini-oidc` is an extremely basic OIDC provider which can be used with the `incus` command line. It doesn't use web authentication and instead just automatically approves any authentication request. Usage: ```shell mini-oidc [] ``` By default, it will authenticate everyone as `unknown`, but this can be overridden by writing the username to be returned in the file named in the 2nd (optional) argument. This effectively allows scripting a variety of users without having to deal with actual login. The `storage` sub-package is a based on https://github.com/zitadel/oidc/tree/main/example/server/storage. The following changes were made: - Added IncusDeviceClient - Added option to configure access token and refresh token expiry --- ### Overview (`shared/resources/usbid/README.md`) This Go package is a variant of `usbid` from https://github.com/google/gousb. It was written by the gousb maintainers and contributors and then adapted for our use. Main changes: - Doesn't load on import (reduced memory footprint). - Uses system-local USB database (to always match `lsusb`). - Doesn't import anything outside of built-in Go packages. - Doesn't use or indirectly rely on CGO as this code is often used in cross-built static binaries. Most users will want to stick to the upstream google/gousb version instead. This fork is really meant for our special use case and may be further trimmed down in the future. --- ### Doc Index (`doc/index.md`) # Incus Incus is a modern, secure and powerful system container and virtual machine manager. % Include content from [../README.md](../README.md) ```{include} ../README.md :start-after: :end-before: ``` ## Security % Include content from [../README.md](../README.md) ```{include} ../README.md :start-after: :end-before: ``` See [Security](security.md) for detailed information. ````{important} % Include content from [../README.md](../README.md) ```{include} ../README.md :start-after: :end-before: ``` ```` ## Project and community Incus is free software and developed under the [Apache 2 license](https://www.apache.org/licenses/LICENSE-2.0). It’s an open source project that warmly welcomes community projects, contributions, suggestions, fixes and constructive feedback. - [Code of Conduct](https://github.com/lxc/incus/blob/main/CODE_OF_CONDUCT.md) - [Contribute to the project](contributing.md) - [Release announcements](https://discuss.linuxcontainers.org/c/news/13) - [Release tarballs](https://github.com/lxc/incus/releases/) - [Get support](support.md) - [Watch tutorials and announcements on YouTube](https://www.youtube.com/@TheZabbly) - [Ask and answer questions on the forum](https://discuss.linuxcontainers.org) ```{toctree} :hidden: :titlesonly: self Getting Started General Client Server Instances Storage Networks Images Projects Clustering API Security Internals Contributing External resources ``` --- ### Doc .Wordlist (`doc/.wordlist.txt`) AAAA AAVMF ABI ACL ACLs AI AIO allocator AMD Ansible Ansible's API APIs AppArmor ARMv ARP ASN AXFR backend backends backported Backports balancer balancers benchmarking BFD BGP bibi BitLocker BMC bool bootable BPF Btrfs bugfix bugfixes Centos Ceph CephFS Ceph's CFS cgroup cgroupfs cgroups checksum checksums Chocolatey CIDR CLI Colima COPR Cowsql CPUs CRIU CRL cron CSV CUDA customizable dataset DCO dereferenced devtmpfs DHCP DHCPv Diátaxis Diffie diskful diskless Diskless Distrobuilder DMA DNAT DNS dnsmasq DNSSEC DoS DRBD DRM EB Ebit eBPF ECDHE EAB ECDSA ECMP EDK EiB Eibit endian EPEL ES ESA ETag failover formatters FQDNs FreeBSD frontend Furo gapped GARM GARP GbE Gbit Geneve GiB Gibit GID GIDs Github Golang goroutines GPG GPUDirect GPUs Grafana GUID HAProxy hardcoded HDDs Hellman HMAC Homebrew hostname hotplug hotplugged hotplugging HTTPS hwdata ICMP idmap idmapped idmaps Incant incrementing Incus Incus' InfiniBand InfluxDB init initramfs integrations IOMMU IOPS IOV IPAM IPs IPv IPVLAN iSCSI JIT jq JSON JWT kB kbit KiB kibi Kibit Kubernetes KVM LINBIT LINSTOR LINSTOR's LLM LLMs lookups Loongarch LRU LTS LV LVM LXC LXCFS LXC's LXD LXD's macOS macvlan Makefile manpages Mbit mDNS MiB Mibit MicroCeph MicroCloud misconfigured MCS MII MITM MTU Mullvad multicast multiport MyST namespace namespaced namespaces NATed natively NDP NetBSD netmask NFS NIC NIC's NICs NixOS NTP NUMA NVMe NVRAM OCI OData OIDC OpenFGA OpenID OpenMetrics OpenSSL openSUSE OpenSUSE OpenTofu OSD overcommit overcommitting overlayfs OVMF OVN OVS PackageHub passthrough Pbit PEM PCI PCIe PDU peerings Permalink PFs PiB Pibit PID PKI PNG Podman Pongo POSIX PPA pre preselects preseed proxied Proxmox proxying PTS PXE qdisc QEMU qgroup qgroups QMP RADOS RBAC RBD RDNSS README reconfiguring requestor resolvers RESTful RHEL rootfs RSA rST RTC runtime SATA scalable scriptlet SDN SeaBIOS Seccomp SELinux SEV SFTP SHA shiftfs SIGHUP SIGTERM simplestreams SLAAC SMTP Snapcraft snapshotted SNAT SOA Solaris SPAs SPL SquashFS SSDs SSL Starlark stateful stderr stdin stdout STP struct structs subcommand subcommands subitem subnet subnets subpage substep subtree subtrees subvolume subvolumes superset SVG symlink symlinks syscall syscalls sysfs syslog systemd Tbit TCP Telegraf Terraform TiB Tibit TLS tmpfs toolchain topologies TPM TrueNAS TSIG TTL UDP UEFI UFW UI UID UIDs uncomment unconfigured unevictable unixgram unmanaged unmount unmounting uplink uptime URI URIs userspace UUID vCPU vCPUs VE VDPA VFs VFS VirtIO virtualize virtualized VLAN VLANs VM VMs VPD VPN VPS VRF vSwitch VXLAN webhook WebSocket WebSockets Winget XFS XHR YAML YAML's Zabbly Zettabyte ZFS zpool zpools --- ### Doc Api Extensions (`doc/api-extensions.md`) # API extensions The changes below were introduced to the Incus API after the 1.0 API was finalized. They are all backward compatible and can be detected by client tools by looking at the `api_extensions` field in `GET /1.0`. ## `storage_zfs_remove_snapshots` A `storage.zfs_remove_snapshots` daemon configuration key was introduced. It's a Boolean that defaults to `false` and that when set to `true` instructs Incus to remove any needed snapshot when attempting to restore another. This is needed as ZFS will only let you restore the latest snapshot. ## `container_host_shutdown_timeout` A `boot.host_shutdown_timeout` container configuration key was introduced. It's an integer which indicates how long Incus should wait for the container to stop before killing it. Its value is only used on clean Incus daemon shutdown. It defaults to 30s. ## `container_stop_priority` A `boot.stop.priority` container configuration key was introduced. It's an integer which indicates the priority of a container during shutdown. Containers will shutdown starting with the highest priority level. Containers with the same priority will shutdown in parallel. It defaults to 0. ## `container_syscall_filtering` A number of new syscalls related container configuration keys were introduced. * `security.syscalls.blacklist_default` * `security.syscalls.blacklist_compat` * `security.syscalls.blacklist` * `security.syscalls.whitelist` See [Instance configuration](instance-config) for how to use them. ## `auth_pki` This indicates support for PKI authentication mode. In this mode, the client and server both must use certificates issued by the same PKI. See [Security](security.md) for details. ## `container_last_used_at` A `last_used_at` field was added to the `GET /1.0/containers/` endpoint. It is a timestamp of the last time the container was started. If a container has been created but not started yet, `last_used_at` field will be `1970-01-01T00:00:00Z` ## `etag` Add support for the ETag header on all relevant endpoints. This adds the following HTTP header on answers to GET: * ETag (SHA-256 of user modifiable content) And adds support for the following HTTP header on PUT requests: * If-Match (ETag value retrieved through previous GET) This makes it possible to GET an Incus object, modify it and PUT it without risking to hit a race condition where Incus or another client modified the object in the meantime. ## `patch` Add support for the HTTP PATCH method. PATCH allows for partial update of an object in place of PUT. ## `usb_devices` Add support for USB hotplug. ## `https_allowed_credentials` To use Incus API with all Web Browsers (via SPAs) you must send credentials (certificate) with each XHR (in order for this to happen, you should set [`withCredentials=true`](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/withCredentials) flag to each XHR Request). Some browsers like Firefox and Safari can't accept server response without `Access-Control-Allow-Credentials: true` header. To ensure that the server will return a response with that header, set `core.https_allowed_credentials=true`. ## `image_compression_algorithm` This adds support for a `compression_algorithm` property when creating an image (`POST /1.0/images`). Setting this property overrides the server default value (`images.compression_algorithm`). ## `directory_manipulation` This allows for creating and listing directories via the Incus API, and exports the file type via the X-Incus-type header, which can be either `file` or `directory` right now. ## `container_cpu_time` This adds support for retrieving CPU time for a running container. ## `storage_zfs_use_refquota` Introduces a new server property `storage.zfs_use_refquota` which instructs Incus to set the `refquota` property instead of `quota` when setting a size limit on a container. Incus will also then use `usedbydataset` in place of `used` when being queried about disk utilization. This effectively controls whether disk usage by snapshots should be considered as part of the container's disk space usage. ## `storage_lvm_mount_options` Adds a new `storage.lvm_mount_options` daemon configuration option which defaults to `discard` and allows the user to set addition mount options for the file system used by the LVM LV. ## `network` Network management API for Incus. This includes: * Addition of the `managed` property on `/1.0/networks` entries * All the network configuration options (see [Network configuration](networks.md) for details) * `POST /1.0/networks` (see [RESTful API](rest-api.md) for details) * `PUT /1.0/networks/` (see [RESTful API](rest-api.md) for details) * `PATCH /1.0/networks/` (see [RESTful API](rest-api.md) for details) * `DELETE /1.0/networks/` (see [RESTful API](rest-api.md) for details) * `ipv4.address` property on `nic` type devices (when `nictype` is `bridged`) * `ipv6.address` property on `nic` type devices (when `nictype` is `bridged`) * `security.mac_filtering` property on `nic` type devices (when `nictype` is `bridged`) ## `profile_usedby` Adds a new `used_by` field to profile entries listing the containers that are using it. ## `container_push` When a container is created in push mode, the client serves as a proxy between the source and target server. This is useful in cases where the target server is behind a NAT or firewall and cannot directly communicate with the source server and operate in pull mode. ## `container_exec_recording` Introduces a new Boolean `record-output`, parameter to `/1.0/containers//exec` which when set to `true` and combined with with `wait-for-websocket` set to `false`, will record stdout and stderr to disk and make them available through the logs interface. The URL to the recorded output is included in the operation metadata once the command is done running. That output will expire similarly to other log files, typically after 48 hours. ## `certificate_update` Adds the following to the REST API: * ETag header on GET of a certificate * PUT of certificate entries * PATCH of certificate entries ## `container_exec_signal_handling` Adds support `/1.0/containers//exec` for forwarding signals sent to the client to the processes executing in the container. Currently SIGTERM and SIGHUP are forwarded. Further signals that can be forwarded might be added later. ## `gpu_devices` Enables adding GPUs to a container. ## `container_image_properties` Introduces a new `image` configuration key space. Read-only, includes the properties of the parent image. ## `migration_progress` Transfer progress is now exported as part of the operation, on both sending and receiving ends. This shows up as a `fs_progress` attribute in the operation metadata. ## `id_map` Enables setting the `security.idmap.isolated` and `security.idmap.isolated`, `security.idmap.size`, and `raw.id_map` fields. ## `network_firewall_filtering` Add two new keys, `ipv4.firewall` and `ipv6.firewall` which if set to `false` will turn off the generation of `iptables` FORWARDING rules. NAT rules will still be added so long as the matching `ipv4.nat` or `ipv6.nat` key is set to `true`. Rules necessary for `dnsmasq` to work (DHCP/DNS) will always be applied if `dnsmasq` is enabled on the bridge. ## `network_routes` Introduces `ipv4.routes` and `ipv6.routes` which allow routing additional subnets to an Incus bridge. ## `storage` Storage management API for Incus. This includes: * `GET /1.0/storage-pools` * `POST /1.0/storage-pools` (see [RESTful API](rest-api.md) for details) * `GET /1.0/storage-pools/` (see [RESTful API](rest-api.md) for details) * `POST /1.0/storage-pools/` (see [RESTful API](rest-api.md) for details) * `PUT /1.0/storage-pools/` (see [RESTful API](rest-api.md) for details) * `PATCH /1.0/storage-pools/` (see [RESTful API](rest-api.md) for details) * `DELETE /1.0/storage-pools/` (see [RESTful API](rest-api.md) for details) * `GET /1.0/storage-pools//volumes` (see [RESTful API](rest-api.md) for details) * `GET /1.0/storage-pools//volumes/` (see [RESTful API](rest-api.md) for details) * `POST /1.0/storage-pools//volumes/` (see [RESTful API](rest-api.md) for details) * `GET /1.0/storage-pools//volumes//` (see [RESTful API](rest-api.md) for details) * `POST /1.0/storage-pools//volumes//` (see [RESTful API](rest-api.md) for details) * `PUT /1.0/storage-pools//volumes//` (see [RESTful API](rest-api.md) for details) * `PATCH /1.0/storage-pools//volumes//` (see [RESTful API](rest-api.md) for details) * `DELETE /1.0/storage-pools//volumes//` (see [RESTful API](rest-api.md) for details) * All storage configuration options (see [Storage configuration](storage.md) for details) ## `file_delete` Implements `DELETE` in `/1.0/containers//files` ## `file_append` Implements the `X-Incus-write` header which can be one of `overwrite` or `append`. ## `network_dhcp_expiry` Introduces `ipv4.dhcp.expiry` and `ipv6.dhcp.expiry` allowing to set the DHCP lease expiry time. ## `storage_lvm_vg_rename` Introduces the ability to rename a volume group by setting `storage.lvm.vg_name`. ## `storage_lvm_thinpool_rename` Introduces the ability to rename a thin pool name by setting `storage.thinpool_name`. ## `network_vlan` This adds a new `vlan` property to `macvlan` network devices. When set, this will instruct Incus to attach to the specified VLAN. Incus will look for an existing interface for that VLAN on the host. If one can't be found it will create one itself and then use that as the macvlan parent. ## `image_create_aliases` Adds a new `aliases` field to `POST /1.0/images` allowing for aliases to be set at image creation/import time. ## `container_stateless_copy` This introduces a new `live` attribute in `POST /1.0/containers/`. Setting it to `false` tells Incus not to attempt running state transfer. ## `container_only_migration` Introduces a new Boolean `container_only` attribute. When set to `true` only the container will be copied or moved. ## `storage_zfs_clone_copy` Introduces a new Boolean `storage_zfs_clone_copy` property for ZFS storage pools. When set to `false` copying a container will be done through `zfs send` and receive. This will make the target container independent of its source container thus avoiding the need to keep dependent snapshots in the ZFS pool around. However, this also entails less efficient storage usage for the affected pool. The default value for this property is `true`, i.e. space-efficient snapshots will be used unless explicitly set to `false`. ## `unix_device_rename` Introduces the ability to rename the `unix-block`/`unix-char` device inside container by setting `path`, and the `source` attribute is added to specify the device on host. If `source` is set without a `path`, we should assume that `path` will be the same as `source`. If `path` is set without `source` and `major`/`minor` isn't set, we should assume that `source` will be the same as `path`. So at least one of them must be set. ## `storage_rsync_bwlimit` When `rsync` has to be invoked to transfer storage entities setting `rsync.bwlimit` places an upper limit on the amount of socket I/O allowed. ## `network_vxlan_interface` This introduces a new `tunnel.NAME.interface` option for networks. This key control what host network interface is used for a VXLAN tunnel. ## `storage_btrfs_mount_options` This introduces the `btrfs.mount_options` property for Btrfs storage pools. This key controls what mount options will be used for the Btrfs storage pool. ## `entity_description` This adds descriptions to entities like containers, snapshots, networks, storage pools and volumes. ## `image_force_refresh` This allows forcing a refresh for an existing image. ## `storage_lvm_lv_resizing` This introduces the ability to resize logical volumes by setting the `size` property in the containers root disk device. ## `id_map_base` This introduces a new `security.idmap.base` allowing the user to skip the map auto-selection process for isolated containers and specify what host UID/GID to use as the base. ## `file_symlinks` This adds support for transferring symlinks through the file API. X-Incus-type can now be `symlink` with the request content being the target path. ## `container_push_target` This adds the `target` field to `POST /1.0/containers/` which can be used to have the source Incus host connect to the target during migration. ## `network_vlan_physical` Allows use of `vlan` property with `physical` network devices. When set, this will instruct Incus to attach to the specified VLAN on the `parent` interface. Incus will look for an existing interface for that `parent` and VLAN on the host. If one can't be found it will create one itself. Then, Incus will directly attach this interface to the container. ## `storage_images_delete` This enabled the storage API to delete storage volumes for images from a specific storage pool. ## `container_edit_metadata` This adds support for editing a container `metadata.yaml` and related templates via API, by accessing URLs under `/1.0/containers//metadata`. It can be used to edit a container before publishing an image from it. ## `container_snapshot_stateful_migration` This enables migrating stateful container snapshots to new containers. ## `storage_driver_ceph` This adds a Ceph storage driver. ## `storage_ceph_user_name` This adds the ability to specify the Ceph user. ## `instance_types` This adds the `instance_type` field to the container creation request. Its value is expanded to Incus resource limits. ## `storage_volatile_initial_source` This records the actual source passed to Incus during storage pool creation. ## `storage_ceph_force_osd_reuse` This introduces the `ceph.osd.force_reuse` property for the Ceph storage driver. When set to `true` Incus will reuse an OSD storage pool that is already in use by another Incus instance. ## `storage_block_filesystem_btrfs` This adds support for Btrfs as a storage volume file system, in addition to `ext4` and `xfs`. ## `resources` This adds support for querying an Incus daemon for the system resources it has available. ## `kernel_limits` This adds support for setting process limits such as maximum number of open files for the container via `nofile`. The format is `limits.kernel.[limit name]`. ## `storage_api_volume_rename` This adds support for renaming custom storage volumes. ## `network_sriov` This adds support for SR-IOV enabled network devices. ## `console` This adds support to interact with the container console device and console log. ## `restrict_dev_incus` A new `security.guestapi` container configuration key was introduced. The key controls whether the `/dev/incus` interface is made available to the container. If set to `false`, this effectively prevents the container from interacting with the Incus daemon. ## `migration_pre_copy` This adds support for optimized memory transfer during live migration. ## `infiniband` This adds support to use InfiniBand network devices. ## `dev_incus_events` This adds a WebSocket API to the `/dev/incus` socket. When connecting to `/1.0/events` over the `/dev/incus` socket, you will now be getting a stream of events over WebSocket. ## `proxy` This adds a new `proxy` device type to containers, allowing forwarding of connections between the host and container. ## `network_dhcp_gateway` Introduces a new `ipv4.dhcp.gateway` network configuration key to set an alternate gateway. ## `file_get_symlink` This makes it possible to retrieve symlinks using the file API. ## `network_leases` Adds a new `/1.0/networks/NAME/leases` API endpoint to query the lease database on bridges which run an Incus-managed DHCP server. ## `unix_device_hotplug` This adds support for the `required` property for Unix devices. ## `storage_api_local_volume_handling` This add the ability to copy and move custom storage volumes locally in the same and between storage pools. ## `operation_description` Adds a `description` field to all operations. ## `clustering` Clustering API for Incus. This includes the following new endpoints (see [RESTful API](rest-api.md) for details): * `GET /1.0/cluster` * `UPDATE /1.0/cluster` * `GET /1.0/cluster/members` * `GET /1.0/cluster/members/` * `POST /1.0/cluster/members/` * `DELETE /1.0/cluster/members/` The following existing endpoints have been modified: * `POST /1.0/containers` accepts a new `target` query parameter * `POST /1.0/storage-pools` accepts a new `target` query parameter * `GET /1.0/storage-pool/` accepts a new `target` query parameter * `POST /1.0/storage-pool//volumes/` accepts a new `target` query parameter * `GET /1.0/storage-pool//volumes//` accepts a new `target` query parameter * `POST /1.0/storage-pool//volumes//` accepts a new `target` query parameter * `PUT /1.0/storage-pool//volumes//` accepts a new `target` query parameter * `PATCH /1.0/storage-pool//volumes//` accepts a new `target` query parameter * `DELETE /1.0/storage-pool//volumes//` accepts a new `target` query parameter * `POST /1.0/networks` accepts a new `target` query parameter * `GET /1.0/networks/` accepts a new `target` query parameter ## `event_lifecycle` This adds a new `lifecycle` message type to the events API. ## `storage_api_remote_volume_handling` This adds the ability to copy and move custom storage volumes between remote. ## `nvidia_runtime` Adds a `nvidia_runtime` configuration option for containers, setting this to `true` will have the NVIDIA runtime and CUDA libraries passed to the container. ## `container_mount_propagation` This adds a new `propagation` option to the disk device type, allowing the configuration of kernel mount propagation. ## `container_backup` Add container backup support. This includes the following new endpoints (see [RESTful API](rest-api.md) for details): * `GET /1.0/containers//backups` * `POST /1.0/containers//backups` * `GET /1.0/containers//backups/` * `POST /1.0/containers//backups/` * `DELETE /1.0/containers//backups/` * `GET /1.0/containers//backups//export` The following existing endpoint has been modified: * `POST /1.0/containers` accepts the new source type `backup` ## `dev_incus_images` Adds a `security.guestapi.images` configuration option for containers which controls the availability of a `/1.0/images/FINGERPRINT/export` API over `/dev/incus`. This can be used by a container running nested Incus to access raw images from the host. ## `container_local_cross_pool_handling` This enables copying or moving containers between storage pools on the same Incus instance. ## `proxy_unix` Add support for both Unix sockets and abstract Unix sockets in proxy devices. They can be used by specifying the address as `unix:/path/to/unix.sock` (normal socket) or `unix:@/tmp/unix.sock` (abstract socket). Supported connections are now: * `TCP <-> TCP` * `UNIX <-> UNIX` * `TCP <-> UNIX` * `UNIX <-> TCP` ## `proxy_udp` Add support for UDP in proxy devices. Supported connections are now: * `TCP <-> TCP` * `UNIX <-> UNIX` * `TCP <-> UNIX` * `UNIX <-> TCP` * `UDP <-> UDP` * `TCP <-> UDP` * `UNIX <-> UDP` ## `clustering_join` This makes `GET /1.0/cluster` return information about which storage pools and networks are required to be created by joining nodes and which node-specific configuration keys they are required to use when creating them. Likewise the `PUT /1.0/cluster` endpoint now accepts the same format to pass information about storage pools and networks to be automatically created before attempting to join a cluster. ## `proxy_tcp_udp_multi_port_handling` Adds support for forwarding traffic for multiple ports. Forwarding is allowed between a range of ports if the port range is equal for source and target (for example `1.2.3.4 0-1000 -> 5.6.7.8 1000-2000`) and between a range of source ports and a single target port (for example `1.2.3.4 0-1000 -> 5.6.7.8 1000`). ## `network_state` Adds support for retrieving a network's state. This adds the following new endpoint (see [RESTful API](rest-api.md) for details): * `GET /1.0/networks//state` ## `proxy_unix_dac_properties` This adds support for GID, UID, and mode properties for non-abstract Unix sockets. ## `container_protection_delete` Enables setting the `security.protection.delete` field which prevents containers from being deleted if set to `true`. Snapshots are not affected by this setting. ## `proxy_priv_drop` Adds `security.uid` and `security.gid` for the proxy devices, allowing privilege dropping and effectively changing the UID/GID used for connections to Unix sockets too. ## `pprof_http` This adds a new `core.debug_address` configuration option to start a debugging HTTP server. That server currently includes a `pprof` API and replaces the old `cpu-profile`, `memory-profile` and `print-goroutines` debug options. ## `proxy_haproxy_protocol` Adds a `proxy_protocol` key to the proxy device which controls the use of the HAProxy PROXY protocol header. ## `network_hwaddr` Adds a `bridge.hwaddr` key to control the MAC address of the bridge. ## `proxy_nat` This adds optimized UDP/TCP proxying. If the configuration allows, proxying will be done via `iptables` instead of proxy devices. ## `network_nat_order` This introduces the `ipv4.nat.order` and `ipv6.nat.order` configuration keys for Incus bridges. Those keys control whether to put the Incus rules before or after any pre-existing rules in the chain. ## `container_full` This introduces a new `recursion=2` mode for `GET /1.0/containers` which allows for the retrieval of all container structs, including the state, snapshots and backup structs. This effectively allows for [`incus list`](incus_list.md) to get all it needs in one query. ## `backup_compression` This introduces a new `backups.compression_algorithm` configuration key which allows configuration of backup compression. ## `nvidia_runtime_config` This introduces a few extra configuration keys when using `nvidia.runtime` and the `libnvidia-container` library. Those keys translate pretty much directly to the matching NVIDIA container environment variables: * `nvidia.driver.capabilities` => `NVIDIA_DRIVER_CAPABILITIES` * `nvidia.require.cuda` => `NVIDIA_REQUIRE_CUDA` * `nvidia.require.driver` => `NVIDIA_REQUIRE_DRIVER` ## `storage_api_volume_snapshots` Add support for storage volume snapshots. They work like container snapshots, only for volumes. This adds the following new endpoint (see [RESTful API](rest-api.md) for details): * `GET /1.0/storage-pools//volumes///snapshots` * `POST /1.0/storage-pools//volumes///snapshots` * `GET /1.0/storage-pools//volumes///snapshots/` * `PUT /1.0/storage-pools//volumes///snapshots/` * `POST /1.0/storage-pools//volumes///snapshots/` * `DELETE /1.0/storage-pools//volumes///snapshots/` ## `storage_unmapped` Introduces a new `security.unmapped` Boolean on storage volumes. Setting it to `true` will flush the current map on the volume and prevent any further idmap tracking and remapping on the volume. This can be used to share data between isolated containers after attaching it to the container which requires write access. ## `projects` Add a new project API, supporting creation, update and deletion of projects. Projects can hold containers, profiles or images at this point and let you get a separate view of your Incus resources by switching to it. ## `network_vxlan_ttl` This adds a new `tunnel.NAME.ttl` network configuration option which makes it possible to raise the TTL on VXLAN tunnels. ## `container_incremental_copy` This adds support for incremental container copy. When copying a container using the `--refresh` flag, only the missing or outdated files will be copied over. Should the target container not exist yet, a normal copy operation is performed. ## `usb_optional_vendorid` As the name implies, the `vendorid` field on USB devices attached to containers has now been made optional, allowing for all USB devices to be passed to a container (similar to what's done for GPUs). ## `snapshot_scheduling` This adds support for snapshot scheduling. It introduces three new configuration keys: `snapshots.schedule`, `snapshots.schedule.stopped`, and `snapshots.pattern`. Snapshots can be created automatically up to every minute. ## `snapshots_schedule_aliases` Snapshot schedule can be configured by a comma-separated list of schedule aliases. Available aliases are `<@hourly> <@daily> <@midnight> <@weekly> <@monthly> <@annually> <@yearly> <@startup>` for instances, and `<@hourly> <@daily> <@midnight> <@weekly> <@monthly> <@annually> <@yearly>` for storage volumes. ## `container_copy_project` Introduces a `project` field to the container source JSON object, allowing for copy/move of containers between projects. ## `clustering_server_address` This adds support for configuring a server network address which differs from the REST API client network address. When bootstrapping a new cluster, clients can set the new `cluster.https_address` configuration key to specify the address of the initial server. When joining a new server, clients can set the `core.https_address` configuration key of the joining server to the REST API address the joining server should listen at, and set the `server_address` key in the `PUT /1.0/cluster` API to the address the joining server should use for clustering traffic (the value of `server_address` will be automatically copied to the `cluster.https_address` configuration key of the joining server). ## `clustering_image_replication` Enable image replication across the nodes in the cluster. A new `cluster.images_minimal_replica` configuration key was introduced can be used to specify to the minimal numbers of nodes for image replication. ## `container_protection_shift` Enables setting the `security.protection.shift` option which prevents containers from having their file system shifted. ## `snapshot_expiry` This adds support for snapshot expiration. The task is run minutely. The configuration option `snapshots.expiry` takes an expression in the form of `1M 2H 3d 4w 5m 6y` (1 minute, 2 hours, 3 days, 4 weeks, 5 months, 6 years), however not all parts have to be used. Snapshots which are then created will be given an expiry date based on the expression. This expiry date, defined by `expires_at`, can be manually edited using the API or [`incus config edit`](incus_config_edit.md). Snapshots with a valid expiry date will be removed when the task in run. Expiry can be disabled by setting `expires_at` to an empty string or `0001-01-01T00:00:00Z` (zero time). This is the default if `snapshots.expiry` is not set. This adds the following new endpoint (see [RESTful API](rest-api.md) for details): * `PUT /1.0/containers//snapshots/` ## `snapshot_expiry_creation` Adds `expires_at` to container creation, allowing for override of a snapshot's expiry at creation time. ## `network_leases_location` Introduces a `Location` field in the leases list. This is used when querying a cluster to show what node a particular lease was found on. ## `resources_cpu_socket` Add Socket field to CPU resources in case we get out of order socket information. ## `resources_gpu` Add a new GPU struct to the server resources, listing all usable GPUs on the system. ## `resources_numa` Shows the NUMA node for all CPUs and GPUs. ## `kernel_features` Exposes the state of optional kernel features through the server environment. ## `id_map_current` This introduces a new internal `volatile.idmap.current` key which is used to track the current mapping for the container. This effectively gives us: * `volatile.last_state.idmap` => On-disk idmap * `volatile.idmap.current` => Current kernel map * `volatile.idmap.next` => Next on-disk idmap This is required to implement environments where the on-disk map isn't changed but the kernel map is (e.g. `idmapped mounts`). ## `event_location` Expose the location of the generation of API events. ## `storage_api_remote_volume_snapshots` This allows migrating storage volumes including their snapshots. ## `network_nat_address` This introduces the `ipv4.nat.address` and `ipv6.nat.address` configuration keys for Incus bridges. Those keys control the source address used for outbound traffic from the bridge. ## `container_nic_routes` This introduces the `ipv4.routes` and `ipv6.routes` properties on `nic` type devices. This allows adding static routes on host to container's NIC. ## `cluster_internal_copy` This makes it possible to do a normal `POST /1.0/containers` to copy a container between cluster nodes with Incus internally detecting whether a migration is required. ## `seccomp_notify` If the kernel supports `seccomp`-based syscall interception Incus can be notified by a container that a registered syscall has been performed. Incus can then decide to trigger various actions. ## `lxc_features` This introduces the `lxc_features` section output from the [`incus info`](incus_info.md) command via the `GET /1.0` route. It outputs the result of checks for key features being present in the underlying LXC library. ## `container_nic_ipvlan` This introduces the `ipvlan` `nic` device type. ## `network_vlan_sriov` This introduces VLAN (`vlan`) and MAC filtering (`security.mac_filtering`) support for SR-IOV devices. ## `storage_cephfs` Add support for CephFS as a storage pool driver. This can only be used for custom volumes, images and containers should be on Ceph (RBD) instead. ## `container_nic_ipfilter` This introduces container IP filtering (`security.ipv4_filtering` and `security.ipv6_filtering`) support for `bridged` NIC devices. ## `resources_v2` Rework the resources API at `/1.0/resources`, especially: * CPU * Fix reporting to track sockets, cores and threads * Track NUMA node per core * Track base and turbo frequency per socket * Track current frequency per core * Add CPU cache information * Export the CPU architecture * Show online/offline status of threads * Memory * Add huge-pages tracking * Track memory consumption per NUMA node too * GPU * Split DRM information to separate struct * Export device names and nodes in DRM struct * Export device name and node in NVIDIA struct * Add SR-IOV VF tracking ## `container_exec_user_group_cwd` Adds support for specifying `User`, `Group` and `Cwd` during `POST /1.0/containers/NAME/exec`. ## `container_syscall_intercept` Adds the `security.syscalls.intercept.*` configuration keys to control what system calls will be intercepted by Incus and processed with elevated permissions. ## `container_disk_shift` Adds the `shift` property on `disk` devices which controls the use of the `idmapped mounts` overlay. ## `storage_shifted` Introduces a new `security.shifted` Boolean on storage volumes. Setting it to `true` will allow multiple isolated containers to attach the same storage volume while keeping the file system writable from all of them. This makes use of `idmapped mounts` as an overlay file system. ## `resources_infiniband` Export InfiniBand character device information (`issm`, `umad`, `uverb`) as part of the resources API. ## `daemon_storage` This introduces two new configuration keys `storage.images_volume` and `storage.backups_volume` to allow for a storage volume on an existing pool be used for storing the daemon-wide images and backups artifacts. ## `instances` This introduces the concept of instances, of which currently the only type is `container`. ## `image_types` This introduces support for a new Type field on images, indicating what type of images they are. ## `resources_disk_sata` Extends the disk resource API struct to include: * Proper detection of SATA devices (type) * Device path * Drive RPM * Block size * Firmware version * Serial number ## `clustering_roles` This adds a new `roles` attribute to cluster entries, exposing a list of roles that the member serves in the cluster. ## `images_expiry` This allows for editing of the expiry date on images. ## `resources_network_firmware` Adds a `FirmwareVersion` field to network card entries. ## `backup_compression_algorithm` This adds support for a `compression_algorithm` property when creating a backup (`POST /1.0/containers//backups`). Setting this property overrides the server default value (`backups.compression_algorithm`). ## `ceph_data_pool_name` This adds support for an optional argument (`ceph.osd.data_pool_name`) when creating storage pools using Ceph RBD, when this argument is used the pool will store it's actual data in the pool specified with `data_pool_name` while keeping the metadata in the pool specified by `pool_name`. ## `container_syscall_intercept_mount` Adds the `security.syscalls.intercept.mount`, `security.syscalls.intercept.mount.allowed`, and `security.syscalls.intercept.mount.shift` configuration keys to control whether and how the `mount` system call will be intercepted by Incus and processed with elevated permissions. ## `compression_squashfs` Adds support for importing/exporting of images/backups using SquashFS file system format. ## `container_raw_mount` This adds support for passing in raw mount options for disk devices. ## `container_nic_routed` This introduces the `routed` `nic` device type. ## `container_syscall_intercept_mount_fuse` Adds the `security.syscalls.intercept.mount.fuse` key. It can be used to redirect file-system mounts to their fuse implementation. To this end, set e.g. `security.syscalls.intercept.mount.fuse=ext4=fuse2fs`. ## `container_disk_ceph` This allows for existing a Ceph RBD or CephFS to be directly connected to an Incus container. ## `virtual-machines` Add virtual machine support. ## `image_profiles` Allows a list of profiles to be applied to an image when launching a new container. ## `clustering_architecture` This adds a new `architecture` attribute to cluster members which indicates a cluster member's architecture. ## `resources_disk_id` Add a new `device_id` field in the disk entries on the resources API. ## `storage_lvm_stripes` This adds the ability to use LVM stripes on normal volumes and thin pool volumes. ## `vm_boot_priority` Adds a `boot.priority` property on NIC and disk devices to control the boot order. ## `unix_hotplug_devices` Adds support for Unix char and block device hotplugging. ## `api_filtering` Adds support for filtering the result of a GET request for instances and images. ## `instance_nic_network` Adds support for the `network` property on a NIC device to allow a NIC to be linked to a managed network. This allows it to inherit some of the network's settings and allows better validation of IP settings. ## `clustering_sizing` Support specifying a custom values for database voters and standbys. The new `cluster.max_voters` and `cluster.max_standby` configuration keys were introduced to specify to the ideal number of database voter and standbys. ## `firewall_driver` Adds the `Firewall` property to the `ServerEnvironment` struct indicating the firewall driver being used. ## `storage_lvm_vg_force_reuse` Introduces the ability to create a storage pool from an existing non-empty volume group. This option should be used with care, as Incus can then not guarantee that volume name conflicts won't occur with non-Incus created volumes in the same volume group. This could also potentially lead to Incus deleting a non-Incus volume should name conflicts occur. ## `container_syscall_intercept_hugetlbfs` When mount syscall interception is enabled and `hugetlbfs` is specified as an allowed file system type Incus will mount a separate `hugetlbfs` instance for the container with the UID and GID mount options set to the container's root UID and GID. This ensures that processes in the container can use huge pages. ## `limits_hugepages` This allows to limit the number of huge pages a container can use through the `hugetlb` cgroup. This means the `hugetlb` cgroup needs to be available. Note, that limiting huge pages is recommended when intercepting the mount syscall for the `hugetlbfs` file system to avoid allowing the container to exhaust the host's huge pages resources. ## `container_nic_routed_gateway` This introduces the `ipv4.gateway` and `ipv6.gateway` NIC configuration keys that can take a value of either `auto` or `none`. The default value for the key if unspecified is `auto`. This will cause the current behavior of a default gateway being added inside the container and the same gateway address being added to the host-side interface. If the value is set to `none` then no default gateway nor will the address be added to the host-side interface. This allows multiple routed NIC devices to be added to a container. ## `projects_restrictions` This introduces support for the `restricted` configuration key on project, which can prevent the use of security-sensitive features in a project. ## `custom_volume_snapshot_expiry` This allows custom volume snapshots to expiry. Expiry dates can be set individually, or by setting the `snapshots.expiry` configuration key on the parent custom volume which then automatically applies to all created snapshots. ## `volume_snapshot_scheduling` This adds support for custom volume snapshot scheduling. It introduces two new configuration keys: `snapshots.schedule` and `snapshots.pattern`. Snapshots can be created automatically up to every minute. ## `trust_ca_certificates` This allows for checking client certificates trusted by the provided CA (`server.ca`). It can be enabled by setting `core.trust_ca_certificates` to `true`. If enabled, it will perform the check, and bypass the trusted password if `true`. An exception will be made if the connecting client certificate is in the provided CRL (`ca.crl`). In this case, it will ask for the password. ## `snapshot_disk_usage` This adds a new `size` field to the output of `/1.0/instances//snapshots/` which represents the disk usage of the snapshot. ## `clustering_edit_roles` This adds a writable endpoint for cluster members, allowing the editing of their roles. ## `container_nic_routed_host_address` This introduces the `ipv4.host_address` and `ipv6.host_address` NIC configuration keys that can be used to control the host-side `veth` interface's IP addresses. This can be useful when using multiple routed NICs at the same time and needing a predictable next-hop address to use. This also alters the behavior of `ipv4.gateway` and `ipv6.gateway` NIC configuration keys. When they are set to `auto` the container will have its default gateway set to the value of `ipv4.host_address` or `ipv6.host_address` respectively. The default values are: `ipv4.host_address`: `169.254.0.1` `ipv6.host_address`: `fe80::1` This is backward compatible with the previous default behavior. ## `container_nic_ipvlan_gateway` This introduces the `ipv4.gateway` and `ipv6.gateway` NIC configuration keys that can take a value of either `auto` or `none`. The default value for the key if unspecified is `auto`. This will cause the current behavior of a default gateway being added inside the container and the same gateway address being added to the host-side interface. If the value is set to `none` then no default gateway nor will the address be added to the host-side interface. This allows multiple IPVLAN NIC devices to be added to a container. ## `resources_usb_pci` This adds USB and PCI devices to the output of `/1.0/resources`. ## `resources_cpu_threads_numa` This indicates that the `numa_node` field is now recorded per-thread rather than per core as some hardware apparently puts threads in different NUMA domains. ## `resources_cpu_core_die` Exposes the `die_id` information on each core. ## `api_os` This introduces two new fields in `/1.0`, `os` and `os_version`. Those are taken from the OS-release data on the system. ## `container_nic_routed_host_table` This introduces the `ipv4.host_table` and `ipv6.host_table` NIC configuration keys that can be used to add static routes for the instance's IPs to a custom policy routing table by ID. ## `container_nic_ipvlan_host_table` This introduces the `ipv4.host_table` and `ipv6.host_table` NIC configuration keys that can be used to add static routes for the instance's IPs to a custom policy routing table by ID. ## `container_nic_ipvlan_mode` This introduces the `mode` NIC configuration key that can be used to switch the `ipvlan` mode into either `l2` or `l3s`. If not specified, the default value is `l3s` (which is the old behavior). In `l2` mode the `ipv4.address` and `ipv6.address` keys will accept addresses in either CIDR or singular formats. If singular format is used, the default subnet size is taken to be /24 and /64 for IPv4 and IPv6 respectively. In `l2` mode the `ipv4.gateway` and `ipv6.gateway` keys accept only a singular IP address. ## `resources_system` This adds system information to the output of `/1.0/resources`. ## `images_push_relay` This adds the push and relay modes to image copy. It also introduces the following new endpoint: * `POST 1.0/images//export` ## `network_dns_search` This introduces the `dns.search` configuration option on networks. ## `container_nic_routed_limits` This introduces `limits.ingress`, `limits.egress` and `limits.max` for routed NICs. ## `instance_nic_bridged_vlan` This introduces the `vlan` and `vlan.tagged` settings for `bridged` NICs. `vlan` specifies the non-tagged VLAN to join, and `vlan.tagged` is a comma-delimited list of tagged VLANs to join. ## `network_state_bond_bridge` This adds a `bridge` and `bond` section to the `/1.0/networks/NAME/state` API. Those contain additional state information relevant to those particular types. Bond: * Mode * Transmit hash * Up delay * Down delay * MII frequency * MII state * Lower devices Bridge: * ID * Forward delay * STP mode * Default VLAN * VLAN filtering * Upper devices ## `resources_cpu_isolated` Add an `Isolated` property on CPU threads to indicate if the thread is physically `Online` but is configured not to accept tasks. ## `usedby_consistency` This extension indicates that `UsedBy` should now be consistent with suitable `?project=` and `?target=` when appropriate. The 5 entities that have `UsedBy` are: * Profiles * Projects * Networks * Storage pools * Storage volumes ## `custom_block_volumes` This adds support for creating and attaching custom block volumes to instances. It introduces the new `--type` flag when creating custom storage volumes, and accepts the values `fs` and `block`. ## `clustering_failure_domains` This extension adds a new `failure_domain` field to the `PUT /1.0/cluster/` API, which can be used to set the failure domain of a node. ## `container_syscall_filtering_allow_deny_syntax` A number of new syscalls related container configuration keys were updated. * `security.syscalls.deny_default` * `security.syscalls.deny_compat` * `security.syscalls.deny` * `security.syscalls.allow` ## `resources_gpu_mdev` Expose available mediated device profiles and devices in `/1.0/resources`. ## `console_vga_type` This extends the `/1.0/console` endpoint to take a `?type=` argument, which can be set to `console` (default) or `vga` (the new type added by this extension). When doing a `POST` to `/1.0//console?type=vga` the data WebSocket returned by the operation in the metadata field will be a bidirectional proxy attached to a SPICE Unix socket of the target virtual machine. ## `projects_limits_disk` Add `limits.disk` to the available project configuration keys. If set, it limits the total amount of disk space that instances volumes, custom volumes and images volumes can use in the project. ## `network_type_macvlan` Adds support for additional network type `macvlan` and adds `parent` configuration key for this network type to specify which parent interface should be used for creating NIC device interfaces on top of. Also adds `network` configuration key support for `macvlan` NICs to allow them to specify the associated network of the same type that they should use as the basis for the NIC device. ## `network_type_sriov` Adds support for additional network type `sriov` and adds `parent` configuration key for this network type to specify which parent interface should be used for creating NIC device interfaces on top of. Also adds `network` configuration key support for `sriov` NICs to allow them to specify the associated network of the same type that they should use as the basis for the NIC device. ## `container_syscall_intercept_bpf_devices` This adds support to intercept the `bpf` syscall in containers. Specifically, it allows to manage device cgroup `bpf` programs. ## `network_type_ovn` Adds support for additional network type `ovn` with the ability to specify a `bridge` type network as the `parent`. Introduces a new NIC device type of `ovn` which allows the `network` configuration key to specify which `ovn` type network they should connect to. Also introduces two new global configuration keys that apply to all `ovn` networks and NIC devices: * `network.ovn.integration_bridge` - the OVS integration bridge to use. * `network.ovn.northbound_connection` - the OVN northbound database connection string. ## `projects_networks` Adds the `features.networks` configuration key to projects and the ability for a project to hold networks. ## `projects_networks_restricted_uplinks` Adds the `restricted.networks.uplinks` project configuration key to indicate (as a comma-delimited list) which networks the networks created inside the project can use as their uplink network. ## `custom_volume_backup` Add custom volume backup support. This includes the following new endpoints (see [RESTful API](rest-api.md) for details): * `GET /1.0/storage-pools////backups` * `POST /1.0/storage-pools////backups` * `GET /1.0/storage-pools////backups/` * `POST /1.0/storage-pools////backups/` * `DELETE /1.0/storage-pools////backups/` * `GET /1.0/storage-pools////backups//export` The following existing endpoint has been modified: * `POST /1.0/storage-pools///` accepts the new source type `backup` ## `backup_override_name` Adds `Name` field to `InstanceBackupArgs` to allow specifying a different instance name when restoring a backup. Adds `Name` and `PoolName` fields to `StoragePoolVolumeBackupArgs` to allow specifying a different volume name when restoring a custom volume backup. ## `storage_rsync_compression` Adds `rsync.compression` configuration key to storage pools. This key can be used to disable compression in `rsync` while migrating storage pools. ## `network_type_physical` Adds support for additional network type `physical` that can be used as an uplink for `ovn` networks. The interface specified by `parent` on the `physical` network will be connected to the `ovn` network's gateway. ## `network_ovn_external_subnets` Adds support for `ovn` networks to use external subnets from uplink networks. Introduces the `ipv4.routes` and `ipv6.routes` setting on `physical` networks that defines the external routes allowed to be used in child OVN networks in their `ipv4.routes.external` and `ipv6.routes.external` settings. Introduces the `restricted.networks.subnets` project setting that specifies which external subnets are allowed to be used by OVN networks inside the project (if not set then all routes defined on the uplink network are allowed). ## `network_ovn_nat` Adds support for `ipv4.nat` and `ipv6.nat` settings on `ovn` networks. When creating the network if these settings are unspecified, and an equivalent IP address is being generated for the subnet, then the appropriate NAT setting will added set to `true`. If the setting is missing then the value is taken as `false`. ## `network_ovn_external_routes_remove` Removes the settings `ipv4.routes.external` and `ipv6.routes.external` from `ovn` networks. The equivalent settings on the `ovn` NIC type can be used instead for this, rather than having to specify them both at the network and NIC level. ## `tpm_device_type` This introduces the `tpm` device type. ## `storage_zfs_clone_copy_rebase` This introduces `rebase` as a value for `zfs.clone_copy` causing Incus to track down any `image` dataset in the ancestry line and then perform send/receive on top of that. ## `gpu_mdev` This adds support for virtual GPUs. It introduces the `mdev` configuration key for GPU devices which takes a supported `mdev` type, e.g. `i915-GVTg_V5_4`. ## `resources_pci_iommu` This adds the `IOMMUGroup` field for PCI entries in the resources API. ## `resources_network_usb` Adds the `usb_address` field to the network card entries in the resources API. ## `resources_disk_address` Adds the `usb_address` and `pci_address` fields to the disk entries in the resources API. ## `network_physical_ovn_ingress_mode` Adds `ovn.ingress_mode` setting for `physical` networks. Sets the method that OVN NIC external IPs will be advertised on uplink network. Either `l2proxy` (proxy ARP/NDP) or `routed`. ## `network_ovn_dhcp` Adds `ipv4.dhcp` and `ipv6.dhcp` settings for `ovn` networks. Allows DHCP (and RA for IPv6) to be disabled. Defaults to on. ## `network_physical_routes_anycast` Adds `ipv4.routes.anycast` and `ipv6.routes.anycast` Boolean settings for `physical` networks. Defaults to `false`. Allows OVN networks using physical network as uplink to relax external subnet/route overlap detection when used with `ovn.ingress_mode=routed`. ## `projects_limits_instances` Adds `limits.instances` to the available project configuration keys. If set, it limits the total number of instances (VMs and containers) that can be used in the project. ## `network_state_vlan` This adds a `vlan` section to the `/1.0/networks/NAME/state` API. Those contain additional state information relevant to VLAN interfaces: * `lower_device` * `vid` ## `instance_nic_bridged_port_isolation` This adds the `security.port_isolation` field for bridged NIC instances. ## `instance_bulk_state_change` Adds the following endpoint for bulk state change (see [RESTful API](rest-api.md) for details): * `PUT /1.0/instances` ## `network_gvrp` This adds an optional `gvrp` property to `macvlan` and `physical` networks, and to `ipvlan`, `macvlan`, `routed` and `physical` NIC devices. When set, this specifies whether the VLAN should be registered using GARP VLAN Registration Protocol. Defaults to `false`. ## `instance_pool_move` This adds a `pool` field to the `POST /1.0/instances/NAME` API, allowing for easy move of an instance root disk between pools. ## `gpu_sriov` This adds support for SR-IOV enabled GPUs. It introduces the `sriov` GPU type property. ## `pci_device_type` This introduces the `pci` device type. ## `storage_volume_state` Add new `/1.0/storage-pools/POOL/volumes/VOLUME/state` API endpoint to get usage data on a volume. ## `network_acl` This adds the concept of network ACLs to API under the API endpoint prefix `/1.0/network-acls`. ## `migration_stateful` Add a new `migration.stateful` configuration key. ## `disk_state_quota` This introduces the `size.state` device configuration key on `disk` devices. ## `storage_ceph_features` Adds a new `ceph.rbd.features` configuration key on storage pools to control the RBD features used for new volumes. ## `projects_compression` Adds new `backups.compression_algorithm` and `images.compression_algorithm` configuration keys which allows configuration of backup and image compression per-project. ## `projects_images_remote_cache_expiry` Add new `images.remote_cache_expiry` configuration key to projects, allowing for set number of days after which an unused cached remote image will be flushed. ## `certificate_project` Adds a new `restricted` property to certificates in the API as well as `projects` holding a list of project names that the certificate has access to. ## `network_ovn_acl` Adds a new `security.acls` property to OVN networks and OVN NICs, allowing Network ACLs to be applied. ## `projects_images_auto_update` Adds new `images.auto_update_cached` and `images.auto_update_interval` configuration keys which allows configuration of images auto update in projects ## `projects_restricted_cluster_target` Adds new `restricted.cluster.target` configuration key to project which prevent the user from using --target to specify what cluster member to place a workload on or the ability to move a workload between members. ## `images_default_architecture` Adds new `images.default_architecture` global configuration key and matching per-project key which lets user tell Incus what architecture to go with when no specific one is specified as part of the image request. ## `network_ovn_acl_defaults` Adds new `security.acls.default.{in,e}gress.action` and `security.acls.default.{in,e}gress.logged` configuration keys for OVN networks and NICs. This replaces the removed ACL `default.action` and `default.logged` keys. ## `gpu_mig` This adds support for NVIDIA MIG. It introduces the `mig` GPU type and associated configuration keys. ## `project_usage` Adds an API endpoint to get current resource allocations in a project. Accessible at API `GET /1.0/projects//state`. ## `network_bridge_acl` Adds a new `security.acls` configuration key to `bridge` networks, allowing Network ACLs to be applied. Also adds `security.acls.default.{in,e}gress.action` and `security.acls.default.{in,e}gress.logged` configuration keys for specifying the default behavior for unmatched traffic. ## `warnings` Warning API for Incus. This includes the following endpoints (see [Restful API](rest-api.md) for details): * `GET /1.0/warnings` * `GET /1.0/warnings/` * `PUT /1.0/warnings/` * `DELETE /1.0/warnings/` ## `projects_restricted_backups_and_snapshots` Adds new `restricted.backups` and `restricted.snapshots` configuration keys to project which prevents the user from creation of backups and snapshots. ## `clustering_join_token` Adds `POST /1.0/cluster/members` API endpoint for requesting a join token used when adding new cluster members without using the trust password. ## `clustering_description` Adds an editable description to the cluster members. ## `server_trusted_proxy` This introduces support for `core.https_trusted_proxy` which has Incus parse a HAProxy style connection header on such connections and if present, will rewrite the request's source address to that provided by the proxy server. ## `clustering_update_cert` Adds `PUT /1.0/cluster/certificate` endpoint for updating the cluster certificate across the whole cluster ## `storage_api_project` This adds support for copy/move custom storage volumes between projects. ## `server_instance_driver_operational` This modifies the `driver` output for the `/1.0` endpoint to only include drivers which are actually supported and operational on the server (as opposed to being included in Incus but not operational on the server). ## `server_supported_storage_drivers` This adds supported storage driver info to server environment info. ## `event_lifecycle_requestor_address` Adds a new address field to `lifecycle` requestor. ## `resources_gpu_usb` Add a new `USBAddress` (`usb_address`) field to `ResourcesGPUCard` (GPU entries) in the resources API. ## `clustering_evacuation` Adds `POST /1.0/cluster/members//state` endpoint for evacuating and restoring cluster members. It also adds the configuration keys `cluster.evacuate` and `volatile.evacuate.origin` for setting the evacuation method (`auto`, `stop` or `migrate`) and the origin of any migrated instance respectively. ## `network_ovn_nat_address` This introduces the `ipv4.nat.address` and `ipv6.nat.address` configuration keys for Incus `ovn` networks. Those keys control the source address used for outbound traffic from the OVN virtual network. These keys can only be specified when the OVN network's uplink network has `ovn.ingress_mode=routed`. ## `network_bgp` This introduces support for Incus acting as a BGP router to advertise routes to `bridge` and `ovn` networks. This comes with the addition to global configuration of: * `core.bgp_address` * `core.bgp_asn` * `core.bgp_routerid` The following network configurations keys (`bridge` and `physical`): * `bgp.peers..address` * `bgp.peers..asn` * `bgp.peers..password` The `nexthop` configuration keys (`bridge`): * `bgp.ipv4.nexthop` * `bgp.ipv6.nexthop` And the following NIC-specific configuration keys (`bridged` NIC type): * `ipv4.routes.external` * `ipv6.routes.external` ## `network_forward` This introduces the networking address forward functionality. Allowing for `bridge` and `ovn` networks to define external IP addresses that can be forwarded to internal IP(s) inside their respective networks. ## `custom_volume_refresh` Adds support for refresh during volume migration. ## `network_counters_errors_dropped` This adds the received and sent errors as well as inbound and outbound dropped packets to the network counters. ## `metrics` This adds metrics to Incus. It returns metrics of running instances using the OpenMetrics format. This includes the following endpoints: * `GET /1.0/metrics` ## `image_source_project` Adds a new `project` field to `POST /1.0/images` allowing for the source project to be set at image copy time. ## `clustering_config` Adds new `config` property to cluster members with configurable key/value pairs. ## `network_peer` This adds network peering to allow traffic to flow between OVN networks without leaving the OVN subsystem. ## `linux_sysctl` Adds new `linux.sysctl.*` configuration keys allowing users to modify certain kernel parameters within containers. ## `network_dns` Introduces a built-in DNS server and zones API to provide DNS records for Incus instances. This introduces the following server configuration key: * `core.dns_address` The following network configuration key: * `dns.zone.forward` * `dns.zone.reverse.ipv4` * `dns.zone.reverse.ipv6` And the following project configuration key: * `restricted.networks.zones` A new REST API is also introduced to manage DNS zones: * `/1.0/network-zones` (GET, POST) * `/1.0/network-zones/` (GET, PUT, PATCH, DELETE) ## `ovn_nic_acceleration` Adds new `acceleration` configuration key to OVN NICs which can be used for enabling hardware offloading. It takes the values `none` or `sriov`. ## `certificate_self_renewal` This adds support for renewing a client's own trust certificate. ## `instance_project_move` This adds a `project` field to the `POST /1.0/instances/NAME` API, allowing for easy move of an instance between projects. ## `storage_volume_project_move` This adds support for moving storage volume between projects. ## `cloud_init` This adds a new `cloud-init` configuration key namespace which contains the following keys: * `cloud-init.vendor-data` * `cloud-init.user-data` * `cloud-init.network-config` It also adds a new endpoint `/1.0/devices` to `/dev/incus` which shows an instance's devices. ## `network_dns_nat` This introduces `network.nat` as a configuration option on network zones (DNS). It defaults to the current behavior of generating records for all instances NICs but if set to `false`, it will instruct Incus to only generate records for externally reachable addresses. ## `database_leader` Adds new `database-leader` role which is assigned to cluster leader. ## `instance_all_projects` This adds support for displaying instances from all projects. ## `clustering_groups` Add support for grouping cluster members. This introduces the following new endpoints: * `/1.0/cluster/groups` (GET, POST) * `/1.0/cluster/groups/` (GET, POST, PUT, PATCH, DELETE) The following project restriction is added: * `restricted.cluster.groups` ## `ceph_rbd_du` Adds a new `ceph.rbd.du` Boolean on Ceph storage pools which allows disabling the use of the potentially slow `rbd du` calls. ## `instance_get_full` This introduces a new `recursion=1` mode for `GET /1.0/instances/{name}` which allows for the retrieval of all instance structs, including the state, snapshots and backup structs. ## `qemu_metrics` This adds a new `security.agent.metrics` Boolean which defaults to `true`. When set to `false`, it doesn't connect to the `incus-agent` for metrics and other state information, but relies on stats from QEMU. ## `gpu_mig_uuid` Adds support for the new MIG UUID format used by NVIDIA `470+` drivers (for example, `MIG-74c6a31a-fde5-5c61-973b-70e12346c202`), the `MIG-` prefix can be omitted This extension supersedes old `mig.gi` and `mig.ci` parameters which are kept for compatibility with old drivers and cannot be set together. ## `event_project` Expose the project an API event belongs to. ## `clustering_evacuation_live` This adds `live-migrate` as a configuration option to `cluster.evacuate`, which forces live-migration of instances during cluster evacuation. ## `instance_allow_inconsistent_copy` Adds `allow_inconsistent` field to instance source on `POST /1.0/instances`. If `true`, `rsync` will ignore the `Partial transfer due to vanished source files` (code 24) error when creating an instance from a copy. ## `network_state_ovn` This adds an `ovn` section to the `/1.0/networks/NAME/state` API which contains additional state information relevant to OVN networks: * chassis ## `storage_volume_api_filtering` Adds support for filtering the result of a GET request for storage volumes. ## `image_restrictions` This extension adds on to the image properties to include image restrictions/host requirements. These requirements help determine the compatibility between an instance and the host system. ## `storage_zfs_export` Introduces the ability to disable zpool export when unmounting pool by setting `zfs.export`. ## `network_dns_records` This extends the network zones (DNS) API to add the ability to create and manage custom records. This adds: * `GET /1.0/network-zones/ZONE/records` * `POST /1.0/network-zones/ZONE/records` * `GET /1.0/network-zones/ZONE/records/RECORD` * `PUT /1.0/network-zones/ZONE/records/RECORD` * `PATCH /1.0/network-zones/ZONE/records/RECORD` * `DELETE /1.0/network-zones/ZONE/records/RECORD` ## `network_zones_all_projects` This adds support for listing network zones across all projects through the `all-projects` parameter on the `GET /1.0/network-zones`API. ## `storage_zfs_reserve_space` Adds ability to set the `reservation`/`refreservation` ZFS property along with `quota`/`refquota`. ## `network_acl_log` Adds a new `GET /1.0/networks-acls/NAME/log` API to retrieve ACL firewall logs. ## `storage_zfs_blocksize` Introduces a new `zfs.blocksize` property for ZFS storage volumes which allows to set volume block size. ## `metrics_cpu_seconds` This is used to detect whether Incus was fixed to output used CPU time in seconds rather than as milliseconds. ## `instance_snapshot_never` Adds a `@never` option to `snapshots.schedule` which allows disabling inheritance. ## `certificate_token` This adds token-based certificate addition to the trust store as a safer alternative to a trust password. It adds the `token` field to `POST /1.0/certificates`. ## `instance_nic_routed_neighbor_probe` This adds the ability to disable the `routed` NIC IP neighbor probing for availability on the parent network. Adds the `ipv4.neighbor_probe` and `ipv6.neighbor_probe` NIC settings. Defaulting to `true` if not specified. ## `event_hub` This adds support for `event-hub` cluster member role and the `ServerEventMode` environment field. ## `agent_nic_config` If set to `true`, on VM start-up the `incus-agent` will apply NIC configuration to change the names and MTU of the instance NIC devices. ## `projects_restricted_intercept` Adds new `restricted.container.intercept` configuration key to allow usually safe system call interception options. ## `metrics_authentication` Introduces a new `core.metrics_authentication` server configuration option to allow for the `/1.0/metrics` endpoint to be generally available without client authentication. ## `images_target_project` Adds ability to copy image to a project different from the source. ## `images_all_projects` This adds support for listing images across all projects through the `all-projects` parameter on the `GET /1.0/images`API. ## `cluster_migration_inconsistent_copy` Adds `allow_inconsistent` field to `POST /1.0/instances/`. Set to `true` to allow inconsistent copying between cluster members. ## `cluster_ovn_chassis` Introduces a new `ovn-chassis` cluster role which allows for specifying what cluster member should act as an OVN chassis. ## `container_syscall_intercept_sched_setscheduler` Adds the `security.syscalls.intercept.sched_setscheduler` to allow advanced process priority management in containers. ## `storage_lvm_thinpool_metadata_size` Introduces the ability to specify the thin pool metadata volume size via `storage.thinpool_metadata_size`. If this is not specified then the default is to let LVM pick an appropriate thin pool metadata volume size. ## `storage_volume_state_total` This adds `total` field to the `GET /1.0/storage-pools/{name}/volumes/{type}/{volume}/state` API. ## `instance_file_head` Implements HEAD on `/1.0/instances/NAME/file`. ## `instances_nic_host_name` This introduces the `instances.nic.host_name` server configuration key that can take a value of either `random` or `mac`. The default value for the key if unspecified is `random`. If it is set to random then use the random host interface names. If it's set to `mac`, then generate a name in the form `inc1122334455`. ## `image_copy_profile` Adds ability to modify the set of profiles when image is copied. ## `container_syscall_intercept_sysinfo` Adds the `security.syscalls.intercept.sysinfo` to allow the `sysinfo` syscall to be populated with cgroup-based resource usage information. ## `clustering_evacuation_mode` This introduces a `mode` field to the evacuation request which allows for overriding the evacuation mode traditionally set through `cluster.evacuate`. ## `resources_pci_vpd` Adds a new VPD struct to the PCI resource entries. This struct extracts vendor provided data including the full product name and additional key/value configuration pairs. ## `qemu_raw_conf` Introduces a `raw.qemu.conf` configuration key to override select sections of the generated `qemu.conf`. ## `storage_cephfs_fscache` Add support for `fscache`/`cachefilesd` on CephFS pools through a new `cephfs.fscache` configuration option. ## `network_load_balancer` This introduces the networking load balancer functionality. Allowing `ovn` networks to define port(s) on external IP addresses that can be forwarded to one or more internal IP(s) inside their respective networks. ## `vsock_api` This introduces a bidirectional `vsock` interface which allows the `incus-agent` and the Incus server to communicate better. ## `instance_ready_state` This introduces a new `Ready` state for instances which can be set using `/dev/incus`. ## `network_bgp_holdtime` This introduces a new `bgp.peers..holdtime` configuration key to control the BGP hold time for a particular peer. ## `storage_volumes_all_projects` This introduces the ability to list storage volumes from all projects. ## `metrics_memory_oom_total` This introduces a new `incus_memory_OOM_kills_total` metric to the `/1.0/metrics` API. It reports the number of times the out of memory killer (`OOM`) has been triggered. ## `storage_buckets` This introduces the storage bucket API. It allows the management of S3 object storage buckets for storage pools. ## `storage_buckets_create_credentials` This updates the storage bucket API to return initial admin credentials at bucket creation time. ## `metrics_cpu_effective_total` This introduces a new `incus_cpu_effective_total` metric to the `/1.0/metrics` API. It reports the total number of effective CPUs. ## `projects_networks_restricted_access` Adds the `restricted.networks.access` project configuration key to indicate (as a comma-delimited list) which networks can be accessed inside the project. If not specified, all networks are accessible (assuming it is also allowed by the `restricted.devices.nic` setting, described below). This also introduces a change whereby network access is controlled by the project's `restricted.devices.nic` setting: * If `restricted.devices.nic` is set to `managed` (the default if not specified), only managed networks are accessible. * If `restricted.devices.nic` is set to `allow`, all networks are accessible (dependent on the `restricted.networks.access` setting). * If `restricted.devices.nic` is set to `block`, no networks are accessible. ## `storage_buckets_local` This introduces the ability to use storage buckets on local storage pools by setting the new `core.storage_buckets_address` global configuration setting. ## `loki` This adds support for sending life cycle and logging events to a Loki server. It adds the following global configuration keys: * `loki.api.ca_cert`: CA certificate which can be used when sending events to the Loki server * `loki.api.url`: URL to the Loki server (protocol, name or IP and port) * `loki.auth.username` and `loki.auth.password`: Used if Loki is behind a reverse proxy with basic authentication enabled * `loki.labels`: Comma-separated list of values which are to be used as labels for Loki events. * `loki.loglevel`: Minimum log level for events sent to the Loki server. * `loki.types`: Types of events which are to be sent to the Loki server (`lifecycle` and/or `logging`). ## `acme` This adds ACME support, which allows [Let's Encrypt](https://letsencrypt.org/) or other ACME services to issue certificates. It adds the following global configuration keys: * `acme.domain`: The domain for which the certificate should be issued. * `acme.email`: The email address used for the account of the ACME service. * `acme.ca_url`: The directory URL of the ACME service, defaults to `https://acme-v02.api.letsencrypt.org/directory`. It also adds the following endpoint, which is required for the HTTP-01 challenge: * `/.well-known/acme-challenge/` ## `internal_metrics` This adds internal metrics to the list of metrics. These include: * Total running operations * Total active warnings * Daemon uptime in seconds * Go memory stats * Number of goroutines ## `cluster_join_token_expiry` This adds an expiry to cluster join tokens which defaults to 3 hours, but can be changed by setting the `cluster.join_token_expiry` configuration key. ## `remote_token_expiry` This adds an expiry to remote add join tokens. It can be set in the `core.remote_token_expiry` configuration key, and default to no expiry. ## `storage_volumes_created_at` This change adds support for storing the creation date and time of storage volumes and their snapshots. This adds the `CreatedAt` field to the `StorageVolume` and `StorageVolumeSnapshot` API types. ## `cpu_hotplug` This adds CPU hotplugging for VMs. Hotplugging is disabled when using CPU pinning, because this would require hotplugging NUMA devices as well, which is not possible. ## `projects_networks_zones` This adds support for the `features.networks.zones` project feature, which changes which project network zones are associated with when they are created. Previously network zones were tied to the value of `features.networks`, meaning they were created in the same project as networks were. Now this has been decoupled from `features.networks` to allow projects that share a network in the default project (i.e those with `features.networks=false`) to have their own project level DNS zones that give a project oriented "view" of the addresses on that shared network (which only includes addresses from instances in their project). This also introduces a change to the network `dns.zone.forward` setting, which now accepts a comma-separated of DNS zone names (a maximum of one per project) in order to associate a shared network with multiple zones. No change to the `dns.zone.reverse.*` settings have been made, they still only allow a single DNS zone to be set. However the resulting zone content that is generated now includes `PTR` records covering addresses from all projects that are referencing that network via one of their forward zones. Existing projects that have `features.networks=true` will have `features.networks.zones=true` set automatically, but new projects will need to specify this explicitly. ## `instance_nic_txqueuelength` Adds a `txqueuelen` key to control the `txqueuelen` parameter of the NIC device. ## `cluster_member_state` Adds `GET /1.0/cluster/members//state` API endpoint and associated `ClusterMemberState` API response type. ## `instances_placement_scriptlet` Adds support for a Starlark scriptlet to be provided to Incus to allow customized logic that controls placement of new instances in a cluster. The Starlark scriptlet is provided to Incus via the new global configuration option `instances.placement.scriptlet`. ## `storage_pool_source_wipe` Adds support for a `source.wipe` Boolean on the storage pool, indicating that Incus should wipe partition headers off the requested disk rather than potentially fail due to pre-existing file systems. ## `zfs_block_mode` This adds support for using ZFS block {spellexception}`filesystem` volumes allowing the use of different file systems on top of ZFS. This adds the following new configuration options for ZFS storage pools: * `volume.zfs.block_mode` * `volume.block.mount_options` * `volume.block.filesystem` ## `instance_generation_id` Adds support for instance generation ID. The VM or container generation ID will change whenever the instance's place in time moves backwards. As of now, the generation ID is only exposed through to VM type instances. This allows for the VM guest OS to reinitialize any state it needs to avoid duplicating potential state that has already occurred: * `volatile.uuid.generation` ## `disk_io_cache` This introduces a new `io.cache` property to disk devices which can be used to override the VM caching behavior. ## `amd_sev` Adds support for AMD SEV (Secure Encrypted Virtualization) that can be used to encrypt the memory of a guest VM. This adds the following new configuration options for SEV encryption: * `security.sev` : (bool) is SEV enabled for this VM * `security.sev.policy.es` : (bool) is SEV-ES enabled for this VM * `security.sev.session.dh` : (string) guest owner's `base64`-encoded Diffie-Hellman key * `security.sev.session.data` : (string) guest owner's `base64`-encoded session blob ## `storage_pool_loop_resize` This allows growing loop file backed storage pools by changing the `size` setting of the pool. ## `migration_vm_live` This adds support for performing VM QEMU to QEMU live migration for both shared storage (clustered Ceph) and non-shared storage pools. This also adds the `CRIUType_VM_QEMU` value of `3` for the migration `CRIUType` `protobuf` field. ## `ovn_nic_nesting` This adds support for nesting an `ovn` NIC inside another `ovn` NIC on the same instance. This allows for an OVN logical switch port to be tunneled inside another OVN NIC using VLAN tagging. This feature is configured by specifying the parent NIC name using the `nested` property and the VLAN ID to use for tunneling with the `vlan` property. ## `oidc` This adds support for OpenID Connect (OIDC) authentication. This adds the following new configuration keys: * `oidc.issuer` * `oidc.client.id` * `oidc.audience` ## `network_ovn_l3only` This adds the ability to set an `ovn` network into "layer 3 only" mode. This mode can be enabled at IPv4 or IPv6 level using `ipv4.l3only` and `ipv6.l3only` configuration options respectively. With this mode enabled the following changes are made to the network: * The virtual router's internal port address will be configured with a single host netmask (e.g. /32 for IPv4 or /128 for IPv6). * Static routes for active instance NIC addresses will be added to the virtual router. * A discard route for the entire internal subnet will be added to the virtual router to prevent packets destined for inactive addresses from escaping to the uplink network. * The DHCPv4 server will be configured to indicate that a netmask of 255.255.255.255 be used for instance configuration. ## `ovn_nic_acceleration_vdpa` This updates the `ovn_nic_acceleration` API extension. The `acceleration` configuration key for OVN NICs can now takes the value `vdpa` to support Virtual Data Path Acceleration (VDPA). ## `cluster_healing` This adds cluster healing which automatically evacuates offline cluster members. This adds the following new configuration key: * `cluster.healing_threshold` The configuration key takes an integer, and can be disabled by setting it to 0 (default). If set, the value represents the threshold after which an offline cluster member is to be evacuated. In case the value is lower than `cluster.offline_threshold`, that value will be used instead. When the offline cluster member is evacuated, only remote-backed instances will be migrated. Local instances will be ignored as there is no way of migrating them once the cluster member is offline. ## `instances_state_total` This extension adds a new `total` field to `InstanceStateDisk` and `InstanceStateMemory`, both part of the instance's state API. ## `auth_user` Add current user details to the main API endpoint. This introduces: * `auth_user_name` * `auth_user_method` ## `security_csm` Introduce a new `security.csm` configuration key to control the use of `CSM` (Compatibility Support Module) to allow legacy operating systems to be run in Incus VMs. ## `instances_rebuild` This extension adds the ability to rebuild an instance with the same origin image, alternate image or as empty. A new `POST /1.0/instances//rebuild?project=` API endpoint has been added as well as a new CLI command [`incus rebuild`](incus_rebuild.md). ## `numa_cpu_placement` This adds the possibility to place a set of CPUs in a desired set of NUMA nodes. This adds the following new configuration key: * `limits.cpu.nodes` : (string) comma-separated list of NUMA node IDs or NUMA node ID ranges to place the CPUs (chosen with a dynamic value of `limits.cpu`) in. ## `custom_volume_iso` This adds the possibility to import ISO images as custom storage volumes. This adds the `--type` flag to [`incus storage volume import`](incus_storage_volume_import.md). ## `network_allocations` This adds the possibility to list an Incus deployment's network allocations. Through the [`incus network list-allocations`](incus_network_list-allocations.md) command and the `--project | --all-projects` flags, you can list all the used IP addresses, hardware addresses (for instances), resource URIs and whether it uses NAT for each `instance`, `network`, `network forward` and `network load-balancer`. ## `zfs_delegate` This implements a new `zfs.delegate` volume Boolean for volumes on a ZFS storage driver. When enabled and a suitable system is in use (requires ZFS 2.2 or higher), the ZFS dataset will be delegated to the container, allowing for its use through the `zfs` command line tool. ## `storage_api_remote_volume_snapshot_copy` This allows copying storage volume snapshots to and from remotes. ## `operations_get_query_all_projects` This introduces support for the `all-projects` query parameter for the GET API calls to both `/1.0/operations` and `/1.0/operations?recursion=1`. This parameter allows bypassing the project name filter. ## `metadata_configuration` Adds the `GET /1.0/metadata/configuration` API endpoint to retrieve the generated metadata configuration in a JSON format. The JSON structure adopts the structure ```"configs" > `ENTITY` > `ENTITY_SECTION` > "keys" > [, , ...]```. ## `syslog_socket` This introduces a syslog socket that can receive syslog formatted log messages. These can be viewed in the events API and `incus monitor`, and can be forwarded to Loki. To enable this feature, set `core.syslog_socket` to `true`. ## `event_lifecycle_name_and_project` This adds the fields `Name` and `Project` to `lifecycle` events. ## `instances_nic_limits_priority` This introduces a new per-NIC `limits.priority` option that works with both cgroup1 and cgroup2 unlike the deprecated `limits.network.priority` instance setting, which only worked with cgroup1. ## `disk_initial_volume_configuration` This API extension provides the capability to set initial volume configurations for instance root devices. Initial volume configurations are prefixed with `initial.` and can be specified either through profiles or directly during instance initialization using the `--device` flag. Note that these configuration are applied only at the time of instance creation and subsequent modifications have no effect on existing devices. ## `operation_wait` This API extension indicates that the `/1.0/operations/{id}/wait` endpoint exists on the server. This indicates to the client that the endpoint can be used to wait for an operation to complete rather than waiting for an operation event via the `/1.0/events` endpoint. ## `image_restriction_privileged` This extension adds a new image restriction, `requirements.privileged` which when `false` indicates that an image cannot be run in a privileged container. ## `cluster_internal_custom_volume_copy` This extension adds support for copying and moving custom storage volumes within a cluster with a single API call. Calling `POST /1.0/storage-pools//custom?target=` will copy the custom volume specified in the `source` part of the request. Calling `POST /1.0/storage-pools//custom/?target=` will move the custom volume from the source, specified in the `source` part of the request, to the target. ## `disk_io_bus` This introduces a new `io.bus` property to disk devices which can be used to override the bus the disk is attached to. ## `storage_cephfs_create_missing` This introduces the configuration keys `cephfs.create_missing`, `cephfs.osd_pg_num`, `cephfs.meta_pool` and `cephfs.osd_pool` to be used when adding a `cephfs` storage pool to instruct Incus to create the necessary entities for the storage pool, if they do not exist. ## `instance_move_config` This API extension provides the ability to use flags `--profile`, `--no-profile`, `--device`, and `--config` when moving an instance between projects and/or storage pools. ## `ovn_ssl_config` This introduces new server configuration keys to provide the SSL CA and client key pair to access the OVN databases. The new configuration keys are `network.ovn.ca_cert`, `network.ovn.client_cert` and `network.ovn.client_key`. ## `certificate_description` Adds a `description` field to certificate. ## `disk_io_bus_virtio_blk` Adds a new `virtio-blk` value for `io.bus` on `disk` devices which allows for the attached disk to be connected to the `virtio-blk` bus. ## `loki_config_instance` Adds a new `loki.instance` server configuration key to customize the `instance` field in Loki events. This can be used to expose the name of the cluster rather than the individual system name sending the event as that's usually already covered by the `location` field. ## `instance_create_start` Adds a new `start` field to the `POST /1.0/instances` API which when set to `true` will have the instance automatically start upon creation. In this scenario, the creation and startup is part of a single background operation. ## `clustering_evacuation_stop_options` This introduces new options for the `cluster.evacuate` option: * `stateful-stop` has the instance store its state to disk to be resume on restore. * `force-stop` has the instance immediately stopped without waiting for it to shut down. ## `boot_host_shutdown_action` This introduces a new `boot.host_shutdown_action` instance configuration key which can be used to override the default `stop` behavior on system shutdown. It supports the value `stop`, `stateful-stop` and `force-stop`. ## `agent_config_drive` This introduces a new `agent:config` disk `source` which can be used to expose an ISO to the VM guest containing the agent and its configuration. ## `network_state_ovn_lr` Adds a new `LogicalRouter` field to the `NetworkStateOVN` struct which is part of the `GET /1.0/networks/NAME/state` API. This is used to get the OVN logical router name. ## `image_template_permissions` This adds `uid`, `gid` and `mode` fields to the image metadata template entries. ## `storage_bucket_backup` Add storage bucket backup support. This includes the following new endpoints (see [RESTful API](rest-api.md) for details): * `GET /1.0/storage-pools//buckets//backups` * `POST /1.0/storage-pools//buckets//backups` * `GET /1.0/storage-pools//buckets//backups/` * `POST /1.0/storage-pools//buckets//backups/` * `DELETE /1.0/storage-pools//buckets//backups/` * `GET /1.0/storage-pools//buckets//backups//export` ## `storage_lvm_cluster` This adds a new `lvmcluster` storage driver which makes use of LVM shared VG through `lvmlockd`. With this, it's possible to have a single shared LVM pool across multiple servers so long as they all see the same backing device(s). ## `shared_custom_block_volumes` This adds a new configuration key `security.shared` to custom block volumes. If unset or `false`, the custom block volume cannot be attached to multiple instances. This feature was added to prevent data loss which can happen when custom block volumes are attached to multiple instances at once. ## `auth_tls_jwt` This adds the ability to use a signed `JSON Web Token` (`JWT`) instead of using the TLS client certificate directly. In this scenario, the client derives a `JWT` from their own TLS client certificate providing it as a `bearer` token through the `Authorization` HTTP header. The `JWT` must have the certificate's fingerprint as its `Subject` and must be signed by the client's private key. ## `oidc_claim` This introduces a new `oidc.claim` server configuration key which can be used to specify what OpenID Connect claim to use as the username. ## `device_usb_serial` This adds a new configuration key `serial` for device type `usb`. Feature has been added, to make it possible to distinguish between devices with identical `vendorid` and `productid`. ## `numa_cpu_balanced` This adds `balanced` as a new value for `limits.cpu.nodes`. When set to `balanced`, Incus will attempt to select the least busy NUMA node at startup time for the instance, trying to keep the load spread across NUMA nodes on the system. ## `image_restriction_nesting` This extension adds a new image restriction, `requirements.nesting` which when `true` indicates that an image cannot be run without nesting. ## `network_integrations` Adds the concept of network integrations and initial support for OVN Interconnection. New API: * `/1.0/network-integrations` (GET, POST) * `/1.0/network-integrations/NAME` (GET, PUT, PATCH, DELETE, POST) Each integration is made of: * name * description * type (only `ovn` for now) * configuration * `ovn.northbound_connection` (database connection string for the OVN Interconnection database) * `ovn.ca_cert` (optional, SSL CA certificate for the OVN Interconnection database) * `ovn.client_cert` (optional, SSL client certificate to connect to the OVN Interconnection database) * `ovn.client_key` (optional, SSL client key to connect to the OVN Interconnection database) * `ovn.transit.pattern` (Pongo2 template to generate the transit switch name) Those integrations attach to network peers through some new fields: * `type` (`local` for current behavior, `remote` for integrations) * `target_integration` (reference to the integration) ## `instance_memory_swap_bytes` This extends `limits.memory.swap` to allow for a total limit in bytes. ## `network_bridge_external_create` This adds the ability for `bridge.external_interfaces` to create a parent interface using a `interface/parent/vlan` syntax. ## `storage_zfs_vdev` This adds support for `mirror`, `raidz1` and `raidz2` ZFS `vdev` types by extending storage `source` configuration. ## `container_migration_stateful` A `migration.stateful` configuration key was introduced. It's a Boolean flag set to true whenever the container is in a stateful mode during the start, stop, and snapshot functions. This makes it less likely for users to run into CRIU errors when copying containers to another system. ## `profiles_all_projects` This adds support for listing profiles across all projects through the `all-projects` parameter on the `GET /1.0/profiles`API. ## `instances_scriptlet_get_instances` This allows the instance scriptlet to fetch a list of instances given an optional Project or Location filter. ## `instances_scriptlet_get_cluster_members` This allows the instance scriptlet to fetch a list of cluster members given an optional cluster group. ## `instances_scriptlet_get_project` This allows the instance scriptlet to fetch a project given name of a project. ## `network_acl_stateless` This adds support for stateless rules in network ACLs. ## `instance_state_started_at` This adds a `started_at` timestamp to the instance state API. ## `networks_all_projects` This adds support for listing networks across all projects through the `all-projects` parameter on the `GET /1.0/networks`API. ## `network_acls_all_projects` This adds support for listing network ACLs across all projects through the `all-projects` parameter on the `GET /1.0/network-acls`API. ## `storage_buckets_all_projects` This adds support for listing storage buckets across all projects through the `all-projects` parameter on the `GET /1.0/storage-pools/POOL/buckets`API. ## `resources_load` Add a new Load section to the resources API. ## `instance_access` This introduces a new API endpoint at `GET /1.0/instances/NAME/access` which exposes who can interact with the instance and what role they have. ## `project_access` This introduces a new API endpoint at `GET /1.0/projects/NAME/access` which exposes who can interact with the project and what role they have. ## `projects_force_delete` This extends `DELETE /1.0/projects` to allow `?force=true` which will delete everything inside of the project along with the project itself. ## `resources_cpu_flags` This exposes the CPU flags/extensions in our resources API to check the CPU features. ## `disk_io_bus_cache_filesystem` This adds support for both `io.bus` and `io.cache` to disks that are backed by a file system. ## `instance_oci` Adds initial support for running OCI containers. ## `clustering_groups_config` This introduces a standard key/value `config` option to clustering groups which will allow placing some restrictions or configuration on those groups. ## `instances_lxcfs_per_instance` This introduces a new `instances.lxcfs.per_instance` server configuration key to control whether to run LXCFS per instance instead of globally on the system. ## `clustering_groups_vm_cpu_definition` This introduces a few new configuration options to control the virtual machine CPU definitions through cluster group configuration. The new configuration keys are: * `instances.vm.cpu.ARCHITECTURE.baseline` * `instances.vm.cpu.ARCHITECTURE.flag` ## `disk_volume_subpath` This introduces the ability to access the sub-path of a file system custom volume by using the `source=volume/path` syntax. ## `projects_limits_disk_pool` This introduces per-pool project disk limits, introducing a `limits.disk.pool.NAME` configuration option to the project limits. ## `network_ovn_isolated` This allows using `none` as the uplink network for an OVN network, making the network isolated. ## `qemu_raw_qmp` This adds new configuration options to virtual machines to directly issue QMP commands at various stages of startup: * `raw.qemu.qmp.early` * `raw.qemu.qmp.pre-start` * `raw.qemu.qmp.post-start` ## `network_load_balancer_health_check` This adds the ability to perform health checks for load balancer backends. The following new configuration options are introduced: * `healthcheck` * `healthcheck.interval` * `healthcheck.timeout` * `healthcheck.failure_count` * `healthcheck.success_count` ## `oidc_scopes` This introduces a new `oidc.scopes` server configuration key which can take a comma separate list of OIDC scopes to request from the identity provider. ## `network_integrations_peer_name` This extends `ovn.transit.pattern` to allow `peerName` as a template variable. ## `qemu_scriptlet` This adds the ability to run a scriptlet at various stages of startup: using the `raw.qemu.scriptlet` configuration key. ## `instance_auto_restart` This introduces a new `boot.autorestart` configuration key which when set to `true` will have the instance automatically be restarted upon unexpected exit for up to 10 times over a 1 minute period. ## `storage_lvm_metadatasize` This introduces a new `lvm.metadata_size` option for LVM storage pools which allows overriding the default metadata size when creating a new LVM physical volume. ## `ovn_nic_promiscuous` This implements a new `security.promiscuous` configuration option on OVN NICs. ## `ovn_nic_ip_address_none` This adds `none` as a value for `ipv4.address` and `ipv6.address` for OVN NICs. ## `instances_state_os_info` This extension adds a pointer to an `InstanceStateOSInfo` struct to the instance's state API. ## `network_load_balancer_state` This adds a new `/1.0/networks/NAME/load-balancers/IP/state` API endpoint which returns load-balancer health check information (when configured). ## `instance_nic_macvlan_mode` This adds a `mode` configuration key on `macvlan` network interfaces which allows for configuring the Macvlan mode. ## `storage_lvm_cluster_create` Allow for creating new LVM cluster pools by setting the `source` to the shared block device. ## `network_ovn_external_interfaces` This adds support for `bridge.external_interfaces` on OVN networks. ## `instances_scriptlet_get_instances_count` This allows the instance scriptlet to fetch the count instances given an optional Project or Location filter as well as including pending instances. ## `cluster_rebalance` This adds automatic live-migration to balance load on cluster again. As part of this, the following configuration options have been added: * `cluster.rebalance.batch` * `cluster.rebalance.cooldown` * `cluster.rebalance.interval` * `cluster.rebalance.threshold` ## `custom_volume_refresh_exclude_older_snapshots` This adds support for excluding source snapshots earlier than latest target snapshot. ## `storage_initial_owner` This adds ability to set the initial owner of a custom volume. The following configuration options have been added: * `initial.gid` * `initial.mode` * `initial.uid` ## `storage_live_migration` This adds support for virtual-machines live-migration between storage pools. ## `instance_console_screenshot` This adds support to take screenshots of the current VGA console of a VM. ## `image_import_alias` Adds a new `X-Incus-aliases` HTTP header to set aliases while uploading an image. ## `authorization_scriptlet` This adds the ability to define a scriptlet in a new configuration key, `authorization.scriptlet`, managing authorization on the Incus cluster. ## `console_force` This adds support for forcing a connection to the console, even if there is already an active session. It introduces the new `--force` flag for connecting to the instance console. ## `network_ovn_state_addresses` This adds extra fields to the OVN network state struct for the IPv4 and IPv6 addresses used on the uplink. ## `qemu_scriptlet_config` This extends the QEMU scriptlet feature by allowing to modify QEMU configuration before a VM starts, and passing information about the instance to the scriptlet. ## `network_bridge_acl_devices` This adds support for device ACLs when attached to a bridged network. ## `instance_debug_memory` Add new memory dump API at `/1.0/instances/NAME/debug/memory`. ## `init_preseed_storage_volumes` This API extension provides the ability to configure storage volumes in preseed init. ## `init_preseed_profile_project` This API extension provides the ability to specify the project as part of profile definitions in preseed init. ## `instance_nic_routed_host_address` Adds support for specifying the VRF to add the routes to. ## `instance_smbios11` A new category of configuration options, `smbios11.XYZ` has been added which allows passing key/value pairs through `SMBIOS Type 11` on systems that support it. ## `api_filtering_extended` This extends the API filtering mechanism to all API collections. ## `acme_dns01` Adds support for `DNS-01` challenge to the Incus ACME support for certificate generation. ## `security_iommu` Introduce a new `security.iommu` configuration key to control whether to enable IOMMU emulation. This is done through `virtio_iommu` on Linux and the emulated Intel IOMMU on Windows. ## `network_ipv4_dhcp_routes` Introduces a new `ipv4.dhcp.routes` configuration option on bridged and OVN networks. This allows specifying pairs of CIDR networks and gateway address to be announced by the DHCP server. ## `network_state_ovn_ls` Adds a new `LogicalSwitch` field to the `NetworkStateOVN` struct which is part of the `GET /1.0/networks/NAME/state` API. This is used to get the OVN logical switch name. ## `network_dns_nameservers` Introduces the `dns.nameservers` configuration option on bridged and OVN networks. This allows specifying IPv4 and IPv6 DNS server addresses to be announced by the DHCP server and via Router Advertisements. ## `acme_http01_port` Adds `acme.http.port` to control an alternative HTTP port for `HTTP-01` validation. ## `network_ovn_ipv4_dhcp_expiry` Introduces `ipv4.dhcp.expiry` for OVN networks. ## `instance_state_cpu_time` This adds an `allocated_time` field below `CPU` in the instance state API. ## `network_io_bus` This introduces a new `io.bus` property for compatible network devices allowing to choose between `virtio` (default) and `usb`. ## `disk_io_bus_usb` Adds a new `usb` value for `io.bus` on `disk` devices. ## `storage_driver_linstor` This adds a LINSTOR storage driver. ## `instance_oci_entrypoint` This introduces a set of new configuration options on the container to configure the OCI entry point: * `oci.entrypoint` * `oci.cwd` * `oci.uid` * `oci.gid` Those are initialized at creation time using the values from the OCI image. ## `network_address_set` This adds the concept of network address sets to API under the API endpoint prefix `/1.0/network-address-sets`. ## `server_logging` This implements a new set of `logging` configuration keys on the server, allowing for multiple logging targets. The former `loki` configuration keys are being transitioned over as part of this. ## `network_forward_snat` Adds a `snat` configuration option for network forwards which will cause any DNAT to get a matching SNAT applied. So new connections from the target will appear as coming from the network forward address. This is limited to bridged networks as OVN doesn't support flexible enough SNAT for this. ## `memory_hotplug` This adds memory hotplugging for VMs, allowing them to add memory at runtime without rebooting. ## `instance_nic_routed_host_tables` This adds support for specifying host-routing tables on `nic` devices that use the routed mode. ## `instance_publish_split` This adds support for creating a split format image out of an existing instance. ## `init_preseed_certificates` This API extension provides the ability to configure certificates in preseed init. ## `custom_volume_sftp` This adds the SFTP API to custom storage volumes. ## `network_ovn_external_nic_address` This adds support for configuring a custom external IPv4 or IPv6 address for a given instance so long as that address is available through a network forward. ## `network_physical_gateway_hwaddr` Allows setting the MAC address of the IPv4 and IPv6 gateways when used with OVN. ## `backup_s3_upload` Adds support for immediately uploading instance or volume backups to an S3 compatible endpoint. ## `snapshot_manual_expiry` Introduces a `snapshots.expiry.manual` configuration key to both instances and storage volumes which allows overriding the default expiry value for snapshots created directly by the user as opposed to created on schedule. ## `resources_cpu_address_sizes` This adds tracking of CPU address sizes in the resources API. The main use of this is within clusters to calculate a cluster-wide maximum memory amount for hotplugging into virtual machines. ## `disk_attached` This introduces a new `attached` property to disk devices describing whether disks are attached or ejected. ## `limits_memory_hotplug` The `limits.memory.hotplug` option controls how memory hotplug is handled for the virtual machine. It can be set to `false` to completely disable memory hotplugging. Alternatively, it can be set to a value that defines the maximum amount of memory the VM can reach through hotplug. This value must be greater than or equal to `limits.memory`. ## `disk_wwn` Add support for setting the disk World Wide Name property through the new `wwn` disk configuration option. ## `server_logging_webhook` This adds support for basic webhook as a logging target. It can be selected through `logging.NAME.target.type` with the `webhook` value. The following target keys are supported: * `logging.NAME.target.address` (URL of the target) * `logging.NAME.target.ca_cert` (Certificate when using an HTTPS target with a self-signed certificate) * `logging.NAME.target.username` (Username for HTTP authentication) * `logging.NAME.target.password` (Password for HTTP authentication) * `logging.NAME.target.retry` (How many times to retry the transmission) The webhook data matches what's sent over the existing events API. ## `storage_driver_truenas` This adds a TrueNAS storage driver. ## `container_disk_tmpfs` This adds tmpfs support for disk devices. * `source=tmpfs:` mounts a tmpfs file system, respecting `size`, `uid`, `gid` and `mode` options * `source=tmpfs-overlay:` same as tmpfs but with additional overlayfs behavior ## `instance_limits_oom` This adds a new `limits.memory.oom_priority` configuration option to configure the Out Of Memory score for the container or virtual-machine. ## `backup_override_config` This adds support for overriding both configuration and devices during backup import by using the `X-Incus-config` and `X-Incus-devices` HTTP headers. ## `network_ovn_tunnels` This adds support for network tunnels to OVN networks. ## `init_preseed_cluster_groups` This API extension provides the ability to configure cluster groups in preseed init. ## `usb_attached` This introduces a new `attached` property to USB devices describing whether they are plugged in or not. ## `backup_iso` This allows to backup ISO custom volumes by simply copying them. It enables exporting ISO volumes with `incus storage volume export` and get an ISO back. ## `instance_systemd_credentials` This adds two categories of configuration options, `systemd.credential.*` and `systemd.credential-binary.*`, which allow passing systemd credentials through a bind-mounted directory for containers and `SMBIOS Type 11` for virtual machines. ## `cluster_group_usedby` A `used_by` field was added to the `GET /1.0/cluster/groups/{name}` endpoint. `used_by` holds the URLs of all instances and projects using the cluster group. ## `bpf_token_delegation` This adds support for [eBPF token delegation](https://docs.ebpf.io/linux/concepts/token/). ## `file_storage_volume` This adds file transfer API on the custom volumes. Implements `DELETE`, `GET`, `HEAD`, `POST` operations on the `/1.0/storage-pools/{poolName}/volumes/{type}/{volumeName}/files` endpoint. ## `network_hwaddr_pattern` This adds `network.hwaddr_pattern` global and per-project configuration keys to customize MAC address allocation. ## `storage_volume_full` This adds `?recursion=1` support to `/1.0/storage-pools/POOL/volumes/VOLUME` and `?recursion=2` support to `/1.0/storage-pools/POOL/volumes`. As well as a matching `StorageVolumeFull` struct. ## `storage_bucket_full` This adds `?recursion=1` support to `/1.0/storage-pools/POOL/buckets/VOLUME` and `?recursion=2` support to `/1.0/storage-pools/POOL/buckets`. As well as a matching `StorageBucketFull` struct. ## `device_pci_firmware` This extension introduces a new boolean `firmware` option for PCI devices. When set to `false`, Incus tells QEMU to disable the ROM-BAR (`rombar=0`) for the device. ## `resources_serial` This adds serial device tracking to the resources API. ## `ovn_nic_limits` This adds support for `limits.egress`, `limits.ingress`, `limits.max` and `limits.priority` on `ovn` type network devices. ## `storage_lvmcluster_qcow2` This adds support for running virtual machines from `QCOW2` images. Support is currently limited to `lvmcluster` storage driver. ## `oidc_allowed_subnets` This indicates support by Incus for the custom `incus.allowed_subnets` OIDC claim. ## `file_delete_force` This adds a new `X-Incus-force` HTTP header that when set to `true` allows for recursive deletion of instance or custom volume files. ## `nic_sriov_select_ext` Adds support for selecting an SR-IOV network interface by vendor ID, product ID, or PCI address. ## `network_zones_dns_contact` Adds a `dns.contact` configuration key to network zones. ## `nic_attached_connected` This introduces two new properties for NICs: * `attached`, behaving like the `attached` key for disk and USB devices; * `connected`, setting the up/down link state for the NIC (when supported). ## `nic_sriov_security_trusted` This introduces a new property for SR-IOV NICs: * `security.trusted` allows setting the `trusted` flag for the virtual function if the parent NIC supports it. ## `direct_backup` This allows to perform backups of instances, custom storage volumes and storage buckets without disk buffering when the `Accept` header is set to `application/octet-stream` in the respective POST API endpoints. Doing so, said POST endpoints return the data stream instead of creating a backup entry in the database. ## `instance_snapshot_disk_only_restore` This adds support for only restoring the disk of a snapshotted instance. ## `unix_hotplug_pci` Adds a `pci` configuration key on the `unix-hotplug` device to allow filtering based on the USB controller PCI address. ## `cluster_evacuating_restoring` Introduces new `Evacuating` and `Restoring` cluster member states. ## `projects_restricted_image_servers` Introduces a new `restricted.images.servers` project configuration option. This allows specifying a comma separate list of image server domains from which the user may download images. ## `storage_lvmcluster_size` Adds support for specifying the `size` parameter when updating `lvmcluster` storage pool. ## `authorization_scriptlet_cert` This adds two fields, `Chain` and `Certificate`, to the `details` argument of the authorization scriptlet, to give more information about the client to authorize. ## `lvmcluster_remove_snapshots` It is a Boolean that defaults to `false`. When set to `true`, it instructs Incus to remove any required snapshots when attempting to restore another snapshot. This is necessary because `lvmcluster` only allows restoring the latest snapshot. ## `daemon_storage_logs` This adds `storage.logs_volume` alongside the existing `storage.backups_volume` and `storage.images_volume`. ## `instances_debug_repair` This adds a new API at `POST /1.0/instances/NAME/debug/repair` to trigger low-level repair actions. ## `network_io_bus_ovn` This ports the `io.bus` property available on most NIC devices to non-accelerated OVN NICs. ## `dependent` This introduces a new `dependent` configuration key for disk devices and custom volumes. Volumes marked as `dependent` are tied to the `lifecycle` of the instance they are attached to. Snapshot operations on the instance are automatically propagated to the dependent volume. Creating an instance snapshot creates a corresponding snapshot on the dependent volume, and deleting the instance snapshot removes it as well. Direct snapshot creation or deletion on a dependent volume is not allowed. Exporting and importing an instance also includes dependent volumes. ## `metrics_project_resources` This adds project-level metrics to the `/1.0/metrics` endpoint including resource counts, configured limits, and current usage per project. ## `storage_volume_nbd` This adds a few new APIs for storage volumes and instances: * `POST /1.0/storage-pools/POOL/volumes/TYPE/VOLUME/nbd` * `GET /1.0/storage-pools/POOL/volumes/TYPE/VOLUME/bitmaps` * `POST /1.0/storage-pools/POOL/volumes/TYPE/VOLUME/bitmaps` * `GET /1.0/storage-pools/POOL/volumes/TYPE/VOLUME/bitmaps/NAME` * `DELETE /1.0/storage-pools/POOL/volumes/TYPE/VOLUME/bitmaps/NAME` * `POST /1.0/instances/NAME/bitmaps` This introduces the ability to get a raw `NBD` connection to an Incus block storage volume. It also allows interacting with dirty bitmaps on those volumes which can then be used as the basis for incremental backups. The instance level endpoint allows for consistent bitmap creation across the VM and its dependent volumes. ## `projects_restricted_storage_pool_access` Adds the `restricted.storage-pools.access` project configuration key to indicate (as a comma-delimited list) which storage pools can be accessed inside the project. If not specified, all storage pools are accessible. Storage pools that are not in the list are treated as equivalent to having a pool size limit of 0 (`limits.disk.pool.POOLNAME=0`), making them inaccessible and hidden from the project. ## `server_shutdown_action` This adds a new `core.shutdown_action` server configuration option. Its default value of `shutdown` keeps the current behavior but `evacuate` can be set to have a clustered server attempt an evacuation on shutdown with any instance left after it getting shutdown. ## `instances_placement_scriptlet_rebalance` Add a new placement scriptlet trigger for cluster re-balancing. ## `network_bridge_multicast_snooping` This adds a new `bridge.multicast_snooping` configuration option for managed bridge networks. When set to `false`, multicast snooping is disabled on the bridge, which can help with IPv6 protocols that rely on multicast such as NDP. ## `storage_create_options` This introduces a new `block.create_options` configuration option to allow controlling the `mkfs` arguments when creating volumes on block devices. `btrfs.create_options` is also introduced to similarly control `mkfs.btrfs` options when creating a new storage pool. ## `instances_tpm_platform_cert` This introduces a couple of new configuration options: * `instances.tpm.platform_cert` * `instances.tpm.platform_key` Those can be used to provide a TPM platform CA certificate. When set, new TPM devices will be getting an Endorsement Key that's signed by that CA. This can be used to validate that the VM is in fact running in the expected Incus environment, providing a root of trust for those TPM devices. ## `linstor_raw` This introduces `linstor.raw.*` configuration keys for both LINSTOR storage pools and storage volumes, allowing to set low-level properties to resource groups and resource definitions. ## `network_address_set_ip_ranges` This adds support for IP ranges (e.g. `10.0.0.120-10.0.0.130`) in network address sets, matching what's already allowed in network ACL rules. A single range may expand to at most 256 addresses. Larger sets of addresses should be expressed using CIDR notation instead. ## `storage_volumes_rebuild` This extension adds the ability to rebuild a custom storage volume. The underlying volume is deleted and a new empty one is created with the same configuration. The rebuild is only allowed when the volume has no snapshots. ## `api_fragments` URL fragments in API paths are to be treated by clients as references to configuration keys for the objects said paths point to. For example, `/1.0?target=foo#storage.logs_volume` refers to the `storage.logs_volume` configuration key for the cluster member `foo`. ## `instance_limits_cpu_topology` This extends the `limits.cpu` configuration key for virtual machines to allow specifying an explicit CPU topology of the form `sockets=2,cores=4,threads=2`. ## `instance_nbd` This adds a new `GET /1.0/instances/{name}/nbd` endpoint which acts similarly to the equivalent storage volume endpoint but allows concurrent access to all disks attached to the VM. ## `network_bridge_bgp_instances` This adds the `bgp.ipv4.instances` and `bgp.ipv6.instances` configuration keys to managed bridge networks. When enabled, Incus advertises a `/32` (IPv4) or `/128` (IPv6) route over BGP for each running instance connected to the network, withdrawing it again when the instance stops. ## `core_https_allowed_websocket_origin` This adds a new `core.https_allowed_websocket_origin` server configuration key. It can be set to a comma-separate list of allowed origins or to the `*` wildcard. ## `storage_btrfs_compression` This adds a new `btrfs.compression` storage volume configuration key for the `btrfs` driver. It maps to the Btrfs `compression` property and takes the same values (for example `zstd`, `lzo`, `zlib` or `none`). ## `oci_network_config` This allows for static network configuration of OCI application containers. The NIC `ipv4.address` and `ipv6.address` keys can now be set to a CIDR value to statically configure the address inside the container and the new `ipv4.gateway` and `ipv6.gateway` keys can be used to set the default gateway. Setting either address to `none` prevents any configuration for that address family and stops the built-in DHCP client from running on it. For DNS, the `oci.dns.nameservers`, `oci.dns.domain` and `oci.dns.search` instance configuration keys can be used to set the initial content of the container's `resolv.conf`, which is then extended with any value received over DHCP. All of those keys are only valid for OCI containers. ## `infiniband_sriov_guid` This adds two new configuration keys, `node_guid` and `port_guid`, to `infiniband` devices using the `sriov` `nictype`. When set, the matching GUID of the allocated virtual function is changed to the provided value when the instance starts and restored to its original value when the instance stops. ## `instance_selinux` This adds per-instance SELinux integration for containers and virtual machines, with automatic MCS level allocation for isolation between instances on the same host. The following instance configuration keys are added: * `security.selinux.domain`: Override the SELinux process domain. * `security.selinux.type`: Override the SELinux file type used for the instance storage. * `security.selinux.level`: Override the SELinux MCS level. * `security.selinux.label_rootfs`: Control rootfs labeling behavior (`auto`, `always` or `never`). The computed context is persisted in the `volatile.selinux.context` key so that MCS ranges stay stable across restarts. ## `network_bgp_peer_interface` This adds a new `bgp.peers.NAME.interface` configuration key to `bridge` and `physical` networks as an alternative to `bgp.peers.NAME.address`. When set, the BGP session is established over the given interface using BGP unnumbered (the peer's IPv6 link-local address is discovered automatically and IPv4 routes are exchanged using extended next-hop). ## `projects_restricted_virtual_machines_nesting` This adds a new `restricted.virtual-machines.nesting` project configuration key. When set to `block`, all virtual machines in the project must have `security.nesting` set to `false`, turning off nested virtualization. ## `authorization_config` This moves the authorization configuration keys under a general `authorization.*` namespace. The following server configuration keys replace the former `openfga.*` keys: * `authorization.openfga.api.url`: URL of the OpenFGA server * `authorization.openfga.api.token`: API token of the OpenFGA server * `authorization.openfga.store.id`: ID of the OpenFGA permission store Existing `openfga.*` values are automatically migrated to the new keys on upgrade. ## `network_allocations_network` Adds the `network` field to the network allocations API response. ## `gpu_native_context` This adds a new `native-context` `gputype` for `gpu` devices on virtual machines. Instead of passing through a PCI device, it gives the VM an accelerated `virtio-gpu` that uses DRM native context, so the guest's own driver drives the host GPU directly, with graphics and video acceleration. One host GPU can be shared by several VMs and the host keeps using it. The `blob.size` device option sets the host-visible blob window (default 2GiB). Requires QEMU 11.0.0 or newer. ## `instance_port_forward` This adds a new `POST /1.0/instances/NAME/port-forward` API endpoint which upgrades the connection to a raw TCP connection to the specified address and port inside of the instance. For containers, the connection is established directly by the server from within the container's network namespace. For virtual machines, the request is forwarded to the `incus-agent` which then handles the connection, this is controlled by the new `port-forward` agent feature. A matching `incus port-forward` command is added to the client, providing a local TCP listener which forwards every connection to the instance. ## `unix_block_limits` This adds the `limits.read` and `limits.write` configuration keys to `unix-block` devices. These behave similarly to their `disk` device equivalents, accepting either a byte/s value or an IOPS value. ## `authorization_client_routing` This allows loading multiple authorization drivers at once and routing each request to one of them based on the authentication class of the client. The following server configuration keys are added: * `authorization.client.default`: driver for clients without a more specific class route * `authorization.client.unix`: driver for local (`unix` socket) clients * `authorization.client.tls`: driver for unrestricted TLS clients * `authorization.client.tls-restricted`: driver for restricted (project-scoped) TLS clients * `authorization.client.oidc`: driver for OIDC-authenticated clients Each key accepts one of `allow`, `deny`, `openfga` or `scriptlet`. A per-class key falls back to `authorization.client.default` when unset. `authorization.client.tls-restricted` additionally accepts `tls`, as the TLS authorization method exists to enforce the per-certificate project restrictions that only apply to restricted certificates. The following server configuration key is also added: * `authorization.openfga.tls.identifier`: certificate attribute (`fingerprint` or `name`) used as the OpenFGA user when a TLS client is authorized by OpenFGA (defaults to `name`). ## `instance_nvram` This adds new endpoints to manage virtual machines’ UEFI variables: * `GET /1.0/instances/{name}/nvram`, to get all UEFI variables * `GET /1.0/instances/{name}/nvram/{guid}`, to get UEFI variables under the given GUID * `GET /1.0/instances/{name}/nvram/{guid}/{var}`, to get specific UEFI variables * `DELETE /1.0/instances/{name}/nvram/{guid}/{var}`, to delete specific UEFI variables * `PUT /1.0/instances/{name}/nvram/{guid}/{var}`, to set/update specific UEFI variables It also adds `rebuild-nvram` as an instance debug action. ## `disk_io_limits_combined` This makes it possible to set both a byte/s limit and an IOPS limit at the same time in the I/O limit keys of `disk` and `unix-block` devices by using a comma separated list of values (for example, `limits.read=30MiB,1000iops`). ## `resources_cpu_cluster` Adds a `cluster` field to CPU core entries in the resources API, telling apart cores with identical identifiers on systems where core identifiers are only unique within a CPU cluster (ARM big.LITTLE). ## `acme_eab` Adds support for ACME External Account Binding (EAB) through two new server configuration keys: * `acme.eab.kid` * `acme.eab.hmac` ## `instance_protection_start` Adds a new `security.protection.start` instance configuration key which when set to `true` prevents the instance from being started. ## `instance_nvram_bulk_update` This adds a new `PATCH /1.0/instances/{name}/nvram` endpoint to update NVRAM in bulk. ## `instance_refresh_migration` Adds support for refresh migration, where a running container is moved to another cluster member through a series of incremental transfers, only being stopped for the last one. This adds a `refresh` field to `POST /1.0/instances/{name}`. When set to `true` on a stateless migration (`migration` set to `true` and `live` set to `false`), a running container is migrated this way. It also adds `refresh-migrate` as a configuration option to `cluster.evacuate` and as an evacuation mode, which forces refresh migration of instances during cluster evacuation. ## `image_locations` Adds a new `locations` field to the image API, listing the names of the cluster members which have a local copy of the image. ## `network_ovn_multicast` Adds multicast configuration to OVN networks through two new configuration keys: * `bridge.multicast_snooping` * `bridge.multicast_relay` ## `device_burst_limits` This adds burst support to the I/O limits of `disk` and `nic` devices. For `disk` devices (virtual machines only), the new `limits.read.burst`, `limits.write.burst` and `limits.max.burst` keys take the same syntax as their sustained counterparts and define the rate the device may reach while bursting. The new `limits.read.burst.length`, `limits.write.burst.length` and `limits.max.burst.length` keys define how long the burst rate may be sustained for, defaulting to one second. For `nic` devices (`bridged`, `p2p`, `routed` and `ovn`), the new `limits.ingress.bucket`, `limits.egress.bucket` and `limits.max.bucket` keys define the amount of data, in bit, that may be sent in excess of the sustained limit. The `bridged`, `p2p` and `routed` types also take `limits.ingress.burst`, `limits.egress.burst` and `limits.max.burst`, defining the bit/s rate at which that bucket may be spent. A direction needs both keys to burst. The `ovn` type only takes the bucket keys, as OVN has no separate burst rate. ## `network_ipv6_ra` Adds a new `ipv6.ra` configuration key to both `bridge` and `ovn` networks, controlling whether IPv6 router advertisements are sent on the network. ## `qemu_scriptlet_nvram` This extends the QEMU scriptlet feature by adding functions to explore and edit the NVRAM: * `get_nvram_var` * `has_nvram_var` * `set_nvram_var` * `unset_nvram_var` * `get_raw_nvram_var` * `set_raw_nvram_var` * `list_nvram_vars` ## `storage_ceph_rbd_backend` Adds a new `ceph.rbd.backend` configuration key to the Ceph RBD storage driver. When set to `librbd`, volumes are accessed through `librbd` rather than the RBD kernel driver, with `qemu-nbd` being used whenever a block device is needed on the host. ## `instance_nvram_config` This adds three categories of initial configuration options, `initial.nvram.*`, `initial.nvram-binary.*` and `initial.secureboot`, which allow defining default NVRAM variables and Secure Boot certificates and keys when rebuilding the NVRAM. ## `storage_cephobject_endpoint_cert` Adds a new `cephobject.radosgw.endpoint_cert` configuration key to the Ceph Object storage driver, holding the TLS certificate (PEM bundle) to trust when communicating with the `radosgw` endpoint. This replaces `cephobject.radosgw.endpoint_cert_file`, existing values are converted automatically. ## `device_queue_disc` Adds new `queue.discipline` and `queue.discipline.attach` configuration keys to `nic` devices of type `bridged`, `p2p` and `routed`, controlling the queuing discipline used on the host side of the NIC and where it gets attached. ## `network_bridge_dns_include_hosts` Adds a new `dns.include_hosts` configuration key to `bridge` networks, controlling whether the network's `dnsmasq` serves records from the host's `/etc/hosts` file. Setting it to `false` avoids exposing host-only entries to instances while keeping AppArmor confinement. ## `gpu_physical_clique` Adds a new `nvidia.clique` configuration key to `gpu` devices of type `physical` when used with virtual machines. It sets the NVIDIA GPUDirect P2P clique ID (0 to 15) advertised to the guest driver, allowing peer-to-peer DMA between GPUs passed through to the same virtual machine when they share a clique ID. ## `authorization_scriptlet_claims` This adds a `Claims` field to the `details` argument of the authorization scriptlet, holding the validated OIDC token claims of the client as a dictionary. This allows writing rules based on claims such as `groups`. ## `instance_project_move_live` This allows a running instance to change project as part of a live migration to another cluster member. It also lets instances with `dependent` disks attached change project. ## `metrics_cluster_members` This adds cluster member metrics to the `/1.0/metrics` endpoint: * `incus_cluster_member` with the member's architecture and failure domain * `incus_cluster_member_status` with one sample per possible status * `incus_cluster_member_role` with one sample per role held * `incus_cluster_member_group` with one sample per group the member belongs to ## `instance_security_tags` Adds a new `security.tags` configuration key to instances, holding a comma-separated list of tags. The tags are exposed to the authorization backend, with OpenFGA maintaining a `security_tag` object for each tag along with a `tag` relation to every instance carrying it. ## `network_ovn_parent` Adds a new `parent` configuration key to `ovn` networks, referencing another `ovn` network in the same project. ## `disk_initial_copy` This adds the `initial.copy` property to `disk` devices backed by a custom storage volume. When set to `true` on a container, the content already present at the device `path` inside the container is copied into the volume the first time the volume is used, if it's empty. ## `internal_debug_pprof` This adds `/internal/debug/pprof/` to the API, serving the same `pprof` profiles as the `core.debug_address` listener but over the regular listeners, restricted to trusted clients with server administration rights. ## `instance_state_disk_counters` Adds a `counters` field to each entry of the `disk` section of an instance's state, holding `bytes_read`, `bytes_written`, `reads_completed` and `writes_completed`. These are only populated for virtual machines, using the block I/O statistics reported by QEMU. Containers do not report them because cgroup I/O accounting is keyed by host block device rather than by Incus device name, and a container's root file system and custom volumes routinely share a single host block device, making per-device attribution impossible. ## `instances_state_tpm` Adds a `tpm` section to the instance state, listing every `tpm` device along with its endorsement keys, each with a type, size, SHA-256 fingerprint and PEM encoded public key. The endorsement keys are only known for TPM devices provisioned by Incus, that is when `instances.tpm.platform_cert` was set at the time the device was first started. ## `ovn_nic_security_filtering` This adds the `security.mac_filtering`, `security.ipv4_filtering` and `security.ipv6_filtering` options to OVN NICs. They are implemented through OVN port security on the instance's logical switch port, restricting the source MAC address and, when IP filtering is enabled, the IP addresses (including any static routes) the instance may use. ## `authorization_scriptlet_request` This adds the following fields to the `details` argument of the authorization scriptlet: * `URL`, holding the raw client request URL (`?`) * `Path`, holding the API path requested by the client * `Query`, holding the dictionary of query parameters (note: the values are lists of strings and not single strings) * `Method`, holding the HTTP method used in the request ## `vm_boot_priority_skip` This adds the ability to define `boot.priority` to `skip` on NIC and disk devices, to skip them from the boot sequence. The new default for NICs is to be skipped. This additionally adds a `boot.pxe` VM configuration key, defaulting to `false`, to enable PXE on all the NICs included in the boot sequence. This doesn’t affect HTTP boot, which is always enabled for those NICs. ## `network_ovn_nat_address_l2proxy` Allows `ipv4.nat.address` and `ipv6.nat.address` on an `ovn` network whose uplink uses `ovn.ingress_mode=l2proxy`, previously only supported with `ovn.ingress_mode=routed`. ## `oidc_id_token` This adds support for OIDC providers that issue opaque (non-JWT) access tokens. When the access token isn't a JWT, Incus verifies the ID token instead, checking that it was issued by the configured issuer for the configured client ID. Clients send the ID token alongside the opaque access token through the `X-Incus-OIDC-idtoken` header. ## `certificate_ca_trust` A CA certificate added to the trust store now also trusts the client certificates it issued. Those clients are identified by the CA entry's fingerprint and inherit its type, restrictions and projects. ## `network_bgp_communities` This adds a `bgp.communities` network configuration key for `bridge` and `ovn` networks. It takes a comma-separated list of standard BGP communities (`ASN:value`) which are attached to all prefixes advertised by the network. --- ### Doc Api (`doc/api.md`) # REST API ```{toctree} :maxdepth: 1 Main API documentation rest-api-spec Main API extensions Instance API documentation Events API documentation Metrics API documentation ``` --- ### Doc Architectures (`doc/architectures.md`) (architectures)= # Architectures Incus can run on just about any architecture that is supported by the Linux kernel and by Go. Some entities in Incus are tied to an architecture, for example, the instances, instance snapshots and images. The following table lists all supported architectures including their unique identifier and the name used to refer to them. The architecture names are typically aligned with the Linux kernel architecture names. | ID | Kernel name | Description | Personalities | | :--- | :--- | :--- | :--- | | 1 | `i686` | 32bit Intel x86 | | | 2 | `x86_64` | 64bit Intel x86 | `x86` | | 3 | `armv7l` | 32bit ARMv7 little-endian | | | 4 | `aarch64` | 64bit ARMv8 little-endian | `armv7l` (optional) | | 5 | `ppc` | 32bit PowerPC big-endian | | | 6 | `ppc64` | 64bit PowerPC big-endian | `powerpc` | | 7 | `ppc64le` | 64bit PowerPC little-endian | | | 8 | `s390x` | 64bit ESA/390 big-endian | | | 9 | `mips` | 32bit MIPS | | | 10 | `mips64` | 64bit MIPS | `mips` | | 11 | `riscv32` | 32bit RISC-V little-endian | | | 12 | `riscv64` | 64bit RISC-V little-endian | | | 13 | `armv6l` | 32bit ARMv6 little-endian | | | 14 | `armv8l` | 32bit ARMv8 little-endian | | | 15 | `loongarch64` | 64bit Loongarch | | ```{note} Incus cares only about the kernel architecture, not the particular userspace flavor as determined by the toolchain. That means that Incus considers ARMv7 hard-float to be the same as ARMv7 soft-float and refers to both as `armv7l`. If useful to the user, the exact userspace ABI may be set as an image and container property, allowing easy query. ``` ## Virtual-machine support Incus only supports running virtual-machines on the following host architectures: - `x86_64` - `aarch64` - `ppc64le` - `s390x` - `loongarch64` The virtual machine guest architecture can usually be the 32bit personality of the host architecture, so long as the virtual machine firmware is capable of booting it. --- ### Doc Authentication (`doc/authentication.md`) (authentication)= # Remote API authentication Remote communications with the Incus daemon happen using JSON over HTTPS. To be able to access the remote API, clients must authenticate with the Incus server. The following authentication methods are supported: - {ref}`authentication-tls-certs` - {ref}`authentication-openid` (authentication-tls-certs)= ## TLS client certificates When using {abbr}`TLS (Transport Layer Security)` client certificates for authentication, both the client and the server will generate a key pair the first time they're launched. The server will use that key pair for all HTTPS connections to the Incus socket. The client will use its certificate as a client certificate for any client-server communication. To cause certificates to be regenerated, simply remove the old ones. On the next connection, a new certificate is generated. ### Communication protocol The supported protocol must be TLS 1.3 or better. It's possible to force Incus to accept TLS 1.2 by setting the `INCUS_INSECURE_TLS` environment variable on both client and server. However this isn't a supported setup and should only ever be used when forced to use an outdated corporate proxy. All communications must use perfect forward secrecy, and ciphers must be limited to strong elliptic curve ones (such as ECDHE-RSA or ECDHE-ECDSA). Any generated key should be at least 4096 bit RSA, preferably 384 bit ECDSA. When using signatures, only SHA-2 signatures should be trusted. Since we control both client and server, there is no reason to support any backward compatibility to broken protocol or ciphers. (authentication-trusted-clients)= ### Trusted TLS clients You can obtain the list of TLS certificates trusted by an Incus server with [`incus config trust list`](incus_config_trust_list.md). Trusted clients can be added in either of the following ways: - {ref}`authentication-add-certs` - {ref}`authentication-token` The workflow to authenticate with the server is similar to that of SSH, where an initial connection to an unknown server triggers a prompt: 1. When the user adds a server with [`incus remote add`](incus_remote_add.md), the server is contacted over HTTPS, its certificate is downloaded and the fingerprint is shown to the user. 1. The user is asked to confirm that this is indeed the server's fingerprint, which they can manually check by connecting to the server or by asking someone with access to the server to run the info command and compare the fingerprints. 1. The server attempts to authenticate the client: - If the client certificate is in the server's trust store, the connection is granted. - If the client certificate is not in the server's trust store, the server prompts the user for a token. If the provided token matches, the client certificate is added to the server's trust store and the connection is granted. Otherwise, the connection is rejected. It is possible to restrict a TLS client's access to Incus via {ref}`authorization-tls`. To revoke trust to a client, remove its certificate from the server with [`incus config trust remove `](incus_config_trust_remove.md). (authentication-tls-jwt)= #### Using `JSON Web Token` (`JWT`) to perform TLS authentication As an alternative to directly using the client's TLS certificate for authentication, Incus also supports the user derive a `bearer` token and use it through the HTTP `Authorization` header. To do this, the user must generate a signed `JWT` which has its `Subject` field set to the full fingerprint of their client certificate, it must have valid `NotBefore` and `NotAfter` fields and be signed by the client certificate's private key. (authentication-add-certs)= #### Adding trusted certificates to the server The preferred way to add trusted clients is to directly add their certificates to the trust store on the server. To do so, copy the client certificate to the server and register it using [`incus config trust add-certificate `](incus_config_trust_add-certificate.md). (authentication-ca-certs)= #### Trusting a certificate authority Instead of adding every client certificate, you can add a CA certificate to the trust store. Any client certificate issued by that CA is then trusted as if the CA entry itself had been presented: the client is identified by the fingerprint of the CA entry and inherits its type, restrictions and projects. The client must present any intermediate certificate between its own certificate and the trusted CA, and its certificate must allow the `clientAuth` extended key usage if it specifies any. This makes it possible to use short-lived client certificates issued by an internal PKI without updating the trust store on every renewal. Revocation lists aren't consulted for such certificates, so remove the CA entry from the trust store to revoke all the certificates it issued. Certificates that are directly present in the trust store always take precedence over a matching CA entry. (authentication-token)= #### Adding client certificates using tokens You can also add new clients by using tokens. Tokens expire after a configurable time ({config:option}`server-core:core.remote_token_expiry`) or once they've been used. To use this method, generate a token for each client by calling [`incus config trust add`](incus_config_trust_add.md), which will prompt for the client name. The clients can then add their certificates to the server's trust store by providing the generated token when prompted. ```{note} If your Incus server is behind NAT, you must specify its external public address when adding it as a remote for a client: incus remote add When generating the token on the server, Incus includes a list of IP addresses that the client can use to access the server. However, if the server is behind NAT, these addresses might be local addresses that the client cannot connect to. In this case, you must specify the external address manually. ``` Alternatively, the clients can provide the token directly when adding the remote: [`incus remote add `](incus_remote_add.md). ### Using a PKI system In a {abbr}`PKI (Public key infrastructure)` setup, a system administrator manages a central PKI that issues client certificates for all the Incus clients and server certificates for all the Incus daemons. To enable PKI mode, complete the following steps: 1. Add the {abbr}`CA (Certificate authority)` certificate to all machines: - Place the `client.ca` file in the clients' configuration directories (`~/.config/incus`). - Place the `server.ca` file in the server's configuration directory (`/var/lib/incus`). 1. Place the certificates issued by the CA on the clients and the server, replacing the automatically generated ones. 1. Restart the server. In that mode, any connection to an Incus daemon will be done using the pre-seeded CA certificate. If the server certificate isn't signed by the CA, the connection will simply go through the normal authentication mechanism. If the server certificate is valid and signed by the CA, then the connection continues without prompting the user for the certificate. Note that the generated certificates are not automatically trusted. You must still add them to the server in one of the ways described in {ref}`authentication-trusted-clients`. In that mode, only client certificates issued by the CA can be added to the trust store. Alternatively, you can set {config:option}`server-core:core.trust_ca_certificates` to `true` to have the server automatically trust any valid client certificate issued by the CA. Such certificates don't need to be added to the trust store and get full access to the server, unless they are added to the trust store as restricted certificates. Certificates listed in a `ca.crl` file placed in the server's configuration directory are rejected. ```{caution} Only enable `core.trust_ca_certificates` with a CA dedicated to Incus, as anyone holding a certificate issued by that CA can then administer the server. ``` ### Encrypting local keys The `incus` client also supports encrypted client keys. Keys generated via the methods above can be encrypted with a password, using: ``` ssh-keygen -p -o -f .config/incus/client.key ``` ```{note} Unless you enable [`keepalive` mode](remote-keepalive), then every single call to Incus will cause the prompt which may get a bit annoying: $ incus list remote-host: Password for client.key: ┌──────┬───────┬──────┬──────┬──────┬───────────┐ │ NAME │ STATE │ IPV4 │ IPV6 │ TYPE │ SNAPSHOTS │ └──────┴───────┴──────┴──────┴──────┴───────────┘ ``` ```{note} While the `incus` command line supports encrypted keys, tools such as [Ansible's connection plugin](https://docs.ansible.com/ansible/latest/collections/community/general/incus_connection.html) do not. ``` (authentication-openid)= ## OpenID Connect authentication Incus supports using [OpenID Connect](https://openid.net/connect/) to authenticate users through an {abbr}`OIDC (OpenID Connect)` Identity Provider. ```{note} Authentication through OpenID Connect is supported, but there is no user role handling in place so far. Any user that authenticates through the configured OIDC Identity Provider gets full access to Incus. ``` To configure Incus to use OIDC authentication, set the [`oidc.*`](server-options-oidc) server configuration options. Your OIDC provider must be configured to enable the [Device Authorization Grant](https://oauth.net/2/device-flow/) type. To add a remote pointing to an Incus server configured with OIDC authentication, run [`incus remote add `](incus_remote_add.md). You are then prompted to authenticate through your web browser, where you must confirm the device code that Incus uses. The Incus client then retrieves and stores the access and refresh tokens and provides those to Incus for all interactions. Incus verifies JWT access tokens offline using the provider's signing keys. Providers that issue opaque access tokens are also supported, in which case Incus verifies the ID token instead and checks that it was issued for the configured client ID. The client ID used by Incus must therefore not be shared with any other application registered with the provider. Setting `oidc.audience` disables this fallback as the audience can't be verified on an opaque access token. Incus supports a custom OIDC claim of `incus.allowed_subnets` (list of strings), if the claim is set, the user will only be allowed if connecting from an IP address that's part of one of the CIDR subnets listed in the claim. ```{important} Any user that authenticates through the configured OIDC Identity Provider gets full access to Incus. To restrict user access, you must also configure {ref}`authorization`. Currently, the only authorization method that is compatible with OIDC is {ref}`authorization-openfga`. ``` (authentication-server-certificate)= ## TLS server certificate Incus supports issuing server certificates using {abbr}`ACME (Automatic Certificate Management Environment)` services, for example, [Let's Encrypt](https://letsencrypt.org/). To enable this feature, set the [relevant server configuration options](server-options-acme). Incus supports both `HTTP-01` and `DNS-01` challenges. The set of configuration option varies between the two. For `DNS-01`, the relevant {config:option}`server-acme:acme.provider` and {config:option}`server-acme:acme.provider.environment` values can be found directly in the [documentation of `lego`](https://go-acme.github.io/lego/dns/index.html), the ACME client that Incus uses behind the scenes. For `HTTP-01`, Incus will cause `lego` to temporarily listen on port `80` so the HTTP challenge can go through. In a cluster, the challenge runs on the leader while the other members temporarily listen on that same port and forward the challenge requests to the leader. This allows for a certificate covering the names of all cluster members. If your Incus server sits behind a reverse proxy, you'll need that reverse proxy to redirect HTTP traffic to HTTPS. For ACME services that require External Account Binding (EAB), set both {config:option}`server-acme:acme.eab.kid` and {config:option}`server-acme:acme.eab.hmac` to the values provided by your ACME service. ## Failure scenarios In the following scenarios, authentication is expected to fail. ### Server certificate changed The server certificate might change in the following cases: - The server was fully reinstalled and therefore got a new certificate. - The connection is being intercepted ({abbr}`MITM (Machine in the middle)`). In such cases, the client will refuse to connect to the server because the certificate fingerprint does not match the fingerprint in the configuration for this remote. It is then up to the user to contact the server administrator to check if the certificate did in fact change. If it did, the certificate can be replaced by the new one, or the remote can be removed altogether and re-added. ### Server trust relationship revoked The server trust relationship is revoked for a client if another trusted client or the local server administrator removes the trust entry for the client on the server. In this case, the server still uses the same certificate, but all API calls return a 403 code with an error indicating that the client isn't trusted. --- ### Doc Authorization (`doc/authorization.md`) (authorization)= # Authorization When interacting with Incus over the Unix socket, members of the `incus-admin` group will have full access to the Incus API. Those who are only members of the `incus` group will instead be restricted to a single project tied to their user. When interacting with Incus over the network (see {ref}`server-expose` for instructions), it is possible to further authenticate and restrict user access. There are three supported authorization methods: - {ref}`authorization-tls` - {ref}`authorization-openfga` - {ref}`authorization-scriptlet` By default, the method used for a request is determined automatically from the client's authentication protocol. This can be customized so that different kinds of clients are handled by different methods; see {ref}`authorization-client-routing`. (authorization-tls)= ## TLS authorization Incus natively supports restricting {ref}`authentication-trusted-clients` to one or more projects. When a client certificate is restricted, the client will also be prevented from performing global configuration changes or altering the configuration (limits, restrictions) of the projects it's allowed access to. To restrict access, use [`incus config trust edit `](incus_config_trust_edit.md). Set the `restricted` key to `true` and specify a list of projects to restrict the client to. If the list of projects is empty, the client will not be allowed access to any of them. By default, this authorization method is used for clients authenticating with a restricted TLS certificate. Clients using an unrestricted certificate are granted full access instead. Both can be changed, see {ref}`authorization-client-routing`. (authorization-openfga)= ## Open Fine-Grained Authorization (OpenFGA) Incus supports integrating with [{abbr}`OpenFGA (Open Fine-Grained Authorization)`](https://openfga.dev). This authorization method is highly granular. For example, it can be used to restrict user access to a single instance. To use OpenFGA for authorization, you must configure and run an OpenFGA server yourself. Incus will connect to the OpenFGA server, write the {ref}`openfga-model`, and query this server for authorization for all subsequent requests. To enable this authorization method in Incus, set the [`authorization.openfga.*`](server-options-authorization) server configuration options. All options must be set in order to enable OpenFGA. Though, you do not have to create the authorization-model yourself, Incus will generate it including the initial tuple to allow only authenticated users: `server:incus#authenticated@user:*`. ```{warning} Setting the `authorization.openfga.*` options only makes OpenFGA available. It does not on its own cause any request to be authorized by it. OpenFGA is only consulted for clients whose class is routed to it with an `authorization.client.*` option, see {ref}`authorization-client-routing`. ``` (openfga-model)= ### OpenFGA model With OpenFGA, access to a particular API resource is determined by the user's relationship to it. These relationships are determined by an [OpenFGA authorization model](https://openfga.dev/docs/concepts#what-is-an-authorization-model). The Incus OpenFGA authorization model describes API resources in terms of their relationship to other resources, and a relationship a user or group might have with that resource. The full Incus OpenFGA authorization model is defined in `internal/server/auth/driver_openfga_model.openfga`: ```{literalinclude} ../internal/server/auth/driver_openfga_model.openfga :language: none ``` (openfga-security-tags)= ### Security tags Instances can be tagged through the `security.tags` configuration key, which holds a comma-separated list of tags. Incus does not use those tags in its own authorization decisions. Instead, every tag is exposed in OpenFGA as a `security_tag` object related to the server and holding a `tag` relation with every instance carrying it (for example, `security_tag:pci tag instance:default/web1`). This makes the tags visible to external tooling working directly with the OpenFGA store. ```{important} Users that you do not trust with root access to the host should not be granted the following relations: - `server -> admin` - `server -> operator` - `server -> can_edit` - `server -> can_create_storage_pools` - `server -> can_create_projects` - `server -> can_create_certificates` - `certificate -> can_edit` - `storage_pool -> can_edit` - `project -> admin` The remaining relations may be granted. However, you must apply appropriate {ref}`project-restrictions`. ``` (authorization-scriptlet)= ## Scriptlet authorization Incus supports defining a scriptlet to manage fine-grained authorization, allowing to write precise authorization rules with no dependency on external tools. ```{warning} Setting `authorization.scriptlet` only makes the scriptlet available. It does not on its own cause any request to be authorized by it. The scriptlet is only run for clients whose class is routed to it with an `authorization.client.*` option, see {ref}`authorization-client-routing`. ``` To use scriptlet authorization, you can write a scriptlet in the `authorization.scriptlet` server configuration option implementing a function `authorize`, which takes three arguments: - `details`, an object with the following attributes: - `Username`: the user name or certificate fingerprint - `Protocol`: the authentication protocol - `IsAllProjectsRequest`: whether the request is made on all projects - `ProjectName`: the project name - `Chain`: the certificate chain as a list of dissected x509 certificates - `Certificate`: the certificate data stored in the database - `Claims`: the validated OIDC token claims as a dictionary (empty for non-OIDC clients) - `URL`: the raw client request URL (`?`) - `Path`: the API path requested by the client - `Query`: the dictionary of query parameters (note: the values are lists of strings and not single strings) - `Method`: the HTTP method used in the request - `object`, the object on which the user requests authorization - `entitlement`, the authorization level asked by the user This function must return a Boolean indicating whether the user has access or not to the given object with the given entitlement. Additionally, two optional functions can be defined so that users can be listed through the access API: - `get_instance_access`, with two arguments (`project_name` and `instance_name`), returning a list of users able to access a given instance - `get_project_access`, with one argument (`project_name`), returning a list of users able to access a given project (authorization-client-routing)= ## Client routing More than one authorization method can be loaded at the same time, with each request routed to exactly one of them based on the authentication class of the client. This is controlled through the `authorization.client.*` server configuration options, one per client class: - `authorization.client.unix`: local clients connecting over the Unix socket - `authorization.client.tls`: clients using an unrestricted client certificate - `authorization.client.tls-restricted`: clients using a restricted (project-scoped) client certificate - `authorization.client.oidc`: OIDC-authenticated clients - `authorization.client.default`: any client class not set above Each option accepts one of the following values: - `allow`: unconditionally grant access - `deny`: unconditionally refuse access - `tls`: use {ref}`authorization-tls`, only valid for `authorization.client.tls-restricted` - `openfga`: use {ref}`authorization-openfga` - `scriptlet`: use {ref}`authorization-scriptlet` A per-class option falls back to `authorization.client.default` when unset. When that is unset too, the following built-in routing applies: | Client class | Built-in route | |------------------|----------------| | `unix` | `allow` | | `tls` | `allow` | | `tls-restricted` | `tls` | | `oidc` | `allow` | | `default` | `deny` | This routing is fixed. ```{warning} The per-certificate project restrictions described in {ref}`authorization-tls` are enforced by the TLS authorization method only. If `authorization.client.tls-restricted` is set to any value other than `tls`, the project list of the client certificate are no longer consulted, and a restricted certificate may gain access well beyond the projects it is limited to. Only set this option to something other than `tls` if the method you route to reproduces the restrictions you rely on. ``` ```{warning} Always set `authorization.client.default` last, once you have confirmed that every explicitly configured client class behaves as expected. The default applies to every class that is not set explicitly, so a single mistake affects all of them at once. If it is set to `openfga` or `scriptlet` while that method is misconfigured or unreachable, or to `deny`, every remote client is refused. The `root` user is always allowed to interact with the API over the Unix socket, regardless of the configured authorization method. This is what allows a server locked out by a bad routing configuration to be recovered. ``` For example, to have OpenFGA authorize OIDC clients while restricted TLS clients keep using TLS authorization: incus config set authorization.client.oidc=openfga incus config set authorization.client.tls-restricted=scriptlet --- ### Doc Backup (`doc/backup.md`) (backups)= # How to back up an Incus server In a production setup, you should always back up the contents of your Incus server. The Incus server contains a variety of different entities, and when choosing your backup strategy, you must decide which of these entities you want to back up and how frequently you want to save them. ## What to back up The various contents of your Incus server are located on your file system and, in addition, recorded in the {ref}`Incus database `. Therefore, only backing up the database or only backing up the files on disk does not give you a full functional backup. Your Incus server contains the following entities: - Instances (database records and file systems) - Images (database records, image files, and file systems) - Networks (database records and state files) - Profiles (database records) - Storage volumes (database records and file systems) Consider which of these you need to back up. For example, if you don't use custom images, you don't need to back up your images since they are available on the image server. If you use only the `default` profile, or only the standard `incusbr0` network bridge, you might not need to worry about backing them up, because they can easily be re-created. ## Full backup To create a full backup of all contents of your Incus server, back up the `/var/lib/incus` directory. This directory contains your local storage, the Incus database, and your configuration. It does not contain separate storage devices, however. That means that whether the directory also contains the data of your instances depends on the storage drivers that you use. ```{important} If your Incus server uses any external storage (for example, LVM volume groups, ZFS zpools, or any other resource that isn't directly self-contained to Incus), you must back this up separately. ``` To back up your data, create a tarball of `/var/lib/incus`. If your system uses `/etc/subuid` and `/etc/subgid` file, you should also back up these files. Restoring them avoids needless shifting of instance file systems. To restore your data, complete the following steps: 1. Stop Incus on your server (for example, with `sudo systemctl stop incus.service incus.socket`). 1. Delete the directory (`/var/lib/incus/`). 1. Restore the directory from the backup. 1. Delete and restore any external storage devices. 1. Restore the `/etc/subuid` and `/etc/subgid` files if present. 1. Restart Incus (for example, with `sudo systemctl start incus.socket incus.service` or by restarting your machine). ## Partial backup If you decide to only back up specific entities, you have different options for how to do this. You should consider doing some of these partial backups even if you are doing full backups in addition. It can be easier and safer to, for example, restore a single instance or reconfigure a profile than to restore the full Incus server. ### Back up instances and volumes Instances and storage volumes are backed up in a very similar way (because when backing up an instance, you basically back up its instance volume, see {ref}`storage-volume-types`). See {ref}`instances-backup` and {ref}`howto-storage-backup-volume` for detailed information. The following sections give a brief summary of the options you have for backing up instances and volumes. #### Secondary backup Incus server Incus supports copying and moving instances and storage volumes between two hosts. See {ref}`move-instances` and {ref}`howto-storage-move-volume` for instructions. So if you have a spare server, you can regularly copy your instances and storage volumes to that secondary server to back them up. If needed, you can either switch over to the secondary server or copy your instances or storage volumes back from it. If you use the secondary server as a pure storage server, it doesn't need to be as powerful as your main Incus server. #### Export tarballs You can use the `export` command to export instances and volumes to a backup tarball. By default, those tarballs include all snapshots. You can use an optimized export option, which is usually quicker and results in a smaller size of the tarball. However, you must then use the same storage driver when restoring the backup tarball. See {ref}`instances-backup-export` and {ref}`storage-backup-export` for instructions. #### Snapshots Snapshots save the state of an instance or volume at a specific point in time. However, they are stored in the same storage pool and are therefore likely to be lost if the original data is deleted or lost. This means that while snapshots are very quick and easy to create and restore, they don't constitute a secure backup. See {ref}`instances-snapshots` and {ref}`storage-backup-snapshots` for more information. (backup-database)= ### Back up the database While there is no trivial method to restore the contents of the {ref}`Incus database `, it can still be very convenient to keep a backup of its content. Such a backup can make it much easier to re-create, for example, networks or profiles if the need arises. Use the following command to dump the content of the local database to a file: ```bash incus admin sql local .dump > ``` Use the following command to dump the content of the global database to a file: ```bash incus admin sql global .dump > ``` You should include these two commands in your regular Incus backup. --- ### Doc Client Config (`doc/client-config.md`) (client-config)= # CLI configuration file The Incus CLI client uses a configuration file to store environment data and modify its behavior. This documentation describes the file structure and its options. ```{note} Changing this configuration file manually may leave the CLI in an inconsistent state, so you should only edit it via CLI commands such as [`incus remote`](incus_remote.md) and [`incus alias`](incus_alias.md). ``` ## The configuration file By default, the Incus CLI will use the `$HOME/.config/incus/config.yml`, but it can also use the `config.yml` in an arbitrary directory provided by the `INCUS_CONF` environment variable. If the file or its path don't exist, it will use the default: ```yaml default-remote: local remotes: images: protocol: simplestreams public: true addr: https://images.linuxcontainers.org local: protocol: incus public: false addr: unix:// aliases: {} defaults: list_format: "" console_type: "" console_spice_command: "" no_color: false ``` ## `remotes` Incus remote information managed by [`incus remote`](incus_remote.md) commands is stored in the `remotes` section. Remotes may have the following configuration keys. ### `addr` Address of the remote server. Accepts IPs, FQDNs and URLs. Multiple addresses can be provided, separated with commas `addr1,addr2,...`. ### `last_working_address` Used by the CLI to register the last working address of a remote, if the remote has multiple addresses. ### `auth_type` Method that the client will use to authenticate to the server API. Accepts `tls` and `oidc`. ### `keepalive` Timeout in seconds that the `keepalive` feature will maintain the connection opened. ### `project` Default project used by the CLI when using that remote. It's defined by the [`incus project switch`](incus_project_switch.md) command. It's overwritten by the `--project` flag and the `INCUS_PROJECT` environment variable. ### `protocol` The protocol the CLI will use to communicate with the remote. This defines if the remote will be used as an Incus or image server. Accepts the following options: - `incus`: private Incus server accessible over the network - `oci`: [Open Container Initiative (OCI)](https://opencontainers.org/) server that provides application container images - `public`: Incus server accessible over the network to provide images only - `simplestream`: [Simple Stream](https://git.launchpad.net/simplestreams/tree/) server that provide images ### `credentials_helper` Defines a helper command that will handle OCI service authentication. ### `public` Defines if the remote is an image-only server (public). ## `default_remotes` Defines the default Incus remote used by the CLI to execute commands. It's defined by the [`incus remote switch`](incus_remote_switch.md) command. This configuration is overwritten by the environment variable `INCUS_REMOTE` or by the `:` prefix. ## `aliases` Stores the alias managed by the [`incus alias`](incus_alias.md) command. Each alias is a key/value pair that consists of the alias name and the alias command (enclosed by quotes). ## `defaults` Determines the default CLI behavior for certain CLI commands. ### `list_format` Defines how list outputs will be displayed. Equivalent to the `--format` (`-f`) flag. Accepts `csv`, `json`, `table`, `yaml`, `compact` and `markdown` formats. ### `console_type` Defines the default console type used by the [`incus console`](incus_console.md) command. Available options are: - `console`: text based console - `vga`: graphic UI console ### `console_spice_command` Defines an alternative SPICE command to provide the VGA console used by [`incus console`](incus_console.md). ### `no_color` Disables CLI colors. --- ### Doc Client (`doc/client.md`) (incus-client)= # Incus client ```{toctree} :maxdepth: 1 Add remote servers Add command aliases CLI configuration file /reference/manpages ``` --- ### Doc Cloud Init (`doc/cloud-init.md`) --- relatedlinks: https://cloudinit.readthedocs.org/ --- (cloud-init)= # How to use `cloud-init` [`cloud-init`](https://cloud-init.io/) is a tool for automatically initializing and customizing an instance of a Linux distribution. By adding `cloud-init` configuration to your instance, you can instruct `cloud-init` to execute specific actions at the first start of an instance. Possible actions include, for example: * Updating and installing packages * Applying certain configurations * Adding users * Enabling services * Running commands or scripts * Automatically growing the file system of a VM to the size of the disk See the {ref}`cloud-init:index` for detailed information. ```{note} The `cloud-init` actions are run only once on the first start of the instance. Rebooting the instance does not re-trigger the actions. ``` ## `cloud-init` support in images To use `cloud-init`, you must base your instance on an image that has `cloud-init` installed. Images from the [`images` remote](https://images.linuxcontainers.org/) have `cloud-init`-enabled variants, which are usually bigger in size than the default variant. The cloud variants use the `/cloud` suffix, for example, `images:debian/12/cloud`. ## `cloud-init` and virtual machines For `cloud-init` to work inside of a virtual machine, you need to either have a functional `incus-agent` in the VM or need to provide the `cloud-init` data through a special extra disk. All images coming from the `images:` remote will have the agent already setup and so are good to go from the start. For instances that do not have the `incus-agent`, you can pass in the extra `cloud-init` disk with: incus config device add INSTANCE cloud-init disk source=cloud-init:config ```{note} When relying on the `incus-agent` to apply the `cloud-init` configuration, the instance needs to have a set of file templates in place for: * `/var/lib/cloud/seed/nocloud-net/meta-data` * `/var/lib/cloud/seed/nocloud-net/network-config` * `/var/lib/cloud/seed/nocloud-net/user-data` * `/var/lib/cloud/seed/nocloud-net/vendor-data` When making a new image, it's easiest to replicate those from one of our own `cloud-init` enabled images. Those file templates get applied by the `incus-agent` on first boot. The agent then triggers a reboot, causing `cloud-init` to run cleanly and apply the configuration to the virtual machine. ``` ## Configuration options Incus supports two different sets of configuration options for configuring `cloud-init`: `cloud-init.*` and `user.*`. Which of these sets you must use depends on the `cloud-init` support in the image that you use. As a rule of thumb, newer images support the `cloud-init.*` configuration options, while older images support `user.*`. However, there might be exceptions to that rule. The following configuration options are supported: * `cloud-init.vendor-data` or `user.vendor-data` (see {ref}`cloud-init:vendor-data`) * `cloud-init.user-data` or `user.user-data` (see {ref}`cloud-init:user_data_formats`) * `cloud-init.network-config` or `user.network-config` (see {ref}`cloud-init:network_config`) For more information about the configuration options, see the [`cloud-init` instance options](instance-options-cloud-init). ### Vendor data and user data Both `vendor-data` and `user-data` are used to provide cloud configuration data to `cloud-init`. The main idea is that `vendor-data` is used for the general default configuration, while `user-data` is used for instance-specific configuration. This means that you should specify `vendor-data` in a profile and `user-data` in the instance configuration. Incus does not enforce this method, but allows using both `vendor-data` and `user-data` in profiles and in the instance configuration. If both `vendor-data` and `user-data` are supplied for an instance, `cloud-init` merges the two configurations. However, if you use the same keys in both configurations, merging might not be possible. In this case, configure how `cloud-init` should merge the provided data. See the {ref}`cloud-init documentation ` for instructions. ## How to configure `cloud-init` To configure `cloud-init` for an instance, add the corresponding configuration options to a {ref}`profile ` that the instance uses or directly to the {ref}`instance configuration `. When configuring `cloud-init` directly for an instance, keep in mind that `cloud-init` runs only on the first start of the instance. That means that you must configure `cloud-init` before you start the instance. To do so, create the instance with [`incus init`](incus_create.md) instead of [`incus launch`](incus_launch.md), and then start it after completing the configuration. ### YAML format for `cloud-init` configuration The `cloud-init` options require YAML's [literal style format](https://spec.yaml.io/main/spec/1.2.2/#812-literal-style). You use a pipe symbol (`|`) to indicate that all indented text after the pipe should be passed to `cloud-init` as a single string, with new lines and indentation preserved. The `vendor-data` and `user-data` options usually start with `#cloud-config`. For example: ```yaml config: cloud-init.user-data: | #cloud-config package_upgrade: true packages: - package1 - package2 ``` ```{tip} See {ref}`How to validate user data ` for information on how to check whether the syntax is correct. ``` ## How to check the `cloud-init` status `cloud-init` runs automatically on the first start of an instance. Depending on the configured actions, it might take a while until it finishes. To check the `cloud-init` status, log on to the instance and enter the following command: cloud-init status If the result is `status: running`, `cloud-init` is still working. If the result is `status: done`, it has finished. Alternatively, use the `--wait` flag to be notified only when `cloud-init` is finished: ```{terminal} :input: cloud-init status --wait :user: root :host: instance ..................................... status: done ``` ## How to specify user or vendor data The `user-data` and `vendor-data` configuration can be used to, for example, upgrade or install packages, add users, or run commands. The provided values must have a first line that indicates what type of {ref}`user data format ` is being passed to `cloud-init`. For activities like upgrading packages or setting up a user, `#cloud-config` is the data format to use. The configuration data is stored in the following files in the instance's root file system: * `/var/lib/cloud/instance/cloud-config.txt` * `/var/lib/cloud/instance/user-data.txt` ### Examples See the following sections for the user data (or vendor data) configuration for different example use cases. You can find more advanced {ref}`examples ` in the `cloud-init` documentation. #### Upgrade packages To trigger a package upgrade from the repositories for the instance right after the instance is created, use the `package_upgrade` key: ```yaml config: cloud-init.user-data: | #cloud-config package_upgrade: true ``` #### Install packages To install specific packages when the instance is set up, use the `packages` key and specify the package names as a list: ```yaml config: cloud-init.user-data: | #cloud-config packages: - git - openssh-server ``` #### Set the time zone To set the time zone for the instance on instance creation, use the `timezone` key: ```yaml config: cloud-init.user-data: | #cloud-config timezone: Europe/Rome ``` #### Run commands To run a command (such as writing a marker file), use the `runcmd` key and specify the commands as a list: ```yaml config: cloud-init.user-data: | #cloud-config runcmd: - [touch, /run/cloud.init.ran] ``` #### Add a user account To add a user account, use the `users` key. See the {ref}`cloud-init:reference/examples:including users and groups` example in the `cloud-init` documentation for details about default users and which keys are supported. ```yaml config: cloud-init.user-data: | #cloud-config users: - name: documentation_example ``` ## How to specify network configuration data By default, `cloud-init` configures a DHCP client on an instance's `eth0` interface. You can define your own network configuration using the `network-config` option to override the default configuration (this is due to how the template is structured). `cloud-init` then renders the relevant network configuration on the system using either `ifupdown` or `netplan`, depending on the distribution. The configuration data is stored in the following files in the instance's root file system: * `/var/lib/cloud/seed/nocloud-net/network-config` * `/etc/network/interfaces.d/50-cloud-init.cfg` (if using `ifupdown`) * `/etc/netplan/50-cloud-init.yaml` (if using `netplan`) ### Example To configure a specific network interface with a static IPv4 address and also use a custom name server, use the following configuration: ```yaml config: cloud-init.network-config: | version: 2 ethernets: eth1: addresses: - 10.10.101.20/24 gateway4: 10.10.101.1 nameservers: addresses: - 10.10.10.254 ``` --- ### Doc Clustering (`doc/clustering.md`) (clustering)= # Clustering ```{toctree} :titlesonly: explanation/clustering.md Form a cluster Access a cluster Manage a cluster Recover a cluster Manage cluster groups Manage instances Configure storage Configure networks reference/cluster_member_config ``` --- ### Doc Config Options (`doc/config_options.txt`) // Code generated by generate-config from the incus project; DO NOT EDIT. ```{config:option} scheduler.instance cluster-cluster :defaultdesc: "`all`" :shortdesc: "Controls how instances are scheduled to run on this member" :type: "string" Possible values are `all`, `manual`, and `group`. See {ref}`clustering-instance-placement` for more information. ``` ```{config:option} user.* cluster-cluster :shortdesc: "Free form user key/value storage" :type: "string" User keys can be used in search. ``` ```{config:option} instances.vm.cpu.ARCHITECTURE.baseline cluster_group-common :shortdesc: "CPU base architecture name" :type: "string" The CPU base architecture name as can be found through `qemu -cpu ?`. This can be a generic definition like `qemu64` or `kvm64`, or it can be a specific hardware architecture like `EPYC-v2`. It's important to ensure that all servers in the group match that baseline. ``` ```{config:option} instances.vm.cpu.ARCHITECTURE.flags cluster_group-common :shortdesc: "CPU flags to add/remove to/from the baseline" :type: "string" A comma separated list of CPU flags to add on top of CPU baseline or a list of flags to remove from it. To remove a flag, use `-flag`. ``` ```{config:option} user.* cluster_group-common :shortdesc: "Free form user key/value storage" :type: "string" User keys can be used in search. ``` ```{config:option} attached devices-disk :default: "`true`" :required: "no" :shortdesc: "Only for VMs: Whether the disk is attached or ejected" :type: "bool" ``` ```{config:option} boot.priority devices-disk :default: "`0`" :required: "no" :shortdesc: "Boot priority for VMs (higher value boots first), `skip` to skip" :type: "string" ``` ```{config:option} ceph.cluster_name devices-disk :default: "`ceph`" :required: "no" :shortdesc: "The cluster name of the Ceph cluster (required for Ceph or CephFS sources)" :type: "string" ``` ```{config:option} ceph.user_name devices-disk :default: "`admin`" :required: "no" :shortdesc: "The user name of the Ceph cluster (required for Ceph or CephFS sources)" :type: "string" ``` ```{config:option} dependent devices-disk :default: "`false`" :required: "no" :shortdesc: "Specifies if the disk is instance dependent" :type: "bool" ``` ```{config:option} initial.* devices-disk :required: "no" :shortdesc: "Initial volume configuration for instance root disk devices" :type: "string" For root disk devices, this is used to override the storage pool's default volume configuration when creating the instance's root volume. For custom volumes, only `initial.uid`, `initial.gid`, `initial.mode` and `initial.copy` are accepted. The first three are used when auto-creating sub-directories inside the custom volume (when the `source` includes a sub-path that doesn't exist). `initial.copy` copies the container's existing content at `path` into the volume the first time it's used, if the volume is empty. `initial.uid`, `initial.gid` and `initial.mode` are also used to set the ownership and mode of the file system when the `source` is `tmpfs:` or `tmpfs-overlay:`. ``` ```{config:option} io.bus devices-disk :default: "`virtio-scsi` for block, `auto` for file system" :required: "no" :shortdesc: "Only for VMs: Override the bus for the device" :type: "string" This controls what bus a disk device should be attached to. For block devices (disks), this is one of: - `nvme` - `virtio-blk` - `virtio-scsi` (default) - `usb` For file systems (shared directories or custom volumes), this is one of: - `9p` - `auto` (default) (`virtiofs` if possible, else `9p`) - `virtiofs` `9p` doesn't support hotplugging and `virtiofs` doesn't support live migration. `auto` tries to use `virtiofs` if possible (`migration.stateful` not set to `true` and host support for `virtiofsd`) and falls back to `9p` otherwise. ``` ```{config:option} io.cache devices-disk :default: "`none`" :required: "no" :shortdesc: "Only for VMs: Override the caching mode for the device" :type: "string" This controls what bus a disk device should be attached to. For block devices (disks), this is one of: - `none` (default) - `writeback` - `unsafe` For file systems (shared directories or custom volumes), this is one of: - `none` (default) - `metadata` - `unsafe` ``` ```{config:option} limits.max devices-disk :required: "no" :shortdesc: "I/O limit in byte/s and/or IOPS for both read and write (same as setting both `limits.read` and `limits.write`)" :type: "string" ``` ```{config:option} limits.max.burst devices-disk :required: "no" :shortdesc: "I/O limit in byte/s and/or IOPS that both reads and writes may burst up to (same as setting both `limits.read.burst` and `limits.write.burst`)" :type: "string" ``` ```{config:option} limits.max.burst.length devices-disk :default: "`1s`" :required: "no" :shortdesc: "How long the burst limits may be sustained for (same as setting both `limits.read.burst.length` and `limits.write.burst.length`)" :type: "string" ``` ```{config:option} limits.read devices-disk :required: "no" :shortdesc: "I/O limit in byte/s (various suffixes supported, see {ref}`instances-limit-units`) and/or in IOPS (must be suffixed with `iops`), comma separated when specifying both - see also {ref}`storage-configure-IO`" :type: "string" ``` ```{config:option} limits.read.burst devices-disk :required: "no" :shortdesc: "I/O limit in byte/s and/or IOPS that reads may burst up to for `limits.read.burst.length` (only for virtual machines) - see also {ref}`storage-configure-IO`" :type: "string" ``` ```{config:option} limits.read.burst.length devices-disk :default: "`1s`" :required: "no" :shortdesc: "How long `limits.read.burst` may be sustained for before falling back to the sustained limit" :type: "string" ``` ```{config:option} limits.write devices-disk :required: "no" :shortdesc: "I/O limit in byte/s (various suffixes supported, see {ref}`instances-limit-units`) and/or in IOPS (must be suffixed with `iops`), comma separated when specifying both - see also {ref}`storage-configure-IO`" :type: "string" ``` ```{config:option} limits.write.burst devices-disk :required: "no" :shortdesc: "I/O limit in byte/s and/or IOPS that writes may burst up to for `limits.write.burst.length` (only for virtual machines) - see also {ref}`storage-configure-IO`" :type: "string" ``` ```{config:option} limits.write.burst.length devices-disk :default: "`1s`" :required: "no" :shortdesc: "How long `limits.write.burst` may be sustained for before falling back to the sustained limit" :type: "string" ``` ```{config:option} path devices-disk :required: "yes" :shortdesc: "Path inside the instance where the disk will be mounted (only for file system disk devices)" :type: "string" This controls which path inside the instance the disk should be mounted on. With containers, this option supports mounting file system disk devices, and paths and single files within them. With VMs, this option supports mounting file system disk devices and paths within them. Mounting single files is not supported. ``` ```{config:option} pool devices-disk :required: "no" :shortdesc: "The storage pool to which the disk device belongs (only applicable for storage volumes managed by Incus)" :type: "string" ``` ```{config:option} propagation devices-disk :required: "no" :shortdesc: "Controls how a bind-mount is shared between the instance and the host (can be one of `private`, the default, or `shared`, `slave`, `unbindable`, `rshared`, `rslave`, `runbindable`, `rprivate`; see the Linux Kernel [shared subtree](https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt) documentation for a full explanation)" :type: "string" ``` ```{config:option} raw.mount.options devices-disk :required: "no" :shortdesc: "File system specific mount options" :type: "string" ``` ```{config:option} readonly devices-disk :default: "`false`" :required: "no" :shortdesc: "Controls whether to make the mount read-only" :type: "bool" ``` ```{config:option} recursive devices-disk :default: "`false`" :required: "no" :shortdesc: "Controls whether to recursively mount the source path" :type: "bool" ``` ```{config:option} required devices-disk :default: "`true`" :required: "no" :shortdesc: "Controls whether to fail if the source doesn't exist" :type: "bool" ``` ```{config:option} shift devices-disk :default: "`false`" :required: "no" :shortdesc: "Sets up a shifting overlay to translate the source UID/GID to match the instance (only for containers)" :type: "bool" ``` ```{config:option} size devices-disk :required: "no" :shortdesc: "Disk size in bytes (various suffixes supported, see {ref}`instances-limit-units`) - only supported for the `rootfs` (`/`)" :type: "string" ``` ```{config:option} size.state devices-disk :required: "no" :shortdesc: "Same as `size`, but applies to the file-system volume used for saving runtime state in VMs" :type: "string" ``` ```{config:option} source devices-disk :required: "yes" :shortdesc: "Source of a file system or block device (see {ref}`devices-disk-types` for details)" :type: "string" ``` ```{config:option} wwn devices-disk :default: "``" :required: "no" :shortdesc: "Only for VMs: Set the disk World Wide Name (only supported on `virtio-scsi` bus)" :type: "bool" ``` ```{config:option} id devices-gpu_mdev :required: "no" :shortdesc: "The DRM card ID of the GPU device" :type: "string" ``` ```{config:option} mdev devices-gpu_mdev :required: "yes" :shortdesc: "The mediated device profile to use (required - for example, `i915-GVTg_V5_4`)" :type: "string" ``` ```{config:option} productid devices-gpu_mdev :required: "no" :shortdesc: "The product ID of the GPU device" :type: "string" ``` ```{config:option} vendorid devices-gpu_mdev :required: "no" :shortdesc: "The vendor ID of the GPU device" :type: "string" ``` ```{config:option} id devices-gpu_mig :required: "no" :shortdesc: "The DRM card ID of the GPU device" :type: "string" ``` ```{config:option} mig.ci devices-gpu_mig :required: "no" :shortdesc: "Existing MIG compute instance ID" :type: "int" ``` ```{config:option} mig.gi devices-gpu_mig :required: "no" :shortdesc: "Existing MIG GPU instance ID" :type: "int" ``` ```{config:option} mig.uuid devices-gpu_mig :required: "no" :shortdesc: "Existing MIG device UUID (MIG- prefix can be omitted)" :type: "string" ``` ```{config:option} pci devices-gpu_mig :required: "no" :shortdesc: "The PCI address of the GPU device" :type: "string" ``` ```{config:option} productid devices-gpu_mig :required: "no" :shortdesc: "The product ID of the GPU device" :type: "string" ``` ```{config:option} vendorid devices-gpu_mig :required: "no" :shortdesc: "The vendor ID of the GPU device" :type: "string" ``` ```{config:option} blob.size devices-gpu_native_context :default: "2GiB" :required: "no" :shortdesc: "Size of the host-visible blob memory window for the `virtio-gpu` device" :type: "string" ``` ```{config:option} id devices-gpu_native_context :required: "no" :shortdesc: "The DRM card ID of the GPU device" :type: "string" ``` ```{config:option} pci devices-gpu_native_context :required: "no" :shortdesc: "The PCI address of the GPU device" :type: "string" ``` ```{config:option} productid devices-gpu_native_context :required: "no" :shortdesc: "The product ID of the GPU device" :type: "string" ``` ```{config:option} vendorid devices-gpu_native_context :required: "no" :shortdesc: "The vendor ID of the GPU device" :type: "string" ``` ```{config:option} gid devices-gpu_physical :default: "0" :required: "no" :shortdesc: "GID of the device owner in the instance (container only)" :type: "int" ``` ```{config:option} id devices-gpu_physical :required: "no" :shortdesc: "The DRM card ID of the GPU device" :type: "string" ``` ```{config:option} mode devices-gpu_physical :default: "0660" :required: "no" :shortdesc: "Mode of the device in the instance (container only)" :type: "int" ``` ```{config:option} nvidia.clique devices-gpu_physical :required: "no" :shortdesc: "NVIDIA GPUDirect P2P clique ID (0 to 15) shared by GPUs allowed to DMA to each other (VM only)" :type: "int" ``` ```{config:option} pci devices-gpu_physical :required: "no" :shortdesc: "The PCI address of the GPU device" :type: "string" ``` ```{config:option} productid devices-gpu_physical :required: "no" :shortdesc: "The product ID of the GPU device" :type: "string" ``` ```{config:option} uid devices-gpu_physical :default: "0" :required: "no" :shortdesc: "UID of the device owner in the instance (container only)" :type: "int" ``` ```{config:option} vendorid devices-gpu_physical :required: "no" :shortdesc: "The vendor ID of the GPU device" :type: "string" ``` ```{config:option} id devices-gpu_sriov :required: "no" :shortdesc: "The DRM card ID of the parent GPU device" :type: "string" ``` ```{config:option} pci devices-gpu_sriov :required: "no" :shortdesc: "The PCI address of the parent GPU device" :type: "string" ``` ```{config:option} productid devices-gpu_sriov :required: "no" :shortdesc: "The product ID of the parent GPU device" :type: "string" ``` ```{config:option} vendorid devices-gpu_sriov :required: "no" :shortdesc: "The vendor ID of the parent GPU device" :type: "string" ``` ```{config:option} hwaddr devices-infiniband :defaultdesc: "randomly assigned" :required: "no" :shortdesc: "The MAC address of the new interface (can be either the full 20-byte variant or the short 8-byte variant, which will only modify the last 8 bytes of the parent device)" :type: "string" ``` ```{config:option} mtu devices-infiniband :defaultdesc: "parent MTU" :required: "no" :shortdesc: "The MTU of the new interface" :type: "integer" ``` ```{config:option} name devices-infiniband :defaultdesc: "kernel assigned" :required: "no" :shortdesc: "The name of the interface inside the instance" :type: "string" ``` ```{config:option} nictype devices-infiniband :required: "yes" :shortdesc: "The device type (one of `physical` or `sriov`)" :type: "string" ``` ```{config:option} node_guid devices-infiniband :defaultdesc: "kernel assigned" :required: "no" :shortdesc: "The node GUID to assign to the virtual function (8 bytes of hex separated by colons)" :type: "string" ``` ```{config:option} parent devices-infiniband :defaultdesc: "kernel assigned" :required: "no" :shortdesc: "The name of the interface inside the instance" :type: "string" ``` ```{config:option} port_guid devices-infiniband :defaultdesc: "kernel assigned" :required: "no" :shortdesc: "The port GUID to assign to the virtual function (8 bytes of hex separated by colons)" :type: "string" ``` ```{config:option} attached devices-nic_bridged :default: "`true`" :required: "no" :shortdesc: "Whether the NIC is plugged in or not" :type: "bool" ``` ```{config:option} boot.priority devices-nic_bridged :default: "`skip`" :managed: "no" :shortdesc: "Boot priority for VMs (higher value boots first), `skip` to skip" :type: "string" ``` ```{config:option} connected devices-nic_bridged :default: "`true`" :required: "no" :shortdesc: "Whether the NIC is connected to the host network" :type: "bool" ``` ```{config:option} host_name devices-nic_bridged :default: "randomly assigned" :managed: "no" :shortdesc: "The name of the interface on the host" :type: "string" ``` ```{config:option} hwaddr devices-nic_bridged :default: "randomly assigned" :managed: "no" :shortdesc: "The MAC address of the new interface" :type: "string" ``` ```{config:option} io.bus devices-nic_bridged :default: "`virtio`" :managed: "no" :shortdesc: "Override the bus for the device (can be `virtio` or `usb`) (VM only)" :type: "string" ``` ```{config:option} ipv4.address devices-nic_bridged :managed: "no" :shortdesc: "An IPv4 address to assign to the instance through DHCP (can be `none` to disable DHCPv4 inside an OCI container and restrict all IPv4 traffic when `security.ipv4_filtering` is set, or a CIDR value to statically configure the address inside an OCI container)" :type: "string" ``` ```{config:option} ipv4.gateway devices-nic_bridged :managed: "no" :shortdesc: "IPv4 default gateway to statically configure inside an OCI container (`none` to prevent a default gateway from being applied)" :type: "string" ``` ```{config:option} ipv4.routes devices-nic_bridged :managed: "no" :shortdesc: "Comma-delimited list of IPv4 static routes to add on host to NIC" :type: "string" ``` ```{config:option} ipv4.routes.external devices-nic_bridged :managed: "no" :shortdesc: "Comma-delimited list of IPv4 static routes to route to the NIC and publish on uplink network (BGP)" :type: "string" ``` ```{config:option} ipv6.address devices-nic_bridged :managed: "no" :shortdesc: "An IPv6 address to assign to the instance through DHCP (can be `none` to disable DHCPv6 and SLAAC inside an OCI container and restrict all IPv6 traffic when `security.ipv6_filtering` is set, or a CIDR value to statically configure the address inside an OCI container)" :type: "string" ``` ```{config:option} ipv6.gateway devices-nic_bridged :managed: "no" :shortdesc: "IPv6 default gateway to statically configure inside an OCI container (`none` to prevent a default gateway from being applied)" :type: "string" ``` ```{config:option} ipv6.routes devices-nic_bridged :managed: "no" :shortdesc: "Comma-delimited list of IPv6 static routes to add on host to NIC" :type: "string" ``` ```{config:option} ipv6.routes.external devices-nic_bridged :managed: "no" :shortdesc: "Comma-delimited list of IPv6 static routes to route to the NIC and publish on uplink network (BGP)" :type: "string" ``` ```{config:option} limits.egress devices-nic_bridged :managed: "no" :shortdesc: "I/O limit in bit/s for outgoing traffic (various suffixes supported, see {ref}`instances-limit-units`)" :type: "string" ``` ```{config:option} limits.egress.bucket devices-nic_bridged :managed: "no" :shortdesc: "Amount of data in bit that outgoing traffic may send in excess of `limits.egress`, requires `limits.egress.burst`" :type: "string" ``` ```{config:option} limits.egress.burst devices-nic_bridged :managed: "no" :shortdesc: "I/O limit in bit/s that outgoing traffic may burst up to, requires `limits.egress.bucket`" :type: "string" ``` ```{config:option} limits.ingress devices-nic_bridged :managed: "no" :shortdesc: "I/O limit in bit/s for incoming traffic (various suffixes supported, see {ref}`instances-limit-units`)" :type: "string" ``` ```{config:option} limits.ingress.bucket devices-nic_bridged :managed: "no" :shortdesc: "Amount of data in bit that incoming traffic may send in excess of `limits.ingress`, requires `limits.ingress.burst`" :type: "string" ``` ```{config:option} limits.ingress.burst devices-nic_bridged :managed: "no" :shortdesc: "I/O limit in bit/s that incoming traffic may burst up to, requires `limits.ingress.bucket`" :type: "string" ``` ```{config:option} limits.max devices-nic_bridged :managed: "no" :shortdesc: "I/O limit in bit/s for both incoming and outgoing traffic (same as setting both limits.ingress and limits.egress)" :type: "string" ``` ```{config:option} limits.max.bucket devices-nic_bridged :managed: "no" :shortdesc: "Amount of data in bit that traffic may send in excess of the sustained limit (same as setting both `limits.ingress.bucket` and `limits.egress.bucket`)" :type: "string" ``` ```{config:option} limits.max.burst devices-nic_bridged :managed: "no" :shortdesc: "I/O limit in bit/s that both incoming and outgoing traffic may burst up to (same as setting both `limits.ingress.burst` and `limits.egress.burst`)" :type: "string" ``` ```{config:option} limits.priority devices-nic_bridged :managed: "no" :shortdesc: "The priority for outgoing traffic, to be used by the kernel queuing discipline to prioritize network packets" :type: "integer" ``` ```{config:option} mtu devices-nic_bridged :default: "MTU of the parent device" :managed: "yes" :shortdesc: "The Maximum Transmit Unit (MTU) of the new interface" :type: "integer" ``` ```{config:option} name devices-nic_bridged :default: "kernel assigned" :managed: "no" :shortdesc: "The name of the interface inside the instance" :type: "string" ``` ```{config:option} network devices-nic_bridged :managed: "no" :shortdesc: "The managed network to link the device to (instead of specifying the `nictype` directly)" :type: "string" ``` ```{config:option} parent devices-nic_bridged :managed: "yes" :shortdesc: "The name of the parent host device (required if specifying the `nictype` directly)" :type: "string" ``` ```{config:option} queue.discipline devices-nic_bridged :managed: "no" :shortdesc: "The queuing discipline to set on the NIC, applied to the rate limit's class when a limit is set" :type: "string" ``` ```{config:option} queue.discipline.attach devices-nic_bridged :default: "`queue`" :managed: "no" :shortdesc: "Only for VMs: Whether to attach the queuing discipline to each transmit queue (`queue`) or to the interface root (`root`)" :type: "string" ``` ```{config:option} queue.tx.length devices-nic_bridged :managed: "no" :shortdesc: "The transmit queue length for the NIC" :type: "integer" ``` ```{config:option} security.acls devices-nic_bridged :managed: "no" :shortdesc: "Comma-separated list of network ACLs to apply" :type: "string" ``` ```{config:option} security.acls.default.egress.action devices-nic_bridged :default: "drop" :managed: "no" :shortdesc: "Action to use for egress traffic that doesn't match any ACL rule" :type: "string" ``` ```{config:option} security.acls.default.egress.logged devices-nic_bridged :default: "false" :managed: "no" :shortdesc: "Whether to log egress traffic that doesn't match any ACL rule" :type: "bool" ``` ```{config:option} security.acls.default.ingress.action devices-nic_bridged :default: "drop" :managed: "no" :shortdesc: "Action to use for ingress traffic that doesn't match any ACL rule" :type: "string" ``` ```{config:option} security.acls.default.ingress.logged devices-nic_bridged :default: "false" :managed: "no" :shortdesc: "Whether to log ingress traffic that doesn't match any ACL rule" :type: "bool" ``` ```{config:option} security.ipv4_filtering devices-nic_bridged :default: "false" :managed: "no" :shortdesc: "Prevent the instance from spoofing another instance's IPv4 address (enables `security.mac_filtering`)" :type: "bool" ``` ```{config:option} security.ipv6_filtering devices-nic_bridged :default: "false" :managed: "no" :shortdesc: "Prevent the instance from spoofing another instance's IPv6 address (enables `security.mac_filtering`)" :type: "bool" ``` ```{config:option} security.mac_filtering devices-nic_bridged :default: "false" :managed: "no" :shortdesc: "Prevent the instance from spoofing another instance's MAC address" :type: "bool" ``` ```{config:option} security.port_isolation devices-nic_bridged :default: "false" :managed: "no" :shortdesc: "Prevent the NIC from communicating with other NICs in the network that have port isolation enabled" :type: "bool" ``` ```{config:option} vlan devices-nic_bridged :managed: "no" :shortdesc: "The VLAN ID to use for non-tagged traffic (can be none to remove port from default VLAN)" :type: "integer" ``` ```{config:option} vlan.tagged devices-nic_bridged :managed: "no" :shortdesc: "Comma-delimited list of VLAN IDs or VLAN ranges to join for tagged traffic" :type: "integer" ``` ```{config:option} attached devices-nic_ipvlan :default: "`true`" :required: "no" :shortdesc: "Whether the NIC is plugged in or not" :type: "bool" ``` ```{config:option} gvrp devices-nic_ipvlan :default: "false" :shortdesc: "Register VLAN using GARP VLAN Registration Protocol" :type: "bool" ``` ```{config:option} hwaddr devices-nic_ipvlan :default: "randomly assigned" :shortdesc: "The MAC address of the new interface" :type: "string" ``` ```{config:option} ipv4.address devices-nic_ipvlan :shortdesc: "Comma-delimited list of IPv4 static addresses to add to the instance (in l2 mode, these can be specified as CIDR values or singular addresses using a subnet of /24)" :type: "string" ``` ```{config:option} ipv4.gateway devices-nic_ipvlan :default: "`auto` (in `l3s` mode), `-` (in `l2` mode)" :shortdesc: "In `l3s` mode, whether to add an automatic default IPv4 gateway (can be `auto` or `none`). In `l2` mode, the IPv4 address of the gateway" :type: "string" ``` ```{config:option} ipv4.host_table devices-nic_ipvlan :shortdesc: "The custom policy routing table ID to add IPv4 static routes to (in addition to the main routing table)" :type: "integer" ``` ```{config:option} ipv6.address devices-nic_ipvlan :shortdesc: "Comma-delimited list of IPv6 static addresses to add to the instance (in `l2` mode, these can be specified as CIDR values or singular addresses using a subnet of /64)" :type: "string" ``` ```{config:option} ipv6.gateway devices-nic_ipvlan :default: "`auto` (in `l3s` mode), `-` (in `l2` mode)" :shortdesc: "In `l3s` mode, whether to add an automatic default IPv6 gateway (can be `auto` or `none`). In `l2` mode, the IPv6 address of the gateway" :type: "string" ``` ```{config:option} ipv6.host_table devices-nic_ipvlan :shortdesc: "The custom policy routing table ID to add IPv6 static routes to (in addition to the main routing table)" :type: "integer" ``` ```{config:option} mode devices-nic_ipvlan :default: "`l3s`" :shortdesc: "The IPVLAN mode (either `l2` or `l3s`)" :type: "string" ``` ```{config:option} mtu devices-nic_ipvlan :default: "MTU of the parent device" :shortdesc: "The Maximum Transmit Unit (MTU) of the new interface" :type: "integer" ``` ```{config:option} name devices-nic_ipvlan :default: "kernel assigned" :shortdesc: "The name of the interface inside the instance" :type: "string" ``` ```{config:option} parent devices-nic_ipvlan :shortdesc: "The name of the host device (required)" :type: "string" ``` ```{config:option} vlan devices-nic_ipvlan :shortdesc: "The VLAN ID to attach to" :type: "integer" ``` ```{config:option} attached devices-nic_macvlan :default: "`true`" :required: "no" :shortdesc: "Whether the NIC is plugged in or not" :type: "bool" ``` ```{config:option} boot.priority devices-nic_macvlan :default: "`skip`" :managed: "no" :shortdesc: "Boot priority for VMs (higher value boots first), `skip` to skip" :type: "string" ``` ```{config:option} connected devices-nic_macvlan :default: "`true`" :required: "no" :shortdesc: "Whether the NIC is connected to the host network (VM only)" :type: "bool" ``` ```{config:option} gvrp devices-nic_macvlan :default: "false" :managed: "no" :shortdesc: "Register VLAN using GARP VLAN Registration Protocol" :type: "bool" ``` ```{config:option} hwaddr devices-nic_macvlan :default: "randomly assigned" :managed: "no" :shortdesc: "The MAC address of the new interface" :type: "string" ``` ```{config:option} io.bus devices-nic_macvlan :default: "`virtio`" :managed: "no" :shortdesc: "Override the bus for the device (can be `virtio` or `usb`) (VM only)" :type: "string" ``` ```{config:option} mode devices-nic_macvlan :default: "bridge" :managed: "no" :shortdesc: "Macvlan mode (one of `bridge`, `vepa`, `passthru` or `private`)" :type: "string" ``` ```{config:option} mtu devices-nic_macvlan :default: "MTU of the parent device" :managed: "yes" :shortdesc: "The Maximum Transmit Unit (MTU) of the new interface" :type: "integer" ``` ```{config:option} name devices-nic_macvlan :default: "kernel assigned" :managed: "no" :shortdesc: "The name of the interface inside the instance" :type: "string" ``` ```{config:option} network devices-nic_macvlan :managed: "no" :shortdesc: "The managed network to link the device to (instead of specifying the `nictype` directly)" :type: "string" ``` ```{config:option} parent devices-nic_macvlan :managed: "yes" :shortdesc: "The name of the parent host device (required if specifying the `nictype` directly)" :type: "string" ``` ```{config:option} vlan devices-nic_macvlan :managed: "no" :shortdesc: "The VLAN ID to attach to" :type: "integer" ``` ```{config:option} acceleration devices-nic_ovn :default: "none" :managed: "no" :shortdesc: "Enable hardware offloading (either `none`, `sriov` or `vdpa`)" :type: "string" ``` ```{config:option} attached devices-nic_ovn :default: "`true`" :required: "no" :shortdesc: "Whether the NIC is plugged in or not" :type: "bool" ``` ```{config:option} boot.priority devices-nic_ovn :default: "`skip`" :managed: "no" :shortdesc: "Boot priority for VMs (higher value boots first), `skip` to skip" :type: "string" ``` ```{config:option} connected devices-nic_ovn :default: "`true`" :required: "no" :shortdesc: "Whether the NIC is connected to the host network (requires `acceleration` set to `none`)" :type: "bool" ``` ```{config:option} host_name devices-nic_ovn :default: "randomly assigned" :managed: "no" :shortdesc: "The name of the interface inside the host" :type: "string" ``` ```{config:option} hwaddr devices-nic_ovn :default: "randomly assigned" :managed: "no" :shortdesc: "The MAC address of the new interface" :type: "string" ``` ```{config:option} io.bus devices-nic_ovn :default: "`virtio`" :managed: "no" :shortdesc: "Override the bus for the device (can be `virtio` or `usb`, requires `acceleration` set to `none`) (VM only)" :type: "string" ``` ```{config:option} ipv4.address devices-nic_ovn :managed: "no" :shortdesc: "An IPv4 address to assign to the instance through DHCP, `none` can be used to disable IP allocation (or a CIDR value to statically configure the address inside an OCI container)" :type: "string" ``` ```{config:option} ipv4.address.external devices-nic_ovn :managed: "no" :shortdesc: "Select a specific external address (typically from a network forward)" :type: "string" ``` ```{config:option} ipv4.gateway devices-nic_ovn :managed: "no" :shortdesc: "IPv4 default gateway to statically configure inside an OCI container (`none` to prevent a default gateway from being applied)" :type: "string" ``` ```{config:option} ipv4.routes devices-nic_ovn :managed: "no" :shortdesc: "Comma-delimited list of IPv4 static routes to route to the NIC" :type: "string" ``` ```{config:option} ipv4.routes.external devices-nic_ovn :managed: "no" :shortdesc: "Comma-delimited list of IPv4 static routes to route to the NIC and publish on uplink network" :type: "string" ``` ```{config:option} ipv6.address devices-nic_ovn :managed: "no" :shortdesc: "An IPv6 address to assign to the instance through DHCP, `none` can be used to disable IP allocation (or a CIDR value to statically configure the address inside an OCI container)" :type: "string" ``` ```{config:option} ipv6.address.external devices-nic_ovn :managed: "no" :shortdesc: "Select a specific external address (typically from a network forward)" :type: "string" ``` ```{config:option} ipv6.gateway devices-nic_ovn :managed: "no" :shortdesc: "IPv6 default gateway to statically configure inside an OCI container (`none` to prevent a default gateway from being applied)" :type: "string" ``` ```{config:option} ipv6.routes devices-nic_ovn :managed: "no" :shortdesc: "Comma-delimited list of IPv6 static routes to route to the NIC" :type: "string" ``` ```{config:option} ipv6.routes.external devices-nic_ovn :managed: "no" :shortdesc: "Comma-delimited list of IPv6 static routes to route to the NIC and publish on uplink network" :type: "string" ``` ```{config:option} limits.egress devices-nic_ovn :managed: "no" :shortdesc: "I/O limit in bit/s for outgoing traffic (various suffixes supported, see {ref}`instances-limit-units`)" :type: "string" ``` ```{config:option} limits.egress.bucket devices-nic_ovn :managed: "no" :shortdesc: "Amount of data in bit that outgoing traffic may send in excess of `limits.egress` (mutually exclusive with `limits.max.bucket`)" :type: "string" ``` ```{config:option} limits.ingress devices-nic_ovn :managed: "no" :shortdesc: "I/O limit in bit/s for incoming traffic (various suffixes supported, see {ref}`instances-limit-units`)" :type: "string" ``` ```{config:option} limits.ingress.bucket devices-nic_ovn :managed: "no" :shortdesc: "Amount of data in bit that incoming traffic may send in excess of `limits.ingress` (mutually exclusive with `limits.max.bucket`)" :type: "string" ``` ```{config:option} limits.max devices-nic_ovn :managed: "no" :shortdesc: "I/O limit in bit/s for both incoming and outgoing traffic. (same as setting both limits.ingress and limits.egress / mutually exclusive with limits.ingress and limits.egress)" :type: "string" ``` ```{config:option} limits.max.bucket devices-nic_ovn :managed: "no" :shortdesc: "Amount of data in bit that traffic may send in excess of the sustained limit (same as setting both `limits.ingress.bucket` and `limits.egress.bucket` / mutually exclusive with them)" :type: "string" ``` ```{config:option} limits.priority devices-nic_ovn :default: "100" :managed: "no" :shortdesc: "The priority for outgoing traffic, to be used by the kernel queuing discipline to prioritize network packets" :type: "integer" ``` ```{config:option} mtu devices-nic_ovn :default: "MTU of the parent network" :managed: "yes" :shortdesc: "The Maximum Transmit Unit (MTU) of the new interface" :type: "integer" ``` ```{config:option} name devices-nic_ovn :default: "kernel assigned" :managed: "no" :shortdesc: "The name of the interface inside the instance" :type: "string" ``` ```{config:option} nested devices-nic_ovn :managed: "no" :shortdesc: "The parent NIC name to nest this NIC under (see also `vlan`)" :type: "string" ``` ```{config:option} network devices-nic_ovn :managed: "yes" :shortdesc: "The managed network to link the device to (required)" :type: "string" ``` ```{config:option} security.acls devices-nic_ovn :managed: "no" :shortdesc: "Comma-separated list of network ACLs to apply" :type: "string" ``` ```{config:option} security.acls.default.egress.action devices-nic_ovn :default: "reject" :managed: "no" :shortdesc: "Action to use for egress traffic that doesn't match any ACL rule" :type: "string" ``` ```{config:option} security.acls.default.egress.logged devices-nic_ovn :default: "false" :managed: "no" :shortdesc: "Whether to log egress traffic that doesn't match any ACL rule" :type: "bool" ``` ```{config:option} security.acls.default.ingress.action devices-nic_ovn :default: "reject" :managed: "no" :shortdesc: "Action to use for ingress traffic that doesn't match any ACL rule" :type: "string" ``` ```{config:option} security.acls.default.ingress.logged devices-nic_ovn :default: "false" :managed: "no" :shortdesc: "Whether to log ingress traffic that doesn't match any ACL rule" :type: "bool" ``` ```{config:option} security.ipv4_filtering devices-nic_ovn :default: "false" :managed: "no" :shortdesc: "Prevent the instance from spoofing another instance's IPv4 address (enables `security.mac_filtering`)" :type: "bool" ``` ```{config:option} security.ipv6_filtering devices-nic_ovn :default: "false" :managed: "no" :shortdesc: "Prevent the instance from spoofing another instance's IPv6 address (enables `security.mac_filtering`)" :type: "bool" ``` ```{config:option} security.mac_filtering devices-nic_ovn :default: "false" :managed: "no" :shortdesc: "Prevent the instance from spoofing another instance's MAC address" :type: "bool" ``` ```{config:option} security.promiscuous devices-nic_ovn :default: "false" :managed: "no" :shortdesc: "Have OVN send unknown network traffic to this network interface (required for some nesting cases)" :type: "bool" ``` ```{config:option} vlan devices-nic_ovn :managed: "no" :shortdesc: "The VLAN ID to use when nesting (see also `nested`)" :type: "integer" ``` ```{config:option} attached devices-nic_p2p :default: "`true`" :required: "no" :shortdesc: "Whether the NIC is plugged in or not" :type: "bool" ``` ```{config:option} boot.priority devices-nic_p2p :default: "`skip`" :shortdesc: "Boot priority for VMs (higher value boots first), `skip` to skip" :type: "string" ``` ```{config:option} connected devices-nic_p2p :default: "`true`" :required: "no" :shortdesc: "Whether the NIC is connected to the host network" :type: "bool" ``` ```{config:option} host_name devices-nic_p2p :default: "randomly assigned" :shortdesc: "The name of the interface on the host" :type: "string" ``` ```{config:option} hwaddr devices-nic_p2p :default: "randomly assigned" :shortdesc: "The MAC address of the new interface" :type: "string" ``` ```{config:option} io.bus devices-nic_p2p :default: "`virtio`" :shortdesc: "Override the bus for the device (can be `virtio` or `usb`) (VM only)" :type: "string" ``` ```{config:option} ipv4.routes devices-nic_p2p :shortdesc: "Comma-delimited list of IPv4 static routes to add on host to NIC" :type: "string" ``` ```{config:option} ipv6.routes devices-nic_p2p :shortdesc: "Comma-delimited list of IPv6 static routes to add on host to NIC" :type: "string" ``` ```{config:option} limits.egress devices-nic_p2p :shortdesc: "I/O limit in bit/s for outgoing traffic (various suffixes supported, see {ref}`instances-limit-units`)" :type: "string" ``` ```{config:option} limits.egress.bucket devices-nic_p2p :managed: "no" :shortdesc: "Amount of data in bit that outgoing traffic may send in excess of `limits.egress`, requires `limits.egress.burst`" :type: "string" ``` ```{config:option} limits.egress.burst devices-nic_p2p :managed: "no" :shortdesc: "I/O limit in bit/s that outgoing traffic may burst up to, requires `limits.egress.bucket`" :type: "string" ``` ```{config:option} limits.ingress devices-nic_p2p :shortdesc: "I/O limit in bit/s for incoming traffic (various suffixes supported, see {ref}`instances-limit-units`)" :type: "string" ``` ```{config:option} limits.ingress.bucket devices-nic_p2p :managed: "no" :shortdesc: "Amount of data in bit that incoming traffic may send in excess of `limits.ingress`, requires `limits.ingress.burst`" :type: "string" ``` ```{config:option} limits.ingress.burst devices-nic_p2p :managed: "no" :shortdesc: "I/O limit in bit/s that incoming traffic may burst up to, requires `limits.ingress.bucket`" :type: "string" ``` ```{config:option} limits.max devices-nic_p2p :shortdesc: "I/O limit in bit/s for both incoming and outgoing traffic (same as setting both limits.ingress and limits.egress)" :type: "string" ``` ```{config:option} limits.max.bucket devices-nic_p2p :managed: "no" :shortdesc: "Amount of data in bit that traffic may send in excess of the sustained limit (same as setting both `limits.ingress.bucket` and `limits.egress.bucket`)" :type: "string" ``` ```{config:option} limits.max.burst devices-nic_p2p :managed: "no" :shortdesc: "I/O limit in bit/s that both incoming and outgoing traffic may burst up to (same as setting both `limits.ingress.burst` and `limits.egress.burst`)" :type: "string" ``` ```{config:option} limits.priority devices-nic_p2p :shortdesc: "The priority for outgoing traffic, to be used by the kernel queuing discipline to prioritize network packets" :type: "integer" ``` ```{config:option} mtu devices-nic_p2p :default: "kernel assigned" :shortdesc: "The Maximum Transmit Unit (MTU) of the new interface" :type: "integer" ``` ```{config:option} name devices-nic_p2p :default: "kernel assigned" :shortdesc: "The name of the interface inside the instance" :type: "string" ``` ```{config:option} queue.discipline devices-nic_p2p :shortdesc: "The queuing discipline to set on the NIC, applied to the rate limit's class when a limit is set" :type: "string" ``` ```{config:option} queue.discipline.attach devices-nic_p2p :default: "`queue`" :shortdesc: "Only for VMs: Whether to attach the queuing discipline to each transmit queue (`queue`) or to the interface root (`root`)" :type: "string" ``` ```{config:option} queue.tx.length devices-nic_p2p :shortdesc: "The transmit queue length for the NIC" :type: "integer" ``` ```{config:option} attached devices-nic_physical :default: "`true`" :required: "no" :shortdesc: "Whether the NIC is plugged in or not" :type: "bool" ``` ```{config:option} boot.priority devices-nic_physical :default: "`skip`" :managed: "no" :shortdesc: "Boot priority for VMs (higher value boots first), `skip` to skip" :type: "string" ``` ```{config:option} gvrp devices-nic_physical :condition: "container" :default: "false" :managed: "no" :shortdesc: "Register VLAN using GARP VLAN Registration Protocol" :type: "bool" ``` ```{config:option} hwaddr devices-nic_physical :condition: "container" :default: "randomly assigned" :managed: "no" :shortdesc: "The MAC address of the new interface" :type: "string" ``` ```{config:option} mtu devices-nic_physical :condition: "container" :default: "MTU of the parent device" :managed: "no" :shortdesc: "The Maximum Transmit Unit (MTU) of the new interface" :type: "integer" ``` ```{config:option} name devices-nic_physical :default: "kernel assigned" :managed: "no" :shortdesc: "The name of the interface inside the instance" :type: "string" ``` ```{config:option} network devices-nic_physical :managed: "no" :shortdesc: "The managed network to link the device to (instead of specifying the `nictype` directly)" :type: "string" ``` ```{config:option} parent devices-nic_physical :managed: "yes" :shortdesc: "The name of the parent host device (required if specifying the `nictype` directly)" :type: "string" ``` ```{config:option} vlan devices-nic_physical :condition: "container" :managed: "no" :shortdesc: "The VLAN ID to attach to" :type: "integer" ``` ```{config:option} vlan.tagged devices-nic_physical :condition: "container" :managed: "no" :shortdesc: "Comma-delimited list of VLAN IDs or VLAN ranges to join for tagged traffic" :type: "integer" ``` ```{config:option} attached devices-nic_routed :default: "`true`" :required: "no" :shortdesc: "Whether the NIC is plugged in or not" :type: "bool" ``` ```{config:option} connected devices-nic_routed :default: "`true`" :required: "no" :shortdesc: "Whether the NIC is connected to the host network" :type: "bool" ``` ```{config:option} gvrp devices-nic_routed :default: "false" :shortdesc: "Register VLAN using GARP VLAN Registration Protocol" :type: "bool" ``` ```{config:option} host_name devices-nic_routed :default: "randomly assigned" :shortdesc: "The name of the interface on the host" :type: "string" ``` ```{config:option} hwaddr devices-nic_routed :default: "randomly assigned" :shortdesc: "The MAC address of the new interface" :type: "string" ``` ```{config:option} io.bus devices-nic_routed :default: "`virtio`" :shortdesc: "Override the bus for the device (can be `virtio` or `usb`) (VM only)" :type: "string" ``` ```{config:option} ipv4.address devices-nic_routed :shortdesc: "Comma-delimited list of IPv4 static addresses to add to the instance" :type: "string" ``` ```{config:option} ipv4.gateway devices-nic_routed :default: "auto" :shortdesc: "Whether to add an automatic default IPv4 gateway (can be `auto` or `none`)" :type: "string" ``` ```{config:option} ipv4.host_address devices-nic_routed :default: "`169.254.0.1`" :shortdesc: "The IPv4 address to add to the host-side `veth` interface" :type: "string" ``` ```{config:option} ipv4.host_table devices-nic_routed :shortdesc: "Deprecated: Use `ipv4.host_tables` instead" :type: "integer" The custom policy routing table ID to add IPv4 static routes to (in addition to the main routing table) ``` ```{config:option} ipv4.host_tables devices-nic_routed :default: "254" :shortdesc: "Comma-delimited list of routing tables IDs to add IPv4 static routes to" :type: "string" ``` ```{config:option} ipv4.neighbor_probe devices-nic_routed :default: "true" :shortdesc: "Whether to probe the parent network for IP address availability" :type: "bool" ``` ```{config:option} ipv4.routes devices-nic_routed :shortdesc: "Comma-delimited list of IPv4 static routes to add on host to NIC (without L2 ARP/NDP proxy)" :type: "string" ``` ```{config:option} ipv6.address devices-nic_routed :shortdesc: "Comma-delimited list of IPv6 static addresses to add to the instance" :type: "string" ``` ```{config:option} ipv6.gateway devices-nic_routed :default: "auto" :shortdesc: "Whether to add an automatic default IPv6 gateway (can be `auto` or `none`)" :type: "string" ``` ```{config:option} ipv6.host_address devices-nic_routed :default: "`fe80::1`" :shortdesc: "The IPv6 address to add to the host-side `veth` interface" :type: "string" ``` ```{config:option} ipv6.host_table devices-nic_routed :shortdesc: "Deprecated: Use `ipv6.host_tables` instead" :type: "integer" The custom policy routing table ID to add IPv6 static routes to (in addition to the main routing table) ``` ```{config:option} ipv6.host_tables devices-nic_routed :default: "254" :shortdesc: "Comma-delimited list of routing tables IDs to add IPv6 static routes to" :type: "string" ``` ```{config:option} ipv6.neighbor_probe devices-nic_routed :default: "true" :shortdesc: "Whether to probe the parent network for IP address availability" :type: "bool" ``` ```{config:option} ipv6.routes devices-nic_routed :shortdesc: "Comma-delimited list of IPv6 static routes to add on host to NIC (without L2 ARP/NDP proxy)" :type: "string" ``` ```{config:option} limits.egress devices-nic_routed :shortdesc: "I/O limit in bit/s for outgoing traffic (various suffixes supported, see {ref}`instances-limit-units`)" :type: "string" ``` ```{config:option} limits.egress.bucket devices-nic_routed :managed: "no" :shortdesc: "Amount of data in bit that outgoing traffic may send in excess of `limits.egress`, requires `limits.egress.burst`" :type: "string" ``` ```{config:option} limits.egress.burst devices-nic_routed :managed: "no" :shortdesc: "I/O limit in bit/s that outgoing traffic may burst up to, requires `limits.egress.bucket`" :type: "string" ``` ```{config:option} limits.ingress devices-nic_routed :shortdesc: "I/O limit in bit/s for incoming traffic (various suffixes supported, see {ref}`instances-limit-units`)" :type: "string" ``` ```{config:option} limits.ingress.bucket devices-nic_routed :managed: "no" :shortdesc: "Amount of data in bit that incoming traffic may send in excess of `limits.ingress`, requires `limits.ingress.burst`" :type: "string" ``` ```{config:option} limits.ingress.burst devices-nic_routed :managed: "no" :shortdesc: "I/O limit in bit/s that incoming traffic may burst up to, requires `limits.ingress.bucket`" :type: "string" ``` ```{config:option} limits.max devices-nic_routed :shortdesc: "I/O limit in bit/s for both incoming and outgoing traffic (same as setting both limits.ingress and limits.egress)" :type: "string" ``` ```{config:option} limits.max.bucket devices-nic_routed :managed: "no" :shortdesc: "Amount of data in bit that traffic may send in excess of the sustained limit (same as setting both `limits.ingress.bucket` and `limits.egress.bucket`)" :type: "string" ``` ```{config:option} limits.max.burst devices-nic_routed :managed: "no" :shortdesc: "I/O limit in bit/s that both incoming and outgoing traffic may burst up to (same as setting both `limits.ingress.burst` and `limits.egress.burst`)" :type: "string" ``` ```{config:option} limits.priority devices-nic_routed :shortdesc: "The priority for outgoing traffic, to be used by the kernel queuing discipline to prioritize network packets" :type: "integer" ``` ```{config:option} mtu devices-nic_routed :default: "parent MTU" :shortdesc: "The Maximum Transmit Unit (MTU) of the new interface" :type: "integer" ``` ```{config:option} name devices-nic_routed :default: "kernel assigned" :shortdesc: "The name of the interface inside the instance" :type: "string" ``` ```{config:option} parent devices-nic_routed :shortdesc: "The name of the parent host device to join the instance to" :type: "string" ``` ```{config:option} queue.discipline devices-nic_routed :shortdesc: "The queuing discipline to set on the NIC, applied to the rate limit's class when a limit is set" :type: "string" ``` ```{config:option} queue.discipline.attach devices-nic_routed :default: "`queue`" :shortdesc: "Only for VMs: Whether to attach the queuing discipline to each transmit queue (`queue`) or to the interface root (`root`)" :type: "string" ``` ```{config:option} queue.tx.length devices-nic_routed :shortdesc: "The transmit queue length for the NIC" :type: "integer" ``` ```{config:option} vlan devices-nic_routed :shortdesc: "The VLAN ID to attach to" :type: "integer" ``` ```{config:option} vrf devices-nic_routed :shortdesc: "The VRF on the host in which the host-side interface and routes are created" :type: "string" ``` ```{config:option} attached devices-nic_sriov :default: "`true`" :required: "no" :shortdesc: "Whether the NIC is plugged in or not" :type: "bool" ``` ```{config:option} boot.priority devices-nic_sriov :default: "`skip`" :managed: "no" :shortdesc: "Boot priority for VMs (higher value boots first), `skip` to skip" :type: "string" ``` ```{config:option} hwaddr devices-nic_sriov :default: "randomly assigned" :managed: "no" :shortdesc: "The MAC address of the new interface" :type: "string" ``` ```{config:option} mtu devices-nic_sriov :default: "kernel assigned" :managed: "yes" :shortdesc: "The Maximum Transmit Unit (MTU) of the new interface" :type: "integer" ``` ```{config:option} name devices-nic_sriov :default: "kernel assigned" :managed: "no" :shortdesc: "The name of the interface inside the instance" :type: "string" ``` ```{config:option} network devices-nic_sriov :managed: "no" :shortdesc: "The managed network to link the device to (instead of specifying the `nictype` directly)" :type: "string" ``` ```{config:option} parent devices-nic_sriov :managed: "yes" :shortdesc: "The name of the parent host device (required if specifying the `nictype` directly)" :type: "string" ``` ```{config:option} pci devices-nic_sriov :required: "no" :shortdesc: "The PCI address of the parent host device" :type: "string" ``` ```{config:option} productid devices-nic_sriov :required: "no" :shortdesc: "The product ID of the parent host device" :type: "string" ``` ```{config:option} security.mac_filtering devices-nic_sriov :default: "false" :managed: "no" :shortdesc: "Prevent the instance from spoofing another instance's MAC address" :type: "bool" ``` ```{config:option} security.trusted devices-nic_sriov :default: "false, if supported by parent device" :managed: "no" :shortdesc: "Allows the instance to configure the NIC in ways that may negatively impact security." :type: "bool" ``` ```{config:option} vendorid devices-nic_sriov :required: "no" :shortdesc: "The vendor ID of the parent host device" :type: "string" ``` ```{config:option} vlan devices-nic_sriov :managed: "no" :shortdesc: "The VLAN ID to attach to" :type: "integer" ``` ```{config:option} address devices-pci :required: "yes" :shortdesc: "PCI address of the device" :type: "string" ``` ```{config:option} firmware devices-pci :default: "true" :required: "no" :shortdesc: "Whether to expose the device's option ROM to the VM" :type: "bool" ``` ```{config:option} bind devices-proxy :default: "`host`" :required: "no" :shortdesc: "Which side to bind on (`host`/`instance`)" :type: "string" ``` ```{config:option} connect devices-proxy :required: "yes" :shortdesc: "The address and port to connect to (`::[-][,]`)" :type: "string" ``` ```{config:option} gid devices-proxy :default: "`0`" :required: "no" :shortdesc: "GID of the owner of the listening Unix socket" :type: "int" ``` ```{config:option} listen devices-proxy :required: "yes" :shortdesc: "The address and port to bind and listen (`::[-][,]`)" :type: "string" ``` ```{config:option} mode devices-proxy :default: "`0644`" :required: "no" :shortdesc: "Mode for the listening Unix socket" :type: "int" ``` ```{config:option} nat devices-proxy :default: "`false`" :required: "no" :shortdesc: "Whether to optimize proxying via NAT" :type: "bool" ``` ```{config:option} proxy_protocol devices-proxy :default: "`false`" :required: "no" :shortdesc: "Whether to use the HAProxy PROXY protocol to transmit sender information" :type: "bool" ``` ```{config:option} security.gid devices-proxy :default: "`0`" :required: "no" :shortdesc: "What GID to drop privilege to" :type: "int" ``` ```{config:option} security.uid devices-proxy :default: "`0`" :required: "no" :shortdesc: "What UID to drop privilege to" :type: "int" ``` ```{config:option} uid devices-proxy :default: "`0`" :required: "no" :shortdesc: "UID of the owner of the listening Unix socket" :type: "int" ``` ```{config:option} path devices-tpm :default: "-" :required: "for containers" :shortdesc: "Only for containers: path inside the instance (for example, `/dev/tpm0`)" :type: "string" ``` ```{config:option} pathrm devices-tpm :default: "-" :required: "for containers" :shortdesc: "Only for containers: resource manager path inside the instance (for example, `/dev/tpmrm0`)" :type: "string" ``` ```{config:option} gid devices-unix-char-block :default: "0" :shortdesc: "GID of the device owner in the instance" :type: "int" ``` ```{config:option} limits.read devices-unix-char-block :shortdesc: "I/O limit in byte/s and/or IOPS (comma separated) for read operations" :type: "string" ``` ```{config:option} limits.write devices-unix-char-block :shortdesc: "I/O limit in byte/s and/or IOPS (comma separated) for write operations" :type: "string" ``` ```{config:option} major devices-unix-char-block :default: "device on host" :shortdesc: "Device major number" :type: "int" ``` ```{config:option} minor devices-unix-char-block :default: "device on host" :shortdesc: "Device minor number" :type: "int" ``` ```{config:option} mode devices-unix-char-block :default: "0660" :shortdesc: "Mode of the device in the instance" :type: "int" ``` ```{config:option} path devices-unix-char-block :shortdesc: "Path inside the instance (one of `source` and `path` must be set)" :type: "string" ``` ```{config:option} required devices-unix-char-block :default: "true" :shortdesc: "Whether this device is required to start the instance" :type: "bool" ``` ```{config:option} source devices-unix-char-block :shortdesc: "Path on the host (one of `source` and `path` must be set)" :type: "string" ``` ```{config:option} uid devices-unix-char-block :default: "0" :shortdesc: "UID of the device owner in the instance" :type: "int" ``` ```{config:option} gid devices-unix-hotplug :default: "0" :shortdesc: "GID of the device owner in the instance" :type: "int" ``` ```{config:option} mode devices-unix-hotplug :default: "0660" :shortdesc: "Mode of the device in the instance" :type: "int" ``` ```{config:option} pci devices-unix-hotplug :shortdesc: "The PCI address of a USB controller to monitor" :type: "string" ``` ```{config:option} productid devices-unix-hotplug :shortdesc: "The product ID of the USB device" :type: "string" ``` ```{config:option} required devices-unix-hotplug :default: "true" :shortdesc: "Whether this device is required to start the instance" :type: "bool" ``` ```{config:option} uid devices-unix-hotplug :default: "0" :shortdesc: "UID of the device owner in the instance" :type: "int" ``` ```{config:option} vendorid devices-unix-hotplug :shortdesc: "The vendor ID of the USB device" :type: "string" ``` ```{config:option} attached devices-usb :default: "`true`" :required: "no" :shortdesc: "Whether the USB device is plugged in or not" :type: "bool" ``` ```{config:option} busnum devices-usb :shortdesc: "The bus number of which the USB device is attached" :type: "int" ``` ```{config:option} devnum devices-usb :shortdesc: "The device number of the USB device" :type: "int" ``` ```{config:option} gid devices-usb :defaultdesc: "`0`" :shortdesc: "Only for containers: GID of the device owner in the instance" :type: "int" ``` ```{config:option} mode devices-usb :defaultdesc: "`0660`" :shortdesc: "Only for containers: Mode of the device in the instance" :type: "int" ``` ```{config:option} productid devices-usb :shortdesc: "The product ID of the USB device" :type: "string" ``` ```{config:option} required devices-usb :defaultdesc: "`false`" :shortdesc: "Whether this device is required to start the instance (the default is `false`, and all devices can be hotplugged)" :type: "bool" ``` ```{config:option} serial devices-usb :shortdesc: "The serial number of the USB device" :type: "string" ``` ```{config:option} uid devices-usb :defaultdesc: "`0`" :shortdesc: "Only for containers: UID of the device owner in the instance" :type: "int" ``` ```{config:option} vendorid devices-usb :shortdesc: "The vendor ID of the USB device" :type: "string" ``` ```{config:option} requirements.cdrom_agent image-requirements :shortdesc: "If set to `true`, indicates that the VM requires an `agent:config` disk be added." :type: "bool" ``` ```{config:option} requirements.cdrom_cloud_init image-requirements :shortdesc: "If set to `true`, indicates that the VM requires a `cloud-init:config` disk be present every time `cloud-init` should be run." :type: "bool" ``` ```{config:option} requirements.cgroup image-requirements :shortdesc: "If set to `v1`, indicates that the image requires the host to run cgroup v1." :type: "string" ``` ```{config:option} requirements.nesting image-requirements :shortdesc: "If set to `true`, indicates that the image cannot work without nesting enabled." :type: "bool" ``` ```{config:option} requirements.privileged image-requirements :shortdesc: "If set to `false`, indicates that the image cannot work as a privileged container." :type: "bool" ``` ```{config:option} requirements.secureboot image-requirements :shortdesc: "If set to `false`, indicates that the image cannot boot under secure boot." :type: "bool" ``` ```{config:option} boot.autorestart instance-boot :liveupdate: "no" :shortdesc: "Whether to automatically restart an instance on unexpected exit" :type: "bool" If set to `true` will attempt up to 10 restarts over a 1 minute period upon unexpected instance exit. ``` ```{config:option} boot.autostart instance-boot :liveupdate: "no" :shortdesc: "Whether to always start the instance when the daemon starts" :type: "bool" If unset or set to `last-state`, restores the last state. ``` ```{config:option} boot.autostart.delay instance-boot :defaultdesc: "0" :liveupdate: "no" :shortdesc: "Delay after starting the instance" :type: "integer" The number of seconds to wait after the instance started before starting the next one. ``` ```{config:option} boot.autostart.priority instance-boot :liveupdate: "no" :shortdesc: "What order to start the instances in" :type: "integer" The instance with the highest value is started first. Instances without a priority set will be started (with some parallelism) ahead of instances with a priority set. ``` ```{config:option} boot.host_shutdown_action instance-boot :defaultdesc: "stop" :liveupdate: "yes" :shortdesc: "What action to take on the instance when the host is shut down" :type: "string" Action to take on host shut down Valid values are: `stop`, `force-stop` or `stateful-stop` ``` ```{config:option} boot.host_shutdown_timeout instance-boot :defaultdesc: "30" :liveupdate: "yes" :shortdesc: "How long to wait for the instance to shut down" :type: "integer" Number of seconds to wait for the instance to shut down before it is force-stopped. ``` ```{config:option} boot.pxe instance-boot :defaultdesc: "no" :liveupdate: "yes" :shortdesc: "Whether to enable PXE boot" :type: "bool" ``` ```{config:option} boot.stop.priority instance-boot :defaultdesc: "0" :liveupdate: "no" :shortdesc: "What order to shut down the instances in" :type: "integer" The instance with the highest value is shut down first. ``` ```{config:option} cloud-init.network-config instance-cloud-init :condition: "If supported by image" :defaultdesc: "`DHCP on eth0`" :liveupdate: "no" :shortdesc: "Network configuration for `cloud-init`" :type: "string" The content is used as seed value for `cloud-init`. ``` ```{config:option} cloud-init.user-data instance-cloud-init :condition: "If supported by image" :defaultdesc: "`#cloud-config`" :liveupdate: "no" :shortdesc: "User data for `cloud-init`" :type: "string" The content is used as seed value for `cloud-init`. ``` ```{config:option} cloud-init.vendor-data instance-cloud-init :condition: "If supported by image" :defaultdesc: "`#cloud-config`" :liveupdate: "no" :shortdesc: "Vendor data for `cloud-init`" :type: "string" The content is used as seed value for `cloud-init`. ``` ```{config:option} user.network-config instance-cloud-init :condition: "If supported by image" :defaultdesc: "`DHCP on eth0`" :liveupdate: "no" :shortdesc: "Legacy version of `cloud-init.network-config`" :type: "string" ``` ```{config:option} user.user-data instance-cloud-init :condition: "If supported by image" :defaultdesc: "`#cloud-config`" :liveupdate: "no" :shortdesc: "Legacy version of `cloud-init.user-data`" :type: "string" ``` ```{config:option} user.vendor-data instance-cloud-init :condition: "If supported by image" :defaultdesc: "`#cloud-config`" :liveupdate: "no" :shortdesc: "Legacy version of `cloud-init.vendor-data`" :type: "string" ``` ```{config:option} migration.incremental.memory instance-migration :condition: "container" :defaultdesc: "`false`" :liveupdate: "yes" :shortdesc: "Whether to use incremental memory transfer" :type: "bool" Using incremental memory transfer of the instance's memory can reduce downtime. ``` ```{config:option} migration.incremental.memory.goal instance-migration :condition: "container" :defaultdesc: "`70`" :liveupdate: "yes" :shortdesc: "Percentage of memory to have in sync before stopping the instance" :type: "integer" ``` ```{config:option} migration.incremental.memory.iterations instance-migration :condition: "container" :defaultdesc: "`10`" :liveupdate: "yes" :shortdesc: "Maximum number of transfer operations to go through before stopping the instance" :type: "integer" ``` ```{config:option} migration.stateful instance-migration :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to allow for stateful stop/start and snapshots" :type: "bool" Enabling this option prevents the use of some features that are incompatible with it. ``` ```{config:option} agent.nic_config instance-miscellaneous :condition: "virtual machine" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to use the name and MTU of the default network interfaces" :type: "bool" For containers, the name and MTU of the default network interfaces is used for the instance devices. For virtual machines, set this option to `true` to set the name and MTU of the default network interfaces to be the same as the instance devices. ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ```{config:option} environment.* instance-miscellaneous :liveupdate: "yes" :shortdesc: "Free-form environment key/value" :type: "string" Extra environment variables to set on boot and during exec. ``` ```{config:option} linux.kernel_modules instance-miscellaneous :condition: "container" :liveupdate: "yes" :shortdesc: "Kernel modules to load before starting the instance" :type: "string" Specify the kernel modules as a comma-separated list. ``` ```{config:option} linux.sysctl.* instance-miscellaneous :condition: "container" :liveupdate: "no" :shortdesc: "Override for the corresponding `sysctl` setting in the container" :type: "string" ``` ```{config:option} smbios11.* instance-miscellaneous :liveupdate: "yes" :shortdesc: "Free-form `SMBIOS Type 11` key/value" :type: "string" `SMBIOS Type 11` configuration keys. ``` ```{config:option} systemd.credential-binary.* instance-miscellaneous :liveupdate: "yes" :shortdesc: "Systemd credential key/value, where value is Base64 encoded" :type: "string" Systemd credential key/value pair passed as a read-only bind mount in containers and as `SMBIOS Type 11` data in virtual machines. The value is Base64 encoded. ``` ```{config:option} systemd.credential.* instance-miscellaneous :liveupdate: "yes" :shortdesc: "Systemd credential key/value" :type: "string" Systemd credential key/value pair passed as a read-only bind mount in containers and as `SMBIOS Type 11` data in virtual machines. ``` ```{config:option} user.* instance-miscellaneous :liveupdate: "yes" :shortdesc: "Free-form user key/value storage" :type: "string" User keys can be used in search. ``` ```{config:option} nvidia.driver.capabilities instance-nvidia :condition: "container" :defaultdesc: "`compute,utility`" :liveupdate: "no" :shortdesc: "What driver capabilities the instance needs" :type: "string" The specified driver capabilities are used to set `libnvidia-container NVIDIA_DRIVER_CAPABILITIES`. ``` ```{config:option} nvidia.require.cuda instance-nvidia :condition: "container" :liveupdate: "no" :shortdesc: "Required CUDA version" :type: "string" The specified version expression is used to set `libnvidia-container NVIDIA_REQUIRE_CUDA`. ``` ```{config:option} nvidia.require.driver instance-nvidia :condition: "container" :liveupdate: "no" :shortdesc: "Required driver version" :type: "string" The specified version expression is used to set `libnvidia-container NVIDIA_REQUIRE_DRIVER`. ``` ```{config:option} nvidia.runtime instance-nvidia :condition: "container" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to pass the host NVIDIA and CUDA runtime libraries into the instance" :type: "bool" ``` ```{config:option} oci.cwd instance-oci :condition: "OCI container" :liveupdate: "no" :shortdesc: "Working directory" :type: "string" Override the working directory of an OCI container. ``` ```{config:option} oci.dns.domain instance-oci :condition: "OCI container" :liveupdate: "no" :shortdesc: "DNS domain" :type: "string" Domain name for `resolv.conf`. When set, the domain name received over DHCP is ignored. ``` ```{config:option} oci.dns.nameservers instance-oci :condition: "OCI container" :liveupdate: "no" :shortdesc: "DNS name servers" :type: "string" Comma-separated list of name server addresses for `resolv.conf`. When set, name servers received over DHCP are ignored. ``` ```{config:option} oci.dns.search instance-oci :condition: "OCI container" :liveupdate: "no" :shortdesc: "DNS search domains" :type: "string" Comma-separated list of search domains for `resolv.conf`. When set, search domains received over DHCP are ignored. ``` ```{config:option} oci.entrypoint instance-oci :condition: "OCI container" :liveupdate: "no" :shortdesc: "Entry point" :type: "string" Override the entry point of an OCI container. ``` ```{config:option} oci.gid instance-oci :condition: "OCI container" :liveupdate: "no" :shortdesc: "Process GID" :type: "string" Override the GID of the process run in an OCI container. ``` ```{config:option} oci.uid instance-oci :condition: "OCI container" :liveupdate: "no" :shortdesc: "Process UID" :type: "string" Override the UID of the process run in an OCI container. ``` ```{config:option} initial.nvram-binary.. instance-raw :condition: "virtual machine" :liveupdate: "yes" :shortdesc: "Base64-encoded NVRAM variable value to set when generating the NVRAM, optionally prefixed by its attributes" :type: "blob" Base64-encoded variable value to set when generating the NVRAM, optionally prefixed by `:`, with `` an integer. ``` ```{config:option} initial.nvram.. instance-raw :condition: "virtual machine" :liveupdate: "yes" :shortdesc: "Dissected NVRAM variable value to set when generating the NVRAM" :type: "blob" ``` ```{config:option} raw.apparmor instance-raw :liveupdate: "yes" :shortdesc: "AppArmor profile entries" :type: "blob" The specified entries are appended to the generated profile. ``` ```{config:option} raw.idmap instance-raw :condition: "unprivileged container" :liveupdate: "no" :shortdesc: "Raw idmap configuration" :type: "blob" For example: `both 1000 1000` ``` ```{config:option} raw.lxc instance-raw :condition: "container" :liveupdate: "no" :shortdesc: "Raw LXC configuration to be appended to the generated one" :type: "blob" ``` ```{config:option} raw.qemu instance-raw :condition: "virtual machine" :liveupdate: "no" :shortdesc: "Raw QEMU configuration to be appended to the generated command line" :type: "blob" ``` ```{config:option} raw.qemu.conf instance-raw :condition: "virtual machine" :liveupdate: "no" :shortdesc: "Addition/override to the generated `qemu.conf` file" :type: "blob" See {ref}`instance-options-qemu` for more information. ``` ```{config:option} raw.qemu.qmp.early instance-raw :condition: "virtual machine" :liveupdate: "no" :shortdesc: "QMP commands to run before Incus QEMU initialization" :type: "blob" ``` ```{config:option} raw.qemu.qmp.post-start instance-raw :condition: "virtual machine" :liveupdate: "no" :shortdesc: "QMP commands to run after the VM has started" :type: "blob" ``` ```{config:option} raw.qemu.qmp.pre-start instance-raw :condition: "virtual machine" :liveupdate: "no" :shortdesc: "QMP commands to run after Incus QEMU initialization and before the VM has started" :type: "blob" ``` ```{config:option} raw.qemu.scriptlet instance-raw :condition: "virtual machine" :liveupdate: "no" :shortdesc: "QEMU scriptlet to run at early, pre-start and post-start stages" :type: "string" ``` ```{config:option} raw.seccomp instance-raw :condition: "container" :liveupdate: "no" :shortdesc: "Raw Seccomp configuration" :type: "blob" ``` ```{config:option} limits.cpu instance-resource-limits :defaultdesc: "1 (VMs)" :liveupdate: "yes" :shortdesc: "Which CPUs to expose to the instance" :type: "string" A number or a specific range of CPUs to expose to the instance. For virtual machines, a CPU topology of the form `sockets=2,cores=4,threads=2` may also be provided. See {ref}`instance-options-limits-cpu` for more information. ``` ```{config:option} limits.cpu.allowance instance-resource-limits :condition: "container" :defaultdesc: "100%" :liveupdate: "yes" :shortdesc: "How much of the CPU can be used" :type: "string" To control how much of the CPU can be used, specify either a percentage (`50%`) for a soft limit or a chunk of time (`25ms/100ms`) for a hard limit. See {ref}`instance-options-limits-cpu-container` for more information. ``` ```{config:option} limits.cpu.nodes instance-resource-limits :liveupdate: "yes" :shortdesc: "Which NUMA nodes to place the instance CPUs on" :type: "string" A comma-separated list of NUMA node IDs or ranges to place the instance CPUs on. Alternatively, the value `balanced` may be used to have Incus pick the least busy NUMA node on startup. See {ref}`instance-options-limits-cpu-container` for more information. ``` ```{config:option} limits.cpu.priority instance-resource-limits :condition: "container" :defaultdesc: "`10` (maximum)" :liveupdate: "yes" :shortdesc: "CPU scheduling priority compared to other instances" :type: "integer" When overcommitting resources, specify the CPU scheduling priority compared to other instances that share the same CPUs. Specify an integer between 0 and 10. See {ref}`instance-options-limits-cpu-container` for more information. ``` ```{config:option} limits.disk.priority instance-resource-limits :defaultdesc: "`5` (medium)" :liveupdate: "yes" :shortdesc: "Priority of the instance's I/O requests" :type: "integer" Controls how much priority to give to the instance's I/O requests when under load. Specify an integer between 0 and 10. ``` ```{config:option} limits.hugepages.1GB instance-resource-limits :condition: "container" :liveupdate: "yes" :shortdesc: "Limit for the number of 1 GB huge pages" :type: "string" Fixed value (in bytes) to limit the number of 1 GB huge pages. Various suffixes are supported (see {ref}`instances-limit-units`). See {ref}`instance-options-limits-hugepages` for more information. ``` ```{config:option} limits.hugepages.1MB instance-resource-limits :condition: "container" :liveupdate: "yes" :shortdesc: "Limit for the number of 1 MB huge pages" :type: "string" Fixed value (in bytes) to limit the number of 1 MB huge pages. Various suffixes are supported (see {ref}`instances-limit-units`). See {ref}`instance-options-limits-hugepages` for more information. ``` ```{config:option} limits.hugepages.2MB instance-resource-limits :condition: "container" :liveupdate: "yes" :shortdesc: "Limit for the number of 2 MB huge pages" :type: "string" Fixed value (in bytes) to limit the number of 2 MB huge pages. Various suffixes are supported (see {ref}`instances-limit-units`). See {ref}`instance-options-limits-hugepages` for more information. ``` ```{config:option} limits.hugepages.64KB instance-resource-limits :condition: "container" :liveupdate: "yes" :shortdesc: "Limit for the number of 64 KB huge pages" :type: "string" Fixed value (in bytes) to limit the number of 64 KB huge pages. Various suffixes are supported (see {ref}`instances-limit-units`). See {ref}`instance-options-limits-hugepages` for more information. ``` ```{config:option} limits.memory instance-resource-limits :defaultdesc: "`1GiB` (VMs)" :liveupdate: "yes" :shortdesc: "Usage limit for the host's memory" :type: "string" Percentage of the host's memory or a fixed value in bytes. Various suffixes are supported. See {ref}`instances-limit-units` for details. ``` ```{config:option} limits.memory.enforce instance-resource-limits :condition: "container" :defaultdesc: "`hard`" :liveupdate: "yes" :shortdesc: "Whether the memory limit is `hard` or `soft`" :type: "string" If the instance's memory limit is `hard`, the instance cannot exceed its limit. If it is `soft`, the instance can exceed its memory limit when extra host memory is available. ``` ```{config:option} limits.memory.hotplug instance-resource-limits :condition: "virtual machine" :defaultdesc: "`true`" :liveupdate: "yes" :shortdesc: "Control upper limit for hotplugged memory or disable memory hotplug." :type: "string" If this option is set to `false`, disable memory hotplug entirely. Alternatively, it can be set to a bytes value which will define an upper limit for hotplugged memory. The value must be greater than or equal to limits.memory. ``` ```{config:option} limits.memory.hugepages instance-resource-limits :condition: "virtual machine" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to back the instance using huge pages" :type: "bool" If this option is set to `false`, regular system memory is used. ``` ```{config:option} limits.memory.oom_priority instance-resource-limits :defaultdesc: "`0`" :liveupdate: "yes" :shortdesc: "Out Of Memory killer priority adjustment for the instance" :type: "integer" Specify an integer between -1000 and 1000. A negative value makes the instance less likely to be killed by the Out Of Memory killer, while a positive value makes it more likely to be killed. The default value of 0 means no adjustment to the Out Of Memory score. ``` ```{config:option} limits.memory.swap instance-resource-limits :condition: "container" :defaultdesc: "`true`" :liveupdate: "yes" :shortdesc: "Control swap usage by the instance" :type: "string" When set to `true` or `false`, it controls whether the container is likely to get some of its memory swapped by the kernel. Alternatively, it can be set to a bytes value which will then allow the container to make use of additional memory through swap. Support for this is limited on cgroup2 systems due to lack of swap priority control. ``` ```{config:option} limits.memory.swap.priority instance-resource-limits :condition: "container" :defaultdesc: "`10` (maximum)" :liveupdate: "yes" :shortdesc: "Prevents the instance from being swapped to disk" :type: "integer" Specify an integer between 0 and 10. The higher the value, the less likely the instance is to be swapped to disk. This currently doesn't have any effect on cgroup2 systems. ``` ```{config:option} limits.processes instance-resource-limits :condition: "container" :defaultdesc: "empty" :liveupdate: "yes" :shortdesc: "Maximum number of processes that can run in the instance" :type: "integer" If left empty, no limit is set. ``` ```{config:option} initial.secureboot.db instance-security :condition: "virtual machine" :liveupdate: "yes" :shortdesc: "Initial authorized signature database entries to enroll when generating the NVRAM (PEM certificate and signature bundle)" :type: "string" This option accepts signatures armored with `SIGNATURE` in addition to certificates. ``` ```{config:option} initial.secureboot.dbt instance-security :condition: "virtual machine" :liveupdate: "yes" :shortdesc: "Initial authorized timestamp signature database certificates to enroll when generating the NVRAM (PEM certificate bundle)" :type: "string" ``` ```{config:option} initial.secureboot.dbx instance-security :condition: "virtual machine" :liveupdate: "yes" :shortdesc: "Initial forbidden signature database entries to enroll when generating the NVRAM (PEM certificate and signature bundle)" :type: "string" This option accepts signatures armored with `SIGNATURE` in addition to certificates. ``` ```{config:option} initial.secureboot.kek instance-security :condition: "virtual machine" :liveupdate: "yes" :shortdesc: "Initial key exchange keys to enroll when generating the NVRAM (PEM certificate bundle)" :type: "string" ``` ```{config:option} initial.secureboot.mok instance-security :condition: "virtual machine" :liveupdate: "yes" :shortdesc: "Initial machine owner key entries to enroll when generating the NVRAM (PEM certificate and signature bundle)" :type: "string" This option accepts signatures armored with `SIGNATURE` in addition to certificates. ``` ```{config:option} initial.secureboot.pk instance-security :condition: "virtual machine" :liveupdate: "yes" :shortdesc: "Initial platform key to enroll when generating the NVRAM (PEM certificate)" :type: "string" ``` ```{config:option} security.agent.metrics instance-security :condition: "virtual machine" :defaultdesc: "`true`" :liveupdate: "no" :shortdesc: "Whether the `incus-agent` is queried for state information and metrics" :type: "bool" ``` ```{config:option} security.bpffs.delegate_attachs instance-security :condition: "unprivileged container" :liveupdate: "no" :shortdesc: "What BPF attach types to delegate" :type: "string" See {ref}`bpf-tokens` for more information. ``` ```{config:option} security.bpffs.delegate_cmds instance-security :condition: "unprivileged container" :liveupdate: "no" :shortdesc: "What BPF command types to delegate" :type: "string" See {ref}`bpf-tokens` for more information. ``` ```{config:option} security.bpffs.delegate_maps instance-security :condition: "unprivileged container" :liveupdate: "no" :shortdesc: "What BPF map types to delegate" :type: "string" See {ref}`bpf-tokens` for more information. ``` ```{config:option} security.bpffs.delegate_progs instance-security :condition: "unprivileged container" :liveupdate: "no" :shortdesc: "What BPF program types to delegate" :type: "string" See {ref}`bpf-tokens` for more information. ``` ```{config:option} security.bpffs.path instance-security :condition: "unprivileged container" :defaultdesc: "`/sys/fs/bpf`" :liveupdate: "no" :shortdesc: "The path to mount the BPF file system at" :type: "string" The specified path must exist in the container. The BPF file system is only mounted if any of the `security.bpffs.delegate_*` options are set. See {ref}`bpf-tokens` for more information. ``` ```{config:option} security.csm instance-security :condition: "virtual machine" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to use a firmware that supports UEFI-incompatible operating systems" :type: "bool" When enabling this option, set {config:option}`instance-security:security.secureboot` to `false`. ``` ```{config:option} security.guestapi instance-security :defaultdesc: "`true`" :liveupdate: "no" :shortdesc: "Whether `/dev/incus` is present in the instance" :type: "bool" See {ref}`dev-incus` for more information. ``` ```{config:option} security.guestapi.images instance-security :condition: "container" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Controls the availability of the `/1.0/images` API over `guestapi`" :type: "bool" ``` ```{config:option} security.idmap.base instance-security :condition: "unprivileged container" :liveupdate: "no" :shortdesc: "The base host ID to use for the ID map" :type: "integer" The range can be shared by other containers using the same base and can't be combined with `security.idmap.isolated`. ``` ```{config:option} security.idmap.isolated instance-security :condition: "unprivileged container" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to use a unique idmap for this instance" :type: "bool" If specified, the idmap used for this instance is unique among instances that have this option set. ``` ```{config:option} security.idmap.size instance-security :condition: "unprivileged container" :liveupdate: "no" :shortdesc: "The size of the idmap to use" :type: "integer" ``` ```{config:option} security.iommu instance-security :condition: "virtual machine" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to enable virtual IOMMU, useful for device passthrough and nesting" :type: "bool" ``` ```{config:option} security.nesting instance-security :defaultdesc: "`false` (containers), `true` (VMs)" :liveupdate: "yes (containers only)" :shortdesc: "Whether to allow nesting inside of the instance" :type: "bool" For containers, this controls whether Incus (nested) can be run inside of the instance. For virtual machines, setting this to `false` disables nested virtualization (turns off the `svm` and `vmx` CPU flags). Explicitly setting this to `true` on a virtual machine also makes it use non-shared memory (allowing transparent huge pages) unless a device requires shared memory, see {ref}`performance-tuning-vm-memory`. ``` ```{config:option} security.privileged instance-security :condition: "container" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to run the instance in privileged mode" :type: "bool" ``` ```{config:option} security.protection.delete instance-security :defaultdesc: "`false`" :liveupdate: "yes" :shortdesc: "Prevents the instance from being deleted" :type: "bool" ``` ```{config:option} security.protection.shift instance-security :condition: "container" :defaultdesc: "`false`" :liveupdate: "yes" :shortdesc: "Whether to protect the file system from being UID/GID shifted" :type: "bool" Set this option to `true` to prevent the instance's file system from being UID/GID shifted on startup. ``` ```{config:option} security.protection.start instance-security :defaultdesc: "`false`" :liveupdate: "yes" :shortdesc: "Prevents the instance from being started" :type: "bool" ``` ```{config:option} security.secureboot instance-security :condition: "virtual machine" :defaultdesc: "`true`" :liveupdate: "no" :shortdesc: "Whether UEFI secure boot is enforced with the default Microsoft keys" :type: "bool" When disabling this option, consider enabling {config:option}`instance-security:security.csm`. ``` ```{config:option} security.selinux.domain instance-security :condition: "container or virtual machine" :defaultdesc: "auto-detected (`container_init_t` for containers, `qemu_t` for VMs)" :liveupdate: "no" :shortdesc: "SELinux process domain override" :type: "string" Override the SELinux process domain for the instance. ``` ```{config:option} security.selinux.label_rootfs instance-security :condition: "container" :defaultdesc: "`auto`" :liveupdate: "no" :shortdesc: "SELinux rootfs labeling mode (auto, always, never)" :type: "string" Control SELinux rootfs labeling behavior. The default (`auto`) will label rootfs files if it is required. Labeling will be skipped only if `security.selinux.level` is explicitly set for the instance, it is not started for the first time and the persisted SELinux context is still valid. Setting to `always` will label rootfs files on every start and `never` will never touch any file labels in the rootfs. ``` ```{config:option} security.selinux.level instance-security :condition: "container or virtual machine" :defaultdesc: "auto-generated" :liveupdate: "no" :shortdesc: "SELinux MCS level override" :type: "string" Override the SELinux MCS level for the instance. This key must only be set on individual instances, never on profiles, since using the same MCS level across instances breaks isolation. Values set via profiles are ignored. ``` ```{config:option} security.selinux.type instance-security :condition: "container or virtual machine" :defaultdesc: "auto-detected (`container_file_t` for containers, `qemu_image_t` for VMs)" :liveupdate: "no" :shortdesc: "SELinux file type override" :type: "string" Override the SELinux file type used for labeling instance storage. ``` ```{config:option} security.sev instance-security :condition: "virtual machine" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether AMD SEV (Secure Encrypted Virtualization) is enabled for this VM" :type: "bool" ``` ```{config:option} security.sev.policy.es instance-security :condition: "virtual machine" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether AMD SEV-ES (SEV Encrypted State) is enabled for this VM" :type: "bool" ``` ```{config:option} security.sev.session.data instance-security :condition: "virtual machine" :defaultdesc: "`true`" :liveupdate: "no" :shortdesc: "The guest owner's `base64`-encoded session blob" :type: "string" ``` ```{config:option} security.sev.session.dh instance-security :condition: "virtual machine" :defaultdesc: "`true`" :liveupdate: "no" :shortdesc: "The guest owner's `base64`-encoded Diffie-Hellman key" :type: "string" ``` ```{config:option} security.syscalls.allow instance-security :condition: "container" :liveupdate: "no" :shortdesc: "List of syscalls to allow" :type: "string" A `\n`-separated list of syscalls to allow. This list must be mutually exclusive with `security.syscalls.deny*`. ``` ```{config:option} security.syscalls.deny instance-security :condition: "container" :liveupdate: "no" :shortdesc: "List of syscalls to deny" :type: "string" A `\n`-separated list of syscalls to deny. This list must be mutually exclusive with `security.syscalls.allow`. ``` ```{config:option} security.syscalls.deny_compat instance-security :condition: "container" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to block `compat_*` syscalls (`x86_64` only)" :type: "bool" On `x86_64`, this option controls whether to block `compat_*` syscalls. On other architectures, the option is ignored. ``` ```{config:option} security.syscalls.deny_default instance-security :condition: "container" :defaultdesc: "`true`" :liveupdate: "no" :shortdesc: "Whether to enable the default syscall deny" :type: "bool" ``` ```{config:option} security.syscalls.intercept.bpf instance-security :condition: "container" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to handle the `bpf()` system call" :type: "bool" ``` ```{config:option} security.syscalls.intercept.bpf.devices instance-security :condition: "container" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to allow BPF programs" :type: "bool" This option controls whether to allow BPF programs for the devices cgroup in the unified hierarchy to be loaded. ``` ```{config:option} security.syscalls.intercept.mknod instance-security :condition: "container" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to handle the `mknod` and `mknodat` system calls" :type: "bool" These system calls allow creation of a limited subset of char/block devices. ``` ```{config:option} security.syscalls.intercept.mount instance-security :condition: "container" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to handle the `mount` system call" :type: "bool" ``` ```{config:option} security.syscalls.intercept.mount.allowed instance-security :condition: "container" :liveupdate: "yes" :shortdesc: "File systems that can be mounted" :type: "string" Specify a comma-separated list of file systems that are safe to mount for processes inside the instance. ``` ```{config:option} security.syscalls.intercept.mount.fuse instance-security :condition: "container" :liveupdate: "yes" :shortdesc: "File system that should be redirected to FUSE implementation" :type: "string" Specify the mounts of a given file system that should be redirected to their FUSE implementation (for example, `ext4=fuse2fs`). ``` ```{config:option} security.syscalls.intercept.mount.shift instance-security :condition: "container" :defaultdesc: "`false`" :liveupdate: "yes" :shortdesc: "Whether to use idmapped mounts for syscall interception" :type: "bool" ``` ```{config:option} security.syscalls.intercept.sched_setscheduler instance-security :condition: "container" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to handle the `sched_setscheduler` system call" :type: "bool" This system call allows increasing process priority. ``` ```{config:option} security.syscalls.intercept.setxattr instance-security :condition: "container" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to handle the `setxattr` system call" :type: "bool" This system call allows setting a limited subset of restricted extended attributes. ``` ```{config:option} security.syscalls.intercept.sysinfo instance-security :condition: "container" :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to handle the `sysinfo` system call" :type: "bool" This system call can be used to get cgroup-based resource usage information. ``` ```{config:option} security.tags instance-security :liveupdate: "yes" :shortdesc: "Comma-separated list of security tags for the authorization backend" :type: "string" Tags are made of lowercase letters, digits, dots, dashes and underscores and are at most 64 characters long. They are exposed to the authorization backend but don't affect Incus' own authorization decisions. ``` ```{config:option} snapshots.expiry instance-snapshots :liveupdate: "no" :shortdesc: "When newly created snapshots are to be deleted" :type: "string" Specify an expression like `1M 2H 3d 4w 5m 6y`. This value is used to compute the expiry date of newly created snapshots. It is added to the current time when a snapshot is taken, and the resulting timestamp is stored as that snapshot's individual expiry date. Changing this value only affects snapshots created after the change; the expiry date of existing snapshots is not modified. The supported units are `S` (seconds), `M` (minutes), `H` (hours), `d` (days), `w` (weeks), `m` (months) and `y` (years). ``` ```{config:option} snapshots.expiry.manual instance-snapshots :liveupdate: "no" :shortdesc: "When newly created snapshots are to be deleted (for those not created through scheduling)" :type: "string" Specify an expression like `1M 2H 3d 4w 5m 6y`. This value is used to compute the expiry date of newly created snapshots that are not created through scheduling. It is added to the current time when a snapshot is taken, and the resulting timestamp is stored as that snapshot's individual expiry date. Changing this value only affects snapshots created after the change; the expiry date of existing snapshots is not modified. See {config:option}`instance-snapshots:snapshots.expiry` for the supported units. ``` ```{config:option} snapshots.pattern instance-snapshots :defaultdesc: "`snap%d`" :liveupdate: "no" :shortdesc: "Template for the snapshot name" :type: "string" Specify a Pongo2 template string that represents the snapshot name. This template is used for scheduled snapshots and for unnamed snapshots. See {ref}`instance-options-snapshots-names` for more information. ``` ```{config:option} snapshots.schedule instance-snapshots :defaultdesc: "empty" :liveupdate: "no" :shortdesc: "Schedule for automatic instance snapshots" :type: "string" Specify either a cron expression (` `), a comma-and-space-separated list of schedule aliases (`@startup`, `@hourly`, `@daily`, `@midnight`, `@weekly`, `@monthly`, `@annually`, `@yearly`), or leave empty to disable automatic snapshots. Note that unlike most other configuration keys, this one must be comma-and-space-separated and not just comma-separated as cron expression can themselves contain commas. ``` ```{config:option} snapshots.schedule.stopped instance-snapshots :defaultdesc: "`false`" :liveupdate: "no" :shortdesc: "Whether to automatically snapshot stopped instances" :type: "bool" ``` ```{config:option} volatile..apply_quota instance-volatile :shortdesc: "Disk quota" :type: "string" The disk quota is applied the next time the instance starts. ``` ```{config:option} volatile..ceph_rbd instance-volatile :shortdesc: "RBD device path for Ceph disk devices" :type: "string" ``` ```{config:option} volatile..host_name instance-volatile :shortdesc: "Network device name on the host" :type: "string" ``` ```{config:option} volatile..hwaddr instance-volatile :shortdesc: "Network device MAC address" :type: "string" The network device MAC address is used when no `hwaddr` property is set on the device itself. ``` ```{config:option} volatile..io.bus instance-volatile :shortdesc: "IO bus in use" :type: "string" The IO bus stores the actual IO bus being used, checked in case `io.bus=auto`. ``` ```{config:option} volatile..last_state.created instance-volatile :shortdesc: "Whether the network device physical device was created" :type: "string" Possible values are `true` or `false`. ``` ```{config:option} volatile..last_state.hwaddr instance-volatile :shortdesc: "Network device original MAC" :type: "string" The original MAC that was used when moving a physical device into an instance. ``` ```{config:option} volatile..last_state.ip_addresses instance-volatile :shortdesc: "Last used IP addresses" :type: "string" Comma-separated list of the last used IP addresses of the network device. ``` ```{config:option} volatile..last_state.mtu instance-volatile :shortdesc: "Network device original MTU" :type: "string" The original MTU that was used when moving a physical device into an instance. ``` ```{config:option} volatile..last_state.pci.driver instance-volatile :shortdesc: "PCI original host driver" :type: "string" The original host driver for the PCI device. ``` ```{config:option} volatile..last_state.pci.parent instance-volatile :shortdesc: "PCI parent host device" :type: "string" The parent host device used when allocating a PCI device to an instance. ``` ```{config:option} volatile..last_state.pci.slot.name instance-volatile :shortdesc: "PCI parent slot name" :type: "string" The parent host device PCI slot name. ``` ```{config:option} volatile..last_state.usb.bus instance-volatile :shortdesc: "USB bus address" :type: "string" The original USB bus address. ``` ```{config:option} volatile..last_state.usb.device instance-volatile :shortdesc: "USB device identifier" :type: "string" The original USB device identifier. ``` ```{config:option} volatile..last_state.vdpa.name instance-volatile :shortdesc: "VDPA device name" :type: "string" The VDPA device name used when moving a VDPA device file descriptor into an instance. ``` ```{config:option} volatile..last_state.vf.hwaddr instance-volatile :shortdesc: "SR-IOV virtual function original MAC" :type: "string" The original MAC used when moving a VF into an instance. ``` ```{config:option} volatile..last_state.vf.id instance-volatile :shortdesc: "SR-IOV virtual function ID" :type: "string" The ID used when moving a VF into an instance. ``` ```{config:option} volatile..last_state.vf.parent instance-volatile :shortdesc: "SR-IOV parent host device" :type: "string" The parent host device used when allocating a VF into an instance. ``` ```{config:option} volatile..last_state.vf.spoofcheck instance-volatile :shortdesc: "SR-IOV virtual function original spoof check setting" :type: "string" The original spoof check setting used when moving a VF into an instance. ``` ```{config:option} volatile..last_state.vf.trusted instance-volatile :shortdesc: "SR-IOV virtual function original trusted setting" :type: "string" The original trusted setting used when moving a VF into an instance. ``` ```{config:option} volatile..last_state.vf.vlan instance-volatile :shortdesc: "SR-IOV virtual function original VLAN" :type: "string" The original VLAN used when moving a VF into an instance. ``` ```{config:option} volatile..mig.uuid instance-volatile :shortdesc: "MIG instance UUID" :type: "string" The NVIDIA MIG instance UUID. ``` ```{config:option} volatile..name instance-volatile :shortdesc: "Network interface name inside of the instance" :type: "string" The network interface name inside of the instance when no `name` property is set on the device itself. ``` ```{config:option} volatile..vgpu.uuid instance-volatile :shortdesc: "virtual GPU instance UUID" :type: "string" The NVIDIA virtual GPU instance UUID. ``` ```{config:option} volatile.apply_nvram instance-volatile :shortdesc: "Whether to regenerate VM NVRAM the next time the instance starts" :type: "bool" ``` ```{config:option} volatile.apply_template instance-volatile :shortdesc: "Template hook" :type: "string" The template with the given name is triggered upon next startup. ``` ```{config:option} volatile.base_image instance-volatile :shortdesc: "Hash of the base image" :type: "string" The hash of the image that the instance was created from (empty if the instance was not created from an image). ``` ```{config:option} volatile.cloud_init.instance-id instance-volatile :shortdesc: "`instance-id` (UUID) exposed to `cloud-init`" :type: "string" ``` ```{config:option} volatile.cluster.group instance-volatile :shortdesc: "The original cluster group for the instance" :type: "string" The cluster group(s) that the instance was restricted to at creation time. This is used during re-scheduling events like an evacuation to keep the instance within the requested set. ``` ```{config:option} volatile.container.oci instance-volatile :defaultdesc: "`false`" :shortdesc: "Whether the container is an OCI application container" :type: "bool" ``` ```{config:option} volatile.cpu.nodes instance-volatile :shortdesc: "Instance NUMA node" :type: "string" The NUMA node that was selected for the instance. ``` ```{config:option} volatile.evacuate.origin instance-volatile :shortdesc: "The origin of the evacuated instance" :type: "string" The cluster member that the instance lived on before evacuation. ``` ```{config:option} volatile.idmap.base instance-volatile :shortdesc: "The first ID in the instance's primary idmap range" :type: "integer" ``` ```{config:option} volatile.idmap.current instance-volatile :shortdesc: "The idmap currently in use by the instance" :type: "string" ``` ```{config:option} volatile.idmap.next instance-volatile :shortdesc: "The idmap to use the next time the instance starts" :type: "string" ``` ```{config:option} volatile.last_state.agent instance-volatile :shortdesc: "Last recorded instance agent state" :type: "string" ``` ```{config:option} volatile.last_state.agent.once instance-volatile :shortdesc: "Whether the agent has been seen at least once since last boot" :type: "bool" ``` ```{config:option} volatile.last_state.idmap instance-volatile :shortdesc: "Serialized instance UID/GID map" :type: "string" ``` ```{config:option} volatile.last_state.power instance-volatile :shortdesc: "Instance state as of last host shutdown" :type: "string" ``` ```{config:option} volatile.last_state.ready instance-volatile :shortdesc: "Instance marked itself as ready" :type: "string" ``` ```{config:option} volatile.rebalance.last_move instance-volatile :shortdesc: "Timestamp of last move by automatic live-migration" :type: "integer" ``` ```{config:option} volatile.uuid instance-volatile :shortdesc: "Instance UUID" :type: "string" The instance UUID is globally unique across all servers and projects. ``` ```{config:option} volatile.uuid.generation instance-volatile :shortdesc: "Instance generation UUID" :type: "string" The instance generation UUID changes whenever the instance's place in time moves backwards. It is globally unique across all servers and projects. ``` ```{config:option} volatile.vm.boot_state instance-volatile :shortdesc: "JSON encoded VM properties used during live migration and other state restoration." :type: "string" ``` ```{config:option} volatile.vm.needs_reset instance-volatile :shortdesc: "Indicates that the VM needs a full reset on next reboot" :type: "bool" ``` ```{config:option} volatile.vm.rtc_adjustment instance-volatile :shortdesc: "Real Time Clock change adjustment" :type: "int64" Real Time Clock adjustment time to allow virtual machines to run on a different base than the host. ``` ```{config:option} volatile.vm.rtc_offset instance-volatile :shortdesc: "Real Time Clock change offset" :type: "int64" Real Time Clock offset to allow virtual machines to run on a different base than the host. ``` ```{config:option} volatile.vsock_id instance-volatile :shortdesc: "Instance `vsock ID` used as of last start" :type: "string" ``` ```{config:option} limits.kernel.as kernel-limits :resource: "`RLIMIT_AS`" :shortdesc: "Maximum size of the process's virtual memory" :type: "string" ``` ```{config:option} limits.kernel.core kernel-limits :resource: "`RLIMIT_CORE`" :shortdesc: "Maximum size of the process's core dump file" :type: "string" ``` ```{config:option} limits.kernel.cpu kernel-limits :resource: "`RLIMIT_CPU`" :shortdesc: "Limit in seconds on the amount of CPU time the process can consume" :type: "string" ``` ```{config:option} limits.kernel.data kernel-limits :resource: "`RLIMIT_DATA`" :shortdesc: "Maximum size of the process's data segment" :type: "string" ``` ```{config:option} limits.kernel.fsize kernel-limits :resource: "`RLIMIT_FSIZE`" :shortdesc: "Maximum size of files the process may create" :type: "string" ``` ```{config:option} limits.kernel.locks kernel-limits :resource: "`RLIMIT_LOCKS`" :shortdesc: "Limit on the number of file locks that this process may establish" :type: "string" ``` ```{config:option} limits.kernel.memlock kernel-limits :resource: "`RLIMIT_MEMLOCK`" :shortdesc: "Limit on the number of bytes of memory that the process may lock in RAM" :type: "string" ``` ```{config:option} limits.kernel.nice kernel-limits :resource: "`RLIMIT_NICE`" :shortdesc: "Maximum value to which the process's nice value can be raised" :type: "string" ``` ```{config:option} limits.kernel.nofile kernel-limits :resource: "`RLIMIT_NOFILE`" :shortdesc: "Maximum number of open files for the process" :type: "string" ``` ```{config:option} limits.kernel.nproc kernel-limits :resource: "`RLIMIT_NPROC`" :shortdesc: "Maximum number of processes that can be created for the user of the calling process" :type: "string" ``` ```{config:option} limits.kernel.rtprio kernel-limits :resource: "`RLIMIT_RTPRIO`" :shortdesc: "Maximum value on the real-time-priority that may be set for this process" :type: "string" ``` ```{config:option} limits.kernel.sigpending kernel-limits :resource: "`RLIMIT_SIGPENDING`" :shortdesc: "Limit on the number of bytes of memory that the process may lock in RAM" :type: "string" ``` ```{config:option} user.* network_address_set-common :shortdesc: "Free form user key/value storage" :type: "string" User keys can be used in search. ``` ```{config:option} bgp.peers.NAME.address network_bridge-bgp :condition: "BGP server" :defaultdesc: "-" :shortdesc: "Peer address (IPv4 or IPv6) for use by `ovn` downstream networks" :type: "string" ``` ```{config:option} bgp.peers.NAME.asn network_bridge-bgp :condition: "BGP server" :defaultdesc: "-" :shortdesc: "Peer AS number for use by `ovn` downstream networks" :type: "integer" ``` ```{config:option} bgp.peers.NAME.holdtime network_bridge-bgp :condition: "BGP server" :defaultdesc: "`180`" :shortdesc: "Peer session hold time (in seconds; optional)" :type: "integer" ``` ```{config:option} bgp.peers.NAME.interface network_bridge-bgp :condition: "BGP server" :defaultdesc: "-" :shortdesc: "Interface name for BGP unnumbered peering (alternative to the peer address)" :type: "string" ``` ```{config:option} bgp.peers.NAME.password network_bridge-bgp :condition: "BGP server" :defaultdesc: "- (no password)" :shortdesc: "Peer session password (optional) for use by `ovn` downstream networks" :type: "string" ``` ```{config:option} bgp.communities network_bridge-common :condition: "BGP server" :default: "-" :scope: "global" :shortdesc: "Comma-separated BGP communities to attach to advertised prefixes" :type: "string" ``` ```{config:option} bgp.ipv4.instances network_bridge-common :condition: "BGP server" :default: "`false`" :shortdesc: "Whether to advertise a /32 route for the IPv4 address of each running instance" :type: "bool" ``` ```{config:option} bgp.ipv4.nexthop network_bridge-common :condition: "BGP server" :default: "local address" :scope: "local" :shortdesc: "Override the next-hop for advertised prefixes" :type: "string" ``` ```{config:option} bgp.ipv6.instances network_bridge-common :condition: "BGP server" :default: "`false`" :shortdesc: "Whether to advertise a /128 route for the IPv6 address of each running instance" :type: "bool" ``` ```{config:option} bgp.ipv6.nexthop network_bridge-common :condition: "BGP server" :default: "local address" :scope: "local" :shortdesc: "Override the next-hop for advertised prefixes" :type: "string" ``` ```{config:option} bridge.driver network_bridge-common :condition: "-" :default: "`native`" :shortdesc: "Bridge driver: `native` or `openvswitch`" :type: "string" ``` ```{config:option} bridge.external_interfaces network_bridge-common :condition: "-" :default: "-" :shortdesc: "Comma-separated list of unconfigured network interfaces to include in the bridge" :type: "string" ``` ```{config:option} bridge.hwaddr network_bridge-common :condition: "-" :default: "-" :shortdesc: "MAC address for the bridge" :type: "string" ``` ```{config:option} bridge.mtu network_bridge-common :condition: "-" :default: "`1500`" :shortdesc: "Bridge MTU (default varies if tunnel in use)" :type: "integer" ``` ```{config:option} bridge.multicast_snooping network_bridge-common :condition: "native bridge" :default: "`true`" :shortdesc: "Whether to enable multicast snooping on the bridge" :type: "bool" ``` ```{config:option} dns.domain network_bridge-common :condition: "-" :default: "`incus`" :shortdesc: "Domain to advertise to DHCP clients and use for DNS resolution" :type: "string" ``` ```{config:option} dns.include_hosts network_bridge-common :condition: "-" :default: "`true`" :shortdesc: "Whether to serve records from the host's `/etc/hosts` file" :type: "bool" ``` ```{config:option} dns.mode network_bridge-common :condition: "-" :default: "`managed`" :shortdesc: "DNS registration mode: none for no DNS record, managed for Incus-generated static records or dynamic for client-generated records" :type: "string" ``` ```{config:option} dns.nameservers network_bridge-common :condition: "-" :default: "IPv4 and IPv6 address" :shortdesc: "DNS server IPs to advertise to DHCP clients and via Router Advertisements. Both IPv4 and IPv6 addresses get pushed via DHCP, and IPv6 addresses are also advertised as RDNSS via RA." :type: "string" ``` ```{config:option} dns.search network_bridge-common :condition: "-" :default: "-" :shortdesc: "Full comma-separated domain search list, defaulting to `dns.domain` value" :type: "string" ``` ```{config:option} dns.zone.forward network_bridge-common :condition: "-" :default: "`managed`" :shortdesc: "Comma-separated list of DNS zone names for forward DNS records" :type: "string" ``` ```{config:option} dns.zone.reverse.ipv4 network_bridge-common :condition: "-" :default: "`managed`" :shortdesc: "DNS zone name for IPv4 reverse DNS records" :type: "string" ``` ```{config:option} dns.zone.reverse.ipv6 network_bridge-common :condition: "-" :default: "`managed`" :shortdesc: "DNS zone name for IPv6 reverse DNS records" :type: "string" ``` ```{config:option} ipv4.address network_bridge-common :condition: "standard mode" :default: "- (initial value on creation: `auto`)" :shortdesc: "IPv4 address for the bridge (use `none` to turn off IPv4 or `auto` to generate a new random unused subnet) (CIDR)" :type: "string" ``` ```{config:option} ipv4.dhcp network_bridge-common :condition: "IPv4 address" :default: "`true`" :shortdesc: "Whether to allocate addresses using DHCP" :type: "bool" ``` ```{config:option} ipv4.dhcp.expiry network_bridge-common :condition: "IPv4 DHCP" :default: "`1h`" :shortdesc: "When to expire DHCP leases" :type: "string" ``` ```{config:option} ipv4.dhcp.gateway network_bridge-common :condition: "IPv4 DHCP" :default: "IPv4 address" :shortdesc: "Address of the gateway for the subnet (use `none` to turn off gateway announcement)" :type: "string" ``` ```{config:option} ipv4.dhcp.ranges network_bridge-common :condition: "IPv4 DHCP" :default: "all addresses" :shortdesc: "Comma-separated list of IP ranges to use for DHCP (FIRST-LAST format)" :type: "string" ``` ```{config:option} ipv4.dhcp.routes network_bridge-common :condition: "IPv4 DHCP" :default: "-" :shortdesc: "Static routes to provide via DHCP option 121, as a comma-separated list of alternating subnets (CIDR) and gateway addresses (same syntax as dnsmasq)" :type: "string" ``` ```{config:option} ipv4.firewall network_bridge-common :condition: "IPv4 address" :default: "`true`" :shortdesc: "Whether to generate filtering firewall rules for this network" :type: "bool" ``` ```{config:option} ipv4.nat network_bridge-common :condition: "IPv4 address" :default: "`false`(initial value on creation if `ipv4.address` is set to `auto`: `true`)" :shortdesc: "Whether to NAT" :type: "bool" ``` ```{config:option} ipv4.nat.address network_bridge-common :condition: "IPv4 address" :default: "-" :shortdesc: "The source address used for outbound traffic from the bridge" :type: "string" ``` ```{config:option} ipv4.nat.order network_bridge-common :condition: "IPv4 address" :default: "`before`" :shortdesc: "Whether to add the required NAT rules before or after any pre-existing rules (no effect with the `nftables` firewall driver)" :type: "string" ``` ```{config:option} ipv4.ovn.ranges network_bridge-common :condition: "-" :default: "-" :shortdesc: "Comma-separated list of IPv4 ranges to use for child OVN network routers (FIRST-LAST format)" :type: "string" ``` ```{config:option} ipv4.routes network_bridge-common :condition: "IPv4 address" :default: "-" :shortdesc: "Comma-separated list of additional IPv4 CIDR subnets to route to the bridge" :type: "string" ``` ```{config:option} ipv4.routing network_bridge-common :condition: "IPv4 DHCP" :default: "`true`" :shortdesc: "Whether to route traffic in and out of the bridge" :type: "bool" ``` ```{config:option} ipv6.address network_bridge-common :condition: "standard mode" :default: "- (initial value on creation: `auto`)" :shortdesc: "IPv6 address for the bridge (use `none` to turn off IPv6 or `auto` to generate a new random unused subnet) (CIDR)" :type: "string" ``` ```{config:option} ipv6.dhcp network_bridge-common :condition: "IPv6 DHCP" :default: "`true`" :shortdesc: "Whether to provide additional network configuration over DHCP" :type: "bool" ``` ```{config:option} ipv6.dhcp.expiry network_bridge-common :condition: "IPv6 DHCP" :default: "`1h`" :shortdesc: "When to expire DHCP leases" :type: "string" ``` ```{config:option} ipv6.dhcp.ranges network_bridge-common :condition: "IPv6 stateful DHCP" :default: "all addresses" :shortdesc: "Comma-separated list of IPv6 ranges to use for DHCP (FIRST-LAST format)" :type: "string" ``` ```{config:option} ipv6.dhcp.stateful network_bridge-common :condition: "IPv6 DHCP" :default: "`false`" :shortdesc: "Whether to allocate addresses using DHCP" :type: "bool" ``` ```{config:option} ipv6.firewall network_bridge-common :condition: "IPv6 address" :default: "`true`" :shortdesc: "Whether to generate filtering firewall rules for this network" :type: "bool" ``` ```{config:option} ipv6.nat network_bridge-common :condition: "IPv6 address" :default: "`false` (initial value on creation if `ipv6.address` is set to `auto`: `true`)" :shortdesc: "Whether to NAT" :type: "bool" ``` ```{config:option} ipv6.nat.address network_bridge-common :condition: "IPv6 address" :default: "-" :shortdesc: "The source address used for outbound traffic from the bridge" :type: "string" ``` ```{config:option} ipv6.nat.order network_bridge-common :condition: "IPv6 address" :default: "`before`" :shortdesc: "Whether to add the required NAT rules before or after any pre-existing rules (no effect with the `nftables` firewall driver)" :type: "string" ``` ```{config:option} ipv6.ovn.ranges network_bridge-common :condition: "-" :default: "-" :shortdesc: "Comma-separated list of IPv6 ranges to use for child OVN network routers (FIRST-LAST format)" :type: "string" ``` ```{config:option} ipv6.ra network_bridge-common :condition: "IPv6 address" :default: "`true`" :shortdesc: "Whether to send IPv6 router advertisements" :type: "bool" ``` ```{config:option} ipv6.routes network_bridge-common :condition: "IPv6 address" :default: "-" :shortdesc: "Comma-separated list of additional IPv6 CIDR subnets to route to the bridge" :type: "string" ``` ```{config:option} ipv6.routing network_bridge-common :condition: "IPv6 address" :default: "`true`" :shortdesc: "Whether to route traffic in and out of the bridge" :type: "bool" ``` ```{config:option} raw.dnsmasq network_bridge-common :condition: "-" :default: "-" :shortdesc: "Additional dnsmasq configuration to append to the configuration file" :type: "string" ``` ```{config:option} security.acls network_bridge-common :condition: "-" :default: "-" :shortdesc: "Comma-separated list of Network ACLs to apply to NICs connected to this network (see {ref}`network-acls-bridge-limitations`)" :type: "string" ``` ```{config:option} security.acls.default.egress.action network_bridge-common :condition: "`security.acls`" :default: "`reject`" :shortdesc: "Action to use for egress traffic that doesn't match any ACL rule" :type: "string" ``` ```{config:option} security.acls.default.egress.logged network_bridge-common :condition: "`security.acls`" :default: "`false`" :shortdesc: "Whether to log egress traffic that doesn't match any ACL rule" :type: "bool" ``` ```{config:option} security.acls.default.ingress.action network_bridge-common :condition: "`security.acls`" :default: "`reject`" :shortdesc: "Action to use for ingress traffic that doesn't match any ACL rule" :type: "string" ``` ```{config:option} security.acls.default.ingress.logged network_bridge-common :condition: "`security.acls`" :default: "`false`" :shortdesc: "Whether to log ingress traffic that doesn't match any ACL rule" :type: "bool" ``` ```{config:option} tunnel.NAME.group network_bridge-common :condition: "`vxlan`" :default: "`239.0.0.1`" :shortdesc: "Multicast address for `vxlan` (used if local and remote aren't set)" :type: "string" ``` ```{config:option} tunnel.NAME.id network_bridge-common :condition: "`vxlan`" :default: "`0`" :shortdesc: "Specific tunnel ID to use for the `vxlan` tunnel" :type: "integer" ``` ```{config:option} tunnel.NAME.interface network_bridge-common :condition: "`vxlan`" :default: "-" :shortdesc: "Specific host interface to use for the tunnel" :type: "string" ``` ```{config:option} tunnel.NAME.local network_bridge-common :condition: "`gre` or `vxlan`" :default: "-" :shortdesc: "Local address for the tunnel (not necessary for multicast `vxlan`)" :type: "string" ``` ```{config:option} tunnel.NAME.port network_bridge-common :condition: "`vxlan`" :default: "`0`" :shortdesc: "Specific port to use for the `vxlan` tunnel" :type: "integer" ``` ```{config:option} tunnel.NAME.protocol network_bridge-common :condition: "standard mode" :default: "-" :shortdesc: "Tunneling protocol: `vxlan` or `gre`" :type: "string" ``` ```{config:option} tunnel.NAME.remote network_bridge-common :condition: "`gre` or `vxlan`" :default: "-" :shortdesc: "Remote address for the tunnel (not necessary for multicast `vxlan`)" :type: "string" ``` ```{config:option} tunnel.NAME.ttl network_bridge-common :condition: "`vxlan`" :default: "`1`" :shortdesc: "Specific TTL to use for multicast routing topologies" :type: "integer" ``` ```{config:option} user.* network_bridge-common :condition: "-" :default: "-" :shortdesc: "User-provided free-form key/value pairs" :type: "string" ``` ```{config:option} target_address network_forward-common :shortdesc: "Default target address for anything not covered through a port definition" :type: "string" ``` ```{config:option} user.* network_forward-common :shortdesc: "User defined key/value configuration" :type: "string" ``` ```{config:option} user.* network_integration-common :shortdesc: "Free form user key/value storage" :type: "string" User keys can be used in search. ``` ```{config:option} ovn.ca_cert network_integration-ovn :scope: "global" :shortdesc: "OVN SSL certificate authorities for the inter-connection database (PEM bundle)" :type: "string" ``` ```{config:option} ovn.client_cert network_integration-ovn :scope: "global" :shortdesc: "OVN SSL client certificate (PEM bundle)" :type: "string" ``` ```{config:option} ovn.client_key network_integration-ovn :scope: "global" :shortdesc: "OVN SSL client key (PEM private key)" :type: "string" ``` ```{config:option} ovn.northbound_connection network_integration-ovn :scope: "global" :shortdesc: "OVN northbound inter-connection connection string" :type: "string" ``` ```{config:option} ovn.southbound_connection network_integration-ovn :scope: "global" :shortdesc: "OVN southbound inter-connection connection string" :type: "string" ``` ```{config:option} ovn.transit.pattern network_integration-ovn :defaultdesc: "`ts-incus-{{ integrationName }}-{{ projectName }}-{{ networkName }}`" :shortdesc: "Template for the transit switch name" :type: "string" Specify a Pongo2 template string that represents the transit switch name. This template gets access to the project name (`projectName`), integration name (`integrationName`), network name (`networkName`) and peer name (`peerName`). ``` ```{config:option} healthcheck network_load_balancer-common :defaultdesc: "`false`" :shortdesc: "Whether to perform checks on the backends" :type: "bool" ``` ```{config:option} healthcheck.failure_count network_load_balancer-common :defaultdesc: "`3`" :shortdesc: "Number of failed tests to consider the backend offline" :type: "integer" ``` ```{config:option} healthcheck.interval network_load_balancer-common :defaultdesc: "`10`" :shortdesc: "Interval in seconds between health checks" :type: "integer" ``` ```{config:option} healthcheck.success_count network_load_balancer-common :defaultdesc: "`3`" :shortdesc: "Number of successful tests to consider the backend online" :type: "integer" ``` ```{config:option} healthcheck.timeout network_load_balancer-common :defaultdesc: "`30`" :shortdesc: "Test timeout" :type: "integer" ``` ```{config:option} user.* network_load_balancer-common :shortdesc: "Free form user key/value storage" :type: "string" User keys can be used in search. ``` ```{config:option} gvrp network_macvlan-common :condition: "-" :default: "`false`" :shortdesc: "Register VLAN using GARP VLAN Registration Protocol" :type: "bool" ``` ```{config:option} mtu network_macvlan-common :condition: "-" :shortdesc: "The MTU of the new interface" :type: "int" ``` ```{config:option} parent network_macvlan-common :condition: "-" :scope: "local" :shortdesc: "Parent interface to create macvlan NICs on" :type: "string" ``` ```{config:option} user.* network_macvlan-common :shortdesc: "User-provided free-form key/value pairs" :type: "string" ``` ```{config:option} vlan network_macvlan-common :condition: "-" :shortdesc: "The VLAN ID to attach to" :type: "int" ``` ```{config:option} bgp.communities network_ovn-common :condition: "BGP server" :default: "-" :shortdesc: "Comma-separated BGP communities to attach to advertised prefixes" :type: "string" ``` ```{config:option} bridge.external_interfaces network_ovn-common :scope: "local" :shortdesc: "Comma-separated list of unconfigured network interfaces to include in the bridge" :type: "string" ``` ```{config:option} bridge.hwaddr network_ovn-common :shortdesc: "MAC address for the virtual bridge interface" :type: "string" ``` ```{config:option} bridge.mtu network_ovn-common :default: "`1442`" :shortdesc: "Bridge MTU (default allows host to host Geneve tunnels)" :type: "integer" ``` ```{config:option} bridge.multicast_relay network_ovn-common :default: "`false`" :shortdesc: "Whether to relay multicast traffic through the network's logical router" :type: "bool" ``` ```{config:option} bridge.multicast_snooping network_ovn-common :default: "`true`" :shortdesc: "Whether to enable multicast snooping on the virtual network" :type: "bool" ``` ```{config:option} dns.domain network_ovn-common :default: "`incus`" :shortdesc: "Domain to advertise to DHCP clients and use for DNS resolution" :type: "string" ``` ```{config:option} dns.mode network_ovn-common :condition: "-" :default: "`managed`" :shortdesc: "DNS registration mode: none for no DNS record, managed for OVN managed records" :type: "string" ``` ```{config:option} dns.nameservers network_ovn-common :default: "Uplink DNS servers (IPv4 and IPv6 address if no uplink is configured)" :shortdesc: "DNS server IPs to advertise to DHCP clients and via Router Advertisements. Both IPv4 and IPv6 addresses get pushed via DHCP, and the first IPv6 address is also advertised as RDNSS via RA." :type: "string" ``` ```{config:option} dns.search network_ovn-common :shortdesc: "Full comma-separated domain search list, defaulting to `dns.domain` value" :type: "string" ``` ```{config:option} dns.zone.forward network_ovn-common :shortdesc: "Comma-separated list of DNS zone names for forward DNS records" :type: "string" ``` ```{config:option} dns.zone.reverse.ipv4 network_ovn-common :shortdesc: "DNS zone name for IPv4 reverse DNS records" :type: "string" ``` ```{config:option} dns.zone.reverse.ipv6 network_ovn-common :shortdesc: "DNS zone name for IPv6 reverse DNS records" :type: "string" ``` ```{config:option} ipv4.address network_ovn-common :condition: "standard mode" :default: "(initial value on creation: `auto`)" :shortdesc: "IPv4 address for the bridge (use `none` to turn off IPv4 or `auto` to generate a new random unused subnet) (CIDR)" :type: "string" ``` ```{config:option} ipv4.dhcp network_ovn-common :condition: "IPv4 address" :default: "`true`" :shortdesc: "Whether to allocate addresses using DHCP" :type: "bool" ``` ```{config:option} ipv4.dhcp.expiry network_ovn-common :condition: "IPv4 DHCP" :default: "`1h`" :shortdesc: "When to expire DHCP leases" :type: "string" ``` ```{config:option} ipv4.dhcp.gateway network_ovn-common :condition: "IPv4 DHCP" :default: "IPv4 address" :shortdesc: "Address of the gateway for the subnet (use `none` to turn off gateway announcement)" :type: "string" ``` ```{config:option} ipv4.dhcp.ranges network_ovn-common :condition: "IPv4 DHCP" :default: "all addresses" :shortdesc: "Comma-separated list of IP ranges to use for DHCP (FIRST-LAST format)" :type: "string" ``` ```{config:option} ipv4.dhcp.routes network_ovn-common :condition: "IPv4 DHCP" :shortdesc: "Static routes to provide via DHCP option 121, as a comma-separated list of alternating subnets (CIDR) and gateway addresses (same syntax as dnsmasq and OVN)" :type: "string" ``` ```{config:option} ipv4.l3only network_ovn-common :condition: "IPv4 address" :default: "`false`" :shortdesc: "Whether to enable layer 3 only mode." :type: "bool" ``` ```{config:option} ipv4.nat network_ovn-common :condition: "IPv4 address" :default: "`false` initial value on creation if `ipv4.address` is set to `auto: true`)" :shortdesc: "Whether to NAT" :type: "bool" ``` ```{config:option} ipv4.nat.address network_ovn-common :condition: "IPv4 address" :shortdesc: "The source address used for outbound traffic from the network" :type: "string" ``` ```{config:option} ipv6.address network_ovn-common :condition: "standard mode" :default: "(initial value on creation: `auto`)" :shortdesc: "IPv6 address for the bridge (use `none` to turn off IPv6 or `auto` to generate a new random unused subnet) (CIDR)" :type: "string" ``` ```{config:option} ipv6.dhcp network_ovn-common :condition: "IPv6 address" :default: "`true`" :shortdesc: "Whether to provide additional network configuration over DHCP" :type: "bool" ``` ```{config:option} ipv6.dhcp.stateful network_ovn-common :condition: "IPv6 DHCP" :default: "`false`" :shortdesc: "Whether to allocate addresses using DHCP" :type: "bool" ``` ```{config:option} ipv6.l3only network_ovn-common :condition: "IPv6 DHCP stateful" :default: "`false`" :shortdesc: "Whether to enable layer 3 only mode." :type: "bool" ``` ```{config:option} ipv6.nat network_ovn-common :condition: "IPv6 address" :default: "`false` (initial value on creation if `ipv6.address` is set to `auto: true`)" :shortdesc: "Whether to NAT" :type: "bool" ``` ```{config:option} ipv6.nat.address network_ovn-common :condition: "IPv6 address" :shortdesc: "The source address used for outbound traffic from the network" :type: "string" ``` ```{config:option} ipv6.ra network_ovn-common :condition: "IPv6 address" :default: "`true`" :shortdesc: "Whether to send IPv6 router advertisements" :type: "bool" ``` ```{config:option} network network_ovn-common :shortdesc: "Uplink network to use for external network access or `none` to keep isolated" :type: "string" ``` ```{config:option} parent network_ovn-common :shortdesc: "Parent OVN network whose logical router and uplink to share" :type: "string" ``` ```{config:option} security.acls network_ovn-common :shortdesc: "Comma-separated list of Network ACLs to apply to NICs connected to this network" :type: "string" ``` ```{config:option} security.acls.default.egress.action network_ovn-common :condition: "`security.acls`" :default: "`reject`" :shortdesc: "Action to use for egress traffic that doesn't match any ACL rule" :type: "string" ``` ```{config:option} security.acls.default.egress.logged network_ovn-common :condition: "`security.acls`" :default: "`false`" :shortdesc: "Whether to log egress traffic that doesn't match any ACL rule" :type: "bool" ``` ```{config:option} security.acls.default.ingress.action network_ovn-common :condition: "`security.acls`" :default: "`reject`" :shortdesc: "Action to use for ingress traffic that doesn't match any ACL rule" :type: "string" ``` ```{config:option} security.acls.default.ingress.logged network_ovn-common :condition: "`security.acls`" :default: "`false`" :shortdesc: "Whether to log ingress traffic that doesn't match any ACL rule" :type: "bool" ``` ```{config:option} tunnel.NAME.group network_ovn-common :condition: "`vxlan`" :default: "`239.0.0.1`" :shortdesc: "Multicast address for `vxlan`" :type: "string" ``` ```{config:option} tunnel.NAME.id network_ovn-common :condition: "`vxlan`" :default: "`0`" :shortdesc: "Specific tunnel ID to use for the `vxlan` tunnel" :type: "integer" ``` ```{config:option} tunnel.NAME.interface network_ovn-common :condition: "`vxlan`" :default: "-" :shortdesc: "Specific host interface to use for the tunnel" :type: "string" ``` ```{config:option} tunnel.NAME.local network_ovn-common :condition: "`gre`" :default: "-" :shortdesc: "Local address for the tunnel" :type: "string" ``` ```{config:option} tunnel.NAME.port network_ovn-common :condition: "`vxlan`" :default: "`0`" :shortdesc: "Specific port to use for the `vxlan` tunnel" :type: "integer" ``` ```{config:option} tunnel.NAME.protocol network_ovn-common :condition: "standard mode" :default: "-" :shortdesc: "Tunneling protocol: `vxlan` or `gre`" :type: "string" ``` ```{config:option} tunnel.NAME.remote network_ovn-common :condition: "`gre`" :default: "-" :shortdesc: "Remote address for the tunnel" :type: "string" ``` ```{config:option} tunnel.NAME.ttl network_ovn-common :condition: "`vxlan`" :default: "`1`" :shortdesc: "Specific TTL to use for multicast routing topologies" :type: "integer" ``` ```{config:option} user.* network_ovn-common :shortdesc: "User-provided free-form key/value pairs" :type: "string" ``` ```{config:option} bgp.peers.NAME.address network_physical-bgp :condition: "BGP server" :defaultdesc: "-" :shortdesc: "Peer address (IPv4 or IPv6) for use by `ovn` downstream networks" :type: "string" ``` ```{config:option} bgp.peers.NAME.asn network_physical-bgp :condition: "BGP server" :defaultdesc: "-" :shortdesc: "Peer AS number for use by `ovn` downstream networks" :type: "integer" ``` ```{config:option} bgp.peers.NAME.holdtime network_physical-bgp :condition: "BGP server" :defaultdesc: "`180`" :shortdesc: "Peer session hold time (in seconds; optional)" :type: "integer" ``` ```{config:option} bgp.peers.NAME.interface network_physical-bgp :condition: "BGP server" :defaultdesc: "-" :shortdesc: "Interface name for BGP unnumbered peering (alternative to the peer address)" :type: "string" ``` ```{config:option} bgp.peers.NAME.password network_physical-bgp :condition: "BGP server" :defaultdesc: "- (no password)" :shortdesc: "Peer session password (optional) for use by `ovn` downstream networks" :type: "string" ``` ```{config:option} gvrp network_physical-common :condition: "-" :defaultdesc: "'false'" :shortdesc: "Register VLAN using GARP VLAN Registration Protocol" :type: "bool" ``` ```{config:option} mtu network_physical-common :condition: "-" :shortdesc: "The MTU of the new interface" :type: "integer" ``` ```{config:option} parent network_physical-common :condition: "-" :scope: "local" :shortdesc: "Existing interface to use for network" :type: "string" ``` ```{config:option} vlan network_physical-common :condition: "-" :shortdesc: "The VLAN ID to attach to" :type: "integer" ``` ```{config:option} vlan.tagged network_physical-common :condition: "Parent must be an existing bridge" :shortdesc: "Comma-delimited list of VLAN IDs or VLAN ranges to join for tagged traffic" :type: "integer" ``` ```{config:option} dns.nameservers network_physical-dns :condition: "standard mode" :shortdesc: "List of DNS server IPs on `physical` network" :type: "string" ``` ```{config:option} ipv4.gateway network_physical-ipv4 :condition: "standard mode" :shortdesc: "IPv4 address for the gateway and network (CIDR)" :type: "string" ``` ```{config:option} ipv4.gateway.hwaddr network_physical-ipv4 :shortdesc: "MAC address of the gateway (to avoid discovery)" :type: "string" ``` ```{config:option} ipv4.ovn.ranges network_physical-ipv4 :condition: "-" :shortdesc: "Comma-separated list of IPv4 ranges to use for child OVN network routers (FIRST-LAST format)" :type: "string" ``` ```{config:option} ipv4.routes network_physical-ipv4 :condition: "IPv4 address" :shortdesc: "Comma-separated list of additional IPv4 CIDR subnets that can be used with child OVN networks `ipv4.routes.external` setting" :type: "string" ``` ```{config:option} ipv4.routes.anycast network_physical-ipv4 :condition: "IPv4 address" :defaultdesc: "'false'" :shortdesc: "Allow the overlapping routes to be used on multiple networks/NIC at the same time" :type: "bool" ``` ```{config:option} ipv6.gateway network_physical-ipv6 :condition: "standard mode" :shortdesc: "IPv6 address for the gateway and network (CIDR)" :type: "string" ``` ```{config:option} ipv6.gateway.hwaddr network_physical-ipv6 :shortdesc: "MAC address of the gateway (to avoid discovery)" :type: "string" ``` ```{config:option} ipv6.ovn.ranges network_physical-ipv6 :condition: "-" :shortdesc: "Comma-separated list of IPv6 ranges to use for child OVN network routers (FIRST-LAST format)" :type: "string" ``` ```{config:option} ipv6.routes network_physical-ipv6 :condition: "IPv6 address" :shortdesc: "Comma-separated list of additional IPv6 CIDR subnets that can be used with child OVN networks `ipv6.routes.external` setting" :type: "string" ``` ```{config:option} ipv6.routes.anycast network_physical-ipv6 :condition: "IPv6 address" :defaultdesc: "'false'" :shortdesc: "Allow the overlapping routes to be used on multiple networks/NIC at the same time" :type: "bool" ``` ```{config:option} ovn.ingress_mode network_physical-ovn :condition: "standard mode" :defaultdesc: "`l2proxy`" :shortdesc: "Sets the method how OVN NIC external IPs will be advertised on uplink network: `l2proxy` (proxy ARP/NDP) or `routed`" :type: "string" ``` ```{config:option} mtu network_sriov-common :condition: "-" :shortdesc: "The MTU of the new interface" :type: "integer" ``` ```{config:option} parent network_sriov-common :condition: "-" :scope: "local" :shortdesc: "Parent interface to create `sriov` NICs on" :type: "string" ``` ```{config:option} user.* network_sriov-common :condition: "-" :shortdesc: "User-provided free-form key/value pairs" :type: "string" ``` ```{config:option} vlan network_sriov-common :condition: "-" :shortdesc: "The VLAN ID to attach to" :type: "integer" ``` ```{config:option} dns.contact network_zone-common :required: "no" :shortdesc: "Admin contact email for DNS server" :type: "string" ``` ```{config:option} dns.nameservers network_zone-common :required: "no" :shortdesc: "Comma-separated list of DNS server FQDNs (for NS records)" :type: "string set" ``` ```{config:option} network.nat network_zone-common :defaultdesc: "`true`" :required: "no" :shortdesc: "Whether to generate records for NAT-ed subnets" :type: "bool" ``` ```{config:option} peers.NAME.address network_zone-common :required: "no" :shortdesc: "IP address of a DNS server" :type: "string" ``` ```{config:option} peers.NAME.key network_zone-common :required: "no" :shortdesc: "TSIG key for the server" :type: "string" ``` ```{config:option} user.* network_zone-common :required: "no" :shortdesc: "User-provided free-form key/value pairs" :type: "string" ``` ```{config:option} features.images project-features :defaultdesc: "`false`" :initialvaluedesc: "`true`" :shortdesc: "Whether to use a separate set of images for the project" :type: "bool" This setting applies to both images and image aliases. ``` ```{config:option} features.networks project-features :defaultdesc: "`false`" :initialvaluedesc: "`false`" :shortdesc: "Whether to use a separate set of networks for the project" :type: "bool" This feature requires the server to be configured for OVN. ``` ```{config:option} features.networks.zones project-features :defaultdesc: "`false`" :initialvaluedesc: "`false`" :shortdesc: "Whether to use a separate set of network zones for the project" :type: "bool" ``` ```{config:option} features.profiles project-features :defaultdesc: "`false`" :initialvaluedesc: "`true`" :shortdesc: "Whether to use a separate set of profiles for the project" :type: "bool" ``` ```{config:option} features.storage.buckets project-features :defaultdesc: "`false`" :initialvaluedesc: "`true`" :shortdesc: "Whether to use a separate set of storage buckets for the project" :type: "bool" ``` ```{config:option} features.storage.volumes project-features :defaultdesc: "`false`" :initialvaluedesc: "`true`" :shortdesc: "Whether to use a separate set of storage volumes for the project" :type: "bool" ``` ```{config:option} limits.containers project-limits :shortdesc: "Maximum number of containers that can be created in the project" :type: "integer" ``` ```{config:option} limits.cpu project-limits :shortdesc: "Maximum number of CPUs to use in the project" :type: "integer" This value is the maximum value for the sum of the individual {config:option}`instance-resource-limits:limits.cpu` configurations set on the instances of the project. ``` ```{config:option} limits.disk project-limits :shortdesc: "Maximum disk space used by the project" :type: "string" This value is the maximum value of the aggregate disk space used by all instance volumes, custom volumes, and images of the project. ``` ```{config:option} limits.disk.pool.POOL_NAME project-limits :shortdesc: "Maximum disk space used by the project on this pool" :type: "string" This value is the maximum value of the aggregate disk space used by all instance volumes, custom volumes, and images of the project on this specific storage pool. ``` ```{config:option} limits.instances project-limits :shortdesc: "Maximum number of instances that can be created in the project" :type: "integer" ``` ```{config:option} limits.memory project-limits :shortdesc: "Usage limit for the host's memory for the project" :type: "string" The value is the maximum value for the sum of the individual {config:option}`instance-resource-limits:limits.memory` configurations set on the instances of the project. ``` ```{config:option} limits.networks project-limits :shortdesc: "Maximum number of networks that the project can have" :type: "integer" ``` ```{config:option} limits.processes project-limits :shortdesc: "Maximum number of processes within the project" :type: "integer" This value is the maximum value for the sum of the individual {config:option}`instance-resource-limits:limits.processes` configurations set on the instances of the project. ``` ```{config:option} limits.virtual-machines project-limits :shortdesc: "Maximum number of VMs that can be created in the project" :type: "integer" ``` ```{config:option} restricted project-restricted :defaultdesc: "`false`" :shortdesc: "Whether to block access to security-sensitive features" :type: "bool" This option must be enabled to allow the `restricted.*` keys to take effect. To temporarily remove the restrictions, you can disable this option instead of clearing the related keys. ``` ```{config:option} restricted.backups project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent creating instance or volume backups" :type: "string" Possible values are `allow` or `block`. ``` ```{config:option} restricted.cluster.groups project-restricted :shortdesc: "Cluster groups that can be targeted" :type: "string" If specified, this option prevents targeting cluster groups other than the provided ones. ``` ```{config:option} restricted.cluster.target project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent targeting of cluster members" :type: "string" Possible values are `allow` or `block`. When set to `allow`, this option allows targeting of cluster members (either directly or via a group) when creating or moving instances. ``` ```{config:option} restricted.containers.interception project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent using system call interception options" :type: "string" Possible values are `allow`, `block`, or `full`. When set to `allow`, interception options that are usually safe are allowed. File system mounting remains blocked. ``` ```{config:option} restricted.containers.lowlevel project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent using low-level container options" :type: "string" Possible values are `allow` or `block`. When set to `allow`, low-level container options like {config:option}`instance-raw:raw.lxc`, {config:option}`instance-raw:raw.idmap`, `volatile.*`, etc. can be used. ``` ```{config:option} restricted.containers.nesting project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent running nested Incus" :type: "string" Possible values are `allow` or `block`. When set to `allow`, {config:option}`instance-security:security.nesting` can be set to `true` for an instance. ``` ```{config:option} restricted.containers.privilege project-restricted :defaultdesc: "`unprivileged`" :shortdesc: "Which settings for privileged containers to prevent" :type: "string" Possible values are `unprivileged`, `isolated`, and `allow`. - When set to `unprivileged`, this option prevents setting {config:option}`instance-security:security.privileged` to `true`. - When set to `isolated`, this option prevents setting {config:option}`instance-security:security.privileged` to `true` and {config:option}`instance-security:security.idmap.isolated` to `false`. - When set to `allow`, there is no restriction. ``` ```{config:option} restricted.devices.disk project-restricted :defaultdesc: "`managed`" :shortdesc: "Which disk devices can be used" :type: "string" Possible values are `allow`, `block`, or `managed`. - When set to `block`, this option prevents using all disk devices except the root one. - When set to `managed`, this option allows using disk devices only if `pool=` is set. - When set to `allow`, there is no restriction on which disk devices can be used. ``` ```{config:option} restricted.devices.disk.paths project-restricted :shortdesc: "Which `source` can be used for `disk` devices" :type: "string" If {config:option}`project-restricted:restricted.devices.disk` is set to `allow`, this option controls which `source` can be used for `disk` devices. Specify a comma-separated list of path prefixes that restrict the `source` setting. If this option is left empty, all paths are allowed. ``` ```{config:option} restricted.devices.gpu project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent using devices of type `gpu`" :type: "string" Possible values are `allow` or `block`. ``` ```{config:option} restricted.devices.infiniband project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent using devices of type `infiniband`" :type: "string" Possible values are `allow` or `block`. ``` ```{config:option} restricted.devices.nic project-restricted :defaultdesc: "`managed`" :shortdesc: "Which network devices can be used" :type: "string" Possible values are `allow`, `block`, or `managed`. - When set to `block`, this option prevents using all network devices. - When set to `managed`, this option allows using network devices only if `network=` is set. - When set to `allow`, there is no restriction on which network devices can be used. ``` ```{config:option} restricted.devices.pci project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent using devices of type `pci`" :type: "string" Possible values are `allow` or `block`. ``` ```{config:option} restricted.devices.proxy project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent using devices of type `proxy`" :type: "string" Possible values are `allow` or `block`. ``` ```{config:option} restricted.devices.unix-block project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent using devices of type `unix-block`" :type: "string" Possible values are `allow` or `block`. ``` ```{config:option} restricted.devices.unix-char project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent using devices of type `unix-char`" :type: "string" Possible values are `allow` or `block`. ``` ```{config:option} restricted.devices.unix-hotplug project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent using devices of type `unix-hotplug`" :type: "string" Possible values are `allow` or `block`. ``` ```{config:option} restricted.devices.usb project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent using devices of type `usb`" :type: "string" Possible values are `allow` or `block`. ``` ```{config:option} restricted.idmap.gid project-restricted :shortdesc: "Which host GID ranges are allowed in `raw.idmap`" :type: "string" This option specifies the host GID ranges that are allowed in the instance's {config:option}`instance-raw:raw.idmap` setting. ``` ```{config:option} restricted.idmap.uid project-restricted :shortdesc: "Which host UID ranges are allowed in `raw.idmap`" :type: "string" This option specifies the host UID ranges that are allowed in the instance's {config:option}`instance-raw:raw.idmap` setting. ``` ```{config:option} restricted.images.servers project-restricted :shortdesc: "Which image servers (HTTP host) are allowed for us in this project" :type: "string" Specify a comma-delimited list of image servers domains that are allowed for use in this project. If this option is not set, all image servers are accessible. ``` ```{config:option} restricted.networks.access project-restricted :shortdesc: "Which network names are allowed for use in this project" :type: "string" Specify a comma-delimited list of network names that are allowed for use in this project. If this option is not set, all networks are accessible. In restricted projects with {config:option}`project-features:features.networks` enabled, the listed networks from the default project are shared into the project and their names can't be used for the project's own networks. Note that this setting depends on the {config:option}`project-restricted:restricted.devices.nic` setting. ``` ```{config:option} restricted.networks.integrations project-restricted :shortdesc: "Which network integrations can be used in this project" :type: "string" Specify a comma-delimited list of network integrations that can be used by networks in this project. ``` ```{config:option} restricted.networks.subnets project-restricted :defaultdesc: "`block`" :shortdesc: "Which network subnets are allocated for use in this project" :type: "string" Specify a comma-delimited list of network subnets from the uplink networks that are allocated for use in this project. Use the form `:`. ``` ```{config:option} restricted.networks.uplinks project-restricted :shortdesc: "Which network names can be used as uplink in this project" :type: "string" Specify a comma-delimited list of network names that can be used as uplink for networks in this project. ``` ```{config:option} restricted.networks.zones project-restricted :defaultdesc: "`block`" :shortdesc: "Which network zones can be used in this project" :type: "string" Specify a comma-delimited list of network zones that can be used (or something under them) in this project. ``` ```{config:option} restricted.snapshots project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent creating instance or volume snapshots" :type: "string" ``` ```{config:option} restricted.storage-pools.access project-restricted :shortdesc: "Which storage pool names are allowed for use in this project" :type: "string" Specify a comma-delimited list of storage pool names that are allowed for use in this project. If this option is not set, all storage pools are accessible. ``` ```{config:option} restricted.virtual-machines.lowlevel project-restricted :defaultdesc: "`block`" :shortdesc: "Whether to prevent using low-level VM options" :type: "string" Possible values are `allow` or `block`. When set to `allow`, low-level VM options like {config:option}`instance-raw:raw.qemu`, `volatile.*`, etc. can be used. ``` ```{config:option} restricted.virtual-machines.nesting project-restricted :defaultdesc: "`allow`" :shortdesc: "Whether to prevent using nested virtualization" :type: "string" Possible values are `allow` or `block`. When set to `block`, {config:option}`instance-security:security.nesting` must be set to `false` on all virtual machines, turning off nested virtualization. ``` ```{config:option} backups.compression_algorithm project-specific :shortdesc: "Compression algorithm to use for backups" :type: "string" Specify which compression algorithm to use for backups in this project. Possible values are `bzip2`, `gzip`, `lz4`, `lzma`, `xz`, `zstd` or `none`. ``` ```{config:option} images.auto_update_cached project-specific :shortdesc: "Whether to automatically update cached images in the project" :type: "bool" ``` ```{config:option} images.auto_update_interval project-specific :shortdesc: "Interval at which to look for updates to cached images" :type: "integer" Specify the interval in hours. To disable looking for updates to cached images, set this option to `0`. ``` ```{config:option} images.compression_algorithm project-specific :shortdesc: "Compression algorithm to use for new images in the project" :type: "string" Possible values are `bzip2`, `gzip`, `lz4`, `lzma`, `xz`, `zstd` or `none`. ``` ```{config:option} images.default_architecture project-specific :shortdesc: "Default architecture to use in a mixed-architecture cluster" :type: "string" ``` ```{config:option} images.remote_cache_expiry project-specific :shortdesc: "When an unused cached remote image is flushed in the project" :type: "integer" Specify the number of days after which the unused cached image expires. ``` ```{config:option} network.hwaddr_pattern project-specific :scope: "global" :shortdesc: "MAC address template" :type: "string" Specify a MAC address template, e.g. `10:66:6a:xx:xx:xx`, to use within the cluster. Every `x` in the template will be replaced by a random character in `0`–`f`. Beware of the birthday paradox! A single `xx` block leads to a 10% collision probability with only 8 addresses; for a double `xx:xx` block, 118 addresses; for a triple `xx:xx:xx` block, 1881; for a quadruple `xx:xx:xx:xx` block, 30084. We provide absolutely no guardrail against that. ``` ```{config:option} user.* project-specific :shortdesc: "User-provided free-form key/value pairs" :type: "string" ``` ```{config:option} acme.agree_tos server-acme :defaultdesc: "`false`" :scope: "global" :shortdesc: "Agree to ACME terms of service" :type: "bool" ``` ```{config:option} acme.ca_url server-acme :defaultdesc: "`https://acme-v02.api.letsencrypt.org/directory`" :scope: "global" :shortdesc: "URL to the directory resource of the ACME service" :type: "string" ``` ```{config:option} acme.challenge server-acme :defaultdesc: "`HTTP-01`" :scope: "global" :shortdesc: "ACME challenge type to use" :type: "string" Possible values are `DNS-01` and `HTTP-01`. ``` ```{config:option} acme.domain server-acme :scope: "global" :shortdesc: "Domain for which the certificate is issued" :type: "string" Comma separated list of domains, wildcard domains (`*.example.com`) require the `DNS-01` challenge. ``` ```{config:option} acme.eab.hmac server-acme :scope: "global" :shortdesc: "HMAC key for External Account Binding" :type: "string" ``` ```{config:option} acme.eab.kid server-acme :scope: "global" :shortdesc: "Key identifier for External Account Binding" :type: "string" ``` ```{config:option} acme.email server-acme :scope: "global" :shortdesc: "Email address used for the account registration" :type: "string" ``` ```{config:option} acme.http.port server-acme :defaultdesc: "`:80`" :scope: "global" :shortdesc: "Port and interface for HTTP server (used by HTTP-01)" :type: "string" Set the port and interface to use for HTTP-01 based challenges to listen on ``` ```{config:option} acme.provider server-acme :defaultdesc: "``" :scope: "global" :shortdesc: "Backend provider for the challenge (used by DNS-01)" :type: "string" ``` ```{config:option} acme.provider.environment server-acme :defaultdesc: "``" :scope: "global" :shortdesc: "Environment variables to set during the challenge (used by DNS-01)" :type: "string" ``` ```{config:option} acme.provider.resolvers server-acme :defaultdesc: "``" :scope: "global" :shortdesc: "Comma-separated list of DNS resolvers (used by DNS-01)" :type: "string" DNS resolvers to use for performing (recursive) `CNAME` resolving and apex domain determination during DNS-01 challenge. ``` ```{config:option} authorization.client.default server-authorization :scope: "global" :shortdesc: "Authorization driver for clients without a more specific class route" :type: "string" Routes clients that do not match a more specific class to an authorization driver. Possible values are `allow`, `deny`, `openfga` and `scriptlet`. ``` ```{config:option} authorization.client.oidc server-authorization :scope: "global" :shortdesc: "Authorization driver for OIDC-authenticated clients" :type: "string" Routes OIDC-authenticated clients to an authorization driver. Possible values are `allow`, `deny`, `openfga` and `scriptlet`. ``` ```{config:option} authorization.client.tls server-authorization :scope: "global" :shortdesc: "Authorization driver for unrestricted TLS clients" :type: "string" Routes clients using an unrestricted client certificate to an authorization driver. Possible values are `allow`, `deny`, `openfga` and `scriptlet`. ``` ```{config:option} authorization.client.tls-restricted server-authorization :scope: "global" :shortdesc: "Authorization driver for restricted TLS clients" :type: "string" Routes clients using a restricted (project-scoped) client certificate to an authorization driver. Possible values are `allow`, `deny`, `tls`, `openfga` and `scriptlet`. ``` ```{config:option} authorization.client.unix server-authorization :scope: "global" :shortdesc: "Authorization driver for local (`unix` socket) clients" :type: "string" Routes local clients connecting over the `unix` socket to an authorization driver. Possible values are `allow`, `deny`, `openfga` and `scriptlet`. ``` ```{config:option} authorization.openfga.api.token server-authorization :scope: "global" :shortdesc: "API token of the OpenFGA server" :type: "string" ``` ```{config:option} authorization.openfga.api.url server-authorization :scope: "global" :shortdesc: "URL of the OpenFGA server" :type: "string" ``` ```{config:option} authorization.openfga.store.id server-authorization :scope: "global" :shortdesc: "ID of the OpenFGA permission store" :type: "string" ``` ```{config:option} authorization.openfga.tls.identifier server-authorization :defaultdesc: "`name`" :scope: "global" :shortdesc: "Certificate attribute used as the OpenFGA user for TLS clients" :type: "string" When a TLS client is authorized by OpenFGA, this selects the certificate attribute used as the OpenFGA user: `fingerprint` or `name`. ``` ```{config:option} authorization.scriptlet server-authorization :scope: "global" :shortdesc: "Authorization scriptlet" :type: "string" When using scriptlet-based authorization, this option stores the scriptlet. ``` ```{config:option} cluster.healing_threshold server-cluster :defaultdesc: "`0`" :scope: "global" :shortdesc: "Threshold when to evacuate an offline cluster member" :type: "integer" Specify the number of seconds after which an offline cluster member is to be evacuated. To disable evacuating offline members, set this option to `0`. ``` ```{config:option} cluster.https_address server-cluster :scope: "local" :shortdesc: "Address to use for clustering traffic" :type: "string" See {ref}`cluster-https-address`. ``` ```{config:option} cluster.images_minimal_replica server-cluster :defaultdesc: "`3`" :scope: "global" :shortdesc: "Number of cluster members that replicate an image" :type: "integer" Specify the minimal number of cluster members that keep a copy of a particular image. Set this option to `1` for no replication, or to `-1` to replicate images on all members. ``` ```{config:option} cluster.join_token_expiry server-cluster :defaultdesc: "`3H`" :scope: "global" :shortdesc: "Time after which a cluster join token expires" :type: "string" ``` ```{config:option} cluster.max_standby server-cluster :defaultdesc: "`2`" :scope: "global" :shortdesc: "Number of database stand-by members" :type: "integer" Specify the maximum number of cluster members that are assigned the database stand-by role. This must be a number between `0` and `5`. ``` ```{config:option} cluster.max_voters server-cluster :defaultdesc: "`3`" :scope: "global" :shortdesc: "Number of database voter members" :type: "integer" Specify the maximum number of cluster members that are assigned the database voter role. This must be an odd number >= `3`. ``` ```{config:option} cluster.offline_threshold server-cluster :defaultdesc: "`20`" :scope: "global" :shortdesc: "Threshold when an unresponsive member is considered offline" :type: "integer" Specify the number of seconds after which an unresponsive member is considered offline. ``` ```{config:option} cluster.rebalance.batch server-cluster :defaultdesc: "`1`" :scope: "global" :shortdesc: "Maximum number of instances to move during one re-balancing run" :type: "integer" ``` ```{config:option} cluster.rebalance.cooldown server-cluster :defaultdesc: "`6H`" :scope: "global" :shortdesc: "Amount of time during which an instance will not be moved again" :type: "string" ``` ```{config:option} cluster.rebalance.interval server-cluster :defaultdesc: "`0`" :scope: "global" :shortdesc: "How often (in minutes) to consider re-balancing things. 0 to disable (default)" :type: "integer" ``` ```{config:option} cluster.rebalance.threshold server-cluster :defaultdesc: "`20`" :scope: "global" :shortdesc: "Percentage load difference between most and least busy server needed to trigger a migration" :type: "integer" ``` ```{config:option} core.bgp_address server-core :scope: "local" :shortdesc: "Address to bind the BGP server to" :type: "string" See {ref}`network-bgp`. ``` ```{config:option} core.bgp_asn server-core :scope: "global" :shortdesc: "BGP Autonomous System Number for the local server" :type: "string" ``` ```{config:option} core.bgp_routerid server-core :scope: "local" :shortdesc: "A unique identifier for the BGP server" :type: "string" The identifier must be formatted as an IPv4 address. ``` ```{config:option} core.debug_address server-core :scope: "local" :shortdesc: "Address to bind the `pprof` debug server to (HTTP)" :type: "string" ``` ```{config:option} core.dns_address server-core :scope: "local" :shortdesc: "Address to bind the authoritative DNS server to" :type: "string" See {ref}`network-dns-server`. ``` ```{config:option} core.https_address server-core :scope: "local" :shortdesc: "Address to bind for the remote API (HTTPS)" :type: "string" See {ref}`server-expose`. ``` ```{config:option} core.https_allowed_credentials server-core :defaultdesc: "`false`" :scope: "global" :shortdesc: "Whether to set `Access-Control-Allow-Credentials`" :type: "bool" If enabled, the `Access-Control-Allow-Credentials` HTTP header value is set to `true`. ``` ```{config:option} core.https_allowed_headers server-core :scope: "global" :shortdesc: "`Access-Control-Allow-Headers` HTTP header value" :type: "string" ``` ```{config:option} core.https_allowed_methods server-core :scope: "global" :shortdesc: "`Access-Control-Allow-Methods` HTTP header value" :type: "string" ``` ```{config:option} core.https_allowed_origin server-core :scope: "global" :shortdesc: "`Access-Control-Allow-Origin` HTTP header value" :type: "string" ``` ```{config:option} core.https_allowed_websocket_origin server-core :scope: "global" :shortdesc: "Allowed `Origin` values for WebSocket connections" :type: "string" ``` ```{config:option} core.https_trusted_proxy server-core :scope: "global" :shortdesc: "Trusted servers to provide the client's address" :type: "string" Specify a comma-separated list of IP addresses of trusted servers that provide the client's address through the proxy connection header. ``` ```{config:option} core.metrics_address server-core :scope: "local" :shortdesc: "Address to bind the metrics server to (HTTPS)" :type: "string" See {ref}`metrics`. ``` ```{config:option} core.metrics_authentication server-core :defaultdesc: "`true`" :scope: "global" :shortdesc: "Whether to enforce authentication on the metrics endpoint" :type: "bool" ``` ```{config:option} core.proxy_http server-core :scope: "global" :shortdesc: "HTTP proxy to use" :type: "string" If this option is not specified, the daemon falls back to the `HTTP_PROXY` environment variable (if set). ``` ```{config:option} core.proxy_https server-core :scope: "global" :shortdesc: "HTTPS proxy to use" :type: "string" If this option is not specified, the daemon falls back to the `HTTPS_PROXY` environment variable (if set). ``` ```{config:option} core.proxy_ignore_hosts server-core :scope: "global" :shortdesc: "Hosts that don't need the proxy" :type: "string" Specify this option in a similar format to `NO_PROXY` (for example, `1.2.3.4,1.2.3.5`) If this option is not specified, the daemon falls back to the `NO_PROXY` environment variable (if set). ``` ```{config:option} core.remote_token_expiry server-core :defaultdesc: "no expiry" :scope: "global" :shortdesc: "Time after which a remote add token expires" :type: "string" ``` ```{config:option} core.shutdown_action server-core :defaultdesc: "`shutdown`" :scope: "global" :shortdesc: "Action to perform on server shutdown" :type: "string" Specify the action to take when the daemon is being shut down. Supported values are `shutdown` (stop all instances) and `evacuate` (attempt to evacuate the clustered server). ``` ```{config:option} core.shutdown_timeout server-core :defaultdesc: "`5`" :scope: "global" :shortdesc: "How long to wait before shutdown" :type: "integer" Specify the number of minutes to wait for running operations to complete before the daemon shuts down. ``` ```{config:option} core.storage_buckets_address server-core :scope: "local" :shortdesc: "Address to bind the storage object server to (HTTPS)" :type: "string" See {ref}`howto-storage-buckets`. ``` ```{config:option} core.syslog_socket server-core :defaultdesc: "`false`" :scope: "local" :shortdesc: "Whether to enable the syslog unixgram socket listener" :type: "bool" Set this option to `true` to enable the syslog unixgram socket to receive log messages from external processes. ``` ```{config:option} core.trust_ca_certificates server-core :defaultdesc: "`false`" :scope: "global" :shortdesc: "Whether to automatically trust clients signed by the CA" :type: "bool" This only applies when a `server.ca` file is present. When enabled, any valid client certificate issued by the CA and not listed in `ca.crl` is trusted without being added to the trust store. Such certificates get full access to the server unless they are added to the trust store as restricted certificates. When disabled, certificates issued by the CA still need to be added to the trust store. ``` ```{config:option} images.auto_update_cached server-images :defaultdesc: "`true`" :scope: "global" :shortdesc: "Whether to automatically update cached images" :type: "bool" ``` ```{config:option} images.auto_update_interval server-images :defaultdesc: "`6`" :scope: "global" :shortdesc: "Interval at which to look for updates to cached images" :type: "integer" Specify the interval in hours. To disable looking for updates to cached images, set this option to `0`. ``` ```{config:option} images.compression_algorithm server-images :defaultdesc: "`gzip`" :scope: "global" :shortdesc: "Compression algorithm to use for new images" :type: "string" Possible values are `bzip2`, `gzip`, `lz4`, `lzma`, `xz`, `zstd` or `none`. ``` ```{config:option} images.default_architecture server-images :shortdesc: "Default architecture to use in a mixed-architecture cluster" :type: "string" ``` ```{config:option} images.remote_cache_expiry server-images :defaultdesc: "`10`" :scope: "global" :shortdesc: "When an unused cached remote image is flushed" :type: "integer" Specify the number of days after which the unused cached image expires. ``` ```{config:option} logging.NAME.lifecycle.projects server-logging :scope: "global" :shortdesc: "Comma separate list of projects, empty means all" :type: "string" ``` ```{config:option} logging.NAME.lifecycle.types server-logging :scope: "global" :shortdesc: "E.g., `instance`, comma separate, empty means all" :type: "string" ``` ```{config:option} logging.NAME.logging.level server-logging :defaultdesc: "`info`" :scope: "global" :shortdesc: "Minimum log level to send to the logger" :type: "string" ``` ```{config:option} logging.NAME.target.address server-logging :scope: "global" :shortdesc: "Address of the logger" :type: "string" Specify the protocol, name or IP and port. For example `tcp://syslog01.int.example.net:514`. ``` ```{config:option} logging.NAME.target.ca_cert server-logging :scope: "global" :shortdesc: "CA certificates for the server (PEM bundle)" :type: "string" ``` ```{config:option} logging.NAME.target.facility server-logging :scope: "global" :shortdesc: "The syslog facility defines the category of the log message" :type: "string" ``` ```{config:option} logging.NAME.target.instance server-logging :defaultdesc: "Local server host name or cluster member name" :scope: "global" :shortdesc: "Name to use as the instance field in Loki events." :type: "string" This allows replacing the default instance value (server host name) by a more relevant value like a cluster identifier. ``` ```{config:option} logging.NAME.target.labels server-logging :scope: "global" :shortdesc: "Labels for a Loki log entry" :type: "string" Specify a comma-separated list of values that should be used as labels for a Loki log entry. ``` ```{config:option} logging.NAME.target.password server-logging :scope: "global" :shortdesc: "Password used for authentication" :type: "string" ``` ```{config:option} logging.NAME.target.retry server-logging :scope: "global" :shortdesc: "number of delivery retries, default 3" :type: "integer" ``` ```{config:option} logging.NAME.target.type server-logging :scope: "global" :shortdesc: "The type of the logger. One of `loki`, `syslog` or `webhook`." :type: "string" ``` ```{config:option} logging.NAME.target.username server-logging :scope: "global" :shortdesc: "User name used for authentication" :type: "string" ``` ```{config:option} logging.NAME.types server-logging :defaultdesc: "`lifecycle,logging`" :scope: "global" :shortdesc: "Events to send to the logger" :type: "string" Specify a comma-separated list of events to send to the logger. The events can be any combination of `lifecycle`, `logging`, and `network-acl`. ``` ```{config:option} loki.api.ca_cert server-loki :scope: "global" :shortdesc: "CA certificates for the Loki server (PEM bundle)" :type: "string" ``` ```{config:option} loki.api.url server-loki :scope: "global" :shortdesc: "URL to the Loki server" :type: "string" Specify the protocol, name or IP and port. For example `https://loki.example.com:3100`. Incus will automatically add the `/loki/api/v1/push` suffix so there's no need to add it here. ``` ```{config:option} loki.auth.password server-loki :scope: "global" :shortdesc: "Password used for Loki authentication" :type: "string" ``` ```{config:option} loki.auth.username server-loki :scope: "global" :shortdesc: "User name used for Loki authentication" :type: "string" ``` ```{config:option} loki.instance server-loki :defaultdesc: "Local server host name or cluster member name" :scope: "global" :shortdesc: "Name to use as the instance field in Loki events." :type: "string" This allows replacing the default instance value (server host name) by a more relevant value like a cluster identifier. ``` ```{config:option} loki.labels server-loki :scope: "global" :shortdesc: "Labels for a Loki log entry" :type: "string" Specify a comma-separated list of values that should be used as labels for a Loki log entry. ``` ```{config:option} loki.loglevel server-loki :defaultdesc: "`info`" :scope: "global" :shortdesc: "Minimum log level to send to the Loki server" :type: "string" ``` ```{config:option} loki.types server-loki :defaultdesc: "`lifecycle,logging`" :scope: "global" :shortdesc: "Events to send to the Loki server" :type: "string" Specify a comma-separated list of events to send to the Loki server. The events can be any combination of `lifecycle`, `logging`, and `network-acl`. ``` ```{config:option} backups.compression_algorithm server-miscellaneous :defaultdesc: "`gzip`" :scope: "global" :shortdesc: "Compression algorithm to use for backups" :type: "string" Possible values are `bzip2`, `gzip`, `lz4`, `lzma`, `xz`, `zstd` or `none`. ``` ```{config:option} instances.lxcfs.per_instance server-miscellaneous :defaultdesc: "`false`" :scope: "global" :shortdesc: "Whether to run LXCFS on a per-instance basis" :type: "bool" LXCFS is used to provide overlays for common `/proc` and `/sys` files which reflect the resource limits applied to the container. It normally operates through a single file system mount on the host which is then shared by all containers. This is very efficient but comes with the downside that a crash of LXCFS will break all containers. With this option, it's now possible to run a LXCFS instance per container instead, using more system resources but reducing the impact of a crash. ``` ```{config:option} instances.nic.host_name server-miscellaneous :defaultdesc: "`random`" :scope: "global" :shortdesc: "How to set the host name for a NIC" :type: "string" Possible values are `random` and `mac`. If set to `random`, use the random host interface name as the host name. If set to `mac`, generate a host name in the form `inc` (MAC without leading two digits). ``` ```{config:option} instances.placement.scriptlet server-miscellaneous :scope: "global" :shortdesc: "Instance placement scriptlet for automatic instance placement" :type: "string" When using custom automatic instance placement logic, this option stores the scriptlet. See {ref}`clustering-instance-placement-scriptlet` for more information. ``` ```{config:option} instances.tpm.platform_cert server-miscellaneous :scope: "global" :shortdesc: "Platform CA certificate used to provision TPM Endorsement Keys (PEM bundle)" :type: "string" PEM encoded platform CA certificate used to sign the Endorsement Key and platform certificate of newly created TPM devices. Must be set together with `instances.tpm.platform_key`. ``` ```{config:option} instances.tpm.platform_key server-miscellaneous :scope: "global" :shortdesc: "Private key for the TPM platform CA certificate (PEM private key)" :type: "string" PEM encoded private key matching `instances.tpm.platform_cert`. Used to sign the Endorsement Key and platform certificate of newly created TPM devices. ``` ```{config:option} network.ovn.ca_cert server-miscellaneous :defaultdesc: "Content of `/etc/ovn/ovn-central.crt` if present" :scope: "global" :shortdesc: "OVN SSL certificate authorities (PEM bundle)" :type: "string" ``` ```{config:option} network.ovn.client_cert server-miscellaneous :defaultdesc: "Content of `/etc/ovn/cert_host` if present" :scope: "global" :shortdesc: "OVN SSL client certificate (PEM bundle)" :type: "string" ``` ```{config:option} network.ovn.client_key server-miscellaneous :defaultdesc: "Content of `/etc/ovn/key_host` if present" :scope: "global" :shortdesc: "OVN SSL client key (PEM private key)" :type: "string" ``` ```{config:option} network.ovn.integration_bridge server-miscellaneous :defaultdesc: "`br-int`" :scope: "global" :shortdesc: "OVS integration bridge to use for OVN networks" :type: "string" ``` ```{config:option} network.ovn.northbound_connection server-miscellaneous :defaultdesc: "`unix:/run/ovn/ovnnb_db.sock`" :scope: "global" :shortdesc: "OVN northbound database connection string" :type: "string" ``` ```{config:option} network.ovs.connection server-miscellaneous :defaultdesc: "`unix:/run/openvswitch/db.sock`" :scope: "global" :shortdesc: "OVS socket path" :type: "string" ``` ```{config:option} storage.backups_volume server-miscellaneous :scope: "local" :shortdesc: "Volume to use to store backup tarballs" :type: "string" Specify the volume using the syntax `POOL/VOLUME`. ``` ```{config:option} storage.images_volume server-miscellaneous :scope: "local" :shortdesc: "Volume to use to store the image tarballs" :type: "string" Specify the volume using the syntax `POOL/VOLUME`. ``` ```{config:option} storage.linstor.ca_cert server-miscellaneous :scope: "global" :shortdesc: "LINSTOR SSL certificate authorities (PEM bundle)" :type: "string" ``` ```{config:option} storage.linstor.client_cert server-miscellaneous :scope: "global" :shortdesc: "LINSTOR SSL client certificate (PEM bundle)" :type: "string" ``` ```{config:option} storage.linstor.client_key server-miscellaneous :scope: "global" :shortdesc: "LINSTOR SSL client key (PEM private key)" :type: "string" ``` ```{config:option} storage.linstor.controller_connection server-miscellaneous :scope: "global" :shortdesc: "LINSTOR controller connection string" :type: "string" ``` ```{config:option} storage.linstor.satellite.name server-miscellaneous :scope: "global" :shortdesc: "LINSTOR satellite node name override" :type: "string" Set this option to the name of the local LINSTOR satellite node, should it be different from the Incus server name. ``` ```{config:option} storage.logs_volume server-miscellaneous :scope: "local" :shortdesc: "Volume to use to store instance log directories" :type: "string" Specify the volume using the syntax `POOL/VOLUME`. ``` ```{config:option} network.hwaddr_pattern server-network :defaultdesc: "`10:66:6a:xx:xx:xx`" :scope: "global" :shortdesc: "MAC address template" :type: "string" Specify a MAC address template, e.g. `10:66:6a:xx:xx:xx`, to use within the cluster. Every `x` in the template will be replaced by a random character in `0`–`f`. Beware of the birthday paradox! A single `xx` block leads to a 10% collision probability with only 8 addresses; for a double `xx:xx` block, 118 addresses; for a triple `xx:xx:xx` block, 1881; for a quadruple `xx:xx:xx:xx` block, 30084. We provide absolutely no guardrail against that. ``` ```{config:option} oidc.audience server-oidc :scope: "global" :shortdesc: "Expected audience value for the application" :type: "string" This value is required by some providers. Opaque access tokens are rejected when set as the audience can't be verified on those. ``` ```{config:option} oidc.claim server-oidc :scope: "global" :shortdesc: "OpenID Connect claim to use as the username" :type: "string" Note that the claim must be contained in the access token or, for opaque access tokens, in the ID token. ``` ```{config:option} oidc.client.id server-oidc :scope: "global" :shortdesc: "OpenID Connect client ID" :type: "string" ``` ```{config:option} oidc.issuer server-oidc :scope: "global" :shortdesc: "OpenID Connect Discovery URL for the provider" :type: "string" ``` ```{config:option} oidc.scopes server-oidc :scope: "global" :shortdesc: "Comma separated list of OpenID Connect scopes" :type: "string" ``` ```{config:option} btrfs.create_options storage_btrfs-common :default: "-" :scope: "global" :shortdesc: "Additional options to pass to `mkfs.btrfs` when creating the pool" :type: "string" ``` ```{config:option} btrfs.mount_options storage_btrfs-common :default: "`user_subvol_rm_allowed`" :scope: "global" :shortdesc: "Mount options for block devices" :type: "string" ``` ```{config:option} size storage_btrfs-common :default: "auto (20% of free disk space, >= 5 GiB and <= 30 GiB)" :scope: "local" :shortdesc: "Size of the storage pool when creating loop-based pools (in bytes, suffixes supported, can be increased to grow storage pool)" :type: "string" ``` ```{config:option} source storage_btrfs-common :default: "-" :scope: "local" :shortdesc: "Path to an existing block device, loop file or Btrfs subvolume" :type: "string" ``` ```{config:option} source.wipe storage_btrfs-common :default: "`false`" :scope: "local" :shortdesc: "Wipe the block device specified in `source` prior to creating the storage pool" :type: "bool" ``` ```{config:option} size storage_bucket_btrfs-common :condition: "appropriate driver" :default: "same as `volume.size`" :shortdesc: "Size/quota of the storage bucket" :type: "string" ``` ```{config:option} size storage_bucket_cephobject-common :default: "-" :shortdesc: "Quota of the storage bucket" :type: "string" ``` ```{config:option} size storage_bucket_lvm-common :condition: "appropriate driver" :default: "same as `volume.size`" :shortdesc: "Size/quota of the storage bucket" :type: "string" ``` ```{config:option} size storage_bucket_zfs-common :condition: "appropriate driver" :default: "same as `volume.size`" :shortdesc: "Size/quota of the storage bucket" :type: "string" ``` ```{config:option} ceph.cluster_name storage_ceph-common :default: "`ceph`" :scope: "global" :shortdesc: "Name of the Ceph cluster in which to create new storage pools" :type: "string" ``` ```{config:option} ceph.osd.data_pool_name storage_ceph-common :default: "-" :scope: "global" :shortdesc: "Name of the OSD data pool" :type: "string" ``` ```{config:option} ceph.osd.force_reuse storage_ceph-common :default: "-" :scope: "global" :shortdesc: "Deprecated, should not be used." :type: "bool" ``` ```{config:option} ceph.osd.pg_name storage_ceph-common :default: "`32`" :scope: "global" :shortdesc: "Number of placement groups for the OSD storage pool" :type: "string" ``` ```{config:option} ceph.osd.pool_name storage_ceph-common :default: "name of the pool" :scope: "global" :shortdesc: "Name of the OSD storage pool" :type: "string" ``` ```{config:option} ceph.rbd.backend storage_ceph-common :default: "`krbd`" :scope: "global" :shortdesc: "Whether to use the RBD kernel driver (`krbd`) or `librbd` through `qemu-nbd` (`librbd`)" :type: "string" ``` ```{config:option} ceph.rbd.clone_copy storage_ceph-common :default: "`true`" :scope: "global" :shortdesc: "Whether to use RBD lightweight clones rather than full dataset copies" :type: "bool" When disabled, new volumes are still cloned from their source but flattened in the background right after creation, so the source can only be fully removed once that has completed. ``` ```{config:option} ceph.rbd.du storage_ceph-common :default: "`true`" :scope: "global" :shortdesc: "Whether to use RBD `du` to obtain disk usage data for stopped instances" :type: "bool" ``` ```{config:option} ceph.rbd.features storage_ceph-common :default: "`layering`" :scope: "global" :shortdesc: "Comma-separated list of RBD features to enable on the volumes" :type: "string" ``` ```{config:option} ceph.user.name storage_ceph-common :default: "`admin`" :scope: "global" :shortdesc: "The Ceph user to use when creating storage pools and volumes" :type: "string" ``` ```{config:option} source storage_ceph-common :default: "-" :scope: "local" :shortdesc: "Existing OSD storage pool to use" :type: "string" ``` ```{config:option} volatile.pool.pristine storage_ceph-common :default: "`true`" :scope: "global" :shortdesc: "Whether the pool was empty on creation time" :type: "string" ``` ```{config:option} cephfs.cluster_name storage_cephfs-common :default: "`ceph`" :scope: "global" :shortdesc: "Name of the Ceph cluster that contains the CephFS file system" :type: "string" ``` ```{config:option} cephfs.create_missing storage_cephfs-common :default: "`false`" :scope: "global" :shortdesc: "Create the file system and the missing data and metadata OSD pools" :type: "bool" ``` ```{config:option} cephfs.data_pool storage_cephfs-common :default: "-" :scope: "global" :shortdesc: "Data OSD pool name to create for the file system" :type: "string" ``` ```{config:option} cephfs.fscache storage_cephfs-common :default: "`false`" :scope: "global" :shortdesc: "Enable use of kernel `fscache` and `cachefilesd`" :type: "bool" ``` ```{config:option} cephfs.meta_pool storage_cephfs-common :default: "-" :scope: "global" :shortdesc: "Metadata OSD pool name to create for the file system" :type: "string" ``` ```{config:option} cephfs.osd_pg_num storage_cephfs-common :default: "-" :scope: "global" :shortdesc: "OSD pool `pg_num` to use when creating missing OSD pools" :type: "string" ``` ```{config:option} cephfs.path storage_cephfs-common :default: "`/`" :scope: "global" :shortdesc: "The base path for the CephFS mount" :type: "string" ``` ```{config:option} cephfs.user.name storage_cephfs-common :default: "`admin`" :scope: "global" :shortdesc: "The Ceph user to use" :type: "string" ``` ```{config:option} source storage_cephfs-common :default: "-" :scope: "local" :shortdesc: "Existing CephFS file system or file system path to use" :type: "string" ``` ```{config:option} volatile.pool.pristine storage_cephfs-common :default: "`true`" :scope: "global" :shortdesc: "Whether the CephFS file system was empty on creation time" :type: "string" ``` ```{config:option} cephobject.bucket_name_prefix storage_cephobject-common :default: "-" :scope: "global" :shortdesc: "Prefix to add to bucket names in Ceph" :type: "string" ``` ```{config:option} cephobject.cluster_name storage_cephobject-common :default: "`ceph`" :scope: "global" :shortdesc: "The Ceph cluster to use" :type: "string" ``` ```{config:option} cephobject.radosgw.endpoint storage_cephobject-common :default: "-" :scope: "global" :shortdesc: "URL of the `radosgw` gateway process" :type: "string" ``` ```{config:option} cephobject.radosgw.endpoint_cert storage_cephobject-common :default: "-" :scope: "global" :shortdesc: "TLS certificate (PEM bundle) to trust for endpoint communication" :type: "string" ``` ```{config:option} cephobject.user.name storage_cephobject-common :default: "`admin`" :scope: "global" :shortdesc: "The Ceph user to use" :type: "string" ``` ```{config:option} volatile.pool.pristine storage_cephobject-common :default: "`true`" :scope: "global" :shortdesc: "Whether the `radosgw` `incus-admin` user existed at creation time" :type: "string" ``` ```{config:option} rsync.bwlimit storage_dir-common :default: "`0` (no limit)" :scope: "global" :shortdesc: "The upper limit to be placed on the socket I/O when `rsync` must be used to transfer storage entities" :type: "string" ``` ```{config:option} rsync.compression storage_dir-common :default: "`true`" :scope: "global" :shortdesc: "Whether to use compression while migrating storage pools" :type: "bool" ``` ```{config:option} source storage_dir-common :default: "-" :scope: "local" :shortdesc: "Path to an existing directory" :type: "string" ``` ```{config:option} drbd.auto_add_quorum_tiebreaker storage_linstor-common :default: "`true`" :scope: "global" :shortdesc: "Whether to allow LINSTOR to automatically create diskless resources to act as quorum tiebreakers if needed (applied to the resource group)" :type: "bool" ``` ```{config:option} drbd.auto_diskful storage_linstor-common :default: "-" :scope: "global" :shortdesc: "A duration string describing the time after which a primary diskless resource can be converted to diskful if storage is available on the node (applied to the resource group)" :type: "string" ``` ```{config:option} drbd.on_no_quorum storage_linstor-common :default: "-" :scope: "global" :shortdesc: "The DRBD policy to use on resources when quorum is lost (applied to the resource group)" :type: "string" ``` ```{config:option} linstor.raw.* storage_linstor-common :scope: "global" :shortdesc: "Extra LINSTOR properties to set. For example, `BCache/PoolName` is encoded as `linstor.raw.BCache/PoolName`." :type: "string" ``` ```{config:option} linstor.resource_group.name storage_linstor-common :default: "`incus`" :scope: "global" :shortdesc: "Name of the LINSTOR resource group that will be used for the storage pool" :type: "string" ``` ```{config:option} linstor.resource_group.place_count storage_linstor-common :default: "`2`" :scope: "global" :shortdesc: "Number of diskful replicas that should be created for resources in the resource group. Increasing the value of this option on a pool that already has volumes will result in LINSTOR creating new diskful replicas for all existing resources to match the new value" :type: "int" ``` ```{config:option} linstor.resource_group.storage_pool storage_linstor-common :default: "-" :scope: "global" :shortdesc: "The storage pool name in which resources should be placed on satellite nodes" :type: "string" ``` ```{config:option} linstor.volume.prefix storage_linstor-common :default: "`incus-volume-`" :scope: "global" :shortdesc: "The prefix to use for the internal names of LINSTOR-managed volumes. Cannot be updated after the storage pool is created" :type: "string" ``` ```{config:option} source storage_linstor-common :default: "`incus`" :scope: "global" :shortdesc: "LINSTOR storage pool name. Alias for `linstor.resource_group.name`. Use either either one or the other or make sure they have the same value." :type: "string" ``` ```{config:option} volatile.pool.pristine storage_linstor-common :default: "`true`" :scope: "global" :shortdesc: "Whether the pool was empty on creation time" :type: "string" ``` ```{config:option} block.type storage_lvm-common :condition: "block-based volume" :default: "same as `volume.block.type`" :shortdesc: "Type of the block volume" ``` ```{config:option} lvm.metadata_size storage_lvm-common :default: "`0` (auto)" :scope: "global" :shortdesc: "The size of the metadata space for the physical volume." :type: "string" ``` ```{config:option} lvm.thinpool_metadata_size storage_lvm-common :default: "`0` (auto)" :scope: "global" :shortdesc: "The size of the thin pool metadata volume (the default is to let LVM calculate an appropriate size). Not usable with `lvmcluster`." :type: "string" ``` ```{config:option} lvm.thinpool_name storage_lvm-common :default: "`IncusThinPool`" :scope: "local" :shortdesc: "Thin pool where volumes are created. Not usable with `lvmcluster`." :type: "string" ``` ```{config:option} lvm.use_thinpool storage_lvm-common :default: "`true`" :scope: "global" :shortdesc: "Whether the storage pool uses a thin pool for logical volumes. Not usable with `lvmcluster`." :type: "bool" ``` ```{config:option} lvm.vg.force_reuse storage_lvm-common :default: "`false`" :scope: "local" :shortdesc: "Force using an existing non-empty volume group. Not usable with `lvmcluster`." :type: "bool" ``` ```{config:option} lvm.vg_name storage_lvm-common :default: "name of the pool" :scope: "local" :shortdesc: "Name of the volume group to create." :type: "string" ``` ```{config:option} size storage_lvm-common :default: "auto (20% of free disk space, >= 5 GiB and <= 30 GiB) for `lvm`." :scope: "local" :shortdesc: "Storage pool size (in bytes, suffixes supported, can be increased to grow storage pool). `lvmcluster` pools: cannot set at creation, but can be updated." :type: "string" ``` ```{config:option} source storage_lvm-common :default: "-" :scope: "local" :shortdesc: "Path to an existing block device, loop file or LVM volume group." :type: "string" ``` ```{config:option} source.wipe storage_lvm-common :default: "`false`" :scope: "local" :shortdesc: "Wipe the block device specified in `source` prior to creating the storage pool." :type: "bool" ``` ```{config:option} source storage_truenas-common :default: "-" :scope: "local" :shortdesc: "ZFS dataset to use on the remote TrueNAS host. Format: `[:][/][/]`. If `host` is omitted here, it must be set via `truenas.host`." :type: "string" ``` ```{config:option} truenas.allow_insecure storage_truenas-common :default: "`false`" :scope: "global" :shortdesc: "If set to `true`, allows insecure (non-TLS) connections to the TrueNAS API." :type: "bool" ``` ```{config:option} truenas.api_key storage_truenas-common :default: "-" :scope: "global" :shortdesc: "API key used to authenticate with the TrueNAS host." :type: "string" ``` ```{config:option} truenas.clone_copy storage_truenas-common :default: "`true`" :scope: "global" :shortdesc: "Whether to use lightweight clones rather than full {spellexception}`dataset` copies." :type: "bool" ``` ```{config:option} truenas.config storage_truenas-common :default: "-" :scope: "global" :shortdesc: "Path to a configuration file for the TrueNAS client tool." :type: "string" ``` ```{config:option} truenas.dataset storage_truenas-common :default: "-" :scope: "global" :shortdesc: "Remote dataset name. Typically inferred from `source`, but can be overridden." :type: "string" ``` ```{config:option} truenas.force_reuse storage_truenas-common :default: "`false`" :scope: "global" :shortdesc: "Allow to use an existing non-empty pool." :type: "bool" ``` ```{config:option} truenas.host storage_truenas-common :default: "-" :scope: "global" :shortdesc: "Hostname or IP address of the remote TrueNAS system. Optional if included in the `source`, or a configuration is used." :type: "string" ``` ```{config:option} truenas.initiator storage_truenas-common :default: "-" :scope: "global" :shortdesc: "iSCSI initiator name used during block volume attachment." :type: "string" ``` ```{config:option} truenas.portal storage_truenas-common :default: "-" :scope: "global" :shortdesc: "iSCSI portal address to use for block volume connections." :type: "string" ``` ```{config:option} btrfs.compression storage_volume_btrfs-common :condition: "appropriate driver" :default: "same as `volume.btrfs.compression`" :shortdesc: "Compression algorithm to set on the volume, mapping to the Btrfs `compression` property (for example `zstd`, `lzo`, `zlib` or `none`)" :type: "string" ``` ```{config:option} initial.gid storage_volume_btrfs-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.gid` or `0`" :shortdesc: "GID of the volume owner in the instance" :type: "int" ``` ```{config:option} initial.mode storage_volume_btrfs-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.mode` or `711`" :shortdesc: "Mode of the volume in the instance" :type: "int" ``` ```{config:option} initial.uid storage_volume_btrfs-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.uid` or `0`" :shortdesc: "UID of the volume owner in the instance" :type: "int" ``` ```{config:option} security.shared storage_volume_btrfs-common :condition: "custom block volume" :default: "same as `volume.security.shared` or `false`" :shortdesc: "Enable sharing the volume across multiple instances" :type: "bool" ``` ```{config:option} security.shifted storage_volume_btrfs-common :condition: "custom volume" :default: "same as `volume.security.shifted` or `false`" :shortdesc: "{{enable_ID_shifting}}" :type: "bool" ``` ```{config:option} security.unmapped storage_volume_btrfs-common :condition: "custom volume" :default: "same as `volume.security.unmapped` or `false`" :shortdesc: "Disable ID mapping for the volume" :type: "bool" ``` ```{config:option} size storage_volume_btrfs-common :condition: "appropriate driver" :default: "same as `volume.size`" :shortdesc: "Size/quota of the storage volume" :type: "string" ``` ```{config:option} snapshots.expiry storage_volume_btrfs-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.expiry.manual storage_volume_btrfs-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry.manual`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.pattern storage_volume_btrfs-common :condition: "custom volume" :default: "same as `volume.snapshot.pattern` or `snap%d`" :shortdesc: "{{snapshot_pattern_format}} [^*]" :type: "string" ``` ```{config:option} snapshots.schedule storage_volume_btrfs-common :condition: "custom volume" :default: "same as `volume.snapshot.schedule`" :shortdesc: "{{snapshot_schedule_format}}" :type: "string" ``` ```{config:option} block.create_options storage_volume_ceph-common :condition: "block-based volume with content type `filesystem`" :default: "same as `volume.block.create_options`" :shortdesc: "Additional options to pass to the file system creation tool when formatting the volume" :type: "string" ``` ```{config:option} block.filesystem storage_volume_ceph-common :condition: "block-based volume with content type `filesystem`" :default: "same as `volume.block.filesystem`" :shortdesc: "{{block_filesystem}}" :type: "string" ``` ```{config:option} block.mount_options storage_volume_ceph-common :condition: "block-based volume with content type `filesystem`" :default: "same as `volume.block.mount_options`" :shortdesc: "Mount options for block-backed file system volumes" :type: "string" ``` ```{config:option} initial.gid storage_volume_ceph-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.gid` or `0`" :shortdesc: "GID of the volume owner in the instance" :type: "int" ``` ```{config:option} initial.mode storage_volume_ceph-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.mode` or `711`" :shortdesc: "Mode of the volume in the instance" :type: "int" ``` ```{config:option} initial.uid storage_volume_ceph-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.uid` or `0`" :shortdesc: "UID of the volume owner in the instance" :type: "int" ``` ```{config:option} security.shared storage_volume_ceph-common :condition: "custom block volume" :default: "same as `volume.security.shared` or `false`" :shortdesc: "Enable sharing the volume across multiple instances" :type: "bool" ``` ```{config:option} security.shifted storage_volume_ceph-common :condition: "custom volume" :default: "same as `volume.security.shifted` or `false`" :shortdesc: "{{enable_ID_shifting}}" :type: "bool" ``` ```{config:option} security.unmapped storage_volume_ceph-common :condition: "custom volume" :default: "same as `volume.security.unmapped` or `false`" :shortdesc: "Disable ID mapping for the volume" :type: "bool" ``` ```{config:option} size storage_volume_ceph-common :condition: "-" :default: "same as `volume.size`" :shortdesc: "Size/quota of the storage volume" :type: "string" ``` ```{config:option} snapshots.expiry storage_volume_ceph-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.expiry.manual storage_volume_ceph-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry.manual`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.pattern storage_volume_ceph-common :condition: "custom volume" :default: "same as `volume.snapshot.pattern` or `snap%d`" :shortdesc: "{{snapshot_pattern_format}} [^*]" :type: "string" ``` ```{config:option} snapshots.schedule storage_volume_ceph-common :condition: "custom volume" :default: "same as `volume.snapshot.schedule`" :shortdesc: "{{snapshot_schedule_format}}" :type: "string" ``` ```{config:option} initial.gid storage_volume_cephfs-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.gid` or `0`" :shortdesc: "GID of the volume owner in the instance" :type: "int" ``` ```{config:option} initial.mode storage_volume_cephfs-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.mode` or `711`" :shortdesc: "Mode of the volume in the instance" :type: "int" ``` ```{config:option} initial.uid storage_volume_cephfs-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.uid` or `0`" :shortdesc: "UID of the volume owner in the instance" :type: "int" ``` ```{config:option} security.shared storage_volume_cephfs-common :condition: "custom block volume" :default: "same as `volume.security.shared` or `false`" :shortdesc: "Enable sharing the volume across multiple instances" :type: "bool" ``` ```{config:option} security.shifted storage_volume_cephfs-common :condition: "custom volume" :default: "same as `volume.security.shifted` or `false`" :shortdesc: "{{enable_ID_shifting}}" :type: "bool" ``` ```{config:option} security.unmapped storage_volume_cephfs-common :condition: "custom volume" :default: "same as `volume.security.unmapped` or `false`" :shortdesc: "Disable ID mapping for the volume" :type: "bool" ``` ```{config:option} size storage_volume_cephfs-common :condition: "appropriate driver" :default: "same as `volume.size`" :shortdesc: "Size/quota of the storage volume" :type: "string" ``` ```{config:option} snapshots.expiry storage_volume_cephfs-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.expiry.manual storage_volume_cephfs-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry.manual`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.pattern storage_volume_cephfs-common :condition: "custom volume" :default: "same as `volume.snapshot.pattern` or `snap%d`" :shortdesc: "{{snapshot_pattern_format}} [^*]" :type: "string" ``` ```{config:option} snapshots.schedule storage_volume_cephfs-common :condition: "custom volume" :default: "same as `volume.snapshot.schedule`" :shortdesc: "{{snapshot_schedule_format}}" :type: "string" ``` ```{config:option} initial.gid storage_volume_dir-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.gid` or `0`" :shortdesc: "GID of the volume owner in the instance" :type: "int" ``` ```{config:option} initial.mode storage_volume_dir-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.mode` or `711`" :shortdesc: "Mode of the volume in the instance" :type: "int" ``` ```{config:option} initial.uid storage_volume_dir-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.uid` or `0`" :shortdesc: "UID of the volume owner in the instance" :type: "int" ``` ```{config:option} security.shared storage_volume_dir-common :condition: "custom block volume" :default: "same as `volume.security.shared` or `false`" :shortdesc: "Enable sharing the volume across multiple instances" :type: "bool" ``` ```{config:option} security.shifted storage_volume_dir-common :condition: "custom volume" :default: "same as `volume.security.shifted` or `false`" :shortdesc: "{{enable_ID_shifting}}" :type: "bool" ``` ```{config:option} security.size storage_volume_dir-common :condition: "appropriate driver" :default: "same as `volume.size`" :shortdesc: "Size/quota of the storage volume" :type: "string" ``` ```{config:option} security.unmapped storage_volume_dir-common :condition: "custom volume" :default: "same as `volume.security.unmapped` or `false`" :shortdesc: "Disable ID mapping for the volume" :type: "bool" ``` ```{config:option} snapshots.expiry storage_volume_dir-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.expiry.manual storage_volume_dir-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry.manual`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.pattern storage_volume_dir-common :condition: "custom volume" :default: "same as `volume.snapshot.pattern` or `snap%d`" :shortdesc: "{{snapshot_pattern_format}} [^*]" :type: "string" ``` ```{config:option} snapshots.schedule storage_volume_dir-common :condition: "custom volume" :default: "same as `volume.snapshot.schedule`" :shortdesc: "{{snapshot_schedule_format}}" :type: "string" ``` ```{config:option} block.create_options storage_volume_linstor-common :condition: "block-based volume with content type `filesystem`" :default: "same as `volume.block.create_options`" :shortdesc: "Additional options to pass to the file system creation tool when formatting the volume" :type: "string" ``` ```{config:option} block.filesystem storage_volume_linstor-common :condition: "block-based volume with content type `filesystem`" :default: "same as `volume.block.filesystem`" :shortdesc: "{{block_filesystem}}" :type: "string" ``` ```{config:option} block.mount_options storage_volume_linstor-common :condition: "block-based volume with content type `filesystem`" :default: "same as `volume.block.mount_options`" :shortdesc: "Mount options for block-backed file system volumes" :type: "string" ``` ```{config:option} drbd.auto_add_quorum_tiebreaker storage_volume_linstor-common :condition: "-" :default: "`true`" :shortdesc: "Whether to allow LINSTOR to automatically create diskless resources to act as quorum tiebreakers if needed (applied to the resource definition)" :type: "bool" ``` ```{config:option} drbd.auto_diskful storage_volume_linstor-common :condition: "-" :default: "-" :shortdesc: "A duration string describing the time after which a primary diskless resource can be converted to diskful if storage is available on the node (applied to the resource definition)" :type: "string" ``` ```{config:option} drbd.on_no_quorum storage_volume_linstor-common :condition: "-" :default: "-" :shortdesc: "The DRBD policy to use on resources when quorum is lost (applied to the resource definition)" :type: "string" ``` ```{config:option} initial.gid storage_volume_linstor-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.gid` or `0`" :shortdesc: "GID of the volume owner in the instance" :type: "int" ``` ```{config:option} initial.mode storage_volume_linstor-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.mode` or `711`" :shortdesc: "Mode of the volume in the instance" :type: "int" ``` ```{config:option} initial.uid storage_volume_linstor-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.uid` or `0`" :shortdesc: "UID of the volume owner in the instance" :type: "int" ``` ```{config:option} linstor.raw.* storage_volume_linstor-common :scope: "global" :shortdesc: "Extra LINSTOR properties to set. For example, `BCache/PoolName` is encoded as `linstor.raw.BCache/PoolName`." :type: "string" ``` ```{config:option} linstor.remove_snapshots storage_volume_linstor-common :condition: "-" :default: "same as `volume.linstor.remove_snapshots` or `false`" :shortdesc: "Remove snapshots as needed" :type: "bool" ``` ```{config:option} security.shared storage_volume_linstor-common :condition: "custom block volume" :default: "same as `volume.security.shared` or `false`" :shortdesc: "Enable sharing the volume across multiple instances" :type: "bool" ``` ```{config:option} security.shifted storage_volume_linstor-common :condition: "custom volume" :default: "same as `volume.security.shifted` or `false`" :shortdesc: "{{enable_ID_shifting}}" :type: "bool" ``` ```{config:option} security.unmapped storage_volume_linstor-common :condition: "custom volume" :default: "same as `volume.security.unmapped` or `false`" :shortdesc: "Disable ID mapping for the volume" :type: "bool" ``` ```{config:option} size storage_volume_linstor-common :condition: "-" :default: "same as `volume.size`" :shortdesc: "Size/quota of the storage volume" :type: "string" ``` ```{config:option} snapshots.expiry storage_volume_linstor-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.expiry.manual storage_volume_linstor-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry.manual`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.pattern storage_volume_linstor-common :condition: "custom volume" :default: "same as `volume.snapshot.pattern` or `snap%d`" :shortdesc: "{{snapshot_pattern_format}} [^*]" :type: "string" ``` ```{config:option} snapshots.schedule storage_volume_linstor-common :condition: "custom volume" :default: "same as `volume.snapshot.schedule`" :shortdesc: "{{snapshot_schedule_format}}" :type: "string" ``` ```{config:option} block.create_options storage_volume_lvm-common :condition: "block-based volume with content type `filesystem`" :default: "same as `volume.block.create_options`" :shortdesc: "Additional options to pass to the file system creation tool when formatting the volume" :type: "string" ``` ```{config:option} block.filesystem storage_volume_lvm-common :condition: "block-based volume with content type `filesystem`" :default: "same as `volume.block.filesystem`" :shortdesc: "{{block_filesystem}}" :type: "string" ``` ```{config:option} block.mount_options storage_volume_lvm-common :condition: "block-based volume with content type `filesystem`" :default: "same as `volume.block.mount_options`" :shortdesc: "Mount options for block-backed file system volumes" :type: "string" ``` ```{config:option} initial.gid storage_volume_lvm-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.gid` or `0`" :shortdesc: "GID of the volume owner in the instance" :type: "int" ``` ```{config:option} initial.mode storage_volume_lvm-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.mode` or `711`" :shortdesc: "Mode of the volume in the instance" :type: "int" ``` ```{config:option} initial.uid storage_volume_lvm-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.uid` or `0`" :shortdesc: "UID of the volume owner in the instance" :type: "int" ``` ```{config:option} lvm.stripes storage_volume_lvm-common :condition: "-" :default: "same as `volume.lvm.stripes`" :shortdesc: "Number of stripes to use for new volumes (or thin pool volume)" :type: "string" ``` ```{config:option} lvm.stripes.size storage_volume_lvm-common :condition: "-" :default: "same as `volume.lvm.stripes.size`" :shortdesc: "Size of stripes to use (at least 4096 bytes and multiple of 512 bytes)" :type: "string" ``` ```{config:option} lvmcluster.remove_snapshots storage_volume_lvm-common :condition: "-" :default: "same as `volume.lvmcluster.remove_snapshots` or `false`" :shortdesc: "Remove snapshots as needed" :type: "bool" ``` ```{config:option} security.shared storage_volume_lvm-common :condition: "custom block volume" :default: "same as `volume.security.shared` or `false`" :shortdesc: "Enable sharing the volume across multiple instances" :type: "bool" ``` ```{config:option} security.shifted storage_volume_lvm-common :condition: "custom volume" :default: "same as `volume.security.shifted` or `false`" :shortdesc: "{{enable_ID_shifting}}" :type: "bool" ``` ```{config:option} security.unmapped storage_volume_lvm-common :condition: "custom volume" :default: "same as `volume.security.unmapped` or `false`" :shortdesc: "Disable ID mapping for the volume" :type: "bool" ``` ```{config:option} size storage_volume_lvm-common :condition: "default: same as `volume.size`" :shortdesc: "Size/quota of the storage volume" :type: "string" ``` ```{config:option} snapshots.expiry storage_volume_lvm-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.expiry.manual storage_volume_lvm-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry.manual`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.pattern storage_volume_lvm-common :condition: "custom volume" :default: "same as `volume.snapshot.pattern` or `snap%d`" :shortdesc: "{{snapshot_pattern_format}} [^*]" :type: "string" ``` ```{config:option} snapshots.schedule storage_volume_lvm-common :condition: "custom volume" :default: "same as `volume.snapshot.schedule`" :shortdesc: "{{snapshot_schedule_format}}" :type: "string" ``` ```{config:option} block.create_options storage_volume_truenas-common :condition: "-" :default: "same as `volume.block.create_options`" :shortdesc: "Additional options to pass to the file system creation tool when formatting the volume" :type: "string" ``` ```{config:option} block.filesystem storage_volume_truenas-common :condition: "-" :default: "same as `volume.block.filesystem`" :shortdesc: "{{block_filesystem}}" :type: "string" ``` ```{config:option} block.mount_options storage_volume_truenas-common :condition: "-" :default: "same as `volume.block.mount_options`" :shortdesc: "Mount options for block-backed file system volumes" :type: "string" ``` ```{config:option} initial.gid storage_volume_truenas-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.gid` or `0`" :shortdesc: "GID of the volume owner in the instance" :type: "int" ``` ```{config:option} initial.mode storage_volume_truenas-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.mode` or `711`" :shortdesc: "Mode of the volume in the instance" :type: "int" ``` ```{config:option} initial.uid storage_volume_truenas-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.uid` or `0`" :shortdesc: "UID of the volume owner in the instance" :type: "int" ``` ```{config:option} security.shared storage_volume_truenas-common :condition: "custom block volume" :default: "same as `volume.security.shared` or `false`" :shortdesc: "Enable sharing the volume across multiple instances" :type: "bool" ``` ```{config:option} security.shifted storage_volume_truenas-common :condition: "custom volume" :default: "same as `volume.security.shifted` or `false`" :shortdesc: "{{enable_ID_shifting}}" :type: "bool" ``` ```{config:option} security.unmapped storage_volume_truenas-common :condition: "custom volume" :default: "same as `volume.security.unmapped` or `false`" :shortdesc: "Disable ID mapping for the volume" :type: "bool" ``` ```{config:option} size storage_volume_truenas-common :condition: "appropriate driver" :default: "same as `volume.size`" :shortdesc: "Size/quota of the storage volume" :type: "string" ``` ```{config:option} snapshots.expiry storage_volume_truenas-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.expiry.manual storage_volume_truenas-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry.manual`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.pattern storage_volume_truenas-common :condition: "custom volume" :default: "same as `volume.snapshot.pattern` or `snap%d`" :shortdesc: "{{snapshot_pattern_format}}" :type: "string" ``` ```{config:option} snapshots.schedule storage_volume_truenas-common :condition: "custom volume" :default: "same as `volume.snapshot.schedule`" :shortdesc: "{{snapshot_schedule_format}}" :type: "string" ``` ```{config:option} truenas.blocksize storage_volume_truenas-common :condition: "-" :default: "same as `volume.truenas.blocksize`" :shortdesc: "Size of the ZFS block in range from 512 bytes to 16 MiB (must be power of 2) - for block volume, a maximum value of 128 KiB will be used even if a higher value is set" :type: "string" ``` ```{config:option} truenas.remove_snapshots storage_volume_truenas-common :condition: "-" :default: "same as `volume.truenas.remove_snapshots` or `false`" :shortdesc: "Remove snapshots as needed" :type: "bool" ``` ```{config:option} truenas.use_refquota storage_volume_truenas-common :condition: "-" :default: "same as `volume.truenas.use_refquota` or `false`" :shortdesc: "Use `refquota` instead of `quota` for space" :type: "bool" ``` ```{config:option} block.create_options storage_volume_zfs-common :condition: "block-based volume with content type `filesystem` (`zfs.block_mode` enabled)" :default: "same as `volume.block.create_options`" :shortdesc: "Additional options to pass to the file system creation tool when formatting the volume" :type: "string" ``` ```{config:option} block.filesystem storage_volume_zfs-common :condition: "block-based volume with content type `filesystem` (`zfs.block_mode` enabled)" :default: "same as `volume.block.filesystem`" :shortdesc: "{{block_filesystem}}" :type: "string" ``` ```{config:option} block.mount_options storage_volume_zfs-common :condition: "block-based volume with content type `filesystem` (`zfs.block_mode` enabled)" :default: "same as `volume.block.mount_options`" :shortdesc: "Mount options for block-backed file system volumes" :type: "string" ``` ```{config:option} initial.gid storage_volume_zfs-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.gid` or `0`" :shortdesc: "GID of the volume owner in the instance" :type: "int" ``` ```{config:option} initial.mode storage_volume_zfs-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.mode` or `711`" :shortdesc: "Mode of the volume in the instance" :type: "int" ``` ```{config:option} initial.uid storage_volume_zfs-common :condition: "custom volume with content type `filesystem`" :default: "same as `volume.initial.uid` or `0`" :shortdesc: "UID of the volume owner in the instance" :type: "int" ``` ```{config:option} security.shared storage_volume_zfs-common :condition: "custom block volume" :default: "same as `volume.security.shared` or `false`" :shortdesc: "Enable sharing the volume across multiple instances" :type: "bool" ``` ```{config:option} security.shifted storage_volume_zfs-common :condition: "custom volume" :default: "same as `volume.security.shifted` or `false`" :shortdesc: "{{enable_ID_shifting}}" :type: "bool" ``` ```{config:option} security.unmapped storage_volume_zfs-common :condition: "custom volume" :default: "same as `volume.security.unmapped` or `false`" :shortdesc: "Disable ID mapping for the volume" :type: "bool" ``` ```{config:option} size storage_volume_zfs-common :condition: "-" :default: "same as `volume.size`" :shortdesc: "Size/quota of the storage volume" :type: "string" ``` ```{config:option} snapshots.expiry storage_volume_zfs-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.expiry.manual storage_volume_zfs-common :condition: "custom volume" :default: "same as `volume.snapshot.expiry.manual`" :shortdesc: "{{snapshot_expiry_format}}" :type: "string" {{snapshot_expiry_detail}} ``` ```{config:option} snapshots.pattern storage_volume_zfs-common :condition: "custom volume" :default: "same as `volume.snapshot.pattern` or `snap%d`" :shortdesc: "{{snapshot_pattern_format}} [^*]" :type: "string" ``` ```{config:option} snapshots.schedule storage_volume_zfs-common :condition: "custom volume" :default: "same as `volume.snapshot.schedule`" :shortdesc: "{{snapshot_schedule_format}}" :type: "string" ``` ```{config:option} zfs.block_mode storage_volume_zfs-common :condition: "-" :default: "same as `volume.zfs.block_mode`" :shortdesc: "Whether to use a formatted `zvol` rather than a {spellexception}`dataset` (`zfs.block_mode` can be set only for custom storage volumes; use `volume.zfs.block_mode` to enable ZFS block mode for all storage volumes in the pool, including instance volumes)" :type: "bool" ``` ```{config:option} zfs.blocksize storage_volume_zfs-common :condition: "-" :default: "same as `volume.zfs.blocksize`" :shortdesc: "Size of the ZFS block in range from 512 bytes to 16 MiB (must be power of 2) - for block volume, a maximum value of 128 KiB will be used even if a higher value is set" :type: "string" ``` ```{config:option} zfs.delegate storage_volume_zfs-common :condition: "ZFS 2.2 or higher" :default: "same as `volume.zfs.delegate`" :shortdesc: "Controls whether to delegate the ZFS dataset and anything underneath it to the container(s) using it. Allows the use of the `zfs` command in the container" :type: "bool" ``` ```{config:option} zfs.remove_snapshots storage_volume_zfs-common :condition: "-" :default: "same as `volume.zfs.remove_snapshots` or `false`" :shortdesc: "Remove snapshots as needed" :type: "bool" ``` ```{config:option} zfs.reserve_space storage_volume_zfs-common :condition: "-" :default: "same as `volume.zfs.reserve_space` or `false`" :shortdesc: "Use `reservation`/`refreservation` along with `quota`/`refquota`" :type: "bool" ``` ```{config:option} zfs.use_refquota storage_volume_zfs-common :condition: "-" :default: "same as `volume.zfsuse_refquota` or `false`" :shortdesc: "Use `refquota` instead of `quota` for space" :type: "bool" ``` ```{config:option} size storage_zfs-common :default: "auto (20% of free disk space, >= 5 GiB and <= 30 GiB)" :scope: "local" :shortdesc: "Size of the storage pool when creating loop-based pools (in bytes, suffixes supported, can be increased to grow storage pool)" :type: "string" ``` ```{config:option} source storage_zfs-common :default: "-" :scope: "local" :shortdesc: "Path to existing block device(s), loop file or ZFS dataset/pool. Multiple block devices should be separated by `,`. When listing block devices, you can also prefix them with `vdev` type. To specify a `vdev` type, use an `=` sign between the `vdev` type and the block devices (e.g., `mirror=/dev/sda,/dev/sdb`). Only `stripe`, `mirror`, `raidz1` and `raidz2` `vdev` types are supported." :type: "string" ``` ```{config:option} source.wipe storage_zfs-common :default: "`false`" :scope: "local" :shortdesc: "Wipe the block device specified in `source` prior to creating the storage pool" :type: "bool" ``` ```{config:option} zfs.clone_copy storage_zfs-common :default: "`true`" :scope: "global" :shortdesc: "Whether to use ZFS lightweight clones rather than full {spellexception}`dataset` copies (Boolean), or `rebase` to copy based on the initial image" :type: "string" ``` ```{config:option} zfs.export storage_zfs-common :default: "`true`" :scope: "global" :shortdesc: "Disable zpool export while unmount performed" :type: "bool" ``` ```{config:option} zfs.pool_name storage_zfs-common :default: "name of the pool" :scope: "local" :shortdesc: "Name of the zpool" :type: "string" ``` --- ### Doc Config Options Cheat Sheet (`doc/config_options_cheat_sheet.md`) --- orphan: true nosearch: true --- # Configuration options ```{important} This page shows how to output configuration option documentation. The content in this page is for demonstration purposes only. ``` Some instance options: ```{config:option} agent.nic_config instance :shortdesc: Set the name and MTU to be the same as the instance devices :default: "`false`" :type: bool :liveupdate: "`no`" :condition: Virtual machine Controls whether to set the name and MTU of the default network interfaces to be the same as the instance devices (this happens automatically for containers) ``` ```{config:option} migration.incremental.memory.iterations instance :shortdesc: Maximum number of transfer operations :condition: container :default: 10 :type: integer :liveupdate: "yes" Maximum number of transfer operations to go through before stopping the instance ``` ```{config:option} cluster.evacuate instance :shortdesc: What to do when evacuating the instance :default: "`auto`" :type: string :liveupdate: "no" Controls what to do when evacuating the instance (`auto`, `migrate`, `live-migrate`, or `stop`) ``` These need the `instance` scope to be specified as second argument. The default scope is `server`, so this argument isn't required. Some server options: ```{config:option} backups.compression_algorithm server :shortdesc: Compression algorithm for images :type: string :scope: global :default: "`gzip`" Compression algorithm to use for new images (`bzip2`, `gzip`, `lzma`, `xz` or `none`) ``` ```{config:option} instances.nic.host_name :shortdesc: How to generate a host name :type: string :scope: global :default: "`random`" If set to `random`, use the random host interface name as the host name; if set to `mac`, generate a host name in the form `inc` (MAC without leading two digits) ``` ```{config:option} instances.placement.scriptlet :shortdesc: Custom automatic instance placement logic :type: string :scope: global Stores the {ref}`clustering-instance-placement-scriptlet` for custom automatic instance placement logic ``` Any other scope is also possible. This scope shows that you can use formatting, mainly in the short description and the description, and the available options. ```{config:option} test1 something :shortdesc: testing Testing. ``` ```{config:option} test2 something :shortdesc: Hello! **bold** and `code` This is the real text. With two paragraphs. And a list: - Item - Item - Item And a table: Key | Type | Scope | Default | Description :-- | :--- | :---- | :------ | :---------- `acme.agree_tos` | bool | global | `false` | Agree to ACME terms of service `acme.ca_url` | string | global | `https://acme-v02.api.letsencrypt.org/directory` | URL to the directory resource of the ACME service `acme.domain` | string | global | - | Domain for which the certificate is issued `acme.email` | string | global | - | Email address used for the account registration ``` ```{config:option} test3 something :shortdesc: testing :default: "`false`" :type: Type :liveupdate: Python parses the options, so "no" is converted to "False" - to prevent this, put quotes around the text ("no" or "`no`") :condition: "yes" :readonly: "`maybe` - also add quotes if the option starts with code" :resource: Resource, :managed: Managed :required: Required :scope: (this is something like "global" or "local", **not** the scope of the option (`server`, `instance`, ...) Content ``` To reference an option, use `{config:option}`. It is not possible to override the link text. Except for server options (default), you must specify the scope. {config:option}`instance:migration.incremental.memory.iterations` {config:option}`something:test1` The index is here: {ref}`config-options` --- ### Doc Container Environment (`doc/container-environment.md`) (container-runtime-environment)= # Container runtime environment Incus attempts to present a consistent environment to all containers it runs. The exact environment will differ slightly based on kernel features and user configuration, but otherwise, it is identical for all containers. ## File system Incus assumes that any image it uses to create a new container comes with at least the following root-level directories: - `/dev` (empty) - `/proc` (empty) - `/sbin/init` (executable) - `/sys` (empty) ## Devices Incus containers have a minimal and ephemeral `/dev` based on a `tmpfs` file system. Since this is a `tmpfs` and not a `devtmpfs` file system, device nodes appear only if manually created. The following standard set of device nodes is set up automatically: - `/dev/console` - `/dev/fd` - `/dev/full` - `/dev/log` - `/dev/null` - `/dev/ptmx` - `/dev/random` - `/dev/stdin` - `/dev/stderr` - `/dev/stdout` - `/dev/tty` - `/dev/urandom` - `/dev/zero` In addition to the standard set of devices, the following devices are also set up for convenience: - `/dev/fuse` - `/dev/net/tun` - `/dev/mqueue` ### Network Incus containers may have any number of network devices attached to them. The naming for those (unless overridden by the user) is `ethX`, where `X` is an incrementing number. ### Container-to-host communication Incus sets up a socket at `/dev/incus/sock` that the root user in the container can use to communicate with Incus on the host. See {doc}`dev-incus` for the API documentation. ## Mounts The following mounts are set up by default: - `/proc` ({spellexception}`proc`) - `/sys` (`sysfs`) - `/sys/fs/cgroup/*` (`cgroupfs`) (only on kernels that lack cgroup namespace support) If they are present on the host, the following paths will also automatically be mounted: - `/proc/sys/fs/binfmt_misc` - `/sys/firmware/efi/efivars` - `/sys/fs/fuse/connections` - `/sys/fs/pstore` - `/sys/kernel/debug` - `/sys/kernel/security` The reason for passing all of those paths is that legacy init systems require them to be mounted, or be mountable, inside the container. The majority of those paths will not be writable (or even readable) from inside an unprivileged container. In privileged containers, they will be blocked by the AppArmor policy. ### LXCFS If LXCFS is present on the host, it is automatically set up for the container. This normally results in a number of `/proc` files being overridden through bind-mounts. ## PID1 Incus spawns whatever is located at `/sbin/init` as the initial process of the container (PID 1). This binary should act as a proper init system, including handling re-parented processes. Incus' communication with PID1 in the container is limited to two signals: - `SIGINT` to trigger a reboot of the container - `SIGPWR` (or alternatively `SIGRTMIN`+3) to trigger a clean shutdown of the container The initial environment of PID1 is blank except for `container=lxc`, which can be used by the init system to detect the runtime. All file descriptors above the default three are closed prior to PID1 being spawned. --- ### Doc Contributing (`doc/contributing.md`) # Contributing to Incus ```{toctree} :maxdepth: 1 Introduction Contribute to the code Contribute to the documentation ``` --- ### Doc Daemon Behavior (`doc/daemon-behavior.md`) (daemon-behavior)= # Daemon behavior This specification covers some of the Incus daemon's behavior. ## Startup On every start, Incus checks that its directory structure exists. If it doesn't, it creates the required directories, generates a key pair and initializes the database. Once the daemon is ready for work, Incus scans the instances table for any instance for which the stored power state differs from the current one. If an instance's power state was recorded as running and the instance isn't running, Incus starts it. ## Signal handling ### `SIGINT`, `SIGQUIT`, `SIGTERM` For those signals, Incus assumes that it's being temporarily stopped and will be restarted at a later time to continue handling the instances. The instances will keep running and Incus will close all connections and exit cleanly. ### `SIGPWR` Indicates to Incus that the host is going down. Incus will attempt a clean shutdown of all the instances. After 30 seconds, it kills any remaining instance. The instance `power_state` in the instances table is kept as it was so that Incus can restore the instances as they were after the host is done rebooting. ### `SIGUSR1` Write a memory profile dump to the file specified with `--memprofile`. --- ### Doc Database (`doc/database.md`) (database)= # About the Incus database Incus uses a distributed database to store the server configuration and state, which allows for quicker queries than if the configuration was stored inside each instance's directory (as it is done by LXC, for example). To understand the advantages, consider a query against the configuration of all instances, like "what instances are using `br0`?". To answer that question without a database, you would have to iterate through every single instance, load and parse its configuration, and then check which network devices are defined in there. With a database, you can run a simple query on the database to retrieve this information. ## Cowsql In an Incus cluster, all members of the cluster must share the same database state. Therefore, Incus uses [Cowsql](https://github.com/cowsql/cowsql), a distributed version of SQLite. Cowsql provides replication, fault-tolerance, and automatic failover without the need of external database processes. When using Incus as a single machine and not as a cluster, the Cowsql database effectively behaves like a regular SQLite database. ## File location The database files are stored in the `database` sub-directory of your Incus data directory (`/var/lib/incus/database/`). Upgrading Incus to a newer version might require updating the database schema. In this case, Incus automatically stores a backup of the database and then runs the update. See {ref}`installing-upgrade` for more information. ## Backup See {ref}`backup-database` for instructions on how to back up the contents of the Incus database. --- ### Doc Debugging (`doc/debugging.md`) # How to debug Incus For information on debugging instance issues, see {ref}`instances-troubleshoot`. ## Debugging `incus` and `incusd` Here are different ways to help troubleshooting `incus` and `incusd` code. ### `incus --debug` Adding `--debug` flag to any client command will give extra information about internals. If there is no useful info, it can be added with the logging call: logger.Debugf("Hello: %s", "Debug") ### `incus monitor` This command will monitor messages as they appear on remote server. ## REST API through local socket On server side the most easy way is to communicate with Incus through local socket. This command accesses `GET /1.0` and formats JSON into human readable form using [jq](https://stedolan.github.io/jq/tutorial/) utility: ```bash curl --unix-socket /var/lib/incus/unix.socket incus/1.0 | jq . ``` See the [RESTful API](rest-api.md) for available API. ## REST API through HTTPS {ref}`HTTPS connection to Incus ` requires valid client certificate that is generated on first [`incus remote add`](incus_remote_add.md). This certificate should be passed to connection tools for authentication and encryption. If desired, `openssl` can be used to examine the certificate (`~/.config/incus/client.crt`): ```bash openssl x509 -text -noout -in client.crt ``` Among the lines you should see: Certificate purposes: SSL client : Yes ### With command line tools ```bash wget --no-check-certificate --certificate=$HOME/.config/incus/client.crt --private-key=$HOME/.config/incus/client.key -qO - https://127.0.0.1:8443/1.0 ``` ### With browser Some browser plugins provide convenient interface to create, modify and replay web requests. To authenticate against Incus server, convert `incus` client certificate into importable format and import it into browser. For example this produces `client.pfx` in Windows-compatible format: ```bash openssl pkcs12 -clcerts -inkey client.key -in client.crt -export -out client.pfx ``` After that, opening [`https://127.0.0.1:8443/1.0`](https://127.0.0.1:8443/1.0) should work as expected. ## Profiling `incusd` The Go `pprof` profiles of `incusd` are available in two ways: - Setting `core.debug_address` starts a plain HTTP listener serving them under `/debug/pprof/`. It has no authentication, so only bind it to a trusted address. - The API serves the same profiles under `/internal/debug/pprof/` on every listener, restricted to clients allowed to administer the server. This works at any time, over HTTPS, without changing the server configuration. For example, to save the heap profile of a remote server through the `incus` client and open it locally: ```bash incus query --raw my-remote:/internal/debug/pprof/heap > heap.pprof go tool pprof heap.pprof ``` The upstream [`pprof`](https://github.com/google/pprof) tool can also fetch the profiles directly with the client certificate (`go tool pprof` doesn't support client certificates): ```bash pprof -tls_cert ~/.config/incus/client.crt -tls_key ~/.config/incus/client.key "https+insecure://192.0.2.10:8443/internal/debug/pprof/profile?seconds=30" ``` ## Debug the Incus database The files of the global {ref}`database ` are stored under the `./database/global` sub-directory of your Incus data directory (`/var/lib/incus/database/global`). Since each member of the cluster also needs to keep some data which is specific to that member, Incus also uses a plain SQLite database (the "local" database), which you can find in `./database/local.db`. Backups of the global database directory and of the local database file are made before upgrades, and are tagged with the `.bak` suffix. You can use those if you need to revert the state as it was before the upgrade. ### Dumping the database content or schema If you want to get a SQL text dump of the content or the schema of the databases, use the `incus admin sql [.dump|.schema]` command, which produces the equivalent output of the `.dump` or `.schema` directives of the `sqlite3` command line tool. ### Running custom queries from the console If you need to perform SQL queries (e.g. `SELECT`, `INSERT`, `UPDATE`) against the local or global database, you can use the `incus admin sql` command (run `incus admin sql --help` for details). You should only need to do that in order to recover from broken updates or bugs. Please consult the Incus team first (creating a [GitHub issue](https://github.com/lxc/incus/issues/new) or [forum](https://discuss.linuxcontainers.org) post). ### Running custom queries at Incus daemon startup In case the Incus daemon fails to start after an upgrade because of SQL data migration bugs or similar problems, it's possible to recover the situation by creating `.sql` files containing queries that repair the broken update. To perform repairs against the local database, write a `./database/patch.local.sql` file containing the relevant queries, and similarly a `./database/patch.global.sql` for global database repairs. Those files will be loaded very early in the daemon startup sequence and deleted if the queries were successful (if they fail, no state will change as they are run in a SQL transaction). As above, please consult the Incus team first. ### Syncing the cluster database to disk If you want to flush the content of the cluster database to disk, use the `incus admin sql global .sync` command, that will write a plain SQLite database file into `./database/global/db.bin`, which you can then inspect with the `sqlite3` command line tool. --- ### Doc Dev Incus (`doc/dev-incus.md`) (dev-incus)= # Communication between instance and host Communication between the hosted workload (instance) and its host while not strictly needed is a pretty useful feature. In Incus, this feature is implemented through a `/dev/incus/sock` node which is created and set up for all Incus instances. This file is a Unix socket which processes inside the instance can connect to. It's multi-threaded so multiple clients can be connected at the same time. ```{note} {config:option}`instance-security:security.guestapi` must be set to `true` (which is the default) for an instance to allow access to the socket. ``` ## Implementation details Incus on the host binds `/var/lib/incus/guestapi/sock` and starts listening for new connections on it. This socket is then exposed into every single instance started by Incus at `/dev/incus/sock`. The single socket is required so we can exceed 4096 instances, otherwise, Incus would have to bind a different socket for every instance, quickly reaching the FD limit. ## Authentication Queries on `/dev/incus/sock` will only return information related to the requesting instance. To figure out where a request comes from, Incus will extract the initial socket's user credentials and compare that to the list of instances it manages. ## Protocol The protocol on `/dev/incus/sock` is plain-text HTTP with JSON messaging, so very similar to the local version of the Incus protocol. Unlike the main Incus API, there is no background operation and no authentication support in the `/dev/incus/sock` API. ## REST-API ### API structure * `/` * `/1.0` * `/1.0/config` * `/1.0/config/{key}` * `/1.0/devices` * `/1.0/events` * `/1.0/images/{fingerprint}/export` * `/1.0/meta-data` ### API details #### `/` ##### GET * Description: List of supported APIs * Return: list of supported API endpoint URLs (by default `['/1.0']`) Return value: ```json [ "/1.0" ] ``` #### `/1.0` ##### GET * Description: Information about the 1.0 API * Return: JSON object Return value: ```json { "api_version": "1.0", "location": "foo.example.com", "instance_type": "container", "state": "Started", } ``` #### PATCH * Description: Update instance state (valid states are `Ready` and `Started`) * Return: none Input: ```json { "state": "Ready" } ``` #### `/1.0/config` ##### GET * Description: List of configuration keys * Return: list of configuration keys URL Note that the configuration key names match those in the instance configuration, however not all configuration namespaces will be exported to `/dev/incus/sock`. Currently only the `cloud-init.*` and `user.*` keys are accessible to the instance. At this time, there also aren't any instance-writable namespace. Return value: ```json [ "/1.0/config/user.a" ] ``` #### `/1.0/config/` ##### GET * Description: Value of that key * Return: Plain-text value Return value: blah #### `/1.0/devices` ##### GET * Description: Map of instance devices * Return: JSON object Return value: ```json { "eth0": { "name": "eth0", "network": "incusbr0", "type": "nic" }, "root": { "path": "/", "pool": "default", "type": "disk" } } ``` #### `/1.0/events` ##### GET * Description: WebSocket upgrade * Return: none (never ending flow of events) Supported arguments are: * type: comma-separated list of notifications to subscribe to (defaults to all) The notification types are: * `config` (changes to any of the `user.*` configuration keys) * `device` (any device addition, change or removal) This never returns. Each notification is sent as a separate JSON object: ```json { "timestamp": "2017-12-21T18:28:26.846603815-05:00", "type": "device", "metadata": { "name": "kvm", "action": "added", "config": { "type": "unix-char", "path": "/dev/kvm" } } } ``` ```json { "timestamp": "2017-12-21T18:28:26.846603815-05:00", "type": "config", "metadata": { "key": "user.foo", "old_value": "", "value": "bar" } } ``` #### `/1.0/images//export` ##### GET * Description: Download a public/cached image from the host * Return: raw image or error * Access: Requires `security.guestapi.images` set to `true` Return value: See /1.0/images//export in the daemon API. #### `/1.0/meta-data` ##### GET * Description: Container meta-data compatible with cloud-init * Return: cloud-init meta-data Return value: #cloud-config instance-id: af6a01c7-f847-4688-a2a4-37fddd744625 local-hostname: abc --- ### Doc Doc Cheat Sheet (`doc/doc-cheat-sheet.md`) --- orphan: true nosearch: true myst: substitutions: reuse_key: "This is **included** text." advanced_reuse_key: "This is a substitution that includes a code block: ``` code block ```" --- # Documentation cheat sheet The documentation files use a mixture of [Markdown](https://commonmark.org/) and [MyST](https://myst-parser.readthedocs.io/) syntax. See the following sections for syntax help and conventions. ## Headings ```{list-table} :header-rows: 1 * - Input - Description * - `# Title` - Page title and H1 heading * - `## Heading` - H2 heading * - `### Heading` - H3 heading * - `#### Heading` - H4 heading * - ... - Further headings ``` Adhere to the following conventions: - Do not use consecutive headings without intervening text. - Use sentence style for headings (capitalize only the first word). - Do not skip levels (for example, always follow an H2 with an H3, not an H4). ## Inline formatting ```{list-table} :header-rows: 1 * - Input - Output * - `` {guilabel}`UI element` `` - {guilabel}`UI element` * - `` `code` `` - `code` * - `` {command}`command` `` - {command}`command` * - `*Italic*` - *Italic* * - `**Bold**` - **Bold** ``` Adhere to the following conventions: - Use italics sparingly. Common uses for italics are titles and names (for example, when referring to a section title that you cannot link to, or when introducing the name for a concept). - Use bold sparingly. A common use for bold is UI elements ("Click **OK**"). Avoid using bold for emphasis and rather rewrite the sentence to get your point across. ## Code blocks Start and end a code block with three back ticks: ``` You can specify the code language after the back ticks to enforce a specific lexer, but in many cases, the default lexer works just fine. ```{list-table} :header-rows: 1 * - Input - Output * - ```` ``` # Demonstrate a code block code: - example: true ``` ```` - ``` # Demonstrate a code block code: - example: true ``` * - ```` ```yaml # Demonstrate a code block code: - example: true ``` ```` - ```yaml # Demonstrate a code block code: - example: true ``` ``` To include back ticks in a code block, increase the number of surrounding back ticks: ```{list-table} :header-rows: 1 * - Input - Output * - ````` ```` ``` ```` ````` - ```` ``` ```` ``` ## Links How to link depends on if you are linking to an external URL or to another page in the documentation. ### External links For external links, use only the URL, or Markdown syntax if you want to override the link text. ```{list-table} :header-rows: 1 * - Input - Output * - `https://linuxcontainers.org/incus` - [{spellexception}`https://linuxcontainers.org/incus`](https://linuxcontainers.org/incus) * - `[Incus](https://linuxcontainers.org/incus)` - [Incus](https://linuxcontainers.org/incus) ``` To display a URL as text and prevent it from being linked, add a ``: ```{list-table} :header-rows: 1 * - Input - Output * - `https://linuxcontainers.org/incus` - {spellexception}`https://linuxcontainers.org/incus` ``` ### Internal references For internal references, both Markdown and MyST syntax are supported. In most cases, you should use MyST syntax though, because it resolves the link text automatically and gives an indication of the link in GitHub rendering. #### Referencing a page To reference a documentation page, use MyST syntax to automatically extract the link text. When overriding the link text, use Markdown syntax. ```{list-table} :header-rows: 1 * - Input - Output - Output on GitHub - Status * - `` {doc}`index` `` - {doc}`index` - {doc}`index` - Preferred. * - `[](index)` - [](index) - - Do not use. * - `[Incus documentation](index)` - [Incus documentation](index) - [Incus documentation](index) - Preferred when overriding the link text. * - `` {doc}`Incus documentation ` `` - {doc}`Incus documentation ` - {doc}`Incus documentation ` - Alternative when overriding the link text. ``` Adhere to the following conventions: - Override the link text only when it is necessary. If you can use the document title as link text, do so, because the text will then update automatically if the title changes. - Never "override" the link text with the same text that would be generated automatically. (a_section_target)= #### Referencing a section To reference a section within the documentation (on the same page or on another page), you can either add a target to it and reference that target, or you can use an automatically generated anchor in combination with the file name. Adhere to the following conventions: - Add targets for sections that are central and a "typical" place to link to, so you expect they will be linked frequently. For "one-off" links, you can use the automatically generated anchors. - Override the link text only when it is necessary. If you can use the section title as link text, do so, because the text will then update automatically if the title changes. - Never "override" the link text with the same text that would be generated automatically. ##### Using a target You can add targets at any place in the documentation. However, if there is no heading or title for the targeted element, you must specify a link text. (a_random_target)= ```{list-table} :header-rows: 1 * - Input - Output - Output on GitHub - Description * - `(target_ID)=` - - \(target_ID\)= - Adds the target ``target_ID``. * - `` {ref}`a_section_target` `` - {ref}`a_section_target` - \{ref\}`a_section_target` - References a target that has a title. * - `` {ref}`link text ` `` - {ref}`link text ` - \{ref\}`link text ` - References a target and specifies a title. * - ``[`option name\](a_random_target)`` - [`option name`](a_random_target) - [`option name`](a_random_target) (link is broken) - Use Markdown syntax if you need markup on the link text. ``` ##### Using an automatically generated anchor You must use Markdown syntax to use automatically generated anchors. You can leave out the file name when linking within the same file. ```{list-table} :header-rows: 1 * - Input - Output - Output on GitHub - Description * - `[](#referencing-a-section)` - [](#referencing-a-section) - - Do not use. * - `[link text](#referencing-a-section)` - [link text](#referencing-a-section) - [link text](#referencing-a-section) - Preferred when overriding the link text. ``` ## Navigation Every documentation page must be included as a subpage to another page in the navigation. This is achieved with the [`toctree`](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-toctree) directive in the parent page: ```` ```{toctree} :hidden: subpage1 subpage2 ``` ```` If a page should not be included in the navigation, you can suppress the resulting build warning by putting the following instruction at the top of the file: ``` --- orphan: true --- ``` Use orphan pages sparingly and only if there is a clear reason for it. ## Lists ```{list-table} :header-rows: 1 * - Input - Output * - ``` - Item 1 - Item 2 - Item 3 ``` - - Item 1 - Item 2 - Item 3 * - ``` 1. Step 1 1. Step 2 1. Step 3 ``` - 1. Step 1 1. Step 2 1. Step 3 * - ``` 1. Step 1 - Item 1 * Subitem - Item 2 1. Step 2 1. Substep 1 1. Substep 2 ``` - 1. Step 1 - Item 1 * Subitem - Item 2 1. Step 2 1. Substep 1 1. Substep 2 ``` Adhere to the following conventions: - In numbered lists, use ``1.`` for all items to generate the step numbers automatically. - Use `-` for unordered lists. When using nested lists, you can use `*` for the nested level. ### Definition lists ```{list-table} :header-rows: 1 * - Input - Output * - ``` Term 1 : Definition Term 2 : Definition ``` - Term 1 : Definition Term 2 : Definition ``` ## Tables You can use standard Markdown tables. However, using the rST [list table](https://docutils.sourceforge.io/docs/ref/rst/directives.html#list-table) syntax is usually much easier. Both markups result in the following output: ```{list-table} :header-rows: 1 * - Header 1 - Header 2 * - Cell 1 Second paragraph cell 1 - Cell 2 * - Cell 3 - Cell 4 ``` ### Markdown tables ``` | Header 1 | Header 2 | |------------------------------------|----------| | Cell 1

2nd paragraph cell 1 | Cell 2 | | Cell 3 | Cell 4 | ``` ### List tables ```` ```{list-table} :header-rows: 1 * - Header 1 - Header 2 * - Cell 1 2nd paragraph cell 1 - Cell 2 * - Cell 3 - Cell 4 ``` ```` ## Notes ```{list-table} :header-rows: 1 * - Input - Output * - ```` ```{note} A note. ``` ```` - ```{note} A note. ``` * - ```` ```{tip} A tip. ``` ```` - ```{tip} A tip. ``` * - ```` ```{important} Important information ``` ```` - ```{important} Important information. ``` * - ```` ```{caution} This might damage your hardware! ``` ```` - ```{caution} This might damage your hardware! ``` ``` Adhere to the following conventions: - Use notes sparingly. - Only use the following note types: `note`, `tip`, `important`, `caution` - Only use a caution if there is a clear hazard of hardware damage or data loss. ## Images ```{list-table} :header-rows: 1 * - Input - Output * - ``` ``` - * - ```` ```{figure} https://linuxcontainers.org/incus/docs/main/_static/tag.png :width: 100px :alt: Alt text Figure caption ``` ```` - ```{figure} https://linuxcontainers.org/incus/docs/main/_static/tag.png :width: 100px :alt: Alt text Figure caption ``` ``` Adhere to the following conventions: - For pictures in the `doc` directory, start the path with `/` (for example, `/images/image.png`). - Use PNG format for screenshots and SVG format for graphics. ## Reuse A big advantage of MyST in comparison to plain Markdown is that it allows to reuse content. ### Substitution To reuse sentences or paragraphs without too much markup and special formatting, use substitutions. Substitutions can be defined in the following locations: - In the `substitutions.yaml` file. Substitutions defined in this file are available in all documentation pages. - At the top of a single file in the following format: ```` --- myst: substitutions: reuse_key: "This is **included** text." advanced_reuse_key: "This is a substitution that includes a code block: ``` code block ```" --- ```` You can combine both options by defining a default substitution in `reuse/substitutions.py` and overriding it at the top of a file. ```{list-table} :header-rows: 1 * - Input - Output * - `{{reuse_key}}` - {{reuse_key}} * - `{{advanced_reuse_key}}` - {{advanced_reuse_key}} ``` Adhere to the following convention: - Substitutions do not work on GitHub. Therefore, use key names that indicate the included text (for example, `note_not_supported` instead of `reuse_note`). ### File inclusion To reuse longer sections or text with more advanced markup, you can put the content in a separate file and include the file or parts of the file in several locations. You cannot put any targets into the content that is being reused (because references to this target would be ambiguous then). You can, however, put a target right before including the file. By combining file inclusion and substitutions, you can even replace parts of the included text. `````{list-table} :header-rows: 1 * - Input - Output * - ```` % Include parts of the content from file [../README.md](../README.md) ```{include} ../README.md :start-after: :end-before: ``` ```` - % Include parts of the content from file [../README.md](../README.md) ```{include} ../README.md :start-after: :end-before: ``` ````` Adhere to the following convention: - File inclusion does not work on GitHub. Therefore, always add a comment linking to the included file. - To select parts of the text, add HTML comments for the start and end points and use `:start-after:` and `:end-before:`, if possible. You can combine `:start-after:` and `:end-before:` with `:start-line:` and `:end-line:` if required. Using only `:start-line:` and `:end-line:` is error-prone though. ## Tabs ``````{list-table} :header-rows: 1 * - Input - Output * - ````` ````{tabs} ```{group-tab} Tab 1 Content Tab 1 ``` ```{group-tab} Tab 2 Content Tab 2 ``` ```` ````` - ````{tabs} ```{group-tab} Tab 1 Content Tab 1 ``` ```{group-tab} Tab 2 Content Tab 2 ``` ```` `````` ## Collapsible sections There is no support for details sections in rST, but you can insert HTML to create them. ```{list-table} :header-rows: 1 * - Input - Output * - ```
Details Content
``` -
Details Content
``` ## Glossary You can define glossary terms in any file. Ideally, all terms should be collected in one glossary file though, and they can then be referenced from any file. `````{list-table} :header-rows: 1 * - Input - Output * - ```` ```{glossary} example term Definition of the example term. ``` ```` - ```{glossary} example term Definition of the example term. ``` * - ``{term}`example term` `` - {term}`example term` ````` ## More useful markup `````{list-table} :header-rows: 1 * - Input - Output * - ```` ```{versionadded} X.Y ``` ```` - ```{versionadded} X.Y ``` * - `` {abbr}`API (Application Programming Interface)` `` - {abbr}`API (Application Programming Interface)` ````` ## Custom extensions The documentation uses some custom extensions. ### Related links You can add links to related websites to the sidebar by adding the following field at the top of the page: relatedlinks: https://github.com/canonical/lxd-sphinx-extensions, [RTFM](https://www.google.com) To override the title, use Markdown syntax. Note that spaces are ignored; if you need spaces in the title, replace them with ` `, and include the value in quotes if Sphinx complains about the metadata value because it starts with `[`. To add a link to a Discourse topic, add the following field at the top of the page (where `12345` is the ID of the Discourse topic): discourse: 12345 ### YouTube links To add a link to a YouTube video, use the following directive: `````{list-table} :header-rows: 1 * - Input - Output * - ```` ```{youtube} https://www.youtube.com/watch?v=iMLiK1fX4I0 :title: Demo ``` ```` - ```{youtube} https://www.youtube.com/watch?v=iMLiK1fX4I0 :title: Demo ``` ````` The video title is extracted automatically and displayed when hovering over the link. To override the title, add the `:title:` option. ### Spelling exceptions If you need to use a word that does not comply to the spelling conventions, but is correct in a certain context, you can exempt it from the spelling checker by surrounding it with `{spellexception}`. ```{list-table} :header-rows: 1 * - Input - Output * - `` {spellexception}`PurposelyWrong` `` - {spellexception}`PurposelyWrong` ``` ### Terminal output To show a terminal view with commands and output, use the following directive: `````{list-table} :header-rows: 1 * - Input - Output * - ```` ```{terminal} :input: command number one :user: root :host: vm output line one output line two :input: another command more output ``` ```` - ```{terminal} :input: command number one :user: root :host: vm output line one output line two :input: another command more output ``` ````` Input is specified as the `:input:` option (or prefixed with `:input:` as part of the main content of the directive). Output is the main content of the directive. To override the prompt (`user@host:~$` by default), specify the `:user:` and/or `:host:` options. To make the terminal scroll horizontally instead of wrapping long lines, add `:scroll:`. --- ### Doc Environment (`doc/environment.md`) # Environment variables The Incus client and daemon respect some environment variables to adapt to the user's environment and to turn some advanced features on and off. ## Common | Name | Description | | :--- | :--- | | `INCUS_DIR` | The Incus data directory | | `INCUS_INSECURE_TLS` | If set to true, allows all default Go ciphers both for client <-> server communication and server <-> image servers (server <-> server and clustering are not affected) | | `PATH` | List of paths to look into when resolving binaries | | `http_proxy` | Proxy server URL for HTTP | | `https_proxy` | Proxy server URL for HTTPS | | `no_proxy` | List of domains, IP addresses or CIDR ranges that don't require the use of a proxy | ## Client environment variable | Name | Description | | :--- | :--- | | `EDITOR` | What text editor to use | | `INCUS_CONF` | Path to the client configuration directory | | `INCUS_GLOBAL_CONF` | Path to the global client configuration directory | | `INCUS_PROJECT` | Name of the project to use (overrides configured default project) | | `INCUS_REMOTE` | Name of the remote to use (overrides configured default remote) | | `VISUAL` | What text editor to use (if `EDITOR` isn't set) | ## Server environment variable | Name | Description | | :--- | :--- | | `INCUS_AGENT_PATH` | Path to the directory including the `incus-agent` builds | | `INCUS_CLUSTER_UPDATE` | Script to call on a cluster update | | `INCUS_DEVMONITOR_DIR` | Path to be monitored by the device monitor. This is primarily for testing | | `INCUS_DOCUMENTATION` | Path to the documentation to serve through the web server | | `INCUS_EDK2_PATH` | Path to EDK2 firmware build including `*_CODE.fd` and `*_VARS.fd` | | `INCUS_EXEC_PATH` | Full path to the Incus binary (used when forking subcommands) | | `INCUS_IDMAPPED_MOUNTS_DISABLE` | Disable idmapped mounts support (useful when testing traditional UID shifting) | | `INCUS_LXC_TEMPLATE_CONFIG` | Path to the LXC template configuration directory | | `INCUS_SECURITY_APPARMOR` | If set to `false`, forces AppArmor off | | `INCUS_SECURITY_SELINUX` | If set to `true`, turns on SELinux integration | | `INCUS_SKIP_INSTANCE_TYPES` | If set to `true`, skip downloading instance type definitions | | `INCUS_UI` | Path to the web UI to serve through the web server | | `INCUS_USBIDS_PATH` | Path to the hwdata `usb.ids` file | --- ### Doc Events (`doc/events.md`) # Events ## Introduction Events are messages about actions that have occurred over Incus. Using the API endpoint `/1.0/events` directly or via [`incus monitor`](incus_monitor.md) will connect to a WebSocket through which logs and life-cycle messages will be streamed. ## Event types Incus Currently supports three event types. - `logging`: Shows all logging messages regardless of the server logging level. - `operation`: Shows all ongoing operations from creation to completion (including updates to their state and progress metadata). - `lifecycle`: Shows an audit trail for specific actions occurring over Incus. ## Event ordering Events are delivered to each listener in the order that Incus generated them. A listener that cannot keep up with the stream is disconnected rather than having events silently skipped. After reconnecting, anything being tracked from the event stream should be re-fetched, as the events that occurred while disconnected are not resent. When using the Go client, handlers registered through `AddHandler` are called concurrently and may therefore observe events out of order. Use `AddChannel` instead to receive the events on a channel, which delivers them one at a time and in the order they arrived, and is closed once the listener ends. Its size argument sets how far behind the reader may fall before the listener is dropped. ## Event structure ### Example ```yaml location: cluster_name metadata: action: network-updated requestor: protocol: unix username: root source: /1.0/networks/incusbr0 timestamp: "2021-03-14T00:00:00Z" type: lifecycle ``` - `location`: The cluster member name (if clustered). - `timestamp`: Time that the event occurred in RFC3339 format. - `type`: The type of event this is (one of `logging`, `operation`, or `lifecycle`). - `metadata`: Information about the specific event type. ### Logging event structure - `message`: The log message. - `level`: The log-level of the log. - `context`: Additional information included in the event. ### Operation event structure - `id`: The UUID of the operation. - `class`: The type of operation (`task`, `token`, or `websocket`). - `description`: A description of the operation. - `created_at`: The operation's creation date. - `updated_at`: The operation's date of last change. - `status`: The current state of the operation. - `status_code`: The operation status code. - `resources`: Resources affected by this operation. - `metadata`: Operation specific metadata. - `may_cancel`: Whether the operation may be canceled. - `err`: Error message of the operation. - `location`: The cluster member name (if clustered). ### Life-cycle event structure - `action`: The life-cycle action that occurred. - `requestor`: Information about who is making the request (if applicable). - `source`: Path to what is being acted upon. - `context`: Additional information included in the event. ## Supported life-cycle events | Name | Description | Additional Information | | :------------------------------------- | :-------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------- | | `certificate-created` | A new certificate has been added to the server trust store. | | | `certificate-deleted` | The certificate has been deleted from the trust store. | | | `certificate-updated` | The certificate's configuration has been updated. | | | `cluster-certificate-updated` | The certificate for the whole cluster has changed. | | | `cluster-disabled` | Clustering has been disabled for this machine. | | | `cluster-enabled` | Clustering has been enabled for this machine. | | | `cluster-group-created` | A new cluster group has been created. | | | `cluster-group-deleted` | A cluster group has been deleted. | | | `cluster-group-renamed` | A cluster group has been renamed. | | | `cluster-group-updated` | A cluster group has been updated. | | | `cluster-member-added` | A new machine has joined the cluster. | | | `cluster-member-removed` | The cluster member has been removed from the cluster. | | | `cluster-member-renamed` | The cluster member has been renamed. | `old_name`: the previous name. | | `cluster-member-updated` | The cluster member's configuration been edited. | | | `cluster-token-created` | A join token for adding a cluster member has been created. | | | `config-updated` | The server configuration has changed. | | | `image-alias-created` | An alias has been created for an existing image. | `target`: the original instance. | | `image-alias-deleted` | An alias has been deleted for an existing image. | `target`: the original instance. | | `image-alias-renamed` | The alias for an existing image has been renamed. | `old_name`: the previous name. | | `image-alias-updated` | The configuration for an image alias has changed. | `target`: the original instance. | | `instance-agent-started` | The instance agent has connected to the host. | | | `instance-agent-stopped` | The instance agent has disconnected from the host. | | | `image-created` | A new image has been added to the image store. | `type`: `container` or `vm`. | | `image-deleted` | The image has been deleted from the image store. | | | `image-refreshed` | The local image copy has updated to the current source image version. | | | `image-retrieved` | The raw image file has been downloaded from the server. | `target`: destination server. | | `image-secret-created` | A one-time key to fetch this image has been created. | | | `image-updated` | The image's configuration has changed. | | | `instance-backup-created` | A backup of the instance has been created. | | | `instance-backup-deleted` | The instance backup has been deleted. | | | `instance-backup-renamed` | The instance backup has been renamed. | `old_name`: the previous name. | | `instance-backup-retrieved` | The raw instance backup file has been downloaded. | | | `instance-console` | Connected to the console of the instance. | `type`: `console` or `vga`. | | `instance-console-reset` | The console buffer has been reset. | | | `instance-console-retrieved` | The console log has been downloaded. | | | `instance-created` | A new instance has been created. | | | `instance-deleted` | The instance has been deleted. | | | `instance-exec` | A command has been executed on the instance. | `command`: the command to be executed. | | `instance-file-deleted` | A file on the instance has been deleted. | `file`: path to the file. | | `instance-file-pushed` | The file has been pushed to the instance. | `file-source`: local file path. `file-destination`: destination file path. `info`: file information. | | `instance-file-retrieved` | The file has been downloaded from the instance. | `file-source`: instance file path. `file-destination`: destination file path. | | `instance-log-deleted` | The instance's specified log file has been deleted. | | | `instance-log-retrieved` | The instance's specified log file has been downloaded. | | | `instance-metadata-retrieved` | The instance's image metadata has been downloaded. | | | `instance-metadata-template-created` | A new image template file for the instance has been created. | `path`: relative file path. | | `instance-metadata-template-deleted` | The image template file for the instance has been deleted. | `path`: relative file path. | | `instance-metadata-template-retrieved` | The image template file for the instance has been downloaded. | `path`: relative file path. | | `instance-metadata-updated` | The instance's image metadata has changed. | | | `instance-migrated` | The instance has been migrated. | | | `instance-paused` | The instance has been put in a paused state. | | | `instance-ready` | The instance is ready. | | | `instance-renamed` | The instance has been renamed. | `old_name`: the previous name. | | `instance-restarted` | The instance has restarted. | | | `instance-restored` | The instance has been restored from a snapshot. | `snapshot`: name of the snapshot being restored. | | `instance-resumed` | The instance has resumed after being paused. | | | `instance-shutdown` | The instance has shut down. | | | `instance-snapshot-created` | A snapshot of the instance has been created. | | | `instance-snapshot-deleted` | The instance snapshot has been deleted. | | | `instance-snapshot-renamed` | The instance snapshot has been renamed. | `old_name`: the previous name. | | `instance-snapshot-updated` | The instance snapshot's configuration has changed. | | | `instance-started` | The instance has started. | | | `instance-stopped` | The instance has stopped. | | | `instance-updated` | The instance's configuration has changed. | | | `network-acl-created` | A new network ACL has been created. | | | `network-acl-deleted` | The network ACL has been deleted. | | | `network-acl-renamed` | The network ACL has been renamed. | `old_name`: the previous name. | | `network-acl-updated` | The network ACL configuration has changed. | | | `network-created` | A network device has been created. | | | `network-deleted` | The network device has been deleted. | | | `network-forward-created` | A new network forward has been created. | | | `network-forward-deleted` | The network forward has been deleted. | | | `network-forward-updated` | The network forward has been updated. | | | `network-load-balancer-created` | A new network load balancer has been created. | | | `network-load-balancer-deleted` | The network load balancer has been deleted. | | | `network-load-balancer-health-changed` | The network load balancer has gone healthy or unhealthy. | `status`: `healthy` or `unhealthy`. `service_monitors`: list of the backends' service monitor statuses. | | `network-load-balancer-updated` | The network load balancer has been updated. | | | `network-peer-created` | A new network peer has been created. | | | `network-peer-deleted` | The network peer has been deleted. | | | `network-peer-updated` | The network peer has been updated. | | | `network-renamed` | The network device has been renamed. | `old_name`: the previous name. | | `network-updated` | The network device's configuration has changed. | | | `network-zone-created` | A new network zone has been created. | | | `network-zone-deleted` | The network zone has been deleted. | | | `network-zone-record-created` | A new network zone record has been created. | | | `network-zone-record-deleted` | The network zone record has been deleted. | | | `network-zone-record-updated` | The network zone record has been updated. | | | `network-zone-updated` | The network zone has been updated. | | | `operation-cancelled` | The operation has been canceled. | | | `profile-created` | A new profile has been created. | | | `profile-deleted` | The profile has been deleted. | | | `profile-renamed` | The profile has been renamed . | `old_name`: the previous name. | | `profile-updated` | The profile's configuration has changed. | | | `project-created` | A new project has been created. | | | `project-deleted` | The project has been deleted. | | | `project-renamed` | The project has been renamed. | `old_name`: the previous name. | | `project-updated` | The project's configuration has changed. | | | `storage-pool-created` | A new storage pool has been created. | `target`: cluster member name. | | `storage-pool-deleted` | The storage pool has been deleted. | | | `storage-pool-updated` | The storage pool's configuration has changed. | `target`: cluster member name. | | `storage-volume-backup-created` | A new backup for the storage volume has been created. | `type`: `container`, `virtual-machine`, `image`, or `custom`. | | `storage-volume-backup-deleted` | The storage volume's backup has been deleted. | | | `storage-volume-backup-renamed` | The storage volume's backup has been renamed. | `old_name`: the previous name. | | `storage-volume-backup-retrieved` | The storage volume's backup has been downloaded. | | | `storage-volume-created` | A new storage volume has been created. | `type`: `container`, `virtual-machine`, `image`, or `custom`. | | `storage-volume-deleted` | The storage volume has been deleted. | | | `storage-volume-renamed` | The storage volume has been renamed. | `old_name`: the previous name. | | `storage-volume-restored` | The storage volume has been restored from a snapshot. | `snapshot`: name of the snapshot being restored. | | `storage-volume-snapshot-created` | A new storage volume snapshot has been created. | `type`: `container`, `virtual-machine`, `image`, or `custom`. | | `storage-volume-snapshot-deleted` | The storage volume's snapshot has been deleted. | | | `storage-volume-snapshot-renamed` | The storage volume's snapshot has been renamed. | `old_name`: the previous name. | | `storage-volume-snapshot-updated` | The configuration for the storage volume's snapshot has changed. | | | `storage-volume-updated` | The storage volume's configuration has changed. | | | `warning-acknowledged` | The warning's status has been set to "acknowledged". | | | `warning-deleted` | The warning has been deleted. | | | `warning-reset` | The warning's status has been set to "new". | | --- ### Doc External Resources (`doc/external_resources.md`) # External resources ```{toctree} :maxdepth: 1 Project repository Image server Third party tools ``` --- ### Doc Faq (`doc/faq.md`) # Frequently asked questions The following sections give answers to frequently asked questions. They explain how to resolve common issues and point you to more detailed information. ## Why do my instances not have network access? Most likely, your firewall blocks network access for your instances. See {ref}`network-bridge-firewall` for more information about the problem and how to fix it. Another frequent reason for connectivity issues is running Incus and Docker on the same host. See {ref}`network-incus-docker` for instructions on how to fix such issues. ## How to enable the Incus server for remote access? By default, the Incus server is not accessible from the network, because it only listens on a local Unix socket. You can enable it for remote access by following the instructions in {ref}`server-expose`. ## When I do a `incus remote add`, it asks for a token? To be able to access the remote API, clients must authenticate with the Incus server. See {ref}`server-authenticate` for instructions on how to authenticate using a trust token. ## Why should I not run privileged containers? A privileged container can do things that affect the entire host - for example, it can use things in `/sys` to reset the network card, which will reset it for the entire host, causing network blips. See {ref}`container-security` for more information. Almost everything can be run in an unprivileged container, or - in cases of things that require unusual privileges, like wanting to mount NFS file systems inside the container - you might need to use bind mounts. ## Can I bind-mount my home directory in a container? Yes, you can do this by using a {ref}`disk device `: incus config device add container-name home disk source=/home/${USER} path=/home/ubuntu For unprivileged containers, you need to make sure that the user in the container has working read/write permissions. Otherwise, all files will show up as the overflow UID/GID (`65536:65536`) and access to anything that's not world-readable will fail. Use either of the following methods to grant the required permissions: - Pass `shift=true` to the [`incus config device add`](incus_config_device_add.md) call. This depends on the kernel and file system supporting either idmapped mounts (see [`incus info`](incus_info.md)). - Add a `raw.idmap` entry (see [Idmaps for user namespace](userns-idmap.md)). - Place recursive POSIX ACLs on your home directory. Privileged containers do not have this issue because all UID/GID in the container are the same as outside. But that's also the cause of most of the security issues with such privileged containers. ## How can I run Docker inside an Incus container? To run Docker inside an Incus container, set the {config:option}`instance-security:security.nesting` property of the container to `true`: incus config set security.nesting=true Note that Incus containers cannot load kernel modules, so depending on your Docker configuration, you might need to have extra kernel modules loaded by the host. You can do so by setting a comma-separated list of kernel modules that your container needs: incus config set linux.kernel_modules= In addition, creating a `/.dockerenv` file in your container can help Docker ignore some errors it's getting due to running in a nested environment. ## Where does the Incus client (`incus`) store its configuration? The [`incus`](incus.md) command stores its configuration under `~/.config/incus`. Various configuration files are stored in that directory, for example: - `client.crt`: client certificate (generated on demand) - `client.key`: client key (generated on demand) - `config.yml`: configuration file (info about `remotes`, `aliases`, etc.) - `clientcerts/`: directory with per-remote client certificates - `servercerts/`: directory with server certificates belonging to `remotes` ## Why can I not ping my Incus instance from another host? Many switches do not allow MAC address changes, and will either drop traffic with an incorrect MAC or disable the port totally. If you can ping an Incus instance from the host, but are not able to ping it from a different host, this could be the cause. The way to diagnose this problem is to run a `tcpdump` on the uplink and you will see either ``ARP Who has `xx.xx.xx.xx` tell `yy.yy.yy.yy` ``, with you sending responses but them not getting acknowledged, or ICMP packets going in and out successfully, but never being received by the other host. (faq-monitor)= ## How can I monitor what Incus is doing? To see detailed information about what Incus is doing and what processes it is running, use the [`incus monitor`](incus_monitor.md) command. For example, to show a human-readable output of all types of messages, enter the following command: incus monitor --pretty See [`incus monitor --help`](incus_monitor.md) for all options, and {doc}`debugging` for more information. ## Why does Incus stall when creating an instance? Check if your storage pool is out of space (by running [`incus storage info `](incus_storage_info.md)). In that case, Incus cannot finish unpacking the image, and the instance that you're trying to create shows up as stopped. To get more insight into what is happening, run [`incus monitor`](incus_monitor.md) (see {ref}`faq-monitor`), and check `sudo dmesg` for any I/O errors. ## Why does starting containers suddenly fail? If starting containers suddenly fails with a cgroup-related error message (`Failed to mount "/sys/fs/cgroup"`), this might be due to running a VPN client on the host. This is a known issue for both [Mullvad VPN](https://github.com/mullvad/mullvadvpn-app/issues/3651) and [Private Internet Access VPN](https://github.com/pia-foss/desktop/issues/50), but might occur for other VPN clients as well. The problem is that the VPN client mounts the `net_cls` cgroup1 over cgroup2 (which Incus uses). The easiest fix for this problem is to stop the VPN client and unmount the `net_cls` cgroup1 with the following command: umount /sys/fs/cgroup/net_cls If you need to keep the VPN client running, mount the `net_cls` cgroup1 in another location and reconfigure your VPN client accordingly. See [this Discourse post](https://discuss.linuxcontainers.org/t/help-help-help-cgroup2-related-issue-on-ubuntu-jammy-with-mullvad-and-privateinternetaccess-vpn/14705/18) for instructions for Mullvad VPN. ## What is this `incusbr0-mtu` device? When setting the `bridge.mtu` option on an Incus managed bridge network, Incus will create a dummy network interface named `BRIDGE-mtu`. That interface will never be used to carry traffic but it has the requested MTU set to it and is bridged into the network bridge. This has the effect of forcing the bridge to adopt that MTU and avoids issues where the bridge's configured MTU would change as interfaces get added to it. --- --- METRICS --- - Files Extracted: 32 - Estimated Token Budget: ~126466 tokens - Canonical Reference: https://codewiki.google/github.com/lxc/incus