## File: README.md # LXD LXD is a modern, secure and powerful system container and virtual machine manager. It provides a unified experience for running and managing full Linux systems inside containers or virtual machines. LXD supports images for a large number of Linux distributions (official Ubuntu images and images provided by the community) and is built around a very powerful, yet pretty simple, REST API. LXD scales from one instance on a single machine to a cluster in a full data center rack, making it suitable for running workloads both for development and in production. LXD allows you to easily set up a system that feels like a small private cloud. You can run any type of workload in an efficient way while keeping your resources optimized. You should consider using LXD if you want to containerize different environments or run virtual machines, or in general run and manage your infrastructure in a cost-effective way. ## Get started See [Getting started](https://canonical.com/lxd/docs/latest/getting_started/) in the LXD documentation for installation instructions and first steps. - Release announcements: [`https://discourse.ubuntu.com/tags/c/project/lxd/news/143/release`](https://discourse.ubuntu.com/tags/c/project/lxd/news/143/release) - Release tarballs: [`https://github.com/canonical/lxd/releases/`](https://github.com/canonical/lxd/releases/) - Documentation: [`https://canonical.com/lxd/docs/latest/`](https://canonical.com/lxd/docs/latest/) ## Status Type | Service | Status --- | --- | --- Tests | GitHub | [](https://github.com/canonical/lxd/actions?query=event%3Apush+branch%3Amain) Go documentation | Godoc | [](https://godoc.org/github.com/canonical/lxd/client) Static analysis | GoReport | [](https://goreportcard.com/report/github.com/canonical/lxd) ## Installing LXD from packages The LXD daemon only works on Linux but the client tool (`lxc`) is available on most platforms. OS | Format | Command --- | --- | --- Linux | [Snap](https://snapcraft.io/lxd) | `snap install lxd` Windows | [Chocolatey](https://chocolatey.org/packages/lxc) | `choco install lxc` macOS | [Homebrew](https://formulae.brew.sh/formula/lxc) | `brew install lxc` The LXD snap packaging files are available in the [snap](snap) and [snapcraft](snapcraft) directories. For more instructions on installing LXD for a wide variety of Linux distributions and operating systems, and to install LXD from source, see [How to install LXD](https://canonical.com/lxd/docs/latest/installing/) in the documentation. ## Client SDK packages The LXD project provides SDK client packages for interacting with LXD servers from your own software. These SDKs are licensed as Apache-2.0. Language | URL --- | --- Go | https://pkg.go.dev/github.com/canonical/lxd/client Python | https://github.com/canonical/pylxd For more information on using the LXD API, see [REST API](https://canonical.com/lxd/docs/latest/restapi_landing/) in the documentation. ## Tools for managing LXD If you are looking for tools (beyond `lxc` CLI) to manage LXD at scale (from single server to wide clusters), the following projects can be useful: Tool | Link --- | --- Ansible - connection plugin | https://galaxy.ansible.com/ui/repo/published/community/general/content/connection/lxd/ Ansible - inventory plugin | https://galaxy.ansible.com/ui/repo/published/community/general/content/inventory/lxd/ Bolt - LXD transport | https://www.puppet.com/docs/bolt/latest/bolt_transports_reference.html#lxd MicroCloud | https://canonical.com/microcloud Packer - LXD builder | https://developer.hashicorp.com/packer/integrations/hashicorp/lxd/latest/components/builder/lxd Terraform provider | https://registry.terraform.io/providers/terraform-lxd/lxd ## Security Consider the following aspects to ensure that your LXD installation is secure: - Keep your operating system up-to-date and install all available security patches. - Use only supported LXD versions (LTS releases or the latest feature release). - Restrict access to the LXD daemon and the remote API. - Configure your network interfaces to be secure. - Do not use privileged containers unless required. If you use privileged containers, put appropriate security measures in place. See [Container security](https://canonical.com/lxd/docs/latest/explanation/security/#container-security) for more information. See [Security](https://canonical.com/lxd/docs/latest/explanation/security/) for detailed information. **IMPORTANT:** Local access to LXD through the Unix socket always grants full access to LXD. This includes the ability to attach file system paths or devices to any instance as well as tweak the security features on any instance. Therefore, you should only give such access to users who you'd trust with root access to your system. ## Community support You can seek support from the LXD developers as well as the wider community through the following channels. ### Forum Ask questions or engage in discussions: [`https://discourse.ubuntu.com/c/project/lxd/`](https://discourse.ubuntu.com/c/project/lxd/126) ### Documentation Access the official documentation: [`https://canonical.com/lxd/docs/latest/`](https://canonical.com/lxd/docs/latest/) ### Bug reports and feature requests To file a new bug or feature request, [submit an issue on GitHub](https://github.com/canonical/lxd/issues/new). ### Other community resources You can find additional resources on the [LXD website](https://canonical.com/lxd), on [YouTube](https://www.youtube.com/channel/UCuP6xPt0WTeZu32CkQPpbvA), and the [community-created tutorials](https://discourse.ubuntu.com/c/project/lxd/tutorials/146). ## Commercial support LTS releases of LXD receive standard support for five years, which means they receive continuous updates. Commercial support for LXD is provided as part of [Ubuntu Pro](https://ubuntu.com/pro) (both Infra-only and full Ubuntu Pro), including for [attached LXD instances running Ubuntu](https://canonical.com/lxd/docs/latest/howto/instances_ubuntu_pro_attach/). See the [full service description](https://ubuntu.com/legal/ubuntu-pro-description) for details. Managed solutions and firefighting support are also available for LXD deployments. See: [Managed services](https://ubuntu.com/managed). ## Contributing Fixes and new features are greatly appreciated. Make sure to read our [contributing guidelines](CONTRIBUTING.md) first! --- ## File: doc/explanation/authorization.md --- discourse: "[Identity and Access Management for LXD](41516)" --- (authorization)= # Remote API authorization When LXD is {ref}`exposed over the network ` it is possible to restrict API access via two mechanisms: - {ref}`restricted-tls-certs` - {ref}`fine-grained-authorization` (restricted-tls-certs)= ## Restricted TLS certificates It is possible to restrict a {ref}`TLS client ` to one or multiple projects. In this case, the client will also be prevented from performing global configuration changes or altering the configuration (limits, restrictions) of the projects it's allowed access to. To restrict access, use [`lxc config trust edit `](lxc_config_trust_edit.md). Set the `restricted` key to `true` and specify a list of projects to restrict the client to. If the list of projects is empty, the client will not be allowed access to any of them. (fine-grained-authorization)= ## Fine-grained authorization It is possible to restrict {ref}`OIDC clients ` and fine-grained TLS identities to granular actions on specific LXD resources. For example, one could restrict a user to be able to view, but not edit, a single instance. There are four key concepts that LXD uses to manage these fine-grained permissions: - **Entitlements**: An entitlement encapsulates an action that can be taken against a LXD API resource type. Some entitlements might apply to many resource types, whereas other entitlements can only apply to a single resource type. For example, the entitlement `can_view` is available for all resource types, but the entitlement `can_exec` is only available for LXD resources of type `instance`. - **Permissions**: A permission is the application of an entitlement to a particular LXD resource. For example, given the entitlement `can_exec` that is only defined for instances, a permission is the combination of `can_exec` and a single instance, as uniquely defined by its API URL (for example, `/1.0/instances/c1?project=foo`). - **Identities (users)**: An identity is any authenticated party that makes requests to LXD, including TLS clients. When an OIDC client adds a LXD server as a remote, the OIDC client is saved in LXD as an identity. Permissions cannot be assigned to identities directly. - **Groups**: A group is a collection of one or more identities. Identities can belong to one or more groups. Permissions can be assigned to groups. TLS clients cannot currently be assigned to groups. (permissions)= ### Explore permissions To discover available permissions that can be assigned to a group, or view permissions that are currently assigned, run the following command: lxc auth permission list --max-entitlements 0 The entity type column displays the LXD API resource type, this value is required when adding a permission to a group. The URL column displays the URL of the LXD API resource. The entitlements column displays all available entitlements for that entity type. If any groups are already assigned permissions on the API resource at the displayed URL, they are listed alongside the entitlements that they have been granted. Some useful permissions at a glance: - The `admin` entitlement on entity type `server` gives full access to LXD. This is equivalent to an unrestricted TLS client or Unix socket access. - The `project_manager` entitlement on entity type `server` grants access to create, edit, and delete projects, and all resources belonging to those projects. However, this permission does not allow access to server configuration, storage pool configuration, or certificate/identity management. - The `operator` entitlement on entity type `project` grants access to create, edit, and delete all resources belonging to the project against which the permission is granted. Members of a group with this permission will not be able to edit the project configuration itself. This is equivalent to a restricted TLS client with access to the same project. - The `user` entitlement on entity type `instance` grants access to view an instance, pull/push files, get a console, and begin a terminal session. Members of a group with this entitlement cannot edit the instance configuration. For a full list, see {ref}`permissions-reference`. ```{note} Due to a limitation in the LXD client, if `can_exec` is granted to a group for a particular instance, members of the group will not be able to start a terminal session unless `can_view_events` is additionally granted for the parent project of the instance. We are working to resolve this. ``` (identities)= ### Explore identities To discover available identities that can be assigned to a group, or view identities that are currently assigned, run the following command: lxc auth identity list The authentication method column displays the method by which the client authenticates with LXD. The type column displays the type of identity. Identity types are a superset of TLS certificate types and additionally include OIDC clients. The name column displays the name of the identity. For TLS clients, this will be the name of the certificate. For OIDC clients this will be the name of the client as given by the {abbr}`IdP (identity provider)` (requested via the [profile scope](https://openid.net/specs/openid-connect-basic-1_0.html#Scopes)). The identifier column displays a unique identifier for the identity within that authentication method. For TLS clients, this will be the certificate fingerprint. For OIDC clients, this will be the email address of the client. The groups column displays any groups that are currently assigned to the identity. Groups cannot currently be assigned to TLS clients. ```{note} OIDC clients will only be displayed in the list of identities once they have authenticated with LXD. ``` (manage-permissions)= ### Manage permissions In LXD, identities cannot be granted permissions directly. Instead, identities are added to groups, and groups are granted permissions. To create a group, run: lxc auth group create To add an identity to a group, run: lxc auth identity group add / For example, for OIDC clients: lxc auth identity group add oidc/ The identity is now a member of the group. To add permissions to the group, run: lxc auth group permission add [] [=...] Here are some examples: - `lxc auth group permission add administrator server admin` grants members of `administrator` the `admin` entitlement on `server`. - `lxc auth group permission add junior-dev project sandbox operator` grants members of `junior-dev` the `operator` entitlement on project `sandbox`. - `lxc auth group permission add my-group instance c1 user project=default` grants members of `my-group` the `user` entitlement on instance `c1` in project `default`. Some entity types require more than one supplementary argument to uniquely specify the entity. For example, entities of type `storage_volume` and `storage_bucket` require an additional `pool=` argument. (identity-provider-groups)= ### Use groups defined by the identity provider It is common practice to manage users, roles, and groups centrally via an identity provider (IdP). In LXD, identity provider groups allow groups that are defined by the IdP to be mapped to LXD groups. When an OIDC client makes a request to LXD, any groups that can be extracted from the client's identity token are mapped to LXD groups, giving the client the same effective permissions. To configure IdP group mappings in LXD, first configure your IdP to add groups to identity and access tokens as a custom claim. This configuration depends on your IdP. In [{spellexception}`Auth0`](https://auth0.com/), for example, you can enable [{abbr}`RBAC (role-based access control)`](https://auth0.com/docs/manage-users/access-control/rbac) which will add a "permissions" claim to tokens. Then, configure {ref}`automatic mapping to LXD authorization groups `. In Keycloak, you can define a [mapper](https://forum.keycloak.org/t/anyway-to-include-user-groups-into-my-jwt-token/8715) to set Keycloak groups in the token. In [Pocket ID](https://pocket-id.org/docs), you can set up {ref}`custom claims ` in your admin dashboard. Then configure LXD to extract this claim. To do so, set the value of the {config:option}`server-oidc:oidc.groups.claim` configuration key to the value of the field name of the custom claim: lxc config set oidc.groups.claim= LXD will then expect the identity and access tokens to contain a claim with this name. The value of the claim must be a JSON array containing a string value for each IdP group name. If the group names are extracted successfully, LXD will be aware of the IdP groups for the duration of the request. Next, configure a mapping between an IdP group and a LXD group as follows: lxc auth identity-provider-group create lxc auth identity-provider-group group add IdP groups can be mapped to multiple LXD groups, and multiple IdP groups can be mapped to the same LXD group. ```{important} LXD does not store the identity provider groups that are extracted from identity or access tokens. This can obfuscate the true permissions of an identity. For example, if an identity belongs to LXD group "foo", an administrator can view the permissions of group "foo" to determine the level of access of the identity. However, if identity provider group mappings are configured, direct group membership alone does not determine their level of access. The command `lxc auth identity info` can be run by any identity to view a full list of their own effective groups and permissions as granted directly or indirectly via IdP groups. ``` --- ## File: doc/explanation/bpf.md (bpf-delegation-token)= # Privilege delegation using BPF Token ## Overview The {config:option}`instance-security:security.delegate_bpf` option enables the {abbr}`BPF (Berkeley Packet Filter)` functionality delegation mechanism, using a [BPF Token](https://docs.ebpf.io/linux/concepts/token). When enabled, LXD mounts a BPF File System (BPFFS) inside a container instance. This file system is configured with the `security.delegate_bpf.*` settings. For example: ``` none on /sys/fs/bpf type bpf (rw,relatime,uid=1000000,gid=1000000, delegate_cmds=map_create:prog_load, delegate_maps=ringbuf, delegate_progs=socket_filter, delegate_attachs=cgroup_inet_ingress) ``` Then, applications inside the container can create a BPF Token file descriptor using that BPFFS mount and the `bpf(BPF_TOKEN_CREATE)` syscall. Later, this File Descriptor can be passed to `bpf(BPF_PROG_LOAD)`, `bpf(BPF_MAP_CREATE)`, or another `bpf()`-command syscall, and the kernel will perform a permission check against the token instead of the current user credentials. To be more precise, current user caps are also checked for `CAP_BPF` but in a current user namespace when `bpf(BPF_TOKEN_CREATE)` is called. It follows that user space applications inside the container must be aware of the BPF Token kernel feature (which appeared in Linux kernel v6.9) and make use of it. In contrast to `security.syscalls.intercept.*` features, this one is not fully transparent and might require updates or modifications to the software inside the container. Fortunately, [the libbpf library](https://docs.kernel.org/bpf/libbpf/libbpf_overview.html) supports BPF tokens. Thus if an application uses libbpf, then to make use of this feature, you might only need to update libbpf. ```{note} Configure the following instance options for the container, depending on its BPF workload: - {config:option}`instance-security:security.delegate_bpf.cmd_types` - {config:option}`instance-security:security.delegate_bpf.map_types` - {config:option}`instance-security:security.delegate_bpf.prog_types` - {config:option}`instance-security:security.delegate_bpf.attach_types` ``` See the [BPF Token documentation page](https://docs.ebpf.io/linux/concepts/token/) on `docs.ebpf.io` for details. ## Example (socket filter) Let's consider an example with a socket filter program from [libbpf-bootstrap](https://github.com/libbpf/libbpf-bootstrap). The following creates an unprivileged container instance and sets all the necessary configuration options to enable BPF delegation: ```bash lxc launch ubuntu:noble bpf-experiments lxc config set bpf-experiments limits.kernel.memlock=unlimited lxc config set bpf-experiments security.delegate_bpf=true lxc config set bpf-experiments security.delegate_bpf.prog_types=socket_filter lxc config set bpf-experiments security.delegate_bpf.attach_types=cgroup_inet_ingress lxc config set bpf-experiments security.delegate_bpf.cmd_types=prog_load:map_create lxc config set bpf-experiments security.delegate_bpf.map_types=ringbuf ``` The following set of commands clones and builds the libbpf-bootstrap.git repository within the example `bpf-experiments` container: ```bash lxc shell bpf-experiments apt install clang build-essential git clone https://github.com/libbpf/libbpf-bootstrap.git git submodule update --init --recursive cd libbpf-bootstrap/examples/c make ``` This experiment completes by running commands from two different shells into the `bpf-experiments` container. From one terminal: ```{terminal} lxc shell bpf-experiments ./sockfilter ``` From another terminal: ```{terminal} lxc shell bpf-experiments ping -c 4 localhost ``` Sample output: ``` ibbpf: loading object 'sockfilter_bpf' from buffer libbpf: elf: section(2) .symtab, size 192, link 1, flags 0, type=2 libbpf: elf: section(3) socket, size 576, link 0, flags 6, type=1 libbpf: sec 'socket': found program 'socket_handler' at insn offset 0 (0 bytes), code size 72 insns (576 bytes) ... libbpf: Kernel doesn't support BTF, skipping uploading it. libbpf: map 'rb': created successfully, fd=3 interface: lo protocol: ICMP 127.0.0.1:2048(src) -> 127.0.0.1:32429(dst) interface: lo protocol: ICMP 127.0.0.1:0(src) -> 127.0.0.1:34477(dst) interface: lo protocol: ICMP 127.0.0.1:2048(src) -> 127.0.0.1:46163(dst) interface: lo protocol: ICMP 127.0.0.1:0(src) -> 127.0.0.1:48211(dst) ``` We can see from this sample output that the ICMP packets were captured by the {abbr}`eBPF (extended Berkeley Capture Filter)` program and logged. ## Finding the right configuration To figure out the right values for the `security.delegate_bpf.cmd_types`, `security.delegate_bpf.map_types`, `security.delegate_bpf.prog_types`, `security.delegate_bpf.attach_types` options, you must know how your application inside the container uses eBPF, such as its program types and map types. You can consult the application's source code, or use the [`strace`](https://github.com/strace/strace) tool to trace `bpf` syscall and see how it is being used. Example using `strace`: ```{terminal} strace -e bpf ./sockfilter ``` Sample output: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` This log shows that `sockfilter` is using: 1. Program types: `BPF_PROG_TYPE_SOCKET_FILTER` 1. Map types: `BPF_MAP_TYPE_RINGBUF` 1. Attachment types: `BPF_CGROUP_INET_INGRESS` 1. BPF commands: `BPF_BTF_LOAD`, `BPF_PROG_LOAD`, `BPF_MAP_CREATE` --- ## File: doc/explanation/clusters.md --- discourse: lxc:[Scriptlet based instance placement scheduler](15728) --- (exp-clusters)= # Clusters ```{youtube} https://www.youtube.com/watch?v=nrOR6yaO_MY :title: Deep dive into LXD clustering ``` To spread the total workload over several servers, LXD can be run in clustering mode. In this scenario, any number of LXD servers share the same distributed database that holds the configuration for the cluster members and their instances. The LXD cluster can be managed uniformly using the [`lxc`](lxc.md) client or the REST API. This feature was introduced as part of the [`clustering`](../api-extensions.md#clustering) API extension and is available since LXD 3.0. ```{tip} If you want to quickly set up a basic LXD cluster, check out [MicroCloud](https://canonical.com/microcloud). ``` (clustering-members)= ## Cluster members A LXD cluster consists of one bootstrap server and at least two further cluster members. It stores its state in a [distributed database](../database.md), which is a [Dqlite](https://canonical.com/dqlite) database replicated using the Raft algorithm. While you could create a cluster with only two members, it is strongly recommended that the number of cluster members be at least three. With this setup, the cluster can survive the loss of at least one member and still be able to establish quorum for its distributed state. When you create the cluster, the Dqlite database runs on only the bootstrap server until a third member joins the cluster. Then both the second and the third server receive a replica of the database. See {ref}`cluster-form` for more information. (clustering-member-roles)= ### Member roles In a cluster with three members, all members replicate the distributed database that stores the state of the cluster. If the cluster has more members, only some of them replicate the database. The remaining members have access to the database, but don't replicate it. At each time, there is an elected cluster leader that monitors the health of the other members. Each member that replicates the database has either the role of a *voter* or of a *stand-by*. If the cluster leader goes offline, one of the voters is elected as the new leader. If a voter member goes offline, a stand-by member is automatically promoted to voter. The database (and hence the cluster) remains available as long as a majority of voters is online. The following roles can be assigned to LXD cluster members. Automatic roles are assigned by LXD itself and cannot be modified by the user. | Role | Automatic | Description | | :--- | :-------- | :---------- | | `database-voter` | yes | Voting member of the distributed database | | `database-leader` | yes | Current leader of the distributed database | | `database-standby` | yes | Stand-by (non-voting) member of the distributed database | | `control-plane` | no | Eligible to participate in Raft as voter, standby, or leader; when control plane mode is active, members without this role are assigned as spares and excluded from automatic promotion | | `ovn-chassis` | no | Uplink gateway candidate for OVN networks | The default number of voter members ({config:option}`server-cluster:cluster.max_voters`) is three. The default number of stand-by members ({config:option}`server-cluster:cluster.max_standby`) is two. With this configuration, your cluster will remain operational as long as you switch off at most one voting member at a time. (clustering-control-plane)= #### Control plane mode The `control-plane` role is optional and is not assigned by default. It designates which members are eligible for database roles (voter, standby, leader), enabling safe auto-scaling with fixed database members and dynamic worker members. Control plane mode activates when at least 3 members have the role assigned. Once active, only members with the `control-plane` role can participate in Raft and be assigned as voters, standbys, or the leader. Members without the `control-plane` role are automatically assigned the `RAFT_SPARE` role and are excluded from automatic promotion to database roles. Spare members can still run instances and act as "worker" members for hosting workloads. You can assign the `control-plane` role to more members than {config:option}`server-cluster:cluster.max_voters` to create a pool of eligible candidates. For example, if you assign `control-plane` to 5 members when `cluster.max_voters` is 3, all 5 members are eligible for database roles, but only 3 will be promoted to voters based on the configuration. If no cluster members have the `control-plane` role assigned (the default), or if fewer than 3 members have the role, all members are eligible for automatic promotion to database roles. When control plane mode is active, members with the `control-plane` role also act as event hubs for internal LXD events. If control plane mode is inactive, the cluster uses full-mesh event connectivity. See {ref}`cluster-manage-control-plane` for instructions on using the `control-plane` role. (clustering-offline-members)= #### Offline members and fault tolerance If a cluster member is down for more than the configured offline threshold, its status is marked as offline. In this case, no operations are possible on this member, and neither are operations that require a state change across all members. As soon as the offline member comes back online, operations are available again. If the member that goes offline is the leader itself, the other members will elect a new leader. If you can't or don't want to bring the server back online, you can [delete it from the cluster](cluster-manage-delete-members). You can tweak the amount of seconds after which a non-responding member is considered offline by setting the {config:option}`server-cluster:cluster.offline_threshold` configuration. The default value is 20 seconds. The minimum value is 10 seconds. To automatically {ref}`evacuate ` instances from an offline member, set the {config:option}`server-cluster:cluster.healing_threshold` configuration to a non-zero value. See {ref}`cluster-recover` for more information. (clustering-failure-domains)= #### Failure domains You can use failure domains to indicate which cluster members should be given preference when assigning roles to a cluster member that has gone offline. For example, if a cluster member that currently has the `database-voter` role is shut down and control plane mode is active, LXD tries to promote another `control-plane` cluster member in the same failure domain to voter, if one is available. If no members have the `control-plane` role assigned (the default), any suitable member in the same failure domain can be promoted instead. See {ref}`cluster-manage-failure-domains` for more information. (clustering-member-config)= ### Member configuration LXD cluster members are generally assumed to be identical systems. This means that all LXD servers joining a cluster must have an identical configuration to the bootstrap server, in terms of storage pools and networks. To accommodate things like slightly different disk ordering or network interface naming, there is an exception for some configuration options related to storage and networks, which are member-specific. When such settings are present in a cluster, any server that is being added must provide a value for them. Most often, this is done through the interactive `lxd init` command, which asks the user for the value for a number of configuration keys related to storage or networks. Those settings typically include: - The source device and size (quota) for a storage pool - The name for a ZFS zpool, LVM thin pool or LVM volume group - External interfaces and BGP next-hop for a bridged network - The name of the parent network device for managed `physical` or `macvlan` networks See {ref}`howto-cluster-storage` and {ref}`cluster-config-networks` for more information. If you want to look up the questions ahead of time (which can be useful for scripting), query the `/1.0/cluster` API endpoint. This can be done through `lxc query /1.0/cluster` or through other API clients. ## Images By default, LXD replicates images on as many cluster members as there are database members. This typically means up to three copies within the cluster. You can increase that number to improve fault tolerance and the likelihood of the image being locally available. To do so, set the {config:option}`server-cluster:cluster.images_minimal_replica` configuration. The special value of `-1` can be used to have the image copied to all cluster members. (cluster-groups)= ## Cluster groups In a LXD cluster, you can add members to cluster groups. You can use these cluster groups to launch instances on a cluster member that belongs to a subset of all available members. For example, you could create a cluster group for all members that have a GPU and then launch all instances that require a GPU on this cluster group. By default, all cluster members belong to the `default` group. See {ref}`howto-cluster-groups` and {ref}`cluster-target-instance` for more information. (exp-cluster-links)= ## Cluster links Cluster links enable communication between separate LXD clusters by pinning the remote cluster's TLS certificate and optionally establishing mutual trust. Cluster links are the foundation for {ref}`replicators `, which use bidirectional links to sync instances across clusters for active-passive disaster recovery. Unidirectional links are suited to scenarios where one cluster needs to access another cluster without granting reciprocal access. ### Link types There are two link types, each suited to different trust and access requirements: `bidirectional` : Either cluster can initiate requests to the other cluster. The clusters authenticate each other using mutual TLS, and both clusters create an identity for the other side. This is the default type. `unidirectional` : Requests can only be sent in one direction: from Cluster A to Cluster B. Cluster A pins B's certificate and uses a token to activate a pending identity that B created for A. B stores only a TLS identity for A (no cluster link record) and can authenticate incoming requests from A, but holds no address for A and cannot initiate requests to it. ### Connection process All link types rely on TLS certificate pinning: Cluster A fetches and pins Cluster B's certificate before making any connection. The link type determines the level (and direction) of trust that is established between Cluster A and Cluster B. #### Bidirectional connection process 1. **Cluster A** creates a pending cluster link and generates a trust token. 1. **Cluster B** uses this token to establish the connection and send its certificate back. 1. Both clusters validate certificates and activate their cluster links. 1. The trust relationship is established and both clusters can communicate. #### Unidirectional connection process 1. **Cluster B** issues a pending identity token using [`lxc auth identity create`](lxc_auth_identity_create.md) `cluster-link/`. 1. **Cluster A** consumes the token with [`lxc cluster link create`](lxc_cluster_link_create.md) ` --token --unidirectional`, pins Cluster B's certificate, and calls back to B to activate the pending identity. 1. Cluster A has an active cluster link to Cluster B with no associated identity. Cluster B has an active TLS identity for Cluster A but no cluster link record. For more information, see: {ref}`howto-cluster-links-create`. (exp-clusters-links-identity)= ### Identity management The identities created depend on the link type: - **Bidirectional**: LXD creates a `Cluster link certificate` identity on each side. The identity can be in one of two states: - **Pending**: A trust token has been generated but the link has not been activated yet. - **Active**: Both clusters have exchanged certificates and the link is operational. - **Unidirectional**: Cluster B creates a TLS identity for Cluster A (no cluster link record); Cluster A stores Cluster B's certificate directly without an associated identity. Identities are managed using {ref}`fine-grained authorization `. ### Security considerations - **Certificate validation**: All connections verify certificate fingerprints. - **Fine-grained permissions**: Linked clusters can be granted specific entitlements (for example, only backup operations). - **Identity isolation**: Each cluster link gets its own identity that can be managed independently. - **Group membership**: Cluster link identities can be assigned to authentication groups for bulk permission management. Together, these controls limit the potential impact of a compromised link by enforcing certificate-based trust and least-privilege access. They also make it possible to revoke a single link's access without impacting other cluster-to-cluster trust relationships. ### Removal Deleting a cluster link revokes the security trust it established. The scope depends on the link type: - **Bidirectional**: Run [`lxc cluster link delete`](lxc_cluster_link_delete.md) on both clusters to fully remove the trust relationship. - **Unidirectional**: Deleting on Cluster A removes only A's link. Cluster B's identity remains until B explicitly revokes it with [`lxc auth identity delete`](lxc_auth_identity_delete.md) `cluster-link/`. ### Member status A cluster link member can have one of the following statuses. Run [`lxc cluster link info`](lxc_cluster_link_info.md) to check member status. (Refer to {ref}`howto-cluster-links-view` for additional details.) - `ACTIVE`: Reachable and authenticated. The link is usable for requests according to the {ref}`entitlements ` you granted. - `UNAUTHENTICATED`: Reachable but not authenticated. The remote cluster cannot use the link yet; resolve the trust exchange before relying on it. - `UNREACHABLE`: Not reachable. Requests that depend on the link will fail until connectivity is restored or the remote cluster is online. Member status reflects connectivity, while [`lxc cluster link list`](lxc_cluster_link_list.md) shows the link identity status and link type, which determine the permissions available to the linked cluster. (clustering-instance-placement)= ## Automatic placement of instances In a cluster setup, each instance lives on one of the cluster members. When you launch an instance, you can target it to a specific cluster member, to a cluster group or have LXD automatically assign it to a cluster member. By default, the automatic assignment picks the cluster member that has the lowest number of instances. If several members have the same amount of instances, one of the members is chosen at random. However, you can control this behavior with the {config:option}`cluster-cluster:scheduler.instance` configuration option: - If `scheduler.instance` is set to `all` for a cluster member, this cluster member is selected for an instance if: - The instance is created without `--target` and the cluster member has the lowest number of instances. - The instance is targeted to live on this cluster member. - The instance is targeted to live on a member of a cluster group that the cluster member is a part of, and the cluster member has the lowest number of instances compared to the other members of the cluster group. - If `scheduler.instance` is set to `manual` for a cluster member, this cluster member is selected for an instance if: - The instance is targeted to live on this cluster member. - If `scheduler.instance` is set to `group` for a cluster member, this cluster member is selected for an instance if: - The instance is targeted to live on this cluster member. - The instance is targeted to live on a member of a cluster group that the cluster member is a part of, and the cluster member has the lowest number of instances compared to the other members of the cluster group. (exp-clusters-placement)= ### Placement groups Placement groups provide declarative control over how instances are distributed across cluster members. They define both a **policy** (how instances should be distributed) and a **rigor** (how strictly the policy is enforced). Placement groups are project-scoped resources, which means different projects can have placement groups with the same name without conflict. See {ref}`cluster-placement-groups` for usage instructions and {ref}`ref-placement-groups` for reference documentation. (clusters-high-availability)= ## High availability Clusters provide two types of high availability (HA): - Control plane HA (ensuring that clients can always access the cluster) - Data plane HA (ensuring that workloads continue to run) (clusters-high-availability-control)= ### High availability of the control plane (client access) Each cluster member can {ref}`expose an API endpoint ` through its {config:option}`server-core:core.https_address`. Through this access point, a remote client can communicate with any cluster member in multiple ways: - Through the API (see {ref}`authentication` and {ref}`rest-api`) - Through the {ref}`LXD web UI client ` - By setting up {ref}`remote servers ` for CLI access Because the cluster database is distributed, access to any member gives you access to the entire control plane. If one server goes down, you can still manage the cluster through the other members. This provides the basis for control plane HA. The limitation is that on the client side, you must either manually switch to another member's access point if your chosen server is unavailable, or implement your own client-side logic to cycle through a list of access points. For a single, highly available access point to the control plane, you can add on a routing service that configures a virtual IP. See our how-to guide: {ref}`howto-cluster-vip`. (clusters-high-availability-data)= ### High availability of the data plane (workloads) LXD clusters enable HA of workloads (instances) in multiple ways: Cluster evacuation : Instances can be manually evacuated from one cluster member to another, providing planned high availability during maintenance. This includes live migration for virtual machines. See: {ref}`cluster-evacuate`. Cluster healing : If a cluster member fails and {config:option}`server-cluster:cluster.healing_threshold` is set, it automatically restarts instances on that member on a healthy member of the cluster. See: {ref}`cluster-healing`. Virtual networking : On clusters using {ref}`OVN networking `, logical switches/routers are distributed across the cluster. This means that instance NICs remain reachable even if the server hosting one OVN chassis goes offline. Storage redundancy : On clusters using Ceph for storage, if a disk or cluster member fails, the data is still available elsewhere in the Ceph cluster. Shared storage : Volumes using the {ref}`Ceph RBD ` and {ref}`CephFS ` storage drivers are accessible from all cluster members. If the member hosting an instance fails, its volumes can be reattached to another member. ## Related topics {{clustering_how}} {{clustering_ref}} --- ## File: doc/explanation/index.md --- myst: html_meta: --- (explanation)= # Explanation The explanatory guides in this section discuss the concepts used in LXD and help you understand how things fit together. (explanation-concepts)= ## Important concepts LXD's core concepts include its relationship with LXC and the instance types it supports: system containers and virtual machines. ```{toctree} :titlesonly: /explanation/lxd_lxc /explanation/instances ``` (explanation-entities)= ## Entities in LXD LXD uses several distinct entity types, including images, storage pools, networks, and projects. To learn how to use them, refer to the {ref}`howtos`. ```{toctree} :titlesonly: /image-handling /explanation/storage /explanation/networks /database /explanation/lxc_show_info ``` (explanation-iam)= ## Access management LXD supports multiple methods for authenticating remote API clients and provides fine-grained authorization controls. Projects can also be used to scope and restrict access. ```{toctree} :titlesonly: /authentication /explanation/authorization /explanation/projects ``` (explanation-production)= ## Production setup For scalable, reliable, and secure LXD deployments, these guides help you understand the key concepts around clustering, performance tuning, and security. ```{toctree} :titlesonly: /explanation/clusters /explanation/replicators /explanation/performance_tuning /explanation/security /explanation/bpf ``` (explanation-csi)= ## The LXD CSI driver The LXD CSI driver is an open source implementation of the Container Storage Interface (CSI) that integrates LXD storage backends with Kubernetes. ```{toctree} :titlesonly: /explanation/csi ``` --- ## File: doc/explanation/instance_config.md (instance-config)= # Instance configuration The instance configuration consists of different categories: Instance properties : Instance properties are specified when the instance is created. They include, for example, the instance name and architecture. Some of the properties are read-only and cannot be changed after creation, while others can be updated by {ref}`setting their property value ` or {ref}`editing the full instance configuration `. In the YAML configuration, properties are on the top level. See {ref}`instance-properties` for a reference of available instance properties. Instance options : Instance options are configuration options that are related directly to the instance. They include, for example, startup options, security settings, hardware limits, kernel modules, snapshots and user keys. These options can be specified as key/value pairs during instance creation (through the `--config key=value` flag). After creation, they can be configured with the [`lxc config set`](lxc_config_set.md) and [`lxc config unset`](lxc_config_unset.md) commands. In the YAML configuration, options are located under the `config` entry. See {ref}`instance-options` for a reference of available instance options, and {ref}`instances-configure-options` for instructions on how to configure the options. Instance devices : Instance devices are attached to an instance. They include, for example, network interfaces, mount points, USB and GPU devices. Devices are usually added after an instance is created with the [`lxc config device add`](lxc_config_device_add.md) command, but they can also be added to a profile or a YAML configuration file that is used to create an instance. Each type of device has its own specific set of options, referred to as *instance device options*. In the YAML configuration, devices are located under the `devices` entry. See {ref}`devices` for a reference of available devices and the corresponding instance device options, and {ref}`instances-configure-devices` for instructions on how to add and configure instance devices. ```{toctree} :maxdepth: 2 :hidden: ../reference/instance_properties.md ../reference/instance_options.md ../reference/devices.md ../reference/instance_units.md ``` ## Related topics {{instances_how}} {{instances_exp}} --- ## File: doc/explanation/instances.md --- discourse: lxc:[Overview - GUI inside Containers](8767),lxc:[Running virtual machines with LXD 4.0](7519),lxc:[Install any OS via ISO in a Virtual machine/VM](9281) relatedlinks: "[LXD virtual machines: an overview](https://ubuntu.com/blog/lxd-virtual-machines-an-overview)" --- (containers-and-vms)= # Containers and VMs LXD provides support for two different types of {ref}`instances `: *system containers* and *virtual machines*. When running a system container, LXD simulates a virtual version of a full operating system. To do this, it uses the functionality provided by the kernel running on the host system. When running a virtual machine, LXD uses the hardware of the host system, but the kernel is provided by the virtual machine. Therefore, virtual machines can be used to run, for example, a different operating system. ## Application containers vs. system containers Application containers (as provided by, for example, Docker) package a single process or application. System containers, on the other hand, simulate a full operating system and let you run multiple processes at the same time. Therefore, application containers are suitable to provide separate components, while system containers provide a full solution of libraries, applications, databases, and so on. In addition, you can use system containers to create different user spaces and isolate all processes belonging to each user space, which is not what application containers are intended for. ## Virtual machines vs. system containers Virtual machines emulate a physical machine, using the hardware of the host system from a full and completely isolated operating system. System containers, on the other hand, use the OS kernel of the host system instead of creating their own environment. If you run several system containers, they all share the same kernel, which makes them faster and more lightweight than virtual machines. With LXD, you can create both system containers and virtual machines. You should use a system container to leverage the smaller size and increased performance if all functionality you require is compatible with the kernel of your host operating system. If you need functionality that is not supported by the OS kernel of your host system or you want to run a completely different OS, use a virtual machine. (expl-instances)= ## Instance types in LXD LXD supports the following types of instances: Containers : Containers are the default type for instances. They are implemented through the use of `liblxc` (LXC). Virtual machines : {abbr}`Virtual machines (VMs)` are natively supported since version 4.0 of LXD. Thanks to a built-in agent, they can be used almost like containers, with a similar set of features. LXD uses `qemu` to provide the VM functionality. ```{note} In the {ref}`instance-options` documentation, some instance options display a `condition` field in their details, with the value of either `container` or `virtual machine`. This indicates the type of instance for which that option is available. If no `condition` field exists in an option's details, that option applies to both types. ``` ## Related topics {{instances_how}} {{instances_ref}} --- ## File: doc/explanation/lxc_show_info.md (lxc-show-info)= # `lxc` `show` and `info` For the entities managed by LXD, the `lxc` command provides a `list` sub-command, and might provide `show` and `info` sub-commands. The purpose of the `info` sub-command is to show current state information, and the purpose of the `show` sub-command is to show configuration information and how the entity is used by other entities. For example, the `lxc network info` command shows IP address and traffic statistics: Name: lxdbr0 MAC address: 00:16:3e:d3:ec:41 MTU: 1500 State: up Ips: inet 192.0.2.1 inet6 2001:db8:f4a1:53d2::1 inet6 fe80::216:3eff:fed3:ec41 Network usage: Bytes received: 127.66kB Bytes sent: 15.54kB Packets received: 1433 Packets sent: 175 The `lxc network show` command, on the other hand, shows how the network is configured, and which entities are using the network: config: ipv4.address: 192.0.2.1/24 ipv4.nat: "true" ipv6.address: 2001:db8:f4a1:53d2::1/64 ipv6.nat: "true" name: lxdbr0 type: bridge used_by: - /1.0/instances/ubuntu - /1.0/profiles/default managed: true status: Created locations: - none Refer to the manual pages for details of the commands for managing entities: - Instances: [`lxc list`](lxc_list.md), [`lxc info`](lxc_info.md) - Images: [`lxc image list`](lxc_image_list.md), [`lxc image info`](lxc_image_info.md), [`lxc image show`](lxc_image_show.md) - Networks: [`lxc network list`](lxc_network_list.md), [`lxc network info`](lxc_network_info.md), [`lxc network show`](lxc_network_show.md) - Profiles: [`lxc profile list`](lxc_profile_list.md), [`lxc profile show`](lxc_profile_show.md) - Projects: [`lxc project list`](lxc_project_list.md), [`lxc project info`](lxc_project_info.md), [`lxc project show`](lxc_project_show.md) - Storage: [`lxc storage list`](lxc_storage_list.md), [`lxc storage info`](lxc_storage_info.md), [`lxc storage show`](lxc_storage_show.md) - Cluster links: [`lxc cluster link list`](lxc_cluster_link_list.md), [`lxc cluster link info`](lxc_cluster_link_info.md), [`lxc cluster link show`](lxc_cluster_link_show.md) --- ## File: doc/explanation/lxd_lxc.md --- discourse: lxc:[Comparing LXD vs. LXC](24) --- (lxd-lxc)= # `lxd` and `lxc` LXD is frequently confused with LXC, and the fact that LXD provides both a `lxd` command and a `lxc` command doesn't make things easier. ## LXD vs. LXC LXD and LXC are two distinct implementations of Linux containers. [LXC](https://linuxcontainers.org/lxc/introduction/) is a low-level user space interface for the Linux kernel containment features. It consists of tools (`lxc-*` commands), templates, and library and language bindings. [LXD](https://canonical.com/lxd) is a more intuitive and user-friendly tool aimed at making it easy to work with Linux containers. It is an alternative to LXC's tools and distribution template system, with the added features that come from being controllable over the network. Under the hood, LXD uses LXC to create and manage the containers. LXD provides a superset of the features that LXC supports, and it is easier to use. Therefore, if you are unsure which of the tools to use, you should go for LXD. LXC should be seen as an alternative for experienced users that want to run Linux containers on distributions that don't support LXD. (lxd-daemon)= ## LXD daemon The central part of LXD is its daemon. It runs persistently in the background, manages the instances, and handles all requests. The daemon provides a REST API that you can access directly or through a client (for example, the default command-line client that comes with LXD). See {ref}`daemon-behavior` for more information about the LXD daemon. ## `lxd` vs. `lxc` To control LXD, you typically use two different commands: `lxd` and `lxc`. LXD daemon : The `lxd` command controls the LXD daemon. Since the daemon is typically started automatically, you hardly ever need to use the `lxd` command. An exception is the `lxd init` subcommand that you run to {ref}`initialize LXD `. There are also some subcommands for debugging and administrating the daemon, but they are intended for advanced users only. See `lxd --help` for an overview of all available subcommands. LXD client : The `lxc` command is a command-line client for LXD, which you can use to interact with the LXD daemon. You use the `lxc` command to manage your instances, the server settings, and overall the entities you create in LXD. See [`lxc --help`](lxc.md) for an overview of all available subcommands. The `lxc` tool is not the only client you can use to interact with the LXD daemon. You can also use the API, the UI, or a custom LXD client. --- ## File: doc/explanation/networks.md (networks)= # Networking setups There are different ways to connect your instances to the Internet. The easiest method is to have LXD create a network bridge during initialization and use this bridge for all instances, but LXD supports many different and advanced setups for networking. ## Network devices To grant direct network access to an instance, you must assign it at least one network device, also called {abbr}`NIC (Network Interface Controller)`. You can configure the network device in one of the following ways: - Use the default network bridge that you set up during the LXD initialization. Check the default profile to see the default configuration: lxc profile show default This method is used if you do not specify a network device for your instance. - Use an existing network interface by adding it as a network device to your instance. This network interface is outside of LXD control. Therefore, you must specify all information that LXD needs to use the network interface. Use a command similar to the following: lxc config device add nic nictype= ... See [Type: `nic`](devices-nic) for a list of available NIC types and their configuration properties. For example, you could add a pre-existing Linux bridge (`br0`) with the following command: lxc config device add eth0 nic nictype=bridged parent=br0 - {doc}`Create a managed network ` and add it as a network device to your instance. With this method, LXD has all required information about the configured network, and you can directly attach it to your instance as a device: lxc network attach See {ref}`network-attach` for more information. (managed-networks)= ## Managed networks Managed networks in LXD are created and configured with the `lxc network [create|edit|set]` command. Depending on the network type, LXD either fully controls the network or just manages an external network interface. Note that not all {ref}`NIC types ` are supported as network types. LXD can only set up some of the types as managed networks. ### Fully controlled networks % Include content from [../reference/networks.md](../reference/networks.md) ```{include} ../reference/networks.md :start-after: :end-before: ``` {ref}`network-bridge` : % Include content from [../reference/network_bridge.md](../reference/network_bridge.md) ```{include} ../reference/network_bridge.md :start-after: :end-before: ``` In LXD context, the `bridge` network type creates an L2 bridge that connects the instances that use it together into a single network L2 segment. This makes it possible to pass traffic between the instances. The bridge can also provide local DHCP and DNS. This is the default network type. {ref}`network-ovn` : % Include content from [../reference/network_ovn.md](../reference/network_ovn.md) ```{include} ../reference/network_ovn.md :start-after: :end-before: ``` In LXD context, the `ovn` network type creates a logical network. To set it up, you must install and configure the OVN tools. In addition, you must create an uplink network that provides the network connection for OVN. As the uplink network, you should use one of the external network types or a managed LXD bridge. ```{tip} Unlike the other network types, you can create and manage an OVN network inside a {ref}`project `. This means that you can create your own OVN network as a non-admin user, even in a restricted project. ``` ### External networks % Include content from [../reference/networks.md](../reference/network_external.md) ```{include} ../reference/networks.md :start-after: :end-before: ``` {ref}`network-macvlan` : % Include content from [../reference/network_macvlan.md](../reference/network_macvlan.md) ```{include} ../reference/network_macvlan.md :start-after: :end-before: ``` In LXD context, the `macvlan` network type provides a preset configuration to use when connecting instances to a parent macvlan interface. {ref}`network-sriov` : % Include content from [../reference/network_sriov.md](../reference/network_sriov.md) ```{include} ../reference/network_sriov.md :start-after: :end-before: ``` In LXD context, the `sriov` network type provides a preset configuration to use when connecting instances to a parent SR-IOV interface. {ref}`network-physical` : % Include content from [../reference/network_physical.md](../reference/network_physical.md) ```{include} ../reference/network_physical.md :start-after: :end-before: ``` It provides a preset configuration to use when connecting OVN networks to a parent interface. ## Recommendations In general, if you can use a managed network, you should do so because networks are easy to configure and you can reuse the same network for several instances without repeating the configuration. Which network type to choose depends on your specific use case. If you choose a fully controlled network, it provides more functionality than using a network device. As a general recommendation: - If you are running LXD on a single system or in a public cloud, use a {ref}`network-bridge`, possibly in connection with the [Ubuntu Fan](https://www.youtube.com/watch?v=5cwd0vZJ5bw). - If you are running LXD in your own private cloud, use an {ref}`network-ovn`. ```{note} OVN requires a shared L2 uplink network for proper operation. Therefore, using OVN is usually not possible if you run LXD in a public cloud. ``` - To connect an instance NIC to a managed network, use the `network` property rather than the `parent` property, if possible. This way, the NIC can inherit the settings from the network and you don't need to specify the `nictype`. ## Related topics {{networks_how}} {{networks_ref}}