# Technical Documentation: kata-containers/qemu > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/kata-containers/qemu) ### Overview (`target/i386/hvf/README.md`) # OS X Hypervisor.framework support in QEMU These sources (and ../hvf-all.c) are adapted from Veertu Inc's vdhh (Veertu Desktop Hosted Hypervisor) (last known location: https://github.com/veertuinc/vdhh) with some minor changes, the most significant of which were: 1. Adapt to our current QEMU's `CPUState` structure and `address_space_rw` API; many struct members have been moved around (emulated x86 state, xsave_buf) due to historical differences + QEMU needing to handle more emulation targets. 2. Removal of `apic_page` and hyperv-related functionality. 3. More relaxed use of `qemu_mutex_lock_iothread`. --- ### COLO FT (`docs/COLO-FT.txt`) COarse-grained LOck-stepping Virtual Machines for Non-stop Service ---------------------------------------- Copyright (c) 2016 Intel Corporation Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. Copyright (c) 2016 Fujitsu, Corp. This work is licensed under the terms of the GNU GPL, version 2 or later. See the COPYING file in the top-level directory. This document gives an overview of COLO's design and how to use it. == Background == Virtual machine (VM) replication is a well known technique for providing application-agnostic software-implemented hardware fault tolerance, also known as "non-stop service". COLO (COarse-grained LOck-stepping) is a high availability solution. Both primary VM (PVM) and secondary VM (SVM) run in parallel. They receive the same request from client, and generate response in parallel too. If the response packets from PVM and SVM are identical, they are released immediately. Otherwise, a VM checkpoint (on demand) is conducted. == Architecture == The architecture of COLO is shown in the diagram below. It consists of a pair of networked physical nodes: The primary node running the PVM, and the secondary node running the SVM to maintain a valid replica of the PVM. PVM and SVM execute in parallel and generate output of response packets for client requests according to the application semantics. The incoming packets from the client or external network are received by the primary node, and then forwarded to the secondary node, so that both the PVM and the SVM are stimulated with the same requests. COLO receives the outbound packets from both the PVM and SVM and compares them before allowing the output to be sent to clients. The SVM is qualified as a valid replica of the PVM, as long as it generates identical responses to all client requests. Once the differences in the outputs are detected between the PVM and SVM, COLO withholds transmission of the outbound packets until it has successfully synchronized the PVM state to the SVM. Primary Node Secondary Node +------------+ +-----------------------+ +------------------------+ +------------+ | | | HeartBeat +<----->+ HeartBeat | | | | Primary VM | +-----------+-----------+ +-----------+------------+ |Secondary VM| | | | | | | | | +-----------|-----------+ +-----------|------------+ | | | | |QEMU +---v----+ | |QEMU +----v---+ | | | | | | |Failover| | | |Failover| | | | | | | +--------+ | | +--------+ | | | | | | +---------------+ | | +---------------+ | | | | | | | VM Checkpoint +-------------->+ VM Checkpoint | | | | | | | +---------------+ | | +---------------+ | | | |Requests<--------------------------\ /-----------------\ /--------------------->Requests| | | | ^ ^ | | | | | | | |Responses+---------------------\ /-|-|------------\ /-------------------------+Responses| | | | | | | | | | | | | | | | | | | | +-----------+ | | | | | | | | | | +----------+ | | | | | | | COLO disk | | | | | | | | | | | | COLO disk| | | | | | | | Manager +---------------------------->| Manager | | | | | | | ++----------+ v v | | | | | v v | +---------++ | | | | | | |+-----------+-+-+-++| | ++-+--+-+---------+ | | | | | | | || COLO Proxy || | | COLO Proxy | | | | | | | | || (compare packet || | |(adjust sequence | | | | | | | | ||and mirror packet)|| | | and ACK) | | | | | | | | |+------------+---+-+| | +-----------------+ | | | | +------------+ +-----------------------+ +------------------------+ +------------+ +------------+ | | | | +------------+ | VM Monitor | | | | | | VM Monitor | +------------+ | | | | +------------+ +---------------------------------------+ +----------------------------------------+ | Kernel | | | | | Kernel | | +---------------------------------------+ +----------------------------------------+ | | | | +--------------v+ +---------v---+--+ +------------------+ +v-------------+ | Storage | |External Network| | External Network | | Storage | +---------------+ +----------------+ +------------------+ +--------------+ == Components introduction == You can see there are several components in COLO's diagram of architecture. Their functions are described below. HeartBeat: Runs on both the primary and secondary nodes, to periodically check platform availability. When the primary node suffers a hardware fail-stop failure, the heartbeat stops responding, the secondary node will trigger a failover as soon as it determines the absence. COLO disk Manager: When primary VM writes data into image, the colo disk manager captures this data and sends it to secondary VM's which makes sure the context of secondary VM's image is consistent with the context of primary VM 's image. For more details, please refer to docs/block-replication.txt. Checkpoint/Failover Controller: Modifications of save/restore flow to realize continuous migration, to make sure the state of VM in Secondary side is always consistent with VM in Primary side. COLO Proxy: Delivers packets to Primary and Secondary, and then compare the responses from both side. Then decide whether to start a checkpoint according to some rules. Please refer to docs/colo-proxy.txt for more information. Note: HeartBeat has not been implemented yet, so you need to trigger failover process by using 'x-colo-lost-heartbeat' command. == COLO operation status == +-----------------+ | | | Start COLO | | | +--------+--------+ | | Main qmp command: | migrate-set-capabilities with x-colo | migrate | v +--------+--------+ | | | COLO running | | | +--------+--------+ | | Main qmp command: | x-colo-lost-heartbeat | or | some error happened v +--------+--------+ | | send qmp event: | COLO failover | COLO_EXIT | | +-----------------+ COLO use the qmp command to switch and report operation status. The diagram just shows the main qmp command, you can get the detail in test procedure. == Test procedure == Note: Here we are running both instances on the same host for testing, change the IP Addresses if you want to run it on two hosts. Initially 127.0.0.1 is the Primary Host and 127.0.0.2 is the Secondary Host. == Startup qemu == 1. Primary: Note: Initially, $imagefolder/primary.qcow2 needs to be copied to all hosts. You don't need to change any IP's here, because 0.0.0.0 listens on any interface. The chardev's with 127.0.0.1 IP's loopback to the local qemu instance. # imagefolder="/mnt/vms/colo-test-primary" # qemu-system-x86_64 -enable-kvm -cpu qemu64,kvmclock=on -m 512 -smp 1 -qmp stdio \ -device piix3-usb-uhci -device usb-tablet -name primary \ -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper \ -device rtl8139,id=e0,netdev=hn0 \ -chardev socket,id=mirror0,host=0.0.0.0,port=9003,server=on,wait=off \ -chardev socket,id=compare1,host=0.0.0.0,port=9004,server=on,wait=on \ -chardev socket,id=compare0,host=127.0.0.1,port=9001,server=on,wait=off \ -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \ -chardev socket,id=compare_out,host=127.0.0.1,port=9005,server=on,wait=off \ -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \ -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \ -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \ -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \ -object iothread,id=iothread1 \ -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,\ outdev=compare_out0,iothread=iothread1 \ -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\ children.0.file.filename=$imagefolder/primary.qcow2,children.0.driver=qcow2 -S 2. Secondary: Note: Active and hidden images need to be created only once and the size should be the same as primary.qcow2. Again, you don't need to change any IP's here, except for the $primary_ip variable. # imagefolder="/mnt/vms/colo-test-secondary" # primary_ip=127.0.0.1 # qemu-img create -f qcow2 $imagefolder/secondary-active.qcow2 10G # qemu-img create -f qcow2 $imagefolder/secondary-hidden.qcow2 10G # qemu-system-x86_64 -enable-kvm -cpu qemu64,kvmclock=on -m 512 -smp 1 -qmp stdio \ -device piix3-usb-uhci -device usb-tablet -name secondary \ -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper \ -device rtl8139,id=e0,netdev=hn0 \ -chardev socket,id=red0,host=$primary_ip,port=9003,reconnect=1 \ -chardev socket,id=red1,host=$primary_ip,port=9004,reconnect=1 \ -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \ -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \ -object filter-rewriter,id=rew0,netdev=hn0,queue=all \ -drive if=none,id=parent0,file.filename=$imagefolder/primary.qcow2,driver=qcow2 \ -drive if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,\ top-id=colo-disk0,file.file.filename=$imagefolder/secondary-active.qcow2,\ file.backing.driver=qcow2,file.backing.file.filename=$imagefolder/secondary-hidden.qcow2,\ file.backing.backing=parent0 \ -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\ children.0=childs0 \ -incoming tcp:0.0.0.0:9998 3. On Secondary VM's QEMU monitor, issue command {"execute":"qmp_capabilities"} {"execute": "nbd-server-start", "arguments": {"addr": {"type": "inet", "data": {"host": "0.0.0.0", "port": "9999"} } } } {"execute": "nbd-server-add", "arguments": {"device": "parent0", "writable": true } } Note: a. The qmp command nbd-server-start and nbd-server-add must be run before running the qmp command migrate on primary QEMU b. Active disk, hidden disk and nbd target's length should be the same. c. It is better to put active disk and hidden disk in ramdisk. They will be merged into the parent disk on failover. 4. On Primary VM's QEMU monitor, issue command: {"execute":"qmp_capabilities"} {"execute": "human-monitor-command", "arguments": {"command-line": "drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.2,file.port=9999,file.export=parent0,node-name=replication0"}} {"execute": "x-blockdev-change", "arguments":{"parent": "colo-disk0", "node": "replication0" } } {"execute": "migrate-set-capabilities", "arguments": {"capabilities": [ {"capability": "x-colo", "state": true } ] } } {"execute": "migrate", "arguments": {"uri": "tcp:127.0.0.2:9998" } } Note: a. There should be only one NBD Client for each primary disk. b. The qmp command line must be run after running qmp command line in secondary qemu. 5. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced. You can issue command '{ "execute": "migrate-set-parameters" , "arguments":{ "x-checkpoint-delay": 2000 } }' to change the idle checkpoint period time 6. Failover test You can kill one of the VMs and Failover on the surviving VM: If you killed the Secondary, then follow "Primary Failover". After that, if you want to resume the replication, follow "Primary resume replication" If you killed the Primary, then follow "Secondary Failover". After that, if you want to resume the replication, follow "Secondary resume replication" == Primary Failover == The Secondary died, resume on the Primary {"execute": "x-blockdev-change", "arguments":{ "parent": "colo-disk0", "child": "children.1"} } {"execute": "human-monitor-command", "arguments":{ "command-line": "drive_del replication0" } } {"execute": "object-del", "arguments":{ "id": "comp0" } } {"execute": "object-del", "arguments":{ "id": "iothread1" } } {"execute": "object-del", "arguments":{ "id": "m0" } } {"execute": "object-del", "arguments":{ "id": "redire0" } } {"execute": "object-del", "arguments":{ "id": "redire1" } } {"execute": "x-colo-lost-heartbeat" } == Secondary Failover == The Primary died, resume on the Secondary and prepare to become the new Primary {"execute": "nbd-server-stop"} {"execute": "x-colo-lost-heartbeat"} {"execute": "object-del", "arguments":{ "id": "f2" } } {"execute": "object-del", "arguments":{ "id": "f1" } } {"execute": "chardev-remove", "arguments":{ "id": "red1" } } {"execute": "chardev-remove", "arguments":{ "id": "red0" } } {"execute": "chardev-add", "arguments":{ "id": "mirror0", "backend": {"type": "socket", "data": {"addr": { "type": "inet", "data": { "host": "0.0.0.0", "port": "9003" } }, "server": true } } } } {"execute": "chardev-add", "arguments":{ "id": "compare1", "backend": {"type": "socket", "data": {"addr": { "type": "inet", "data": { "host": "0.0.0.0", "port": "9004" } }, "server": true } } } } {"execute": "chardev-add", "arguments":{ "id": "compare0", "backend": {"type": "socket", "data": {"addr": { "type": "inet", "data": { "host": "127.0.0.1", "port": "9001" } }, "server": true } } } } {"execute": "chardev-add", "arguments":{ "id": "compare0-0", "backend": {"type": "socket", "data": {"addr": { "type": "inet", "data": { "host": "127.0.0.1", "port": "9001" } }, "server": false } } } } {"execute": "chardev-add", "arguments":{ "id": "compare_out", "backend": {"type": "socket", "data": {"addr": { "type": "inet", "data": { "host": "127.0.0.1", "port": "9005" } }, "server": true } } } } {"execute": "chardev-add", "arguments":{ "id": "compare_out0", "backend": {"type": "socket", "data": {"addr": { "type": "inet", "data": { "host": "127.0.0.1", "port": "9005" } }, "server": false } } } } == Primary resume replication == Resume replication after new Secondary is up. Start the new Secondary (Steps 2 and 3 above), then on the Primary: {"execute": "drive-mirror", "arguments":{ "device": "colo-disk0", "job-id": "resync", "target": "nbd://127.0.0.2:9999/parent0", "mode": "existing", "format": "raw", "sync": "full"} } Wait until disk is synced, then: {"execute": "stop"} {"execute": "block-job-cancel", "arguments":{ "device": "resync"} } {"execute": "human-monitor-command", "arguments":{ "command-line": "drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.2,file.port=9999,file.export=parent0,node-name=replication0"}} {"execute": "x-blockdev-change", "arguments":{ "parent": "colo-disk0", "node": "replication0" } } {"execute": "object-add", "arguments":{ "qom-type": "filter-mirror", "id": "m0", "netdev": "hn0", "queue": "tx", "outdev": "mirror0" } } {"execute": "object-add", "arguments":{ "qom-type": "filter-redirector", "id": "redire0", "netdev": "hn0", "queue": "rx", "indev": "compare_out" } } {"execute": "object-add", "arguments":{ "qom-type": "filter-redirector", "id": "redire1", "netdev": "hn0", "queue": "rx", "outdev": "compare0" } } {"execute": "object-add", "arguments":{ "qom-type": "iothread", "id": "iothread1" } } {"execute": "object-add", "arguments":{ "qom-type": "colo-compare", "id": "comp0", "primary_in": "compare0-0", "secondary_in": "compare1", "outdev": "compare_out0", "iothread": "iothread1" } } {"execute": "migrate-set-capabilities", "arguments":{ "capabilities": [ {"capability": "x-colo", "state": true } ] } } {"execute": "migrate", "arguments":{ "uri": "tcp:127.0.0.2:9998" } } Note: If this Primary previously was a Secondary, then we need to insert the filters before the filter-rewriter by using the ""insert": "before", "position": "id=rew0"" Options. See below. == Secondary resume replication == Become Primary and resume replication after new Secondary is up. Note that now 127.0.0.1 is the Secondary and 127.0.0.2 is the Primary. Start the new Secondary (Steps 2 and 3 above, but with primary_ip=127.0.0.2), then on the old Secondary: {"execute": "drive-mirror", "arguments":{ "device": "colo-disk0", "job-id": "resync", "target": "nbd://127.0.0.1:9999/parent0", "mode": "existing", "format": "raw", "sync": "full"} } Wait until disk is synced, then: {"execute": "stop"} {"execute": "block-job-cancel", "arguments":{ "device": "resync" } } {"execute": "human-monitor-command", "arguments":{ "command-line": "drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,file.port=9999,file.export=parent0,node-name=replication0"}} {"execute": "x-blockdev-change", "arguments":{ "parent": "colo-disk0", "node": "replication0" } } {"execute": "object-add", "arguments":{ "qom-type": "filter-mirror", "id": "m0", "insert": "before", "position": "id=rew0", "netdev": "hn0", "queue": "tx", "outdev": "mirror0" } } {"execute": "object-add", "arguments":{ "qom-type": "filter-redirector", "id": "redire0", "insert": "before", "position": "id=rew0", "netdev": "hn0", "queue": "rx", "indev": "compare_out" } } {"execute": "object-add", "arguments":{ "qom-type": "filter-redirector", "id": "redire1", "insert": "before", "position": "id=rew0", "netdev": "hn0", "queue": "rx", "outdev": "compare0" } } {"execute": "object-add", "arguments":{ "qom-type": "iothread", "id": "iothread1" } } {"execute": "object-add", "arguments":{ "qom-type": "colo-compare", "id": "comp0", "primary_in": "compare0-0", "secondary_in": "compare1", "outdev": "compare_out0", "iothread": "iothread1" } } {"execute": "migrate-set-capabilities", "arguments":{ "capabilities": [ {"capability": "x-colo", "state": true } ] } } {"execute": "migrate", "arguments":{ "uri": "tcp:127.0.0.1:9998" } } == TODO == 1. Support shared storage. 2. Develop the heartbeat part. 3. Reduce checkpoint VM’s downtime while doing checkpoint. --- ### Block Replication (`docs/block-replication.txt`) Block replication ---------------------------------------- Copyright Fujitsu, Corp. 2016 Copyright (c) 2016 Intel Corporation Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. This work is licensed under the terms of the GNU GPL, version 2 or later. See the COPYING file in the top-level directory. Block replication is used for continuous checkpoints. It is designed for COLO (COarse-grain LOck-stepping) where the Secondary VM is running. It can also be applied for FT/HA (Fault-tolerance/High Assurance) scenario, where the Secondary VM is not running. This document gives an overview of block replication's design. == Background == High availability solutions such as micro checkpoint and COLO will do consecutive checkpoints. The VM state of the Primary and Secondary VM is identical right after a VM checkpoint, but becomes different as the VM executes till the next checkpoint. To support disk contents checkpoint, the modified disk contents in the Secondary VM must be buffered, and are only dropped at next checkpoint time. To reduce the network transportation effort during a vmstate checkpoint, the disk modification operations of the Primary disk are asynchronously forwarded to the Secondary node. == Workflow == The following is the image of block replication workflow: +----------------------+ +------------------------+ |Primary Write Requests| |Secondary Write Requests| +----------------------+ +------------------------+ | | | (4) | V | /-------------\ | Copy and Forward | | |---------(1)----------+ | Disk Buffer | | | | | | (3) \-------------/ | speculative ^ | write through (2) | | | V V | +--------------+ +----------------+ | Primary Disk | | Secondary Disk | +--------------+ +----------------+ 1) Primary write requests will be copied and forwarded to Secondary QEMU. 2) Before Primary write requests are written to Secondary disk, the original sector content will be read from Secondary disk and buffered in the Disk buffer, but it will not overwrite the existing sector content (it could be from either "Secondary Write Requests" or previous COW of "Primary Write Requests") in the Disk buffer. 3) Primary write requests will be written to Secondary disk. 4) Secondary write requests will be buffered in the Disk buffer and it will overwrite the existing sector content in the buffer. == Architecture == We are going to implement block replication from many basic blocks that are already in QEMU. virtio-blk || ^ || .---------- | || | Secondary 1 Quorum || '---------- / \ || virtio-blk / \ || ^ Primary 2 filter | disk ^ 7 Quorum | / 3 NBD -------> 3 NBD / client || server 2 filter || ^ ^ --------. || | | Primary | || Secondary disk <--------- hidden-disk 5 <--------- active-disk 4 --------' || | backing ^ backing || | | || | | || '-------------------------' || blockdev-backup sync=none 6 1) The disk on the primary is represented by a block device with two children, providing replication between a primary disk and the host that runs the secondary VM. The read pattern (fifo) for quorum can be extended to make the primary always read from the local disk instead of going through NBD. 2) The new block filter (the name is replication) will control the block replication. 3) The secondary disk receives writes from the primary VM through QEMU's embedded NBD server (speculative write-through). 4) The disk on the secondary is represented by a custom block device (called active-disk). It should start as an empty disk, and the format should support bdrv_make_empty() and backing file. 5) The hidden-disk is created automatically. It buffers the original content that is modified by the primary VM. It should also start as an empty disk, and the driver supports bdrv_make_empty() and backing file. 6) The blockdev-backup job (sync=none) is run to allow hidden-disk to buffer any state that would otherwise be lost by the speculative write-through of the NBD server into the secondary disk. So before block replication, the primary disk and secondary disk should contain the same data. 7) The secondary also has a quorum node, so after secondary failover it can become the new primary and continue replication. == Failure Handling == There are 7 internal errors when block replication is running: 1. I/O error on primary disk 2. Forwarding primary write requests failed 3. Backup failed 4. I/O error on secondary disk 5. I/O error on active disk 6. Making active disk or hidden disk empty failed 7. Doing failover failed In case 1 and 5, we just report the error to the disk layer. In case 2, 3, 4 and 6, we just report block replication's error to FT/HA manager (which decides when to do a new checkpoint, when to do failover). In case 7, if active commit failed, we use replication failover failed state in Secondary's write operation (what decides which target to write). == New block driver interface == We add four block driver interfaces to control block replication: a. replication_start_all() Start block replication, called in migration/checkpoint thread. We must call block_replication_start_all() in secondary QEMU before calling block_replication_start_all() in primary QEMU. The caller must hold the I/O mutex lock if it is in migration/checkpoint thread. b. replication_do_checkpoint_all() This interface is called after all VM state is transferred to Secondary QEMU. The Disk buffer will be dropped in this interface. The caller must hold the I/O mutex lock if it is in migration/checkpoint thread. c. replication_get_error_all() This interface is called to check if error happened in replication. The caller must hold the I/O mutex lock if it is in migration/checkpoint thread. d. replication_stop_all() It is called on failover. We will flush the Disk buffer into Secondary Disk and stop block replication. The vm should be stopped before calling it if you use this API to shutdown the guest, or other things except failover. The caller must hold the I/O mutex lock if it is in migration/checkpoint thread. == Usage == Primary: -drive if=xxx,driver=quorum,read-pattern=fifo,id=colo1,vote-threshold=1,\ children.0.file.filename=1.raw,\ children.0.driver=raw Run qmp command in primary qemu: { "execute": "human-monitor-command", "arguments": { "command-line": "drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=xxxx,file.port=xxxx,file.export=colo1,node-name=nbd_client1" } } { "execute": "x-blockdev-change", "arguments": { "parent": "colo1", "node": "nbd_client1" } } Note: 1. There should be only one NBD Client for each primary disk. 2. host is the secondary physical machine's hostname or IP 3. Each disk must have its own export name. 4. It is all a single argument to -drive and you should ignore the leading whitespace. 5. The qmp command line must be run after running qmp command line in secondary qemu. 6. After primary failover we need remove children.1 (replication driver). Secondary: -drive if=none,driver=raw,file.filename=1.raw,id=colo1 \ -drive if=none,id=childs1,driver=replication,mode=secondary,top-id=top-disk1 file.file.filename=active_disk.qcow2,\ file.driver=qcow2,\ file.backing.file.filename=hidden_disk.qcow2,\ file.backing.driver=qcow2,\ file.backing.backing=colo1 -drive if=xxx,driver=quorum,read-pattern=fifo,id=top-disk1,\ vote-threshold=1,children.0=childs1 Then run qmp command in secondary qemu: { "execute": "nbd-server-start", "arguments": { "addr": { "type": "inet", "data": { "host": "xxx", "port": "xxx" } } } } { "execute": "nbd-server-add", "arguments": { "device": "colo1", "writable": true } } Note: 1. The export name in secondary QEMU command line is the secondary disk's id. 2. The export name for the same disk must be the same 3. The qmp command nbd-server-start and nbd-server-add must be run before running the qmp command migrate on primary QEMU 4. Active disk, hidden disk and nbd target's length should be the same. 5. It is better to put active disk and hidden disk in ramdisk. 6. It is all a single argument to -drive, and you should ignore the leading whitespace. After Failover: Primary: The secondary host is down, so we should run the following qmp command to remove the nbd child from the quorum: { "execute": "x-blockdev-change", "arguments": { "parent": "colo1", "child": "children.1" } } { "execute": "human-monitor-command", "arguments": { "command-line": "drive_del xxxx" } } Note: there is no qmp command to remove the blockdev now Secondary: The primary host is down, so we should do the following thing: { "execute": "nbd-server-stop" } Promote Secondary to Primary: see COLO-FT.txt TODO: 1. Shared disk --- ### Bypass Iommu (`docs/bypass-iommu.txt`) BYPASS IOMMU PROPERTY ===================== Description =========== Traditionally, there is a global switch to enable/disable vIOMMU. All devices in the system can only support go through vIOMMU or not, which is not flexible. We introduce this bypass iommu property to support coexist of devices go through vIOMMU and devices not. This is useful to passthrough devices with no-iommu mode and devices go through vIOMMU in the same virtual machine. PCI host bridges have a bypass_iommu property. This property is used to determine whether the devices attached on the PCI host bridge will bypass virtual iommu. The bypass_iommu property is valid only when there is a virtual iommu in the system, it is implemented to allow some devices to bypass vIOMMU. When bypass_iommu property is not set for a host bridge, the attached devices will go through vIOMMU by default. Usage ===== The bypass iommu feature support PXB host bridge and default main host bridge, we add a bypass_iommu property for PXB and default_bus_bypass_iommu for machine. Note that default_bus_bypass_iommu is available only when the 'q35' machine type on x86 architecture and the 'virt' machine type on AArch64. Other machine types do not support bypass iommu for default root bus. 1. The following is the bypass iommu options: (1) PCI expander bridge qemu -device pxb-pcie,bus_nr=0x10,addr=0x1,bypass_iommu=true (2) Arm default host bridge qemu -machine virt,iommu=smmuv3,default_bus_bypass_iommu=true (3) X86 default root bus bypass iommu: qemu -machine q35,default_bus_bypass_iommu=true 2. Here is the detailed qemu command line for 'virt' machine with PXB on AArch64: qemu-system-aarch64 \ -machine virt,kernel_irqchip=on,iommu=smmuv3,default_bus_bypass_iommu=true \ -device pxb-pcie,bus_nr=0x10,id=pci.10,bus=pcie.0,addr=0x3.0x1 \ -device pxb-pcie,bus_nr=0x20,id=pci.20,bus=pcie.0,addr=0x3.0x2,bypass_iommu=true \ And we got: - a default host bridge which bypass SMMUv3 - a pxb host bridge which go through SMMUv3 - a pxb host bridge which bypass SMMUv3 3. Here is the detailed qemu command line for 'q35' machine with PXB on x86 architecture: qemu-system-x86_64 \ -machine q35,accel=kvm,default_bus_bypass_iommu=true \ -device pxb-pcie,bus_nr=0x10,id=pci.10,bus=pcie.0,addr=0x3 \ -device pxb-pcie,bus_nr=0x20,id=pci.20,bus=pcie.0,addr=0x4,bypass_iommu=true \ -device intel-iommu \ And we got: - a default host bridge which bypass iommu - a pxb host bridge which go through iommu - a pxb host bridge which bypass iommu Limitations =========== There might be potential security risk when devices bypass iommu, because devices might send malicious dma request to virtual machine if there is no iommu isolation. So it would be necessary to only bypass iommu for trusted device. Implementation ============== The bypass iommu feature includes: - Address space Add bypass iommu property check of PCI Host and do not get iommu address space for devices bypass iommu. - Arm SMMUv3 support We traverse all PCI root bus and get bus number ranges, then build explicit RID mapping for devices which do not bypass iommu. - X86 IOMMU support To support Intel iommu, we traverse all PCI host bridge and get information of devices which do not bypass iommu, then fill the DMAR drhd struct with explicit device scope info. To support AMD iommu, add check of bypass iommu when traverse the PCI hsot bridge. - Machine and PXB options We add bypass iommu options in machine option for default root bus, and add option for PXB also. Note that the default value of bypass iommu is false, so that the devices will by default go through iommu if there exist one. --- ### Colo Proxy (`docs/colo-proxy.txt`) COLO-proxy ---------- Copyright (c) 2016 Intel Corporation Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. Copyright (c) 2016 Fujitsu, Corp. This work is licensed under the terms of the GNU GPL, version 2 or later. See the COPYING file in the top-level directory. This document gives an overview of COLO proxy's design. == Background == COLO-proxy is a part of COLO project. It is used to compare the network package to help COLO decide whether to do checkpoint. With COLO-proxy's help, COLO greatly improves the performance. The filter-redirector, filter-mirror, colo-compare and filter-rewriter compose the COLO-proxy. == Architecture == COLO-Proxy is based on qemu netfilter and it's a plugin for qemu netfilter (except colo-compare). It keep Secondary VM connect normally to client and compare packets sent by PVM with sent by SVM. If the packet difference, notify COLO-frame to do checkpoint and send all primary packet has queued. Otherwise just send the queued primary packet and drop the queued secondary packet. Below is a COLO proxy ascii figure: Primary qemu Secondary qemu +--------------------------------------------------------------+ +----------------------------------------------------------------+ | +----------------------------------------------------------+ | | +-----------------------------------------------------------+ | | | | | | | | | | | guest | | | | guest | | | | | | | | | | | +-------^--------------------------+-----------------------+ | | +---------------------+--------+----------------------------+ | | | | | | ^ | | | | | | | | | | | | +------------------------------------------------------+ | | | | |netfilter| | | | | | netfilter | | | | +----------+ +----------------------------+ | | | +-----------------------------------------------------------+ | | | | | | | out | | | | | | filter execute order | | | | | | +-----------------------------+ | | | | | | +-------------------> | | | | | | | | | | | | | | | | TCP | | | | +-----+--+-+ +-----v----+ +-----v----+ |pri +----+----+sec| | | | +------------+ +---+----+---v+rewriter++ +------------+ | | | | | | | | | | |in | |in | | | | | | | | | | | | | | | | filter | | filter | | filter +------> colo <------+ +--------> filter +--> adjust | adjust +--> filter | | | | | | mirror | |redirector| |redirector| | | compare | | | | | | redirector | | ack | seq | | redirector | | | | | | | | | | | | | | | | | | | | | | | | | | | | | +----^-----+ +----+-----+ +----------+ | +---------+ | | | | +------------+ +--------+--------------+ +---+--------+ | | | | | tx | rx rx | | | | | tx all | rx | | | | | | | | | | +-----------------------------------------------------------+ | | | | +--------------+ | | | | | | | | | filter execute order | | | | | | | | | | +----------------> | | | +--------------------------------------------------------+ | | +-----------------------------------------+ | | | | | | | | | +--------------------------------------------------------------+ +----------------------------------------------------------------+ |guest receive | guest send | | +--------+----------------------------v------------------------+ | | NOTE: filter direction is rx/tx/all | tap | rx:receive packets sent to the netdev | | tx:receive packets sent by the netdev +--------------------------------------------------------------+ 1.Guest receive packet route: Primary: Tap --> Mirror Client Filter Mirror client will send packet to guest,at the same time, copy and forward packet to secondary mirror server. Secondary: Mirror Server Filter --> TCP Rewriter If receive packet is TCP packet,we will adjust ack and update TCP checksum, then send to secondary guest. Otherwise directly send to guest. 2.Guest send packet route: Primary: Guest --> Redirect Server Filter Redirect server filter receive primary guest packet but do nothing, just pass to next filter. Redirect Server Filter --> COLO-Compare COLO-compare receive primary guest packet then waiting secondary redirect packet to compare it. If packet same,send queued primary packet and clear queued secondary packet, Otherwise send primary packet and do checkpoint. COLO-Compare --> Another Redirector Filter The redirector get packet from colo-compare by use chardev socket. Redirector Filter --> Tap Send the packet. Secondary: Guest --> TCP Rewriter Filter If the packet is TCP packet,we will adjust seq and update TCP checksum. Then send it to redirect client filter. Otherwise directly send to redirect client filter. Redirect Client Filter --> Redirect Server Filter Forward packet to primary. == Components introduction == Filter-mirror is a netfilter plugin. It gives qemu the ability to mirror packets to a chardev. Filter-redirector is a netfilter plugin. It gives qemu the ability to redirect net packet. Redirector can redirect filter's net packet to outdev, and redirect indev's packet to filter. filter + redirector | +--------------+ | | | | | | | | | indev +---------+ +----------> outdev | | | | | | | | | +--------------+ | v filter COLO-compare, we do packet comparing job. Packets coming from the primary char indev will be sent to outdev. Packets coming from the secondary char dev will be dropped after comparing. COLO-compare needs two input chardevs and one output chardev: primary_in=chardev1-id (source: primary send packet) secondary_in=chardev2-id (source: secondary send packet) outdev=chardev3-id Filter-rewriter will rewrite some of secondary packet to make secondary guest's tcp connection established successfully. In this module we will rewrite tcp packet's ack to the secondary from primary,and rewrite tcp packet's seq to the primary from secondary. == Usage == Here is an example using demonstration IP and port addresses to more clearly describe the usage. Primary(ip:3.3.3.3): -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66 -chardev socket,id=mirror0,host=3.3.3.3,port=9003,server=on,wait=off -chardev socket,id=compare1,host=3.3.3.3,port=9004,server=on,wait=off -chardev socket,id=compare0,host=3.3.3.3,port=9001,server=on,wait=off -chardev socket,id=compare0-0,host=3.3.3.3,port=9001 -chardev socket,id=compare_out,host=3.3.3.3,port=9005,server=on,wait=off -chardev socket,id=compare_out0,host=3.3.3.3,port=9005 -object iothread,id=iothread1 -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,iothread=iothread1 Secondary(ip:3.3.3.8): -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,down script=/etc/qemu-ifdown -device e1000,netdev=hn0,mac=52:a4:00:12:78:66 -chardev socket,id=red0,host=3.3.3.3,port=9003 -chardev socket,id=red1,host=3.3.3.3,port=9004 -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 -object filter-rewriter,id=f3,netdev=hn0,queue=all If you want to use virtio-net-pci or other driver with vnet_header: Primary(ip:3.3.3.3): -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66 -chardev socket,id=mirror0,host=3.3.3.3,port=9003,server=on,wait=off -chardev socket,id=compare1,host=3.3.3.3,port=9004,server=on,wait=off -chardev socket,id=compare0,host=3.3.3.3,port=9001,server=on,wait=off -chardev socket,id=compare0-0,host=3.3.3.3,port=9001 -chardev socket,id=compare_out,host=3.3.3.3,port=9005,server=on,wait=off -chardev socket,id=compare_out0,host=3.3.3.3,port=9005 -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0,vnet_hdr_support -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out,vnet_hdr_support -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0,vnet_hdr_support -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,vnet_hdr_support Secondary(ip:3.3.3.8): -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,down script=/etc/qemu-ifdown -device e1000,netdev=hn0,mac=52:a4:00:12:78:66 -chardev socket,id=red0,host=3.3.3.3,port=9003 -chardev socket,id=red1,host=3.3.3.3,port=9004 -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0,vnet_hdr_support -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1,vnet_hdr_support -object filter-rewriter,id=f3,netdev=hn0,queue=all,vnet_hdr_support Note: a.COLO-proxy must work with COLO-frame and Block-replication. b.Primary COLO must be started firstly, because COLO-proxy needs chardev socket server running before secondary started. c.Filter-rewriter only rewrite tcp packet. --- ### Igd Assign (`docs/igd-assign.txt`) Intel Graphics Device (IGD) assignment with vfio-pci ==================================================== IGD has two different modes for assignment using vfio-pci: 1) Universal Pass-Through (UPT) mode: In this mode the IGD device is added as a *secondary* (ie. non-primary) graphics device in combination with an emulated primary graphics device. This mode *requires* guest driver support to remove the external dependencies generally associated with IGD (see below). Those guest drivers only support this mode for Broadwell and newer IGD, according to Intel. Additionally, this mode by default, and as officially supported by Intel, does not support direct video output. The intention is to use this mode either to provide hardware acceleration to the emulated graphics or to use this mode in combination with guest-based remote access software, for example VNC (see below for optional output support). This mode theoretically has no device specific handling dependencies on vfio-pci or the VM firmware. 2) "Legacy" mode: In this mode the IGD device is intended to be the primary and exclusive graphics device in the VM[1], as such QEMU does not facilitate any sort of remote graphics to the VM in this mode. A connected physical monitor is the intended output device for IGD. This mode includes several requirements and restrictions: * IGD must be given address 02.0 on the PCI root bus in the VM * The host kernel must support vfio extensions for IGD (v4.6) * vfio VGA support very likely needs to be enabled in the host kernel * The VM firmware must support specific fw_cfg enablers for IGD * The VM machine type must support a PCI host bridge at 00.0 (standard) * The VM machine type must provide or allow to be created a special ISA/LPC bridge device (vfio-pci-igd-lpc-bridge) on the root bus at PCI address 1f.0. * The IGD device must have a VGA ROM, either provided via the romfile option or loaded automatically through vfio (standard). rombar=0 will disable legacy mode support. * Hotplug of the IGD device is not supported. * The IGD device must be a SandyBridge or newer model device. For either mode, depending on the host kernel, the i915 driver in the host may generate faults and errors upon re-binding to an IGD device after it has been assigned to a VM. It's therefore generally recommended to prevent such driver binding unless the host driver is known to work well for this. There are numerous ways to do this, i915 can be blacklisted on the host, the driver_override option can be used to ensure that only vfio-pci can bind to the device on the host[2], virsh nodedev-detach can be used to bind the device to vfio drivers and then managed='no' set in the VM xml to prevent re-binding to i915, etc. Also note that IGD is also typically the primary graphics in the host and special options may be required beyond simply blacklisting i915 or using pci-stub/vfio-pci to take ownership of IGD as a PCI class device. Lower level drivers exist that may still claim the device. It may therefore be necessary to use kernel boot options video=vesafb:off or video=efifb:off (depending on host BIOS/UEFI) or these can be combined to a catch-all, video=vesafb:off,efifb:off. Error messages such as: Failed to mmap 0000:00:02.0 BAR <>. Performance may be slow are a good indicator that such a problem exists. The host files /proc/iomem and /proc/ioports are often useful for identifying drivers consuming ranges of the device to cause such conflicts. Additionally, IGD device are known to generate small numbers of DMAR faults when initially assigned. It is believed that this is simply the IGD attempting to access the reserved GTT space after reset, which it no longer has access to when accessed from userspace. So long as the DMAR faults are small in number and most importantly, not ongoing, these are not an indication of an error. Additionally++, analog VGA output (as opposed to digital outputs like HDMI, DVI, or DisplayPort) may be unsupported in some use cases. In the author's experience, even DP to VGA adapters can be troublesome while adapters between digital formats work well. Usage ===== The intention is for IGD assignment to be transparent for users and thus for management tools like libvirt. To make use of legacy mode, simply remove all other graphics options and use "-nographic" and either "-vga none" or "-nodefaults", along with adding the device using vfio-pci: -device vfio-pci,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x2 For UPT mode, retain the default emulated graphics and simply add the vfio-pci device making use of any other bus address other than 02.0. libvirt will default to assigning the device a UPT compatible address while legacy mode users will need to manually edit the XML if using a tool like virt-manager where the VM device address is not expressly specified. An experimental vfio-pci option also exists to enable OpRegion, and thus external monitor support, for UPT mode. This can be enabled by adding "x-igd-opregion=on" to the vfio-pci device options for the IGD device. As with legacy mode, this requires the host to support features introduced in the v4.6 kernel. If Intel chooses to embrace this support, the option may be made non-experimental in the future, opening it to libvirt support. Developer ABI ============= Legacy mode IGD support imposes two fw_cfg requirements on the VM firmware: 1) "etc/igd-opregion" This fw_cfg file exposes the OpRegion for the IGD device. A reserved region should be created below 4GB (recommended 4KB alignment), sized sufficient for the fw_cfg file size, and the content of this file copied to it. The dword based address of this reserved memory region must also be written to the ASLS register at offset 0xFC on the IGD device. It is recommended that firmware should make use of this fw_cfg entry for any PCI class VGA device with Intel vendor ID. Multiple of such devices within a VM is undefined. 2) "etc/igd-bdsm-size" This fw_cfg file contains an 8-byte, little endian integer indicating the size of the reserved memory region required for IGD stolen memory. Firmware must allocate a reserved memory below 4GB with required 1MB alignment equal to this size. Additionally the base address of this reserved region must be written to the dword BDSM register in PCI config space of the IGD device at offset 0x5C. As this support is related to running the IGD ROM, which has other dependencies on the device appearing at guest address 00:02.0, it's expected that this fw_cfg file is only relevant to a single PCI class VGA device with Intel vendor ID, appearing at PCI bus address 00:02.0. Footnotes ========= [1] Nothing precludes adding additional emulated or assigned graphics devices as non-primary, other than the combination typically not working. I only intend to set user expectations, others are welcome to find working combinations or fix whatever issues prevent this from working in the common case. [2] # echo "vfio-pci" > /sys/bus/pci/devices/0000:00:02.0/driver_override --- ### Image Fuzzer (`docs/image-fuzzer.txt`) # Specification for the fuzz testing tool # # Copyright (C) 2014 Maria Kustova # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . Image fuzzer ============ Description ----------- The goal of the image fuzzer is to catch crashes of qemu-io/qemu-img by providing to them randomly corrupted images. Test images are generated from scratch and have valid inner structure with some elements, e.g. L1/L2 tables, having random invalid values. Test runner ----------- The test runner generates test images, executes tests utilizing generated images, indicates their results and collects all test related artifacts (logs, core dumps, test images, backing files). The test means execution of all available commands under test with the same generated test image. By default, the test runner generates new tests and executes them until keyboard interruption. But if a test seed is specified via the '--seed' runner parameter, then only one test with this seed will be executed, after its finish the runner will exit. The runner uses an external image fuzzer to generate test images. An image generator should be specified as a mandatory parameter of the test runner. Details about interactions between the runner and fuzzers see "Module interfaces". The runner activates generation of core dumps during test executions, but it assumes that core dumps will be generated in the current working directory. For comprehensive test results, please, set up your test environment properly. Paths to binaries under test (SUTs) ``qemu-img`` and ``qemu-io`` are retrieved from environment variables. If the environment check fails the runner will use SUTs installed in system paths. ``qemu-img`` is required for creation of backing files, so it's mandatory to set the related environment variable if it's not installed in the system path. For details about environment variables see qemu-iotests/check. The runner accepts a JSON array of fields expected to be fuzzed via the '--config' argument, e.g. '[["feature_name_table"], ["header", "l1_table_offset"]]' Each sublist can have one or two strings defining image structure elements. In the latter case a parent element should be placed on the first position, and a field name on the second one. The runner accepts a list of commands under test as a JSON array via the '--command' argument. Each command is a list containing a SUT and all its arguments, e.g. runner.py -c '[["qemu-io", "$test_img", "-c", "write $off $len"]]' /tmp/test ../qcow2 For variable arguments next aliases can be used: - $test_img for a fuzzed img - $off for an offset in the fuzzed image - $len for a data size Values for last two aliases will be generated based on a size of a virtual disk of the generated image. In case when no commands are specified the runner will execute commands from the default list: - qemu-img check - qemu-img info - qemu-img convert - qemu-io -c read - qemu-io -c write - qemu-io -c aio_read - qemu-io -c aio_write - qemu-io -c flush - qemu-io -c discard - qemu-io -c truncate Qcow2 image generator --------------------- The 'qcow2' generator is a Python package providing 'create_image' method as a single public API. See details in 'Test runner/image fuzzer' chapter of 'Module interfaces'. Qcow2 contains two submodules: fuzz.py and layout.py. 'fuzz.py' contains all fuzzing functions, one per image field. It's assumed that after code analysis every field will have own constraints for its value. For now only universal potentially dangerous values are used, e.g. type limits for integers or unsafe symbols as '%s' for strings. For bitmasks random amount of bits are set to ones. All fuzzed values are checked on non-equality to the current valid value of the field. In case of equality the value will be regenerated. 'layout.py' creates a random valid image, fuzzes a random subset of the image fields by 'fuzz.py' module and writes a fuzzed image to the file specified. If a fuzzer configuration is specified, then it has the next interpretation: 1. If a list contains a parent image element only, then some random portion of fields of this element will be fuzzed every test. The same behavior is applied for the entire image if no configuration is used. This case is useful for the test specialization. 2. If a list contains a parent element and a field name, then a field will be always fuzzed for every test. This case is useful for regression testing. The generator can create header fields, header extensions, L1/L2 tables and refcount table and blocks. Module interfaces ----------------- * Test runner/image fuzzer The runner calls an image generator specifying the path to a test image file, path to a backing file and its format and a fuzzer configuration. An image generator is expected to provide a 'create_image(test_img_path, backing_file_path=None, backing_file_format=None, fuzz_config=None)' method that creates a test image, writes it to the specified file and returns the size of the virtual disk. The file should be created if it doesn't exist or overwritten otherwise. fuzz_config has a form of a list of lists. Every sublist can have one or two elements: first element is a name of a parent image element, second one if exists is a name of a field in this element. Example, [['header', 'l1_table_offset'], ['header', 'nb_snapshots'], ['feature_name_table']] Random seed is set by the runner at every test execution for the regression purpose, so an image generator is not recommended to modify it internally. Overall fuzzer requirements =========================== Input data: ---------- - image template (generator) - work directory - action vector (optional) - seed (optional) - SUT and its arguments (optional) Fuzzer requirements: ------------------- 1. Should be able to inject random data 2. Should be able to select a random value from the manually pregenerated vector (boundary values, e.g. max/min cluster size) 3. Image template should describe a general structure invariant for all test images (image format description) 4. Image template should be autonomous and other fuzzer parts should not rely on it 5. Image template should contain reference rules (not only block+size description) 6. Should generate the test image with the correct structure based on an image template 7. Should accept a seed as an argument (for regression purpose) 8. Should generate a seed if it is not specified as an input parameter. 9. The same seed should generate the same image for the same action vector, specified or generated. 10. Should accept a vector of actions as an argument (for test reproducing and for test case specification, e.g. group of tests for header structure, group of test for snapshots, etc) 11. Action vector should be randomly generated from the pool of available actions, if it is not specified as an input parameter 12. Pool of actions should be defined automatically based on an image template 13. Should accept a SUT and its call parameters as an argument or select them randomly otherwise. As far as it's expected to be rarely changed, the list of all possible test commands can be available in the test runner internally. 14. Should support an external cancellation of a test run 15. Seed should be logged (for regression purpose) 16. All files related to a test result should be collected: a test image, SUT logs, fuzzer logs and crash dumps 17. Should be compatible with python version 2.4-2.7 18. Usage of external libraries should be limited as much as possible. Image formats: ------------- Main target image format is qcow2, but support of image templates should provide an ability to add any other image format. Effectiveness: ------------- The fuzzer can be controlled via template, seed and action vector; it makes the fuzzer itself invariant to an image format and test logic. It should be able to perform rather complex and precise tests, that can be specified via an action vector. Otherwise, knowledge about an image structure allows the fuzzer to generate the pool of all available areas can be fuzzed and randomly select some of them and so compose its own action vector. Also complexity of a template defines complexity of the fuzzer, so its functionality can be varied from simple model-independent fuzzing to smart model-based one. Glossary: -------- Action vector is a sequence of structure elements retrieved from an image format, each of them will be fuzzed for the test image. It's a subset of elements of the action pool. Example: header, refcount table, etc. Action pool is all available elements of an image structure that generated automatically from an image template. Image template is a formal description of an image structure and relations between image blocks. Test image is an output image of the fuzzer defined by the current seed and action vector. --- ### Index (`docs/index.rst`) .. QEMU documentation master file, created by sphinx-quickstart on Thu Jan 31 16:40:14 2019. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. ================================ Welcome to QEMU's documentation! ================================ .. toctree:: :maxdepth: 2 :caption: Contents: about/index system/index user/index tools/index interop/index specs/index devel/index --- ### Memory Hotplug (`docs/memory-hotplug.txt`) QEMU memory hotplug =================== This document explains how to use the memory hotplug feature in QEMU, which is present since v2.1.0. Guest support is required for memory hotplug to work. Basic RAM hotplug ----------------- In order to be able to hotplug memory, QEMU has to be told how many hotpluggable memory slots to create and what is the maximum amount of memory the guest can grow. This is done at startup time by means of the -m command-line option, which has the following format: -m [size=]megs[,slots=n,maxmem=size] Where, - "megs" is the startup RAM. It is the RAM the guest will boot with - "slots" is the number of hotpluggable memory slots - "maxmem" is the maximum RAM size the guest can have For example, the following command-line: qemu [...] -m 1G,slots=3,maxmem=4G Creates a guest with 1GB of memory and three hotpluggable memory slots. The hotpluggable memory slots are empty when the guest is booted, so all memory the guest will see after boot is 1GB. The maximum memory the guest can reach is 4GB. This means that three additional gigabytes can be hotplugged by using any combination of the available memory slots. Two monitor commands are used to hotplug memory: - "object_add": creates a memory backend object - "device_add": creates a front-end pc-dimm device and inserts it into the first empty slot For example, the following commands add another 1GB to the guest discussed earlier: (qemu) object_add memory-backend-ram,id=mem1,size=1G (qemu) device_add pc-dimm,id=dimm1,memdev=mem1 Using the file backend ---------------------- Besides basic RAM hotplug, QEMU also supports using files as a memory backend. This is useful for using hugetlbfs in Linux, which provides access to bigger page sizes. For example, assuming that the host has 1GB hugepages available in the /mnt/hugepages-1GB directory, a 1GB hugepage could be hotplugged into the guest from the previous section with the following commands: (qemu) object_add memory-backend-file,id=mem1,size=1G,mem-path=/mnt/hugepages-1GB (qemu) device_add pc-dimm,id=dimm1,memdev=mem1 It's also possible to start a guest with memory cold-plugged into the hotpluggable memory slots. This might seem counterintuitive at first, but this allows for a lot of flexibility when using the file backend. In the following command-line example, an 8GB guest is created where 6GB comes from regular RAM, 1GB is a 1GB hugepage page and 256MB is from 2MB pages. Also, the guest has additional memory slots to hotplug more 2GB if needed: qemu [...] -m 6GB,slots=4,maxmem=10G \ -object memory-backend-file,id=mem1,size=1G,mem-path=/mnt/hugepages-1G \ -device pc-dimm,id=dimm1,memdev=mem1 \ -object memory-backend-file,id=mem2,size=256M,mem-path=/mnt/hugepages-2MB \ -device pc-dimm,id=dimm2,memdev=mem2 RAM hot-unplug --------------- In order to be able to hot unplug pc-dimm device, QEMU has to be told the ids of pc-dimm device and memory backend object. The ids were assigned when you hot plugged memory. Two monitor commands are used to hot unplug memory: - "device_del": deletes a front-end pc-dimm device - "object_del": deletes a memory backend object For example, assuming that the pc-dimm device with id "dimm1" exists, and its memory backend is "mem1", the following commands tries to remove it. (qemu) device_del dimm1 (qemu) object_del mem1 --- ### Multi Thread Compression (`docs/multi-thread-compression.txt`) Use multiple thread (de)compression in live migration ===================================================== Copyright (C) 2015 Intel Corporation Author: Liang Li This work is licensed under the terms of the GNU GPLv2 or later. See the COPYING file in the top-level directory. Contents: ========= * Introduction * When to use * Performance * Usage * TODO Introduction ============ Instead of sending the guest memory directly, this solution will compress the RAM page before sending; after receiving, the data will be decompressed. Using compression in live migration can help to reduce the data transferred about 60%, this is very useful when the bandwidth is limited, and the total migration time can also be reduced about 70% in a typical case. In addition to this, the VM downtime can be reduced about 50%. The benefit depends on data's compressibility in VM. The process of compression will consume additional CPU cycles, and the extra CPU cycles will increase the migration time. On the other hand, the amount of data transferred will decrease; this factor can reduce the total migration time. If the process of the compression is quick enough, then the total migration time can be reduced, and multiple thread compression can be used to accelerate the compression process. The decompression speed of Zlib is at least 4 times as quick as compression, if the source and destination CPU have equal speed, keeping the compression thread count 4 times the decompression thread count can avoid resource waste. Compression level can be used to control the compression speed and the compression ratio. High compression ratio will take more time, level 0 stands for no compression, level 1 stands for the best compression speed, and level 9 stands for the best compression ratio. Users can select a level number between 0 and 9. When to use the multiple thread compression in live migration ============================================================= Compression of data will consume extra CPU cycles; so in a system with high overhead of CPU, avoid using this feature. When the network bandwidth is very limited and the CPU resource is adequate, use of multiple thread compression will be very helpful. If both the CPU and the network bandwidth are adequate, use of multiple thread compression can still help to reduce the migration time. Performance =========== Test environment: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz Socket Count: 2 RAM: 128G NIC: Intel I350 (10/100/1000Mbps) Host OS: CentOS 7 64-bit Guest OS: RHEL 6.5 64-bit Parameter: qemu-system-x86_64 -accel kvm -smp 4 -m 4096 /share/ia32e_rhel6u5.qcow -monitor stdio There is no additional application is running on the guest when doing the test. Speed limit: 1000Gb/s --------------------------------------------------------------- | original | compress thread: 8 | way | decompress thread: 2 | | compression level: 1 --------------------------------------------------------------- total time(msec): | 3333 | 1833 --------------------------------------------------------------- downtime(msec): | 100 | 27 --------------------------------------------------------------- transferred ram(kB):| 363536 | 107819 --------------------------------------------------------------- throughput(mbps): | 893.73 | 482.22 --------------------------------------------------------------- total ram(kB): | 4211524 | 4211524 --------------------------------------------------------------- There is an application running on the guest which write random numbers to RAM block areas periodically. Speed limit: 1000Gb/s --------------------------------------------------------------- | original | compress thread: 8 | way | decompress thread: 2 | | compression level: 1 --------------------------------------------------------------- total time(msec): | 37369 | 15989 --------------------------------------------------------------- downtime(msec): | 337 | 173 --------------------------------------------------------------- transferred ram(kB):| 4274143 | 1699824 --------------------------------------------------------------- throughput(mbps): | 936.99 | 870.95 --------------------------------------------------------------- total ram(kB): | 4211524 | 4211524 --------------------------------------------------------------- Usage ===== 1. Verify both the source and destination QEMU are able to support the multiple thread compression migration: {qemu} info migrate_capabilities {qemu} ... compress: off ... 2. Activate compression on the source: {qemu} migrate_set_capability compress on 3. Set the compression thread count on source: {qemu} migrate_set_parameter compress_threads 12 4. Set the compression level on the source: {qemu} migrate_set_parameter compress_level 1 5. Set the decompression thread count on destination: {qemu} migrate_set_parameter decompress_threads 3 6. Start outgoing migration: {qemu} migrate -d tcp:destination.host:4444 {qemu} info migrate Capabilities: ... compress: on ... The following are the default settings: compress: off compress_threads: 8 decompress_threads: 2 compress_level: 1 (which means best speed) So, only the first two steps are required to use the multiple thread compression in migration. You can do more if the default settings are not appropriate. TODO ==== Some faster (de)compression method such as LZ4 and Quicklz can help to reduce the CPU consumption when doing (de)compression. If using these faster (de)compression method, less (de)compression threads are needed when doing the migration. --- ### Multiseat (`docs/multiseat.txt`) multiseat howto (with some multihead coverage) ============================================== host devices ------------ First you must compile qemu with a user interface supporting multihead/multiseat and input event routing. Right now this list includes sdl2, gtk (both 2+3) and vnc: ./configure --enable-sdl or ./configure --enable-gtk Next put together the qemu command line (sdk/gtk): qemu -accel kvm -usb $memory $disk $whatever \ -display [ sdl | gtk ] \ -vga std \ -device usb-tablet That is it for the first seat, which will use the standard vga, the standard ps/2 keyboard (implicitly there) and the usb-tablet. Now the additional switches for the second seat: -device pci-bridge,addr=12.0,chassis_nr=2,id=head.2 \ -device secondary-vga,bus=head.2,addr=02.0,id=video.2 \ -device nec-usb-xhci,bus=head.2,addr=0f.0,id=usb.2 \ -device usb-kbd,bus=usb.2.0,port=1,display=video.2 \ -device usb-tablet,bus=usb.2.0,port=2,display=video.2 This places a pci bridge in slot 12, connects a display adapter and xhci (usb) controller to the bridge. Then it adds a usb keyboard and usb mouse, both connected to the xhci and linked to the display. The "display=video2" sets up the input routing. Any input coming from the window which belongs to the video.2 display adapter will be routed to these input devices. Starting with qemu 2.4 and linux kernel 4.1 you can also use virtio for the input devices, using this ... -device pci-bridge,addr=12.0,chassis_nr=2,id=head.2 \ -device secondary-vga,bus=head.2,addr=02.0,id=video.2 \ -device virtio-keyboard-pci,bus=head.2,addr=03.0,display=video.2 \ -device virtio-tablet-pci,bus=head.2,addr=03.0,display=video.2 ... instead of xhci and usb hid devices. host ui ------- The sdl2 ui will start up with two windows, one for each display device. The gtk ui will start with a single window and each display in a separate tab. You can either simply switch tabs to switch heads, or use the "View / Detach tab" menu item to move one of the displays to its own window so you can see both display devices side-by-side. For vnc some additional configuration on the command line is needed. We'll create two vnc server instances, and bind the second one to the second seat, similar to input devices: -display vnc=:1,id=primary \ -display vnc=:2,id=secondary,display=video.2 Connecting to vnc display :1 gives you access to the first seat, and likewise connecting to vnc display :2 shows the second seat. Note on spice: Spice handles multihead just fine. But it can't do multiseat. For tablet events the event source is sent to the spice agent. But qemu can't figure it, so it can't do input routing. Fixing this needs a new or extended input interface between libspice-server and qemu. For keyboard events it is even worse: The event source isn't included in the spice protocol, so the wire protocol must be extended to support this. guest side ---------- You need a pretty recent linux guest. systemd with loginctl. kernel 3.14+ with CONFIG_DRM_BOCHS enabled. Fedora 20 will do. Must be fully updated for the new kernel though, i.e. the live iso doesn't cut it. Now we'll have to configure the guest. Boot and login. "lspci -vt" should list the pci bridge with the display adapter and usb controller: [root@fedora ~]# lspci -vt -[0000:00]-+-00.0 Intel Corporation 440FX - 82441FX PMC [Natoma] [ ... ] \-12.0-[01]--+-02.0 Device 1234:1111 \-0f.0 NEC Corporation USB 3.0 Host Controller Good. Now lets tell the system that the pci bridge and all devices below it belong to a separate seat by dropping a file into /etc/udev/rules.d: [root@fedora ~]# cat /etc/udev/rules.d/70-qemu-autoseat.rules SUBSYSTEMS=="pci", DEVPATH=="*/0000:00:12.0", TAG+="seat", ENV{ID_AUTOSEAT}="1" Reboot. System should come up with two seats. With loginctl you can check the configuration: [root@fedora ~]# loginctl list-seats SEAT seat0 seat-pci-pci-0000_00_12_0 2 seats listed. You can use "loginctl seat-status seat-pci-pci-0000_00_12_0" to list the devices attached to the seat. Background info is here: http://www.freedesktop.org/wiki/Software/systemd/multiseat/ guest side with pci-bridge-seat ------------------------------- QEMU version 2.4 and newer has a new pci-bridge-seat device which can be used instead of pci-bridge. Just swap the device name in the qemu command line above. The only difference between the two devices is the pci id. We can match the pci id instead of the device path with a nice generic rule now, which simplifies the guest configuration: [root@fedora ~]# cat /etc/udev/rules.d/70-qemu-pci-bridge-seat.rules SUBSYSTEM=="pci", ATTR{vendor}=="0x1b36", ATTR{device}=="0x000a", \ TAG+="seat", ENV{ID_AUTOSEAT}="1" Patch with this rule has been submitted to upstream udev/systemd, was accepted and should be included in the next systemd release (222). So, if your guest has this or a newer version, multiseat will work just fine without any manual guest configuration. Enjoy! -- Gerd Hoffmann --- ### Nvdimm (`docs/nvdimm.txt`) QEMU Virtual NVDIMM =================== This document explains the usage of virtual NVDIMM (vNVDIMM) feature which is available since QEMU v2.6.0. The current QEMU only implements the persistent memory mode of vNVDIMM device and not the block window mode. Basic Usage ----------- The storage of a vNVDIMM device in QEMU is provided by the memory backend (i.e. memory-backend-file and memory-backend-ram). A simple way to create a vNVDIMM device at startup time is done via the following command line options: -machine pc,nvdimm=on -m $RAM_SIZE,slots=$N,maxmem=$MAX_SIZE -object memory-backend-file,id=mem1,share=on,mem-path=$PATH,size=$NVDIMM_SIZE,readonly=off -device nvdimm,id=nvdimm1,memdev=mem1,unarmed=off Where, - the "nvdimm" machine option enables vNVDIMM feature. - "slots=$N" should be equal to or larger than the total amount of normal RAM devices and vNVDIMM devices, e.g. $N should be >= 2 here. - "maxmem=$MAX_SIZE" should be equal to or larger than the total size of normal RAM devices and vNVDIMM devices, e.g. $MAX_SIZE should be >= $RAM_SIZE + $NVDIMM_SIZE here. - "object memory-backend-file,id=mem1,share=on,mem-path=$PATH, size=$NVDIMM_SIZE,readonly=off" creates a backend storage of size $NVDIMM_SIZE on a file $PATH. All accesses to the virtual NVDIMM device go to the file $PATH. "share=on/off" controls the visibility of guest writes. If "share=on", then guest writes will be applied to the backend file. If another guest uses the same backend file with option "share=on", then above writes will be visible to it as well. If "share=off", then guest writes won't be applied to the backend file and thus will be invisible to other guests. "readonly=on/off" controls whether the file $PATH is opened read-only or read/write (default). - "device nvdimm,id=nvdimm1,memdev=mem1,unarmed=off" creates a read/write virtual NVDIMM device whose storage is provided by above memory backend device. "unarmed" controls the ACPI NFIT NVDIMM Region Mapping Structure "NVDIMM State Flags" Bit 3 indicating that the device is "unarmed" and cannot accept persistent writes. Linux guest drivers set the device to read-only when this bit is present. Set unarmed to on when the memdev has readonly=on. Multiple vNVDIMM devices can be created if multiple pairs of "-object" and "-device" are provided. For above command line options, if the guest OS has the proper NVDIMM driver (e.g. "CONFIG_ACPI_NFIT=y" under Linux), it should be able to detect a NVDIMM device which is in the persistent memory mode and whose size is $NVDIMM_SIZE. Note: 1. Prior to QEMU v2.8.0, if memory-backend-file is used and the actual backend file size is not equal to the size given by "size" option, QEMU will truncate the backend file by ftruncate(2), which will corrupt the existing data in the backend file, especially for the shrink case. QEMU v2.8.0 and later check the backend file size and the "size" option. If they do not match, QEMU will report errors and abort in order to avoid the data corruption. 2. QEMU v2.6.0 only puts a basic alignment requirement on the "size" option of memory-backend-file, e.g. 4KB alignment on x86. However, QEMU v.2.7.0 puts an additional alignment requirement, which may require a larger value than the basic one, e.g. 2MB on x86. This change breaks the usage of memory-backend-file that only satisfies the basic alignment. QEMU v2.8.0 and later remove the additional alignment on non-s390x architectures, so the broken memory-backend-file can work again. Label ----- QEMU v2.7.0 and later implement the label support for vNVDIMM devices. To enable label on vNVDIMM devices, users can simply add "label-size=$SZ" option to "-device nvdimm", e.g. -device nvdimm,id=nvdimm1,memdev=mem1,label-size=128K Note: 1. The minimal label size is 128KB. 2. QEMU v2.7.0 and later store labels at the end of backend storage. If a memory backend file, which was previously used as the backend of a vNVDIMM device without labels, is now used for a vNVDIMM device with label, the data in the label area at the end of file will be inaccessible to the guest. If any useful data (e.g. the meta-data of the file system) was stored there, the latter usage may result guest data corruption (e.g. breakage of guest file system). Hotplug ------- QEMU v2.8.0 and later implement the hotplug support for vNVDIMM devices. Similarly to the RAM hotplug, the vNVDIMM hotplug is accomplished by two monitor commands "object_add" and "device_add". For example, the following commands add another 4GB vNVDIMM device to the guest: (qemu) object_add memory-backend-file,id=mem2,share=on,mem-path=new_nvdimm.img,size=4G (qemu) device_add nvdimm,id=nvdimm2,memdev=mem2 Note: 1. Each hotplugged vNVDIMM device consumes one memory slot. Users should always ensure the memory option "-m ...,slots=N" specifies enough number of slots, i.e. N >= number of RAM devices + number of statically plugged vNVDIMM devices + number of hotplugged vNVDIMM devices 2. The similar is required for the memory option "-m ...,maxmem=M", i.e. M >= size of RAM devices + size of statically plugged vNVDIMM devices + size of hotplugged vNVDIMM devices Alignment --------- QEMU uses mmap(2) to maps vNVDIMM backends and aligns the mapping address to the page size (getpagesize(2)) by default. However, some types of backends may require an alignment different than the page size. In that case, QEMU v2.12.0 and later provide 'align' option to memory-backend-file to allow users to specify the proper alignment. For device dax (e.g., /dev/dax0.0), this alignment needs to match the alignment requirement of the device dax. The NUM of 'align=NUM' option must be larger than or equal to the 'align' of device dax. We can use one of the following commands to show the 'align' of device dax. ndctl list -X daxctl list -R In order to get the proper 'align' of device dax, you need to install the library 'libdaxctl'. For example, device dax require the 2 MB alignment, so we can use following QEMU command line options to use it (/dev/dax0.0) as the backend of vNVDIMM: -object memory-backend-file,id=mem1,share=on,mem-path=/dev/dax0.0,size=4G,align=2M -device nvdimm,id=nvdimm1,memdev=mem1 Guest Data Persistence ---------------------- Though QEMU supports multiple types of vNVDIMM backends on Linux, the only backend that can guarantee the guest write persistence is: A. DAX device (e.g., /dev/dax0.0, ) or B. DAX file(mounted with dax option) When using B (A file supporting direct mapping of persistent memory) as a backend, write persistence is guaranteed if the host kernel has support for the MAP_SYNC flag in the mmap system call (available since Linux 4.15 and on certain distro kernels) and additionally both 'pmem' and 'share' flags are set to 'on' on the backend. If these conditions are not satisfied i.e. if either 'pmem' or 'share' are not set, if the backend file does not support DAX or if MAP_SYNC is not supported by the host kernel, write persistence is not guaranteed after a system crash. For compatibility reasons, these conditions are ignored if not satisfied. Currently, no way is provided to test for them. For more details, please reference mmap(2) man page: http://man7.org/linux/man-pages/man2/mmap.2.html. When using other types of backends, it's suggested to set 'unarmed' option of '-device nvdimm' to 'on', which sets the unarmed flag of the guest NVDIMM region mapping structure. This unarmed flag indicates guest software that this vNVDIMM device contains a region that cannot accept persistent writes. In result, for example, the guest Linux NVDIMM driver, marks such vNVDIMM device as read-only. Backend File Setup Example -------------------------- Here are two examples showing how to setup these persistent backends on linux using the tool ndctl [3]. A. DAX device Use the following command to set up /dev/dax0.0 so that the entirety of namespace0.0 can be exposed as an emulated NVDIMM to the guest: ndctl create-namespace -f -e namespace0.0 -m devdax The /dev/dax0.0 could be used directly in "mem-path" option. B. DAX file Individual files on a DAX host file system can be exposed as emulated NVDIMMS. First an fsdax block device is created, partitioned, and then mounted with the "dax" mount option: ndctl create-namespace -f -e namespace0.0 -m fsdax (partition /dev/pmem0 with name pmem0p1) mount -o dax /dev/pmem0p1 /mnt (create or copy a disk image file with qemu-img(1), cp(1), or dd(1) in /mnt) Then the new file in /mnt could be used in "mem-path" option. NVDIMM Persistence ------------------ ACPI 6.2 Errata A added support for a new Platform Capabilities Structure which allows the platform to communicate what features it supports related to NVDIMM data persistence. Users can provide a persistence value to a guest via the optional "nvdimm-persistence" machine command line option: -machine pc,accel=kvm,nvdimm,nvdimm-persistence=cpu There are currently two valid values for this option: "mem-ctrl" - The platform supports flushing dirty data from the memory controller to the NVDIMMs in the event of power loss. "cpu" - The platform supports flushing dirty data from the CPU cache to the NVDIMMs in the event of power loss. This implies that the platform also supports flushing dirty data through the memory controller on power loss. If the vNVDIMM backend is in host persistent memory that can be accessed in SNIA NVM Programming Model [1] (e.g., Intel NVDIMM), it's suggested to set the 'pmem' option of memory-backend-file to 'on'. When 'pmem' is 'on' and QEMU is built with libpmem [2] support (configured with --enable-libpmem), QEMU will take necessary operations to guarantee the persistence of its own writes to the vNVDIMM backend(e.g., in vNVDIMM label emulation and live migration). If 'pmem' is 'on' while there is no libpmem support, qemu will exit and report a "lack of libpmem support" message to ensure the persistence is available. For example, if we want to ensure the persistence for some backend file, use the QEMU command line: -object memory-backend-file,id=nv_mem,mem-path=/XXX/yyy,size=4G,pmem=on References ---------- [1] NVM Programming Model (NPM) Version 1.2 https://www.snia.org/sites/default/files/technical_work/final/NVMProgrammingModel_v1.2.pdf [2] Persistent Memory Development Kit (PMDK), formerly known as NVML project, home page: http://pmem.io/pmdk/ [3] ndctl-create-namespace - provision or reconfigure a namespace http://pmem.io/ndctl/ndctl-create-namespace.html --- ### Pci Expander Bridge (`docs/pci_expander_bridge.txt`) PCI EXPANDER BRIDGE (PXB) ========================= Description =========== PXB is a "light-weight" host bridge in the same PCI domain as the main host bridge whose purpose is to enable the main host bridge to support multiple PCI root buses. It is implemented only for i440fx and can be placed only on bus 0 (pci.0). As opposed to PCI-2-PCI bridge's secondary bus, PXB's bus is a primary bus and can be associated with a NUMA node (different from the main host bridge) allowing the guest OS to recognize the proximity of a pass-through device to other resources as RAM and CPUs. Usage ===== A detailed command line would be: [qemu-bin + storage options] -m 2G -object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0 -numa node,nodeid=0,cpus=0,memdev=ram-node0 -object memory-backend-ram,size=1024M,policy=bind,host-nodes=1,id=ram-node1 -numa node,nodeid=1,cpus=1,memdev=ram-node1 -device pxb,id=bridge1,bus=pci.0,numa_node=1,bus_nr=4 -netdev user,id=nd -device e1000,bus=bridge1,addr=0x4,netdev=nd -device pxb,id=bridge2,bus=pci.0,numa_node=0,bus_nr=8 -device e1000,bus=bridge2,addr=0x3 -device pxb,id=bridge3,bus=pci.0,bus_nr=40 -drive if=none,id=drive0,file=[img] -device virtio-blk-pci,drive=drive0,scsi=off,bus=bridge3,addr=1 Here you have: - 2 NUMA nodes for the guest, 0 and 1. (both mapped to the same NUMA node in host, but you can and should put it in different host NUMA nodes) - a pxb host bridge attached to NUMA 1 with an e1000 behind it - a pxb host bridge attached to NUMA 0 with an e1000 behind it - a pxb host bridge not attached to any NUMA with a hard drive behind it. Limitations =========== Please observe that we specified the bus "pci.0" for the second and third pxb. This is because when no bus is given, another pxb can be selected by QEMU as default bus, however, PXBs can be placed only under the root bus. Implementation ============== The PXB is composed by: - HostBridge (TYPE_PXB_HOST) The host bridge allows to register and query the PXB's PCI root bus in QEMU. - PXBDev(TYPE_PXB_DEVICE) It is a regular PCI Device that resides on the piix host-bridge bus and its bus uses the same PCI domain. However, the bus behind is exposed through ACPI as a primary PCI bus and starts a new PCI hierarchy. The interrupts from devices behind the PXB are routed through this device the same as if it were a PCI-2-PCI bridge. The _PRT follows the i440fx model. - PCIBridgeDev(TYPE_PCI_BRIDGE_DEV) Created automatically as part of init sequence. When adding a device to PXB it is attached to the bridge for two reasons: - Using the bridge will enable hotplug support - All the devices behind the bridge will use bridge's IO/MEM windows compacting the PCI address space. --- ### Pcie (`docs/pcie.txt`) PCI EXPRESS GUIDELINES ====================== 1. Introduction ================ The doc proposes best practices on how to use PCI Express (PCIe) / PCI devices in PCI Express based machines and explains the reasoning behind them. Note that the PCIe features are available only when using the 'q35' machine type on x86 architecture and the 'virt' machine type on AArch64. Other machine types do not use PCIe at this time. The following presentations accompany this document: (1) Q35 overview. https://wiki.qemu.org/images/4/4e/Q35.pdf (2) A comparison between PCI and PCI Express technologies. https://wiki.qemu.org/images/f/f6/PCIvsPCIe.pdf Note: The usage examples are not intended to replace the full documentation, please use QEMU help to retrieve all options. 2. Device placement strategy ============================ QEMU does not have a clear socket-device matching mechanism and allows any PCI/PCI Express device to be plugged into any PCI/PCI Express slot. Plugging a PCI device into a PCI Express slot might not always work and is weird anyway since it cannot be done for "bare metal". Plugging a PCI Express device into a PCI slot will hide the Extended Configuration Space thus is also not recommended. The recommendation is to separate the PCI Express and PCI hierarchies. PCI Express devices should be plugged only into PCI Express Root Ports and PCI Express Downstream ports. 2.1 Root Bus (pcie.0) ===================== Place only the following kinds of devices directly on the Root Complex: (1) PCI Devices (e.g. network card, graphics card, IDE controller), not controllers. Place only legacy PCI devices on the Root Complex. These will be considered Integrated Endpoints. Note: Integrated Endpoints are not hot-pluggable. Although the PCI Express spec does not forbid PCI Express devices as Integrated Endpoints, existing hardware mostly integrates legacy PCI devices with the Root Complex. Guest OSes are suspected to behave strangely when PCI Express devices are integrated with the Root Complex. (2) PCI Express Root Ports (ioh3420), for starting exclusively PCI Express hierarchies. (3) PCI Express to PCI Bridge (pcie-pci-bridge), for starting legacy PCI hierarchies. (4) Extra Root Complexes (pxb-pcie), if multiple PCI Express Root Buses are needed. pcie.0 bus ---------------------------------------------------------------------------- | | | | ----------- ------------------ ------------------- -------------- | PCI Dev | | PCIe Root Port | | PCIe-PCI Bridge | | pxb-pcie | ----------- ------------------ ------------------- -------------- 2.1.1 To plug a device into pcie.0 as a Root Complex Integrated Endpoint use: -device [,bus=pcie.0] 2.1.2 To expose a new PCI Express Root Bus use: -device pxb-pcie,id=pcie.1,bus_nr=x[,numa_node=y][,addr=z] PCI Express Root Ports and PCI Express to PCI bridges can be connected to the pcie.1 bus: -device ioh3420,id=root_port1[,bus=pcie.1][,chassis=x][,slot=y][,addr=z] \ -device pcie-pci-bridge,id=pcie_pci_bridge1,bus=pcie.1 2.2 PCI Express only hierarchy ============================== Always use PCI Express Root Ports to start PCI Express hierarchies. A PCI Express Root bus supports up to 32 devices. Since each PCI Express Root Port is a function and a multi-function device may support up to 8 functions, the maximum possible number of PCI Express Root Ports per PCI Express Root Bus is 256. Prefer grouping PCI Express Root Ports into multi-function devices to keep a simple flat hierarchy that is enough for most scenarios. Only use PCI Express Switches (x3130-upstream, xio3130-downstream) if there is no more room for PCI Express Root Ports. Please see section 4. for further justifications. Plug only PCI Express devices into PCI Express Ports. pcie.0 bus ---------------------------------------------------------------------------------- | | | ------------- ------------- ------------- | Root Port | | Root Port | | Root Port | ------------ ------------- ------------- | -------------------------|------------------------ ------------ | ----------------- | | PCIe Dev | | PCI Express | Upstream Port | | ------------ | Switch ----------------- | | | | | | ------------------- ------------------- | | | Downstream Port | | Downstream Port | | | ------------------- ------------------- | -------------|-----------------------|------------ ------------ | PCIe Dev | ------------ 2.2.1 Plugging a PCI Express device into a PCI Express Root Port: -device ioh3420,id=root_port1,chassis=x,slot=y[,bus=pcie.0][,addr=z] \ -device ,bus=root_port1 2.2.2 Using multi-function PCI Express Root Ports: -device ioh3420,id=root_port1,multifunction=on,chassis=x,addr=z.0[,slot=y][,bus=pcie.0] \ -device ioh3420,id=root_port2,chassis=x1,addr=z.1[,slot=y1][,bus=pcie.0] \ -device ioh3420,id=root_port3,chassis=x2,addr=z.2[,slot=y2][,bus=pcie.0] \ 2.2.3 Plugging a PCI Express device into a Switch: -device ioh3420,id=root_port1,chassis=x,slot=y[,bus=pcie.0][,addr=z] \ -device x3130-upstream,id=upstream_port1,bus=root_port1[,addr=x] \ -device xio3130-downstream,id=downstream_port1,bus=upstream_port1,chassis=x1,slot=y1[,addr=z1]] \ -device ,bus=downstream_port1 Notes: - (slot, chassis) pair is mandatory and must be unique for each PCI Express Root Port. slot defaults to 0 when not specified. - 'addr' parameter can be 0 for all the examples above. 2.3 PCI only hierarchy ====================== Legacy PCI devices can be plugged into pcie.0 as Integrated Endpoints, but, as mentioned in section 5, doing so means the legacy PCI device in question will be incapable of hot-unplugging. Besides that use PCI Express to PCI Bridges (pcie-pci-bridge) in combination with PCI-PCI Bridges (pci-bridge) to start PCI hierarchies. Prefer flat hierarchies. For most scenarios a single PCI Express to PCI Bridge (having 32 slots) and several PCI-PCI Bridges attached to it (each supporting also 32 slots) will support hundreds of legacy devices. The recommendation is to populate one PCI-PCI Bridge under the PCI Express to PCI Bridge until is full and then plug a new PCI-PCI Bridge... pcie.0 bus ---------------------------------------------- | | ----------- ------------------- | PCI Dev | | PCIe-PCI Bridge | ----------- ------------------- | | ------------------ ------------------ | PCI-PCI Bridge | | PCI-PCI Bridge | ------------------ ------------------ | | ----------- ----------- | PCI Dev | | PCI Dev | ----------- ----------- 2.3.1 To plug a PCI device into pcie.0 as an Integrated Endpoint use: -device [,bus=pcie.0] 2.3.2 Plugging a PCI device into a PCI-PCI Bridge: -device pcie-pci-bridge,id=pcie_pci_bridge1[,bus=pcie.0] \ -device pci-bridge,id=pci_bridge1,bus=pcie_pci_bridge1[,chassis_nr=x][,addr=y] \ -device ,bus=pci_bridge1[,addr=x] Note that 'addr' cannot be 0 unless shpc=off parameter is passed to the PCI Bridge/PCI Express to PCI Bridge. 3. IO space issues =================== The PCI Express Root Ports and PCI Express Downstream ports are seen by Firmware/Guest OS as PCI-PCI Bridges. As required by the PCI spec, each such Port should be reserved a 4K IO range for, even though only one (multifunction) device can be plugged into each Port. This results in poor IO space utilization. The firmware used by QEMU (SeaBIOS/OVMF) may try further optimizations by not allocating IO space for each PCI Express Root / PCI Express Downstream port if: (1) the port is empty, or (2) the device behind the port has no IO BARs. The IO space is very limited, to 65536 byte-wide IO ports, and may even be fragmented by fixed IO ports owned by platform devices resulting in at most 10 PCI Express Root Ports or PCI Express Downstream Ports per system if devices with IO BARs are used in the PCI Express hierarchy. Using the proposed device placing strategy solves this issue by using only PCI Express devices within PCI Express hierarchy. The PCI Express spec requires that PCI Express devices work properly without using IO ports. The PCI hierarchy has no such limitations. 4. Bus numbers issues ====================== Each PCI domain can have up to only 256 buses and the QEMU PCI Express machines do not support multiple PCI domains even if extra Root Complexes (pxb-pcie) are used. Each element of the PCI Express hierarchy (Root Complexes, PCI Express Root Ports, PCI Express Downstream/Upstream ports) uses one bus number. Since only one (multifunction) device can be attached to a PCI Express Root Port or PCI Express Downstream Port it is advised to plan in advance for the expected number of devices to prevent bus number starvation. Avoiding PCI Express Switches (and thereby striving for a 'flatter' PCI Express hierarchy) enables the hierarchy to not spend bus numbers on Upstream Ports. The bus_nr properties of the pxb-pcie devices partition the 0..255 bus number space. All bus numbers assigned to the buses recursively behind a given pxb-pcie device's root bus must fit between the bus_nr property of that pxb-pcie device, and the lowest of the higher bus_nr properties that the command line sets for other pxb-pcie devices. 5. Hot-plug ============ The PCI Express root buses (pcie.0 and the buses exposed by pxb-pcie devices) do not support hot-plug, so any devices plugged into Root Complexes cannot be hot-plugged/hot-unplugged: (1) PCI Express Integrated Endpoints (2) PCI Express Root Ports (3) PCI Express to PCI Bridges (4) pxb-pcie Be aware that PCI Express Downstream Ports can't be hot-plugged into an existing PCI Express Upstream Port. PCI devices can be hot-plugged into PCI Express to PCI and PCI-PCI Bridges. The PCI hot-plug into PCI-PCI bridge is ACPI based, whereas hot-plug into PCI Express to PCI bridges is SHPC-based. They both can work side by side with the PCI Express native hot-plug. PCI Express devices can be natively hot-plugged/hot-unplugged into/from PCI Express Root Ports (and PCI Express Downstream Ports). 5.1 Planning for hot-plug: (1) PCI hierarchy Leave enough PCI-PCI Bridge slots empty or add one or more empty PCI-PCI Bridges to the PCI Express to PCI Bridge. For each such PCI-PCI Bridge the Guest Firmware is expected to reserve 4K IO space and 2M MMIO range to be used for all devices behind it. Appropriate PCI capability is designed, see pcie_pci_bridge.txt. Because of the hard IO limit of around 10 PCI Bridges (~ 40K space) per system don't use more than 9 PCI-PCI Bridges, leaving 4K for the Integrated Endpoints. (The PCI Express Hierarchy needs no IO space). (2) PCI Express hierarchy: Leave enough PCI Express Root Ports empty. Use multifunction PCI Express Root Ports (up to 8 ports per pcie.0 slot) on the Root Complex(es), for keeping the hierarchy as flat as possible, thereby saving PCI bus numbers. Don't use PCI Express Switches if you don't have to, each one of those uses an extra PCI bus (for its Upstream Port) that could be put to better use with another Root Port or Downstream Port, which may come handy for hot-plugging another device. 5.3 Hot-plug example: Using HMP: (add -monitor stdio to QEMU command line) device_add ,id=,bus= 6. Device assignment ==================== Host devices are mostly PCI Express and should be plugged only into PCI Express Root Ports or PCI Express Downstream Ports. PCI-PCI Bridge slots can be used for legacy PCI host devices. 6.1 How to detect if a device is PCI Express: > lspci -s 03:00.0 -v (as root) 03:00.0 Network controller: Intel Corporation Wireless 7260 (rev 83) Subsystem: Intel Corporation Dual Band Wireless-AC 7260 Flags: bus master, fast devsel, latency 0, IRQ 50 Memory at f0400000 (64-bit, non-prefetchable) [size=8K] Capabilities: [c8] Power Management version 3 Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [40] Express Endpoint, MSI 00 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 7c-7a-91-ff-ff-90-db-20 Capabilities: [14c] Latency Tolerance Reporting Capabilities: [154] Vendor Specific Information: ID=cafe Rev=1 Len=014 If you can see the "Express Endpoint" capability in the output, then the device is indeed PCI Express. 7. Virtio devices ================= Virtio devices plugged into the PCI hierarchy or as Integrated Endpoints will remain PCI and have transitional behaviour as default. Transitional virtio devices work in both IO and MMIO modes depending on the guest support. The Guest firmware will assign both IO and MMIO resources to transitional virtio devices. Virtio devices plugged into PCI Express ports are PCI Express devices and have "1.0" behavior by default without IO support. In both cases disable-legacy and disable-modern properties can be used to override the behaviour. Note that setting disable-legacy=off will enable legacy mode (enabling legacy behavior) for PCI Express virtio devices causing them to require IO space, which, given the limited available IO space, may quickly lead to resource exhaustion, and is therefore strongly discouraged. 8. Conclusion ============== The proposal offers a usage model that is easy to understand and follow and at the same time overcomes the PCI Express architecture limitations. --- ### Pcie Pci Bridge (`docs/pcie_pci_bridge.txt`) Generic PCI Express to PCI Bridge ================================ Description =========== PCIE-to-PCI bridge is a new method for legacy PCI hierarchies creation on Q35 machines. Previously Intel DMI-to-PCI bridge was used for this purpose. But due to its strict limitations - no support of hot-plug, no cross-platform and cross-architecture support - a new generic PCIE-to-PCI bridge should now be used for any legacy PCI device usage with PCI Express machine. This generic PCIE-PCI bridge is a cross-platform device, can be hot-plugged into appropriate root port (requires additional actions, see 'PCIE-PCI bridge hot-plug' section), and supports devices hot-plug into the bridge itself (with some limitations, see below). Hot-plug of legacy PCI devices into the bridge is provided by bridge's built-in Standard hot-plug Controller. Though it still has some limitations, see below. PCIE-PCI bridge hot-plug ======================= Guest OSes require extra efforts to enable PCIE-PCI bridge hot-plug. Motivation - now on init any PCI Express root port which doesn't have any device plugged in, has no free buses reserved to provide any of them to a hot-plugged devices in future. To solve this problem we reserve additional buses on a firmware level. Currently only SeaBIOS is supported. The way of bus number to reserve delivery is special Red Hat vendor-specific PCI capability, added to the root port that is planned to have PCIE-PCI bridge hot-plugged in. Capability layout (defined in include/hw/pci/pci_bridge.h): uint8_t id; Standard PCI capability header field uint8_t next; Standard PCI capability header field uint8_t len; Standard PCI vendor-specific capability header field uint8_t type; Red Hat vendor-specific capability type List of currently existing types: RESOURCE_RESERVE = 1 uint32_t bus_res; Minimum number of buses to reserve uint64_t io; IO space to reserve uint32_t mem Non-prefetchable memory to reserve At most one of the following two fields may be set to a value different from -1: uint32_t mem_pref_32; Prefetchable memory to reserve (32-bit MMIO) uint64_t mem_pref_64; Prefetchable memory to reserve (64-bit MMIO) If any reservation field is -1 then this kind of reservation is not needed and must be ignored by firmware. At the moment this capability is used only in QEMU generic PCIe root port (-device pcie-root-port). Capability construction function takes all reservation fields values from corresponding device properties. By default all of them are set to -1 to leave root port's default behavior unchanged. Usage ===== A detailed command line would be: [qemu-bin + storage options] \ -m 2G \ -device pcie-root-port,bus=pcie.0,id=rp1,slot=1 \ -device pcie-root-port,bus=pcie.0,id=rp2,slot=2 \ -device pcie-root-port,bus=pcie.0,id=rp3,slot=3,bus-reserve=1 \ -device pcie-pci-bridge,id=br1,bus=rp1 \ -device pcie-pci-bridge,id=br2,bus=rp2 \ -device e1000,bus=br1,addr=8 Then in monitor it's OK to execute next commands: device_add pcie-pci-bridge,id=br3,bus=rp3 \ device_add e1000,bus=br2,addr=1 \ device_add e1000,bus=br3,addr=1 Here you have: (1) Cold-plugged: - Root ports: 1 QEMU generic root port with the capability mentioned above, 2 QEMU generic root ports without this capability; - 2 PCIE-PCI bridges plugged into 2 different root ports; - e1000 plugged into the first bridge. (2) Hot-plugged: - PCIE-PCI bridge, plugged into QEMU generic root port; - 2 e1000 cards, one plugged into the cold-plugged PCIE-PCI bridge, another plugged into the hot-plugged bridge. Limitations =========== The PCIE-PCI bridge can be hot-plugged only into pcie-root-port that has proper 'bus-reserve' property value to provide secondary bus for the hot-plugged bridge. Windows 7 and older versions don't support hot-plug devices into the PCIE-PCI bridge. To enable device hot-plug into the bridge on Linux there're 3 ways: 1) Build shpchp module with this patch http://www.spinics.net/lists/linux-pci/msg63052.html 2) Use kernel 4.14+ where the patch mentioned above is already merged. 3) Set 'msi' property to off - this forces the bridge to use legacy INTx, which allows the bridge to notify the OS about hot-plug event without having BUSMASTER set. Implementation ============== The PCIE-PCI bridge is based on PCI-PCI bridge, but also accumulates PCI Express features as a PCI Express device. --- ### Pcie Sriov (`docs/pcie_sriov.txt`) PCI SR/IOV EMULATION SUPPORT ============================ Description =========== SR/IOV (Single Root I/O Virtualization) is an optional extended capability of a PCI Express device. It allows a single physical function (PF) to appear as multiple virtual functions (VFs) for the main purpose of eliminating software overhead in I/O from virtual machines. QEMU now implements the basic common functionality to enable an emulated device to support SR/IOV. Yet no fully implemented devices exists in QEMU, but a proof-of-concept hack of the Intel igb can be found here: git://github.com/knuto/qemu.git sriov_patches_v5 Implementation ============== Implementing emulation of an SR/IOV capable device typically consists of implementing support for two types of device classes; the "normal" physical device (PF) and the virtual device (VF). From QEMU's perspective, the VFs are just like other devices, except that some of their properties are derived from the PF. A virtual function is different from a physical function in that the BAR space for all VFs are defined by the BAR registers in the PFs SR/IOV capability. All VFs have the same BARs and BAR sizes. Accesses to these virtual BARs then is computed as + * + From our emulation perspective this means that there is a separate call for setting up a BAR for a VF. 1) To enable SR/IOV support in the PF, it must be a PCI Express device so you would need to add a PCI Express capability in the normal PCI capability list. You might also want to add an ARI (Alternative Routing-ID Interpretation) capability to indicate that your device supports functions beyond it's "own" function space (0-7), which is necessary to support more than 7 functions, or if functions extends beyond offset 7 because they are placed at an offset > 1 or have stride > 1. ... #include "hw/pci/pcie.h" #include "hw/pci/pcie_sriov.h" pci_your_pf_dev_realize( ... ) { ... int ret = pcie_endpoint_cap_init(d, 0x70); ... pcie_ari_init(d, 0x100, 1); ... /* Add and initialize the SR/IOV capability */ pcie_sriov_pf_init(d, 0x200, "your_virtual_dev", vf_devid, initial_vfs, total_vfs, fun_offset, stride); /* Set up individual VF BARs (parameters as for normal BARs) */ pcie_sriov_pf_init_vf_bar( ... ) ... } For cleanup, you simply call: pcie_sriov_pf_exit(device); which will delete all the virtual functions and associated resources. 2) Similarly in the implementation of the virtual function, you need to make it a PCI Express device and add a similar set of capabilities except for the SR/IOV capability. Then you need to set up the VF BARs as subregions of the PFs SR/IOV VF BARs by calling pcie_sriov_vf_register_bar() instead of the normal pci_register_bar() call: pci_your_vf_dev_realize( ... ) { ... int ret = pcie_endpoint_cap_init(d, 0x60); ... pcie_ari_init(d, 0x100, 1); ... memory_region_init(mr, ... ) pcie_sriov_vf_register_bar(d, bar_nr, mr); ... } Testing on Linux guest ====================== The easiest is if your device driver supports sysfs based SR/IOV enabling. Support for this was added in kernel v.3.8, so not all drivers support it yet. To enable 4 VFs for a device at 01:00.0: modprobe yourdriver echo 4 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs You should now see 4 VFs with lspci. To turn SR/IOV off again - the standard requires you to turn it off before you can enable another VF count, and the emulation enforces this: echo 0 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs Older drivers typically provide a max_vfs module parameter to enable it at load time: modprobe yourdriver max_vfs=4 To disable the VFs again then, you simply have to unload the driver: rmmod yourdriver --- ### Pvrdma (`docs/pvrdma.txt`) Paravirtualized RDMA Device (PVRDMA) ==================================== 1. Description =============== PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications. While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines as peers. It does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe). It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance. A project presentation accompany this document: - https://blog.linuxplumbersconf.org/2017/ocw/system/presentations/4730/original/lpc-2017-pvrdma-marcel-apfelbaum-yuval-shaia.pdf 2. Setup ======== 2.1 Guest setup =============== Fedora 27+ kernels work out of the box, older distributions require updating the kernel to 4.14 to include the pvrdma driver. However the libpvrdma library needed by User Level Software is still not available as part of the distributions, so the rdma-core library needs to be compiled and optionally installed. Please follow the instructions at: https://github.com/linux-rdma/rdma-core.git 2.2 Host Setup ============== The pvrdma backend is an ibdevice interface that can be exposed either by a Soft-RoCE(rxe) device on machines with no RDMA device, or an HCA SRIOV function(VF/PF). Note that ibdevice interfaces can't be shared between pvrdma devices, each one requiring a separate instance (rxe or SRIOV VF). 2.2.1 Soft-RoCE backend(rxe) =========================== A stable version of rxe is required, Fedora 27+ or a Linux Kernel 4.14+ is preferred. The rdma_rxe module is part of the Linux Kernel but not loaded by default. Install the User Level library (librxe) following the instructions from: https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home Associate an ETH interface with rxe by running: rxe_cfg add eth0 An rxe0 ibdevice interface will be created and can be used as pvrdma backend. 2.2.2 RDMA device Virtual Function backend ========================================== Nothing special is required, the pvrdma device can work not only with Ethernet Links, but also Infinibands Links. All is needed is an ibdevice with an active port, for Mellanox cards will be something like mlx5_6 which can be the backend. 2.2.3 QEMU setup ================ Configure QEMU with --enable-rdma flag, installing the required RDMA libraries. 3. Usage ======== 3.1 VM Memory settings ====================== Currently the device is working only with memory backed RAM and it must be mark as "shared": -m 1G \ -object memory-backend-ram,id=mb1,size=1G,share \ -numa node,memdev=mb1 \ 3.2 MAD Multiplexer =================== MAD Multiplexer is a service that exposes MAD-like interface for VMs in order to overcome the limitation where only single entity can register with MAD layer to send and receive RDMA-CM MAD packets. To build rdmacm-mux run # make rdmacm-mux Before running the rdmacm-mux make sure that both ib_cm and rdma_cm kernel modules aren't loaded, otherwise the rdmacm-mux service will fail to start. The application accepts 3 command line arguments and exposes a UNIX socket to pass control and data to it. -d rdma-device-name Name of RDMA device to register with -s unix-socket-path Path to unix socket to listen (default /var/run/rdmacm-mux) -p rdma-device-port Port number of RDMA device to register with (default 1) The final UNIX socket file name is a concatenation of the 3 arguments so for example for device mlx5_0 on port 2 this /var/run/rdmacm-mux-mlx5_0-2 will be created. pvrdma requires this service. Please refer to contrib/rdmacm-mux for more details. 3.3 Service exposed by libvirt daemon ===================================== The control over the RDMA device's GID table is done by updating the device's Ethernet function addresses. Usually the first GID entry is determined by the MAC address, the second by the first IPv6 address and the third by the IPv4 address. Other entries can be added by adding more IP addresses. The opposite is the same, i.e. whenever an address is removed, the corresponding GID entry is removed. The process is done by the network and RDMA stacks. Whenever an address is added the ib_core driver is notified and calls the device driver add_gid function which in turn update the device. To support this in pvrdma device the device hooks into the create_bind and destroy_bind HW commands triggered by pvrdma driver in guest. Whenever changed is made to the pvrdma port's GID table a special QMP messages is sent to be processed by libvirt to update the address of the backend Ethernet device. pvrdma requires that libvirt service will be up. 3.4 PCI devices settings ======================== RoCE device exposes two functions - an Ethernet and RDMA. To support it, pvrdma device is composed of two PCI functions, an Ethernet device of type vmxnet3 on PCI slot 0 and a PVRDMA device on PCI slot 1. The Ethernet function can be used for other Ethernet purposes such as IP. 3.5 Device parameters ===================== - netdev: Specifies the Ethernet device function name on the host for example enp175s0f0. For Soft-RoCE device (rxe) this would be the Ethernet device used to create it. - ibdev: The IB device name on host for example rxe0, mlx5_0 etc. - mad-chardev: The name of the MAD multiplexer char device. - ibport: In case of multi-port device (such as Mellanox's HCA) this specify the port to use. If not set 1 will be used. - dev-caps-max-mr-size: The maximum size of MR. - dev-caps-max-qp: Maximum number of QPs. - dev-caps-max-cq: Maximum number of CQs. - dev-caps-max-mr: Maximum number of MRs. - dev-caps-max-pd: Maximum number of PDs. - dev-caps-max-ah: Maximum number of AHs. Notes: - The first 3 parameters are mandatory settings, the rest have their defaults. - The last 8 parameters (the ones that prefixed by dev-caps) defines the top limits but the final values is adjusted by the backend device limitations. - netdev can be extracted from ibdev's sysfs (/sys/class/infiniband//device/net/) 3.6 Example =========== Define bridge device with vmxnet3 network backend:
Define pvrdma device: 4. Implementation details ========================= 4.1 Overview ============ The device acts like a proxy between the Guest Driver and the host ibdevice interface. On configuration path: - For every hardware resource request (PD/QP/CQ/...) the pvrdma will request a resource from the backend interface, maintaining a 1-1 mapping between the guest and host. On data path: - Every post_send/receive received from the guest will be converted into a post_send/receive for the backend. The buffers data will not be touched or copied resulting in near bare-metal performance for large enough buffers. - Completions from the backend interface will result in completions for the pvrdma device. 4.2 PCI BARs ============ PCI Bars: BAR 0 - MSI-X MSI-X vectors: (0) Command - used when execution of a command is completed. (1) Async - not in use. (2) Completion - used when a completion event is placed in device's CQ ring. BAR 1 - Registers -------------------------------------------------------- | VERSION | DSR | CTL | REQ | ERR | ICR | IMR | MAC | -------------------------------------------------------- DSR - Address of driver/device shared memory used for the command channel, used for passing: - General info such as driver version - Address of 'command' and 'response' - Address of async ring - Address of device's CQ ring - Device capabilities CTL - Device control operations (activate, reset etc) IMG - Set interrupt mask REQ - Command execution register ERR - Operation status BAR 2 - UAR --------------------------------------------------------- | QP_NUM | SEND/RECV Flag || CQ_NUM | ARM/POLL Flag | --------------------------------------------------------- - Offset 0 used for QP operations (send and recv) - Offset 4 used for CQ operations (arm and poll) 4.3 Major flows =============== 4.3.1 Create CQ =============== - Guest driver - Allocates pages for CQ ring - Creates page directory (pdir) to hold CQ ring's pages - Initializes CQ ring - Initializes 'Create CQ' command object (cqe, pdir etc) - Copies the command to 'command' address - Writes 0 into REQ register - Device - Reads the request object from the 'command' address - Allocates CQ object and initialize CQ ring based on pdir - Creates the backend CQ - Writes operation status to ERR register - Posts command-interrupt to guest - Guest driver - Reads the HW response code from ERR register 4.3.2 Create QP =============== - Guest driver - Allocates pages for send and receive rings - Creates page directory(pdir) to hold the ring's pages - Initializes 'Create QP' command object (max_send_wr, send_cq_handle, recv_cq_handle, pdir etc) - Copies the object to 'command' address - Write 0 into REQ register - Device - Reads the request object from 'command' address - Allocates the QP object and initialize - Send and recv rings based on pdir - Send and recv ring state - Creates the backend QP - Writes the operation status to ERR register - Posts command-interrupt to guest - Guest driver - Reads the HW response code from ERR register 4.3.3 Post receive ================== - Guest driver - Initializes a wqe and place it on recv ring - Write to qpn|qp_recv_bit (31) to QP offset in UAR - Device - Extracts qpn from UAR - Walks through the ring and does the following for each wqe - Prepares the backend CQE context to be used when receiving completion from backend (wr_id, op_code, emu_cq_num) - For each sge prepares backend sge - Calls backend's post_recv 4.3.4 Process backend events ============================ - Done by a dedicated thread used to process backend events; at initialization is attached to the device and creates the communication channel. - Thread main loop: - Polls for completions - Extracts QEMU _cq_num, wr_id and op_code from context - Writes CQE to CQ ring - Writes CQ number to device CQ - Sends completion-interrupt to guest - Deallocates context - Acks the event to backend 5. Limitations ============== - The device obviously is limited by the Guest Linux Driver features implementation of the VMware device API. - Memory registration mechanism requires mremap for every page in the buffer in order to map it to a contiguous virtual address range. Since this is not the data path it should not matter much. If the default max mr size is increased, be aware that memory registration can take up to 0.5 seconds for 1GB of memory. - The device requires target page size to be the same as the host page size, otherwise it will fail to init. - QEMU cannot map guest RAM from a file descriptor if a pvrdma device is attached, so it can't work with huge pages. The limitation will be addressed in the future, however QEMU allocates Guest RAM with MADV_HUGEPAGE so if there are enough huge pages available, QEMU will use them. QEMU will fail to init if the requirements are not met. 6. Performance ============== By design the pvrdma device exits on each post-send/receive, so for small buffers the performance is affected; however for medium buffers it will became close to bare metal and from 1MB buffers and up it reaches bare metal performance. (tested with 2 VMs, the pvrdma devices connected to 2 VFs of the same device) All the above assumes no memory registration is done on data path. --- ### Qcow2 Cache (`docs/qcow2-cache.txt`) qcow2 L2/refcount cache configuration ===================================== Copyright (C) 2015, 2018-2020 Igalia, S.L. Author: Alberto Garcia This work is licensed under the terms of the GNU GPL, version 2 or later. See the COPYING file in the top-level directory. Introduction ------------ The QEMU qcow2 driver has two caches that can improve the I/O performance significantly. However, setting the right cache sizes is not a straightforward operation. This document attempts to give an overview of the L2 and refcount caches, and how to configure them. Please refer to the docs/interop/qcow2.txt file for an in-depth technical description of the qcow2 file format. Clusters -------- A qcow2 file is organized in units of constant size called clusters. The cluster size is configurable, but it must be a power of two and its value 512 bytes or higher. QEMU currently defaults to 64 KB clusters, and it does not support sizes larger than 2MB. The 'qemu-img create' command supports specifying the size using the cluster_size option: qemu-img create -f qcow2 -o cluster_size=128K hd.qcow2 4G The L2 tables ------------- The qcow2 format uses a two-level structure to map the virtual disk as seen by the guest to the disk image in the host. These structures are called the L1 and L2 tables. There is one single L1 table per disk image. The table is small and is always kept in memory. There can be many L2 tables, depending on how much space has been allocated in the image. Each table is one cluster in size. In order to read or write data from the virtual disk, QEMU needs to read its corresponding L2 table to find out where that data is located. Since reading the table for each I/O operation can be expensive, QEMU keeps an L2 cache in memory to speed up disk access. The size of the L2 cache can be configured, and setting the right value can improve the I/O performance significantly. The refcount blocks ------------------- The qcow2 format also maintains a reference count for each cluster. Reference counts are used for cluster allocation and internal snapshots. The data is stored in a two-level structure similar to the L1/L2 tables described above. The second level structures are called refcount blocks, are also one cluster in size and the number is also variable and dependent on the amount of allocated space. Each block contains a number of refcount entries. Their size (in bits) is a power of two and must not be higher than 64. It defaults to 16 bits, but a different value can be set using the refcount_bits option: qemu-img create -f qcow2 -o refcount_bits=8 hd.qcow2 4G QEMU keeps a refcount cache to speed up I/O much like the aforementioned L2 cache, and its size can also be configured. Choosing the right cache sizes ------------------------------ In order to choose the cache sizes we need to know how they relate to the amount of allocated space. The part of the virtual disk that can be mapped by the L2 and refcount caches (in bytes) is: disk_size = l2_cache_size * cluster_size / 8 disk_size = refcount_cache_size * cluster_size * 8 / refcount_bits With the default values for cluster_size (64KB) and refcount_bits (16), this becomes: disk_size = l2_cache_size * 8192 disk_size = refcount_cache_size * 32768 So in order to cover n GB of disk space with the default values we need: l2_cache_size = disk_size_GB * 131072 refcount_cache_size = disk_size_GB * 32768 For example, 1MB of L2 cache is needed to cover every 8 GB of the virtual image size (given that the default cluster size is used): 8 GB / 8192 = 1 MB The refcount cache is 4 times the cluster size by default. With the default cluster size of 64 KB, it is 256 KB (262144 bytes). This is sufficient for 8 GB of image size: 262144 * 32768 = 8 GB How to configure the cache sizes -------------------------------- Cache sizes can be configured using the -drive option in the command-line, or the 'blockdev-add' QMP command. There are three options available, and all of them take bytes: "l2-cache-size": maximum size of the L2 table cache "refcount-cache-size": maximum size of the refcount block cache "cache-size": maximum size of both caches combined There are a few things that need to be taken into account: - Both caches must have a size that is a multiple of the cluster size (or the cache entry size: see "Using smaller cache sizes" below). - The maximum L2 cache size is 32 MB by default on Linux platforms (enough for full coverage of 256 GB images, with the default cluster size). This value can be modified using the "l2-cache-size" option. QEMU will not use more memory than needed to hold all of the image's L2 tables, regardless of this max. value. On non-Linux platforms the maximal value is smaller by default (8 MB) and this difference stems from the fact that on Linux the cache can be cleared periodically if needed, using the "cache-clean-interval" option (see below). The minimal L2 cache size is 2 clusters (or 2 cache entries, see below). - The default (and minimum) refcount cache size is 4 clusters. - If only "cache-size" is specified then QEMU will assign as much memory as possible to the L2 cache before increasing the refcount cache size. - At most two of "l2-cache-size", "refcount-cache-size", and "cache-size" can be set simultaneously. Unlike L2 tables, refcount blocks are not used during normal I/O but only during allocations and internal snapshots. In most cases they are accessed sequentially (even during random guest I/O) so increasing the refcount cache size won't have any measurable effect in performance (this can change if you are using internal snapshots, so you may want to think about increasing the cache size if you use them heavily). Before QEMU 2.12 the refcount cache had a default size of 1/4 of the L2 cache size. This resulted in unnecessarily large caches, so now the refcount cache is as small as possible unless overridden by the user. Using smaller cache entries --------------------------- The qcow2 L2 cache can store complete tables. This means that if QEMU needs an entry from an L2 table then the whole table is read from disk and is kept in the cache. If the cache is full then a complete table needs to be evicted first. This can be inefficient with large cluster sizes since it results in more disk I/O and wastes more cache memory. Since QEMU 2.12 you can change the size of the L2 cache entry and make it smaller than the cluster size. This can be configured using the "l2-cache-entry-size" parameter: -drive file=hd.qcow2,l2-cache-size=2097152,l2-cache-entry-size=4096 Since QEMU 4.0 the value of l2-cache-entry-size defaults to 4KB (or the cluster size if it's smaller). Some things to take into account: - The L2 cache entry size has the same restrictions as the cluster size (power of two, at least 512 bytes). - Smaller entry sizes generally improve the cache efficiency and make disk I/O faster. This is particularly true with solid state drives so it's a good idea to reduce the entry size in those cases. With rotating hard drives the situation is a bit more complicated so you should test it first and stay with the default size if unsure. - Try different entry sizes to see which one gives faster performance in your case. The block size of the host filesystem is generally a good default (usually 4096 bytes in the case of ext4, hence the default). - Only the L2 cache can be configured this way. The refcount cache always uses the cluster size as the entry size. - If the L2 cache is big enough to hold all of the image's L2 tables (as explained in the "Choosing the right cache sizes" and "How to configure the cache sizes" sections in this document) then none of this is necessary and you can omit the "l2-cache-entry-size" parameter altogether. In this case QEMU makes the entry size equal to the cluster size by default. Reducing the memory usage ------------------------- It is possible to clean unused cache entries in order to reduce the memory usage during periods of low I/O activity. The parameter "cache-clean-interval" defines an interval (in seconds), after which all the cache entries that haven't been accessed during the interval are removed from memory. Setting this parameter to 0 disables this feature. The following example removes all unused cache entries every 15 minutes: -drive file=hd.qcow2,cache-clean-interval=900 If unset, the default value for this parameter is 600 on platforms which support this functionality, and is 0 (disabled) on other platforms. This functionality currently relies on the MADV_DONTNEED argument for madvise() to actually free the memory. This is a Linux-specific feature, so cache-clean-interval is not supported on other systems. Extended L2 Entries ------------------- All numbers shown in this document are valid for qcow2 images with normal 64-bit L2 entries. Images with extended L2 entries need twice as much L2 metadata, so the L2 cache size must be twice as large for the same disk space. disk_size = l2_cache_size * cluster_size / 16 i.e. l2_cache_size = disk_size * 16 / cluster_size Refcount blocks are not affected by this. --- ### Qdev Device Use (`docs/qdev-device-use.txt`) = How to convert to -device & friends = === Specifying Bus and Address on Bus === In qdev, each device has a parent bus. Some devices provide one or more buses for children. You can specify a device's parent bus with -device parameter bus. A device typically has a device address on its parent bus. For buses where this address can be configured, devices provide a bus-specific property. Examples: bus property name value format PCI addr %x.%x (dev.fn, .fn optional) I2C address %u SCSI scsi-id %u IDE unit %u HDA cad %u virtio-serial-bus nr %u ccid-bus slot %u USB port %d(.%d)* (port.port...) Example: device i440FX-pcihost is on the root bus, and provides a PCI bus named pci.0. To put a FOO device into its slot 4, use -device FOO,bus=/i440FX-pcihost/pci.0,addr=4. The abbreviated form bus=pci.0 also works as long as the bus name is unique. === Block Devices === A QEMU block device (drive) has a host and a guest part. In the general case, the guest device is connected to a controller device. For instance, the IDE controller provides two IDE buses, each of which can have up to two devices, and each device is a guest part, and is connected to a host part. Except we sometimes lump controller, bus(es) and drive device(s) all together into a single device. For instance, the ISA floppy controller is connected to up to two host drives. The old ways to define block devices define host and guest part together. Sometimes, they can even define a controller device in addition to the block device. The new way keeps the parts separate: you create the host part with -drive, and guest device(s) with -device. The various old ways to define drives all boil down to the common form -drive if=TYPE,bus=BUS,unit=UNIT,OPTS... TYPE, BUS and UNIT identify the controller device, which of its buses to use, and the drive's address on that bus. Details depend on TYPE. Instead of bus=BUS,unit=UNIT, you can also say index=IDX. In the new way, this becomes something like -drive if=none,id=DRIVE-ID,HOST-OPTS... -device DEVNAME,drive=DRIVE-ID,DEV-OPTS... The old OPTS get split into HOST-OPTS and DEV-OPTS as follows: * file, format, snapshot, cache, aio, readonly, rerror, werror go into HOST-OPTS. * cyls, head, secs and trans go into HOST-OPTS. Future work: they should go into DEV-OPTS instead. * serial goes into DEV-OPTS, for devices supporting serial numbers. For other devices, it goes nowhere. * media is special. In the old way, it selects disk vs. CD-ROM with if=ide, if=scsi and if=xen. The new way uses DEVNAME for that. Additionally, readonly=on goes into HOST-OPTS. * addr is special, see if=virtio below. The -device argument differs in detail for each type of drive: * if=ide -device DEVNAME,drive=DRIVE-ID,bus=IDE-BUS,unit=UNIT where DEVNAME is either ide-hd or ide-cd, IDE-BUS identifies an IDE bus, normally either ide.0 or ide.1, and UNIT is either 0 or 1. * if=scsi The old way implicitly creates SCSI controllers as needed. The new way makes that explicit: -device lsi53c895a,id=ID As for all PCI devices, you can add bus=PCI-BUS,addr=DEVFN to control the PCI device address. This SCSI controller provides a single SCSI bus, named ID.0. Put a disk on it: -device DEVNAME,drive=DRIVE-ID,bus=ID.0,scsi-id=UNIT where DEVNAME is either scsi-hd, scsi-cd or scsi-generic. * if=floppy -device floppy,unit=UNIT,drive=DRIVE-ID Without any -device floppy,... you get an empty unit 0 and no unit 1. You can use -nodefaults to suppress the default unit 0, see "Default Devices". * if=virtio -device virtio-blk-pci,drive=DRIVE-ID,class=C,vectors=V,ioeventfd=IOEVENTFD This lets you control PCI device class and MSI-X vectors. IOEVENTFD controls whether or not ioeventfd is used for virtqueue notify. It can be set to on (default) or off. As for all PCI devices, you can add bus=PCI-BUS,addr=DEVFN to control the PCI device address. This replaces option addr available with -drive if=virtio. * if=pflash, if=mtd, if=sd, if=xen are not yet available with -device For USB devices, the old way was actually different: -usbdevice disk:format=FMT:FILENAME "Was" because "disk:" is gone since v2.12.0. The old way provided much less control than -drive's OPTS... The new way fixes that: -device usb-storage,drive=DRIVE-ID,removable=RMB The removable parameter gives control over the SCSI INQUIRY removable (RMB) bit. USB thumbdrives usually set removable=on, while USB hard disks set removable=off. Bug: usb-storage pretends to be a block device, but it's really a SCSI controller that can serve only a single device, which it creates automatically. The automatic creation guesses what kind of guest part to create from the host part, like -drive if=scsi. Host and guest part are not cleanly separated. === Character Devices === A QEMU character device has a host and a guest part. The old ways to define character devices define host and guest part together. The new way keeps the parts separate: you create the host part with -chardev, and the guest device with -device. The various old ways to define a character device are all of the general form -FOO FOO-OPTS...,LEGACY-CHARDEV where FOO-OPTS... is specific to -FOO, and the host part LEGACY-CHARDEV is the same everywhere. In the new way, this becomes -chardev HOST-OPTS...,id=CHR-ID -device DEVNAME,chardev=CHR-ID,DEV-OPTS... The appropriate DEVNAME depends on the machine type. For type "pc": * -serial becomes -device isa-serial,iobase=IOADDR,irq=IRQ,index=IDX This lets you control I/O ports and IRQs. * -parallel becomes -device isa-parallel,iobase=IOADDR,irq=IRQ,index=IDX This lets you control I/O ports and IRQs. * -usbdevice braille doesn't support LEGACY-CHARDEV syntax. It always uses "braille". With -device, this useful default is gone, so you have to use something like -device usb-braille,chardev=braille -chardev braille,id=braille * -usbdevice serial::chardev is gone since v2.12.0. It became -device usb-serial,chardev=dev. LEGACY-CHARDEV translates to -chardev HOST-OPTS... as follows: * null becomes -chardev null * pty, msmouse, wctablet, braille, stdio likewise * vc:WIDTHxHEIGHT becomes -chardev vc,width=WIDTH,height=HEIGHT * vc:CxC becomes -chardev vc,cols=,rows= * con: becomes -chardev console * COM becomes -chardev serial,path=COM * file:FNAME becomes -chardev file,path=FNAME * pipe:FNAME becomes -chardev pipe,path=FNAME * tcp:HOST:PORT,OPTS... becomes -chardev socket,host=HOST,port=PORT,OPTS... * telnet:HOST:PORT,OPTS... becomes -chardev socket,host=HOST,port=PORT,OPTS...,telnet=on * udp:HOST:PORT@LOCALADDR:LOCALPORT becomes -chardev udp,host=HOST,port=PORT,localaddr=LOCALADDR,localport=LOCALPORT * unix:FNAME becomes -chardev socket,path=FNAME * /dev/parportN becomes -chardev parport,file=/dev/parportN * /dev/ppiN likewise * Any other /dev/FNAME becomes -chardev tty,path=/dev/FNAME * mon:LEGACY-CHARDEV is special: it multiplexes the monitor onto the character device defined by LEGACY-CHARDEV. -chardev provides more general multiplexing instead: you can connect up to four users to a single host part. You need to pass mux=on to -chardev to enable switching the input focus. QEMU uses LEGACY-CHARDEV syntax not just to set up guest devices, but also in various other places such as -monitor or -net user,guestfwd=... You can use chardev:CHR-ID in place of LEGACY-CHARDEV to refer to a host part defined with -chardev. === Network Devices === Host and guest part of network devices have always been separate. The old way to define the guest part looks like this: -net nic,netdev=NET-ID,macaddr=MACADDR,model=MODEL,name=ID,addr=STR,vectors=V Except for USB it looked like this: -usbdevice net:netdev=NET-ID,macaddr=MACADDR,name=ID "Looked" because "net:" is gone since v2.12.0. The new way is -device: -device DEVNAME,netdev=NET-ID,mac=MACADDR,DEV-OPTS... DEVNAME equals MODEL, except for virtio you have to name the virtio device appropriate for the bus (virtio-net-pci for PCI), and for USB you have to use usb-net. The old name=ID parameter becomes the usual id=ID with -device. For PCI devices, you can add bus=PCI-BUS,addr=DEVFN to control the PCI device address, as usual. The old -net nic provides parameter addr for that, which is silently ignored when the NIC is not a PCI device. For virtio-net-pci, you can control whether or not ioeventfd is used for virtqueue notify by setting ioeventfd= to on or off (default). -net nic accepts vectors=V for all models, but it's silently ignored except for virtio-net-pci (model=virtio). With -device, only devices that support it accept it. Not all devices are available with -device at this time. All PCI devices and ne2k_isa are. Some PCI devices aren't available with -net nic, e.g. i82558a. === Graphics Devices === Host and guest part of graphics devices have always been separate. The old way to define the guest graphics device is -vga VGA. Not all machines support all -vga options. The new way is -device. The mapping from -vga argument to -device depends on the machine type. For machine "pc", it's: std -device VGA cirrus -device cirrus-vga vmware -device vmware-svga qxl -device qxl-vga none -nodefaults disables more than just VGA, see "Default Devices" As for all PCI devices, you can add bus=PCI-BUS,addr=DEVFN to control the PCI device address. -device VGA supports properties bios-offset and bios-size, but they aren't used with machine type "pc". For machine "isapc", it's std -device isa-vga cirrus not yet available with -device none -nodefaults disables more than just VGA, see "Default Devices" Bug: the new way doesn't work for machine types "pc" and "isapc", because it violates obscure device initialization ordering constraints. === Audio Devices === Host and guest part of audio devices have always been separate. The old way to define guest audio devices is -soundhw C1,... The new way is to define each guest audio device separately with -device. Map from -soundhw sound card name to -device: ac97 -device AC97 cs4231a -device cs4231a,iobase=IOADDR,irq=IRQ,dma=DMA es1370 -device ES1370 gus -device gus,iobase=IOADDR,irq=IRQ,dma=DMA,freq=F hda -device intel-hda,msi=MSI -device hda-duplex sb16 -device sb16,iobase=IOADDR,irq=IRQ,dma=DMA,dma16=DMA16,version=V adlib not yet available with -device pcspk not yet available with -device For PCI devices, you can add bus=PCI-BUS,addr=DEVFN to control the PCI device address, as usual. === USB Devices === The old way to define a virtual USB device is -usbdevice DRIVER:OPTS... The new way is -device DEVNAME,DEV-OPTS... Details depend on DRIVER: * ccid -device usb-ccid * keyboard -device usb-kbd * mouse -device usb-mouse * tablet -device usb-tablet * wacom-tablet -device usb-wacom-tablet * u2f -device u2f-{emulated,passthru} * braille See "Character Devices" Until v2.12.0, we additionally had * host:... See "Host Device Assignment" * disk:... See "Block Devices" * serial:... See "Character Devices" * net:... See "Network Devices" === Watchdog Devices === Host and guest part of watchdog devices have always been separate. The old way to define a guest watchdog device is -watchdog DEVNAME. The new way is -device DEVNAME. For PCI devices, you can add bus=PCI-BUS,addr=DEVFN to control the PCI device address, as usual. === Host Device Assignment === QEMU supports assigning host PCI devices (qemu-kvm only at this time) and host USB devices. PCI devices can only be assigned with -device: -device vfio-pci,host=ADDR,id=ID The old way to assign a USB host device -usbdevice host:auto:BUS.ADDR:VID:PRID was removed in v2.12.0. Any of BUS, ADDR, VID, PRID could be the wildcard *. The new way is -device usb-host,hostbus=BUS,hostaddr=ADDR,vendorid=VID,productid=PRID Omitted options match anything. === Default Devices === QEMU creates a number of devices by default, depending on the machine type. -device DEVNAME... and global DEVNAME... suppress default devices for some DEVNAMEs: default device suppressing DEVNAMEs CD-ROM ide-cd, ide-hd, scsi-cd, scsi-hd floppy floppy, isa-fdc parallel isa-parallel serial isa-serial VGA VGA, cirrus-vga, isa-vga, isa-cirrus-vga, vmware-svga, qxl-vga, virtio-vga, ati-vga, vhost-user-vga The default NIC is connected to a default part created along with it. It is *not* suppressed by configuring a NIC with -device (you may call that a bug). -net and -netdev suppress the default NIC. -nodefaults suppresses all the default devices mentioned above, plus a few other things such as default SD-Card drive and default monitor. --- ### Rdma (`docs/rdma.txt`) (RDMA: Remote Direct Memory Access) RDMA Live Migration Specification, Version # 1 ============================================== Wiki: https://wiki.qemu.org/Features/RDMALiveMigration Github: git@github.com:hinesmr/qemu.git, 'rdma' branch Copyright (C) 2013 Michael R. Hines An *exhaustive* paper (2010) shows additional performance details linked on the QEMU wiki above. Contents: ========= * Introduction * Before running * Running * Performance * RDMA Migration Protocol Description * Versioning and Capabilities * QEMUFileRDMA Interface * Migration of VM's ram * Error handling * TODO Introduction: ============= RDMA helps make your migration more deterministic under heavy load because of the significantly lower latency and higher throughput over TCP/IP. This is because the RDMA I/O architecture reduces the number of interrupts and data copies by bypassing the host networking stack. In particular, a TCP-based migration, under certain types of memory-bound workloads, may take a more unpredictable amount of time to complete the migration if the amount of memory tracked during each live migration iteration round cannot keep pace with the rate of dirty memory produced by the workload. RDMA currently comes in two flavors: both Ethernet based (RoCE, or RDMA over Converged Ethernet) as well as Infiniband-based. This implementation of migration using RDMA is capable of using both technologies because of the use of the OpenFabrics OFED software stack that abstracts out the programming model irrespective of the underlying hardware. Refer to openfabrics.org or your respective RDMA hardware vendor for an understanding on how to verify that you have the OFED software stack installed in your environment. You should be able to successfully link against the "librdmacm" and "libibverbs" libraries and development headers for a working build of QEMU to run successfully using RDMA Migration. BEFORE RUNNING: =============== Use of RDMA during migration requires pinning and registering memory with the hardware. This means that memory must be physically resident before the hardware can transmit that memory to another machine. If this is not acceptable for your application or product, then the use of RDMA migration may in fact be harmful to co-located VMs or other software on the machine if there is not sufficient memory available to relocate the entire footprint of the virtual machine. If so, then the use of RDMA is discouraged and it is recommended to use standard TCP migration. Experimental: Next, decide if you want dynamic page registration. For example, if you have an 8GB RAM virtual machine, but only 1GB is in active use, then enabling this feature will cause all 8GB to be pinned and resident in memory. This feature mostly affects the bulk-phase round of the migration and can be enabled for extremely high-performance RDMA hardware using the following command: QEMU Monitor Command: $ migrate_set_capability rdma-pin-all on # disabled by default Performing this action will cause all 8GB to be pinned, so if that's not what you want, then please ignore this step altogether. On the other hand, this will also significantly speed up the bulk round of the migration, which can greatly reduce the "total" time of your migration. Example performance of this using an idle VM in the previous example can be found in the "Performance" section. Note: for very large virtual machines (hundreds of GBs), pinning all *all* of the memory of your virtual machine in the kernel is very expensive may extend the initial bulk iteration time by many seconds, and thus extending the total migration time. However, this will not affect the determinism or predictability of your migration you will still gain from the benefits of advanced pinning with RDMA. RUNNING: ======== First, set the migration speed to match your hardware's capabilities: QEMU Monitor Command: $ migrate_set_parameter max_bandwidth 40g # or whatever is the MAX of your RDMA device Next, on the destination machine, add the following to the QEMU command line: qemu ..... -incoming rdma:host:port Finally, perform the actual migration on the source machine: QEMU Monitor Command: $ migrate -d rdma:host:port PERFORMANCE =========== Here is a brief summary of total migration time and downtime using RDMA: Using a 40gbps infiniband link performing a worst-case stress test, using an 8GB RAM virtual machine: Using the following command: $ apt-get install stress $ stress --vm-bytes 7500M --vm 1 --vm-keep 1. Migration throughput: 26 gigabits/second. 2. Downtime (stop time) varies between 15 and 100 milliseconds. EFFECTS of memory registration on bulk phase round: For example, in the same 8GB RAM example with all 8GB of memory in active use and the VM itself is completely idle using the same 40 gbps infiniband link: 1. rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps 2. rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps These numbers would of course scale up to whatever size virtual machine you have to migrate using RDMA. Enabling this feature does *not* have any measurable affect on migration *downtime*. This is because, without this feature, all of the memory will have already been registered already in advance during the bulk round and does not need to be re-registered during the successive iteration rounds. RDMA Protocol Description: ========================== Migration with RDMA is separated into two parts: 1. The transmission of the pages using RDMA 2. Everything else (a control channel is introduced) "Everything else" is transmitted using a formal protocol now, consisting of infiniband SEND messages. An infiniband SEND message is the standard ibverbs message used by applications of infiniband hardware. The only difference between a SEND message and an RDMA message is that SEND messages cause notifications to be posted to the completion queue (CQ) on the infiniband receiver side, whereas RDMA messages (used for VM's ram) do not (to behave like an actual DMA). Messages in infiniband require two things: 1. registration of the memory that will be transmitted 2. (SEND only) work requests to be posted on both sides of the network before the actual transmission can occur. RDMA messages are much easier to deal with. Once the memory on the receiver side is registered and pinned, we're basically done. All that is required is for the sender side to start dumping bytes onto the link. (Memory is not released from pinning until the migration completes, given that RDMA migrations are very fast.) SEND messages require more coordination because the receiver must have reserved space (using a receive work request) on the receive queue (RQ) before QEMUFileRDMA can start using them to carry all the bytes as a control transport for migration of device state. To begin the migration, the initial connection setup is as follows (migration-rdma.c): 1. Receiver and Sender are started (command line or libvirt): 2. Both sides post two RQ work requests 3. Receiver does listen() 4. Sender does connect() 5. Receiver accept() 6. Check versioning and capabilities (described later) At this point, we define a control channel on top of SEND messages which is described by a formal protocol. Each SEND message has a header portion and a data portion (but together are transmitted as a single SEND message). Header: * Length (of the data portion, uint32, network byte order) * Type (what command to perform, uint32, network byte order) * Repeat (Number of commands in data portion, same type only) The 'Repeat' field is here to support future multiple page registrations in a single message without any need to change the protocol itself so that the protocol is compatible against multiple versions of QEMU. Version #1 requires that all server implementations of the protocol must check this field and register all requests found in the array of commands located in the data portion and return an equal number of results in the response. The maximum number of repeats is hard-coded to 4096. This is a conservative limit based on the maximum size of a SEND message along with empirical observations on the maximum future benefit of simultaneous page registrations. The 'type' field has 12 different command values: 1. Unused 2. Error (sent to the source during bad things) 3. Ready (control-channel is available) 4. QEMU File (for sending non-live device state) 5. RAM Blocks request (used right after connection setup) 6. RAM Blocks result (used right after connection setup) 7. Compress page (zap zero page and skip registration) 8. Register request (dynamic chunk registration) 9. Register result ('rkey' to be used by sender) 10. Register finished (registration for current iteration finished) 11. Unregister request (unpin previously registered memory) 12. Unregister finished (confirmation that unpin completed) A single control message, as hinted above, can contain within the data portion an array of many commands of the same type. If there is more than one command, then the 'repeat' field will be greater than 1. After connection setup, message 5 & 6 are used to exchange ram block information and optionally pin all the memory if requested by the user. After ram block exchange is completed, we have two protocol-level functions, responsible for communicating control-channel commands using the above list of values: Logically: qemu_rdma_exchange_recv(header, expected command type) 1. We transmit a READY command to let the sender know that we are *ready* to receive some data bytes on the control channel. 2. Before attempting to receive the expected command, we post another RQ work request to replace the one we just used up. 3. Block on a CQ event channel and wait for the SEND to arrive. 4. When the send arrives, librdmacm will unblock us. 5. Verify that the command-type and version received matches the one we expected. qemu_rdma_exchange_send(header, data, optional response header & data): 1. Block on the CQ event channel waiting for a READY command from the receiver to tell us that the receiver is *ready* for us to transmit some new bytes. 2. Optionally: if we are expecting a response from the command (that we have not yet transmitted), let's post an RQ work request to receive that data a few moments later. 3. When the READY arrives, librdmacm will unblock us and we immediately post a RQ work request to replace the one we just used up. 4. Now, we can actually post the work request to SEND the requested command type of the header we were asked for. 5. Optionally, if we are expecting a response (as before), we block again and wait for that response using the additional work request we previously posted. (This is used to carry 'Register result' commands #6 back to the sender which hold the rkey need to perform RDMA. Note that the virtual address corresponding to this rkey was already exchanged at the beginning of the connection (described below). All of the remaining command types (not including 'ready') described above all use the aforementioned two functions to do the hard work: 1. After connection setup, RAMBlock information is exchanged using this protocol before the actual migration begins. This information includes a description of each RAMBlock on the server side as well as the virtual addresses and lengths of each RAMBlock. This is used by the client to determine the start and stop locations of chunks and how to register them dynamically before performing the RDMA operations. 2. During runtime, once a 'chunk' becomes full of pages ready to be sent with RDMA, the registration commands are used to ask the other side to register the memory for this chunk and respond with the result (rkey) of the registration. 3. Also, the QEMUFile interfaces also call these functions (described below) when transmitting non-live state, such as devices or to send its own protocol information during the migration process. 4. Finally, zero pages are only checked if a page has not yet been registered using chunk registration (or not checked at all and unconditionally written if chunk registration is disabled. This is accomplished using the "Compress" command listed above. If the page *has* been registered then we check the entire chunk for zero. Only if the entire chunk is zero, then we send a compress command to zap the page on the other side. Versioning and Capabilities =========================== Current version of the protocol is version #1. The same version applies to both for protocol traffic and capabilities negotiation. (i.e. There is only one version number that is referred to by all communication). librdmacm provides the user with a 'private data' area to be exchanged at connection-setup time before any infiniband traffic is generated. Header: * Version (protocol version validated before send/recv occurs), uint32, network byte order * Flags (bitwise OR of each capability), uint32, network byte order There is no data portion of this header right now, so there is no length field. The maximum size of the 'private data' section is only 192 bytes per the Infiniband specification, so it's not very useful for data anyway. This structure needs to remain small. This private data area is a convenient place to check for protocol versioning because the user does not need to register memory to transmit a few bytes of version information. This is also a convenient place to negotiate capabilities (like dynamic page registration). If the version is invalid, we throw an error. If the version is new, we only negotiate the capabilities that the requested version is able to perform and ignore the rest. Currently there is only one capability in Version #1: dynamic page registration Finally: Negotiation happens with the Flags field: If the primary-VM sets a flag, but the destination does not support this capability, it will return a zero-bit for that flag and the primary-VM will understand that as not being an available capability and will thus disable that capability on the primary-VM side. QEMUFileRDMA Interface: ======================= QEMUFileRDMA introduces a couple of new functions: 1. qemu_rdma_get_buffer() (QEMUFileOps rdma_read_ops) 2. qemu_rdma_put_buffer() (QEMUFileOps rdma_write_ops) These two functions are very short and simply use the protocol describe above to deliver bytes without changing the upper-level users of QEMUFile that depend on a bytestream abstraction. Finally, how do we handoff the actual bytes to get_buffer()? Again, because we're trying to "fake" a bytestream abstraction using an analogy not unlike individual UDP frames, we have to hold on to the bytes received from control-channel's SEND messages in memory. Each time we receive a complete "QEMU File" control-channel message, the bytes from SEND are copied into a small local holding area. Then, we return the number of bytes requested by get_buffer() and leave the remaining bytes in the holding area until get_buffer() comes around for another pass. If the buffer is empty, then we follow the same steps listed above and issue another "QEMU File" protocol command, asking for a new SEND message to re-fill the buffer. Migration of VM's ram: ==================== At the beginning of the migration, (migration-rdma.c), the sender and the receiver populate the list of RAMBlocks to be registered with each other into a structure. Then, using the aforementioned protocol, they exchange a description of these blocks with each other, to be used later during the iteration of main memory. This description includes a list of all the RAMBlocks, their offsets and lengths, virtual addresses and possibly includes pre-registered RDMA keys in case dynamic page registration was disabled on the server-side, otherwise not. Main memory is not migrated with the aforementioned protocol, but is instead migrated with normal RDMA Write operations. Pages are migrated in "chunks" (hard-coded to 1 Megabyte right now). Chunk size is not dynamic, but it could be in a future implementation. There's nothing to indicate that this is useful right now. When a chunk is full (or a flush() occurs), the memory backed by the chunk is registered with librdmacm is pinned in memory on both sides using the aforementioned protocol. After pinning, an RDMA Write is generated and transmitted for the entire chunk. Chunks are also transmitted in batches: This means that we do not request that the hardware signal the completion queue for the completion of *every* chunk. The current batch size is about 64 chunks (corresponding to 64 MB of memory). Only the last chunk in a batch must be signaled. This helps keep everything as asynchronous as possible and helps keep the hardware busy performing RDMA operations. Error-handling: =============== Infiniband has what is called a "Reliable, Connected" link (one of 4 choices). This is the mode in which we use for RDMA migration. If a *single* message fails, the decision is to abort the migration entirely and cleanup all the RDMA descriptors and unregister all the memory. After cleanup, the Virtual Machine is returned to normal operation the same way that would happen if the TCP socket is broken during a non-RDMA based migration. TODO: ===== 1. Currently, 'ulimit -l' mlock() limits as well as cgroups swap limits are not compatible with infiniband memory pinning and will result in an aborted migration (but with the source VM left unaffected). 2. Use of the recent /proc//pagemap would likely speed up the use of KSM and ballooning while using RDMA. 3. Also, some form of balloon-device usage tracking would also help alleviate some issues. 4. Use LRU to provide more fine-grained direction of UNREGISTER requests for unpinning memory in an overcommitted environment. 5. Expose UNREGISTER support to the user by way of workload-specific hints about application behavior. --- ### Spice Port Fqdn (`docs/spice-port-fqdn.txt`) A Spice port channel is an arbitrary communication between the Spice server host side and the client side. Thanks to the associated reverse fully qualified domain name (fqdn), a Spice client can handle the various ports appropriately. The following fqdn names are reserved by the QEMU project: org.qemu.monitor.hmp.0 QEMU human monitor org.qemu.monitor.qmp.0: QEMU control monitor org.qemu.console.serial.0 QEMU virtual serial port org.qemu.console.debug.0 QEMU debug console --- ### Throttle (`docs/throttle.txt`) The QEMU throttling infrastructure ================================== Copyright (C) 2016,2020 Igalia, S.L. Author: Alberto Garcia This work is licensed under the terms of the GNU GPL, version 2 or later. See the COPYING file in the top-level directory. Introduction ------------ QEMU includes a throttling module that can be used to set limits to I/O operations. The code itself is generic and independent of the I/O units, but it is currently used to limit the number of bytes per second and operations per second (IOPS) when performing disk I/O. This document explains how to use the throttling code in QEMU, and how it works internally. The implementation is in throttle.c. Using throttling to limit disk I/O ---------------------------------- Two aspects of the disk I/O can be limited: the number of bytes per second and the number of operations per second (IOPS). For each one of them the user can set a global limit or separate limits for read and write operations. This gives us a total of six different parameters. I/O limits can be set using the throttling.* parameters of -drive, or using the QMP 'block_set_io_throttle' command. These are the names of the parameters for both cases: |-----------------------+-----------------------| | -drive | block_set_io_throttle | |-----------------------+-----------------------| | throttling.iops-total | iops | | throttling.iops-read | iops_rd | | throttling.iops-write | iops_wr | | throttling.bps-total | bps | | throttling.bps-read | bps_rd | | throttling.bps-write | bps_wr | |-----------------------+-----------------------| It is possible to set limits for both IOPS and bps at the same time, and for each case we can decide whether to have separate read and write limits or not, but note that if iops-total is set then neither iops-read nor iops-write can be set. The same applies to bps-total and bps-read/write. The default value of these parameters is 0, and it means 'unlimited'. In its most basic usage, the user can add a drive to QEMU with a limit of 100 IOPS with the following -drive line: -drive file=hd0.qcow2,throttling.iops-total=100 We can do the same using QMP. In this case all these parameters are mandatory, so we must set to 0 the ones that we don't want to limit: { "execute": "block_set_io_throttle", "arguments": { "device": "virtio0", "iops": 100, "iops_rd": 0, "iops_wr": 0, "bps": 0, "bps_rd": 0, "bps_wr": 0 } } I/O bursts ---------- In addition to the basic limits we have just seen, QEMU allows the user to do bursts of I/O for a configurable amount of time. A burst is an amount of I/O that can exceed the basic limit. Bursts are useful to allow better performance when there are peaks of activity (the OS boots, a service needs to be restarted) while keeping the average limits lower the rest of the time. Two parameters control bursts: their length and the maximum amount of I/O they allow. These two can be configured separately for each one of the six basic parameters described in the previous section, but in this section we'll use 'iops-total' as an example. The I/O limit during bursts is set using 'iops-total-max', and the maximum length (in seconds) is set with 'iops-total-max-length'. So if we want to configure a drive with a basic limit of 100 IOPS and allow bursts of 2000 IOPS for 60 seconds, we would do it like this (the line is split for clarity): -drive file=hd0.qcow2, throttling.iops-total=100, throttling.iops-total-max=2000, throttling.iops-total-max-length=60 Or, with QMP: { "execute": "block_set_io_throttle", "arguments": { "device": "virtio0", "iops": 100, "iops_rd": 0, "iops_wr": 0, "bps": 0, "bps_rd": 0, "bps_wr": 0, "iops_max": 2000, "iops_max_length": 60, } } With this, the user can perform I/O on hd0.qcow2 at a rate of 2000 IOPS for 1 minute before it's throttled down to 100 IOPS. The user will be able to do bursts again if there's a sufficiently long period of time with unused I/O (see below for details). The default value for 'iops-total-max' is 0 and it means that bursts are not allowed. 'iops-total-max-length' can only be set if 'iops-total-max' is set as well, and its default value is 1 second. Here's the complete list of parameters for configuring bursts: |----------------------------------+-----------------------| | -drive | block_set_io_throttle | |----------------------------------+-----------------------| | throttling.iops-total-max | iops_max | | throttling.iops-total-max-length | iops_max_length | | throttling.iops-read-max | iops_rd_max | | throttling.iops-read-max-length | iops_rd_max_length | | throttling.iops-write-max | iops_wr_max | | throttling.iops-write-max-length | iops_wr_max_length | | throttling.bps-total-max | bps_max | | throttling.bps-total-max-length | bps_max_length | | throttling.bps-read-max | bps_rd_max | | throttling.bps-read-max-length | bps_rd_max_length | | throttling.bps-write-max | bps_wr_max | | throttling.bps-write-max-length | bps_wr_max_length | |----------------------------------+-----------------------| Controlling the size of I/O operations -------------------------------------- When applying IOPS limits all I/O operations are treated equally regardless of their size. This means that the user can take advantage of this in order to circumvent the limits and submit one huge I/O request instead of several smaller ones. QEMU provides a setting called throttling.iops-size to prevent this from happening. This setting specifies the size (in bytes) of an I/O request for accounting purposes. Larger requests will be counted proportionally to this size. For example, if iops-size is set to 4096 then an 8KB request will be counted as two, and a 6KB request will be counted as one and a half. This only applies to requests larger than iops-size: smaller requests will be always counted as one, no matter their size. The default value of iops-size is 0 and it means that the size of the requests is never taken into account when applying IOPS limits. Applying I/O limits to groups of disks -------------------------------------- In all the examples so far we have seen how to apply limits to the I/O performed on individual drives, but QEMU allows grouping drives so they all share the same limits. The way it works is that each drive with I/O limits is assigned to a group named using the throttling.group parameter. If this parameter is not specified, then the device name (i.e. 'virtio0', 'ide0-hd0') will be used as the group name. Limits set using the throttling.* parameters discussed earlier in this document apply to the combined I/O of all members of a group. Consider this example: -drive file=hd1.qcow2,throttling.iops-total=6000,throttling.group=foo -drive file=hd2.qcow2,throttling.iops-total=6000,throttling.group=foo -drive file=hd3.qcow2,throttling.iops-total=3000,throttling.group=bar -drive file=hd4.qcow2,throttling.iops-total=6000,throttling.group=foo -drive file=hd5.qcow2,throttling.iops-total=3000,throttling.group=bar -drive file=hd6.qcow2,throttling.iops-total=5000 Here hd1, hd2 and hd4 are all members of a group named 'foo' with a combined IOPS limit of 6000, and hd3 and hd5 are members of 'bar'. hd6 is left alone (technically it is part of a 1-member group). Limits are applied in a round-robin fashion so if there are concurrent I/O requests on several drives of the same group they will be distributed evenly. When I/O limits are applied to an existing drive using the QMP command 'block_set_io_throttle', the following things need to be taken into account: - I/O limits are shared within the same group, so new values will affect all members and overwrite the previous settings. In other words: if different limits are applied to members of the same group, the last one wins. - If 'group' is unset it is assumed to be the current group of that drive. If the drive is not in a group yet, it will be added to a group named after the device name. - If 'group' is set then the drive will be moved to that group if it was member of a different one. In this case the limits specified in the parameters will be applied to the new group only. - I/O limits can be disabled by setting all of them to 0. In this case the device will be removed from its group and the rest of its members will not be affected. The 'group' parameter is ignored. The Leaky Bucket algorithm -------------------------- I/O limits in QEMU are implemented using the leaky bucket algorithm (specifically the "Leaky bucket as a meter" variant). This algorithm uses the analogy of a bucket that leaks water constantly. The water that gets into the bucket represents the I/O that has been performed, and no more I/O is allowed once the bucket is full. To see the way this corresponds to the throttling parameters in QEMU, consider the following values: iops-total=100 iops-total-max=2000 iops-total-max-length=60 - Water leaks from the bucket at a rate of 100 IOPS. - Water can be added to the bucket at a rate of 2000 IOPS. - The size of the bucket is 2000 x 60 = 120000 - If 'iops-total-max-length' is unset then it defaults to 1 and the size of the bucket is 2000. - If 'iops-total-max' is unset then 'iops-total-max-length' must be unset as well. In this case the bucket size is 100. The bucket is initially empty, therefore water can be added until it's full at a rate of 2000 IOPS (the burst rate). Once the bucket is full we can only add as much water as it leaks, therefore the I/O rate is reduced to 100 IOPS. If we add less water than it leaks then the bucket will start to empty, allowing for bursts again. Note that since water is leaking from the bucket even during bursts, it will take a bit more than 60 seconds at 2000 IOPS to fill it up. After those 60 seconds the bucket will have leaked 60 x 100 = 6000, allowing for 3 more seconds of I/O at 2000 IOPS. Also, due to the way the algorithm works, longer burst can be done at a lower I/O rate, e.g. 1000 IOPS during 120 seconds. The 'throttle' block filter --------------------------- Since QEMU 2.11 it is possible to configure the I/O limits using a 'throttle' block filter. This filter uses the exact same throttling infrastructure described above but can be used anywhere in the node graph, allowing for more flexibility. The user can create an arbitrary number of filters and each one of them must be assigned to a group that contains the actual I/O limits. Different filters can use the same group so the limits are shared as described earlier in "Applying I/O limits to groups of disks". A group can be created using the object-add QMP function: { "execute": "object-add", "arguments": { "qom-type": "throttle-group", "id": "group0", "limits" : { "iops-total": 1000, "bps-write": 2097152 } } } throttle-group has a 'limits' property (of type ThrottleLimits as defined in qapi/block-core.json) which can be set on creation or later with 'qom-set'. A throttle-group can also be created with the -object command line option but at the moment there is no way to pass a 'limits' parameter that contains a ThrottleLimits structure. The solution is to set the individual values directly, like in this example: -object throttle-group,id=group0,x-iops-total=1000,x-bps-write=2097152 Note however that this is not a stable API (hence the 'x-' prefixes) and will disappear when -object gains support for structured options and enables use of 'limits'. Once we have a throttle-group we can use the throttle block filter, where the 'file' property must be set to the block device that we want to filter: { "execute": "blockdev-add", "arguments": { "options": { "driver": "qcow2", "node-name": "disk0", "file": { "driver": "file", "filename": "/path/to/disk.qcow2" } } } } { "execute": "blockdev-add", "arguments": { "driver": "throttle", "node-name": "throttle0", "throttle-group": "group0", "file": "disk0" } } A similar setup can also be done with the command line, for example: -drive driver=throttle,throttle-group=group0, file.driver=qcow2,file.file.filename=/path/to/disk.qcow2 The scenario described so far is very simple but the throttle block filter allows for more complex configurations. For example, let's say that we have three different drives and we want to set I/O limits for each one of them and an additional set of limits for the combined I/O of all three drives. First we would define all throttle groups, one for each one of the drives and one that would apply to all of them: -object throttle-group,id=limits0,x-iops-total=2000 -object throttle-group,id=limits1,x-iops-total=2500 -object throttle-group,id=limits2,x-iops-total=3000 -object throttle-group,id=limits012,x-iops-total=4000 Now we can define the drives, and for each one of them we use two chained throttle filters: the drive's own filter and the combined filter. -drive driver=throttle,throttle-group=limits012, file.driver=throttle,file.throttle-group=limits0 file.file.driver=qcow2,file.file.file.filename=/path/to/disk0.qcow2 -drive driver=throttle,throttle-group=limits012, file.driver=throttle,file.throttle-group=limits1 file.file.driver=qcow2,file.file.file.filename=/path/to/disk1.qcow2 -drive driver=throttle,throttle-group=limits012, file.driver=throttle,file.throttle-group=limits2 file.file.driver=qcow2,file.file.file.filename=/path/to/disk2.qcow2 In this example the individual drives have IOPS limits of 2000, 2500 and 3000 respectively but the total combined I/O can never exceed 4000 IOPS. --- ### U2f (`docs/u2f.txt`) QEMU U2F Key Device Documentation. Contents 1. USB U2F key device 2. Building 3. Using u2f-emulated 4. Using u2f-passthru 5. Libu2f-emu 1. USB U2F key device U2F is an open authentication standard that enables relying parties exposed to the internet to offer a strong second factor option for end user authentication. The standard brings many advantages to both parties, client and server, allowing to reduce over-reliance on passwords, it increases authentication security and simplifies passwords. The second factor is materialized by a device implementing the U2F protocol. In case of a USB U2F security key, it is a USB HID device that implements the U2F protocol. In QEMU, the USB U2F key device offers a dedicated support of U2F, allowing guest USB FIDO/U2F security keys operating in two possible modes: pass-through and emulated. The pass-through mode consists of passing all requests made from the guest to the physical security key connected to the host machine and vice versa. In addition, the dedicated pass-through allows to have a U2F security key shared on several guests which is not possible with a simple host device assignment pass-through. The emulated mode consists of completely emulating the behavior of an U2F device through software part. Libu2f-emu is used for that. 2. Building To ensure the build of the u2f-emulated device variant which depends on libu2f-emu: configuring and building: ./configure --enable-u2f && make The pass-through mode is built by default on Linux. To take advantage of the autoscan option it provides, make sure you have a working libudev installed on the host. 3. Using u2f-emulated To work, an emulated U2F device must have four elements: * ec x509 certificate * ec private key * counter (four bytes value) * 48 bytes of entropy (random bits) To use this type of device, this one has to be configured, and these four elements must be passed one way or another. Assuming that you have a working libu2f-emu installed on the host. There are three possible ways of configurations: * ephemeral * setup directory * manual Ephemeral is the simplest way to configure, it lets the device generate all the elements it needs for a single use of the lifetime of the device. qemu -usb -device u2f-emulated Setup directory allows to configure the device from a directory containing four files: * certificate.pem: ec x509 certificate * private-key.pem: ec private key * counter: counter value * entropy: 48 bytes of entropy qemu -usb -device u2f-emulated,dir=$dir Manual allows to configure the device more finely by specifying each of the elements necessary for the device: * cert * priv * counter * entropy qemu -usb -device u2f-emulated,cert=$DIR1/$FILE1,priv=$DIR2/$FILE2,counter=$DIR3/$FILE3,entropy=$DIR4/$FILE4 4. Using u2f-passthru On the host specify the u2f-passthru device with a suitable hidraw: qemu -usb -device u2f-passthru,hidraw=/dev/hidraw0 Alternately, the u2f-passthru device can autoscan to take the first U2F device it finds on the host (this requires a working libudev): qemu -usb -device u2f-passthru 5. Libu2f-emu The u2f-emulated device uses libu2f-emu for the U2F key emulation. Libu2f-emu implements completely the U2F protocol device part for all specified transport given by the FIDO Alliance. For more information about libu2f-emu see this page: https://github.com/MattGorko/libu2f-emu. --- ### Xbzrle (`docs/xbzrle.txt`) XBZRLE (Xor Based Zero Run Length Encoding) =========================================== Using XBZRLE (Xor Based Zero Run Length Encoding) allows for the reduction of VM downtime and the total live-migration time of Virtual machines. It is particularly useful for virtual machines running memory write intensive workloads that are typical of large enterprise applications such as SAP ERP Systems, and generally speaking for any application that uses a sparse memory update pattern. Instead of sending the changed guest memory page this solution will send a compressed version of the updates, thus reducing the amount of data sent during live migration. In order to be able to calculate the update, the previous memory pages need to be stored on the source. Those pages are stored in a dedicated cache (hash table) and are accessed by their address. The larger the cache size the better the chances are that the page has already been stored in the cache. A small cache size will result in high cache miss rate. Cache size can be changed before and during migration. Format ======= The compression format performs a XOR between the previous and current content of the page, where zero represents an unchanged value. The page data delta is represented by zero and non zero runs. A zero run is represented by its length (in bytes). A non zero run is represented by its length (in bytes) and the new data. The run length is encoded using ULEB128 (http://en.wikipedia.org/wiki/LEB128) There can be more than one valid encoding, the sender may send a longer encoding for the benefit of reducing computation cost. page = zrun nzrun | zrun nzrun page zrun = length nzrun = length byte... length = uleb128 encoded integer On the sender side XBZRLE is used as a compact delta encoding of page updates, retrieving the old page content from the cache (default size of 64MB). The receiving side uses the existing page's content and XBZRLE to decode the new page's content. This work was originally based on research results published VEE 2011: Evaluation of Delta Compression Techniques for Efficient Live Migration of Large Virtual Machines by Benoit, Svard, Tordsson and Elmroth. Additionally the delta encoder XBRLE was improved further using the XBZRLE instead. XBZRLE has a sustained bandwidth of 2-2.5 GB/s for typical workloads making it ideal for in-line, real-time encoding such as is needed for live-migration. Example old buffer: 1001 zeros 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 68 00 00 6b 00 6d 3074 zeros new buffer: 1001 zeros 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 68 00 00 67 00 69 3074 zeros encoded buffer: encoded length 24 e9 07 0f 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 03 01 67 01 01 69 Cache update strategy ===================== Keeping the hot pages in the cache is effective for decreasing cache misses. XBZRLE uses a counter as the age of each page. The counter will increase after each ram dirty bitmap sync. When a cache conflict is detected, XBZRLE will only evict pages in the cache that are older than a threshold. Usage ====================== 1. Verify the destination QEMU version is able to decode the new format. {qemu} info migrate_capabilities {qemu} xbzrle: off , ... 2. Activate xbzrle on both source and destination: {qemu} migrate_set_capability xbzrle on 3. Set the XBZRLE cache size - the cache size is in MBytes and should be a power of 2. The cache default value is 64MBytes. (on source only) {qemu} migrate_set_parameter xbzrle-cache-size 256m 4. Start outgoing migration {qemu} migrate -d tcp:destination.host:4444 {qemu} info migrate capabilities: xbzrle: on Migration status: active transferred ram: A kbytes remaining ram: B kbytes total ram: C kbytes total time: D milliseconds duplicate: E pages normal: F pages normal bytes: G kbytes cache size: H bytes xbzrle transferred: I kbytes xbzrle pages: J pages xbzrle cache miss: K pages xbzrle cache miss rate: L xbzrle encoding rate: M xbzrle overflow: N xbzrle cache miss: the number of cache misses to date - high cache-miss rate indicates that the cache size is set too low. xbzrle overflow: the number of overflows in the decoding which where the delta could not be compressed. This can happen if the changes in the pages are too large or there are many short changes; for example, changing every second byte (half a page). Testing: Testing indicated that live migration with XBZRLE was completed in 110 seconds, whereas without it would not be able to complete. A simple synthetic memory r/w load generator: .. include .. include .. int main() .. { .. char *buf = (char *) calloc(4096, 4096); .. while (1) { .. int i; .. for (i = 0; i < 4096 * 4; i++) { .. buf[i * 4096 / 4]++; .. } .. printf("."); .. } .. } --- ### Xen Save Devices State (`docs/xen-save-devices-state.txt`) = Save Devices = QEMU has code to load/save the state of the guest that it is running. These are two complementary operations. Saving the state just does that, saves the state for each device that the guest is running. These operations are normally used with migration (see migration.txt), however it is also possible to save the state of all devices to file, without saving the RAM or the block devices of the VM. The save operation is available as QMP command xen-save-devices-state. The binary format used in the file is the following: ------------------------------------------- 32 bit big endian: QEMU_VM_FILE_MAGIC 32 bit big endian: QEMU_VM_FILE_VERSION for_each_device { 8 bit: QEMU_VM_SECTION_FULL 32 bit big endian: section_id 8 bit: idstr (ID string) length string: idstr (ID string) 32 bit big endian: instance_id 32 bit big endian: version_id buffer: device specific data } 8 bit: QEMU_VM_EOF --- ### About Build Platforms (`docs/about/build-platforms.rst`) .. _Supported-build-platforms: Supported build platforms ========================= QEMU aims to support building and executing on multiple host OS platforms. This appendix outlines which platforms are the major build targets. These platforms are used as the basis for deciding upon the minimum required versions of 3rd party software QEMU depends on. The supported platforms are the targets for automated testing performed by the project when patches are submitted for review, and tested before and after merge. If a platform is not listed here, it does not imply that QEMU won't work. If an unlisted platform has comparable software versions to a listed platform, there is every expectation that it will work. Bug reports are welcome for problems encountered on unlisted platforms unless they are clearly older vintage than what is described here. Note that when considering software versions shipped in distros as support targets, QEMU considers only the version number, and assumes the features in that distro match the upstream release with the same version. In other words, if a distro backports extra features to the software in their distro, QEMU upstream code will not add explicit support for those backports, unless the feature is auto-detectable in a manner that works for the upstream releases too. The `Repology`_ site is a useful resource to identify currently shipped versions of software in various operating systems, though it does not cover all distros listed below. Supported host architectures ---------------------------- Those hosts are officially supported, with various accelerators: .. list-table:: :header-rows: 1 * - CPU Architecture - Accelerators * - Arm - kvm (64 bit only), tcg, xen * - MIPS (little endian only) - kvm, tcg * - PPC - kvm, tcg * - RISC-V - kvm, tcg * - s390x - kvm, tcg * - SPARC - tcg * - x86 - hax, hvf (64 bit only), kvm, nvmm, tcg, whpx (64 bit only), xen Other host architectures are not supported. It is possible to build QEMU system emulation on an unsupported host architecture using the configure ``--enable-tcg-interpreter`` option to enable the TCI support, but note that this is very slow and is not recommended for normal use. QEMU user emulation requires host-specific support for signal handling, therefore TCI won't help on unsupported host architectures. Non-supported architectures may be removed in the future following the :ref:`deprecation process`. Linux OS, macOS, FreeBSD, NetBSD, OpenBSD ----------------------------------------- The project aims to support the most recent major version at all times. Support for the previous major version will be dropped 2 years after the new major version is released or when the vendor itself drops support, whichever comes first. In this context, third-party efforts to extend the lifetime of a distro are not considered, even when they are endorsed by the vendor (eg. Debian LTS); the same is true of repositories that contain packages backported from later releases (e.g. Debian backports). Within each major release, only the most recent minor release is considered. For the purposes of identifying supported software versions available on Linux, the project will look at CentOS, Debian, Fedora, openSUSE, RHEL, SLES and Ubuntu LTS. Other distros will be assumed to ship similar software versions. For FreeBSD and OpenBSD, decisions will be made based on the contents of the respective ports repository, while NetBSD will use the pkgsrc repository. For macOS, `Homebrew`_ will be used, although `MacPorts`_ is expected to carry similar versions. Windows ------- The project aims to support the two most recent versions of Windows that are still supported by the vendor. The minimum Windows API that is currently targeted is "Windows 8", so theoretically the QEMU binaries can still be run on older versions of Windows, too. However, such old versions of Windows are not tested anymore, so it is recommended to use one of the latest versions of Windows instead. The project supports building QEMU with current versions of the MinGW toolchain, either hosted on Linux (Debian/Fedora) or via `MSYS2`_ on Windows. A more recent Windows version is always preferred as it is less likely to have problems with building via MSYS2. The building process of QEMU involves some Python scripts that call os.symlink() which needs special attention for the build process to successfully complete. On newer versions of Windows 10, unprivileged accounts can create symlinks if Developer Mode is enabled. When Developer Mode is not available/enabled, the SeCreateSymbolicLinkPrivilege privilege is required, or the process must be run as an administrator. .. _Homebrew: https://brew.sh/ .. _MacPorts: https://www.macports.org/ .. _MSYS2: https://www.msys2.org/ .. _Repology: https://repology.org/ --- ### About Deprecated (`docs/about/deprecated.rst`) .. _Deprecated features: Deprecated features =================== In general features are intended to be supported indefinitely once introduced into QEMU. In the event that a feature needs to be removed, it will be listed in this section. The feature will remain functional for the release in which it was deprecated and one further release. After these two releases, the feature is liable to be removed. Deprecated features may also generate warnings on the console when QEMU starts up, or if activated via a monitor command, however, this is not a mandatory requirement. Prior to the 2.10.0 release there was no official policy on how long features would be deprecated prior to their removal, nor any documented list of which features were deprecated. Thus any features deprecated prior to 2.10.0 will be treated as if they were first deprecated in the 2.10.0 release. What follows is a list of all features currently marked as deprecated. System emulator command line arguments -------------------------------------- ``QEMU_AUDIO_`` environment variables and ``-audio-help`` (since 4.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The ``-audiodev`` argument is now the preferred way to specify audio backend settings instead of environment variables. To ease migration to the new format, the ``-audiodev-help`` option can be used to convert the current values of the environment variables to ``-audiodev`` options. Creating sound card devices and vnc without ``audiodev=`` property (since 4.2) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' When not using the deprecated legacy audio config, each sound card should specify an ``audiodev=`` property. Additionally, when using vnc, you should specify an ``audiodev=`` property if you plan to transmit audio through the VNC protocol. ``-chardev`` backend aliases ``tty`` and ``parport`` (since 6.0) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' ``tty`` and ``parport`` are aliases that will be removed. Instead, the actual backend names ``serial`` and ``parallel`` should be used. Short-form boolean options (since 6.0) '''''''''''''''''''''''''''''''''''''' Boolean options such as ``share=on``/``share=off`` could be written in short form as ``share`` and ``noshare``. This is now deprecated and will cause a warning. ``delay`` option for socket character devices (since 6.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The replacement for the ``nodelay`` short-form boolean option is ``nodelay=on`` rather than ``delay=off``. Userspace local APIC with KVM (x86, since 6.0) '''''''''''''''''''''''''''''''''''''''''''''' Using ``-M kernel-irqchip=off`` with x86 machine types that include a local APIC is deprecated. The ``split`` setting is supported, as is using ``-M kernel-irqchip=off`` with the ISA PC machine type. hexadecimal sizes with scaling multipliers (since 6.0) '''''''''''''''''''''''''''''''''''''''''''''''''''''' Input parameters that take a size value should only use a size suffix (such as 'k' or 'M') when the base is written in decimal, and not when the value is hexadecimal. That is, '0x20M' is deprecated, and should be written either as '32M' or as '0x2000000'. ``-spice password=string`` (since 6.0) '''''''''''''''''''''''''''''''''''''' This option is insecure because the SPICE password remains visible in the process listing. This is replaced by the new ``password-secret`` option which lets the password be securely provided on the command line using a ``secret`` object instance. ``-smp`` ("parameter=0" SMP configurations) (since 6.2) ''''''''''''''''''''''''''''''''''''''''''''''''''''''' Specified CPU topology parameters must be greater than zero. In the SMP configuration, users should either provide a CPU topology parameter with a reasonable value (greater than zero) or just omit it and QEMU will compute the missing value. However, historically it was implicitly allowed for users to provide a parameter with zero value, which is meaningless and could also possibly cause unexpected results in the -smp parsing. So support for this kind of configurations (e.g. -smp 8,sockets=0) is deprecated since 6.2 and will be removed in the near future, users have to ensure that all the topology members described with -smp are greater than zero. Plugin argument passing through ``arg=`` (since 6.1) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Passing TCG plugins arguments through ``arg=`` is redundant is makes the command-line less readable, especially when the argument itself consist of a name and a value, e.g. ``-plugin plugin_name,arg="arg_name=arg_value"``. Therefore, the usage of ``arg`` is redundant. Single-word arguments are treated as short-form boolean values, and passed to plugins as ``arg_name=on``. However, short-form booleans are deprecated and full explicit ``arg_name=on`` form is preferred. ``-drive if=none`` for the sifive_u OTP device (since 6.2) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Using ``-drive if=none`` to configure the OTP device of the sifive_u RISC-V machine is deprecated. Use ``-drive if=pflash`` instead. QEMU Machine Protocol (QMP) commands ------------------------------------ ``blockdev-open-tray``, ``blockdev-close-tray`` argument ``device`` (since 2.8) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Use argument ``id`` instead. ``eject`` argument ``device`` (since 2.8) ''''''''''''''''''''''''''''''''''''''''' Use argument ``id`` instead. ``blockdev-change-medium`` argument ``device`` (since 2.8) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Use argument ``id`` instead. ``block_set_io_throttle`` argument ``device`` (since 2.8) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Use argument ``id`` instead. ``blockdev-add`` empty string argument ``backing`` (since 2.10) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Use argument value ``null`` instead. ``block-commit`` arguments ``base`` and ``top`` (since 3.1) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Use arguments ``base-node`` and ``top-node`` instead. ``nbd-server-add`` and ``nbd-server-remove`` (since 5.2) '''''''''''''''''''''''''''''''''''''''''''''''''''''''' Use the more generic commands ``block-export-add`` and ``block-export-del`` instead. As part of this deprecation, where ``nbd-server-add`` used a single ``bitmap``, the new ``block-export-add`` uses a list of ``bitmaps``. ``query-qmp-schema`` return value member ``values`` (since 6.2) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Member ``values`` in return value elements with meta-type ``enum`` is deprecated. Use ``members`` instead. ``drive-backup`` (since 6.2) '''''''''''''''''''''''''''' Use ``blockdev-backup`` in combination with ``blockdev-add`` instead. This change primarily separates the creation/opening process of the backup target with explicit, separate steps. ``blockdev-backup`` uses mostly the same arguments as ``drive-backup``, except the ``format`` and ``mode`` options are removed in favor of using explicit ``blockdev-create`` and ``blockdev-add`` calls. See :doc:`/interop/live-block-operations` for details. Incorrectly typed ``device_add`` arguments (since 6.2) '''''''''''''''''''''''''''''''''''''''''''''''''''''' Due to shortcomings in the internal implementation of ``device_add``, QEMU incorrectly accepts certain invalid arguments: Any object or list arguments are silently ignored. Other argument types are not checked, but an implicit conversion happens, so that e.g. string values can be assigned to integer device properties or vice versa. This is a bug in QEMU that will be fixed in the future so that previously accepted incorrect commands will return an error. Users should make sure that all arguments passed to ``device_add`` are consistent with the documented property types. ``query-sgx`` return value member ``section-size`` (since 7.0) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Member ``section-size`` in return value elements with meta-type ``uint64`` is deprecated. Use ``sections`` instead. ``query-sgx-capabilities`` return value member ``section-size`` (since 7.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Member ``section-size`` in return value elements with meta-type ``uint64`` is deprecated. Use ``sections`` instead. System accelerators ------------------- MIPS ``Trap-and-Emul`` KVM support (since 6.0) '''''''''''''''''''''''''''''''''''''''''''''' The MIPS ``Trap-and-Emul`` KVM host and guest support has been removed from Linux upstream kernel, declare it deprecated. Host Architectures ------------------ BE MIPS (since 7.2) ''''''''''''''''''' As Debian 10 ("Buster") moved into LTS the big endian 32 bit version of MIPS moved out of support making it hard to maintain our cross-compilation CI tests of the architecture. As we no longer have CI coverage support may bitrot away before the deprecation process completes. The little endian variants of MIPS (both 32 and 64 bit) are still a supported host architecture. QEMU API (QAPI) events ---------------------- ``MEM_UNPLUG_ERROR`` (since 6.2) '''''''''''''''''''''''''''''''''''''''''''''''''''''''' Use the more generic event ``DEVICE_UNPLUG_GUEST_ERROR`` instead. System emulator machines ------------------------ Arm ``virt`` machine ``dtb-kaslr-seed`` property '''''''''''''''''''''''''''''''''''''''''''''''' The ``dtb-kaslr-seed`` property on the ``virt`` board has been deprecated; use the new name ``dtb-randomness`` instead. The new name better reflects the way this property affects all random data within the device tree blob, not just the ``kaslr-seed`` node. ``pc-i440fx-1.4`` up to ``pc-i440fx-1.7`` (since 7.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''' These old machine types are quite neglected nowadays and thus might have various pitfalls with regards to live migration. Use a newer machine type instead. Backend options --------------- Using non-persistent backing file with pmem=on (since 6.1) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''' This option is used when ``memory-backend-file`` is consumed by emulated NVDIMM device. However enabling ``memory-backend-file.pmem`` option, when backing file is (a) not DAX capable or (b) not on a filesystem that support direct mapping of persistent memory, is not safe and may lead to data loss or corruption in case of host crash. Options are: - modify VM configuration to set ``pmem=off`` to continue using fake NVDIMM (without persistence guaranties) with backing file on non DAX storage - move backing file to NVDIMM storage and keep ``pmem=on`` (to have NVDIMM with persistence guaranties). Device options -------------- Emulated device options ''''''''''''''''''''''' ``-device virtio-blk,scsi=on|off`` (since 5.0) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The virtio-blk SCSI passthrough feature is a legacy VIRTIO feature. VIRTIO 1.0 and later do not support it because the virtio-scsi device was introduced for full SCSI support. Use virtio-scsi instead when SCSI passthrough is required. Note this also applies to ``-device virtio-blk-pci,scsi=on|off``, which is an alias. ``-device sga`` (since 6.2) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ``sga`` device loads an option ROM for x86 targets which enables SeaBIOS to send messages to the serial console. SeaBIOS 1.11.0 onwards contains native support for this feature and thus use of the option ROM approach is obsolete. The native SeaBIOS support can be activated by using ``-machine graphics=off``. ``-device nvme-ns,eui64-default=on|off`` (since 7.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In QEMU versions 6.1, 6.2 and 7.0, the ``nvme-ns`` generates an EUI-64 identifier that is not globally unique. If an EUI-64 identifier is required, the user must set it explicitly using the ``nvme-ns`` device parameter ``eui64``. ``-device nvme,use-intel-id=on|off`` (since 7.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ``nvme`` device originally used a PCI Vendor/Device Identifier combination from Intel that was not properly allocated. Since version 5.2, the controller has used a properly allocated identifier. Deprecate the ``use-intel-id`` machine compatibility parameter. Block device options '''''''''''''''''''' ``"backing": ""`` (since 2.12) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In order to prevent QEMU from automatically opening an image's backing chain, use ``"backing": null`` instead. ``rbd`` keyvalue pair encoded filenames: ``""`` (since 3.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Options for ``rbd`` should be specified according to its runtime options, like other block drivers. Legacy parsing of keyvalue pair encoded filenames is useful to open images with the old format for backing files; These image files should be updated to use the current format. Example of legacy encoding:: json:{"file.driver":"rbd", "file.filename":"rbd:rbd/name"} The above, converted to the current supported format:: json:{"file.driver":"rbd", "file.pool":"rbd", "file.image":"name"} Backwards compatibility ----------------------- Runnability guarantee of CPU models (since 4.1) ''''''''''''''''''''''''''''''''''''''''''''''' Previous versions of QEMU never changed existing CPU models in ways that introduced additional host software or hardware requirements to the VM. This allowed management software to safely change the machine type of an existing VM without introducing new requirements ("runnability guarantee"). This prevented CPU models from being updated to include CPU vulnerability mitigations, leaving guests vulnerable in the default configuration. The CPU model runnability guarantee won't apply anymore to existing CPU models. Management software that needs runnability guarantees must resolve the CPU model aliases using the ``alias-of`` field returned by the ``query-cpu-definitions`` QMP command. While those guarantees are kept, the return value of ``query-cpu-definitions`` will have existing CPU model aliases point to a version that doesn't break runnability guarantees (specifically, version 1 of those CPU models). In future QEMU versions, aliases will point to newer CPU model versions depending on the machine type, so management software must resolve CPU model aliases before starting a virtual machine. Tools ----- virtiofsd ''''''''' There is a new Rust implementation of ``virtiofsd`` at ``https://gitlab.com/virtio-fs/virtiofsd``; since this is now marked stable, new development should be done on that rather than the existing C version in the QEMU tree. The C version will still accept fixes and patches that are already in development for the moment, but will eventually be deleted from this tree. New deployments should use the Rust version, and existing systems should consider moving to it. The command line and feature set is very close and moving should be simple. QEMU guest agent ---------------- ``--blacklist`` command line option (since 7.2) ''''''''''''''''''''''''''''''''''''''''''''''' ``--blacklist`` has been replaced by ``--block-rpcs`` (which is a better wording for what this option does). The short form ``-b`` still stays the same and thus is the preferred way for scripts that should run with both, older and future versions of QEMU. ``blacklist`` config file option (since 7.2) '''''''''''''''''''''''''''''''''''''''''''' The ``blacklist`` config file option has been renamed to ``block-rpcs`` (to be in sync with the renaming of the corresponding command line option). --- ### About Index (`docs/about/index.rst`) ---------- About QEMU ---------- QEMU is a generic and open source machine emulator and virtualizer. QEMU can be used in several different ways. The most common is for "system emulation", where it provides a virtual model of an entire machine (CPU, memory and emulated devices) to run a guest OS. In this mode the CPU may be fully emulated, or it may work with a hypervisor such as KVM, Xen, Hax or Hypervisor.Framework to allow the guest to run directly on the host CPU. The second supported way to use QEMU is "user mode emulation", where QEMU can launch processes compiled for one CPU on another CPU. In this mode the CPU is always emulated. QEMU also provides a number of standalone commandline utilities, such as the ``qemu-img`` disk image utility that allows you to create, convert and modify disk images. .. toctree:: :maxdepth: 2 build-platforms deprecated removed-features license --- ### About License (`docs/about/license.rst`) .. _License: License ======= QEMU is a trademark of Fabrice Bellard. QEMU is released under the `GNU General Public License `__, version 2. Parts of QEMU have specific licenses, see file `LICENSE `__. --- ### About Removed Features (`docs/about/removed-features.rst`) Removed features ================ What follows is a record of recently removed, formerly deprecated features that serves as a record for users who have encountered trouble after a recent upgrade. System emulator command line arguments -------------------------------------- ``-hdachs`` (removed in 2.12) ''''''''''''''''''''''''''''' The geometry defined by ``-hdachs c,h,s,t`` should now be specified via ``-device ide-hd,drive=dr,cyls=c,heads=h,secs=s,bios-chs-trans=t`` (together with ``-drive if=none,id=dr,...``). ``-net channel`` (removed in 2.12) '''''''''''''''''''''''''''''''''' This option has been replaced by ``-net user,guestfwd=...``. ``-net dump`` (removed in 2.12) ''''''''''''''''''''''''''''''' ``-net dump[,vlan=n][,file=filename][,len=maxlen]`` has been replaced by ``-object filter-dump,id=id,netdev=dev[,file=filename][,maxlen=maxlen]``. Note that the new syntax works with netdev IDs instead of the old "vlan" hubs. ``-no-kvm-pit`` (removed in 2.12) ''''''''''''''''''''''''''''''''' This was just a dummy option that has been ignored, since the in-kernel PIT cannot be disabled separately from the irqchip anymore. A similar effect (which also disables the KVM IOAPIC) can be obtained with ``-M kernel_irqchip=split``. ``-tdf`` (removed in 2.12) '''''''''''''''''''''''''' There is no replacement, the ``-tdf`` option has just been ignored since the behaviour that could be changed by this option in qemu-kvm is now the default when using the KVM PIT. It still can be requested explicitly using ``-global kvm-pit.lost_tick_policy=delay``. ``-drive secs=s``, ``-drive heads=h`` & ``-drive cyls=c`` (removed in 3.0) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The drive geometry should now be specified via ``-device ...,drive=dr,cyls=c,heads=h,secs=s`` (together with ``-drive if=none,id=dr,...``). ``-drive serial=``, ``-drive trans=`` & ``-drive addr=`` (removed in 3.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Use ``-device ...,drive=dr,serial=r,bios-chs-trans=t,addr=a`` instead (together with ``-drive if=none,id=dr,...``). ``-net ...,vlan=x`` (removed in 3.0) '''''''''''''''''''''''''''''''''''' The term "vlan" was very confusing for most users in this context (it's about specifying a hub ID, not about IEEE 802.1Q or something similar), so this has been removed. To connect one NIC frontend with a network backend, either use ``-nic ...`` (e.g. for on-board NICs) or use ``-netdev ...,id=n`` together with ``-device ...,netdev=n`` (for full control over pluggable NICs). To connect multiple NICs or network backends via a hub device (which is what vlan did), use ``-nic hubport,hubid=x,...`` or ``-netdev hubport,id=n,hubid=x,...`` (with ``-device ...,netdev=n``) instead. ``-no-kvm-irqchip`` (removed in 3.0) '''''''''''''''''''''''''''''''''''' Use ``-machine kernel_irqchip=off`` instead. ``-no-kvm-pit-reinjection`` (removed in 3.0) '''''''''''''''''''''''''''''''''''''''''''' Use ``-global kvm-pit.lost_tick_policy=discard`` instead. ``-balloon`` (removed in 3.1) ''''''''''''''''''''''''''''' The ``-balloon virtio`` option has been replaced by ``-device virtio-balloon``. The ``-balloon none`` option was a no-op and has no replacement. ``-bootp`` (removed in 3.1) ''''''''''''''''''''''''''' The ``-bootp /some/file`` argument is replaced by either ``-netdev user,id=x,bootp=/some/file`` (for pluggable NICs, accompanied with ``-device ...,netdev=x``), or ``-nic user,bootp=/some/file`` (for on-board NICs). The new syntax allows different settings to be provided per NIC. ``-redir`` (removed in 3.1) ''''''''''''''''''''''''''' The ``-redir [tcp|udp]:hostport:[guestaddr]:guestport`` option is replaced by either ``-netdev user,id=x,hostfwd=[tcp|udp]:[hostaddr]:hostport-[guestaddr]:guestport`` (for pluggable NICs, accompanied with ``-device ...,netdev=x``) or by the option ``-nic user,hostfwd=[tcp|udp]:[hostaddr]:hostport-[guestaddr]:guestport`` (for on-board NICs). The new syntax allows different settings to be provided per NIC. ``-smb`` (removed in 3.1) ''''''''''''''''''''''''' The ``-smb /some/dir`` argument is replaced by either ``-netdev user,id=x,smb=/some/dir`` (for pluggable NICs, accompanied with ``-device ...,netdev=x``), or ``-nic user,smb=/some/dir`` (for on-board NICs). The new syntax allows different settings to be provided per NIC. ``-tftp`` (removed in 3.1) '''''''''''''''''''''''''' The ``-tftp /some/dir`` argument is replaced by either ``-netdev user,id=x,tftp=/some/dir`` (for pluggable NICs, accompanied with ``-device ...,netdev=x``), or ``-nic user,tftp=/some/dir`` (for embedded NICs). The new syntax allows different settings to be provided per NIC. ``-localtime`` (removed in 3.1) ''''''''''''''''''''''''''''''' Replaced by ``-rtc base=localtime``. ``-nodefconfig`` (removed in 3.1) ''''''''''''''''''''''''''''''''' Use ``-no-user-config`` instead. ``-rtc-td-hack`` (removed in 3.1) ''''''''''''''''''''''''''''''''' Use ``-rtc driftfix=slew`` instead. ``-startdate`` (removed in 3.1) ''''''''''''''''''''''''''''''' Replaced by ``-rtc base=date``. ``-vnc ...,tls=...``, ``-vnc ...,x509=...`` & ``-vnc ...,x509verify=...`` (removed in 3.1) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The "tls-creds" option should be used instead to point to a "tls-creds-x509" object created using "-object". ``-mem-path`` fallback to RAM (removed in 5.0) '''''''''''''''''''''''''''''''''''''''''''''' If guest RAM allocation from file pointed by ``mem-path`` failed, QEMU was falling back to allocating from RAM, which might have resulted in unpredictable behavior since the backing file specified by the user as ignored. Currently, users are responsible for making sure the backing storage specified with ``-mem-path`` can actually provide the guest RAM configured with ``-m`` and QEMU fails to start up if RAM allocation is unsuccessful. ``-net ...,name=...`` (removed in 5.1) '''''''''''''''''''''''''''''''''''''' The ``name`` parameter of the ``-net`` option was a synonym for the ``id`` parameter, which should now be used instead. ``-numa node,mem=...`` (removed in 5.1) ''''''''''''''''''''''''''''''''''''''' The parameter ``mem`` of ``-numa node`` was used to assign a part of guest RAM to a NUMA node. But when using it, it's impossible to manage a specified RAM chunk on the host side (like bind it to a host node, setting bind policy, ...), so the guest ends up with the fake NUMA configuration with suboptiomal performance. However since 2014 there is an alternative way to assign RAM to a NUMA node using parameter ``memdev``, which does the same as ``mem`` and adds means to actually manage node RAM on the host side. Use parameter ``memdev`` with *memory-backend-ram* backend as replacement for parameter ``mem`` to achieve the same fake NUMA effect or a properly configured *memory-backend-file* backend to actually benefit from NUMA configuration. New machine versions (since 5.1) will not accept the option but it will still work with old machine types. User can check the QAPI schema to see if the legacy option is supported by looking at MachineInfo::numa-mem-supported property. ``-numa`` node (without memory specified) (removed in 5.2) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Splitting RAM by default between NUMA nodes had the same issues as ``mem`` parameter with the difference that the role of the user plays QEMU using implicit generic or board specific splitting rule. Use ``memdev`` with *memory-backend-ram* backend or ``mem`` (if it's supported by used machine type) to define mapping explicitly instead. Users of existing VMs, wishing to preserve the same RAM distribution, should configure it explicitly using ``-numa node,memdev`` options. Current RAM distribution can be retrieved using HMP command ``info numa`` and if separate memory devices (pc|nv-dimm) are present use ``info memory-device`` and subtract device memory from output of ``info numa``. ``-smp`` (invalid topologies) (removed in 5.2) '''''''''''''''''''''''''''''''''''''''''''''' CPU topology properties should describe whole machine topology including possible CPUs. However, historically it was possible to start QEMU with an incorrect topology where *n* <= *sockets* * *cores* * *threads* < *maxcpus*, which could lead to an incorrect topology enumeration by the guest. Support for invalid topologies is removed, the user must ensure topologies described with -smp include all possible cpus, i.e. *sockets* * *cores* * *threads* = *maxcpus*. ``-machine enforce-config-section=on|off`` (removed in 5.2) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The ``enforce-config-section`` property was replaced by the ``-global migration.send-configuration={on|off}`` option. ``-no-kvm`` (removed in 5.2) '''''''''''''''''''''''''''' The ``-no-kvm`` argument was a synonym for setting ``-machine accel=tcg``. ``-realtime`` (removed in 6.0) '''''''''''''''''''''''''''''' The ``-realtime mlock=on|off`` argument has been replaced by the ``-overcommit mem-lock=on|off`` argument. ``-show-cursor`` option (removed in 6.0) '''''''''''''''''''''''''''''''''''''''' Use ``-display sdl,show-cursor=on``, ``-display gtk,show-cursor=on`` or ``-display default,show-cursor=on`` instead. ``-tb-size`` option (removed in 6.0) '''''''''''''''''''''''''''''''''''' QEMU 5.0 introduced an alternative syntax to specify the size of the translation block cache, ``-accel tcg,tb-size=``. ``-usbdevice audio`` (removed in 6.0) ''''''''''''''''''''''''''''''''''''' This option lacked the possibility to specify an audio backend device. Use ``-device usb-audio`` now instead (and specify a corresponding USB host controller or ``-usb`` if necessary). ``-vnc acl`` (removed in 6.0) ''''''''''''''''''''''''''''' The ``acl`` option to the ``-vnc`` argument has been replaced by the ``tls-authz`` and ``sasl-authz`` options. ``-mon ...,control=readline,pretty=on|off`` (removed in 6.0) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The ``pretty=on|off`` switch has no effect for HMP monitors and its use is rejected. ``-drive file=json:{...{'driver':'file'}}`` (removed in 6.0) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The 'file' driver for drives is no longer appropriate for character or host devices and will only accept regular files (S_IFREG). The correct driver for these file types is 'host_cdrom' or 'host_device' as appropriate. Floppy controllers' drive properties (removed in 6.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''' Use ``-device floppy,...`` instead. When configuring onboard floppy controllers :: -global isa-fdc.driveA=... -global sysbus-fdc.driveA=... -global SUNW,fdtwo.drive=... become :: -device floppy,unit=0,drive=... and :: -global isa-fdc.driveB=... -global sysbus-fdc.driveB=... become :: -device floppy,unit=1,drive=... When plugging in a floppy controller :: -device isa-fdc,...,driveA=... becomes :: -device isa-fdc,... -device floppy,unit=0,drive=... and :: -device isa-fdc,...,driveB=... becomes :: -device isa-fdc,... -device floppy,unit=1,drive=... ``-drive`` with bogus interface type (removed in 6.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''' Drives with interface types other than ``if=none`` are for onboard devices. Drives the board doesn't pick up can no longer be used with -device. Use ``if=none`` instead. ``-usbdevice ccid`` (removed in 6.0) ''''''''''''''''''''''''''''''''''''' This option was undocumented and not used in the field. Use ``-device usb-ccid`` instead. RISC-V firmware not booted by default (removed in 5.1) '''''''''''''''''''''''''''''''''''''''''''''''''''''' QEMU 5.1 changes the default behaviour from ``-bios none`` to ``-bios default`` for the RISC-V ``virt`` machine and ``sifive_u`` machine. ``-no-quit`` (removed in 7.0) ''''''''''''''''''''''''''''' The ``-no-quit`` was a synonym for ``-display ...,window-close=off`` which should be used instead. ``--enable-fips`` (removed in 7.1) '''''''''''''''''''''''''''''''''' This option restricted usage of certain cryptographic algorithms when the host is operating in FIPS mode. If FIPS compliance is required, QEMU should be built with the ``libgcrypt`` or ``gnutls`` library enabled as a cryptography provider. Neither the ``nettle`` library, or the built-in cryptography provider are supported on FIPS enabled hosts. ``-writeconfig`` (removed in 7.1) ''''''''''''''''''''''''''''''''' The ``-writeconfig`` option was not able to serialize the entire contents of the QEMU command line. It is thus considered a failed experiment and removed without a replacement. ``loaded`` property of ``secret`` and ``secret_keyring`` objects (removed in 7.1) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The ``loaded=on`` option in the command line or QMP ``object-add`` either had no effect (if ``loaded`` was the last option) or caused options to be effectively ignored as if they were not given. The property is therefore useless and should simply be removed. ``opened`` property of ``rng-*`` objects (removed in 7.1) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The ``opened=on`` option in the command line or QMP ``object-add`` either had no effect (if ``opened`` was the last option) or caused errors. The property is therefore useless and should simply be removed. ``-display sdl,window_close=...`` (removed in 7.1) '''''''''''''''''''''''''''''''''''''''''''''''''' Use ``-display sdl,window-close=...`` instead (i.e. with a minus instead of an underscore between "window" and "close"). ``-alt-grab`` and ``-display sdl,alt_grab=on`` (removed in 7.1) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Use ``-display sdl,grab-mod=lshift-lctrl-lalt`` instead. ``-ctrl-grab`` and ``-display sdl,ctrl_grab=on`` (removed in 7.1) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Use ``-display sdl,grab-mod=rctrl`` instead. ``-sdl`` (removed in 7.1) ''''''''''''''''''''''''' Use ``-display sdl`` instead. ``-curses`` (removed in 7.1) '''''''''''''''''''''''''''' Use ``-display curses`` instead. Creating sound card devices using ``-soundhw`` (removed in 7.1) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Sound card devices should be created using ``-device`` or ``-audio``. The exception is ``pcspk`` which can be activated using ``-machine pcspk-audiodev=``. ``-watchdog`` (since 7.2) ''''''''''''''''''''''''' Use ``-device`` instead. QEMU Machine Protocol (QMP) commands ------------------------------------ ``block-dirty-bitmap-add`` "autoload" parameter (removed in 4.2) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The "autoload" parameter has been ignored since 2.12.0. All bitmaps are automatically loaded from qcow2 images. ``cpu-add`` (removed in 5.2) '''''''''''''''''''''''''''' Use ``device_add`` for hotplugging vCPUs instead of ``cpu-add``. See documentation of ``query-hotpluggable-cpus`` for additional details. ``change`` (removed in 6.0) ''''''''''''''''''''''''''' Use ``blockdev-change-medium`` or ``change-vnc-password`` or ``display-update`` instead. ``query-events`` (removed in 6.0) ''''''''''''''''''''''''''''''''' The ``query-events`` command has been superseded by the more powerful and accurate ``query-qmp-schema`` command. ``migrate_set_cache_size`` and ``query-migrate-cache-size`` (removed in 6.0) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Use ``migrate_set_parameter`` and ``info migrate_parameters`` instead. ``migrate_set_downtime`` and ``migrate_set_speed`` (removed in 6.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Use ``migrate_set_parameter`` instead. ``query-cpus`` (removed in 6.0) ''''''''''''''''''''''''''''''' The ``query-cpus`` command is replaced by the ``query-cpus-fast`` command. ``query-cpus-fast`` ``arch`` output member (removed in 6.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The ``arch`` output member of the ``query-cpus-fast`` command is replaced by the ``target`` output member. chardev client socket with ``wait`` option (removed in 6.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Character devices creating sockets in client mode should not specify the 'wait' field, which is only applicable to sockets in server mode ``query-named-block-nodes`` result ``encryption_key_missing`` (removed in 6.0) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Removed with no replacement. ``query-block`` result ``inserted.encryption_key_missing`` (removed in 6.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Removed with no replacement. ``query-named-block-nodes`` and ``query-block`` result dirty-bitmaps[i].status (removed in 6.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The ``status`` field of the ``BlockDirtyInfo`` structure, returned by these commands is removed. Two new boolean fields, ``recording`` and ``busy`` effectively replace it. ``query-block`` result field ``dirty-bitmaps`` (removed in 6.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The ``dirty-bitmaps`` field of the ``BlockInfo`` structure, returned by the query-block command is itself now removed. The ``dirty-bitmaps`` field of the ``BlockDeviceInfo`` struct should be used instead, which is the type of the ``inserted`` field in query-block replies, as well as the type of array items in query-named-block-nodes. ``object-add`` option ``props`` (removed in 6.0) '''''''''''''''''''''''''''''''''''''''''''''''' Specify the properties for the object as top-level arguments instead. Human Monitor Protocol (HMP) commands ------------------------------------- ``usb_add`` and ``usb_remove`` (removed in 2.12) '''''''''''''''''''''''''''''''''''''''''''''''' Replaced by ``device_add`` and ``device_del`` (use ``device_add help`` for a list of available devices). ``host_net_add`` and ``host_net_remove`` (removed in 2.12) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Replaced by ``netdev_add`` and ``netdev_del``. The ``hub_id`` parameter of ``hostfwd_add`` / ``hostfwd_remove`` (removed in 5.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The ``[hub_id name]`` parameter tuple of the 'hostfwd_add' and 'hostfwd_remove' HMP commands has been replaced by ``netdev_id``. ``cpu-add`` (removed in 5.2) '''''''''''''''''''''''''''' Use ``device_add`` for hotplugging vCPUs instead of ``cpu-add``. See documentation of ``query-hotpluggable-cpus`` for additional details. ``change vnc TARGET`` (removed in 6.0) '''''''''''''''''''''''''''''''''''''' No replacement. The ``change vnc password`` and ``change DEVICE MEDIUM`` commands are not affected. ``acl_show``, ``acl_reset``, ``acl_policy``, ``acl_add``, ``acl_remove`` (removed in 6.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The ``acl_show``, ``acl_reset``, ``acl_policy``, ``acl_add``, and ``acl_remove`` commands were removed with no replacement. Authorization for VNC should be performed using the pluggable QAuthZ objects. ``migrate-set-cache-size`` and ``info migrate-cache-size`` (removed in 6.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Use ``migrate-set-parameters`` and ``info migrate-parameters`` instead. ``migrate_set_downtime`` and ``migrate_set_speed`` (removed in 6.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Use ``migrate-set-parameters`` instead. ``info cpustats`` (removed in 6.1) '''''''''''''''''''''''''''''''''' This command didn't produce any output already. Removed with no replacement. Guest Emulator ISAs ------------------- RISC-V ISA privilege specification version 1.09.1 (removed in 5.1) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The RISC-V ISA privilege specification version 1.09.1 has been removed. QEMU supports both the newer version 1.10.0 and the ratified version 1.11.0, these should be used instead of the 1.09.1 version. System emulator CPUS -------------------- KVM guest support on 32-bit Arm hosts (removed in 5.2) '''''''''''''''''''''''''''''''''''''''''''''''''''''' The Linux kernel has dropped support for allowing 32-bit Arm systems to host KVM guests as of the 5.7 kernel. Accordingly, QEMU is deprecating its support for this configuration and will remove it in a future version. Running 32-bit guests on a 64-bit Arm host remains supported. RISC-V ISA Specific CPUs (removed in 5.1) ''''''''''''''''''''''''''''''''''''''''' The RISC-V cpus with the ISA version in the CPU name have been removed. The four CPUs are: ``rv32gcsu-v1.9.1``, ``rv32gcsu-v1.10.0``, ``rv64gcsu-v1.9.1`` and ``rv64gcsu-v1.10.0``. Instead the version can be specified via the CPU ``priv_spec`` option when using the ``rv32`` or ``rv64`` CPUs. RISC-V no MMU CPUs (removed in 5.1) ''''''''''''''''''''''''''''''''''' The RISC-V no MMU cpus have been removed. The two CPUs: ``rv32imacu-nommu`` and ``rv64imacu-nommu`` can no longer be used. Instead the MMU status can be specified via the CPU ``mmu`` option when using the ``rv32`` or ``rv64`` CPUs. ``compat`` property of server class POWER CPUs (removed in 6.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The ``max-cpu-compat`` property of the ``pseries`` machine type should be used instead. ``moxie`` CPU (removed in 6.1) '''''''''''''''''''''''''''''' Nobody was using this CPU emulation in QEMU, and there were no test images available to make sure that the code is still working, so it has been removed without replacement. ``lm32`` CPUs (removed in 6.1) '''''''''''''''''''''''''''''' The only public user of this architecture was the milkymist project, which has been dead for years; there was never an upstream Linux port. Removed without replacement. ``unicore32`` CPUs (removed in 6.1) ''''''''''''''''''''''''''''''''''' Support for this CPU was removed from the upstream Linux kernel, and there is no available upstream toolchain to build binaries for it. Removed without replacement. x86 ``Icelake-Client`` CPU (removed in 7.1) ''''''''''''''''''''''''''''''''''''''''''' There isn't ever Icelake Client CPU, it is some wrong and imaginary one. Use ``Icelake-Server`` instead. System emulator machines ------------------------ ``s390-virtio`` (removed in 2.6) '''''''''''''''''''''''''''''''' Use the ``s390-ccw-virtio`` machine instead. The m68k ``dummy`` machine (removed in 2.9) ''''''''''''''''''''''''''''''''''''''''''' Use the ``none`` machine with the ``loader`` device instead. ``xlnx-ep108`` (removed in 3.0) ''''''''''''''''''''''''''''''' The EP108 was an early access development board that is no longer used. Use the ``xlnx-zcu102`` machine instead. ``spike_v1.9.1`` and ``spike_v1.10`` (removed in 5.1) ''''''''''''''''''''''''''''''''''''''''''''''''''''' The version specific Spike machines have been removed in favour of the generic ``spike`` machine. If you need to specify an older version of the RISC-V spec you can use the ``-cpu rv64gcsu,priv_spec=v1.10.0`` command line argument. mips ``r4k`` platform (removed in 5.2) '''''''''''''''''''''''''''''''''''''' This machine type was very old and unmaintained. Users should use the ``malta`` machine type instead. mips ``fulong2e`` machine alias (removed in 6.0) '''''''''''''''''''''''''''''''''''''''''''''''' This machine has been renamed ``fuloong2e``. ``pc-0.10`` up to ``pc-1.3`` (removed in 4.0 up to 6.0) ''''''''''''''''''''''''''''''''''''''''''''''''''''''' These machine types were very old and likely could not be used for live migration from old QEMU versions anymore. Use a newer machine type instead. Raspberry Pi ``raspi2`` and ``raspi3`` machines (removed in 6.2) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The Raspberry Pi machines come in various models (A, A+, B, B+). To be able to distinguish which model QEMU is implementing, the ``raspi2`` and ``raspi3`` machines have been renamed ``raspi2b`` and ``raspi3b``. Aspeed ``swift-bmc`` machine (removed in 7.0) ''''''''''''''''''''''''''''''''''''''''''''' This machine was removed because it was unused. Alternative AST2500 based OpenPOWER machines are ``witherspoon-bmc`` and ``romulus-bmc``. ppc ``taihu`` machine (removed in 7.2) ''''''''''''''''''''''''''''''''''''''''''''' This machine was removed because it was partially emulated and 405 machines are very similar. Use the ``ref405ep`` machine instead. linux-user mode CPUs -------------------- ``tilegx`` CPUs (removed in 6.0) '''''''''''''''''''''''''''''''' The ``tilegx`` guest CPU support has been removed without replacement. It was only implemented in linux-user mode, but support for this CPU was removed from the upstream Linux kernel in 2018, and it has also been dropped from glibc, so there is no new Linux development taking place with this architecture. For running the old binaries, you can use older versions of QEMU. ``ppc64abi32`` CPUs (removed in 7.0) '''''''''''''''''''''''''''''''''''' The ``ppc64abi32`` architecture has a number of issues which regularly tripped up the CI testing and was suspected to be quite broken. For that reason the maintainers strongly suspected no one actually used it. TCG introspection features -------------------------- TCG trace-events (since 6.2) '''''''''''''''''''''''''''' The ability to add new TCG trace points had bit rotted and as the feature can be replicated with TCG plugins it was removed. If any user is currently using this feature and needs help with converting to using TCG plugins they should contact the qemu-devel mailing list. System emulator devices ----------------------- ``spapr-pci-vfio-host-bridge`` (removed in 2.12) ''''''''''''''''''''''''''''''''''''''''''''''''' The ``spapr-pci-vfio-host-bridge`` device type has been replaced by the ``spapr-pci-host-bridge`` device type. ``ivshmem`` (removed in 4.0) '''''''''''''''''''''''''''' Replaced by either the ``ivshmem-plain`` or ``ivshmem-doorbell``. ``ide-drive`` (removed in 6.0) '''''''''''''''''''''''''''''' The 'ide-drive' device has been removed. Users should use 'ide-hd' or 'ide-cd' as appropriate to get an IDE hard disk or CD-ROM as needed. ``scsi-disk`` (removed in 6.0) '''''''''''''''''''''''''''''' The 'scsi-disk' device has been removed. Users should use 'scsi-hd' or 'scsi-cd' as appropriate to get a SCSI hard disk or CD-ROM as needed. Related binaries ---------------- ``qemu-nbd --partition`` (removed in 5.0) ''''''''''''''''''''''''''''''''''''''''' The ``qemu-nbd --partition $digit`` code (also spelled ``-P``) could only handle MBR partitions, and never correctly handled logical partitions beyond partition 5. Exporting a partition can still be done by utilizing the ``--image-opts`` option with a raw blockdev using the ``offset`` and ``size`` parameters layered on top of any other existing blockdev. For example, if partition 1 is 100MiB long starting at 1MiB, the old command:: qemu-nbd -t -P 1 -f qcow2 file.qcow2 can be rewritten as:: qemu-nbd -t --image-opts driver=raw,offset=1M,size=100M,file.driver=qcow2,file.file.driver=file,file.file.filename=file.qcow2 ``qemu-img convert -n -o`` (removed in 5.1) ''''''''''''''''''''''''''''''''''''''''''' All options specified in ``-o`` are image creation options, so they are now rejected when used with ``-n`` to skip image creation. ``qemu-img create -b bad file $size`` (removed in 5.1) '''''''''''''''''''''''''''''''''''''''''''''''''''''' When creating an image with a backing file that could not be opened, ``qemu-img create`` used to issue a warning about the failure but proceed with the image creation if an explicit size was provided. However, as the ``-u`` option exists for this purpose, it is safer to enforce that any failure to open the backing image (including if the backing file is missing or an incorrect format was specified) is an error when ``-u`` is not used. ``qemu-img amend`` to adjust backing file (removed in 6.1) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The use of ``qemu-img amend`` to modify the name or format of a qcow2 backing image was never fully documented or tested, and interferes with other amend operations that need access to the original backing image (such as deciding whether a v3 zero cluster may be left unallocated when converting to a v2 image). Any changes to the backing chain should be performed with ``qemu-img rebase -u`` either before or after the remaining changes being performed by amend, as appropriate. ``qemu-img`` backing file without format (removed in 6.1) ''''''''''''''''''''''''''''''''''''''''''''''''''''''''' The use of ``qemu-img create``, ``qemu-img rebase``, or ``qemu-img convert`` to create or modify an image that depends on a backing file now requires that an explicit backing format be provided. This is for safety: if QEMU probes a different format than what you thought, the data presented to the guest will be corrupt; similarly, presenting a raw image to a guest allows a potential security exploit if a future probe sees a non-raw image based on guest writes. To avoid creating unsafe backing chains, you must pass ``-o backing_fmt=`` (or the shorthand ``-F`` during create) to specify the intended backing format. You may use ``qemu-img rebase -u`` to retroactively add a backing format to an existing image. However, be aware that there are already potential security risks to blindly using ``qemu-img info`` to probe the format of an untrusted backing image, when deciding what format to add into an existing image. Block devices ------------- VXHS backend (removed in 5.1) ''''''''''''''''''''''''''''' The VXHS code did not compile since v2.12.0. It was removed in 5.1. ``sheepdog`` driver (removed in 6.0) '''''''''''''''''''''''''''''''''''' The corresponding upstream server project is no longer maintained. Users are recommended to switch to an alternative distributed block device driver such as RBD. --- ### Devel Atomics (`docs/devel/atomics.rst`) ========================= Atomic operations in QEMU ========================= CPUs perform independent memory operations effectively in random order. but this can be a problem for CPU-CPU interaction (including interactions between QEMU and the guest). Multi-threaded programs use various tools to instruct the compiler and the CPU to restrict the order to something that is consistent with the expectations of the programmer. The most basic tool is locking. Mutexes, condition variables and semaphores are used in QEMU, and should be the default approach to synchronization. Anything else is considerably harder, but it's also justified more often than one would like; the most performance-critical parts of QEMU in particular require a very low level approach to concurrency, involving memory barriers and atomic operations. The semantics of concurrent memory accesses are governed by the C11 memory model. QEMU provides a header, ``qemu/atomic.h``, which wraps C11 atomics to provide better portability and a less verbose syntax. ``qemu/atomic.h`` provides macros that fall in three camps: - compiler barriers: ``barrier()``; - weak atomic access and manual memory barriers: ``qatomic_read()``, ``qatomic_set()``, ``smp_rmb()``, ``smp_wmb()``, ``smp_mb()``, ``smp_mb_acquire()``, ``smp_mb_release()``, ``smp_read_barrier_depends()``; - sequentially consistent atomic access: everything else. In general, use of ``qemu/atomic.h`` should be wrapped with more easily used data structures (e.g. the lock-free singly-linked list operations ``QSLIST_INSERT_HEAD_ATOMIC`` and ``QSLIST_MOVE_ATOMIC``) or synchronization primitives (such as RCU, ``QemuEvent`` or ``QemuLockCnt``). Bare use of atomic operations and memory barriers should be limited to inter-thread checking of flags and documented thoroughly. Compiler memory barrier ======================= ``barrier()`` prevents the compiler from moving the memory accesses on either side of it to the other side. The compiler barrier has no direct effect on the CPU, which may then reorder things however it wishes. ``barrier()`` is mostly used within ``qemu/atomic.h`` itself. On some architectures, CPU guarantees are strong enough that blocking compiler optimizations already ensures the correct order of execution. In this case, ``qemu/atomic.h`` will reduce stronger memory barriers to simple compiler barriers. Still, ``barrier()`` can be useful when writing code that can be interrupted by signal handlers. Sequentially consistent atomic access ===================================== Most of the operations in the ``qemu/atomic.h`` header ensure *sequential consistency*, where "the result of any execution is the same as if the operations of all the processors were executed in some sequential order, and the operations of each individual processor appear in this sequence in the order specified by its program". ``qemu/atomic.h`` provides the following set of atomic read-modify-write operations:: void qatomic_inc(ptr) void qatomic_dec(ptr) void qatomic_add(ptr, val) void qatomic_sub(ptr, val) void qatomic_and(ptr, val) void qatomic_or(ptr, val) typeof(*ptr) qatomic_fetch_inc(ptr) typeof(*ptr) qatomic_fetch_dec(ptr) typeof(*ptr) qatomic_fetch_add(ptr, val) typeof(*ptr) qatomic_fetch_sub(ptr, val) typeof(*ptr) qatomic_fetch_and(ptr, val) typeof(*ptr) qatomic_fetch_or(ptr, val) typeof(*ptr) qatomic_fetch_xor(ptr, val) typeof(*ptr) qatomic_fetch_inc_nonzero(ptr) typeof(*ptr) qatomic_xchg(ptr, val) typeof(*ptr) qatomic_cmpxchg(ptr, old, new) all of which return the old value of ``*ptr``. These operations are polymorphic; they operate on any type that is as wide as a pointer or smaller. Similar operations return the new value of ``*ptr``:: typeof(*ptr) qatomic_inc_fetch(ptr) typeof(*ptr) qatomic_dec_fetch(ptr) typeof(*ptr) qatomic_add_fetch(ptr, val) typeof(*ptr) qatomic_sub_fetch(ptr, val) typeof(*ptr) qatomic_and_fetch(ptr, val) typeof(*ptr) qatomic_or_fetch(ptr, val) typeof(*ptr) qatomic_xor_fetch(ptr, val) ``qemu/atomic.h`` also provides loads and stores that cannot be reordered with each other:: typeof(*ptr) qatomic_mb_read(ptr) void qatomic_mb_set(ptr, val) However these do not provide sequential consistency and, in particular, they do not participate in the total ordering enforced by sequentially-consistent operations. For this reason they are deprecated. They should instead be replaced with any of the following (ordered from easiest to hardest): - accesses inside a mutex or spinlock - lightweight synchronization primitives such as ``QemuEvent`` - RCU operations (``qatomic_rcu_read``, ``qatomic_rcu_set``) when publishing or accessing a new version of a data structure - other atomic accesses: ``qatomic_read`` and ``qatomic_load_acquire`` for loads, ``qatomic_set`` and ``qatomic_store_release`` for stores, ``smp_mb`` to forbid reordering subsequent loads before a store. Weak atomic access and manual memory barriers ============================================= Compared to sequentially consistent atomic access, programming with weaker consistency models can be considerably more complicated. The only guarantees that you can rely upon in this case are: - atomic accesses will not cause data races (and hence undefined behavior); ordinary accesses instead cause data races if they are concurrent with other accesses of which at least one is a write. In order to ensure this, the compiler will not optimize accesses out of existence, create unsolicited accesses, or perform other similar optimzations. - acquire operations will appear to happen, with respect to the other components of the system, before all the LOAD or STORE operations specified afterwards. - release operations will appear to happen, with respect to the other components of the system, after all the LOAD or STORE operations specified before. - release operations will *synchronize with* acquire operations; see :ref:`acqrel` for a detailed explanation. When using this model, variables are accessed with: - ``qatomic_read()`` and ``qatomic_set()``; these prevent the compiler from optimizing accesses out of existence and creating unsolicited accesses, but do not otherwise impose any ordering on loads and stores: both the compiler and the processor are free to reorder them. - ``qatomic_load_acquire()``, which guarantees the LOAD to appear to happen, with respect to the other components of the system, before all the LOAD or STORE operations specified afterwards. Operations coming before ``qatomic_load_acquire()`` can still be reordered after it. - ``qatomic_store_release()``, which guarantees the STORE to appear to happen, with respect to the other components of the system, after all the LOAD or STORE operations specified before. Operations coming after ``qatomic_store_release()`` can still be reordered before it. Restrictions to the ordering of accesses can also be specified using the memory barrier macros: ``smp_rmb()``, ``smp_wmb()``, ``smp_mb()``, ``smp_mb_acquire()``, ``smp_mb_release()``, ``smp_read_barrier_depends()``. Memory barriers control the order of references to shared memory. They come in six kinds: - ``smp_rmb()`` guarantees that all the LOAD operations specified before the barrier will appear to happen before all the LOAD operations specified after the barrier with respect to the other components of the system. In other words, ``smp_rmb()`` puts a partial ordering on loads, but is not required to have any effect on stores. - ``smp_wmb()`` guarantees that all the STORE operations specified before the barrier will appear to happen before all the STORE operations specified after the barrier with respect to the other components of the system. In other words, ``smp_wmb()`` puts a partial ordering on stores, but is not required to have any effect on loads. - ``smp_mb_acquire()`` guarantees that all the LOAD operations specified before the barrier will appear to happen before all the LOAD or STORE operations specified after the barrier with respect to the other components of the system. - ``smp_mb_release()`` guarantees that all the STORE operations specified *after* the barrier will appear to happen after all the LOAD or STORE operations specified *before* the barrier with respect to the other components of the system. - ``smp_mb()`` guarantees that all the LOAD and STORE operations specified before the barrier will appear to happen before all the LOAD and STORE operations specified after the barrier with respect to the other components of the system. ``smp_mb()`` puts a partial ordering on both loads and stores. It is stronger than both a read and a write memory barrier; it implies both ``smp_mb_acquire()`` and ``smp_mb_release()``, but it also prevents STOREs coming before the barrier from overtaking LOADs coming after the barrier and vice versa. - ``smp_read_barrier_depends()`` is a weaker kind of read barrier. On most processors, whenever two loads are performed such that the second depends on the result of the first (e.g., the first load retrieves the address to which the second load will be directed), the processor will guarantee that the first LOAD will appear to happen before the second with respect to the other components of the system. However, this is not always true---for example, it was not true on Alpha processors. Whenever this kind of access happens to shared memory (that is not protected by a lock), a read barrier is needed, and ``smp_read_barrier_depends()`` can be used instead of ``smp_rmb()``. Note that the first load really has to have a _data_ dependency and not a control dependency. If the address for the second load is dependent on the first load, but the dependency is through a conditional rather than actually loading the address itself, then it's a _control_ dependency and a full read barrier or better is required. Memory barriers and ``qatomic_load_acquire``/``qatomic_store_release`` are mostly used when a data structure has one thread that is always a writer and one thread that is always a reader: +----------------------------------+----------------------------------+ | thread 1 | thread 2 | +==================================+==================================+ | :: | :: | | | | | qatomic_store_release(&a, x); | y = qatomic_load_acquire(&b); | | qatomic_store_release(&b, y); | x = qatomic_load_acquire(&a); | +----------------------------------+----------------------------------+ In this case, correctness is easy to check for using the "pairing" trick that is explained below. Sometimes, a thread is accessing many variables that are otherwise unrelated to each other (for example because, apart from the current thread, exactly one other thread will read or write each of these variables). In this case, it is possible to "hoist" the barriers outside a loop. For example: +------------------------------------------+----------------------------------+ | before | after | +==========================================+==================================+ | :: | :: | | | | | n = 0; | n = 0; | | for (i = 0; i < 10; i++) | for (i = 0; i < 10; i++) | | n += qatomic_load_acquire(&a[i]); | n += qatomic_read(&a[i]); | | | smp_mb_acquire(); | +------------------------------------------+----------------------------------+ | :: | :: | | | | | | smp_mb_release(); | | for (i = 0; i < 10; i++) | for (i = 0; i < 10; i++) | | qatomic_store_release(&a[i], false); | qatomic_set(&a[i], false); | +------------------------------------------+----------------------------------+ Splitting a loop can also be useful to reduce the number of barriers: +------------------------------------------+----------------------------------+ | before | after | +==========================================+==================================+ | :: | :: | | | | | n = 0; | smp_mb_release(); | | for (i = 0; i < 10; i++) { | for (i = 0; i < 10; i++) | | qatomic_store_release(&a[i], false); | qatomic_set(&a[i], false); | | smp_mb(); | smb_mb(); | | n += qatomic_read(&b[i]); | n = 0; | | } | for (i = 0; i < 10; i++) | | | n += qatomic_read(&b[i]); | +------------------------------------------+----------------------------------+ In this case, a ``smp_mb_release()`` is also replaced with a (possibly cheaper, and clearer as well) ``smp_wmb()``: +------------------------------------------+----------------------------------+ | before | after | +==========================================+==================================+ | :: | :: | | | | | | smp_mb_release(); | | for (i = 0; i < 10; i++) { | for (i = 0; i < 10; i++) | | qatomic_store_release(&a[i], false); | qatomic_set(&a[i], false); | | qatomic_store_release(&b[i], false); | smb_wmb(); | | } | for (i = 0; i < 10; i++) | | | qatomic_set(&b[i], false); | +------------------------------------------+----------------------------------+ .. _acqrel: Acquire/release pairing and the *synchronizes-with* relation ------------------------------------------------------------ Atomic operations other than ``qatomic_set()`` and ``qatomic_read()`` have either *acquire* or *release* semantics [#rmw]_. This has two effects: .. [#rmw] Read-modify-write operations can have both---acquire applies to the read part, and release to the write. - within a thread, they are ordered either before subsequent operations (for acquire) or after previous operations (for release). - if a release operation in one thread *synchronizes with* an acquire operation in another thread, the ordering constraints propagates from the first to the second thread. That is, everything before the release operation in the first thread is guaranteed to *happen before* everything after the acquire operation in the second thread. The concept of acquire and release semantics is not exclusive to atomic operations; almost all higher-level synchronization primitives also have acquire or release semantics. For example: - ``pthread_mutex_lock`` has acquire semantics, ``pthread_mutex_unlock`` has release semantics and synchronizes with a ``pthread_mutex_lock`` for the same mutex. - ``pthread_cond_signal`` and ``pthread_cond_broadcast`` have release semantics; ``pthread_cond_wait`` has both release semantics (synchronizing with ``pthread_mutex_lock``) and acquire semantics (synchronizing with ``pthread_mutex_unlock`` and signaling of the condition variable). - ``pthread_create`` has release semantics and synchronizes with the start of the new thread; ``pthread_join`` has acquire semantics and synchronizes with the exiting of the thread. - ``qemu_event_set`` has release semantics, ``qemu_event_wait`` has acquire semantics. For example, in the following example there are no atomic accesses, but still thread 2 is relying on the *synchronizes-with* relation between ``pthread_exit`` (release) and ``pthread_join`` (acquire): +----------------------+-------------------------------+ | thread 1 | thread 2 | +======================+===============================+ | :: | :: | | | | | *a = 1; | | | pthread_exit(a); | pthread_join(thread1, &a); | | | x = *a; | +----------------------+-------------------------------+ Synchronization between threads basically descends from this pairing of a release operation and an acquire operation. Therefore, atomic operations other than ``qatomic_set()`` and ``qatomic_read()`` will almost always be paired with another operation of the opposite kind: an acquire operation will pair with a release operation and vice versa. This rule of thumb is extremely useful; in the case of QEMU, however, note that the other operation may actually be in a driver that runs in the guest! ``smp_read_barrier_depends()``, ``smp_rmb()``, ``smp_mb_acquire()``, ``qatomic_load_acquire()`` and ``qatomic_rcu_read()`` all count as acquire operations. ``smp_wmb()``, ``smp_mb_release()``, ``qatomic_store_release()`` and ``qatomic_rcu_set()`` all count as release operations. ``smp_mb()`` counts as both acquire and release, therefore it can pair with any other atomic operation. Here is an example: +----------------------+------------------------------+ | thread 1 | thread 2 | +======================+==============================+ | :: | :: | | | | | qatomic_set(&a, 1);| | | smp_wmb(); | | | qatomic_set(&b, 2);| x = qatomic_read(&b); | | | smp_rmb(); | | | y = qatomic_read(&a); | +----------------------+------------------------------+ Note that a load-store pair only counts if the two operations access the same variable: that is, a store-release on a variable ``x`` *synchronizes with* a load-acquire on a variable ``x``, while a release barrier synchronizes with any acquire operation. The following example shows correct synchronization: +--------------------------------+--------------------------------+ | thread 1 | thread 2 | +================================+================================+ | :: | :: | | | | | qatomic_set(&a, 1); | | | qatomic_store_release(&b, 2);| x = qatomic_load_acquire(&b);| | | y = qatomic_read(&a); | +--------------------------------+--------------------------------+ Acquire and release semantics of higher-level primitives can also be relied upon for the purpose of establishing the *synchronizes with* relation. Note that the "writing" thread is accessing the variables in the opposite order as the "reading" thread. This is expected: stores before a release operation will normally match the loads after the acquire operation, and vice versa. In fact, this happened already in the ``pthread_exit``/``pthread_join`` example above. Finally, this more complex example has more than two accesses and data dependency barriers. It also does not use atomic accesses whenever there cannot be a data race: +----------------------+------------------------------+ | thread 1 | thread 2 | +======================+==============================+ | :: | :: | | | | | b[2] = 1; | | | smp_wmb(); | | | x->i = 2; | | | smp_wmb(); | | | qatomic_set(&a, x);| x = qatomic_read(&a); | | | smp_read_barrier_depends(); | | | y = x->i; | | | smp_read_barrier_depends(); | | | z = b[y]; | +----------------------+------------------------------+ Comparison with Linux kernel primitives ======================================= Here is a list of differences between Linux kernel atomic operations and memory barriers, and the equivalents in QEMU: - atomic operations in Linux are always on a 32-bit int type and use a boxed ``atomic_t`` type; atomic operations in QEMU are polymorphic and use normal C types. - Originally, ``atomic_read`` and ``atomic_set`` in Linux gave no guarantee at all. Linux 4.1 updated them to implement volatile semantics via ``ACCESS_ONCE`` (or the more recent ``READ``/``WRITE_ONCE``). QEMU's ``qatomic_read`` and ``qatomic_set`` implement C11 atomic relaxed semantics if the compiler supports it, and volatile semantics otherwise. Both semantics prevent the compiler from doing certain transformations; the difference is that atomic accesses are guaranteed to be atomic, while volatile accesses aren't. Thus, in the volatile case we just cross our fingers hoping that the compiler will generate atomic accesses, since we assume the variables passed are machine-word sized and properly aligned. No barriers are implied by ``qatomic_read`` and ``qatomic_set`` in either Linux or QEMU. - atomic read-modify-write operations in Linux are of three kinds: ===================== ========================================= ``atomic_OP`` returns void ``atomic_OP_return`` returns new value of the variable ``atomic_fetch_OP`` returns the old value of the variable ``atomic_cmpxchg`` returns the old value of the variable ===================== ========================================= In QEMU, the second kind is named ``atomic_OP_fetch``. - different atomic read-modify-write operations in Linux imply a different set of memory barriers; in QEMU, all of them enforce sequential consistency. - in QEMU, ``qatomic_read()`` and ``qatomic_set()`` do not participate in the total ordering enforced by sequentially-consistent operations. This is because QEMU uses the C11 memory model. The following example is correct in Linux but not in QEMU: +----------------------------------+--------------------------------+ | Linux (correct) | QEMU (incorrect) | +==================================+================================+ | :: | :: | | | | | a = atomic_fetch_add(&x, 2); | a = qatomic_fetch_add(&x, 2);| | b = READ_ONCE(&y); | b = qatomic_read(&y); | +----------------------------------+--------------------------------+ because the read of ``y`` can be moved (by either the processor or the compiler) before the write of ``x``. Fixing this requires an ``smp_mb()`` memory barrier between the write of ``x`` and the read of ``y``. In the common case where only one thread writes ``x``, it is also possible to write it like this: +--------------------------------+ | QEMU (correct) | +================================+ | :: | | | | a = qatomic_read(&x); | | qatomic_set(&x, a + 2); | | smp_mb(); | | b = qatomic_read(&y); | +--------------------------------+ Sources ======= - ``Documentation/memory-barriers.txt`` from the Linux kernel --- ### Devel Bitops (`docs/devel/bitops.rst`) ================== Bitwise operations ================== The header ``qemu/bitops.h`` provides utility functions for performing bitwise operations. .. kernel-doc:: include/qemu/bitops.h --- ### Devel Blkdebug (`docs/devel/blkdebug.txt`) Block I/O error injection using blkdebug ---------------------------------------- Copyright (C) 2014-2015 Red Hat Inc This work is licensed under the terms of the GNU GPL, version 2 or later. See the COPYING file in the top-level directory. The blkdebug block driver is a rule-based error injection engine. It can be used to exercise error code paths in block drivers including ENOSPC (out of space) and EIO. This document gives an overview of the features available in blkdebug. Background ---------- Block drivers have many error code paths that handle I/O errors. Image formats are especially complex since metadata I/O errors during cluster allocation or while updating tables happen halfway through request processing and require discipline to keep image files consistent. Error injection allows test cases to trigger I/O errors at specific points. This way, all error paths can be tested to make sure they are correct. Rules ----- The blkdebug block driver takes a list of "rules" that tell the error injection engine when to fail an I/O request. Each I/O request is evaluated against the rules. If a rule matches the request then its "action" is executed. Rules can be placed in a configuration file; the configuration file follows the same .ini-like format used by QEMU's -readconfig option, and each section of the file represents a rule. The following configuration file defines a single rule: $ cat blkdebug.conf [inject-error] event = "read_aio" errno = "28" This rule fails all aio read requests with ENOSPC (28). Note that the errno value depends on the host. On Linux, see /usr/include/asm-generic/errno-base.h for errno values. Invoke QEMU as follows: $ qemu-system-x86_64 -drive if=none,cache=none,file=blkdebug:blkdebug.conf:test.img,id=drive0 \ -device virtio-blk-pci,drive=drive0,id=virtio-blk-pci0 Rules support the following attributes: event - which type of operation to match (e.g. read_aio, write_aio, flush_to_os, flush_to_disk). See the "Events" section for information on events. state - (optional) the engine must be in this state number in order for this rule to match. See the "State transitions" section for information on states. errno - the numeric errno value to return when a request matches this rule. The errno values depend on the host since the numeric values are not standardized in the POSIX specification. sector - (optional) a sector number that the request must overlap in order to match this rule once - (optional, default "off") only execute this action on the first matching request immediately - (optional, default "off") return a NULL BlockAIOCB pointer and fail without an errno instead. This exercises the code path where BlockAIOCB fails and the caller's BlockCompletionFunc is not invoked. Events ------ Block drivers provide information about the type of I/O request they are about to make so rules can match specific types of requests. For example, the qcow2 block driver tells blkdebug when it accesses the L1 table so rules can match only L1 table accesses and not other metadata or guest data requests. The core events are: read_aio - guest data read write_aio - guest data write flush_to_os - write out unwritten block driver state (e.g. cached metadata) flush_to_disk - flush the host block device's disk cache See qapi/block-core.json:BlkdebugEvent for the full list of events. You may need to grep block driver source code to understand the meaning of specific events. State transitions ----------------- There are cases where more power is needed to match a particular I/O request in a longer sequence of requests. For example: write_aio flush_to_disk write_aio How do we match the 2nd write_aio but not the first? This is where state transitions come in. The error injection engine has an integer called the "state" that always starts initialized to 1. The state integer is internal to blkdebug and cannot be observed from outside but rules can interact with it for powerful matching behavior. Rules can be conditional on the current state and they can transition to a new state. When a rule's "state" attribute is non-zero then the current state must equal the attribute in order for the rule to match. For example, to match the 2nd write_aio: [set-state] event = "write_aio" state = "1" new_state = "2" [inject-error] event = "write_aio" state = "2" errno = "5" The first write_aio request matches the set-state rule and transitions from state 1 to state 2. Once state 2 has been entered, the set-state rule no longer matches since it requires state 1. But the inject-error rule now matches the next write_aio request and injects EIO (5). State transition rules support the following attributes: event - which type of operation to match (e.g. read_aio, write_aio, flush_to_os, flush_to_disk). See the "Events" section for information on events. state - (optional) the engine must be in this state number in order for this rule to match new_state - transition to this state number Suspend and resume ------------------ Exercising code paths in block drivers may require specific ordering amongst concurrent requests. The "breakpoint" feature allows requests to be halted on a blkdebug event and resumed later. This makes it possible to achieve deterministic ordering when multiple requests are in flight. Breakpoints on blkdebug events are associated with a user-defined "tag" string. This tag serves as an identifier by which the request can be resumed at a later point. See the qemu-io(1) break, resume, remove_break, and wait_break commands for details. --- ### Devel Blkverify (`docs/devel/blkverify.txt`) = Block driver correctness testing with blkverify = == Introduction == This document describes how to use the blkverify protocol to test that a block driver is operating correctly. It is difficult to test and debug block drivers against real guests. Often processes inside the guest will crash because corrupt sectors were read as part of the executable. Other times obscure errors are raised by a program inside the guest. These issues are extremely hard to trace back to bugs in the block driver. Blkverify solves this problem by catching data corruption inside QEMU the first time bad data is read and reporting the disk sector that is corrupted. == How it works == The blkverify protocol has two child block devices, the "test" device and the "raw" device. Read/write operations are mirrored to both devices so their state should always be in sync. The "raw" device is a raw image, a flat file, that has identical starting contents to the "test" image. The idea is that the "raw" device will handle read/write operations correctly and not corrupt data. It can be used as a reference for comparison against the "test" device. After a mirrored read operation completes, blkverify will compare the data and raise an error if it is not identical. This makes it possible to catch the first instance where corrupt data is read. == Example == Imagine raw.img has 0xcd repeated throughout its first sector: $ ./qemu-io -c 'read -v 0 512' raw.img 00000000: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 00000010: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ [...] 000001e0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ 000001f0: cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd ................ read 512/512 bytes at offset 0 512.000000 bytes, 1 ops; 0.0000 sec (97.656 MiB/sec and 200000.0000 ops/sec) And test.img is corrupt, its first sector is zeroed when it shouldn't be: $ ./qemu-io -c 'read -v 0 512' test.img 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [...] 000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ read 512/512 bytes at offset 0 512.000000 bytes, 1 ops; 0.0000 sec (81.380 MiB/sec and 166666.6667 ops/sec) This error is caught by blkverify: $ ./qemu-io -c 'read 0 512' blkverify:a.img:b.img blkverify: read sector_num=0 nb_sectors=4 contents mismatch in sector 0 A more realistic scenario is verifying the installation of a guest OS: $ ./qemu-img create raw.img 16G $ ./qemu-img create -f qcow2 test.qcow2 16G $ ./qemu-system-x86_64 -cdrom debian.iso \ -drive file=blkverify:raw.img:test.qcow2 If the installation is aborted when blkverify detects corruption, use qemu-io to explore the contents of the disk image at the sector in question. --- ### Devel Block Coroutine Wrapper (`docs/devel/block-coroutine-wrapper.rst`) ======================= block-coroutine-wrapper ======================= A lot of functions in QEMU block layer (see ``block/*``) can only be called in coroutine context. Such functions are normally marked by the coroutine_fn specifier. Still, sometimes we need to call them from non-coroutine context; for this we need to start a coroutine, run the needed function from it and wait for the coroutine to finish in a BDRV_POLL_WHILE() loop. To run a coroutine we need a function with one void* argument. So for each coroutine_fn function which needs a non-coroutine interface, we should define a structure to pack the parameters, define a separate function to unpack the parameters and call the original function and finally define a new interface function with same list of arguments as original one, which will pack the parameters into a struct, create a coroutine, run it and wait in BDRV_POLL_WHILE() loop. It's boring to create such wrappers by hand, so we have a script to generate them. Usage ===== Assume we have defined the ``coroutine_fn`` function ``bdrv_co_foo()`` and need a non-coroutine interface for it, called ``bdrv_foo()``. In this case the script can help. To trigger the generation: 1. You need ``bdrv_foo`` declaration somewhere (for example, in ``block/coroutines.h``) with the ``generated_co_wrapper`` mark, like this: .. code-block:: c int generated_co_wrapper bdrv_foo(); 2. You need to feed this declaration to block-coroutine-wrapper script. For this, add the .h (or .c) file with the declaration to the ``input: files(...)`` list of ``block_gen_c`` target declaration in ``block/meson.build`` You are done. During the build, coroutine wrappers will be generated in ``/block/block-gen.c``. Links ===== 1. The script location is ``scripts/block-coroutine-wrapper.py``. 2. Generic place for private ``generated_co_wrapper`` declarations is ``block/coroutines.h``, for public declarations: ``include/block/block.h`` 3. The core API of generated coroutine wrappers is placed in (not generated) ``block/block-gen.h`` --- ### Devel Build System (`docs/devel/build-system.rst`) ================================== The QEMU build system architecture ================================== This document aims to help developers understand the architecture of the QEMU build system. As with projects using GNU autotools, the QEMU build system has two stages, first the developer runs the "configure" script to determine the local build environment characteristics, then they run "make" to build the project. There is about where the similarities with GNU autotools end, so try to forget what you know about them. Stage 1: configure ================== The QEMU configure script is written directly in shell, and should be compatible with any POSIX shell, hence it uses #!/bin/sh. An important implication of this is that it is important to avoid using bash-isms on development platforms where bash is the primary host. In contrast to autoconf scripts, QEMU's configure is expected to be silent while it is checking for features. It will only display output when an error occurs, or to show the final feature enablement summary on completion. Because QEMU uses the Meson build system under the hood, only VPATH builds are supported. There are two general ways to invoke configure & perform a build: - VPATH, build artifacts outside of QEMU source tree entirely:: cd ../ mkdir build cd build ../qemu/configure make - VPATH, build artifacts in a subdir of QEMU source tree:: mkdir build cd build ../configure make The configure script automatically recognizes command line options for which a same-named Meson option exists; dashes in the command line are replaced with underscores. Many checks on the compilation environment are still found in configure rather than ``meson.build``, but new checks should be added directly to ``meson.build``. Patches are also welcome to move existing checks from the configure phase to ``meson.build``. When doing so, ensure that ``meson.build`` does not use anymore the keys that you have removed from ``config-host.mak``. Typically these will be replaced in ``meson.build`` by boolean variables, ``get_option('optname')`` invocations, or ``dep.found()`` expressions. In general, the remaining checks have little or no interdependencies, so they can be moved one by one. Helper functions ---------------- The configure script provides a variety of helper functions to assist developers in checking for system features: ``do_cc $ARGS...`` Attempt to run the system C compiler passing it $ARGS... ``do_cxx $ARGS...`` Attempt to run the system C++ compiler passing it $ARGS... ``compile_object $CFLAGS`` Attempt to compile a test program with the system C compiler using $CFLAGS. The test program must have been previously written to a file called $TMPC. The replacement in Meson is the compiler object ``cc``, which has methods such as ``cc.compiles()``, ``cc.check_header()``, ``cc.has_function()``. ``compile_prog $CFLAGS $LDFLAGS`` Attempt to compile a test program with the system C compiler using $CFLAGS and link it with the system linker using $LDFLAGS. The test program must have been previously written to a file called $TMPC. The replacement in Meson is ``cc.find_library()`` and ``cc.links()``. ``has $COMMAND`` Determine if $COMMAND exists in the current environment, either as a shell builtin, or executable binary, returning 0 on success. The replacement in Meson is ``find_program()``. ``check_define $NAME`` Determine if the macro $NAME is defined by the system C compiler ``check_include $NAME`` Determine if the include $NAME file is available to the system C compiler. The replacement in Meson is ``cc.has_header()``. ``write_c_skeleton`` Write a minimal C program main() function to the temporary file indicated by $TMPC ``error_exit $MESSAGE $MORE...`` Print $MESSAGE to stderr, followed by $MORE... and then exit from the configure script with non-zero status ``query_pkg_config $ARGS...`` Run pkg-config passing it $ARGS. If QEMU is doing a static build, then --static will be automatically added to $ARGS Stage 2: Meson ============== The Meson build system is currently used to describe the build process for: 1) executables, which include: - Tools - ``qemu-img``, ``qemu-nbd``, ``qga`` (guest agent), etc - System emulators - ``qemu-system-$ARCH`` - Userspace emulators - ``qemu-$ARCH`` - Unit tests 2) documentation 3) ROMs, which can be either installed as binary blobs or compiled 4) other data files, such as icons or desktop files All executables are built by default, except for some ``contrib/`` binaries that are known to fail to build on some platforms (for example 32-bit or big-endian platforms). Tests are also built by default, though that might change in the future. The source code is highly modularized, split across many files to facilitate building of all of these components with as little duplicated compilation as possible. Using the Meson "sourceset" functionality, ``meson.build`` files group the source files in rules that are enabled according to the available system libraries and to various configuration symbols. Sourcesets belong to one of four groups: Subsystem sourcesets: Various subsystems that are common to both tools and emulators have their own sourceset, for example ``block_ss`` for the block device subsystem, ``chardev_ss`` for the character device subsystem, etc. These sourcesets are then turned into static libraries as follows:: libchardev = static_library('chardev', chardev_ss.sources(), name_suffix: 'fa', build_by_default: false) chardev = declare_dependency(link_whole: libchardev) As of Meson 0.55.1, the special ``.fa`` suffix should be used for everything that is used with ``link_whole``, to ensure that the link flags are placed correctly in the command line. Target-independent emulator sourcesets: Various general purpose helper code is compiled only once and the .o files are linked into all output binaries that need it. This includes error handling infrastructure, standard data structures, platform portability wrapper functions, etc. Target-independent code lives in the ``common_ss``, ``softmmu_ss`` and ``user_ss`` sourcesets. ``common_ss`` is linked into all emulators, ``softmmu_ss`` only in system emulators, ``user_ss`` only in user-mode emulators. Target-independent sourcesets must exercise particular care when using ``if_false`` rules. The ``if_false`` rule will be used correctly when linking emulator binaries; however, when *compiling* target-independent files into .o files, Meson may need to pick *both* the ``if_true`` and ``if_false`` sides to cater for targets that want either side. To achieve that, you can add a special rule using the ``CONFIG_ALL`` symbol:: # Some targets have CONFIG_ACPI, some don't, so this is not enough softmmu_ss.add(when: 'CONFIG_ACPI', if_true: files('acpi.c'), if_false: files('acpi-stub.c')) # This is required as well: softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('acpi-stub.c')) Target-dependent emulator sourcesets: In the target-dependent set lives CPU emulation, some device emulation and much glue code. This sometimes also has to be compiled multiple times, once for each target being built. Target-dependent files are included in the ``specific_ss`` sourceset. Each emulator also includes sources for files in the ``hw/`` and ``target/`` subdirectories. The subdirectory used for each emulator comes from the target's definition of ``TARGET_BASE_ARCH`` or (if missing) ``TARGET_ARCH``, as found in ``default-configs/targets/*.mak``. Each subdirectory in ``hw/`` adds one sourceset to the ``hw_arch`` dictionary, for example:: arm_ss = ss.source_set() arm_ss.add(files('boot.c'), fdt) ... hw_arch += {'arm': arm_ss} The sourceset is only used for system emulators. Each subdirectory in ``target/`` instead should add one sourceset to each of the ``target_arch`` and ``target_softmmu_arch``, which are used respectively for all emulators and for system emulators only. For example:: arm_ss = ss.source_set() arm_softmmu_ss = ss.source_set() ... target_arch += {'arm': arm_ss} target_softmmu_arch += {'arm': arm_softmmu_ss} Module sourcesets: There are two dictionaries for modules: ``modules`` is used for target-independent modules and ``target_modules`` is used for target-dependent modules. When modules are disabled the ``module`` source sets are added to ``softmmu_ss`` and the ``target_modules`` source sets are added to ``specific_ss``. Both dictionaries are nested. One dictionary is created per subdirectory, and these per-subdirectory dictionaries are added to the toplevel dictionaries. For example:: hw_display_modules = {} qxl_ss = ss.source_set() ... hw_display_modules += { 'qxl': qxl_ss } modules += { 'hw-display': hw_display_modules } Utility sourcesets: All binaries link with a static library ``libqemuutil.a``. This library is built from several sourcesets; most of them however host generated code, and the only two of general interest are ``util_ss`` and ``stub_ss``. The separation between these two is purely for documentation purposes. ``util_ss`` contains generic utility files. Even though this code is only linked in some binaries, sometimes it requires hooks only in some of these and depend on other functions that are not fully implemented by all QEMU binaries. ``stub_ss`` links dummy stubs that will only be linked into the binary if the real implementation is not present. In a way, the stubs can be thought of as a portable implementation of the weak symbols concept. The following files concur in the definition of which files are linked into each emulator: ``default-configs/devices/*.mak`` The files under ``default-configs/devices/`` control the boards and devices that are built into each QEMU system emulation targets. They merely contain a list of config variable definitions such as:: include arm-softmmu.mak CONFIG_XLNX_ZYNQMP_ARM=y CONFIG_XLNX_VERSAL=y ``*/Kconfig`` These files are processed together with ``default-configs/devices/*.mak`` and describe the dependencies between various features, subsystems and device models. They are described in :ref:`kconfig` ``default-configs/targets/*.mak`` These files mostly define symbols that appear in the ``*-config-target.h`` file for each emulator [#cfgtarget]_. However, the ``TARGET_ARCH`` and ``TARGET_BASE_ARCH`` will also be used to select the ``hw/`` and ``target/`` subdirectories that are compiled into each target. .. [#cfgtarget] This header is included by ``qemu/osdep.h`` when compiling files from the target-specific sourcesets. These files rarely need changing unless you are adding a completely new target, or enabling new devices or hardware for a particular system/userspace emulation target Adding checks ------------- New checks should be added to Meson. Compiler checks can be as simple as the following:: config_host_data.set('HAVE_BTRFS_H', cc.has_header('linux/btrfs.h')) A more complex task such as adding a new dependency usually comprises the following tasks: - Add a Meson build option to meson_options.txt. - Add code to perform the actual feature check. - Add code to include the feature status in ``config-host.h`` - Add code to print out the feature status in the configure summary upon completion. Taking the probe for SDL2_Image as an example, we have the following in ``meson_options.txt``:: option('sdl_image', type : 'feature', value : 'auto', description: 'SDL Image support for icons') Unless the option was given a non-``auto`` value (on the configure command line), the detection code must be performed only if the dependency will be used:: sdl_image = not_found if not get_option('sdl_image').auto() or have_system sdl_image = dependency('SDL2_image', required: get_option('sdl_image'), method: 'pkg-config', static: enable_static) endif This avoids warnings on static builds of user-mode emulators, for example. Most of the libraries used by system-mode emulators are not available for static linking. The other supporting code is generally simple:: # Create config-host.h (if applicable) config_host_data.set('CONFIG_SDL_IMAGE', sdl_image.found()) # Summary summary_info += {'SDL image support': sdl_image.found()} For the configure script to parse the new option, the ``scripts/meson-buildoptions.sh`` file must be up-to-date; ``make update-buildoptions`` (or just ``make``) will take care of updating it. Support scripts --------------- Meson has a special convention for invoking Python scripts: if their first line is ``#! /usr/bin/env python3`` and the file is *not* executable, find_program() arranges to invoke the script under the same Python interpreter that was used to invoke Meson. This is the most common and preferred way to invoke support scripts from Meson build files, because it automatically uses the value of configure's --python= option. In case the script is not written in Python, use a ``#! /usr/bin/env ...`` line and make the script executable. Scripts written in Python, where it is desirable to make the script executable (for example for test scripts that developers may want to invoke from the command line, such as tests/qapi-schema/test-qapi.py), should be invoked through the ``python`` variable in meson.build. For example:: test('QAPI schema regression tests', python, args: files('test-qapi.py'), env: test_env, suite: ['qapi-schema', 'qapi-frontend']) This is needed to obey the --python= option passed to the configure script, which may point to something other than the first python3 binary on the path. Stage 3: makefiles ================== The use of GNU make is required with the QEMU build system. The output of Meson is a build.ninja file, which is used with the Ninja build system. QEMU uses a different approach, where Makefile rules are synthesized from the build.ninja file. The main Makefile includes these rules and wraps them so that e.g. submodules are built before QEMU. The resulting build system is largely non-recursive in nature, in contrast to common practices seen with automake. Tests are also ran by the Makefile with the traditional ``make check`` phony target, while benchmarks are run with ``make bench``. Meson test suites such as ``unit`` can be ran with ``make check-unit`` too. It is also possible to run tests defined in meson.build with ``meson test``. Useful make targets ------------------- ``help`` Print a help message for the most common build targets. ``print-VAR`` Print the value of the variable VAR. Useful for debugging the build system. Important files for the build system ==================================== Statically defined files ------------------------ The following key files are statically defined in the source tree, with the rules needed to build QEMU. Their behaviour is influenced by a number of dynamically created files listed later. ``Makefile`` The main entry point used when invoking make to build all the components of QEMU. The default 'all' target will naturally result in the build of every component. Makefile takes care of recursively building submodules directly via a non-recursive set of rules. ``*/meson.build`` The meson.build file in the root directory is the main entry point for the Meson build system, and it coordinates the configuration and build of all executables. Build rules for various subdirectories are included in other meson.build files spread throughout the QEMU source tree. ``tests/Makefile.include`` Rules for external test harnesses. These include the TCG tests, ``qemu-iotests`` and the Avocado-based integration tests. ``tests/docker/Makefile.include`` Rules for Docker tests. Like tests/Makefile, this file is included directly by the top level Makefile, anything defined in this file will influence the entire build system. ``tests/vm/Makefile.include`` Rules for VM-based tests. Like tests/Makefile, this file is included directly by the top level Makefile, anything defined in this file will influence the entire build system. Dynamically created files ------------------------- The following files are generated dynamically by configure in order to control the behaviour of the statically defined makefiles. This avoids the need for QEMU makefiles to go through any pre-processing as seen with autotools, where Makefile.am generates Makefile.in which generates Makefile. Built by configure: ``config-host.mak`` When configure has determined the characteristics of the build host it will write a long list of variables to config-host.mak file. This provides the various install directories, compiler / linker flags and a variety of ``CONFIG_*`` variables related to optionally enabled features. This is imported by the top level Makefile and meson.build in order to tailor the build output. config-host.mak is also used as a dependency checking mechanism. If make sees that the modification timestamp on configure is newer than that on config-host.mak, then configure will be re-run. The variables defined here are those which are applicable to all QEMU build outputs. Variables which are potentially different for each emulator target are defined by the next file... Built by Meson: ``${TARGET-NAME}-config-devices.mak`` TARGET-NAME is again the name of a system or userspace emulator. The config-devices.mak file is automatically generated by make using the scripts/make_device_config.sh program, feeding it the default-configs/$TARGET-NAME file as input. ``config-host.h``, ``$TARGET_NAME-config-target.h``, ``$TARGET_NAME-config-devices.h`` These files are used by source code to determine what features are enabled. They are generated from the contents of the corresponding ``*.mak`` files using Meson's ``configure_file()`` function. ``build.ninja`` The build rules. Built by Makefile: ``Makefile.ninja`` A Makefile include that bridges to ninja for the actual build. The Makefile is mostly a list of targets that Meson included in build.ninja. ``Makefile.mtest`` The Makefile definitions that let "make check" run tests defined in meson.build. The rules are produced from Meson's JSON description of tests (obtained with "meson introspect --tests") through the script scripts/mtest2make.py. --- ### Devel Ci (`docs/devel/ci.rst`) .. _ci: == CI == Most of QEMU's CI is run on GitLab's infrastructure although a number of other CI services are used for specialised purposes. The most up to date information about them and their status can be found on the `project wiki testing page `_. .. include:: ci-definitions.rst.inc .. include:: ci-jobs.rst.inc .. include:: ci-runners.rst.inc --- ### Devel Clocks (`docs/devel/clocks.rst`) Modelling a clock tree in QEMU ============================== What are clocks? ---------------- Clocks are QOM objects developed for the purpose of modelling the distribution of clocks in QEMU. They allow us to model the clock distribution of a platform and detect configuration errors in the clock tree such as badly configured PLL, clock source selection or disabled clock. The object is *Clock* and its QOM name is ``clock`` (in C code, the macro ``TYPE_CLOCK``). Clocks are typically used with devices where they are used to model inputs and outputs. They are created in a similar way to GPIOs. Inputs and outputs of different devices can be connected together. In these cases a Clock object is a child of a Device object, but this is not a requirement. Clocks can be independent of devices. For example it is possible to create a clock outside of any device to model the main clock source of a machine. Here is an example of clocks:: +---------+ +----------------------+ +--------------+ | Clock 1 | | Device B | | Device C | | | | +-------+ +-------+ | | +-------+ | | |>>-+-->>|Clock 2| |Clock 3|>>--->>|Clock 6| | +---------+ | | | (in) | | (out) | | | | (in) | | | | +-------+ +-------+ | | +-------+ | | | +-------+ | +--------------+ | | |Clock 4|>> | | | (out) | | +--------------+ | | +-------+ | | Device D | | | +-------+ | | +-------+ | | | |Clock 5|>>--->>|Clock 7| | | | | (out) | | | | (in) | | | | +-------+ | | +-------+ | | +----------------------+ | | | | +-------+ | +----------------------------->>|Clock 8| | | | (in) | | | +-------+ | +--------------+ Clocks are defined in the ``include/hw/clock.h`` header and device related functions are defined in the ``include/hw/qdev-clock.h`` header. The clock state --------------- The state of a clock is its period; it is stored as an integer representing it in units of 2 :sup:`-32` ns. The special value of 0 is used to represent the clock being inactive or gated. The clocks do not model the signal itself (pin toggling) or other properties such as the duty cycle. All clocks contain this state: outputs as well as inputs. This allows the current period of a clock to be fetched at any time. When a clock is updated, the value is immediately propagated to all connected clocks in the tree. To ease interaction with clocks, helpers with a unit suffix are defined for every clock state setter or getter. The suffixes are: - ``_ns`` for handling periods in nanoseconds - ``_hz`` for handling frequencies in hertz The 0 period value is converted to 0 in hertz and vice versa. 0 always means that the clock is disabled. Adding a new clock ------------------ Adding clocks to a device must be done during the init method of the Device instance. To add an input clock to a device, the function ``qdev_init_clock_in()`` must be used. It takes the name, a callback, an opaque parameter for the callback and a mask of events when the callback should be called (this will be explained in a following section). Output is simpler; only the name is required. Typically:: qdev_init_clock_in(DEVICE(dev), "clk_in", clk_in_callback, dev, ClockUpdate); qdev_init_clock_out(DEVICE(dev), "clk_out"); Both functions return the created Clock pointer, which should be saved in the device's state structure for further use. These objects will be automatically deleted by the QOM reference mechanism. Note that it is possible to create a static array describing clock inputs and outputs. The function ``qdev_init_clocks()`` must be called with the array as parameter to initialize the clocks: it has the same behaviour as calling the ``qdev_init_clock_in/out()`` for each clock in the array. To ease the array construction, some macros are defined in ``include/hw/qdev-clock.h``. As an example, the following creates 2 clocks to a device: one input and one output. .. code-block:: c /* device structure containing pointers to the clock objects */ typedef struct MyDeviceState { DeviceState parent_obj; Clock *clk_in; Clock *clk_out; } MyDeviceState; /* * callback for the input clock (see "Callback on input clock * change" section below for more information). */ static void clk_in_callback(void *opaque, ClockEvent event); /* * static array describing clocks: * + a clock input named "clk_in", whose pointer is stored in * the clk_in field of a MyDeviceState structure with callback * clk_in_callback. * + a clock output named "clk_out" whose pointer is stored in * the clk_out field of a MyDeviceState structure. */ static const ClockPortInitArray mydev_clocks = { QDEV_CLOCK_IN(MyDeviceState, clk_in, clk_in_callback, ClockUpdate), QDEV_CLOCK_OUT(MyDeviceState, clk_out), QDEV_CLOCK_END }; /* device initialization function */ static void mydev_init(Object *obj) { /* cast to MyDeviceState */ MyDeviceState *mydev = MYDEVICE(obj); /* create and fill the pointer fields in the MyDeviceState */ qdev_init_clocks(mydev, mydev_clocks); [...] } An alternative way to create a clock is to simply call ``object_new(TYPE_CLOCK)``. In that case the clock will neither be an input nor an output of a device. After the whole QOM hierarchy of the clock has been set ``clock_setup_canonical_path()`` should be called. At creation, the period of the clock is 0: the clock is disabled. You can change it using ``clock_set_ns()`` or ``clock_set_hz()``. Note that if you are creating a clock with a fixed period which will never change (for example the main clock source of a board), then you'll have nothing else to do. This value will be propagated to other clocks when connecting the clocks together and devices will fetch the right value during the first reset. Clock callbacks --------------- You can give a clock a callback function in several ways: * by passing it as an argument to ``qdev_init_clock_in()`` * as an argument to the ``QDEV_CLOCK_IN()`` macro initializing an array to be passed to ``qdev_init_clocks()`` * by directly calling the ``clock_set_callback()`` function The callback function must be of this type: .. code-block:: c typedef void ClockCallback(void *opaque, ClockEvent event); The ``opaque`` argument is the pointer passed to ``qdev_init_clock_in()`` or ``clock_set_callback()``; for ``qdev_init_clocks()`` it is the ``dev`` device pointer. The ``event`` argument specifies why the callback has been called. When you register the callback you specify a mask of ClockEvent values that you are interested in. The callback will only be called for those events. The events currently supported are: * ``ClockPreUpdate`` : called when the input clock's period is about to update. This is useful if the device needs to do some action for which it needs to know the old value of the clock period. During this callback, Clock API functions like ``clock_get()`` or ``clock_ticks_to_ns()`` will use the old period. * ``ClockUpdate`` : called after the input clock's period has changed. During this callback, Clock API functions like ``clock_ticks_to_ns()`` will use the new period. Note that a clock only has one callback: it is not possible to register different functions for different events. You must register a single callback which listens for all of the events you are interested in, and use the ``event`` argument to identify which event has happened. Retrieving clocks from a device ------------------------------- ``qdev_get_clock_in()`` and ``dev_get_clock_out()`` are available to get the clock inputs or outputs of a device. For example: .. code-block:: c Clock *clk = qdev_get_clock_in(DEVICE(mydev), "clk_in"); or: .. code-block:: c Clock *clk = qdev_get_clock_out(DEVICE(mydev), "clk_out"); Connecting two clocks together ------------------------------ To connect two clocks together, use the ``clock_set_source()`` function. Given two clocks ``clk1``, and ``clk2``, ``clock_set_source(clk2, clk1);`` configures ``clk2`` to follow the ``clk1`` period changes. Every time ``clk1`` is updated, ``clk2`` will be updated too. When connecting clock between devices, prefer using the ``qdev_connect_clock_in()`` function to set the source of an input device clock. For example, to connect the input clock ``clk2`` of ``devB`` to the output clock ``clk1`` of ``devA``, do: .. code-block:: c qdev_connect_clock_in(devB, "clk2", qdev_get_clock_out(devA, "clk1")) We used ``qdev_get_clock_out()`` above, but any clock can drive an input clock, even another input clock. The following diagram shows some examples of connections. Note also that a clock can drive several other clocks. :: +------------+ +--------------------------------------------------+ | Device A | | Device B | | | | +---------------------+ | | | | | Device C | | | +-------+ | | +-------+ | +-------+ +-------+ | +-------+ | | |Clock 1|>>-->>|Clock 2|>>+-->>|Clock 3| |Clock 5|>>>>|Clock 6|>> | | (out) | | | | (in) | | | | (in) | | (out) | | | (out) | | | +-------+ | | +-------+ | | +-------+ +-------+ | +-------+ | +------------+ | | +---------------------+ | | | | | | +--------------+ | | | | Device D | | | | | +-------+ | | | +-->>|Clock 4| | | | | | (in) | | | | | +-------+ | | | +--------------+ | +--------------------------------------------------+ In the above example, when *Clock 1* is updated by *Device A*, three clocks get the new clock period value: *Clock 2*, *Clock 3* and *Clock 4*. It is not possible to disconnect a clock or to change the clock connection after it is connected. Clock multiplier and divider settings ------------------------------------- By default, when clocks are connected together, the child clocks run with the same period as their source (parent) clock. The Clock API supports a built-in period multiplier/divider mechanism so you can configure a clock to make its children run at a different period from its own. If you call the ``clock_set_mul_div()`` function you can specify the clock's multiplier and divider values. The children of that clock will all run with a period of ``parent_period * multiplier / divider``. For instance, if the clock has a frequency of 8MHz and you set its multiplier to 2 and its divider to 3, the child clocks will run at 12MHz. You can change the multiplier and divider of a clock at runtime, so you can use this to model clock controller devices which have guest-programmable frequency multipliers or dividers. Note that ``clock_set_mul_div()`` does not automatically call ``clock_propagate()``. If you make a runtime change to the multiplier or divider you must call clock_propagate() yourself. Unconnected input clocks ------------------------ A newly created input clock is disabled (period of 0). This means the clock will be considered as disabled until the period is updated. If the clock remains unconnected it will always keep its initial value of 0. If this is not the desired behaviour, ``clock_set()``, ``clock_set_ns()`` or ``clock_set_hz()`` should be called on the Clock object during device instance init. For example: .. code-block:: c clk = qdev_init_clock_in(DEVICE(dev), "clk-in", clk_in_callback, dev, ClockUpdate); /* set initial value to 10ns / 100MHz */ clock_set_ns(clk, 10); To enforce that the clock is wired up by the board code, you can call ``clock_has_source()`` in your device's realize method: .. code-block:: c if (!clock_has_source(s->clk)) { error_setg(errp, "MyDevice: clk input must be connected"); return; } Note that this only checks that the clock has been wired up; it is still possible that the output clock connected to it is disabled or has not yet been configured, in which case the period will be zero. You should use the clock callback to find out when the clock period changes. Fetching clock frequency/period ------------------------------- To get the current state of a clock, use the functions ``clock_get()`` or ``clock_get_hz()``. ``clock_get()`` returns the period of the clock in its fully precise internal representation, as an unsigned 64-bit integer in units of 2^-32 nanoseconds. (For many purposes ``clock_ticks_to_ns()`` will be more convenient; see the section below on expiry deadlines.) ``clock_get_hz()`` returns the frequency of the clock, rounded to the next lowest integer. This implies some inaccuracy due to the rounding, so be cautious about using it in calculations. It is also possible to register a callback on clock frequency changes. Here is an example, which assumes that ``clock_callback`` has been specified as the callback for the ``ClockUpdate`` event: .. code-block:: c void clock_callback(void *opaque, ClockEvent event) { MyDeviceState *s = (MyDeviceState *) opaque; /* * 'opaque' is the argument passed to qdev_init_clock_in(); * usually this will be the device state pointer. */ /* do something with the new period */ fprintf(stdout, "device new period is %" PRIu64 "* 2^-32 ns\n", clock_get(dev->my_clk_input)); } If you are only interested in the frequency for displaying it to humans (for instance in debugging), use ``clock_display_freq()``, which returns a prettified string-representation, e.g. "33.3 MHz". The caller must free the string with g_free() after use. Calculating expiry deadlines ---------------------------- A commonly required operation for a clock is to calculate how long it will take for the clock to tick N times; this can then be used to set a timer expiry deadline. Use the function ``clock_ticks_to_ns()``, which takes an unsigned 64-bit count of ticks and returns the length of time in nanoseconds required for the clock to tick that many times. It is important not to try to calculate expiry deadlines using a shortcut like multiplying a "period of clock in nanoseconds" value by the tick count, because clocks can have periods which are not a whole number of nanoseconds, and the accumulated error in the multiplication can be significant. For a clock with a very long period and a large number of ticks, the result of this function could in theory be too large to fit in a 64-bit value. To avoid overflow in this case, ``clock_ticks_to_ns()`` saturates the result to INT64_MAX (because this is the largest valid input to the QEMUTimer APIs). Since INT64_MAX nanoseconds is almost 300 years, anything with an expiry later than that is in the "will never happen" category. Callers of ``clock_ticks_to_ns()`` should therefore generally not special-case the possibility of a saturated result but just allow the timer to be set to that far-future value. (If you are performing further calculations on the returned value rather than simply passing it to a QEMUTimer function like ``timer_mod_ns()`` then you should be careful to avoid overflow in those calculations, of course.) Obtaining tick counts --------------------- For calculations where you need to know the number of ticks in a given duration, use ``clock_ns_to_ticks()``. This function handles possible non-whole-number-of-nanoseconds periods and avoids potential rounding errors. It will return '0' if the clock is stopped (i.e. it has period zero). If the inputs imply a tick count that overflows a 64-bit value (a very long duration for a clock with a very short period) the output value is truncated, so effectively the 64-bit output wraps around. Changing a clock period ----------------------- A device can change its outputs using the ``clock_update()``, ``clock_update_ns()`` or ``clock_update_hz()`` function. It will trigger updates on every connected input. For example, let's say that we have an output clock *clkout* and we have a pointer to it in the device state because we did the following in init phase: .. code-block:: c dev->clkout = qdev_init_clock_out(DEVICE(dev), "clkout"); Then at any time (apart from the cases listed below), it is possible to change the clock value by doing: .. code-block:: c clock_update_hz(dev->clkout, 1000 * 1000 * 1000); /* 1GHz */ Because updating a clock may trigger any side effects through connected clocks and their callbacks, this operation must be done while holding the qemu io lock. For the same reason, one can update clocks only when it is allowed to have side effects on other objects. In consequence, it is forbidden: * during migration, * and in the enter phase of reset. Note that calling ``clock_update[_ns|_hz]()`` is equivalent to calling ``clock_set[_ns|_hz]()`` (with the same arguments) then ``clock_propagate()`` on the clock. Thus, setting the clock value can be separated from triggering the side-effects. This is often required to factorize code to handle reset and migration in devices. Aliasing clocks --------------- Sometimes, one needs to forward, or inherit, a clock from another device. Typically, when doing device composition, a device might expose a sub-device's clock without interfering with it. The function ``qdev_alias_clock()`` can be used to achieve this behaviour. Note that it is possible to expose the clock under a different name. ``qdev_alias_clock()`` works for both input and output clocks. For example, if device B is a child of device A, ``device_a_instance_init()`` may do something like this: .. code-block:: c void device_a_instance_init(Object *obj) { AState *A = DEVICE_A(obj); BState *B; /* create object B as child of A */ [...] qdev_alias_clock(B, "clk", A, "b_clk"); /* * Now A has a clock "b_clk" which is an alias to * the clock "clk" of its child B. */ } This function does not return any clock object. The new clock has the same direction (input or output) as the original one. This function only adds a link to the existing clock. In the above example, object B remains the only object allowed to use the clock and device A must not try to change the clock period or set a callback to the clock. This diagram describes the example with an input clock:: +--------------------------+ | Device A | | +--------------+ | | | Device B | | | | +-------+ | | >>"b_clk">>>| "clk" | | | | (in) | | (in) | | | | | +-------+ | | | +--------------+ | +--------------------------+ Migration --------- Clock state is not migrated automatically. Every device must handle its clock migration. Alias clocks must not be migrated. To ensure clock states are restored correctly during migration, there are two solutions. Clock states can be migrated by adding an entry into the device vmstate description. You should use the ``VMSTATE_CLOCK`` macro for this. This is typically used to migrate an input clock state. For example: .. code-block:: c MyDeviceState { DeviceState parent_obj; [...] /* some fields */ Clock *clk; }; VMStateDescription my_device_vmstate = { .name = "my_device", .fields = (VMStateField[]) { [...], /* other migrated fields */ VMSTATE_CLOCK(clk, MyDeviceState), VMSTATE_END_OF_LIST() } }; The second solution is to restore the clock state using information already at our disposal. This can be used to restore output clock states using the device state. The functions ``clock_set[_ns|_hz]()`` can be used during the ``post_load()`` migration callback. When adding clock support to an existing device, if you care about migration compatibility you will need to be careful, as simply adding a ``VMSTATE_CLOCK()`` line will break compatibility. Instead, you can put the ``VMSTATE_CLOCK()`` line into a vmstate subsection with a suitable ``needed`` function, and use ``clock_set()`` in a ``pre_load()`` function to set the default value that will be used if the source virtual machine in the migration does not send the clock state. Care should be taken not to use ``clock_update[_ns|_hz]()`` or ``clock_propagate()`` during the whole migration procedure because it will trigger side effects to other devices in an unknown state. --- ### Devel Code Of Conduct (`docs/devel/code-of-conduct.rst`) Code of Conduct =============== The QEMU community is made up of a mixture of professionals and volunteers from all over the world. Diversity is one of our strengths, but it can also lead to communication issues and unhappiness. To that end, we have a few ground rules that we ask people to adhere to. * Be welcoming. We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, or nationality. * Be respectful. Not all of us will agree all the time. Disagreements, both social and technical, happen all the time and the QEMU community is no exception. When we disagree, we try to understand why. It is important that we resolve disagreements and differing views constructively. Members of the QEMU community should be respectful when dealing with other contributors as well as with people outside the QEMU community and with users of QEMU. Harassment and other exclusionary behavior are not acceptable. A community where people feel uncomfortable or threatened is neither welcoming nor respectful. Examples of unacceptable behavior by participants include: * The use of sexualized language or imagery * Personal attacks * Trolling or insulting/derogatory comments * Public or private harassment * Publishing other's private information, such as physical or electronic addresses, without explicit permission This isn't an exhaustive list of things that you can't do. Rather, take it in the spirit in which it's intended: a guide to make it easier to be excellent to each other. This code of conduct applies to all spaces managed by the QEMU project. This includes IRC, the mailing lists, the issue tracker, community events, and any other forums created by the project team which the community uses for communication. This code of conduct also applies outside these spaces, when an individual acts as a representative or a member of the project or its community. By adopting this code of conduct, project maintainers commit themselves to fairly and consistently applying these principles to every aspect of managing this project. If you believe someone is violating the code of conduct, please read the :ref:`conflict-resolution` document for information about how to proceed. Sources ------- This document is based on the `Fedora Code of Conduct `__ (as of April 2021) and the `Contributor Covenant version 1.3.0 `__. --- ### Devel Conflict Resolution (`docs/devel/conflict-resolution.rst`) .. _conflict-resolution: Conflict Resolution Policy ========================== Conflicts in the community can take many forms, from someone having a bad day and using harsh and hurtful language on the mailing list to more serious code of conduct violations (including sexist/racist statements or threats of violence), and everything in between. For the vast majority of issues, we aim to empower individuals to first resolve conflicts themselves, asking for help when needed, and only after that fails to escalate further. This approach gives people more control over the outcome of their dispute. How we resolve conflicts ------------------------ If you are experiencing conflict, please consider first addressing the perceived conflict directly with other involved parties, preferably through a real-time medium such as IRC. You could also try to get a third-party (e.g. a mutual friend, and/or someone with background on the issue, but not involved in the conflict) to intercede or mediate. If this fails or if you do not feel comfortable proceeding this way, or if the problem requires immediate escalation, report the issue to the QEMU leadership committee by sending an email to qemu@sfconservancy.org, providing references to the misconduct. For very urgent topics, you can also inform one or more members through IRC. The up-to-date list of members is `available on the QEMU wiki `__. Your report will be treated confidentially by the leadership committee and not be published without your agreement. The QEMU leadership committee will then do its best to review the incident in a timely manner, and will either seek further information, or will make a determination on next steps. Remedies -------- Escalating an issue to the QEMU leadership committee may result in actions impacting one or more involved parties. In the event the leadership committee has to intervene, here are some of the ways they might respond: 1. Take no action. For example, if the leadership committee determines the complaint has not been substantiated or is being made in bad faith, or if it is deemed to be outside its purview. 2. A private reprimand, explaining the consequences of continued behavior, to one or more involved individuals. 3. A private reprimand and request for a private or public apology 4. A public reprimand and request for a public apology 5. A public reprimand plus a mandatory cooling off period. The cooling off period may require, for example, one or more of the following: abstaining from maintainer duties; not interacting with people involved, including unsolicited interaction with those enforcing the guidelines and interaction on social media; being denied participation to in-person events. The cooling off period is voluntary but may escalate to a temporary ban in order to enforce it. 6. A temporary or permanent ban from some or all current and future QEMU spaces (mailing lists, IRC, wiki, etc.), possibly including in-person events. In the event of severe harassment, the leadership committee may advise that the matter be escalated to the relevant local law enforcement agency. It is however not the role of the leadership committee to initiate contact with law enforcement on behalf of any of the community members involved in an incident. Sources ------- This document was developed based on the `Drupal Conflict Resolution Policy and Process `__ and the `Mozilla Consequence Ladder `__ --- ### Devel Control Flow Integrity (`docs/devel/control-flow-integrity.rst`) ============================ Control-Flow Integrity (CFI) ============================ This document describes the current control-flow integrity (CFI) mechanism in QEMU. How it can be enabled, its benefits and deficiencies, and how it affects new and existing code in QEMU Basics ------ CFI is a hardening technique that focusing on guaranteeing that indirect function calls have not been altered by an attacker. The type used in QEMU is a forward-edge control-flow integrity that ensures function calls performed through function pointers, always call a "compatible" function. A compatible function is a function with the same signature of the function pointer declared in the source code. This type of CFI is entirely compiler-based and relies on the compiler knowing the signature of every function and every function pointer used in the code. As of now, the only compiler that provides support for CFI is Clang. CFI is best used on production binaries, to protect against unknown attack vectors. In case of a CFI violation (i.e. call to a non-compatible function) QEMU will terminate abruptly, to stop the possible attack. Building with CFI ----------------- NOTE: CFI requires the use of link-time optimization. Therefore, when CFI is selected, LTO will be automatically enabled. To build with CFI, the minimum requirement is Clang 6+. If you are planning to also enable fuzzing, then Clang 11+ is needed (more on this later). Given the use of LTO, a version of AR that supports LLVM IR is required. The easies way of doing this is by selecting the AR provided by LLVM:: AR=llvm-ar-9 CC=clang-9 CXX=clang++-9 /path/to/configure --enable-cfi CFI is enabled on every binary produced. If desired, an additional flag to increase the verbosity of the output in case of a CFI violation is offered (``--enable-debug-cfi``). Using QEMU built with CFI ------------------------- A binary with CFI will work exactly like a standard binary. In case of a CFI violation, the binary will terminate with an illegal instruction signal. Incompatible code with CFI -------------------------- As mentioned above, CFI is entirely compiler-based and therefore relies on compile-time knowledge of the code. This means that, while generally supported for most code, some specific use pattern can break CFI compatibility, and create false-positives. The two main patterns that can cause issues are: * Just-in-time compiled code: since such code is created at runtime, the jump to the buffer containing JIT code will fail. * Libraries loaded dynamically, e.g. with dlopen/dlsym, since the library was not known at compile time. Current areas of QEMU that are not entirely compatible with CFI are: 1. TCG, since the idea of TCG is to pre-compile groups of instructions at runtime to speed-up interpretation, quite similarly to a JIT compiler 2. TCI, where the interpreter has to interpret the generic *call* operation 3. Plugins, since a plugin is implemented as an external library 4. Modules, since they are implemented as an external library 5. Directly calling signal handlers from the QEMU source code, since the signal handler may have been provided by an external library or even plugged at runtime. Disabling CFI for a specific function ------------------------------------- If you are working on function that is performing a call using an incompatible way, as described before, you can selectively disable CFI checks for such function by using the decorator ``QEMU_DISABLE_CFI`` at function definition, and add an explanation on why the function is not compatible with CFI. An example of the use of ``QEMU_DISABLE_CFI`` is provided here:: /* * Disable CFI checks. * TCG creates binary blobs at runtime, with the transformed code. * A TB is a blob of binary code, created at runtime and called with an * indirect function call. Since such function did not exist at compile time, * the CFI runtime has no way to verify its signature and would fail. * TCG is not considered a security-sensitive part of QEMU so this does not * affect the impact of CFI in environment with high security requirements */ QEMU_DISABLE_CFI static inline tcg_target_ulong cpu_tb_exec(CPUState *cpu, TranslationBlock *itb) NOTE: CFI needs to be disabled at the **caller** function, (i.e. a compatible cfi function that calls a non-compatible one), since the check is performed when the function call is performed. CFI and fuzzing --------------- There is generally no advantage of using CFI and fuzzing together, because they target different environments (production for CFI, debug for fuzzing). CFI could be used in conjunction with fuzzing to identify a broader set of bugs that may not end immediately in a segmentation fault or triggering an assertion. However, other sanitizers such as address and ub sanitizers can identify such bugs in a more precise way than CFI. There is, however, an interesting use case in using CFI in conjunction with fuzzing, that is to make sure that CFI is not triggering any false positive in remote-but-possible parts of the code. CFI can be enabled with fuzzing, but with some caveats: 1. Fuzzing relies on the linker performing function wrapping at link-time. The standard BFD linker does not support function wrapping when LTO is also enabled. The workaround is to use LLVM's lld linker. 2. Fuzzing also relies on a custom linker script, which is only supported by lld with version 11+. In other words, to compile with fuzzing and CFI, clang 11+ is required, and lld needs to be used as a linker:: AR=llvm-ar-11 CC=clang-11 CXX=clang++-11 /path/to/configure --enable-cfi \ -enable-fuzzing --extra-ldflags="-fuse-ld=lld" and then, compile the fuzzers as usual. --- ### Devel Decodetree (`docs/devel/decodetree.rst`) ======================== Decodetree Specification ======================== A *decodetree* is built from instruction *patterns*. A pattern may represent a single architectural instruction or a group of same, depending on what is convenient for further processing. Each pattern has both *fixedbits* and *fixedmask*, the combination of which describes the condition under which the pattern is matched:: (insn & fixedmask) == fixedbits Each pattern may have *fields*, which are extracted from the insn and passed along to the translator. Examples of such are registers, immediates, and sub-opcodes. In support of patterns, one may declare *fields*, *argument sets*, and *formats*, each of which may be re-used to simplify further definitions. Fields ====== Syntax:: field_def := '%' identifier ( unnamed_field )* ( !function=identifier )? unnamed_field := number ':' ( 's' ) number For *unnamed_field*, the first number is the least-significant bit position of the field and the second number is the length of the field. If the 's' is present, the field is considered signed. If multiple ``unnamed_fields`` are present, they are concatenated. In this way one can define disjoint fields. If ``!function`` is specified, the concatenated result is passed through the named function, taking and returning an integral value. One may use ``!function`` with zero ``unnamed_fields``. This case is called a *parameter*, and the named function is only passed the ``DisasContext`` and returns an integral value extracted from there. A field with no ``unnamed_fields`` and no ``!function`` is in error. Field examples: +---------------------------+---------------------------------------------+ | Input | Generated code | +===========================+=============================================+ | %disp 0:s16 | sextract(i, 0, 16) | +---------------------------+---------------------------------------------+ | %imm9 16:6 10:3 | extract(i, 16, 6) << 3 | extract(i, 10, 3) | +---------------------------+---------------------------------------------+ | %disp12 0:s1 1:1 2:10 | sextract(i, 0, 1) << 11 | | | | extract(i, 1, 1) << 10 | | | | extract(i, 2, 10) | +---------------------------+---------------------------------------------+ | %shimm8 5:s8 13:1 | expand_shimm8(sextract(i, 5, 8) << 1 | | | !function=expand_shimm8 | extract(i, 13, 1)) | +---------------------------+---------------------------------------------+ Argument Sets ============= Syntax:: args_def := '&' identifier ( args_elt )+ ( !extern )? args_elt := identifier (':' identifier)? Each *args_elt* defines an argument within the argument set. If the form of the *args_elt* contains a colon, the first identifier is the argument name and the second identifier is the argument type. If the colon is missing, the argument type will be ``int``. Each argument set will be rendered as a C structure "arg_$name" with each of the fields being one of the member arguments. If ``!extern`` is specified, the backing structure is assumed to have been already declared, typically via a second decoder. Argument sets are useful when one wants to define helper functions for the translator functions that can perform operations on a common set of arguments. This can ensure, for instance, that the ``AND`` pattern and the ``OR`` pattern put their operands into the same named structure, so that a common ``gen_logic_insn`` may be able to handle the operations common between the two. Argument set examples:: ®3 ra rb rc &loadstore reg base offset &longldst reg base offset:int64_t Formats ======= Syntax:: fmt_def := '@' identifier ( fmt_elt )+ fmt_elt := fixedbit_elt | field_elt | field_ref | args_ref fixedbit_elt := [01.-]+ field_elt := identifier ':' 's'? number field_ref := '%' identifier | identifier '=' '%' identifier args_ref := '&' identifier Defining a format is a handy way to avoid replicating groups of fields across many instruction patterns. A *fixedbit_elt* describes a contiguous sequence of bits that must be 1, 0, or don't care. The difference between '.' and '-' is that '.' means that the bit will be covered with a field or a final 0 or 1 from the pattern, and '-' means that the bit is really ignored by the cpu and will not be specified. A *field_elt* describes a simple field only given a width; the position of the field is implied by its position with respect to other *fixedbit_elt* and *field_elt*. If any *fixedbit_elt* or *field_elt* appear, then all bits must be defined. Padding with a *fixedbit_elt* of all '.' is an easy way to accomplish that. A *field_ref* incorporates a field by reference. This is the only way to add a complex field to a format. A field may be renamed in the process via assignment to another identifier. This is intended to allow the same argument set be used with disjoint named fields. A single *args_ref* may specify an argument set to use for the format. The set of fields in the format must be a subset of the arguments in the argument set. If an argument set is not specified, one will be inferred from the set of fields. It is recommended, but not required, that all *field_ref* and *args_ref* appear at the end of the line, not interleaving with *fixedbit_elf* or *field_elt*. Format examples:: @opr ...... ra:5 rb:5 ... 0 ....... rc:5 @opi ...... ra:5 lit:8 1 ....... rc:5 Patterns ======== Syntax:: pat_def := identifier ( pat_elt )+ pat_elt := fixedbit_elt | field_elt | field_ref | args_ref | fmt_ref | const_elt fmt_ref := '@' identifier const_elt := identifier '=' number The *fixedbit_elt* and *field_elt* specifiers are unchanged from formats. A pattern that does not specify a named format will have one inferred from a referenced argument set (if present) and the set of fields. A *const_elt* allows a argument to be set to a constant value. This may come in handy when fields overlap between patterns and one has to include the values in the *fixedbit_elt* instead. The decoder will call a translator function for each pattern matched. Pattern examples:: addl_r 010000 ..... ..... .... 0000000 ..... @opr addl_i 010000 ..... ..... .... 0000000 ..... @opi which will, in part, invoke:: trans_addl_r(ctx, &arg_opr, insn) and:: trans_addl_i(ctx, &arg_opi, insn) Pattern Groups ============== Syntax:: group := overlap_group | no_overlap_group overlap_group := '{' ( pat_def | group )+ '}' no_overlap_group := '[' ( pat_def | group )+ ']' A *group* begins with a lone open-brace or open-bracket, with all subsequent lines indented two spaces, and ending with a lone close-brace or close-bracket. Groups may be nested, increasing the required indentation of the lines within the nested group to two spaces per nesting level. Patterns within overlap groups are allowed to overlap. Conflicts are resolved by selecting the patterns in order. If all of the fixedbits for a pattern match, its translate function will be called. If the translate function returns false, then subsequent patterns within the group will be matched. Patterns within no-overlap groups are not allowed to overlap, just the same as ungrouped patterns. Thus no-overlap groups are intended to be nested inside overlap groups. The following example from PA-RISC shows specialization of the *or* instruction:: { { nop 000010 ----- ----- 0000 001001 0 00000 copy 000010 00000 r1:5 0000 001001 0 rt:5 } or 000010 rt2:5 r1:5 cf:4 001001 0 rt:5 } When the *cf* field is zero, the instruction has no side effects, and may be specialized. When the *rt* field is zero, the output is discarded and so the instruction has no effect. When the *rt2* field is zero, the operation is ``reg[r1] | 0`` and so encodes the canonical register copy operation. The output from the generator might look like:: switch (insn & 0xfc000fe0) { case 0x08000240: /* 000010.. ........ ....0010 010..... */ if ((insn & 0x0000f000) == 0x00000000) { /* 000010.. ........ 00000010 010..... */ if ((insn & 0x0000001f) == 0x00000000) { /* 000010.. ........ 00000010 01000000 */ extract_decode_Fmt_0(&u.f_decode0, insn); if (trans_nop(ctx, &u.f_decode0)) return true; } if ((insn & 0x03e00000) == 0x00000000) { /* 00001000 000..... 00000010 010..... */ extract_decode_Fmt_1(&u.f_decode1, insn); if (trans_copy(ctx, &u.f_decode1)) return true; } } extract_decode_Fmt_2(&u.f_decode2, insn); if (trans_or(ctx, &u.f_decode2)) return true; return false; } --- ### Devel Ebpf Rss (`docs/devel/ebpf_rss.rst`) =========================== eBPF RSS virtio-net support =========================== RSS(Receive Side Scaling) is used to distribute network packets to guest virtqueues by calculating packet hash. Usually every queue is processed then by a specific guest CPU core. For now there are 2 RSS implementations in qemu: - 'in-qemu' RSS (functions if qemu receives network packets, i.e. vhost=off) - eBPF RSS (can function with also with vhost=on) eBPF support (CONFIG_EBPF) is enabled by 'configure' script. To enable eBPF RSS support use './configure --enable-bpf'. If steering BPF is not set for kernel's TUN module, the TUN uses automatic selection of rx virtqueue based on lookup table built according to calculated symmetric hash of transmitted packets. If steering BPF is set for TUN the BPF code calculates the hash of packet header and returns the virtqueue number to place the packet to. Simplified decision formula: .. code:: C queue_index = indirection_table[hash()%] Not for all packets, the hash can/should be calculated. Note: currently, eBPF RSS does not support hash reporting. eBPF RSS turned on by different combinations of vhost-net, vitrio-net and tap configurations: - eBPF is used: tap,vhost=off & virtio-net-pci,rss=on,hash=off - eBPF is used: tap,vhost=on & virtio-net-pci,rss=on,hash=off - 'in-qemu' RSS is used: tap,vhost=off & virtio-net-pci,rss=on,hash=on - eBPF is used, hash population feature is not reported to the guest: tap,vhost=on & virtio-net-pci,rss=on,hash=on If CONFIG_EBPF is not set then only 'in-qemu' RSS is supported. Also 'in-qemu' RSS, as a fallback, is used if the eBPF program failed to load or set to TUN. RSS eBPF program ---------------- RSS program located in ebpf/rss.bpf.skeleton.h generated by bpftool. So the program is part of the qemu binary. Initially, the eBPF program was compiled by clang and source code located at tools/ebpf/rss.bpf.c. Prerequisites to recompile the eBPF program (regenerate ebpf/rss.bpf.skeleton.h): llvm, clang, kernel source tree, bpftool Adjust Makefile.ebpf to reflect the location of the kernel source tree $ cd tools/ebpf $ make -f Makefile.ebpf Current eBPF RSS implementation uses 'bounded loops' with 'backward jump instructions' which present in the last kernels. Overall eBPF RSS works on kernels 5.8+. eBPF RSS implementation ----------------------- eBPF RSS loading functionality located in ebpf/ebpf_rss.c and ebpf/ebpf_rss.h. The ``struct EBPFRSSContext`` structure that holds 4 file descriptors: - ctx - pointer of the libbpf context. - program_fd - file descriptor of the eBPF RSS program. - map_configuration - file descriptor of the 'configuration' map. This map contains one element of 'struct EBPFRSSConfig'. This configuration determines eBPF program behavior. - map_toeplitz_key - file descriptor of the 'Toeplitz key' map. One element of the 40byte key prepared for the hashing algorithm. - map_indirections_table - 128 elements of queue indexes. ``struct EBPFRSSConfig`` fields: - redirect - "boolean" value, should the hash be calculated, on false - ``default_queue`` would be used as the final decision. - populate_hash - for now, not used. eBPF RSS doesn't support hash reporting. - hash_types - binary mask of different hash types. See ``VIRTIO_NET_RSS_HASH_TYPE_*`` defines. If for packet hash should not be calculated - ``default_queue`` would be used. - indirections_len - length of the indirections table, maximum 128. - default_queue - the queue index that used for packet that shouldn't be hashed. For some packets, the hash can't be calculated(g.e ARP). Functions: - ``ebpf_rss_init()`` - sets ctx to NULL, which indicates that EBPFRSSContext is not loaded. - ``ebpf_rss_load()`` - creates 3 maps and loads eBPF program from the rss.bpf.skeleton.h. Returns 'true' on success. After that, program_fd can be used to set steering for TAP. - ``ebpf_rss_set_all()`` - sets values for eBPF maps. ``indirections_table`` length is in EBPFRSSConfig. ``toeplitz_key`` is VIRTIO_NET_RSS_MAX_KEY_SIZE aka 40 bytes array. - ``ebpf_rss_unload()`` - close all file descriptors and set ctx to NULL. Simplified eBPF RSS workflow: .. code:: C struct EBPFRSSConfig config; config.redirect = 1; config.hash_types = VIRTIO_NET_RSS_HASH_TYPE_UDPv4 | VIRTIO_NET_RSS_HASH_TYPE_TCPv4; config.indirections_len = VIRTIO_NET_RSS_MAX_TABLE_LEN; config.default_queue = 0; uint16_t table[VIRTIO_NET_RSS_MAX_TABLE_LEN] = {...}; uint8_t key[VIRTIO_NET_RSS_MAX_KEY_SIZE] = {...}; struct EBPFRSSContext ctx; ebpf_rss_init(&ctx); ebpf_rss_load(&ctx); ebpf_rss_set_all(&ctx, &config, table, key); if (net_client->info->set_steering_ebpf != NULL) { net_client->info->set_steering_ebpf(net_client, ctx->program_fd); } ... ebpf_unload(&ctx); NetClientState SetSteeringEBPF() ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For now, ``set_steering_ebpf()`` method supported by Linux TAP NetClientState. The method requires an eBPF program file descriptor as an argument. --- ### Devel Fuzzing (`docs/devel/fuzzing.rst`) ======== Fuzzing ======== This document describes the virtual-device fuzzing infrastructure in QEMU and how to use it to implement additional fuzzers. Basics ------ Fuzzing operates by passing inputs to an entry point/target function. The fuzzer tracks the code coverage triggered by the input. Based on these findings, the fuzzer mutates the input and repeats the fuzzing. To fuzz QEMU, we rely on libfuzzer. Unlike other fuzzers such as AFL, libfuzzer is an *in-process* fuzzer. For the developer, this means that it is their responsibility to ensure that state is reset between fuzzing-runs. Building the fuzzers -------------------- *NOTE*: If possible, build a 32-bit binary. When forking, the 32-bit fuzzer is much faster, since the page-map has a smaller size. This is due to the fact that AddressSanitizer maps ~20TB of memory, as part of its detection. This results in a large page-map, and a much slower ``fork()``. To build the fuzzers, install a recent version of clang: Configure with (substitute the clang binaries with the version you installed). Here, enable-sanitizers, is optional but it allows us to reliably detect bugs such as out-of-bounds accesses, use-after-frees, double-frees etc.:: CC=clang-8 CXX=clang++-8 /path/to/configure --enable-fuzzing \ --enable-sanitizers Fuzz targets are built similarly to system targets:: make qemu-fuzz-i386 This builds ``./qemu-fuzz-i386`` The first option to this command is: ``--fuzz-target=FUZZ_NAME`` To list all of the available fuzzers run ``qemu-fuzz-i386`` with no arguments. For example:: ./qemu-fuzz-i386 --fuzz-target=virtio-scsi-fuzz Internally, libfuzzer parses all arguments that do not begin with ``"--"``. Information about these is available by passing ``-help=1`` Now the only thing left to do is wait for the fuzzer to trigger potential crashes. Useful libFuzzer flags ---------------------- As mentioned above, libFuzzer accepts some arguments. Passing ``-help=1`` will list the available arguments. In particular, these arguments might be helpful: * ``CORPUS_DIR/`` : Specify a directory as the last argument to libFuzzer. libFuzzer stores each "interesting" input in this corpus directory. The next time you run libFuzzer, it will read all of the inputs from the corpus, and continue fuzzing from there. You can also specify multiple directories. libFuzzer loads existing inputs from all specified directories, but will only write new ones to the first one specified. * ``-max_len=4096`` : specify the maximum byte-length of the inputs libFuzzer will generate. * ``-close_fd_mask={1,2,3}`` : close, stderr, or both. Useful for targets that trigger many debug/error messages, or create output on the serial console. * ``-jobs=4 -workers=4`` : These arguments configure libFuzzer to run 4 fuzzers in parallel (4 fuzzing jobs in 4 worker processes). Alternatively, with only ``-jobs=N``, libFuzzer automatically spawns a number of workers less than or equal to half the available CPU cores. Replace 4 with a number appropriate for your machine. Make sure to specify a ``CORPUS_DIR``, which will allow the parallel fuzzers to share information about the interesting inputs they find. * ``-use_value_profile=1`` : For each comparison operation, libFuzzer computes ``(caller_pc&4095) | (popcnt(Arg1 ^ Arg2) << 12)`` and places this in the coverage table. Useful for targets with "magic" constants. If Arg1 came from the fuzzer's input and Arg2 is a magic constant, then each time the Hamming distance between Arg1 and Arg2 decreases, libFuzzer adds the input to the corpus. * ``-shrink=1`` : Tries to make elements of the corpus "smaller". Might lead to better coverage performance, depending on the target. Note that libFuzzer's exact behavior will depend on the version of clang and libFuzzer used to build the device fuzzers. Generating Coverage Reports --------------------------- Code coverage is a crucial metric for evaluating a fuzzer's performance. libFuzzer's output provides a "cov: " column that provides a total number of unique blocks/edges covered. To examine coverage on a line-by-line basis we can use Clang coverage: 1. Configure libFuzzer to store a corpus of all interesting inputs (see CORPUS_DIR above) 2. ``./configure`` the QEMU build with :: --enable-fuzzing \ --extra-cflags="-fprofile-instr-generate -fcoverage-mapping" 3. Re-run the fuzzer. Specify $CORPUS_DIR/* as an argument, telling libfuzzer to execute all of the inputs in $CORPUS_DIR and exit. Once the process exits, you should find a file, "default.profraw" in the working directory. 4. Execute these commands to generate a detailed HTML coverage-report:: llvm-profdata merge -output=default.profdata default.profraw llvm-cov show ./path/to/qemu-fuzz-i386 -instr-profile=default.profdata \ --format html -output-dir=/path/to/output/report Adding a new fuzzer ------------------- Coverage over virtual devices can be improved by adding additional fuzzers. Fuzzers are kept in ``tests/qtest/fuzz/`` and should be added to ``tests/qtest/fuzz/meson.build`` Fuzzers can rely on both qtest and libqos to communicate with virtual devices. 1. Create a new source file. For example ``tests/qtest/fuzz/foo-device-fuzz.c``. 2. Write the fuzzing code using the libqtest/libqos API. See existing fuzzers for reference. 3. Add the fuzzer to ``tests/qtest/fuzz/meson.build``. Fuzzers can be more-or-less thought of as special qtest programs which can modify the qtest commands and/or qtest command arguments based on inputs provided by libfuzzer. Libfuzzer passes a byte array and length. Commonly the fuzzer loops over the byte-array interpreting it as a list of qtest commands, addresses, or values. The Generic Fuzzer ------------------ Writing a fuzz target can be a lot of effort (especially if a device driver has not be built-out within libqos). Many devices can be fuzzed to some degree, without any device-specific code, using the generic-fuzz target. The generic-fuzz target is capable of fuzzing devices over their PIO, MMIO, and DMA input-spaces. To apply the generic-fuzz to a device, we need to define two env-variables, at minimum: * ``QEMU_FUZZ_ARGS=`` is the set of QEMU arguments used to configure a machine, with the device attached. For example, if we want to fuzz the virtio-net device attached to a pc-i440fx machine, we can specify:: QEMU_FUZZ_ARGS="-M pc -nodefaults -netdev user,id=user0 \ -device virtio-net,netdev=user0" * ``QEMU_FUZZ_OBJECTS=`` is a set of space-delimited strings used to identify the MemoryRegions that will be fuzzed. These strings are compared against MemoryRegion names and MemoryRegion owner names, to decide whether each MemoryRegion should be fuzzed. These strings support globbing. For the virtio-net example, we could use one of :: QEMU_FUZZ_OBJECTS='virtio-net' QEMU_FUZZ_OBJECTS='virtio*' QEMU_FUZZ_OBJECTS='virtio* pcspk' # Fuzz the virtio devices and the speaker QEMU_FUZZ_OBJECTS='*' # Fuzz the whole machine`` The ``"info mtree"`` and ``"info qom-tree"`` monitor commands can be especially useful for identifying the ``MemoryRegion`` and ``Object`` names used for matching. As a generic rule-of-thumb, the more ``MemoryRegions``/Devices we match, the greater the input-space, and the smaller the probability of finding crashing inputs for individual devices. As such, it is usually a good idea to limit the fuzzer to only a few ``MemoryRegions``. To ensure that these env variables have been configured correctly, we can use:: ./qemu-fuzz-i386 --fuzz-target=generic-fuzz -runs=0 The output should contain a complete list of matched MemoryRegions. OSS-Fuzz -------- QEMU is continuously fuzzed on `OSS-Fuzz `_. By default, the OSS-Fuzz build will try to fuzz every fuzz-target. Since the generic-fuzz target requires additional information provided in environment variables, we pre-define some generic-fuzz configs in ``tests/qtest/fuzz/generic_fuzz_configs.h``. Each config must specify: - ``.name``: To identify the fuzzer config - ``.args`` OR ``.argfunc``: A string or pointer to a function returning a string. These strings are used to specify the ``QEMU_FUZZ_ARGS`` environment variable. ``argfunc`` is useful when the config relies on e.g. a dynamically created temp directory, or a free tcp/udp port. - ``.objects``: A string that specifies the ``QEMU_FUZZ_OBJECTS`` environment variable. To fuzz additional devices/device configuration on OSS-Fuzz, send patches for either a new device-specific fuzzer or a new generic-fuzz config. Build details: - The Dockerfile that sets up the environment for building QEMU's fuzzers on OSS-Fuzz can be fund in the OSS-Fuzz repository __(https://github.com/google/oss-fuzz/blob/master/projects/qemu/Dockerfile) - The script responsible for building the fuzzers can be found in the QEMU source tree at ``scripts/oss-fuzz/build.sh`` Building Crash Reproducers ----------------------------------------- When we find a crash, we should try to create an independent reproducer, that can be used on a non-fuzzer build of QEMU. This filters out any potential false-positives, and improves the debugging experience for developers. Here are the steps for building a reproducer for a crash found by the generic-fuzz target. - Ensure the crash reproduces:: qemu-fuzz-i386 --fuzz-target... ./crash-... - Gather the QTest output for the crash:: QEMU_FUZZ_TIMEOUT=0 QTEST_LOG=1 FUZZ_SERIALIZE_QTEST=1 \ qemu-fuzz-i386 --fuzz-target... ./crash-... &> /tmp/trace - Reorder and clean-up the resulting trace:: scripts/oss-fuzz/reorder_fuzzer_qtest_trace.py /tmp/trace > /tmp/reproducer - Get the arguments needed to start qemu, and provide a path to qemu:: less /tmp/trace # The args should be logged at the top of this file export QEMU_ARGS="-machine ..." export QEMU_PATH="path/to/qemu-system" - Ensure the crash reproduces in qemu-system:: $QEMU_PATH $QEMU_ARGS -qtest stdio < /tmp/reproducer - From the crash output, obtain some string that identifies the crash. This can be a line in the stack-trace, for example:: export CRASH_TOKEN="hw/usb/hcd-xhci.c:1865" - Minimize the reproducer:: scripts/oss-fuzz/minimize_qtest_trace.py -M1 -M2 \ /tmp/reproducer /tmp/reproducer-minimized - Confirm that the minimized reproducer still crashes:: $QEMU_PATH $QEMU_ARGS -qtest stdio < /tmp/reproducer-minimized - Create a one-liner reproducer that can be sent over email:: ./scripts/oss-fuzz/output_reproducer.py -bash /tmp/reproducer-minimized - Output the C source code for a test case that will reproduce the bug:: ./scripts/oss-fuzz/output_reproducer.py -owner "John Smith "\ -name "test_function_name" /tmp/reproducer-minimized - Report the bug and send a patch with the C reproducer upstream Implementation Details / Fuzzer Lifecycle ----------------------------------------- The fuzzer has two entrypoints that libfuzzer calls. libfuzzer provides it's own ``main()``, which performs some setup, and calls the entrypoints: ``LLVMFuzzerInitialize``: called prior to fuzzing. Used to initialize all of the necessary state ``LLVMFuzzerTestOneInput``: called for each fuzzing run. Processes the input and resets the state at the end of each run. In more detail: ``LLVMFuzzerInitialize`` parses the arguments to the fuzzer (must start with two dashes, so they are ignored by libfuzzer ``main()``). Currently, the arguments select the fuzz target. Then, the qtest client is initialized. If the target requires qos, qgraph is set up and the QOM/LIBQOS modules are initialized. Then the QGraph is walked and the QEMU cmd_line is determined and saved. After this, the ``vl.c:main`` is called to set up the guest. There are target-specific hooks that can be called before and after main, for additional setup(e.g. PCI setup, or VM snapshotting). ``LLVMFuzzerTestOneInput``: Uses qtest/qos functions to act based on the fuzz input. It is also responsible for manually calling ``main_loop_wait`` to ensure that bottom halves are executed and any cleanup required before the next input. Since the same process is reused for many fuzzing runs, QEMU state needs to be reset at the end of each run. There are currently two implemented options for resetting state: - Reboot the guest between runs. - *Pros*: Straightforward and fast for simple fuzz targets. - *Cons*: Depending on the device, does not reset all device state. If the device requires some initialization prior to being ready for fuzzing (common for QOS-based targets), this initialization needs to be done after each reboot. - *Example target*: ``i440fx-qtest-reboot-fuzz`` - Run each test case in a separate forked process and copy the coverage information back to the parent. This is fairly similar to AFL's "deferred" fork-server mode [3] - *Pros*: Relatively fast. Devices only need to be initialized once. No need to do slow reboots or vmloads. - *Cons*: Not officially supported by libfuzzer. Does not work well for devices that rely on dedicated threads. - *Example target*: ``virtio-net-fork-fuzz`` --- ### Devel Index Api (`docs/devel/index-api.rst`) Internal QEMU APIs ------------------ Details about how QEMU's various internal APIs. Most of these are generated from in-code annotations to function prototypes. .. toctree:: :maxdepth: 2 bitops loads-stores memory modules ui --- ### Devel Index Build (`docs/devel/index-build.rst`) QEMU Build and Test System -------------------------- Details about how QEMU's build system works and how it is integrated into our testing infrastructure. You will need to understand some of the basics if you are adding new files and targets to the build. .. toctree:: :maxdepth: 3 build-system kconfig testing qtest ci qapi-code-gen fuzzing control-flow-integrity --- ### Devel Index Internals (`docs/devel/index-internals.rst`) Internal Subsystem Information ------------------------------ Details about QEMU's various subsystems including how to add features to them. .. toctree:: :maxdepth: 2 qom atomics block-coroutine-wrapper clocks ebpf_rss migration multi-process reset s390-dasd-ipl tracing vfio-migration writing-monitor-commands virtio-backends --- ### Devel Index Process (`docs/devel/index-process.rst`) QEMU Community Processes ------------------------ Notes about how to interact with the community and how and where to submit patches. .. toctree:: :maxdepth: 2 code-of-conduct conflict-resolution style submitting-a-patch trivial-patches stable-process submitting-a-pull-request secure-coding-practices --- ### Devel Index Tcg (`docs/devel/index-tcg.rst`) TCG Emulation ------------- Details about QEMU's Tiny Code Generator and the infrastructure associated with emulation. You do not need to worry about this if you are only implementing things for HW accelerated hypervisors. .. toctree:: :maxdepth: 2 tcg decodetree multi-thread-tcg tcg-icount tcg-plugins replay --- ### Devel Index (`docs/devel/index.rst`) --------------------- Developer Information --------------------- This section of the manual documents various parts of the internals of QEMU. You only need to read it if you are interested in reading or modifying QEMU's source code. .. toctree:: :maxdepth: 1 index-process index-build index-api index-internals index-tcg --- ### Devel Kconfig (`docs/devel/kconfig.rst`) .. _kconfig: ================ QEMU and Kconfig ================ QEMU is a very versatile emulator; it can be built for a variety of targets, where each target can emulate various boards and at the same time different targets can share large amounts of code. For example, a POWER and an x86 board can run the same code to emulate a PCI network card, even though the boards use different PCI host bridges, and they can run the same code to emulate a SCSI disk while using different SCSI adapters. Arm, s390 and x86 boards can all present a virtio-blk disk to their guests, but with three different virtio guest interfaces. Each QEMU target enables a subset of the boards, devices and buses that are included in QEMU's source code. As a result, each QEMU executable only links a small subset of the files that form QEMU's source code; anything that is not needed to support a particular target is culled. QEMU uses a simple domain-specific language to describe the dependencies between components. This is useful for two reasons: * new targets and boards can be added without knowing in detail the architecture of the hardware emulation subsystems. Boards only have to list the components they need, and the compiled executable will include all the required dependencies and all the devices that the user can add to that board; * users can easily build reduced versions of QEMU that support only a subset of boards or devices. For example, by default most targets will include all emulated PCI devices that QEMU supports, but the build process is configurable and it is easy to drop unnecessary (or otherwise unwanted) code to make a leaner binary. This domain-specific language is based on the Kconfig language that originated in the Linux kernel, though it was heavily simplified and the handling of dependencies is stricter in QEMU. Unlike Linux, there is no user interface to edit the configuration, which is instead specified in per-target files under the ``default-configs/`` directory of the QEMU source tree. This is because, unlike Linux, configuration and dependencies can be treated as a black box when building QEMU; the default configuration that QEMU ships with should be okay in almost all cases. The Kconfig language -------------------- Kconfig defines configurable components in files named ``hw/*/Kconfig``. Note that configurable components are _not_ visible in C code as preprocessor symbols; they are only visible in the Makefile. Each configurable component defines a Makefile variable whose name starts with ``CONFIG_``. All elements have boolean (true/false) type; truth is written as ``y``, while falsehood is written ``n``. They are defined in a Kconfig stanza like the following:: config ARM_VIRT bool imply PCI_DEVICES imply VFIO_AMD_XGBE imply VFIO_XGMAC select A15MPCORE select ACPI select ARM_SMMUV3 The ``config`` keyword introduces a new configuration element. In the example above, Makefiles will have access to a variable named ``CONFIG_ARM_VIRT``, with value ``y`` or ``n`` (respectively for boolean true and false). Boolean expressions can be used within the language, whenever ```` is written in the remainder of this section. The ``&&``, ``||`` and ``!`` operators respectively denote conjunction (AND), disjunction (OR) and negation (NOT). The ``bool`` data type declaration is optional, but it is suggested to include it for clarity and future-proofing. After ``bool`` the following directives can be included: **dependencies**: ``depends on `` This defines a dependency for this configurable element. Dependencies evaluate an expression and force the value of the variable to false if the expression is false. **reverse dependencies**: ``select [if ]`` While ``depends on`` can force a symbol to false, reverse dependencies can be used to force another symbol to true. In the following example, ``CONFIG_BAZ`` will be true whenever ``CONFIG_FOO`` is true:: config FOO select BAZ The optional expression will prevent ``select`` from having any effect unless it is true. Note that unlike Linux's Kconfig implementation, QEMU will detect contradictions between ``depends on`` and ``select`` statements and prevent you from building such a configuration. **default value**: ``default [if ]`` Default values are assigned to the config symbol if no other value was set by the user via ``default-configs/*.mak`` files, and only if ``select`` or ``depends on`` directives do not force the value to true or false respectively. ```` can be ``y`` or ``n``; it cannot be an arbitrary Boolean expression. However, a condition for applying the default value can be added with ``if``. A configuration element can have any number of default values (usually, if more than one default is present, they will have different conditions). If multiple default values satisfy their condition, only the first defined one is active. **reverse default** (weak reverse dependency): ``imply [if ]`` This is similar to ``select`` as it applies a lower limit of ``y`` to another symbol. However, the lower limit is only a default and the "implied" symbol's value may still be set to ``n`` from a ``default-configs/*.mak`` files. The following two examples are equivalent:: config FOO bool imply BAZ config BAZ bool default y if FOO The next section explains where to use ``imply`` or ``default y``. Guidelines for writing Kconfig files ------------------------------------ Configurable elements in QEMU fall under five broad groups. Each group declares its dependencies in different ways: **subsystems**, of which **buses** are a special case Example:: config SCSI bool Subsystems always default to false (they have no ``default`` directive) and are never visible in ``default-configs/*.mak`` files. It's up to other symbols to ``select`` whatever subsystems they require. They sometimes have ``select`` directives to bring in other required subsystems or buses. For example, ``AUX`` (the DisplayPort auxiliary channel "bus") selects ``I2C`` because it can act as an I2C master too. **devices** Example:: config MEGASAS_SCSI_PCI bool default y if PCI_DEVICES depends on PCI select SCSI Devices are the most complex of the five. They can have a variety of directives that cooperate so that a default configuration includes all the devices that can be accessed from QEMU. Devices *depend on* the bus that they lie on, for example a PCI device would specify ``depends on PCI``. An MMIO device will likely have no ``depends on`` directive. Devices also *select* the buses that the device provides, for example a SCSI adapter would specify ``select SCSI``. Finally, devices are usually ``default y`` if and only if they have at least one ``depends on``; the default could be conditional on a device group. Devices also select any optional subsystem that they use; for example a video card might specify ``select EDID`` if it needs to build EDID information and publish it to the guest. **device groups** Example:: config PCI_DEVICES bool Device groups provide a convenient mechanism to enable/disable many devices in one go. This is useful when a set of devices is likely to be enabled/disabled by several targets. Device groups usually need no directive and are not used in the Makefile either; they only appear as conditions for ``default y`` directives. QEMU currently has three device groups, ``PCI_DEVICES``, ``I2C_DEVICES``, and ``TEST_DEVICES``. PCI devices usually have a ``default y if PCI_DEVICES`` directive rather than just ``default y``. This lets some boards (notably s390) easily support a subset of PCI devices, for example only VFIO (passthrough) and virtio-pci devices. ``I2C_DEVICES`` is similar to ``PCI_DEVICES``. It contains i2c devices that users might reasonably want to plug in to an i2c bus on any board (and not ones which are very board-specific or that need to be wired up in a way that can't be done on the command line). ``TEST_DEVICES`` instead is used for devices that are rarely used on production virtual machines, but provide useful hooks to test QEMU or KVM. **boards** Example:: config SUN4M bool imply TCX imply CG3 select CS4231 select ECCMEMCTL select EMPTY_SLOT select ESCC select ESP select FDC select SLAVIO select LANCE select M48T59 select STP2000 Boards specify their constituent devices using ``imply`` and ``select`` directives. A device should be listed under ``select`` if the board cannot be started at all without it. It should be listed under ``imply`` if (depending on the QEMU command line) the board may or may not be started without it. Boards also default to false; they are enabled by the ``default-configs/*.mak`` for the target they apply to. **internal elements** Example:: config ECCMEMCTL bool select ECC Internal elements group code that is useful in several boards or devices. They are usually enabled with ``select`` and in turn select other elements; they are never visible in ``default-configs/*.mak`` files, and often not even in the Makefile. Writing and modifying default configurations -------------------------------------------- In addition to the Kconfig files under hw/, each target also includes a file called ``default-configs/TARGETNAME-softmmu.mak``. These files initialize some Kconfig variables to non-default values and provide the starting point to turn on devices and subsystems. A file in ``default-configs/`` looks like the following example:: # Default configuration for alpha-softmmu # Uncomment the following lines to disable these optional devices: # #CONFIG_PCI_DEVICES=n #CONFIG_TEST_DEVICES=n # Boards: # CONFIG_DP264=y The first part, consisting of commented-out ``=n`` assignments, tells the user which devices or device groups are implied by the boards. The second part, consisting of ``=y`` assignments, tells the user which boards are supported by the target. The user will typically modify the default configuration by uncommenting lines in the first group, or commenting out lines in the second group. It is also possible to run QEMU's configure script with the ``--without-default-devices`` option. When this is done, everything defaults to ``n`` unless it is ``select``ed or explicitly switched on in the ``.mak`` files. In other words, ``default`` and ``imply`` directives are disabled. When QEMU is built with this option, the user will probably want to change some lines in the first group, for example like this:: CONFIG_PCI_DEVICES=y #CONFIG_TEST_DEVICES=n and/or pick a subset of the devices in those device groups. Right now there is no single place that lists all the optional devices for ``CONFIG_PCI_DEVICES`` and ``CONFIG_TEST_DEVICES``. In the future, we expect that ``.mak`` files will be automatically generated, so that they will include all these symbols and some help text on what they do. ``Kconfig.host`` ---------------- In some special cases, a configurable element depends on host features that are detected by QEMU's configure or ``meson.build`` scripts; for example some devices depend on the availability of KVM or on the presence of a library on the host. These symbols should be listed in ``Kconfig.host`` like this:: config TPM bool and also listed as follows in the top-level meson.build's host_kconfig variable:: host_kconfig = \ (have_tpm ? ['CONFIG_TPM=y'] : []) + \ ('CONFIG_SPICE' in config_host ? ['CONFIG_SPICE=y'] : []) + \ (have_ivshmem ? ['CONFIG_IVSHMEM=y'] : []) + \ ... --- ### Devel Loads Stores (`docs/devel/loads-stores.rst`) .. Copyright (c) 2017 Linaro Limited Written by Peter Maydell =================== Load and Store APIs =================== QEMU internally has multiple families of functions for performing loads and stores. This document attempts to enumerate them all and indicate when to use them. It does not provide detailed documentation of each API -- for that you should look at the documentation comments in the relevant header files. ``ld*_p and st*_p`` ~~~~~~~~~~~~~~~~~~~ These functions operate on a host pointer, and should be used when you already have a pointer into host memory (corresponding to guest ram or a local buffer). They deal with doing accesses with the desired endianness and with correctly handling potentially unaligned pointer values. Function names follow the pattern: load: ``ld{sign}{size}_{endian}_p(ptr)`` store: ``st{size}_{endian}_p(ptr, val)`` ``sign`` - (empty) : for 32 or 64 bit sizes - ``u`` : unsigned - ``s`` : signed ``size`` - ``b`` : 8 bits - ``w`` : 16 bits - ``l`` : 32 bits - ``q`` : 64 bits ``endian`` - ``he`` : host endian - ``be`` : big endian - ``le`` : little endian The ``_{endian}`` infix is omitted for target-endian accesses. The target endian accessors are only available to source files which are built per-target. There are also functions which take the size as an argument: load: ``ldn{endian}_p(ptr, sz)`` which performs an unsigned load of ``sz`` bytes from ``ptr`` as an ``{endian}`` order value and returns it in a uint64_t. store: ``stn{endian}_p(ptr, sz, val)`` which stores ``val`` to ``ptr`` as an ``{endian}`` order value of size ``sz`` bytes. Regexes for git grep - ``\`` - ``\`` - ``\`` - ``\`` ``cpu_{ld,st}*_mmu`` ~~~~~~~~~~~~~~~~~~~~ These functions operate on a guest virtual address, plus a context known as a "mmu index" which controls how that virtual address is translated, plus a ``MemOp`` which contains alignment requirements among other things. The ``MemOp`` and mmu index are combined into a single argument of type ``MemOpIdx``. The meaning of the indexes are target specific, but specifying a particular index might be necessary if, for instance, the helper requires a "always as non-privileged" access rather than the default access for the current state of the guest CPU. These functions may cause a guest CPU exception to be taken (e.g. for an alignment fault or MMU fault) which will result in guest CPU state being updated and control longjmp'ing out of the function call. They should therefore only be used in code that is implementing emulation of the guest CPU. The ``retaddr`` parameter is used to control unwinding of the guest CPU state in case of a guest CPU exception. This is passed to ``cpu_restore_state()``. Therefore the value should either be 0, to indicate that the guest CPU state is already synchronized, or the result of ``GETPC()`` from the top level ``HELPER(foo)`` function, which is a return address into the generated code [#gpc]_. .. [#gpc] Note that ``GETPC()`` should be used with great care: calling it in other functions that are *not* the top level ``HELPER(foo)`` will cause unexpected behavior. Instead, the value of ``GETPC()`` should be read from the helper and passed if needed to the functions that the helper calls. Function names follow the pattern: load: ``cpu_ld{size}{end}_mmu(env, ptr, oi, retaddr)`` store: ``cpu_st{size}{end}_mmu(env, ptr, val, oi, retaddr)`` ``size`` - ``b`` : 8 bits - ``w`` : 16 bits - ``l`` : 32 bits - ``q`` : 64 bits ``end`` - (empty) : for target endian, or 8 bit sizes - ``_be`` : big endian - ``_le`` : little endian Regexes for git grep: - ``\`` - ``\`` ``cpu_{ld,st}*_mmuidx_ra`` ~~~~~~~~~~~~~~~~~~~~~~~~~~ These functions work like the ``cpu_{ld,st}_mmu`` functions except that the ``mmuidx`` parameter is not combined with a ``MemOp``, and therefore there is no required alignment supplied or enforced. Function names follow the pattern: load: ``cpu_ld{sign}{size}{end}_mmuidx_ra(env, ptr, mmuidx, retaddr)`` store: ``cpu_st{size}{end}_mmuidx_ra(env, ptr, val, mmuidx, retaddr)`` ``sign`` - (empty) : for 32 or 64 bit sizes - ``u`` : unsigned - ``s`` : signed ``size`` - ``b`` : 8 bits - ``w`` : 16 bits - ``l`` : 32 bits - ``q`` : 64 bits ``end`` - (empty) : for target endian, or 8 bit sizes - ``_be`` : big endian - ``_le`` : little endian Regexes for git grep: - ``\`` - ``\`` ``cpu_{ld,st}*_data_ra`` ~~~~~~~~~~~~~~~~~~~~~~~~ These functions work like the ``cpu_{ld,st}_mmuidx_ra`` functions except that the ``mmuidx`` parameter is taken from the current mode of the guest CPU, as determined by ``cpu_mmu_index(env, false)``. These are generally the preferred way to do accesses by guest virtual address from helper functions, unless the access should be performed with a context other than the default, or alignment should be enforced for the access. Function names follow the pattern: load: ``cpu_ld{sign}{size}{end}_data_ra(env, ptr, ra)`` store: ``cpu_st{size}{end}_data_ra(env, ptr, val, ra)`` ``sign`` - (empty) : for 32 or 64 bit sizes - ``u`` : unsigned - ``s`` : signed ``size`` - ``b`` : 8 bits - ``w`` : 16 bits - ``l`` : 32 bits - ``q`` : 64 bits ``end`` - (empty) : for target endian, or 8 bit sizes - ``_be`` : big endian - ``_le`` : little endian Regexes for git grep: - ``\`` - ``\`` ``cpu_{ld,st}*_data`` ~~~~~~~~~~~~~~~~~~~~~ These functions work like the ``cpu_{ld,st}_data_ra`` functions except that the ``retaddr`` parameter is 0, and thus does not unwind guest CPU state. This means they must only be used from helper functions where the translator has saved all necessary CPU state. These functions are the right choice for calls made from hooks like the CPU ``do_interrupt`` hook or when you know for certain that the translator had to save all the CPU state anyway. Function names follow the pattern: load: ``cpu_ld{sign}{size}{end}_data(env, ptr)`` store: ``cpu_st{size}{end}_data(env, ptr, val)`` ``sign`` - (empty) : for 32 or 64 bit sizes - ``u`` : unsigned - ``s`` : signed ``size`` - ``b`` : 8 bits - ``w`` : 16 bits - ``l`` : 32 bits - ``q`` : 64 bits ``end`` - (empty) : for target endian, or 8 bit sizes - ``_be`` : big endian - ``_le`` : little endian Regexes for git grep - ``\`` - ``\`` ``cpu_ld*_code`` ~~~~~~~~~~~~~~~~ These functions perform a read for instruction execution. The ``mmuidx`` parameter is taken from the current mode of the guest CPU, as determined by ``cpu_mmu_index(env, true)``. The ``retaddr`` parameter is 0, and thus does not unwind guest CPU state, because CPU state is always synchronized while translating instructions. Any guest CPU exception that is raised will indicate an instruction execution fault rather than a data read fault. In general these functions should not be used directly during translation. There are wrapper functions that are to be used which also take care of plugins for tracing. Function names follow the pattern: load: ``cpu_ld{sign}{size}_code(env, ptr)`` ``sign`` - (empty) : for 32 or 64 bit sizes - ``u`` : unsigned - ``s`` : signed ``size`` - ``b`` : 8 bits - ``w`` : 16 bits - ``l`` : 32 bits - ``q`` : 64 bits Regexes for git grep: - ``\`` ``translator_ld*`` ~~~~~~~~~~~~~~~~~~ These functions are a wrapper for ``cpu_ld*_code`` which also perform any actions required by any tracing plugins. They are only to be called during the translator callback ``translate_insn``. There is a set of functions ending in ``_swap`` which, if the parameter is true, returns the value in the endianness that is the reverse of the guest native endianness, as determined by ``TARGET_BIG_ENDIAN``. Function names follow the pattern: load: ``translator_ld{sign}{size}(env, ptr)`` swap: ``translator_ld{sign}{size}_swap(env, ptr, swap)`` ``sign`` - (empty) : for 32 or 64 bit sizes - ``u`` : unsigned - ``s`` : signed ``size`` - ``b`` : 8 bits - ``w`` : 16 bits - ``l`` : 32 bits - ``q`` : 64 bits Regexes for git grep - ``\`` ``helper_*_{ld,st}*_mmu`` ~~~~~~~~~~~~~~~~~~~~~~~~~ These functions are intended primarily to be called by the code generated by the TCG backend. They may also be called by target CPU helper function code. Like the ``cpu_{ld,st}_mmuidx_ra`` functions they perform accesses by guest virtual address, with a given ``mmuidx``. These functions specify an ``opindex`` parameter which encodes (among other things) the mmu index to use for the access. This parameter should be created by calling ``make_memop_idx()``. The ``retaddr`` parameter should be the result of GETPC() called directly from the top level HELPER(foo) function (or 0 if no guest CPU state unwinding is required). **TODO** The names of these functions are a bit odd for historical reasons because they were originally expected to be called only from within generated code. We should rename them to bring them more in line with the other memory access functions. The explicit endianness is the only feature they have beyond ``*_mmuidx_ra``. load: ``helper_{endian}_ld{sign}{size}_mmu(env, addr, opindex, retaddr)`` store: ``helper_{endian}_st{size}_mmu(env, addr, val, opindex, retaddr)`` ``sign`` - (empty) : for 32 or 64 bit sizes - ``u`` : unsigned - ``s`` : signed ``size`` - ``b`` : 8 bits - ``w`` : 16 bits - ``l`` : 32 bits - ``q`` : 64 bits ``endian`` - ``le`` : little endian - ``be`` : big endian - ``ret`` : target endianness Regexes for git grep - ``\`` - ``\`` ``address_space_*`` ~~~~~~~~~~~~~~~~~~~ These functions are the primary ones to use when emulating CPU or device memory accesses. They take an AddressSpace, which is the way QEMU defines the view of memory that a device or CPU has. (They generally correspond to being the "master" end of a hardware bus or bus fabric.) Each CPU has an AddressSpace. Some kinds of CPU have more than one AddressSpace (for instance Arm guest CPUs have an AddressSpace for the Secure world and one for NonSecure if they implement TrustZone). Devices which can do DMA-type operations should generally have an AddressSpace. There is also a "system address space" which typically has all the devices and memory that all CPUs can see. (Some older device models use the "system address space" rather than properly modelling that they have an AddressSpace of their own.) Functions are provided for doing byte-buffer reads and writes, and also for doing one-data-item loads and stores. In all cases the caller provides a MemTxAttrs to specify bus transaction attributes, and can check whether the memory transaction succeeded using a MemTxResult return code. ``address_space_read(address_space, addr, attrs, buf, len)`` ``address_space_write(address_space, addr, attrs, buf, len)`` ``address_space_rw(address_space, addr, attrs, buf, len, is_write)`` ``address_space_ld{sign}{size}_{endian}(address_space, addr, attrs, txresult)`` ``address_space_st{size}_{endian}(address_space, addr, val, attrs, txresult)`` ``sign`` - (empty) : for 32 or 64 bit sizes - ``u`` : unsigned (No signed load operations are provided.) ``size`` - ``b`` : 8 bits - ``w`` : 16 bits - ``l`` : 32 bits - ``q`` : 64 bits ``endian`` - ``le`` : little endian - ``be`` : big endian The ``_{endian}`` suffix is omitted for byte accesses. Regexes for git grep - ``\`` - ``\`` - ``\`` ``address_space_write_rom`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~ This function performs a write by physical address like ``address_space_write``, except that if the write is to a ROM then the ROM contents will be modified, even though a write by the guest CPU to the ROM would be ignored. This is used for non-guest writes like writes from the gdb debug stub or initial loading of ROM contents. Note that portions of the write which attempt to write data to a device will be silently ignored -- only real RAM and ROM will be written to. Regexes for git grep - ``address_space_write_rom`` ``{ld,st}*_phys`` ~~~~~~~~~~~~~~~~~ These are functions which are identical to ``address_space_{ld,st}*``, except that they always pass ``MEMTXATTRS_UNSPECIFIED`` for the transaction attributes, and ignore whether the transaction succeeded or failed. The fact that they ignore whether the transaction succeeded means they should not be used in new code, unless you know for certain that your code will only be used in a context where the CPU or device doing the access has no way to report such an error. ``load: ld{sign}{size}_{endian}_phys`` ``store: st{size}_{endian}_phys`` ``sign`` - (empty) : for 32 or 64 bit sizes - ``u`` : unsigned (No signed load operations are provided.) ``size`` - ``b`` : 8 bits - ``w`` : 16 bits - ``l`` : 32 bits - ``q`` : 64 bits ``endian`` - ``le`` : little endian - ``be`` : big endian The ``_{endian}_`` infix is omitted for byte accesses. Regexes for git grep - ``\`` - ``\`` ``cpu_physical_memory_*`` ~~~~~~~~~~~~~~~~~~~~~~~~~ These are convenience functions which are identical to ``address_space_*`` but operate specifically on the system address space, always pass a ``MEMTXATTRS_UNSPECIFIED`` set of memory attributes and ignore whether the memory transaction succeeded or failed. For new code they are better avoided: * there is likely to be behaviour you need to model correctly for a failed read or write operation * a device should usually perform operations on its own AddressSpace rather than using the system address space ``cpu_physical_memory_read`` ``cpu_physical_memory_write`` ``cpu_physical_memory_rw`` Regexes for git grep - ``\`` ``cpu_memory_rw_debug`` ~~~~~~~~~~~~~~~~~~~~~~~ Access CPU memory by virtual address for debug purposes. This function is intended for use by the GDB stub and similar code. It takes a virtual address, converts it to a physical address via an MMU lookup using the current settings of the specified CPU, and then performs the access (using ``address_space_rw`` for reads or ``cpu_physical_memory_write_rom`` for writes). This means that if the access is a write to a ROM then this function will modify the contents (whereas a normal guest CPU access would ignore the write attempt). ``cpu_memory_rw_debug`` ``dma_memory_*`` ~~~~~~~~~~~~~~~~ These behave like ``address_space_*``, except that they perform a DMA barrier operation first. **TODO**: We should provide guidance on when you need the DMA barrier operation and when it's OK to use ``address_space_*``, and make sure our existing code is doing things correctly. ``dma_memory_read`` ``dma_memory_write`` ``dma_memory_rw`` Regexes for git grep - ``\`` - ``\`` - ``\`` ``pci_dma_*`` and ``{ld,st}*_pci_dma`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ These functions are specifically for PCI device models which need to perform accesses where the PCI device is a bus master. You pass them a ``PCIDevice *`` and they will do ``dma_memory_*`` operations on the correct address space for that device. ``pci_dma_read`` ``pci_dma_write`` ``pci_dma_rw`` ``load: ld{sign}{size}_{endian}_pci_dma`` ``store: st{size}_{endian}_pci_dma`` ``sign`` - (empty) : for 32 or 64 bit sizes - ``u`` : unsigned (No signed load operations are provided.) ``size`` - ``b`` : 8 bits - ``w`` : 16 bits - ``l`` : 32 bits - ``q`` : 64 bits ``endian`` - ``le`` : little endian - ``be`` : big endian The ``_{endian}_`` infix is omitted for byte accesses. Regexes for git grep - ``\`` - ``\`` - ``\`` --- ### Devel Lockcnt (`docs/devel/lockcnt.txt`) DOCUMENTATION FOR LOCKED COUNTERS (aka QemuLockCnt) =================================================== QEMU often uses reference counts to track data structures that are being accessed and should not be freed. For example, a loop that invoke callbacks like this is not safe: QLIST_FOREACH_SAFE(ioh, &io_handlers, next, pioh) { if (ioh->revents & G_IO_OUT) { ioh->fd_write(ioh->opaque); } } QLIST_FOREACH_SAFE protects against deletion of the current node (ioh) by stashing away its "next" pointer. However, ioh->fd_write could actually delete the next node from the list. The simplest way to avoid this is to mark the node as deleted, and remove it from the list in the above loop: QLIST_FOREACH_SAFE(ioh, &io_handlers, next, pioh) { if (ioh->deleted) { QLIST_REMOVE(ioh, next); g_free(ioh); } else { if (ioh->revents & G_IO_OUT) { ioh->fd_write(ioh->opaque); } } } If however this loop must also be reentrant, i.e. it is possible that ioh->fd_write invokes the loop again, some kind of counting is needed: walking_handlers++; QLIST_FOREACH_SAFE(ioh, &io_handlers, next, pioh) { if (ioh->deleted) { if (walking_handlers == 1) { QLIST_REMOVE(ioh, next); g_free(ioh); } } else { if (ioh->revents & G_IO_OUT) { ioh->fd_write(ioh->opaque); } } } walking_handlers--; One may think of using the RCU primitives, rcu_read_lock() and rcu_read_unlock(); effectively, the RCU nesting count would take the place of the walking_handlers global variable. Indeed, reference counting and RCU have similar purposes, but their usage in general is complementary: - reference counting is fine-grained and limited to a single data structure; RCU delays reclamation of *all* RCU-protected data structures; - reference counting works even in the presence of code that keeps a reference for a long time; RCU critical sections in principle should be kept short; - reference counting is often applied to code that is not thread-safe but is reentrant; in fact, usage of reference counting in QEMU predates the introduction of threads by many years. RCU is generally used to protect readers from other threads freeing memory after concurrent modifications to a data structure. - reclaiming data can be done by a separate thread in the case of RCU; this can improve performance, but also delay reclamation undesirably. With reference counting, reclamation is deterministic. This file documents QemuLockCnt, an abstraction for using reference counting in code that has to be both thread-safe and reentrant. QemuLockCnt concepts -------------------- A QemuLockCnt comprises both a counter and a mutex; it has primitives to increment and decrement the counter, and to take and release the mutex. The counter notes how many visits to the data structures are taking place (the visits could be from different threads, or there could be multiple reentrant visits from the same thread). The basic rules governing the counter/mutex pair then are the following: - Data protected by the QemuLockCnt must not be freed unless the counter is zero and the mutex is taken. - A new visit cannot be started while the counter is zero and the mutex is taken. Most of the time, the mutex protects all writes to the data structure, not just frees, though there could be cases where this is not necessary. Reads, instead, can be done without taking the mutex, as long as the readers and writers use the same macros that are used for RCU, for example qatomic_rcu_read, qatomic_rcu_set, QLIST_FOREACH_RCU, etc. This is because the reads are done outside a lock and a set or QLIST_INSERT_HEAD can happen concurrently with the read. The RCU API ensures that the processor and the compiler see all required memory barriers. This could be implemented simply by protecting the counter with the mutex, for example: // (1) qemu_mutex_lock(&walking_handlers_mutex); walking_handlers++; qemu_mutex_unlock(&walking_handlers_mutex); ... // (2) qemu_mutex_lock(&walking_handlers_mutex); if (--walking_handlers == 0) { QLIST_FOREACH_SAFE(ioh, &io_handlers, next, pioh) { if (ioh->deleted) { QLIST_REMOVE(ioh, next); g_free(ioh); } } } qemu_mutex_unlock(&walking_handlers_mutex); Here, no frees can happen in the code represented by the ellipsis. If another thread is executing critical section (2), that part of the code cannot be entered, because the thread will not be able to increment the walking_handlers variable. And of course during the visit any other thread will see a nonzero value for walking_handlers, as in the single-threaded code. Note that it is possible for multiple concurrent accesses to delay the cleanup arbitrarily; in other words, for the walking_handlers counter to never become zero. For this reason, this technique is more easily applicable if concurrent access to the structure is rare. However, critical sections are easy to forget since you have to do them for each modification of the counter. QemuLockCnt ensures that all modifications of the counter take the lock appropriately, and it can also be more efficient in two ways: - it avoids taking the lock for many operations (for example incrementing the counter while it is non-zero); - on some platforms, one can implement QemuLockCnt to hold the lock and the mutex in a single word, making the fast path no more expensive than simply managing a counter using atomic operations (see docs/devel/atomics.rst). This can be very helpful if concurrent access to the data structure is expected to be rare. Using the same mutex for frees and writes can still incur some small inefficiencies; for example, a visit can never start if the counter is zero and the mutex is taken---even if the mutex is taken by a write, which in principle need not block a visit of the data structure. However, these are usually not a problem if any of the following assumptions are valid: - concurrent access is possible but rare - writes are rare - writes are frequent, but this kind of write (e.g. appending to a list) has a very small critical section. For example, QEMU uses QemuLockCnt to manage an AioContext's list of bottom halves and file descriptor handlers. Modifications to the list of file descriptor handlers are rare. Creation of a new bottom half is frequent and can happen on a fast path; however: 1) it is almost never concurrent with a visit to the list of bottom halves; 2) it only has three instructions in the critical path, two assignments and a smp_wmb(). QemuLockCnt API --------------- The QemuLockCnt API is described in include/qemu/thread.h. QemuLockCnt usage ----------------- This section explains the typical usage patterns for QemuLockCnt functions. Setting a variable to a non-NULL value can be done between qemu_lockcnt_lock and qemu_lockcnt_unlock: qemu_lockcnt_lock(&xyz_lockcnt); if (!xyz) { new_xyz = g_new(XYZ, 1); ... qatomic_rcu_set(&xyz, new_xyz); } qemu_lockcnt_unlock(&xyz_lockcnt); Accessing the value can be done between qemu_lockcnt_inc and qemu_lockcnt_dec: qemu_lockcnt_inc(&xyz_lockcnt); if (xyz) { XYZ *p = qatomic_rcu_read(&xyz); ... /* Accesses can now be done through "p". */ } qemu_lockcnt_dec(&xyz_lockcnt); Freeing the object can similarly use qemu_lockcnt_lock and qemu_lockcnt_unlock, but you also need to ensure that the count is zero (i.e. there is no concurrent visit). Because qemu_lockcnt_inc takes the QemuLockCnt's lock, the count cannot become non-zero while the object is being freed. Freeing an object looks like this: qemu_lockcnt_lock(&xyz_lockcnt); if (!qemu_lockcnt_count(&xyz_lockcnt)) { g_free(xyz); xyz = NULL; } qemu_lockcnt_unlock(&xyz_lockcnt); If an object has to be freed right after a visit, you can combine the decrement, the locking and the check on count as follows: qemu_lockcnt_inc(&xyz_lockcnt); if (xyz) { XYZ *p = qatomic_rcu_read(&xyz); ... /* Accesses can now be done through "p". */ } if (qemu_lockcnt_dec_and_lock(&xyz_lockcnt)) { g_free(xyz); xyz = NULL; qemu_lockcnt_unlock(&xyz_lockcnt); } QemuLockCnt can also be used to access a list as follows: qemu_lockcnt_inc(&io_handlers_lockcnt); QLIST_FOREACH_RCU(ioh, &io_handlers, pioh) { if (ioh->revents & G_IO_OUT) { ioh->fd_write(ioh->opaque); } } if (qemu_lockcnt_dec_and_lock(&io_handlers_lockcnt)) { QLIST_FOREACH_SAFE(ioh, &io_handlers, next, pioh) { if (ioh->deleted) { QLIST_REMOVE(ioh, next); g_free(ioh); } } qemu_lockcnt_unlock(&io_handlers_lockcnt); } Again, the RCU primitives are used because new items can be added to the list during the walk. QLIST_FOREACH_RCU ensures that the processor and the compiler see the appropriate memory barriers. An alternative pattern uses qemu_lockcnt_dec_if_lock: qemu_lockcnt_inc(&io_handlers_lockcnt); QLIST_FOREACH_SAFE_RCU(ioh, &io_handlers, next, pioh) { if (ioh->deleted) { if (qemu_lockcnt_dec_if_lock(&io_handlers_lockcnt)) { QLIST_REMOVE(ioh, next); g_free(ioh); qemu_lockcnt_inc_and_unlock(&io_handlers_lockcnt); } } else { if (ioh->revents & G_IO_OUT) { ioh->fd_write(ioh->opaque); } } } qemu_lockcnt_dec(&io_handlers_lockcnt); Here you can use qemu_lockcnt_dec instead of qemu_lockcnt_dec_and_lock, because there is no special task to do if the count goes from 1 to 0. --- ### Devel Memory (`docs/devel/memory.rst`) ============== The memory API ============== The memory API models the memory and I/O buses and controllers of a QEMU machine. It attempts to allow modelling of: - ordinary RAM - memory-mapped I/O (MMIO) - memory controllers that can dynamically reroute physical memory regions to different destinations The memory model provides support for - tracking RAM changes by the guest - setting up coalesced memory for kvm - setting up ioeventfd regions for kvm Memory is modelled as an acyclic graph of MemoryRegion objects. Sinks (leaves) are RAM and MMIO regions, while other nodes represent buses, memory controllers, and memory regions that have been rerouted. In addition to MemoryRegion objects, the memory API provides AddressSpace objects for every root and possibly for intermediate MemoryRegions too. These represent memory as seen from the CPU or a device's viewpoint. Types of regions ---------------- There are multiple types of memory regions (all represented by a single C type MemoryRegion): - RAM: a RAM region is simply a range of host memory that can be made available to the guest. You typically initialize these with memory_region_init_ram(). Some special purposes require the variants memory_region_init_resizeable_ram(), memory_region_init_ram_from_file(), or memory_region_init_ram_ptr(). - MMIO: a range of guest memory that is implemented by host callbacks; each read or write causes a callback to be called on the host. You initialize these with memory_region_init_io(), passing it a MemoryRegionOps structure describing the callbacks. - ROM: a ROM memory region works like RAM for reads (directly accessing a region of host memory), and forbids writes. You initialize these with memory_region_init_rom(). - ROM device: a ROM device memory region works like RAM for reads (directly accessing a region of host memory), but like MMIO for writes (invoking a callback). You initialize these with memory_region_init_rom_device(). - IOMMU region: an IOMMU region translates addresses of accesses made to it and forwards them to some other target memory region. As the name suggests, these are only needed for modelling an IOMMU, not for simple devices. You initialize these with memory_region_init_iommu(). - container: a container simply includes other memory regions, each at a different offset. Containers are useful for grouping several regions into one unit. For example, a PCI BAR may be composed of a RAM region and an MMIO region. A container's subregions are usually non-overlapping. In some cases it is useful to have overlapping regions; for example a memory controller that can overlay a subregion of RAM with MMIO or ROM, or a PCI controller that does not prevent card from claiming overlapping BARs. You initialize a pure container with memory_region_init(). - alias: a subsection of another region. Aliases allow a region to be split apart into discontiguous regions. Examples of uses are memory banks used when the guest address space is smaller than the amount of RAM addressed, or a memory controller that splits main memory to expose a "PCI hole". You can also create aliases to avoid trying to add the original region to multiple parents via `memory_region_add_subregion`. Aliases may point to any type of region, including other aliases, but an alias may not point back to itself, directly or indirectly. You initialize these with memory_region_init_alias(). - reservation region: a reservation region is primarily for debugging. It claims I/O space that is not supposed to be handled by QEMU itself. The typical use is to track parts of the address space which will be handled by the host kernel when KVM is enabled. You initialize these by passing a NULL callback parameter to memory_region_init_io(). It is valid to add subregions to a region which is not a pure container (that is, to an MMIO, RAM or ROM region). This means that the region will act like a container, except that any addresses within the container's region which are not claimed by any subregion are handled by the container itself (ie by its MMIO callbacks or RAM backing). However it is generally possible to achieve the same effect with a pure container one of whose subregions is a low priority "background" region covering the whole address range; this is often clearer and is preferred. Subregions cannot be added to an alias region. Migration --------- Where the memory region is backed by host memory (RAM, ROM and ROM device memory region types), this host memory needs to be copied to the destination on migration. These APIs which allocate the host memory for you will also register the memory so it is migrated: - memory_region_init_ram() - memory_region_init_rom() - memory_region_init_rom_device() For most devices and boards this is the correct thing. If you have a special case where you need to manage the migration of the backing memory yourself, you can call the functions: - memory_region_init_ram_nomigrate() - memory_region_init_rom_nomigrate() - memory_region_init_rom_device_nomigrate() which only initialize the MemoryRegion and leave handling migration to the caller. The functions: - memory_region_init_resizeable_ram() - memory_region_init_ram_from_file() - memory_region_init_ram_from_fd() - memory_region_init_ram_ptr() - memory_region_init_ram_device_ptr() are for special cases only, and so they do not automatically register the backing memory for migration; the caller must manage migration if necessary. Region names ------------ Regions are assigned names by the constructor. For most regions these are only used for debugging purposes, but RAM regions also use the name to identify live migration sections. This means that RAM region names need to have ABI stability. Region lifecycle ---------------- A region is created by one of the memory_region_init*() functions and attached to an object, which acts as its owner or parent. QEMU ensures that the owner object remains alive as long as the region is visible to the guest, or as long as the region is in use by a virtual CPU or another device. For example, the owner object will not die between an address_space_map operation and the corresponding address_space_unmap. After creation, a region can be added to an address space or a container with memory_region_add_subregion(), and removed using memory_region_del_subregion(). Various region attributes (read-only, dirty logging, coalesced mmio, ioeventfd) can be changed during the region lifecycle. They take effect as soon as the region is made visible. This can be immediately, later, or never. Destruction of a memory region happens automatically when the owner object dies. If however the memory region is part of a dynamically allocated data structure, you should call object_unparent() to destroy the memory region before the data structure is freed. For an example see VFIOMSIXInfo and VFIOQuirk in hw/vfio/pci.c. You must not destroy a memory region as long as it may be in use by a device or CPU. In order to do this, as a general rule do not create or destroy memory regions dynamically during a device's lifetime, and only call object_unparent() in the memory region owner's instance_finalize callback. The dynamically allocated data structure that contains the memory region then should obviously be freed in the instance_finalize callback as well. If you break this rule, the following situation can happen: - the memory region's owner had a reference taken via memory_region_ref (for example by address_space_map) - the region is unparented, and has no owner anymore - when address_space_unmap is called, the reference to the memory region's owner is leaked. There is an exception to the above rule: it is okay to call object_unparent at any time for an alias or a container region. It is therefore also okay to create or destroy alias and container regions dynamically during a device's lifetime. This exceptional usage is valid because aliases and containers only help QEMU building the guest's memory map; they are never accessed directly. memory_region_ref and memory_region_unref are never called on aliases or containers, and the above situation then cannot happen. Exploiting this exception is rarely necessary, and therefore it is discouraged, but nevertheless it is used in a few places. For regions that "have no owner" (NULL is passed at creation time), the machine object is actually used as the owner. Since instance_finalize is never called for the machine object, you must never call object_unparent on regions that have no owner, unless they are aliases or containers. Overlapping regions and priority -------------------------------- Usually, regions may not overlap each other; a memory address decodes into exactly one target. In some cases it is useful to allow regions to overlap, and sometimes to control which of an overlapping regions is visible to the guest. This is done with memory_region_add_subregion_overlap(), which allows the region to overlap any other region in the same container, and specifies a priority that allows the core to decide which of two regions at the same address are visible (highest wins). Priority values are signed, and the default value is zero. This means that you can use memory_region_add_subregion_overlap() both to specify a region that must sit 'above' any others (with a positive priority) and also a background region that sits 'below' others (with a negative priority). If the higher priority region in an overlap is a container or alias, then the lower priority region will appear in any "holes" that the higher priority region has left by not mapping subregions to that area of its address range. (This applies recursively -- if the subregions are themselves containers or aliases that leave holes then the lower priority region will appear in these holes too.) For example, suppose we have a container A of size 0x8000 with two subregions B and C. B is a container mapped at 0x2000, size 0x4000, priority 2; C is an MMIO region mapped at 0x0, size 0x6000, priority 1. B currently has two of its own subregions: D of size 0x1000 at offset 0 and E of size 0x1000 at offset 0x2000. As a diagram:: 0 1000 2000 3000 4000 5000 6000 7000 8000 |------|------|------|------|------|------|------|------| A: [ ] C: [CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC] B: [ ] D: [DDDDD] E: [EEEEE] The regions that will be seen within this address range then are:: [CCCCCCCCCCCC][DDDDD][CCCCC][EEEEE][CCCCC] Since B has higher priority than C, its subregions appear in the flat map even where they overlap with C. In ranges where B has not mapped anything C's region appears. If B had provided its own MMIO operations (ie it was not a pure container) then these would be used for any addresses in its range not handled by D or E, and the result would be:: [CCCCCCCCCCCC][DDDDD][BBBBB][EEEEE][BBBBB] Priority values are local to a container, because the priorities of two regions are only compared when they are both children of the same container. This means that the device in charge of the container (typically modelling a bus or a memory controller) can use them to manage the interaction of its child regions without any side effects on other parts of the system. In the example above, the priorities of D and E are unimportant because they do not overlap each other. It is the relative priority of B and C that causes D and E to appear on top of C: D and E's priorities are never compared against the priority of C. Visibility ---------- The memory core uses the following rules to select a memory region when the guest accesses an address: - all direct subregions of the root region are matched against the address, in descending priority order - if the address lies outside the region offset/size, the subregion is discarded - if the subregion is a leaf (RAM or MMIO), the search terminates, returning this leaf region - if the subregion is a container, the same algorithm is used within the subregion (after the address is adjusted by the subregion offset) - if the subregion is an alias, the search is continued at the alias target (after the address is adjusted by the subregion offset and alias offset) - if a recursive search within a container or alias subregion does not find a match (because of a "hole" in the container's coverage of its address range), then if this is a container with its own MMIO or RAM backing the search terminates, returning the container itself. Otherwise we continue with the next subregion in priority order - if none of the subregions match the address then the search terminates with no match found Example memory map ------------------ :: system_memory: container@0-2^48-1 | +---- lomem: alias@0-0xdfffffff ---> #ram (0-0xdfffffff) | +---- himem: alias@0x100000000-0x11fffffff ---> #ram (0xe0000000-0xffffffff) | +---- vga-window: alias@0xa0000-0xbffff ---> #pci (0xa0000-0xbffff) | (prio 1) | +---- pci-hole: alias@0xe0000000-0xffffffff ---> #pci (0xe0000000-0xffffffff) pci (0-2^32-1) | +--- vga-area: container@0xa0000-0xbffff | | | +--- alias@0x00000-0x7fff ---> #vram (0x010000-0x017fff) | | | +--- alias@0x08000-0xffff ---> #vram (0x020000-0x027fff) | +---- vram: ram@0xe1000000-0xe1ffffff | +---- vga-mmio: mmio@0xe2000000-0xe200ffff ram: ram@0x00000000-0xffffffff This is a (simplified) PC memory map. The 4GB RAM block is mapped into the system address space via two aliases: "lomem" is a 1:1 mapping of the first 3.5GB; "himem" maps the last 0.5GB at address 4GB. This leaves 0.5GB for the so-called PCI hole, that allows a 32-bit PCI bus to exist in a system with 4GB of memory. The memory controller diverts addresses in the range 640K-768K to the PCI address space. This is modelled using the "vga-window" alias, mapped at a higher priority so it obscures the RAM at the same addresses. The vga window can be removed by programming the memory controller; this is modelled by removing the alias and exposing the RAM underneath. The pci address space is not a direct child of the system address space, since we only want parts of it to be visible (we accomplish this using aliases). It has two subregions: vga-area models the legacy vga window and is occupied by two 32K memory banks pointing at two sections of the framebuffer. In addition the vram is mapped as a BAR at address e1000000, and an additional BAR containing MMIO registers is mapped after it. Note that if the guest maps a BAR outside the PCI hole, it would not be visible as the pci-hole alias clips it to a 0.5GB range. MMIO Operations --------------- MMIO regions are provided with ->read() and ->write() callbacks, which are sufficient for most devices. Some devices change behaviour based on the attributes used for the memory transaction, or need to be able to respond that the access should provoke a bus error rather than completing successfully; those devices can use the ->read_with_attrs() and ->write_with_attrs() callbacks instead. In addition various constraints can be supplied to control how these callbacks are called: - .valid.min_access_size, .valid.max_access_size define the access sizes (in bytes) which the device accepts; accesses outside this range will have device and bus specific behaviour (ignored, or machine check) - .valid.unaligned specifies that the *device being modelled* supports unaligned accesses; if false, unaligned accesses will invoke the appropriate bus or CPU specific behaviour. - .impl.min_access_size, .impl.max_access_size define the access sizes (in bytes) supported by the *implementation*; other access sizes will be emulated using the ones available. For example a 4-byte write will be emulated using four 1-byte writes, if .impl.max_access_size = 1. - .impl.unaligned specifies that the *implementation* supports unaligned accesses; if false, unaligned accesses will be emulated by two aligned accesses. API Reference ------------- .. kernel-doc:: include/exec/memory.h --- ### Devel Migration (`docs/devel/migration.rst`) ========= Migration ========= QEMU has code to load/save the state of the guest that it is running. These are two complementary operations. Saving the state just does that, saves the state for each device that the guest is running. Restoring a guest is just the opposite operation: we need to load the state of each device. For this to work, QEMU has to be launched with the same arguments the two times. I.e. it can only restore the state in one guest that has the same devices that the one it was saved (this last requirement can be relaxed a bit, but for now we can consider that configuration has to be exactly the same). Once that we are able to save/restore a guest, a new functionality is requested: migration. This means that QEMU is able to start in one machine and being "migrated" to another machine. I.e. being moved to another machine. Next was the "live migration" functionality. This is important because some guests run with a lot of state (specially RAM), and it can take a while to move all state from one machine to another. Live migration allows the guest to continue running while the state is transferred. Only while the last part of the state is transferred has the guest to be stopped. Typically the time that the guest is unresponsive during live migration is the low hundred of milliseconds (notice that this depends on a lot of things). Transports ========== The migration stream is normally just a byte stream that can be passed over any transport. - tcp migration: do the migration using tcp sockets - unix migration: do the migration using unix sockets - exec migration: do the migration using the stdin/stdout through a process. - fd migration: do the migration using a file descriptor that is passed to QEMU. QEMU doesn't care how this file descriptor is opened. In addition, support is included for migration using RDMA, which transports the page data using ``RDMA``, where the hardware takes care of transporting the pages, and the load on the CPU is much lower. While the internals of RDMA migration are a bit different, this isn't really visible outside the RAM migration code. All these migration protocols use the same infrastructure to save/restore state devices. This infrastructure is shared with the savevm/loadvm functionality. Debugging ========= The migration stream can be analyzed thanks to ``scripts/analyze-migration.py``. Example usage: .. code-block:: shell $ qemu-system-x86_64 -display none -monitor stdio (qemu) migrate "exec:cat > mig" (qemu) q $ ./scripts/analyze-migration.py -f mig { "ram (3)": { "section sizes": { "pc.ram": "0x0000000008000000", ... See also ``analyze-migration.py -h`` help for more options. Common infrastructure ===================== The files, sockets or fd's that carry the migration stream are abstracted by the ``QEMUFile`` type (see ``migration/qemu-file.h``). In most cases this is connected to a subtype of ``QIOChannel`` (see ``io/``). Saving the state of one device ============================== For most devices, the state is saved in a single call to the migration infrastructure; these are *non-iterative* devices. The data for these devices is sent at the end of precopy migration, when the CPUs are paused. There are also *iterative* devices, which contain a very large amount of data (e.g. RAM or large tables). See the iterative device section below. General advice for device developers ------------------------------------ - The migration state saved should reflect the device being modelled rather than the way your implementation works. That way if you change the implementation later the migration stream will stay compatible. That model may include internal state that's not directly visible in a register. - When saving a migration stream the device code may walk and check the state of the device. These checks might fail in various ways (e.g. discovering internal state is corrupt or that the guest has done something bad). Consider carefully before asserting/aborting at this point, since the normal response from users is that *migration broke their VM* since it had apparently been running fine until then. In these error cases, the device should log a message indicating the cause of error, and should consider putting the device into an error state, allowing the rest of the VM to continue execution. - The migration might happen at an inconvenient point, e.g. right in the middle of the guest reprogramming the device, during guest reboot or shutdown or while the device is waiting for external IO. It's strongly preferred that migrations do not fail in this situation, since in the cloud environment migrations might happen automatically to VMs that the administrator doesn't directly control. - If you do need to fail a migration, ensure that sufficient information is logged to identify what went wrong. - The destination should treat an incoming migration stream as hostile (which we do to varying degrees in the existing code). Check that offsets into buffers and the like can't cause overruns. Fail the incoming migration in the case of a corrupted stream like this. - Take care with internal device state or behaviour that might become migration version dependent. For example, the order of PCI capabilities is required to stay constant across migration. Another example would be that a special case handled by subsections (see below) might become much more common if a default behaviour is changed. - The state of the source should not be changed or destroyed by the outgoing migration. Migrations timing out or being failed by higher levels of management, or failures of the destination host are not unusual, and in that case the VM is restarted on the source. Note that the management layer can validly revert the migration even though the QEMU level of migration has succeeded as long as it does it before starting execution on the destination. - Buses and devices should be able to explicitly specify addresses when instantiated, and management tools should use those. For example, when hot adding USB devices it's important to specify the ports and addresses, since implicit ordering based on the command line order may be different on the destination. This can result in the device state being loaded into the wrong device. VMState ------- Most device data can be described using the ``VMSTATE`` macros (mostly defined in ``include/migration/vmstate.h``). An example (from hw/input/pckbd.c) .. code:: c static const VMStateDescription vmstate_kbd = { .name = "pckbd", .version_id = 3, .minimum_version_id = 3, .fields = (VMStateField[]) { VMSTATE_UINT8(write_cmd, KBDState), VMSTATE_UINT8(status, KBDState), VMSTATE_UINT8(mode, KBDState), VMSTATE_UINT8(pending, KBDState), VMSTATE_END_OF_LIST() } }; We are declaring the state with name "pckbd". The ``version_id`` is 3, and the fields are 4 uint8_t in a KBDState structure. We registered this with: .. code:: c vmstate_register(NULL, 0, &vmstate_kbd, s); For devices that are ``qdev`` based, we can register the device in the class init function: .. code:: c dc->vmsd = &vmstate_kbd_isa; The VMState macros take care of ensuring that the device data section is formatted portably (normally big endian) and make some compile time checks against the types of the fields in the structures. VMState macros can include other VMStateDescriptions to store substructures (see ``VMSTATE_STRUCT_``), arrays (``VMSTATE_ARRAY_``) and variable length arrays (``VMSTATE_VARRAY_``). Various other macros exist for special cases. Note that the format on the wire is still very raw; i.e. a VMSTATE_UINT32 ends up with a 4 byte bigendian representation on the wire; in the future it might be possible to use a more structured format. Legacy way ---------- This way is going to disappear as soon as all current users are ported to VMSTATE; although converting existing code can be tricky, and thus 'soon' is relative. Each device has to register two functions, one to save the state and another to load the state back. .. code:: c int register_savevm_live(const char *idstr, int instance_id, int version_id, SaveVMHandlers *ops, void *opaque); Two functions in the ``ops`` structure are the ``save_state`` and ``load_state`` functions. Notice that ``load_state`` receives a version_id parameter to know what state format is receiving. ``save_state`` doesn't have a version_id parameter because it always uses the latest version. Note that because the VMState macros still save the data in a raw format, in many cases it's possible to replace legacy code with a carefully constructed VMState description that matches the byte layout of the existing code. Changing migration data structures ---------------------------------- When we migrate a device, we save/load the state as a series of fields. Sometimes, due to bugs or new functionality, we need to change the state to store more/different information. Changing the migration state saved for a device can break migration compatibility unless care is taken to use the appropriate techniques. In general QEMU tries to maintain forward migration compatibility (i.e. migrating from QEMU n->n+1) and there are users who benefit from backward compatibility as well. Subsections ----------- The most common structure change is adding new data, e.g. when adding a newer form of device, or adding that state that you previously forgot to migrate. This is best solved using a subsection. A subsection is "like" a device vmstate, but with a particularity, it has a Boolean function that tells if that values are needed to be sent or not. If this functions returns false, the subsection is not sent. Subsections have a unique name, that is looked for on the receiving side. On the receiving side, if we found a subsection for a device that we don't understand, we just fail the migration. If we understand all the subsections, then we load the state with success. There's no check that a subsection is loaded, so a newer QEMU that knows about a subsection can (with care) load a stream from an older QEMU that didn't send the subsection. If the new data is only needed in a rare case, then the subsection can be made conditional on that case and the migration will still succeed to older QEMUs in most cases. This is OK for data that's critical, but in some use cases it's preferred that the migration should succeed even with the data missing. To support this the subsection can be connected to a device property and from there to a versioned machine type. The 'pre_load' and 'post_load' functions on subsections are only called if the subsection is loaded. One important note is that the outer post_load() function is called "after" loading all subsections, because a newer subsection could change the same value that it uses. A flag, and the combination of outer pre_load and post_load can be used to detect whether a subsection was loaded, and to fall back on default behaviour when the subsection isn't present. Example: .. code:: c static bool ide_drive_pio_state_needed(void *opaque) { IDEState *s = opaque; return ((s->status & DRQ_STAT) != 0) || (s->bus->error_status & BM_STATUS_PIO_RETRY); } const VMStateDescription vmstate_ide_drive_pio_state = { .name = "ide_drive/pio_state", .version_id = 1, .minimum_version_id = 1, .pre_save = ide_drive_pio_pre_save, .post_load = ide_drive_pio_post_load, .needed = ide_drive_pio_state_needed, .fields = (VMStateField[]) { VMSTATE_INT32(req_nb_sectors, IDEState), VMSTATE_VARRAY_INT32(io_buffer, IDEState, io_buffer_total_len, 1, vmstate_info_uint8, uint8_t), VMSTATE_INT32(cur_io_buffer_offset, IDEState), VMSTATE_INT32(cur_io_buffer_len, IDEState), VMSTATE_UINT8(end_transfer_fn_idx, IDEState), VMSTATE_INT32(elementary_transfer_size, IDEState), VMSTATE_INT32(packet_transfer_size, IDEState), VMSTATE_END_OF_LIST() } }; const VMStateDescription vmstate_ide_drive = { .name = "ide_drive", .version_id = 3, .minimum_version_id = 0, .post_load = ide_drive_post_load, .fields = (VMStateField[]) { .... several fields .... VMSTATE_END_OF_LIST() }, .subsections = (const VMStateDescription*[]) { &vmstate_ide_drive_pio_state, NULL } }; Here we have a subsection for the pio state. We only need to save/send this state when we are in the middle of a pio operation (that is what ``ide_drive_pio_state_needed()`` checks). If DRQ_STAT is not enabled, the values on that fields are garbage and don't need to be sent. Connecting subsections to properties ------------------------------------ Using a condition function that checks a 'property' to determine whether to send a subsection allows backward migration compatibility when new subsections are added, especially when combined with versioned machine types. For example: a) Add a new property using ``DEFINE_PROP_BOOL`` - e.g. support-foo and default it to true. b) Add an entry to the ``hw_compat_`` for the previous version that sets the property to false. c) Add a static bool support_foo function that tests the property. d) Add a subsection with a .needed set to the support_foo function e) (potentially) Add an outer pre_load that sets up a default value for 'foo' to be used if the subsection isn't loaded. Now that subsection will not be generated when using an older machine type and the migration stream will be accepted by older QEMU versions. Not sending existing elements ----------------------------- Sometimes members of the VMState are no longer needed: - removing them will break migration compatibility - making them version dependent and bumping the version will break backward migration compatibility. Adding a dummy field into the migration stream is normally the best way to preserve compatibility. If the field really does need to be removed then: a) Add a new property/compatibility/function in the same way for subsections above. b) replace the VMSTATE macro with the _TEST version of the macro, e.g.: ``VMSTATE_UINT32(foo, barstruct)`` becomes ``VMSTATE_UINT32_TEST(foo, barstruct, pre_version_baz)`` Sometime in the future when we no longer care about the ancient versions these can be killed off. Note that for backward compatibility it's important to fill in the structure with data that the destination will understand. Any difference in the predicates on the source and destination will end up with different fields being enabled and data being loaded into the wrong fields; for this reason conditional fields like this are very fragile. Versions -------- Version numbers are intended for major incompatible changes to the migration of a device, and using them breaks backward-migration compatibility; in general most changes can be made by adding Subsections (see above) or _TEST macros (see above) which won't break compatibility. Each version is associated with a series of fields saved. The ``save_state`` always saves the state as the newer version. But ``load_state`` sometimes is able to load state from an older version. You can see that there are two version fields: - ``version_id``: the maximum version_id supported by VMState for that device. - ``minimum_version_id``: the minimum version_id that VMState is able to understand for that device. VMState is able to read versions from minimum_version_id to version_id. There are *_V* forms of many ``VMSTATE_`` macros to load fields for version dependent fields, e.g. .. code:: c VMSTATE_UINT16_V(ip_id, Slirp, 2), only loads that field for versions 2 and newer. Saving state will always create a section with the 'version_id' value and thus can't be loaded by any older QEMU. Massaging functions ------------------- Sometimes, it is not enough to be able to save the state directly from one structure, we need to fill the correct values there. One example is when we are using kvm. Before saving the cpu state, we need to ask kvm to copy to QEMU the state that it is using. And the opposite when we are loading the state, we need a way to tell kvm to load the state for the cpu that we have just loaded from the QEMUFile. The functions to do that are inside a vmstate definition, and are called: - ``int (*pre_load)(void *opaque);`` This function is called before we load the state of one device. - ``int (*post_load)(void *opaque, int version_id);`` This function is called after we load the state of one device. - ``int (*pre_save)(void *opaque);`` This function is called before we save the state of one device. - ``int (*post_save)(void *opaque);`` This function is called after we save the state of one device (even upon failure, unless the call to pre_save returned an error). Example: You can look at hpet.c, that uses the first three functions to massage the state that is transferred. The ``VMSTATE_WITH_TMP`` macro may be useful when the migration data doesn't match the stored device data well; it allows an intermediate temporary structure to be populated with migration data and then transferred to the main structure. If you use memory API functions that update memory layout outside initialization (i.e., in response to a guest action), this is a strong indication that you need to call these functions in a ``post_load`` callback. Examples of such memory API functions are: - memory_region_add_subregion() - memory_region_del_subregion() - memory_region_set_readonly() - memory_region_set_nonvolatile() - memory_region_set_enabled() - memory_region_set_address() - memory_region_set_alias_offset() Iterative device migration -------------------------- Some devices, such as RAM, Block storage or certain platform devices, have large amounts of data that would mean that the CPUs would be paused for too long if they were sent in one section. For these devices an *iterative* approach is taken. The iterative devices generally don't use VMState macros (although it may be possible in some cases) and instead use qemu_put_*/qemu_get_* macros to read/write data to the stream. Specialist versions exist for high bandwidth IO. An iterative device must provide: - A ``save_setup`` function that initialises the data structures and transmits a first section containing information on the device. In the case of RAM this transmits a list of RAMBlocks and sizes. - A ``load_setup`` function that initialises the data structures on the destination. - A ``save_live_pending`` function that is called repeatedly and must indicate how much more data the iterative data must save. The core migration code will use this to determine when to pause the CPUs and complete the migration. - A ``save_live_iterate`` function (called after ``save_live_pending`` when there is significant data still to be sent). It should send a chunk of data until the point that stream bandwidth limits tell it to stop. Each call generates one section. - A ``save_live_complete_precopy`` function that must transmit the last section for the device containing any remaining data. - A ``load_state`` function used to load sections generated by any of the save functions that generate sections. - ``cleanup`` functions for both save and load that are called at the end of migration. Note that the contents of the sections for iterative migration tend to be open-coded by the devices; care should be taken in parsing the results and structuring the stream to make them easy to validate. Device ordering --------------- There are cases in which the ordering of device loading matters; for example in some systems where a device may assert an interrupt during loading, if the interrupt controller is loaded later then it might lose the state. Some ordering is implicitly provided by the order in which the machine definition creates devices, however this is somewhat fragile. The ``MigrationPriority`` enum provides a means of explicitly enforcing ordering. Numerically higher priorities are loaded earlier. The priority is set by setting the ``priority`` field of the top level ``VMStateDescription`` for the device. Stream structure ================ The stream tries to be word and endian agnostic, allowing migration between hosts of different characteristics running the same VM. - Header - Magic - Version - VM configuration section - Machine type - Target page bits - List of sections Each section contains a device, or one iteration of a device save. - section type - section id - ID string (First section of each device) - instance id (First section of each device) - version id (First section of each device) - - Footer mark - EOF mark - VM Description structure Consisting of a JSON description of the contents for analysis only The ``device data`` in each section consists of the data produced by the code described above. For non-iterative devices they have a single section; iterative devices have an initial and last section and a set of parts in between. Note that there is very little checking by the common code of the integrity of the ``device data`` contents, that's up to the devices themselves. The ``footer mark`` provides a little bit of protection for the case where the receiving side reads more or less data than expected. The ``ID string`` is normally unique, having been formed from a bus name and device address, PCI devices and storage devices hung off PCI controllers fit this pattern well. Some devices are fixed single instances (e.g. "pc-ram"). Others (especially either older devices or system devices which for some reason don't have a bus concept) make use of the ``instance id`` for otherwise identically named devices. Return path ----------- Only a unidirectional stream is required for normal migration, however a ``return path`` can be created when bidirectional communication is desired. This is primarily used by postcopy, but is also used to return a success flag to the source at the end of migration. ``qemu_file_get_return_path(QEMUFile* fwdpath)`` gives the QEMUFile* for the return path. Source side Forward path - written by migration thread Return path - opened by main thread, read by return-path thread Destination side Forward path - read by main thread Return path - opened by main thread, written by main thread AND postcopy thread (protected by rp_mutex) Postcopy ======== 'Postcopy' migration is a way to deal with migrations that refuse to converge (or take too long to converge) its plus side is that there is an upper bound on the amount of migration traffic and time it takes, the down side is that during the postcopy phase, a failure of *either* side or the network connection causes the guest to be lost. In postcopy the destination CPUs are started before all the memory has been transferred, and accesses to pages that are yet to be transferred cause a fault that's translated by QEMU into a request to the source QEMU. Postcopy can be combined with precopy (i.e. normal migration) so that if precopy doesn't finish in a given time the switch is made to postcopy. Enabling postcopy ----------------- To enable postcopy, issue this command on the monitor (both source and destination) prior to the start of migration: ``migrate_set_capability postcopy-ram on`` The normal commands are then used to start a migration, which is still started in precopy mode. Issuing: ``migrate_start_postcopy`` will now cause the transition from precopy to postcopy. It can be issued immediately after migration is started or any time later on. Issuing it after the end of a migration is harmless. Blocktime is a postcopy live migration metric, intended to show how long the vCPU was in state of interruptible sleep due to pagefault. That metric is calculated both for all vCPUs as overlapped value, and separately for each vCPU. These values are calculated on destination side. To enable postcopy blocktime calculation, enter following command on destination monitor: ``migrate_set_capability postcopy-blocktime on`` Postcopy blocktime can be retrieved by query-migrate qmp command. postcopy-blocktime value of qmp command will show overlapped blocking time for all vCPU, postcopy-vcpu-blocktime will show list of blocking time per vCPU. .. note:: During the postcopy phase, the bandwidth limits set using ``migrate_set_parameter`` is ignored (to avoid delaying requested pages that the destination is waiting for). Postcopy device transfer ------------------------ Loading of device data may cause the device emulation to access guest RAM that may trigger faults that have to be resolved by the source, as such the migration stream has to be able to respond with page data *during* the device load, and hence the device data has to be read from the stream completely before the device load begins to free the stream up. This is achieved by 'packaging' the device data into a blob that's read in one go. Source behaviour ---------------- Until postcopy is entered the migration stream is identical to normal precopy, except for the addition of a 'postcopy advise' command at the beginning, to tell the destination that postcopy might happen. When postcopy starts the source sends the page discard data and then forms the 'package' containing: - Command: 'postcopy listen' - The device state A series of sections, identical to the precopy streams device state stream containing everything except postcopiable devices (i.e. RAM) - Command: 'postcopy run' The 'package' is sent as the data part of a Command: ``CMD_PACKAGED``, and the contents are formatted in the same way as the main migration stream. During postcopy the source scans the list of dirty pages and sends them to the destination without being requested (in much the same way as precopy), however when a page request is received from the destination, the dirty page scanning restarts from the requested location. This causes requested pages to be sent quickly, and also causes pages directly after the requested page to be sent quickly in the hope that those pages are likely to be used by the destination soon. Destination behaviour --------------------- Initially the destination looks the same as precopy, with a single thread reading the migration stream; the 'postcopy advise' and 'discard' commands are processed to change the way RAM is managed, but don't affect the stream processing. :: ------------------------------------------------------------------------------ 1 2 3 4 5 6 7 main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN ) thread | | | (page request) | \___ v \ listen thread: --- page -- page -- page -- page -- page -- a b c ------------------------------------------------------------------------------ - On receipt of ``CMD_PACKAGED`` (1) All the data associated with the package - the ( ... ) section in the diagram - is read into memory, and the main thread recurses into qemu_loadvm_state_main to process the contents of the package (2) which contains commands (3,6) and devices (4...) - On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package) a new thread (a) is started that takes over servicing the migration stream, while the main thread carries on loading the package. It loads normal background page data (b) but if during a device load a fault happens (5) the returned page (c) is loaded by the listen thread allowing the main threads device load to carry on. - The last thing in the ``CMD_PACKAGED`` is a 'RUN' command (6) letting the destination CPUs start running. At the end of the ``CMD_PACKAGED`` (7) the main thread returns to normal running behaviour and is no longer used by migration, while the listen thread carries on servicing page data until the end of migration. Postcopy states --------------- Postcopy moves through a series of states (see postcopy_state) from ADVISE->DISCARD->LISTEN->RUNNING->END - Advise Set at the start of migration if postcopy is enabled, even if it hasn't had the start command; here the destination checks that its OS has the support needed for postcopy, and performs setup to ensure the RAM mappings are suitable for later postcopy. The destination will fail early in migration at this point if the required OS support is not present. (Triggered by reception of POSTCOPY_ADVISE command) - Discard Entered on receipt of the first 'discard' command; prior to the first Discard being performed, hugepages are switched off (using madvise) to ensure that no new huge pages are created during the postcopy phase, and to cause any huge pages that have discards on them to be broken. - Listen The first command in the package, POSTCOPY_LISTEN, switches the destination state to Listen, and starts a new thread (the 'listen thread') which takes over the job of receiving pages off the migration stream, while the main thread carries on processing the blob. With this thread able to process page reception, the destination now 'sensitises' the RAM to detect any access to missing pages (on Linux using the 'userfault' system). - Running POSTCOPY_RUN causes the destination to synchronise all state and start the CPUs and IO devices running. The main thread now finishes processing the migration package and now carries on as it would for normal precopy migration (although it can't do the cleanup it would do as it finishes a normal migration). - End The listen thread can now quit, and perform the cleanup of migration state, the migration is now complete. Source side page maps --------------------- The source side keeps two bitmaps during postcopy; 'the migration bitmap' and 'unsent map'. The 'migration bitmap' is basically the same as in the precopy case, and holds a bit to indicate that page is 'dirty' - i.e. needs sending. During the precopy phase this is updated as the CPU dirties pages, however during postcopy the CPUs are stopped and nothing should dirty anything any more. The 'unsent map' is used for the transition to postcopy. It is a bitmap that has a bit cleared whenever a page is sent to the destination, however during the transition to postcopy mode it is combined with the migration bitmap to form a set of pages that: a) Have been sent but then redirtied (which must be discarded) b) Have not yet been sent - which also must be discarded to cause any transparent huge pages built during precopy to be broken. Note that the contents of the unsentmap are sacrificed during the calculation of the discard set and thus aren't valid once in postcopy. The dirtymap is still valid and is used to ensure that no page is sent more than once. Any request for a page that has already been sent is ignored. Duplicate requests such as this can happen as a page is sent at about the same time the destination accesses it. Postcopy with hugepages ----------------------- Postcopy now works with hugetlbfs backed memory: a) The linux kernel on the destination must support userfault on hugepages. b) The huge-page configuration on the source and destination VMs must be identical; i.e. RAMBlocks on both sides must use the same page size. c) Note that ``-mem-path /dev/hugepages`` will fall back to allocating normal RAM if it doesn't have enough hugepages, triggering (b) to fail. Using ``-mem-prealloc`` enforces the allocation using hugepages. d) Care should be taken with the size of hugepage used; postcopy with 2MB hugepages works well, however 1GB hugepages are likely to be problematic since it takes ~1 second to transfer a 1GB hugepage across a 10Gbps link, and until the full page is transferred the destination thread is blocked. Postcopy with shared memory --------------------------- Postcopy migration with shared memory needs explicit support from the other processes that share memory and from QEMU. There are restrictions on the type of memory that userfault can support shared. The Linux kernel userfault support works on ``/dev/shm`` memory and on ``hugetlbfs`` (although the kernel doesn't provide an equivalent to ``madvise(MADV_DONTNEED)`` for hugetlbfs which may be a problem in some configurations). The vhost-user code in QEMU supports clients that have Postcopy support, and the ``vhost-user-bridge`` (in ``tests/``) and the DPDK package have changes to support postcopy. The client needs to open a userfaultfd and register the areas of memory that it maps with userfault. The client must then pass the userfaultfd back to QEMU together with a mapping table that allows fault addresses in the clients address space to be converted back to RAMBlock/offsets. The client's userfaultfd is added to the postcopy fault-thread and page requests are made on behalf of the client by QEMU. QEMU performs 'wake' operations on the client's userfaultfd to allow it to continue after a page has arrived. .. note:: There are two future improvements that would be nice: a) Some way to make QEMU ignorant of the addresses in the clients address space b) Avoiding the need for QEMU to perform ufd-wake calls after the pages have arrived Retro-fitting postcopy to existing clients is possible: a) A mechanism is needed for the registration with userfault as above, and the registration needs to be coordinated with the phases of postcopy. In vhost-user extra messages are added to the existing control channel. b) Any thread that can block due to guest memory accesses must be identified and the implication understood; for example if the guest memory access is made while holding a lock then all other threads waiting for that lock will also be blocked. Firmware ======== Migration migrates the copies of RAM and ROM, and thus when running on the destination it includes the firmware from the source. Even after resetting a VM, the old firmware is used. Only once QEMU has been restarted is the new firmware in use. - Changes in firmware size can cause changes in the required RAMBlock size to hold the firmware and thus migration can fail. In practice it's best to pad firmware images to convenient powers of 2 with plenty of space for growth. - Care should be taken with device emulation code so that newer emulation code can work with older firmware to allow forward migration. - Care should be taken with newer firmware so that backward migration to older systems with older device emulation code will work. In some cases it may be best to tie specific firmware versions to specific versioned machine types to cut down on the combinations that will need support. This is also useful when newer versions of firmware outgrow the padding. --- ### Devel Modules (`docs/devel/modules.rst`) ============ QEMU modules ============ .. kernel-doc:: include/qemu/module.h --- ### Devel Multi Process (`docs/devel/multi-process.rst`) Multi-process QEMU =================== .. note:: This is the design document for multi-process QEMU. It does not necessarily reflect the status of the current implementation, which may lack features or be considerably different from what is described in this document. This document is still useful as a description of the goals and general direction of this feature. Please refer to the following wiki for latest details: https://wiki.qemu.org/Features/MultiProcessQEMU QEMU is often used as the hypervisor for virtual machines running in the Oracle cloud. Since one of the advantages of cloud computing is the ability to run many VMs from different tenants in the same cloud infrastructure, a guest that compromised its hypervisor could potentially use the hypervisor's access privileges to access data it is not authorized for. QEMU can be susceptible to security attacks because it is a large, monolithic program that provides many features to the VMs it services. Many of these features can be configured out of QEMU, but even a reduced configuration QEMU has a large amount of code a guest can potentially attack. Separating QEMU reduces the attack surface by aiding to limit each component in the system to only access the resources that it needs to perform its job. QEMU services ------------- QEMU can be broadly described as providing three main services. One is a VM control point, where VMs can be created, migrated, re-configured, and destroyed. A second is to emulate the CPU instructions within the VM, often accelerated by HW virtualization features such as Intel's VT extensions. Finally, it provides IO services to the VM by emulating HW IO devices, such as disk and network devices. A multi-process QEMU ~~~~~~~~~~~~~~~~~~~~ A multi-process QEMU involves separating QEMU services into separate host processes. Each of these processes can be given only the privileges it needs to provide its service, e.g., a disk service could be given access only to the disk images it provides, and not be allowed to access other files, or any network devices. An attacker who compromised this service would not be able to use this exploit to access files or devices beyond what the disk service was given access to. A QEMU control process would remain, but in multi-process mode, will have no direct interfaces to the VM. During VM execution, it would still provide the user interface to hot-plug devices or live migrate the VM. A first step in creating a multi-process QEMU is to separate IO services from the main QEMU program, which would continue to provide CPU emulation. i.e., the control process would also be the CPU emulation process. In a later phase, CPU emulation could be separated from the control process. Separating IO services ---------------------- Separating IO services into individual host processes is a good place to begin for a couple of reasons. One is the sheer number of IO devices QEMU can emulate provides a large surface of interfaces which could potentially be exploited, and, indeed, have been a source of exploits in the past. Another is the modular nature of QEMU device emulation code provides interface points where the QEMU functions that perform device emulation can be separated from the QEMU functions that manage the emulation of guest CPU instructions. The devices emulated in the separate process are referred to as remote devices. QEMU device emulation ~~~~~~~~~~~~~~~~~~~~~ QEMU uses an object oriented SW architecture for device emulation code. Configured objects are all compiled into the QEMU binary, then objects are instantiated by name when used by the guest VM. For example, the code to emulate a device named "foo" is always present in QEMU, but its instantiation code is only run when the device is included in the target VM. (e.g., via the QEMU command line as *-device foo*) The object model is hierarchical, so device emulation code names its parent object (such as "pci-device" for a PCI device) and QEMU will instantiate a parent object before calling the device's instantiation code. Current separation models ~~~~~~~~~~~~~~~~~~~~~~~~~ In order to separate the device emulation code from the CPU emulation code, the device object code must run in a different process. There are a couple of existing QEMU features that can run emulation code separately from the main QEMU process. These are examined below. vhost user model ^^^^^^^^^^^^^^^^ Virtio guest device drivers can be connected to vhost user applications in order to perform their IO operations. This model uses special virtio device drivers in the guest and vhost user device objects in QEMU, but once the QEMU vhost user code has configured the vhost user application, mission-mode IO is performed by the application. The vhost user application is a daemon process that can be contacted via a known UNIX domain socket. vhost socket '''''''''''' As mentioned above, one of the tasks of the vhost device object within QEMU is to contact the vhost application and send it configuration information about this device instance. As part of the configuration process, the application can also be sent other file descriptors over the socket, which then can be used by the vhost user application in various ways, some of which are described below. vhost MMIO store acceleration ''''''''''''''''''''''''''''' VMs are often run using HW virtualization features via the KVM kernel driver. This driver allows QEMU to accelerate the emulation of guest CPU instructions by running the guest in a virtual HW mode. When the guest executes instructions that cannot be executed by virtual HW mode, execution returns to the KVM driver so it can inform QEMU to emulate the instructions in SW. One of the events that can cause a return to QEMU is when a guest device driver accesses an IO location. QEMU then dispatches the memory operation to the corresponding QEMU device object. In the case of a vhost user device, the memory operation would need to be sent over a socket to the vhost application. This path is accelerated by the QEMU virtio code by setting up an eventfd file descriptor that the vhost application can directly receive MMIO store notifications from the KVM driver, instead of needing them to be sent to the QEMU process first. vhost interrupt acceleration '''''''''''''''''''''''''''' Another optimization used by the vhost application is the ability to directly inject interrupts into the VM via the KVM driver, again, bypassing the need to send the interrupt back to the QEMU process first. The QEMU virtio setup code configures the KVM driver with an eventfd that triggers the device interrupt in the guest when the eventfd is written. This irqfd file descriptor is then passed to the vhost user application program. vhost access to guest memory '''''''''''''''''''''''''''' The vhost application is also allowed to directly access guest memory, instead of needing to send the data as messages to QEMU. This is also done with file descriptors sent to the vhost user application by QEMU. These descriptors can be passed to ``mmap()`` by the vhost application to map the guest address space into the vhost application. IOMMUs introduce another level of complexity, since the address given to the guest virtio device to DMA to or from is not a guest physical address. This case is handled by having vhost code within QEMU register as a listener for IOMMU mapping changes. The vhost application maintains a cache of IOMMMU translations: sending translation requests back to QEMU on cache misses, and in turn receiving flush requests from QEMU when mappings are purged. applicability to device separation '''''''''''''''''''''''''''''''''' Much of the vhost model can be re-used by separated device emulation. In particular, the ideas of using a socket between QEMU and the device emulation application, using a file descriptor to inject interrupts into the VM via KVM, and allowing the application to ``mmap()`` the guest should be re used. There are, however, some notable differences between how a vhost application works and the needs of separated device emulation. The most basic is that vhost uses custom virtio device drivers which always trigger IO with MMIO stores. A separated device emulation model must work with existing IO device models and guest device drivers. MMIO loads break vhost store acceleration since they are synchronous - guest progress cannot continue until the load has been emulated. By contrast, stores are asynchronous, the guest can continue after the store event has been sent to the vhost application. Another difference is that in the vhost user model, a single daemon can support multiple QEMU instances. This is contrary to the security regime desired, in which the emulation application should only be allowed to access the files or devices the VM it's running on behalf of can access. #### qemu-io model ``qemu-io`` is a test harness used to test changes to the QEMU block backend object code (e.g., the code that implements disk images for disk driver emulation). ``qemu-io`` is not a device emulation application per se, but it does compile the QEMU block objects into a separate binary from the main QEMU one. This could be useful for disk device emulation, since its emulation applications will need to include the QEMU block objects. New separation model based on proxy objects ------------------------------------------- A different model based on proxy objects in the QEMU program communicating with remote emulation programs could provide separation while minimizing the changes needed to the device emulation code. The rest of this section is a discussion of how a proxy object model would work. Remote emulation processes ~~~~~~~~~~~~~~~~~~~~~~~~~~ The remote emulation process will run the QEMU object hierarchy without modification. The device emulation objects will be also be based on the QEMU code, because for anything but the simplest device, it would not be a tractable to re-implement both the object model and the many device backends that QEMU has. The processes will communicate with the QEMU process over UNIX domain sockets. The processes can be executed either as standalone processes, or be executed by QEMU. In both cases, the host backends the emulation processes will provide are specified on its command line, as they would be for QEMU. For example: :: disk-proc -blockdev driver=file,node-name=file0,filename=disk-file0 \ -blockdev driver=qcow2,node-name=drive0,file=file0 would indicate process *disk-proc* uses a qcow2 emulated disk named *file0* as its backend. Emulation processes may emulate more than one guest controller. A common configuration might be to put all controllers of the same device class (e.g., disk, network, etc.) in a single process, so that all backends of the same type can be managed by a single QMP monitor. communication with QEMU ^^^^^^^^^^^^^^^^^^^^^^^ The first argument to the remote emulation process will be a Unix domain socket that connects with the Proxy object. This is a required argument. :: disk-proc remote process QMP monitor ^^^^^^^^^^^^^^^^^^^^^^^^^^ Remote emulation processes can be monitored via QMP, similar to QEMU itself. The QMP monitor socket is specified the same as for a QEMU process: :: disk-proc -qmp unix:/tmp/disk-mon,server can be monitored over the UNIX socket path */tmp/disk-mon*. QEMU command line ~~~~~~~~~~~~~~~~~ Each remote device emulated in a remote process on the host is represented as a *-device* of type *pci-proxy-dev*. A socket sub-option to this option specifies the Unix socket that connects to the remote process. An *id* sub-option is required, and it should be the same id as used in the remote process. :: qemu-system-x86_64 ... -device pci-proxy-dev,id=lsi0,socket=3 can be used to add a device emulated in a remote process QEMU management of remote processes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ QEMU is not aware of the type of type of the remote PCI device. It is a pass through device as far as QEMU is concerned. communication with emulation process ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ primary channel ''''''''''''''' The primary channel (referred to as com in the code) is used to bootstrap the remote process. It is also used to pass on device-agnostic commands like reset. per-device channels ''''''''''''''''''' Each remote device communicates with QEMU using a dedicated communication channel. The proxy object sets up this channel using the primary channel during its initialization. QEMU device proxy objects ~~~~~~~~~~~~~~~~~~~~~~~~~ QEMU has an object model based on sub-classes inherited from the "object" super-class. The sub-classes that are of interest here are the "device" and "bus" sub-classes whose child sub-classes make up the device tree of a QEMU emulated system. The proxy object model will use device proxy objects to replace the device emulation code within the QEMU process. These objects will live in the same place in the object and bus hierarchies as the objects they replace. i.e., the proxy object for an LSI SCSI controller will be a sub-class of the "pci-device" class, and will have the same PCI bus parent and the same SCSI bus child objects as the LSI controller object it replaces. It is worth noting that the same proxy object is used to mediate with all types of remote PCI devices. object initialization ^^^^^^^^^^^^^^^^^^^^^ The Proxy device objects are initialized in the exact same manner in which any other QEMU device would be initialized. In addition, the Proxy objects perform the following two tasks: - Parses the "socket" sub option and connects to the remote process using this channel - Uses the "id" sub-option to connect to the emulated device on the separate process class\_init ''''''''''' The ``class_init()`` method of a proxy object will, in general behave similarly to the object it replaces, including setting any static properties and methods needed by the proxy. instance\_init / realize '''''''''''''''''''''''' The ``instance_init()`` and ``realize()`` functions would only need to perform tasks related to being a proxy, such are registering its own MMIO handlers, or creating a child bus that other proxy devices can be attached to later. Other tasks will be device-specific. For example, PCI device objects will initialize the PCI config space in order to make a valid PCI device tree within the QEMU process. address space registration ^^^^^^^^^^^^^^^^^^^^^^^^^^ Most devices are driven by guest device driver accesses to IO addresses or ports. The QEMU device emulation code uses QEMU's memory region function calls (such as ``memory_region_init_io()``) to add callback functions that QEMU will invoke when the guest accesses the device's areas of the IO address space. When a guest driver does access the device, the VM will exit HW virtualization mode and return to QEMU, which will then lookup and execute the corresponding callback function. A proxy object would need to mirror the memory region calls the actual device emulator would perform in its initialization code, but with its own callbacks. When invoked by QEMU as a result of a guest IO operation, they will forward the operation to the device emulation process. PCI config space ^^^^^^^^^^^^^^^^ PCI devices also have a configuration space that can be accessed by the guest driver. Guest accesses to this space is not handled by the device emulation object, but by its PCI parent object. Much of this space is read-only, but certain registers (especially BAR and MSI-related ones) need to be propagated to the emulation process. PCI parent proxy '''''''''''''''' One way to propagate guest PCI config accesses is to create a "pci-device-proxy" class that can serve as the parent of a PCI device proxy object. This class's parent would be "pci-device" and it would override the PCI parent's ``config_read()`` and ``config_write()`` methods with ones that forward these operations to the emulation program. interrupt receipt ^^^^^^^^^^^^^^^^^ A proxy for a device that generates interrupts will need to create a socket to receive interrupt indications from the emulation process. An incoming interrupt indication would then be sent up to its bus parent to be injected into the guest. For example, a PCI device object may use ``pci_set_irq()``. live migration ^^^^^^^^^^^^^^ The proxy will register to save and restore any *vmstate* it needs over a live migration event. The device proxy does not need to manage the remote device's *vmstate*; that will be handled by the remote process proxy (see below). QEMU remote device operation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Generic device operations, such as DMA, will be performed by the remote process proxy by sending messages to the remote process. DMA operations ^^^^^^^^^^^^^^ DMA operations would be handled much like vhost applications do. One of the initial messages sent to the emulation process is a guest memory table. Each entry in this table consists of a file descriptor and size that the emulation process can ``mmap()`` to directly access guest memory, similar to ``vhost_user_set_mem_table()``. Note guest memory must be backed by file descriptors, such as when QEMU is given the *-mem-path* command line option. IOMMU operations ^^^^^^^^^^^^^^^^ When the emulated system includes an IOMMU, the remote process proxy in QEMU will need to create a socket for IOMMU requests from the emulation process. It will handle those requests with an ``address_space_get_iotlb_entry()`` call. In order to handle IOMMU unmaps, the remote process proxy will also register as a listener on the device's DMA address space. When an IOMMU memory region is created within the DMA address space, an IOMMU notifier for unmaps will be added to the memory region that will forward unmaps to the emulation process over the IOMMU socket. device hot-plug via QMP ^^^^^^^^^^^^^^^^^^^^^^^ An QMP "device\_add" command can add a device emulated by a remote process. It will also have "rid" option to the command, just as the *-device* command line option does. The remote process may either be one started at QEMU startup, or be one added by the "add-process" QMP command described above. In either case, the remote process proxy will forward the new device's JSON description to the corresponding emulation process. live migration ^^^^^^^^^^^^^^ The remote process proxy will also register for live migration notifications with ``vmstate_register()``. When called to save state, the proxy will send the remote process a secondary socket file descriptor to save the remote process's device *vmstate* over. The incoming byte stream length and data will be saved as the proxy's *vmstate*. When the proxy is resumed on its new host, this *vmstate* will be extracted, and a secondary socket file descriptor will be sent to the new remote process through which it receives the *vmstate* in order to restore the devices there. device emulation in remote process ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The parts of QEMU that the emulation program will need include the object model; the memory emulation objects; the device emulation objects of the targeted device, and any dependent devices; and, the device's backends. It will also need code to setup the machine environment, handle requests from the QEMU process, and route machine-level requests (such as interrupts or IOMMU mappings) back to the QEMU process. initialization ^^^^^^^^^^^^^^ The process initialization sequence will follow the same sequence followed by QEMU. It will first initialize the backend objects, then device emulation objects. The JSON descriptions sent by the QEMU process will drive which objects need to be created. - address spaces Before the device objects are created, the initial address spaces and memory regions must be configured with ``memory_map_init()``. This creates a RAM memory region object (*system\_memory*) and an IO memory region object (*system\_io*). - RAM RAM memory region creation will follow how ``pc_memory_init()`` creates them, but must use ``memory_region_init_ram_from_fd()`` instead of ``memory_region_allocate_system_memory()``. The file descriptors needed will be supplied by the guest memory table from above. Those RAM regions would then be added to the *system\_memory* memory region with ``memory_region_add_subregion()``. - PCI IO initialization will be driven by the JSON descriptions sent from the QEMU process. For a PCI device, a PCI bus will need to be created with ``pci_root_bus_new()``, and a PCI memory region will need to be created and added to the *system\_memory* memory region with ``memory_region_add_subregion_overlap()``. The overlap version is required for architectures where PCI memory overlaps with RAM memory. MMIO handling ^^^^^^^^^^^^^ The device emulation objects will use ``memory_region_init_io()`` to install their MMIO handlers, and ``pci_register_bar()`` to associate those handlers with a PCI BAR, as they do within QEMU currently. In order to use ``address_space_rw()`` in the emulation process to handle MMIO requests from QEMU, the PCI physical addresses must be the same in the QEMU process and the device emulation process. In order to accomplish that, guest BAR programming must also be forwarded from QEMU to the emulation process. interrupt injection ^^^^^^^^^^^^^^^^^^^ When device emulation wants to inject an interrupt into the VM, the request climbs the device's bus object hierarchy until the point where a bus object knows how to signal the interrupt to the guest. The details depend on the type of interrupt being raised. - PCI pin interrupts On x86 systems, there is an emulated IOAPIC object attached to the root PCI bus object, and the root PCI object forwards interrupt requests to it. The IOAPIC object, in turn, calls the KVM driver to inject the corresponding interrupt into the VM. The simplest way to handle this in an emulation process would be to setup the root PCI bus driver (via ``pci_bus_irqs()``) to send a interrupt request back to the QEMU process, and have the device proxy object reflect it up the PCI tree there. - PCI MSI/X interrupts PCI MSI/X interrupts are implemented in HW as DMA writes to a CPU-specific PCI address. In QEMU on x86, a KVM APIC object receives these DMA writes, then calls into the KVM driver to inject the interrupt into the VM. A simple emulation process implementation would be to send the MSI DMA address from QEMU as a message at initialization, then install an address space handler at that address which forwards the MSI message back to QEMU. DMA operations ^^^^^^^^^^^^^^ When a emulation object wants to DMA into or out of guest memory, it first must use dma\_memory\_map() to convert the DMA address to a local virtual address. The emulation process memory region objects setup above will be used to translate the DMA address to a local virtual address the device emulation code can access. IOMMU ^^^^^ When an IOMMU is in use in QEMU, DMA translation uses IOMMU memory regions to translate the DMA address to a guest physical address before that physical address can be translated to a local virtual address. The emulation process will need similar functionality. - IOTLB cache The emulation process will maintain a cache of recent IOMMU translations (the IOTLB). When the translate() callback of an IOMMU memory region is invoked, the IOTLB cache will be searched for an entry that will map the DMA address to a guest PA. On a cache miss, a message will be sent back to QEMU requesting the corresponding translation entry, which be both be used to return a guest address and be added to the cache. - IOTLB purge The IOMMU emulation will also need to act on unmap requests from QEMU. These happen when the guest IOMMU driver purges an entry from the guest's translation table. live migration ^^^^^^^^^^^^^^ When a remote process receives a live migration indication from QEMU, it will set up a channel using the received file descriptor with ``qio_channel_socket_new_fd()``. This channel will be used to create a *QEMUfile* that can be passed to ``qemu_save_device_state()`` to send the process's device state back to QEMU. This method will be reversed on restore - the channel will be passed to ``qemu_loadvm_state()`` to restore the device state. Accelerating device emulation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The messages that are required to be sent between QEMU and the emulation process can add considerable latency to IO operations. The optimizations described below attempt to ameliorate this effect by allowing the emulation process to communicate directly with the kernel KVM driver. The KVM file descriptors created would be passed to the emulation process via initialization messages, much like the guest memory table is done. #### MMIO acceleration Vhost user applications can receive guest virtio driver stores directly from KVM. The issue with the eventfd mechanism used by vhost user is that it does not pass any data with the event indication, so it cannot handle guest loads or guest stores that carry store data. This concept could, however, be expanded to cover more cases. The expanded idea would require a new type of KVM device: *KVM\_DEV\_TYPE\_USER*. This device has two file descriptors: a master descriptor that QEMU can use for configuration, and a slave descriptor that the emulation process can use to receive MMIO notifications. QEMU would create both descriptors using the KVM driver, and pass the slave descriptor to the emulation process via an initialization message. data structures ^^^^^^^^^^^^^^^ - guest physical range The guest physical range structure describes the address range that a device will respond to. It includes the base and length of the range, as well as which bus the range resides on (e.g., on an x86machine, it can specify whether the range refers to memory or IO addresses). A device can have multiple physical address ranges it responds to (e.g., a PCI device can have multiple BARs), so the structure will also include an enumerated identifier to specify which of the device's ranges is being referred to. +--------+----------------------------+ | Name | Description | +========+============================+ | addr | range base address | +--------+----------------------------+ | len | range length | +--------+----------------------------+ | bus | addr type (memory or IO) | +--------+----------------------------+ | id | range ID (e.g., PCI BAR) | +--------+----------------------------+ - MMIO request structure This structure describes an MMIO operation. It includes which guest physical range the MMIO was within, the offset within that range, the MMIO type (e.g., load or store), and its length and data. It also includes a sequence number that can be used to reply to the MMIO, and the CPU that issued the MMIO. +----------+------------------------+ | Name | Description | +==========+========================+ | rid | range MMIO is within | +----------+------------------------+ | offset | offset within *rid* | +----------+------------------------+ | type | e.g., load or store | +----------+------------------------+ | len | MMIO length | +----------+------------------------+ | data | store data | +----------+------------------------+ | seq | sequence ID | +----------+------------------------+ - MMIO request queues MMIO request queues are FIFO arrays of MMIO request structures. There are two queues: pending queue is for MMIOs that haven't been read by the emulation program, and the sent queue is for MMIOs that haven't been acknowledged. The main use of the second queue is to validate MMIO replies from the emulation program. - scoreboard Each CPU in the VM is emulated in QEMU by a separate thread, so multiple MMIOs may be waiting to be consumed by an emulation program and multiple threads may be waiting for MMIO replies. The scoreboard would contain a wait queue and sequence number for the per-CPU threads, allowing them to be individually woken when the MMIO reply is received from the emulation program. It also tracks the number of posted MMIO stores to the device that haven't been replied to, in order to satisfy the PCI constraint that a load to a device will not complete until all previous stores to that device have been completed. - device shadow memory Some MMIO loads do not have device side-effects. These MMIOs can be completed without sending a MMIO request to the emulation program if the emulation program shares a shadow image of the device's memory image with the KVM driver. The emulation program will ask the KVM driver to allocate memory for the shadow image, and will then use ``mmap()`` to directly access it. The emulation program can control KVM access to the shadow image by sending KVM an access map telling it which areas of the image have no side-effects (and can be completed immediately), and which require a MMIO request to the emulation program. The access map can also inform the KVM drive which size accesses are allowed to the image. master descriptor ^^^^^^^^^^^^^^^^^ The master descriptor is used by QEMU to configure the new KVM device. The descriptor would be returned by the KVM driver when QEMU issues a *KVM\_CREATE\_DEVICE* ``ioctl()`` with a *KVM\_DEV\_TYPE\_USER* type. KVM\_DEV\_TYPE\_USER device ops The *KVM\_DEV\_TYPE\_USER* operations vector will be registered by a ``kvm_register_device_ops()`` call when the KVM system in initialized by ``kvm_init()``. These device ops are called by the KVM driver when QEMU executes certain ``ioctl()`` operations on its KVM file descriptor. They include: - create This routine is called when QEMU issues a *KVM\_CREATE\_DEVICE* ``ioctl()`` on its per-VM file descriptor. It will allocate and initialize a KVM user device specific data structure, and assign the *kvm\_device* private field to it. - ioctl This routine is invoked when QEMU issues an ``ioctl()`` on the master descriptor. The ``ioctl()`` commands supported are defined by the KVM device type. *KVM\_DEV\_TYPE\_USER* ones will need several commands: *KVM\_DEV\_USER\_SLAVE\_FD* creates the slave file descriptor that will be passed to the device emulation program. Only one slave can be created by each master descriptor. The file operations performed by this descriptor are described below. The *KVM\_DEV\_USER\_PA\_RANGE* command configures a guest physical address range that the slave descriptor will receive MMIO notifications for. The range is specified by a guest physical range structure argument. For buses that assign addresses to devices dynamically, this command can be executed while the guest is running, such as the case when a guest changes a device's PCI BAR registers. *KVM\_DEV\_USER\_PA\_RANGE* will use ``kvm_io_bus_register_dev()`` to register *kvm\_io\_device\_ops* callbacks to be invoked when the guest performs a MMIO operation within the range. When a range is changed, ``kvm_io_bus_unregister_dev()`` is used to remove the previous instantiation. *KVM\_DEV\_USER\_TIMEOUT* will configure a timeout value that specifies how long KVM will wait for the emulation process to respond to a MMIO indication. - destroy This routine is called when the VM instance is destroyed. It will need to destroy the slave descriptor; and free any memory allocated by the driver, as well as the *kvm\_device* structure itself. slave descriptor ^^^^^^^^^^^^^^^^ The slave descriptor will have its own file operations vector, which responds to system calls on the descriptor performed by the device emulation program. - read A read returns any pending MMIO requests from the KVM driver as MMIO request structures. Multiple structures can be returned if there are multiple MMIO operations pending. The MMIO requests are moved from the pending queue to the sent queue, and if there are threads waiting for space in the pending to add new MMIO operations, they will be woken here. - write A write also consists of a set of MMIO requests. They are compared to the MMIO requests in the sent queue. Matches are removed from the sent queue, and any threads waiting for the reply are woken. If a store is removed, then the number of posted stores in the per-CPU scoreboard is decremented. When the number is zero, and a non side-effect load was waiting for posted stores to complete, the load is continued. - ioctl There are several ioctl()s that can be performed on the slave descriptor. A *KVM\_DEV\_USER\_SHADOW\_SIZE* ``ioctl()`` causes the KVM driver to allocate memory for the shadow image. This memory can later be ``mmap()``\ ed by the emulation process to share the emulation's view of device memory with the KVM driver. A *KVM\_DEV\_USER\_SHADOW\_CTRL* ``ioctl()`` controls access to the shadow image. It will send the KVM driver a shadow control map, which specifies which areas of the image can complete guest loads without sending the load request to the emulation program. It will also specify the size of load operations that are allowed. - poll An emulation program will use the ``poll()`` call with a *POLLIN* flag to determine if there are MMIO requests waiting to be read. It will return if the pending MMIO request queue is not empty. - mmap This call allows the emulation program to directly access the shadow image allocated by the KVM driver. As device emulation updates device memory, changes with no side-effects will be reflected in the shadow, and the KVM driver can satisfy guest loads from the shadow image without needing to wait for the emulation program. kvm\_io\_device ops ^^^^^^^^^^^^^^^^^^^ Each KVM per-CPU thread can handle MMIO operation on behalf of the guest VM. KVM will use the MMIO's guest physical address to search for a matching *kvm\_io\_device* to see if the MMIO can be handled by the KVM driver instead of exiting back to QEMU. If a match is found, the corresponding callback will be invoked. - read This callback is invoked when the guest performs a load to the device. Loads with side-effects must be handled synchronously, with the KVM driver putting the QEMU thread to sleep waiting for the emulation process reply before re-starting the guest. Loads that do not have side-effects may be optimized by satisfying them from the shadow image, if there are no outstanding stores to the device by this CPU. PCI memory ordering demands that a load cannot complete before all older stores to the same device have been completed. - write Stores can be handled asynchronously unless the pending MMIO request queue is full. In this case, the QEMU thread must sleep waiting for space in the queue. Stores will increment the number of posted stores in the per-CPU scoreboard, in order to implement the PCI ordering constraint above. interrupt acceleration ^^^^^^^^^^^^^^^^^^^^^^ This performance optimization would work much like a vhost user application does, where the QEMU process sets up *eventfds* that cause the device's corresponding interrupt to be triggered by the KVM driver. These irq file descriptors are sent to the emulation process at initialization, and are used when the emulation code raises a device interrupt. intx acceleration ''''''''''''''''' Traditional PCI pin interrupts are level based, so, in addition to an irq file descriptor, a re-sampling file descriptor needs to be sent to the emulation program. This second file descriptor allows multiple devices sharing an irq to be notified when the interrupt has been acknowledged by the guest, so they can re-trigger the interrupt if their device has not de-asserted its interrupt. intx irq descriptor The irq descriptors are created by the proxy object ``using event_notifier_init()`` to create the irq and re-sampling *eventds*, and ``kvm_vm_ioctl(KVM_IRQFD)`` to bind them to an interrupt. The interrupt route can be found with ``pci_device_route_intx_to_irq()``. intx routing changes Intx routing can be changed when the guest programs the APIC the device pin is connected to. The proxy object in QEMU will use ``pci_device_set_intx_routing_notifier()`` to be informed of any guest changes to the route. This handler will broadly follow the VFIO interrupt logic to change the route: de-assigning the existing irq descriptor from its route, then assigning it the new route. (see ``vfio_intx_update()``) MSI/X acceleration '''''''''''''''''' MSI/X interrupts are sent as DMA transactions to the host. The interrupt data contains a vector that is programmed by the guest, A device may have multiple MSI interrupts associated with it, so multiple irq descriptors may need to be sent to the emulation program. MSI/X irq descriptor This case will also follow the VFIO example. For each MSI/X interrupt, an *eventfd* is created, a virtual interrupt is allocated by ``kvm_irqchip_add_msi_route()``, and the virtual interrupt is bound to the eventfd with ``kvm_irqchip_add_irqfd_notifier()``. MSI/X config space changes The guest may dynamically update several MSI-related tables in the device's PCI config space. These include per-MSI interrupt enables and vector data. Additionally, MSIX tables exist in device memory space, not config space. Much like the BAR case above, the proxy object must look at guest config space programming to keep the MSI interrupt state consistent between QEMU and the emulation program. -------------- Disaggregated CPU emulation --------------------------- After IO services have been disaggregated, a second phase would be to separate a process to handle CPU instruction emulation from the main QEMU control function. There are no object separation points for this code, so the first task would be to create one. Host access controls -------------------- Separating QEMU relies on the host OS's access restriction mechanisms to enforce that the differing processes can only access the objects they are entitled to. There are a couple types of mechanisms usually provided by general purpose OSs. Discretionary access control ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Discretionary access control allows each user to control who can access their files. In Linux, this type of control is usually too coarse for QEMU separation, since it only provides three separate access controls: one for the same user ID, the second for users IDs with the same group ID, and the third for all other user IDs. Each device instance would need a separate user ID to provide access control, which is likely to be unwieldy for dynamically created VMs. Mandatory access control ~~~~~~~~~~~~~~~~~~~~~~~~ Mandatory access control allows the OS to add an additional set of controls on top of discretionary access for the OS to control. It also adds other attributes to processes and files such as types, roles, and categories, and can establish rules for how processes and files can interact. Type enforcement ^^^^^^^^^^^^^^^^ Type enforcement assigns a *type* attribute to processes and files, and allows rules to be written on what operations a process with a given type can perform on a file with a given type. QEMU separation could take advantage of type enforcement by running the emulation processes with different types, both from the main QEMU process, and from the emulation processes of different classes of devices. For example, guest disk images and disk emulation processes could have types separate from the main QEMU process and non-disk emulation processes, and the type rules could prevent processes other than disk emulation ones from accessing guest disk images. Similarly, network emulation processes can have a type separate from the main QEMU process and non-network emulation process, and only that type can access the host tun/tap device used to provide guest networking. Category enforcement ^^^^^^^^^^^^^^^^^^^^ Category enforcement assigns a set of numbers within a given range to the process or file. The process is granted access to the file if the process's set is a superset of the file's set. This enforcement can be used to separate multiple instances of devices in the same class. For example, if there are multiple disk devices provides to a guest, each device emulation process could be provisioned with a separate category. The different device emulation processes would not be able to access each other's backing disk images. Alternatively, categories could be used in lieu of the type enforcement scheme described above. In this scenario, different categories would be used to prevent device emulation processes in different classes from accessing resources assigned to other classes. --- ### Devel Multi Thread Tcg (`docs/devel/multi-thread-tcg.rst`) .. Copyright (c) 2015-2020 Linaro Ltd. This work is licensed under the terms of the GNU GPL, version 2 or later. See the COPYING file in the top-level directory. ================== Multi-threaded TCG ================== This document outlines the design for multi-threaded TCG (a.k.a MTTCG) system-mode emulation. user-mode emulation has always mirrored the thread structure of the translated executable although some of the changes done for MTTCG system emulation have improved the stability of linux-user emulation. The original system-mode TCG implementation was single threaded and dealt with multiple CPUs with simple round-robin scheduling. This simplified a lot of things but became increasingly limited as systems being emulated gained additional cores and per-core performance gains for host systems started to level off. vCPU Scheduling =============== We introduce a new running mode where each vCPU will run on its own user-space thread. This is enabled by default for all FE/BE combinations where the host memory model is able to accommodate the guest (TCG_GUEST_DEFAULT_MO & ~TCG_TARGET_DEFAULT_MO is zero) and the guest has had the required work done to support this safely (TARGET_SUPPORTS_MTTCG). System emulation will fall back to the original round robin approach if: * forced by --accel tcg,thread=single * enabling --icount mode * 64 bit guests on 32 bit hosts (TCG_OVERSIZED_GUEST) In the general case of running translated code there should be no inter-vCPU dependencies and all vCPUs should be able to run at full speed. Synchronisation will only be required while accessing internal shared data structures or when the emulated architecture requires a coherent representation of the emulated machine state. Shared Data Structures ====================== Main Run Loop ------------- Even when there is no code being generated there are a number of structures associated with the hot-path through the main run-loop. These are associated with looking up the next translation block to execute. These include: tb_jmp_cache (per-vCPU, cache of recent jumps) tb_ctx.htable (global hash table, phys address->tb lookup) As TB linking only occurs when blocks are in the same page this code is critical to performance as looking up the next TB to execute is the most common reason to exit the generated code. DESIGN REQUIREMENT: Make access to lookup structures safe with multiple reader/writer threads. Minimise any lock contention to do it. The hot-path avoids using locks where possible. The tb_jmp_cache is updated with atomic accesses to ensure consistent results. The fall back QHT based hash table is also designed for lockless lookups. Locks are only taken when code generation is required or TranslationBlocks have their block-to-block jumps patched. Global TCG State ---------------- User-mode emulation ~~~~~~~~~~~~~~~~~~~ We need to protect the entire code generation cycle including any post generation patching of the translated code. This also implies a shared translation buffer which contains code running on all cores. Any execution path that comes to the main run loop will need to hold a mutex for code generation. This also includes times when we need flush code or entries from any shared lookups/caches. Structures held on a per-vCPU basis won't need locking unless other vCPUs will need to modify them. DESIGN REQUIREMENT: Add locking around all code generation and TB patching. (Current solution) Code generation is serialised with mmap_lock(). !User-mode emulation ~~~~~~~~~~~~~~~~~~~~ Each vCPU has its own TCG context and associated TCG region, thereby requiring no locking during translation. Translation Blocks ------------------ Currently the whole system shares a single code generation buffer which when full will force a flush of all translations and start from scratch again. Some operations also force a full flush of translations including: - debugging operations (breakpoint insertion/removal) - some CPU helper functions - linux-user spawning its first thread This is done with the async_safe_run_on_cpu() mechanism to ensure all vCPUs are quiescent when changes are being made to shared global structures. More granular translation invalidation events are typically due to a change of the state of a physical page: - code modification (self modify code, patching code) - page changes (new page mapping in linux-user mode) While setting the invalid flag in a TranslationBlock will stop it being used when looked up in the hot-path there are a number of other book-keeping structures that need to be safely cleared. Any TranslationBlocks which have been patched to jump directly to the now invalid blocks need the jump patches reversing so they will return to the C code. There are a number of look-up caches that need to be properly updated including the: - jump lookup cache - the physical-to-tb lookup hash table - the global page table The global page table (l1_map) which provides a multi-level look-up for PageDesc structures which contain pointers to the start of a linked list of all Translation Blocks in that page (see page_next). Both the jump patching and the page cache involve linked lists that the invalidated TranslationBlock needs to be removed from. DESIGN REQUIREMENT: Safely handle invalidation of TBs - safely patch/revert direct jumps - remove central PageDesc lookup entries - ensure lookup caches/hashes are safely updated (Current solution) The direct jump themselves are updated atomically by the TCG tb_set_jmp_target() code. Modification to the linked lists that allow searching for linked pages are done under the protection of tb->jmp_lock, where tb is the destination block of a jump. Each origin block keeps a pointer to its destinations so that the appropriate lock can be acquired before iterating over a jump list. The global page table is a lockless radix tree; cmpxchg is used to atomically insert new elements. The lookup caches are updated atomically and the lookup hash uses QHT which is designed for concurrent safe lookup. Parallel code generation is supported. QHT is used at insertion time as the synchronization point across threads, thereby ensuring that we only keep track of a single TranslationBlock for each guest code block. Memory maps and TLBs -------------------- The memory handling code is fairly critical to the speed of memory access in the emulated system. The SoftMMU code is designed so the hot-path can be handled entirely within translated code. This is handled with a per-vCPU TLB structure which once populated will allow a series of accesses to the page to occur without exiting the translated code. It is possible to set flags in the TLB address which will ensure the slow-path is taken for each access. This can be done to support: - Memory regions (dividing up access to PIO, MMIO and RAM) - Dirty page tracking (for code gen, SMC detection, migration and display) - Virtual TLB (for translating guest address->real address) When the TLB tables are updated by a vCPU thread other than their own we need to ensure it is done in a safe way so no inconsistent state is seen by the vCPU thread. Some operations require updating a number of vCPUs TLBs at the same time in a synchronised manner. DESIGN REQUIREMENTS: - TLB Flush All/Page - can be across-vCPUs - cross vCPU TLB flush may need other vCPU brought to halt - change may need to be visible to the calling vCPU immediately - TLB Flag Update - usually cross-vCPU - want change to be visible as soon as possible - TLB Update (update a CPUTLBEntry, via tlb_set_page_with_attrs) - This is a per-vCPU table - by definition can't race - updated by its own thread when the slow-path is forced (Current solution) We have updated cputlb.c to defer operations when a cross-vCPU operation with async_run_on_cpu() which ensures each vCPU sees a coherent state when it next runs its work (in a few instructions time). A new set up operations (tlb_flush_*_all_cpus) take an additional flag which when set will force synchronisation by setting the source vCPUs work as "safe work" and exiting the cpu run loop. This ensure by the time execution restarts all flush operations have completed. TLB flag updates are all done atomically and are also protected by the corresponding page lock. (Known limitation) Not really a limitation but the wait mechanism is overly strict for some architectures which only need flushes completed by a barrier instruction. This could be a future optimisation. Emulated hardware state ----------------------- Currently thanks to KVM work any access to IO memory is automatically protected by the global iothread mutex, also known as the BQL (Big QEMU Lock). Any IO region that doesn't use global mutex is expected to do its own locking. However IO memory isn't the only way emulated hardware state can be modified. Some architectures have model specific registers that trigger hardware emulation features. Generally any translation helper that needs to update more than a single vCPUs of state should take the BQL. As the BQL, or global iothread mutex is shared across the system we push the use of the lock as far down into the TCG code as possible to minimise contention. (Current solution) MMIO access automatically serialises hardware emulation by way of the BQL. Currently Arm targets serialise all ARM_CP_IO register accesses and also defer the reset/startup of vCPUs to the vCPU context by way of async_run_on_cpu(). Updates to interrupt state are also protected by the BQL as they can often be cross vCPU. Memory Consistency ================== Between emulated guests and host systems there are a range of memory consistency models. Even emulating weakly ordered systems on strongly ordered hosts needs to ensure things like store-after-load re-ordering can be prevented when the guest wants to. Memory Barriers --------------- Barriers (sometimes known as fences) provide a mechanism for software to enforce a particular ordering of memory operations from the point of view of external observers (e.g. another processor core). They can apply to any memory operations as well as just loads or stores. The Linux kernel has an excellent `write-up `_ on the various forms of memory barrier and the guarantees they can provide. Barriers are often wrapped around synchronisation primitives to provide explicit memory ordering semantics. However they can be used by themselves to provide safe lockless access by ensuring for example a change to a signal flag will only be visible once the changes to payload are. DESIGN REQUIREMENT: Add a new tcg_memory_barrier op This would enforce a strong load/store ordering so all loads/stores complete at the memory barrier. On single-core non-SMP strongly ordered backends this could become a NOP. Aside from explicit standalone memory barrier instructions there are also implicit memory ordering semantics which comes with each guest memory access instruction. For example all x86 load/stores come with fairly strong guarantees of sequential consistency whereas Arm has special variants of load/store instructions that imply acquire/release semantics. In the case of a strongly ordered guest architecture being emulated on a weakly ordered host the scope for a heavy performance impact is quite high. DESIGN REQUIREMENTS: Be efficient with use of memory barriers - host systems with stronger implied guarantees can skip some barriers - merge consecutive barriers to the strongest one (Current solution) The system currently has a tcg_gen_mb() which will add memory barrier operations if code generation is being done in a parallel context. The tcg_optimize() function attempts to merge barriers up to their strongest form before any load/store operations. The solution was originally developed and tested for linux-user based systems. All backends have been converted to emit fences when required. So far the following front-ends have been updated to emit fences when required: - target-i386 - target-arm - target-aarch64 - target-alpha - target-mips Memory Control and Maintenance ------------------------------ This includes a class of instructions for controlling system cache behaviour. While QEMU doesn't model cache behaviour these instructions are often seen when code modification has taken place to ensure the changes take effect. Synchronisation Primitives -------------------------- There are two broad types of synchronisation primitives found in modern ISAs: atomic instructions and exclusive regions. The first type offer a simple atomic instruction which will guarantee some sort of test and conditional store will be truly atomic w.r.t. other cores sharing access to the memory. The classic example is the x86 cmpxchg instruction. The second type offer a pair of load/store instructions which offer a guarantee that a region of memory has not been touched between the load and store instructions. An example of this is Arm's ldrex/strex pair where the strex instruction will return a flag indicating a successful store only if no other CPU has accessed the memory region since the ldrex. Traditionally TCG has generated a series of operations that work because they are within the context of a single translation block so will have completed before another CPU is scheduled. However with the ability to have multiple threads running to emulate multiple CPUs we will need to explicitly expose these semantics. DESIGN REQUIREMENTS: - Support classic atomic instructions - Support load/store exclusive (or load link/store conditional) pairs - Generic enough infrastructure to support all guest architectures CURRENT OPEN QUESTIONS: - How problematic is the ABA problem in general? (Current solution) The TCG provides a number of atomic helpers (tcg_gen_atomic_*) which can be used directly or combined to emulate other instructions like Arm's ldrex/strex instructions. While they are susceptible to the ABA problem so far common guests have not implemented patterns where this may be a problem - typically presenting a locking ABI which assumes cmpxchg like semantics. The code also includes a fall-back for cases where multi-threaded TCG ops can't work (e.g. guest atomic width > host atomic width). In this case an EXCP_ATOMIC exit occurs and the instruction is emulated with an exclusive lock which ensures all emulation is serialised. While the atomic helpers look good enough for now there may be a need to look at solutions that can more closely model the guest architectures semantics. --- --- METRICS --- - Files Extracted: 58 - Estimated Token Budget: ~128044 tokens - Canonical Reference: https://codewiki.google/github.com/kata-containers/qemu