qemu Documentation

Define pvrdma device:
qemu:commandline
<qemu:arg value='-object'/>
<qemu:arg value='memory-backend-ram,id=mb1,size=1G,share'/>
<qemu:arg value='-numa'/>
<qemu:arg value='node,memdev=mb1'/>
<qemu:arg value='-chardev'/>
<qemu:arg value='socket,path=/var/run/rdmacm-mux-rxe0-1,id=mads'/>
<qemu:arg value='-device'/>
<qemu:arg value='pvrdma,addr=10.1,ibdev=rxe0,netdev=bridge0,mad-chardev=mads'/>

4. Implementation details

4.1 Overview

The device acts like a proxy between the Guest Driver and the host
ibdevice interface.
On configuration path:

• For every hardware resource request (PD/QP/CQ/...) the pvrdma will request
  a resource from the backend interface, maintaining a 1-1 mapping
  between the guest and host.
  On data path:
• Every post_send/receive received from the guest will be converted into
  a post_send/receive for the backend. The buffers data will not be touched
  or copied resulting in near bare-metal performance for large enough buffers.
• Completions from the backend interface will result in completions for
  the pvrdma device.

4.2 PCI BARs

PCI Bars:
BAR 0 - MSI-X
MSI-X vectors:
(0) Command - used when execution of a command is completed.
(1) Async - not in use.
(2) Completion - used when a completion event is placed in
device's CQ ring.
BAR 1 - Registers
--------------------------------------------------------
| VERSION | DSR | CTL | REQ | ERR | ICR | IMR | MAC |
--------------------------------------------------------
DSR - Address of driver/device shared memory used
for the command channel, used for passing:
- General info such as driver version
- Address of 'command' and 'response'
- Address of async ring
- Address of device's CQ ring
- Device capabilities
CTL - Device control operations (activate, reset etc)
IMG - Set interrupt mask
REQ - Command execution register
ERR - Operation status

BAR 2 - UAR
    ---------------------------------------------------------
    | QP_NUM  | SEND/RECV Flag ||  CQ_NUM |   ARM/POLL Flag |
    ---------------------------------------------------------
	- Offset 0 used for QP operations (send and recv)
	- Offset 4 used for CQ operations (arm and poll)

4.3 Major flows

4.3.1 Create CQ

- Guest driver
    - Allocates pages for CQ ring
    - Creates page directory (pdir) to hold CQ ring's pages
    - Initializes CQ ring
    - Initializes 'Create CQ' command object (cqe, pdir etc)
    - Copies the command to 'command' address
    - Writes 0 into REQ register
- Device
    - Reads the request object from the 'command' address
    - Allocates CQ object and initialize CQ ring based on pdir
    - Creates the backend CQ
    - Writes operation status to ERR register
    - Posts command-interrupt to guest
- Guest driver
    - Reads the HW response code from ERR register

4.3.2 Create QP

- Guest driver
    - Allocates pages for send and receive rings
    - Creates page directory(pdir) to hold the ring's pages
    - Initializes 'Create QP' command object (max_send_wr,
      send_cq_handle, recv_cq_handle, pdir etc)
    - Copies the object to 'command' address
    - Write 0 into REQ register
- Device
    - Reads the request object from 'command' address
    - Allocates the QP object and initialize
        - Send and recv rings based on pdir
        - Send and recv ring state
    - Creates the backend QP
    - Writes the operation status to ERR register
    - Posts command-interrupt to guest
- Guest driver
    - Reads the HW response code from ERR register

4.3.3 Post receive

- Guest driver
    - Initializes a wqe and place it on recv ring
    - Write to qpn|qp_recv_bit (31) to QP offset in UAR
- Device
    - Extracts qpn from UAR
    - Walks through the ring and does the following for each wqe
        - Prepares the backend CQE context to be used when
          receiving completion from backend (wr_id, op_code, emu_cq_num)
        - For each sge prepares backend sge
        - Calls backend's post_recv

4.3.4 Process backend events

- Done by a dedicated thread used to process backend events;
  at initialization is attached to the device and creates
  the communication channel.
- Thread main loop:
    - Polls for completions
    - Extracts QEMU _cq_num, wr_id and op_code from context
    - Writes CQE to CQ ring
    - Writes CQ number to device CQ
    - Sends completion-interrupt to guest
    - Deallocates context
    - Acks the event to backend

5. Limitations

• The device obviously is limited by the Guest Linux Driver features implementation
  of the VMware device API.
• Memory registration mechanism requires mremap for every page in the buffer in order
  to map it to a contiguous virtual address range. Since this is not the data path
  it should not matter much. If the default max mr size is increased, be aware that
  memory registration can take up to 0.5 seconds for 1GB of memory.
• The device requires target page size to be the same as the host page size,
  otherwise it will fail to init.
• QEMU cannot map guest RAM from a file descriptor if a pvrdma device is attached,
  so it can't work with huge pages. The limitation will be addressed in the future,
  however QEMU allocates Guest RAM with MADV_HUGEPAGE so if there are enough huge
  pages available, QEMU will use them. QEMU will fail to init if the requirements
  are not met.

6. Performance

   By design the pvrdma device exits on each post-send/receive, so for small buffers
   the performance is affected; however for medium buffers it will became close to
   bare metal and from 1MB buffers and up it reaches bare metal performance.
   (tested with 2 VMs, the pvrdma devices connected to 2 VFs of the same device)

All the above assumes no memory registration is done on data path.

Qcow2 Cache (`docs/qcow2-cache.txt`) Copy Section

qcow2 L2/refcount cache configuration

Copyright (C) 2015, 2018-2020 Igalia, S.L.
Author: Alberto Garcia

This work is licensed under the terms of the GNU GPL, version 2 or
later. See the COPYING file in the top-level directory.

Introduction Copy Chapter

The QEMU qcow2 driver has two caches that can improve the I/O
performance significantly. However, setting the right cache sizes is
not a straightforward operation.

This document attempts to give an overview of the L2 and refcount
caches, and how to configure them.

Please refer to the docs/interop/qcow2.txt file for an in-depth
technical description of the qcow2 file format.

Clusters Copy Chapter

A qcow2 file is organized in units of constant size called clusters.

The cluster size is configurable, but it must be a power of two and
its value 512 bytes or higher. QEMU currently defaults to 64 KB
clusters, and it does not support sizes larger than 2MB.

The 'qemu-img create' command supports specifying the size using the
cluster_size option:

qemu-img create -f qcow2 -o cluster_size=128K hd.qcow2 4G

The L2 tables Copy Chapter

The qcow2 format uses a two-level structure to map the virtual disk as
seen by the guest to the disk image in the host. These structures are
called the L1 and L2 tables.

There is one single L1 table per disk image. The table is small and is
always kept in memory.

There can be many L2 tables, depending on how much space has been
allocated in the image. Each table is one cluster in size. In order to
read or write data from the virtual disk, QEMU needs to read its
corresponding L2 table to find out where that data is located. Since
reading the table for each I/O operation can be expensive, QEMU keeps
an L2 cache in memory to speed up disk access.

The size of the L2 cache can be configured, and setting the right
value can improve the I/O performance significantly.

The refcount blocks Copy Chapter

The qcow2 format also maintains a reference count for each cluster.
Reference counts are used for cluster allocation and internal
snapshots. The data is stored in a two-level structure similar to the
L1/L2 tables described above.

The second level structures are called refcount blocks, are also one
cluster in size and the number is also variable and dependent on the
amount of allocated space.

Each block contains a number of refcount entries. Their size (in bits)
is a power of two and must not be higher than 64. It defaults to 16
bits, but a different value can be set using the refcount_bits option:

qemu-img create -f qcow2 -o refcount_bits=8 hd.qcow2 4G

QEMU keeps a refcount cache to speed up I/O much like the
aforementioned L2 cache, and its size can also be configured.

Choosing the right cache sizes Copy Chapter

In order to choose the cache sizes we need to know how they relate to
the amount of allocated space.

The part of the virtual disk that can be mapped by the L2 and refcount
caches (in bytes) is:

disk_size = l2_cache_size * cluster_size / 8
disk_size = refcount_cache_size * cluster_size * 8 / refcount_bits

With the default values for cluster_size (64KB) and refcount_bits
(16), this becomes:

disk_size = l2_cache_size * 8192
disk_size = refcount_cache_size * 32768

So in order to cover n GB of disk space with the default values we
need:

l2_cache_size = disk_size_GB * 131072
refcount_cache_size = disk_size_GB * 32768

For example, 1MB of L2 cache is needed to cover every 8 GB of the virtual
image size (given that the default cluster size is used):

8 GB / 8192 = 1 MB

The refcount cache is 4 times the cluster size by default. With the default
cluster size of 64 KB, it is 256 KB (262144 bytes). This is sufficient for
8 GB of image size:

262144 * 32768 = 8 GB

How to configure the cache sizes Copy Chapter

Cache sizes can be configured using the -drive option in the
command-line, or the 'blockdev-add' QMP command.

There are three options available, and all of them take bytes:

"l2-cache-size": maximum size of the L2 table cache
"refcount-cache-size": maximum size of the refcount block cache
"cache-size": maximum size of both caches combined

There are a few things that need to be taken into account:

• Both caches must have a size that is a multiple of the cluster size
  (or the cache entry size: see "Using smaller cache sizes" below).

• The maximum L2 cache size is 32 MB by default on Linux platforms (enough
  for full coverage of 256 GB images, with the default cluster size). This
  value can be modified using the "l2-cache-size" option. QEMU will not use
  more memory than needed to hold all of the image's L2 tables, regardless
  of this max. value.
  On non-Linux platforms the maximal value is smaller by default (8 MB) and
  this difference stems from the fact that on Linux the cache can be cleared
  periodically if needed, using the "cache-clean-interval" option (see below).
  The minimal L2 cache size is 2 clusters (or 2 cache entries, see below).

• The default (and minimum) refcount cache size is 4 clusters.

• If only "cache-size" is specified then QEMU will assign as much
  memory as possible to the L2 cache before increasing the refcount
  cache size.

• At most two of "l2-cache-size", "refcount-cache-size", and "cache-size"
  can be set simultaneously.

Unlike L2 tables, refcount blocks are not used during normal I/O but
only during allocations and internal snapshots. In most cases they are
accessed sequentially (even during random guest I/O) so increasing the
refcount cache size won't have any measurable effect in performance
(this can change if you are using internal snapshots, so you may want
to think about increasing the cache size if you use them heavily).

Before QEMU 2.12 the refcount cache had a default size of 1/4 of the
L2 cache size. This resulted in unnecessarily large caches, so now the
refcount cache is as small as possible unless overridden by the user.

Using smaller cache entries Copy Chapter

The qcow2 L2 cache can store complete tables. This means that if QEMU
needs an entry from an L2 table then the whole table is read from disk
and is kept in the cache. If the cache is full then a complete table
needs to be evicted first.

This can be inefficient with large cluster sizes since it results in
more disk I/O and wastes more cache memory.

Since QEMU 2.12 you can change the size of the L2 cache entry and make
it smaller than the cluster size. This can be configured using the
"l2-cache-entry-size" parameter:

-drive file=hd.qcow2,l2-cache-size=2097152,l2-cache-entry-size=4096

Since QEMU 4.0 the value of l2-cache-entry-size defaults to 4KB (or
the cluster size if it's smaller).

Some things to take into account:

• The L2 cache entry size has the same restrictions as the cluster
  size (power of two, at least 512 bytes).

• Smaller entry sizes generally improve the cache efficiency and make
  disk I/O faster. This is particularly true with solid state drives
  so it's a good idea to reduce the entry size in those cases. With
  rotating hard drives the situation is a bit more complicated so you
  should test it first and stay with the default size if unsure.

• Try different entry sizes to see which one gives faster performance
  in your case. The block size of the host filesystem is generally a
  good default (usually 4096 bytes in the case of ext4, hence the
  default).

• Only the L2 cache can be configured this way. The refcount cache
  always uses the cluster size as the entry size.

• If the L2 cache is big enough to hold all of the image's L2 tables
  (as explained in the "Choosing the right cache sizes" and "How to
  configure the cache sizes" sections in this document) then none of
  this is necessary and you can omit the "l2-cache-entry-size"
  parameter altogether. In this case QEMU makes the entry size
  equal to the cluster size by default.

Reducing the memory usage Copy Chapter

It is possible to clean unused cache entries in order to reduce the
memory usage during periods of low I/O activity.

The parameter "cache-clean-interval" defines an interval (in seconds),
after which all the cache entries that haven't been accessed during the
interval are removed from memory. Setting this parameter to 0 disables this
feature.

The following example removes all unused cache entries every 15 minutes:

-drive file=hd.qcow2,cache-clean-interval=900

If unset, the default value for this parameter is 600 on platforms which
support this functionality, and is 0 (disabled) on other platforms.

This functionality currently relies on the MADV_DONTNEED argument for
madvise() to actually free the memory. This is a Linux-specific feature,
so cache-clean-interval is not supported on other systems.

Extended L2 Entries Copy Chapter

All numbers shown in this document are valid for qcow2 images with normal
64-bit L2 entries.

Images with extended L2 entries need twice as much L2 metadata, so the L2
cache size must be twice as large for the same disk space.

disk_size = l2_cache_size * cluster_size / 16

i.e.

l2_cache_size = disk_size * 16 / cluster_size

Refcount blocks are not affected by this.

Qdev Device Use (`docs/qdev-device-use.txt`) Copy Section

= How to convert to -device & friends =

=== Specifying Bus and Address on Bus ===

In qdev, each device has a parent bus. Some devices provide one or
more buses for children. You can specify a device's parent bus with
-device parameter bus.

A device typically has a device address on its parent bus. For buses
where this address can be configured, devices provide a bus-specific
property. Examples:

bus         property name       value format
PCI         addr                %x.%x    (dev.fn, .fn optional)
I2C         address             %u
SCSI        scsi-id             %u
IDE         unit                %u
HDA         cad                 %u
virtio-serial-bus  nr           %u
ccid-bus    slot                %u
USB         port                %d(.%d)*    (port.port...)

Example: device i440FX-pcihost is on the root bus, and provides a PCI
bus named pci.0. To put a FOO device into its slot 4, use -device
FOO,bus=/i440FX-pcihost/pci.0,addr=4. The abbreviated form bus=pci.0
also works as long as the bus name is unique.

=== Block Devices ===

A QEMU block device (drive) has a host and a guest part.

In the general case, the guest device is connected to a controller
device. For instance, the IDE controller provides two IDE buses, each
of which can have up to two devices, and each device is a guest part,
and is connected to a host part.

Except we sometimes lump controller, bus(es) and drive device(s) all
together into a single device. For instance, the ISA floppy
controller is connected to up to two host drives.

The old ways to define block devices define host and guest part
together. Sometimes, they can even define a controller device in
addition to the block device.

The new way keeps the parts separate: you create the host part with
-drive, and guest device(s) with -device.

The various old ways to define drives all boil down to the common form

-drive if=TYPE,bus=BUS,unit=UNIT,OPTS...

TYPE, BUS and UNIT identify the controller device, which of its buses
to use, and the drive's address on that bus. Details depend on TYPE.

Instead of bus=BUS,unit=UNIT, you can also say index=IDX.

In the new way, this becomes something like

-drive if=none,id=DRIVE-ID,HOST-OPTS...
-device DEVNAME,drive=DRIVE-ID,DEV-OPTS...

The old OPTS get split into HOST-OPTS and DEV-OPTS as follows:

• file, format, snapshot, cache, aio, readonly, rerror, werror go into
  HOST-OPTS.

• cyls, head, secs and trans go into HOST-OPTS. Future work: they
  should go into DEV-OPTS instead.

• serial goes into DEV-OPTS, for devices supporting serial numbers.
  For other devices, it goes nowhere.

• media is special. In the old way, it selects disk vs. CD-ROM with
  if=ide, if=scsi and if=xen. The new way uses DEVNAME for that.
  Additionally, readonly=on goes into HOST-OPTS.

• addr is special, see if=virtio below.

The -device argument differs in detail for each type of drive:

• if=ide

  -device DEVNAME,drive=DRIVE-ID,bus=IDE-BUS,unit=UNIT

  where DEVNAME is either ide-hd or ide-cd, IDE-BUS identifies an IDE
  bus, normally either ide.0 or ide.1, and UNIT is either 0 or 1.

• if=scsi

  The old way implicitly creates SCSI controllers as needed. The new
  way makes that explicit:

  -device lsi53c895a,id=ID

  As for all PCI devices, you can add bus=PCI-BUS,addr=DEVFN to
  control the PCI device address.

  This SCSI controller provides a single SCSI bus, named ID.0. Put a
  disk on it:

  -device DEVNAME,drive=DRIVE-ID,bus=ID.0,scsi-id=UNIT

  where DEVNAME is either scsi-hd, scsi-cd or scsi-generic.

• if=floppy

  -device floppy,unit=UNIT,drive=DRIVE-ID

  Without any -device floppy,... you get an empty unit 0 and no unit

  1. You can use -nodefaults to suppress the default unit 0, see
     "Default Devices".

• if=virtio

  -device virtio-blk-pci,drive=DRIVE-ID,class=C,vectors=V,ioeventfd=IOEVENTFD

  This lets you control PCI device class and MSI-X vectors.

  IOEVENTFD controls whether or not ioeventfd is used for virtqueue
  notify. It can be set to on (default) or off.

  As for all PCI devices, you can add bus=PCI-BUS,addr=DEVFN to
  control the PCI device address. This replaces option addr available
  with -drive if=virtio.

• if=pflash, if=mtd, if=sd, if=xen are not yet available with -device

For USB devices, the old way was actually different:

-usbdevice disk:format=FMT:FILENAME

"Was" because "disk:" is gone since v2.12.0.

The old way provided much less control than -drive's OPTS... The new
way fixes that:

-device usb-storage,drive=DRIVE-ID,removable=RMB

The removable parameter gives control over the SCSI INQUIRY removable
(RMB) bit. USB thumbdrives usually set removable=on, while USB hard
disks set removable=off.

Bug: usb-storage pretends to be a block device, but it's really a SCSI
controller that can serve only a single device, which it creates
automatically. The automatic creation guesses what kind of guest part
to create from the host part, like -drive if=scsi. Host and guest
part are not cleanly separated.

=== Character Devices ===

A QEMU character device has a host and a guest part.

The old ways to define character devices define host and guest part
together.

The new way keeps the parts separate: you create the host part with
-chardev, and the guest device with -device.

The various old ways to define a character device are all of the
general form

-FOO FOO-OPTS...,LEGACY-CHARDEV

where FOO-OPTS... is specific to -FOO, and the host part
LEGACY-CHARDEV is the same everywhere.

In the new way, this becomes

-chardev HOST-OPTS...,id=CHR-ID
-device DEVNAME,chardev=CHR-ID,DEV-OPTS...

The appropriate DEVNAME depends on the machine type. For type "pc":

• -serial becomes -device isa-serial,iobase=IOADDR,irq=IRQ,index=IDX

  This lets you control I/O ports and IRQs.

• -parallel becomes -device isa-parallel,iobase=IOADDR,irq=IRQ,index=IDX

  This lets you control I/O ports and IRQs.

• -usbdevice braille doesn't support LEGACY-CHARDEV syntax. It always
  uses "braille". With -device, this useful default is gone, so you
  have to use something like

  -device usb-braille,chardev=braille -chardev braille,id=braille

• -usbdevice serial::chardev is gone since v2.12.0. It became
  -device usb-serial,chardev=dev.

LEGACY-CHARDEV translates to -chardev HOST-OPTS... as follows:

• null becomes -chardev null

• pty, msmouse, wctablet, braille, stdio likewise

• vc:WIDTHxHEIGHT becomes -chardev vc,width=WIDTH,height=HEIGHT

• vc:CxC becomes -chardev vc,cols=,rows=

• con: becomes -chardev console

• COM becomes -chardev serial,path=COM

• file:FNAME becomes -chardev file,path=FNAME

• pipe:FNAME becomes -chardev pipe,path=FNAME

• tcp:HOST:PORT,OPTS... becomes -chardev socket,host=HOST,port=PORT,OPTS...

• telnet:HOST:PORT,OPTS... becomes
  -chardev socket,host=HOST,port=PORT,OPTS...,telnet=on

• udp:HOST:PORT@LOCALADDR:LOCALPORT becomes
  -chardev udp,host=HOST,port=PORT,localaddr=LOCALADDR,localport=LOCALPORT

• unix:FNAME becomes -chardev socket,path=FNAME

• /dev/parportN becomes -chardev parport,file=/dev/parportN

• /dev/ppiN likewise

• Any other /dev/FNAME becomes -chardev tty,path=/dev/FNAME

• mon:LEGACY-CHARDEV is special: it multiplexes the monitor onto the
  character device defined by LEGACY-CHARDEV. -chardev provides more
  general multiplexing instead: you can connect up to four users to a
  single host part. You need to pass mux=on to -chardev to enable
  switching the input focus.

QEMU uses LEGACY-CHARDEV syntax not just to set up guest devices, but
also in various other places such as -monitor or -net
user,guestfwd=... You can use chardev:CHR-ID in place of
LEGACY-CHARDEV to refer to a host part defined with -chardev.

=== Network Devices ===

Host and guest part of network devices have always been separate.

The old way to define the guest part looks like this:

-net nic,netdev=NET-ID,macaddr=MACADDR,model=MODEL,name=ID,addr=STR,vectors=V

Except for USB it looked like this:

-usbdevice net:netdev=NET-ID,macaddr=MACADDR,name=ID

"Looked" because "net:" is gone since v2.12.0.

The new way is -device:

-device DEVNAME,netdev=NET-ID,mac=MACADDR,DEV-OPTS...

DEVNAME equals MODEL, except for virtio you have to name the virtio
device appropriate for the bus (virtio-net-pci for PCI), and for USB
you have to use usb-net.

The old name=ID parameter becomes the usual id=ID with -device.

For PCI devices, you can add bus=PCI-BUS,addr=DEVFN to control the PCI
device address, as usual. The old -net nic provides parameter addr
for that, which is silently ignored when the NIC is not a PCI device.

For virtio-net-pci, you can control whether or not ioeventfd is used for
virtqueue notify by setting ioeventfd= to on or off (default).

-net nic accepts vectors=V for all models, but it's silently ignored
except for virtio-net-pci (model=virtio). With -device, only devices
that support it accept it.

Not all devices are available with -device at this time. All PCI
devices and ne2k_isa are.

Some PCI devices aren't available with -net nic, e.g. i82558a.

=== Graphics Devices ===

Host and guest part of graphics devices have always been separate.

The old way to define the guest graphics device is -vga VGA. Not all
machines support all -vga options.

The new way is -device. The mapping from -vga argument to -device
depends on the machine type. For machine "pc", it's:

std         -device VGA
cirrus      -device cirrus-vga
vmware      -device vmware-svga
qxl         -device qxl-vga
none        -nodefaults
            disables more than just VGA, see "Default Devices"

As for all PCI devices, you can add bus=PCI-BUS,addr=DEVFN to control
the PCI device address.

-device VGA supports properties bios-offset and bios-size, but they
aren't used with machine type "pc".

For machine "isapc", it's

std         -device isa-vga
cirrus      not yet available with -device
none        -nodefaults
            disables more than just VGA, see "Default Devices"

Bug: the new way doesn't work for machine types "pc" and "isapc",
because it violates obscure device initialization ordering
constraints.

=== Audio Devices ===

Host and guest part of audio devices have always been separate.

The old way to define guest audio devices is -soundhw C1,...

The new way is to define each guest audio device separately with
-device.

Map from -soundhw sound card name to -device:

ac97        -device AC97
cs4231a     -device cs4231a,iobase=IOADDR,irq=IRQ,dma=DMA
es1370      -device ES1370
gus         -device gus,iobase=IOADDR,irq=IRQ,dma=DMA,freq=F
hda         -device intel-hda,msi=MSI -device hda-duplex
sb16        -device sb16,iobase=IOADDR,irq=IRQ,dma=DMA,dma16=DMA16,version=V
adlib       not yet available with -device
pcspk       not yet available with -device

For PCI devices, you can add bus=PCI-BUS,addr=DEVFN to control the PCI
device address, as usual.

=== USB Devices ===

The old way to define a virtual USB device is -usbdevice DRIVER:OPTS...

The new way is -device DEVNAME,DEV-OPTS... Details depend on DRIVER:

• ccid -device usb-ccid
• keyboard -device usb-kbd
• mouse -device usb-mouse
• tablet -device usb-tablet
• wacom-tablet -device usb-wacom-tablet
• u2f -device u2f-{emulated,passthru}
• braille See "Character Devices"

Until v2.12.0, we additionally had

• host:... See "Host Device Assignment"
• disk:... See "Block Devices"
• serial:... See "Character Devices"
• net:... See "Network Devices"

=== Watchdog Devices ===

Host and guest part of watchdog devices have always been separate.

The old way to define a guest watchdog device is -watchdog DEVNAME.
The new way is -device DEVNAME. For PCI devices, you can add
bus=PCI-BUS,addr=DEVFN to control the PCI device address, as usual.

=== Host Device Assignment ===

QEMU supports assigning host PCI devices (qemu-kvm only at this time)
and host USB devices. PCI devices can only be assigned with -device:

-device vfio-pci,host=ADDR,id=ID

The old way to assign a USB host device

-usbdevice host:auto:BUS.ADDR:VID:PRID

was removed in v2.12.0. Any of BUS, ADDR, VID, PRID could be the
wildcard *.

The new way is

-device usb-host,hostbus=BUS,hostaddr=ADDR,vendorid=VID,productid=PRID

Omitted options match anything.

=== Default Devices ===

QEMU creates a number of devices by default, depending on the machine
type.

-device DEVNAME... and global DEVNAME... suppress default devices for
some DEVNAMEs:

default device      suppressing DEVNAMEs
CD-ROM              ide-cd, ide-hd, scsi-cd, scsi-hd
floppy              floppy, isa-fdc
parallel            isa-parallel
serial              isa-serial
VGA                 VGA, cirrus-vga, isa-vga, isa-cirrus-vga,
                    vmware-svga, qxl-vga, virtio-vga, ati-vga,
                    vhost-user-vga

The default NIC is connected to a default part created along with it.
It is not suppressed by configuring a NIC with -device (you may call
that a bug). -net and -netdev suppress the default NIC.

-nodefaults suppresses all the default devices mentioned above, plus a
few other things such as default SD-Card drive and default monitor.

Rdma (`docs/rdma.txt`) Copy Section

(RDMA: Remote Direct Memory Access) RDMA Live Migration Specification, Version # 1

Wiki: https://wiki.qemu.org/Features/RDMALiveMigration
Github: [email protected]:hinesmr/qemu.git, 'rdma' branch

Copyright (C) 2013 Michael R. Hines [email protected]

An exhaustive paper (2010) shows additional performance details
linked on the QEMU wiki above.

Contents:

• Introduction
• Before running
• Running
• Performance
• RDMA Migration Protocol Description
• Versioning and Capabilities
• QEMUFileRDMA Interface
• Migration of VM's ram
• Error handling
• TODO

Introduction:

RDMA helps make your migration more deterministic under heavy load because
of the significantly lower latency and higher throughput over TCP/IP. This is
because the RDMA I/O architecture reduces the number of interrupts and
data copies by bypassing the host networking stack. In particular, a TCP-based
migration, under certain types of memory-bound workloads, may take a more
unpredictable amount of time to complete the migration if the amount of
memory tracked during each live migration iteration round cannot keep pace
with the rate of dirty memory produced by the workload.

RDMA currently comes in two flavors: both Ethernet based (RoCE, or RDMA
over Converged Ethernet) as well as Infiniband-based. This implementation of
migration using RDMA is capable of using both technologies because of
the use of the OpenFabrics OFED software stack that abstracts out the
programming model irrespective of the underlying hardware.

Refer to openfabrics.org or your respective RDMA hardware vendor for
an understanding on how to verify that you have the OFED software stack
installed in your environment. You should be able to successfully link
against the "librdmacm" and "libibverbs" libraries and development headers
for a working build of QEMU to run successfully using RDMA Migration.

BEFORE RUNNING:

Use of RDMA during migration requires pinning and registering memory
with the hardware. This means that memory must be physically resident
before the hardware can transmit that memory to another machine.
If this is not acceptable for your application or product, then the use
of RDMA migration may in fact be harmful to co-located VMs or other
software on the machine if there is not sufficient memory available to
relocate the entire footprint of the virtual machine. If so, then the
use of RDMA is discouraged and it is recommended to use standard TCP migration.

Experimental: Next, decide if you want dynamic page registration.
For example, if you have an 8GB RAM virtual machine, but only 1GB
is in active use, then enabling this feature will cause all 8GB to
be pinned and resident in memory. This feature mostly affects the
bulk-phase round of the migration and can be enabled for extremely
high-performance RDMA hardware using the following command:

QEMU Monitor Command:
$ migrate_set_capability rdma-pin-all on # disabled by default

Performing this action will cause all 8GB to be pinned, so if that's
not what you want, then please ignore this step altogether.

On the other hand, this will also significantly speed up the bulk round
of the migration, which can greatly reduce the "total" time of your migration.
Example performance of this using an idle VM in the previous example
can be found in the "Performance" section.

Note: for very large virtual machines (hundreds of GBs), pinning all
all of the memory of your virtual machine in the kernel is very expensive
may extend the initial bulk iteration time by many seconds,
and thus extending the total migration time. However, this will not
affect the determinism or predictability of your migration you will
still gain from the benefits of advanced pinning with RDMA.

RUNNING:

First, set the migration speed to match your hardware's capabilities:

QEMU Monitor Command:
$ migrate_set_parameter max_bandwidth 40g # or whatever is the MAX of your RDMA device

Next, on the destination machine, add the following to the QEMU command line:

qemu ..... -incoming rdma:host:port

Finally, perform the actual migration on the source machine:

QEMU Monitor Command:
$ migrate -d rdma:host:port

PERFORMANCE

Here is a brief summary of total migration time and downtime using RDMA:
Using a 40gbps infiniband link performing a worst-case stress test,
using an 8GB RAM virtual machine:

Using the following command:
$ apt-get install stress
$ stress --vm-bytes 7500M --vm 1 --vm-keep

1. Migration throughput: 26 gigabits/second.
2. Downtime (stop time) varies between 15 and 100 milliseconds.

EFFECTS of memory registration on bulk phase round:

For example, in the same 8GB RAM example with all 8GB of memory in
active use and the VM itself is completely idle using the same 40 gbps
infiniband link:

1. rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps
2. rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps

These numbers would of course scale up to whatever size virtual machine
you have to migrate using RDMA.

Enabling this feature does not have any measurable affect on
migration downtime. This is because, without this feature, all of the
memory will have already been registered already in advance during
the bulk round and does not need to be re-registered during the successive
iteration rounds.

RDMA Protocol Description:

Migration with RDMA is separated into two parts:

1. The transmission of the pages using RDMA
2. Everything else (a control channel is introduced)

"Everything else" is transmitted using a formal
protocol now, consisting of infiniband SEND messages.

An infiniband SEND message is the standard ibverbs
message used by applications of infiniband hardware.
The only difference between a SEND message and an RDMA
message is that SEND messages cause notifications
to be posted to the completion queue (CQ) on the
infiniband receiver side, whereas RDMA messages (used
for VM's ram) do not (to behave like an actual DMA).

Messages in infiniband require two things:

1. registration of the memory that will be transmitted
2. (SEND only) work requests to be posted on both
   sides of the network before the actual transmission
   can occur.

RDMA messages are much easier to deal with. Once the memory
on the receiver side is registered and pinned, we're
basically done. All that is required is for the sender
side to start dumping bytes onto the link.

(Memory is not released from pinning until the migration
completes, given that RDMA migrations are very fast.)

SEND messages require more coordination because the
receiver must have reserved space (using a receive
work request) on the receive queue (RQ) before QEMUFileRDMA
can start using them to carry all the bytes as
a control transport for migration of device state.

To begin the migration, the initial connection setup is
as follows (migration-rdma.c):

1. Receiver and Sender are started (command line or libvirt):
2. Both sides post two RQ work requests
3. Receiver does listen()
4. Sender does connect()
5. Receiver accept()
6. Check versioning and capabilities (described later)

At this point, we define a control channel on top of SEND messages
which is described by a formal protocol. Each SEND message has a
header portion and a data portion (but together are transmitted
as a single SEND message).

Header:
* Length (of the data portion, uint32, network byte order)
* Type (what command to perform, uint32, network byte order)
* Repeat (Number of commands in data portion, same type only)

The 'Repeat' field is here to support future multiple page registrations
in a single message without any need to change the protocol itself
so that the protocol is compatible against multiple versions of QEMU.
Version #1 requires that all server implementations of the protocol must
check this field and register all requests found in the array of commands located
in the data portion and return an equal number of results in the response.
The maximum number of repeats is hard-coded to 4096. This is a conservative
limit based on the maximum size of a SEND message along with empirical
observations on the maximum future benefit of simultaneous page registrations.

The 'type' field has 12 different command values:
1. Unused
2. Error (sent to the source during bad things)
3. Ready (control-channel is available)
4. QEMU File (for sending non-live device state)
5. RAM Blocks request (used right after connection setup)
6. RAM Blocks result (used right after connection setup)
7. Compress page (zap zero page and skip registration)
8. Register request (dynamic chunk registration)
9. Register result ('rkey' to be used by sender)
10. Register finished (registration for current iteration finished)
11. Unregister request (unpin previously registered memory)
12. Unregister finished (confirmation that unpin completed)

A single control message, as hinted above, can contain within the data
portion an array of many commands of the same type. If there is more than
one command, then the 'repeat' field will be greater than 1.

After connection setup, message 5 & 6 are used to exchange ram block
information and optionally pin all the memory if requested by the user.

After ram block exchange is completed, we have two protocol-level
functions, responsible for communicating control-channel commands
using the above list of values:

Logically:

qemu_rdma_exchange_recv(header, expected command type)

1. We transmit a READY command to let the sender know that
   we are ready to receive some data bytes on the control channel.
2. Before attempting to receive the expected command, we post another
   RQ work request to replace the one we just used up.
3. Block on a CQ event channel and wait for the SEND to arrive.
4. When the send arrives, librdmacm will unblock us.
5. Verify that the command-type and version received matches the one we expected.

qemu_rdma_exchange_send(header, data, optional response header & data):

1. Block on the CQ event channel waiting for a READY command
   from the receiver to tell us that the receiver
   is ready for us to transmit some new bytes.
2. Optionally: if we are expecting a response from the command
   (that we have not yet transmitted), let's post an RQ
   work request to receive that data a few moments later.
3. When the READY arrives, librdmacm will
   unblock us and we immediately post a RQ work request
   to replace the one we just used up.
4. Now, we can actually post the work request to SEND
   the requested command type of the header we were asked for.
5. Optionally, if we are expecting a response (as before),
   we block again and wait for that response using the additional
   work request we previously posted. (This is used to carry
   'Register result' commands #6 back to the sender which
   hold the rkey need to perform RDMA. Note that the virtual address
   corresponding to this rkey was already exchanged at the beginning
   of the connection (described below).

All of the remaining command types (not including 'ready')
described above all use the aforementioned two functions to do the hard work:

1. After connection setup, RAMBlock information is exchanged using
   this protocol before the actual migration begins. This information includes
   a description of each RAMBlock on the server side as well as the virtual addresses
   and lengths of each RAMBlock. This is used by the client to determine the
   start and stop locations of chunks and how to register them dynamically
   before performing the RDMA operations.
2. During runtime, once a 'chunk' becomes full of pages ready to
   be sent with RDMA, the registration commands are used to ask the
   other side to register the memory for this chunk and respond
   with the result (rkey) of the registration.
3. Also, the QEMUFile interfaces also call these functions (described below)
   when transmitting non-live state, such as devices or to send
   its own protocol information during the migration process.
4. Finally, zero pages are only checked if a page has not yet been registered
   using chunk registration (or not checked at all and unconditionally
   written if chunk registration is disabled. This is accomplished using
   the "Compress" command listed above. If the page has been registered
   then we check the entire chunk for zero. Only if the entire chunk is
   zero, then we send a compress command to zap the page on the other side.

Versioning and Capabilities

Current version of the protocol is version #1.

The same version applies to both for protocol traffic and capabilities
negotiation. (i.e. There is only one version number that is referred to
by all communication).

librdmacm provides the user with a 'private data' area to be exchanged
at connection-setup time before any infiniband traffic is generated.

Header:
* Version (protocol version validated before send/recv occurs),
uint32, network byte order
* Flags (bitwise OR of each capability),
uint32, network byte order

There is no data portion of this header right now, so there is
no length field. The maximum size of the 'private data' section
is only 192 bytes per the Infiniband specification, so it's not
very useful for data anyway. This structure needs to remain small.

This private data area is a convenient place to check for protocol
versioning because the user does not need to register memory to
transmit a few bytes of version information.

This is also a convenient place to negotiate capabilities
(like dynamic page registration).

If the version is invalid, we throw an error.

If the version is new, we only negotiate the capabilities that the
requested version is able to perform and ignore the rest.

Currently there is only one capability in Version #1: dynamic page registration

Finally: Negotiation happens with the Flags field: If the primary-VM
sets a flag, but the destination does not support this capability, it
will return a zero-bit for that flag and the primary-VM will understand
that as not being an available capability and will thus disable that
capability on the primary-VM side.

QEMUFileRDMA Interface:

QEMUFileRDMA introduces a couple of new functions:

1. qemu_rdma_get_buffer() (QEMUFileOps rdma_read_ops)
2. qemu_rdma_put_buffer() (QEMUFileOps rdma_write_ops)

These two functions are very short and simply use the protocol
describe above to deliver bytes without changing the upper-level
users of QEMUFile that depend on a bytestream abstraction.

Finally, how do we handoff the actual bytes to get_buffer()?

Again, because we're trying to "fake" a bytestream abstraction
using an analogy not unlike individual UDP frames, we have
to hold on to the bytes received from control-channel's SEND
messages in memory.

Each time we receive a complete "QEMU File" control-channel
message, the bytes from SEND are copied into a small local holding area.

Then, we return the number of bytes requested by get_buffer()
and leave the remaining bytes in the holding area until get_buffer()
comes around for another pass.

If the buffer is empty, then we follow the same steps
listed above and issue another "QEMU File" protocol command,
asking for a new SEND message to re-fill the buffer.

Migration of VM's ram:

At the beginning of the migration, (migration-rdma.c),
the sender and the receiver populate the list of RAMBlocks
to be registered with each other into a structure.
Then, using the aforementioned protocol, they exchange a
description of these blocks with each other, to be used later
during the iteration of main memory. This description includes
a list of all the RAMBlocks, their offsets and lengths, virtual
addresses and possibly includes pre-registered RDMA keys in case dynamic
page registration was disabled on the server-side, otherwise not.

Main memory is not migrated with the aforementioned protocol,
but is instead migrated with normal RDMA Write operations.

Pages are migrated in "chunks" (hard-coded to 1 Megabyte right now).
Chunk size is not dynamic, but it could be in a future implementation.
There's nothing to indicate that this is useful right now.

When a chunk is full (or a flush() occurs), the memory backed by
the chunk is registered with librdmacm is pinned in memory on
both sides using the aforementioned protocol.
After pinning, an RDMA Write is generated and transmitted
for the entire chunk.

Chunks are also transmitted in batches: This means that we
do not request that the hardware signal the completion queue
for the completion of every chunk. The current batch size
is about 64 chunks (corresponding to 64 MB of memory).
Only the last chunk in a batch must be signaled.
This helps keep everything as asynchronous as possible
and helps keep the hardware busy performing RDMA operations.

Error-handling:

Infiniband has what is called a "Reliable, Connected"
link (one of 4 choices). This is the mode in which
we use for RDMA migration.

If a single message fails,
the decision is to abort the migration entirely and
cleanup all the RDMA descriptors and unregister all
the memory.

After cleanup, the Virtual Machine is returned to normal
operation the same way that would happen if the TCP
socket is broken during a non-RDMA based migration.

TODO:

1. Currently, 'ulimit -l' mlock() limits as well as cgroups swap limits
   are not compatible with infiniband memory pinning and will result in
   an aborted migration (but with the source VM left unaffected).
2. Use of the recent /proc//pagemap would likely speed up
   the use of KSM and ballooning while using RDMA.
3. Also, some form of balloon-device usage tracking would also
   help alleviate some issues.
4. Use LRU to provide more fine-grained direction of UNREGISTER
   requests for unpinning memory in an overcommitted environment.
5. Expose UNREGISTER support to the user by way of workload-specific
   hints about application behavior.

Spice Port Fqdn (`docs/spice-port-fqdn.txt`) Copy Section

A Spice port channel is an arbitrary communication between the Spice
server host side and the client side.

Thanks to the associated reverse fully qualified domain name (fqdn),
a Spice client can handle the various ports appropriately.

The following fqdn names are reserved by the QEMU project:

org.qemu.monitor.hmp.0
QEMU human monitor

org.qemu.monitor.qmp.0:
QEMU control monitor

org.qemu.console.serial.0
QEMU virtual serial port

org.qemu.console.debug.0
QEMU debug console

Throttle (`docs/throttle.txt`) Copy Section

The QEMU throttling infrastructure

Copyright (C) 2016,2020 Igalia, S.L.
Author: Alberto Garcia

This work is licensed under the terms of the GNU GPL, version 2 or
later. See the COPYING file in the top-level directory.

Introduction Copy Chapter

QEMU includes a throttling module that can be used to set limits to
I/O operations. The code itself is generic and independent of the I/O
units, but it is currently used to limit the number of bytes per second
and operations per second (IOPS) when performing disk I/O.

This document explains how to use the throttling code in QEMU, and how
it works internally. The implementation is in throttle.c.

Using throttling to limit disk I/O Copy Chapter

Two aspects of the disk I/O can be limited: the number of bytes per
second and the number of operations per second (IOPS). For each one of
them the user can set a global limit or separate limits for read and
write operations. This gives us a total of six different parameters.

I/O limits can be set using the throttling.* parameters of -drive, or
using the QMP 'block_set_io_throttle' command. These are the names of
the parameters for both cases:

|-----------------------+-----------------------|
| -drive | block_set_io_throttle |
|-----------------------+-----------------------|
| throttling.iops-total | iops |
| throttling.iops-read | iops_rd |
| throttling.iops-write | iops_wr |
| throttling.bps-total | bps |
| throttling.bps-read | bps_rd |
| throttling.bps-write | bps_wr |
|-----------------------+-----------------------|

It is possible to set limits for both IOPS and bps at the same time,
and for each case we can decide whether to have separate read and
write limits or not, but note that if iops-total is set then neither
iops-read nor iops-write can be set. The same applies to bps-total and
bps-read/write.

The default value of these parameters is 0, and it means 'unlimited'.

In its most basic usage, the user can add a drive to QEMU with a limit
of 100 IOPS with the following -drive line:

-drive file=hd0.qcow2,throttling.iops-total=100

We can do the same using QMP. In this case all these parameters are
mandatory, so we must set to 0 the ones that we don't want to limit:

{ "execute": "block_set_io_throttle",
"arguments": {
"device": "virtio0",
"iops": 100,
"iops_rd": 0,
"iops_wr": 0,
"bps": 0,
"bps_rd": 0,
"bps_wr": 0
}
}

I/O bursts Copy Chapter

In addition to the basic limits we have just seen, QEMU allows the
user to do bursts of I/O for a configurable amount of time. A burst is
an amount of I/O that can exceed the basic limit. Bursts are useful to
allow better performance when there are peaks of activity (the OS
boots, a service needs to be restarted) while keeping the average
limits lower the rest of the time.

Two parameters control bursts: their length and the maximum amount of
I/O they allow. These two can be configured separately for each one of
the six basic parameters described in the previous section, but in
this section we'll use 'iops-total' as an example.

The I/O limit during bursts is set using 'iops-total-max', and the
maximum length (in seconds) is set with 'iops-total-max-length'. So if
we want to configure a drive with a basic limit of 100 IOPS and allow
bursts of 2000 IOPS for 60 seconds, we would do it like this (the line
is split for clarity):

-drive file=hd0.qcow2,
throttling.iops-total=100,
throttling.iops-total-max=2000,
throttling.iops-total-max-length=60

Or, with QMP:

{ "execute": "block_set_io_throttle",
"arguments": {
"device": "virtio0",
"iops": 100,
"iops_rd": 0,
"iops_wr": 0,
"bps": 0,
"bps_rd": 0,
"bps_wr": 0,
"iops_max": 2000,
"iops_max_length": 60,
}
}

With this, the user can perform I/O on hd0.qcow2 at a rate of 2000
IOPS for 1 minute before it's throttled down to 100 IOPS.

The user will be able to do bursts again if there's a sufficiently
long period of time with unused I/O (see below for details).

The default value for 'iops-total-max' is 0 and it means that bursts
are not allowed. 'iops-total-max-length' can only be set if
'iops-total-max' is set as well, and its default value is 1 second.

Here's the complete list of parameters for configuring bursts:

|----------------------------------+-----------------------|
| -drive | block_set_io_throttle |
|----------------------------------+-----------------------|
| throttling.iops-total-max | iops_max |
| throttling.iops-total-max-length | iops_max_length |
| throttling.iops-read-max | iops_rd_max |
| throttling.iops-read-max-length | iops_rd_max_length |
| throttling.iops-write-max | iops_wr_max |
| throttling.iops-write-max-length | iops_wr_max_length |
| throttling.bps-total-max | bps_max |
| throttling.bps-total-max-length | bps_max_length |
| throttling.bps-read-max | bps_rd_max |
| throttling.bps-read-max-length | bps_rd_max_length |
| throttling.bps-write-max | bps_wr_max |
| throttling.bps-write-max-length | bps_wr_max_length |
|----------------------------------+-----------------------|

Controlling the size of I/O operations Copy Chapter

When applying IOPS limits all I/O operations are treated equally
regardless of their size. This means that the user can take advantage
of this in order to circumvent the limits and submit one huge I/O
request instead of several smaller ones.

QEMU provides a setting called throttling.iops-size to prevent this
from happening. This setting specifies the size (in bytes) of an I/O
request for accounting purposes. Larger requests will be counted
proportionally to this size.

For example, if iops-size is set to 4096 then an 8KB request will be
counted as two, and a 6KB request will be counted as one and a
half. This only applies to requests larger than iops-size: smaller
requests will be always counted as one, no matter their size.

The default value of iops-size is 0 and it means that the size of the
requests is never taken into account when applying IOPS limits.

Applying I/O limits to groups of disks Copy Chapter

In all the examples so far we have seen how to apply limits to the I/O
performed on individual drives, but QEMU allows grouping drives so
they all share the same limits.

The way it works is that each drive with I/O limits is assigned to a
group named using the throttling.group parameter. If this parameter is
not specified, then the device name (i.e. 'virtio0', 'ide0-hd0') will
be used as the group name.

Limits set using the throttling.* parameters discussed earlier in this
document apply to the combined I/O of all members of a group.

Consider this example:

-drive file=hd1.qcow2,throttling.iops-total=6000,throttling.group=foo
-drive file=hd2.qcow2,throttling.iops-total=6000,throttling.group=foo
-drive file=hd3.qcow2,throttling.iops-total=3000,throttling.group=bar
-drive file=hd4.qcow2,throttling.iops-total=6000,throttling.group=foo
-drive file=hd5.qcow2,throttling.iops-total=3000,throttling.group=bar
-drive file=hd6.qcow2,throttling.iops-total=5000

Here hd1, hd2 and hd4 are all members of a group named 'foo' with a
combined IOPS limit of 6000, and hd3 and hd5 are members of 'bar'. hd6
is left alone (technically it is part of a 1-member group).

Limits are applied in a round-robin fashion so if there are concurrent
I/O requests on several drives of the same group they will be
distributed evenly.

When I/O limits are applied to an existing drive using the QMP command
'block_set_io_throttle', the following things need to be taken into
account:

• I/O limits are shared within the same group, so new values will
  affect all members and overwrite the previous settings. In other
  words: if different limits are applied to members of the same
  group, the last one wins.

• If 'group' is unset it is assumed to be the current group of that
  drive. If the drive is not in a group yet, it will be added to a
  group named after the device name.

• If 'group' is set then the drive will be moved to that group if
  it was member of a different one. In this case the limits
  specified in the parameters will be applied to the new group
  only.

• I/O limits can be disabled by setting all of them to 0. In this
  case the device will be removed from its group and the rest of
  its members will not be affected. The 'group' parameter is
  ignored.

The Leaky Bucket algorithm Copy Chapter

I/O limits in QEMU are implemented using the leaky bucket algorithm
(specifically the "Leaky bucket as a meter" variant).

This algorithm uses the analogy of a bucket that leaks water
constantly. The water that gets into the bucket represents the I/O
that has been performed, and no more I/O is allowed once the bucket is
full.

To see the way this corresponds to the throttling parameters in QEMU,
consider the following values:

iops-total=100
iops-total-max=2000
iops-total-max-length=60

• Water leaks from the bucket at a rate of 100 IOPS.
• Water can be added to the bucket at a rate of 2000 IOPS.
• The size of the bucket is 2000 x 60 = 120000
• If 'iops-total-max-length' is unset then it defaults to 1 and the
  size of the bucket is 2000.
• If 'iops-total-max' is unset then 'iops-total-max-length' must be
  unset as well. In this case the bucket size is 100.

The bucket is initially empty, therefore water can be added until it's
full at a rate of 2000 IOPS (the burst rate). Once the bucket is full
we can only add as much water as it leaks, therefore the I/O rate is
reduced to 100 IOPS. If we add less water than it leaks then the
bucket will start to empty, allowing for bursts again.

Note that since water is leaking from the bucket even during bursts,
it will take a bit more than 60 seconds at 2000 IOPS to fill it
up. After those 60 seconds the bucket will have leaked 60 x 100 =
6000, allowing for 3 more seconds of I/O at 2000 IOPS.

Also, due to the way the algorithm works, longer burst can be done at
a lower I/O rate, e.g. 1000 IOPS during 120 seconds.

The 'throttle' block filter Copy Chapter

Since QEMU 2.11 it is possible to configure the I/O limits using a
'throttle' block filter. This filter uses the exact same throttling
infrastructure described above but can be used anywhere in the node
graph, allowing for more flexibility.

The user can create an arbitrary number of filters and each one of
them must be assigned to a group that contains the actual I/O limits.
Different filters can use the same group so the limits are shared as
described earlier in "Applying I/O limits to groups of disks".

A group can be created using the object-add QMP function:

{ "execute": "object-add",
"arguments": {
"qom-type": "throttle-group",
"id": "group0",
"limits" : {
"iops-total": 1000,
"bps-write": 2097152
}
}
}

throttle-group has a 'limits' property (of type ThrottleLimits as
defined in qapi/block-core.json) which can be set on creation or later
with 'qom-set'.

A throttle-group can also be created with the -object command line
option but at the moment there is no way to pass a 'limits' parameter
that contains a ThrottleLimits structure. The solution is to set the
individual values directly, like in this example:

-object throttle-group,id=group0,x-iops-total=1000,x-bps-write=2097152

Note however that this is not a stable API (hence the 'x-' prefixes) and
will disappear when -object gains support for structured options and
enables use of 'limits'.

Once we have a throttle-group we can use the throttle block filter,
where the 'file' property must be set to the block device that we want
to filter:

{ "execute": "blockdev-add",
"arguments": {
"options": {
"driver": "qcow2",
"node-name": "disk0",
"file": {
"driver": "file",
"filename": "/path/to/disk.qcow2"
}
}
}
}

{ "execute": "blockdev-add",
"arguments": {
"driver": "throttle",
"node-name": "throttle0",
"throttle-group": "group0",
"file": "disk0"
}
}

A similar setup can also be done with the command line, for example:

-drive driver=throttle,throttle-group=group0,
file.driver=qcow2,file.file.filename=/path/to/disk.qcow2

The scenario described so far is very simple but the throttle block
filter allows for more complex configurations. For example, let's say
that we have three different drives and we want to set I/O limits for
each one of them and an additional set of limits for the combined I/O
of all three drives.

First we would define all throttle groups, one for each one of the
drives and one that would apply to all of them:

-object throttle-group,id=limits0,x-iops-total=2000
-object throttle-group,id=limits1,x-iops-total=2500
-object throttle-group,id=limits2,x-iops-total=3000
-object throttle-group,id=limits012,x-iops-total=4000

Now we can define the drives, and for each one of them we use two
chained throttle filters: the drive's own filter and the combined
filter.

-drive driver=throttle,throttle-group=limits012,
file.driver=throttle,file.throttle-group=limits0
file.file.driver=qcow2,file.file.file.filename=/path/to/disk0.qcow2
-drive driver=throttle,throttle-group=limits012,
file.driver=throttle,file.throttle-group=limits1
file.file.driver=qcow2,file.file.file.filename=/path/to/disk1.qcow2
-drive driver=throttle,throttle-group=limits012,
file.driver=throttle,file.throttle-group=limits2
file.file.driver=qcow2,file.file.file.filename=/path/to/disk2.qcow2

In this example the individual drives have IOPS limits of 2000, 2500
and 3000 respectively but the total combined I/O can never exceed 4000
IOPS.

U2f (`docs/u2f.txt`) Copy Section

QEMU U2F Key Device Documentation.

Contents

1. USB U2F key device

2. Building

3. Using u2f-emulated

4. Using u2f-passthru

5. Libu2f-emu

6. USB U2F key device

U2F is an open authentication standard that enables relying parties
exposed to the internet to offer a strong second factor option for end
user authentication.

The standard brings many advantages to both parties, client and server,
allowing to reduce over-reliance on passwords, it increases authentication
security and simplifies passwords.

The second factor is materialized by a device implementing the U2F
protocol. In case of a USB U2F security key, it is a USB HID device
that implements the U2F protocol.

In QEMU, the USB U2F key device offers a dedicated support of U2F, allowing
guest USB FIDO/U2F security keys operating in two possible modes:
pass-through and emulated.

The pass-through mode consists of passing all requests made from the guest
to the physical security key connected to the host machine and vice versa.
In addition, the dedicated pass-through allows to have a U2F security key
shared on several guests which is not possible with a simple host device
assignment pass-through.

The emulated mode consists of completely emulating the behavior of an
U2F device through software part. Libu2f-emu is used for that.

2. Building

To ensure the build of the u2f-emulated device variant which depends
on libu2f-emu: configuring and building:

./configure --enable-u2f && make

The pass-through mode is built by default on Linux. To take advantage
of the autoscan option it provides, make sure you have a working libudev
installed on the host.

3. Using u2f-emulated

To work, an emulated U2F device must have four elements:

• ec x509 certificate
• ec private key
• counter (four bytes value)
• 48 bytes of entropy (random bits)

To use this type of device, this one has to be configured, and these
four elements must be passed one way or another.

Assuming that you have a working libu2f-emu installed on the host.
There are three possible ways of configurations:

• ephemeral
• setup directory
• manual

Ephemeral is the simplest way to configure, it lets the device generate
all the elements it needs for a single use of the lifetime of the device.

qemu -usb -device u2f-emulated

Setup directory allows to configure the device from a directory containing
four files:

• certificate.pem: ec x509 certificate

• private-key.pem: ec private key

• counter: counter value

• entropy: 48 bytes of entropy

  qemu -usb -device u2f-emulated,dir=$dir

Manual allows to configure the device more finely by specifying each
of the elements necessary for the device:

• cert

• priv

• counter

• entropy

  qemu -usb -device u2f-emulated,cert=$DIR1/$FILE1,priv=$DIR2/$FILE2,counter=$DIR3/$FILE3,entropy=$DIR4/$FILE4

4. Using u2f-passthru

On the host specify the u2f-passthru device with a suitable hidraw:

qemu -usb -device u2f-passthru,hidraw=/dev/hidraw0

Alternately, the u2f-passthru device can autoscan to take the first
U2F device it finds on the host (this requires a working libudev):

qemu -usb -device u2f-passthru

5. Libu2f-emu

The u2f-emulated device uses libu2f-emu for the U2F key emulation. Libu2f-emu
implements completely the U2F protocol device part for all specified
transport given by the FIDO Alliance.

For more information about libu2f-emu see this page:
https://github.com/MattGorko/libu2f-emu.

Xbzrle (`docs/xbzrle.txt`) Copy Section

XBZRLE (Xor Based Zero Run Length Encoding)

Using XBZRLE (Xor Based Zero Run Length Encoding) allows for the reduction
of VM downtime and the total live-migration time of Virtual machines.
It is particularly useful for virtual machines running memory write intensive
workloads that are typical of large enterprise applications such as SAP ERP
Systems, and generally speaking for any application that uses a sparse memory
update pattern.

Instead of sending the changed guest memory page this solution will send a
compressed version of the updates, thus reducing the amount of data sent during
live migration.
In order to be able to calculate the update, the previous memory pages need to
be stored on the source. Those pages are stored in a dedicated cache
(hash table) and are accessed by their address.
The larger the cache size the better the chances are that the page has already
been stored in the cache.
A small cache size will result in high cache miss rate.
Cache size can be changed before and during migration.

Format

The compression format performs a XOR between the previous and current content
of the page, where zero represents an unchanged value.
The page data delta is represented by zero and non zero runs.
A zero run is represented by its length (in bytes).
A non zero run is represented by its length (in bytes) and the new data.
The run length is encoded using ULEB128 (http://en.wikipedia.org/wiki/LEB128)

There can be more than one valid encoding, the sender may send a longer encoding
for the benefit of reducing computation cost.

page = zrun nzrun
| zrun nzrun page

zrun = length

nzrun = length byte...

length = uleb128 encoded integer

On the sender side XBZRLE is used as a compact delta encoding of page updates,
retrieving the old page content from the cache (default size of 64MB). The
receiving side uses the existing page's content and XBZRLE to decode the new
page's content.

This work was originally based on research results published
VEE 2011: Evaluation of Delta Compression Techniques for Efficient Live
Migration of Large Virtual Machines by Benoit, Svard, Tordsson and Elmroth.
Additionally the delta encoder XBRLE was improved further using the XBZRLE
instead.

XBZRLE has a sustained bandwidth of 2-2.5 GB/s for typical workloads making it
ideal for in-line, real-time encoding such as is needed for live-migration.

Example
old buffer:
1001 zeros
05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 68 00 00 6b 00 6d
3074 zeros

new buffer:
1001 zeros
01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 68 00 00 67 00 69
3074 zeros

encoded buffer:

encoded length 24
e9 07 0f 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 03 01 67 01 01 69

Cache update strategy

Keeping the hot pages in the cache is effective for decreasing cache
misses. XBZRLE uses a counter as the age of each page. The counter will
increase after each ram dirty bitmap sync. When a cache conflict is
detected, XBZRLE will only evict pages in the cache that are older than
a threshold.

Usage

1. Verify the destination QEMU version is able to decode the new format.
   {qemu} info migrate_capabilities
   {qemu} xbzrle: off , ...

2. Activate xbzrle on both source and destination:
   {qemu} migrate_set_capability xbzrle on

3. Set the XBZRLE cache size - the cache size is in MBytes and should be a
   power of 2. The cache default value is 64MBytes. (on source only)
   {qemu} migrate_set_parameter xbzrle-cache-size 256m

4. Start outgoing migration
   {qemu} migrate -d tcp:destination.host:4444
   {qemu} info migrate
   capabilities: xbzrle: on
   Migration status: active
   transferred ram: A kbytes
   remaining ram: B kbytes
   total ram: C kbytes
   total time: D milliseconds
   duplicate: E pages
   normal: F pages
   normal bytes: G kbytes
   cache size: H bytes
   xbzrle transferred: I kbytes
   xbzrle pages: J pages
   xbzrle cache miss: K pages
   xbzrle cache miss rate: L
   xbzrle encoding rate: M
   xbzrle overflow: N

xbzrle cache miss: the number of cache misses to date - high cache-miss rate
indicates that the cache size is set too low.
xbzrle overflow: the number of overflows in the decoding which where the delta
could not be compressed. This can happen if the changes in the pages are too
large or there are many short changes; for example, changing every second byte
(half a page).

Testing: Testing indicated that live migration with XBZRLE was completed in 110
seconds, whereas without it would not be able to complete.

A simple synthetic memory r/w load generator:
.. include <stdlib.h>
.. include <stdio.h>
.. int main()
.. {
.. char *buf = (char *) calloc(4096, 4096);
.. while (1) {
.. int i;
.. for (i = 0; i < 4096 * 4; i++) {
.. buf[i * 4096 / 4]++;
.. }
.. printf(".");
.. }
.. }

Xen Save Devices State (`docs/xen-save-devices-state.txt`) Copy Section

= Save Devices =

QEMU has code to load/save the state of the guest that it is running.
These are two complementary operations. Saving the state just does
that, saves the state for each device that the guest is running.

These operations are normally used with migration (see migration.txt),
however it is also possible to save the state of all devices to file,
without saving the RAM or the block devices of the VM.

The save operation is available as QMP command xen-save-devices-state.

The binary format used in the file is the following:

32 bit big endian: QEMU_VM_FILE_MAGIC
32 bit big endian: QEMU_VM_FILE_VERSION

for_each_device
{
8 bit: QEMU_VM_SECTION_FULL
32 bit big endian: section_id
8 bit: idstr (ID string) length
string: idstr (ID string)
32 bit big endian: instance_id
32 bit big endian: version_id
buffer: device specific data
}

8 bit: QEMU_VM_EOF

About Build Platforms (`docs/about/build-platforms.rst`) Copy Section

.. _Supported-build-platforms:

Supported build platforms

QEMU aims to support building and executing on multiple host OS
platforms. This appendix outlines which platforms are the major build
targets. These platforms are used as the basis for deciding upon the
minimum required versions of 3rd party software QEMU depends on. The
supported platforms are the targets for automated testing performed by
the project when patches are submitted for review, and tested before and
after merge.

If a platform is not listed here, it does not imply that QEMU won't
work. If an unlisted platform has comparable software versions to a
listed platform, there is every expectation that it will work. Bug
reports are welcome for problems encountered on unlisted platforms
unless they are clearly older vintage than what is described here.

Note that when considering software versions shipped in distros as
support targets, QEMU considers only the version number, and assumes the
features in that distro match the upstream release with the same
version. In other words, if a distro backports extra features to the
software in their distro, QEMU upstream code will not add explicit
support for those backports, unless the feature is auto-detectable in a
manner that works for the upstream releases too.

The Repology_ site is a useful resource to identify
currently shipped versions of software in various operating systems,
though it does not cover all distros listed below.

Supported host architectures Copy Chapter

Those hosts are officially supported, with various accelerators:

.. list-table::
:header-rows: 1

• • CPU Architecture
  • Accelerators
• • Arm
  • kvm (64 bit only), tcg, xen
• • MIPS (little endian only)
  • kvm, tcg
• • PPC
  • kvm, tcg
• • RISC-V
  • kvm, tcg
• • s390x
  • kvm, tcg
• • SPARC
  • tcg
• • x86
  • hax, hvf (64 bit only), kvm, nvmm, tcg, whpx (64 bit only), xen

Other host architectures are not supported. It is possible to build QEMU system
emulation on an unsupported host architecture using the configure
--enable-tcg-interpreter option to enable the TCI support, but note that
this is very slow and is not recommended for normal use. QEMU user emulation
requires host-specific support for signal handling, therefore TCI won't help
on unsupported host architectures.

Non-supported architectures may be removed in the future following the
:ref:deprecation process<Deprecated features>.

Linux OS, macOS, FreeBSD, NetBSD, OpenBSD Copy Chapter

The project aims to support the most recent major version at all times. Support
for the previous major version will be dropped 2 years after the new major
version is released or when the vendor itself drops support, whichever comes
first. In this context, third-party efforts to extend the lifetime of a distro
are not considered, even when they are endorsed by the vendor (eg. Debian LTS);
the same is true of repositories that contain packages backported from later
releases (e.g. Debian backports). Within each major release, only the most
recent minor release is considered.

For the purposes of identifying supported software versions available on Linux,
the project will look at CentOS, Debian, Fedora, openSUSE, RHEL, SLES and
Ubuntu LTS. Other distros will be assumed to ship similar software versions.

For FreeBSD and OpenBSD, decisions will be made based on the contents of the
respective ports repository, while NetBSD will use the pkgsrc repository.

For macOS, Homebrew_ will be used, although MacPorts_ is expected to carry
similar versions.

Windows Copy Chapter

The project aims to support the two most recent versions of Windows that are
still supported by the vendor. The minimum Windows API that is currently
targeted is "Windows 8", so theoretically the QEMU binaries can still be run
on older versions of Windows, too. However, such old versions of Windows are
not tested anymore, so it is recommended to use one of the latest versions of
Windows instead.

The project supports building QEMU with current versions of the MinGW
toolchain, either hosted on Linux (Debian/Fedora) or via MSYS2_ on Windows.
A more recent Windows version is always preferred as it is less likely to have
problems with building via MSYS2. The building process of QEMU involves some
Python scripts that call os.symlink() which needs special attention for the
build process to successfully complete. On newer versions of Windows 10,
unprivileged accounts can create symlinks if Developer Mode is enabled.
When Developer Mode is not available/enabled, the SeCreateSymbolicLinkPrivilege
privilege is required, or the process must be run as an administrator.

.. _Homebrew: https://brew.sh/
.. _MacPorts: https://www.macports.org/
.. _MSYS2: https://www.msys2.org/
.. _Repology: https://repology.org/

About Deprecated (`docs/about/deprecated.rst`) Copy Section

.. _Deprecated features:

Deprecated features

In general features are intended to be supported indefinitely once
introduced into QEMU. In the event that a feature needs to be removed,
it will be listed in this section. The feature will remain functional for the
release in which it was deprecated and one further release. After these two
releases, the feature is liable to be removed. Deprecated features may also
generate warnings on the console when QEMU starts up, or if activated via a
monitor command, however, this is not a mandatory requirement.

Prior to the 2.10.0 release there was no official policy on how
long features would be deprecated prior to their removal, nor
any documented list of which features were deprecated. Thus
any features deprecated prior to 2.10.0 will be treated as if
they were first deprecated in the 2.10.0 release.

What follows is a list of all features currently marked as
deprecated.

System emulator command line arguments Copy Chapter

QEMU_AUDIO_ environment variables and -audio-help (since 4.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The -audiodev argument is now the preferred way to specify audio
backend settings instead of environment variables. To ease migration to
the new format, the -audiodev-help option can be used to convert
the current values of the environment variables to -audiodev options.

Creating sound card devices and vnc without audiodev= property (since 4.2)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

When not using the deprecated legacy audio config, each sound card
should specify an audiodev= property. Additionally, when using
vnc, you should specify an audiodev= property if you plan to
transmit audio through the VNC protocol.

-chardev backend aliases tty and parport (since 6.0)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

tty and parport are aliases that will be removed. Instead, the
actual backend names serial and parallel should be used.

Short-form boolean options (since 6.0)
''''''''''''''''''''''''''''''''''''''

Boolean options such as share=on/share=off could be written
in short form as share and noshare. This is now deprecated
and will cause a warning.

delay option for socket character devices (since 6.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The replacement for the nodelay short-form boolean option is nodelay=on
rather than delay=off.

Userspace local APIC with KVM (x86, since 6.0)
''''''''''''''''''''''''''''''''''''''''''''''

Using -M kernel-irqchip=off with x86 machine types that include a local
APIC is deprecated. The split setting is supported, as is using
-M kernel-irqchip=off with the ISA PC machine type.

hexadecimal sizes with scaling multipliers (since 6.0)
''''''''''''''''''''''''''''''''''''''''''''''''''''''

Input parameters that take a size value should only use a size suffix
(such as 'k' or 'M') when the base is written in decimal, and not when
the value is hexadecimal. That is, '0x20M' is deprecated, and should
be written either as '32M' or as '0x2000000'.

-spice password=string (since 6.0)
''''''''''''''''''''''''''''''''''''''

This option is insecure because the SPICE password remains visible in
the process listing. This is replaced by the new password-secret
option which lets the password be securely provided on the command
line using a secret object instance.

-smp ("parameter=0" SMP configurations) (since 6.2)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''

Specified CPU topology parameters must be greater than zero.

In the SMP configuration, users should either provide a CPU topology
parameter with a reasonable value (greater than zero) or just omit it
and QEMU will compute the missing value.

However, historically it was implicitly allowed for users to provide
a parameter with zero value, which is meaningless and could also possibly
cause unexpected results in the -smp parsing. So support for this kind of
configurations (e.g. -smp 8,sockets=0) is deprecated since 6.2 and will
be removed in the near future, users have to ensure that all the topology
members described with -smp are greater than zero.

Plugin argument passing through arg=<string> (since 6.1)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Passing TCG plugins arguments through arg= is redundant is makes the
command-line less readable, especially when the argument itself consist of a
name and a value, e.g. -plugin plugin_name,arg="arg_name=arg_value".
Therefore, the usage of arg is redundant. Single-word arguments are treated
as short-form boolean values, and passed to plugins as arg_name=on.
However, short-form booleans are deprecated and full explicit arg_name=on
form is preferred.

-drive if=none for the sifive_u OTP device (since 6.2)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Using -drive if=none to configure the OTP device of the sifive_u
RISC-V machine is deprecated. Use -drive if=pflash instead.

QEMU Machine Protocol (QMP) commands Copy Chapter

blockdev-open-tray, blockdev-close-tray argument device (since 2.8)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Use argument id instead.

eject argument device (since 2.8)
'''''''''''''''''''''''''''''''''''''''''

Use argument id instead.

blockdev-change-medium argument device (since 2.8)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Use argument id instead.

block_set_io_throttle argument device (since 2.8)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Use argument id instead.

blockdev-add empty string argument backing (since 2.10)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Use argument value null instead.

block-commit arguments base and top (since 3.1)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Use arguments base-node and top-node instead.

nbd-server-add and nbd-server-remove (since 5.2)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Use the more generic commands block-export-add and block-export-del
instead. As part of this deprecation, where nbd-server-add used a
single bitmap, the new block-export-add uses a list of bitmaps.

query-qmp-schema return value member values (since 6.2)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Member values in return value elements with meta-type enum is
deprecated. Use members instead.

drive-backup (since 6.2)
''''''''''''''''''''''''''''

Use blockdev-backup in combination with blockdev-add instead.
This change primarily separates the creation/opening process of the backup
target with explicit, separate steps. blockdev-backup uses mostly the
same arguments as drive-backup, except the format and mode
options are removed in favor of using explicit blockdev-create and
blockdev-add calls. See :doc:/interop/live-block-operations for
details.

Incorrectly typed device_add arguments (since 6.2)
''''''''''''''''''''''''''''''''''''''''''''''''''''''

Due to shortcomings in the internal implementation of device_add, QEMU
incorrectly accepts certain invalid arguments: Any object or list arguments are
silently ignored. Other argument types are not checked, but an implicit
conversion happens, so that e.g. string values can be assigned to integer
device properties or vice versa.

This is a bug in QEMU that will be fixed in the future so that previously
accepted incorrect commands will return an error. Users should make sure that
all arguments passed to device_add are consistent with the documented
property types.

query-sgx return value member section-size (since 7.0)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Member section-size in return value elements with meta-type uint64 is
deprecated. Use sections instead.

query-sgx-capabilities return value member section-size (since 7.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Member section-size in return value elements with meta-type uint64 is
deprecated. Use sections instead.

System accelerators Copy Chapter

MIPS Trap-and-Emul KVM support (since 6.0)
''''''''''''''''''''''''''''''''''''''''''''''

The MIPS Trap-and-Emul KVM host and guest support has been removed
from Linux upstream kernel, declare it deprecated.

Host Architectures Copy Chapter

BE MIPS (since 7.2)
'''''''''''''''''''

As Debian 10 ("Buster") moved into LTS the big endian 32 bit version of
MIPS moved out of support making it hard to maintain our
cross-compilation CI tests of the architecture. As we no longer have
CI coverage support may bitrot away before the deprecation process
completes. The little endian variants of MIPS (both 32 and 64 bit) are
still a supported host architecture.

QEMU API (QAPI) events Copy Chapter

MEM_UNPLUG_ERROR (since 6.2)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Use the more generic event DEVICE_UNPLUG_GUEST_ERROR instead.

System emulator machines Copy Chapter

Arm virt machine dtb-kaslr-seed property
''''''''''''''''''''''''''''''''''''''''''''''''

The dtb-kaslr-seed property on the virt board has been
deprecated; use the new name dtb-randomness instead. The new name
better reflects the way this property affects all random data within
the device tree blob, not just the kaslr-seed node.

pc-i440fx-1.4 up to pc-i440fx-1.7 (since 7.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''

These old machine types are quite neglected nowadays and thus might have
various pitfalls with regards to live migration. Use a newer machine type
instead.

Backend options Copy Chapter

Using non-persistent backing file with pmem=on (since 6.1)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

This option is used when memory-backend-file is consumed by emulated NVDIMM
device. However enabling memory-backend-file.pmem option, when backing file
is (a) not DAX capable or (b) not on a filesystem that support direct mapping
of persistent memory, is not safe and may lead to data loss or corruption in case
of host crash.
Options are:

- modify VM configuration to set ``pmem=off`` to continue using fake NVDIMM
  (without persistence guaranties) with backing file on non DAX storage
- move backing file to NVDIMM storage and keep ``pmem=on``
  (to have NVDIMM with persistence guaranties).

Device options Copy Chapter

Emulated device options
'''''''''''''''''''''''

-device virtio-blk,scsi=on|off (since 5.0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The virtio-blk SCSI passthrough feature is a legacy VIRTIO feature. VIRTIO 1.0
and later do not support it because the virtio-scsi device was introduced for
full SCSI support. Use virtio-scsi instead when SCSI passthrough is required.

Note this also applies to -device virtio-blk-pci,scsi=on|off, which is an
alias.

-device sga (since 6.2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^

The sga device loads an option ROM for x86 targets which enables
SeaBIOS to send messages to the serial console. SeaBIOS 1.11.0 onwards
contains native support for this feature and thus use of the option
ROM approach is obsolete. The native SeaBIOS support can be activated
by using -machine graphics=off.

-device nvme-ns,eui64-default=on|off (since 7.1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In QEMU versions 6.1, 6.2 and 7.0, the nvme-ns generates an EUI-64
identifier that is not globally unique. If an EUI-64 identifier is required, the
user must set it explicitly using the nvme-ns device parameter eui64.

-device nvme,use-intel-id=on|off (since 7.1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The nvme device originally used a PCI Vendor/Device Identifier combination
from Intel that was not properly allocated. Since version 5.2, the controller
has used a properly allocated identifier. Deprecate the use-intel-id
machine compatibility parameter.

Block device options
''''''''''''''''''''

"backing": "" (since 2.12)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In order to prevent QEMU from automatically opening an image's backing
chain, use "backing": null instead.

rbd keyvalue pair encoded filenames: "" (since 3.1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Options for rbd should be specified according to its runtime options,
like other block drivers. Legacy parsing of keyvalue pair encoded
filenames is useful to open images with the old format for backing files;
These image files should be updated to use the current format.

Example of legacy encoding::

json:{"file.driver":"rbd", "file.filename":"rbd:rbd/name"}

The above, converted to the current supported format::

json:{"file.driver":"rbd", "file.pool":"rbd", "file.image":"name"}

Backwards compatibility Copy Chapter

Runnability guarantee of CPU models (since 4.1)
'''''''''''''''''''''''''''''''''''''''''''''''

Previous versions of QEMU never changed existing CPU models in
ways that introduced additional host software or hardware
requirements to the VM. This allowed management software to
safely change the machine type of an existing VM without
introducing new requirements ("runnability guarantee"). This
prevented CPU models from being updated to include CPU
vulnerability mitigations, leaving guests vulnerable in the
default configuration.

The CPU model runnability guarantee won't apply anymore to
existing CPU models. Management software that needs runnability
guarantees must resolve the CPU model aliases using the
alias-of field returned by the query-cpu-definitions QMP
command.

While those guarantees are kept, the return value of
query-cpu-definitions will have existing CPU model aliases
point to a version that doesn't break runnability guarantees
(specifically, version 1 of those CPU models). In future QEMU
versions, aliases will point to newer CPU model versions
depending on the machine type, so management software must
resolve CPU model aliases before starting a virtual machine.

Tools Copy Chapter

virtiofsd
'''''''''

There is a new Rust implementation of virtiofsd at
https://gitlab.com/virtio-fs/virtiofsd;
since this is now marked stable, new development should be done on that
rather than the existing C version in the QEMU tree.
The C version will still accept fixes and patches that
are already in development for the moment, but will eventually
be deleted from this tree.
New deployments should use the Rust version, and existing systems
should consider moving to it. The command line and feature set
is very close and moving should be simple.

QEMU guest agent Copy Chapter

--blacklist command line option (since 7.2)
'''''''''''''''''''''''''''''''''''''''''''''''

--blacklist has been replaced by --block-rpcs (which is a better
wording for what this option does). The short form -b still stays
the same and thus is the preferred way for scripts that should run with
both, older and future versions of QEMU.

blacklist config file option (since 7.2)
''''''''''''''''''''''''''''''''''''''''''''

The blacklist config file option has been renamed to block-rpcs
(to be in sync with the renaming of the corresponding command line
option).

About Index (`docs/about/index.rst`) Copy Section

About QEMU Copy Chapter

QEMU is a generic and open source machine emulator and virtualizer.

QEMU can be used in several different ways. The most common is for
"system emulation", where it provides a virtual model of an
entire machine (CPU, memory and emulated devices) to run a guest OS.
In this mode the CPU may be fully emulated, or it may work with
a hypervisor such as KVM, Xen, Hax or Hypervisor.Framework to
allow the guest to run directly on the host CPU.

The second supported way to use QEMU is "user mode emulation",
where QEMU can launch processes compiled for one CPU on another CPU.
In this mode the CPU is always emulated.

QEMU also provides a number of standalone commandline utilities,
such as the qemu-img disk image utility that allows you to create,
convert and modify disk images.

.. toctree::
:maxdepth: 2

build-platforms
deprecated
removed-features
license

About License (`docs/about/license.rst`) Copy Section

.. _License:

License

QEMU is a trademark of Fabrice Bellard.

QEMU is released under the GNU General Public License <https://www.gnu.org/licenses/gpl-2.0.txt>, version 2. Parts
of QEMU have specific licenses, see file
LICENSE <https://git.qemu.org/?p=qemu.git;a=blob_plain;f=LICENSE>.

About Removed Features (`docs/about/removed-features.rst`) Copy Section

Removed features

What follows is a record of recently removed, formerly deprecated
features that serves as a record for users who have encountered
trouble after a recent upgrade.

System emulator command line arguments Copy Chapter

-hdachs (removed in 2.12)
'''''''''''''''''''''''''''''

The geometry defined by -hdachs c,h,s,t should now be specified via
-device ide-hd,drive=dr,cyls=c,heads=h,secs=s,bios-chs-trans=t
(together with -drive if=none,id=dr,...).

-net channel (removed in 2.12)
''''''''''''''''''''''''''''''''''

This option has been replaced by -net user,guestfwd=....

-net dump (removed in 2.12)
'''''''''''''''''''''''''''''''

-net dump[,vlan=n][,file=filename][,len=maxlen] has been replaced by
-object filter-dump,id=id,netdev=dev[,file=filename][,maxlen=maxlen].
Note that the new syntax works with netdev IDs instead of the old "vlan" hubs.

-no-kvm-pit (removed in 2.12)
'''''''''''''''''''''''''''''''''

This was just a dummy option that has been ignored, since the in-kernel PIT
cannot be disabled separately from the irqchip anymore. A similar effect
(which also disables the KVM IOAPIC) can be obtained with
-M kernel_irqchip=split.

-tdf (removed in 2.12)
''''''''''''''''''''''''''

There is no replacement, the -tdf option has just been ignored since the
behaviour that could be changed by this option in qemu-kvm is now the default
when using the KVM PIT. It still can be requested explicitly using
-global kvm-pit.lost_tick_policy=delay.

-drive secs=s, -drive heads=h & -drive cyls=c (removed in 3.0)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The drive geometry should now be specified via
-device ...,drive=dr,cyls=c,heads=h,secs=s (together with
-drive if=none,id=dr,...).

-drive serial=, -drive trans= & -drive addr= (removed in 3.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Use -device ...,drive=dr,serial=r,bios-chs-trans=t,addr=a instead
(together with -drive if=none,id=dr,...).

-net ...,vlan=x (removed in 3.0)
''''''''''''''''''''''''''''''''''''

The term "vlan" was very confusing for most users in this context (it's about
specifying a hub ID, not about IEEE 802.1Q or something similar), so this
has been removed. To connect one NIC frontend with a network backend, either
use -nic ... (e.g. for on-board NICs) or use -netdev ...,id=n together
with -device ...,netdev=n (for full control over pluggable NICs). To
connect multiple NICs or network backends via a hub device (which is what
vlan did), use -nic hubport,hubid=x,... or
-netdev hubport,id=n,hubid=x,... (with -device ...,netdev=n) instead.

-no-kvm-irqchip (removed in 3.0)
''''''''''''''''''''''''''''''''''''

Use -machine kernel_irqchip=off instead.

-no-kvm-pit-reinjection (removed in 3.0)
''''''''''''''''''''''''''''''''''''''''''''

Use -global kvm-pit.lost_tick_policy=discard instead.

-balloon (removed in 3.1)
'''''''''''''''''''''''''''''

The -balloon virtio option has been replaced by -device virtio-balloon.
The -balloon none option was a no-op and has no replacement.

-bootp (removed in 3.1)
'''''''''''''''''''''''''''

The -bootp /some/file argument is replaced by either
-netdev user,id=x,bootp=/some/file (for pluggable NICs, accompanied with
-device ...,netdev=x), or -nic user,bootp=/some/file (for on-board NICs).
The new syntax allows different settings to be provided per NIC.

-redir (removed in 3.1)
'''''''''''''''''''''''''''

The -redir [tcp|udp]:hostport:[guestaddr]:guestport option is replaced
by either -netdev user,id=x,hostfwd=[tcp|udp]:[hostaddr]:hostport-[guestaddr]:guestport
(for pluggable NICs, accompanied with -device ...,netdev=x) or by the option
-nic user,hostfwd=[tcp|udp]:[hostaddr]:hostport-[guestaddr]:guestport
(for on-board NICs). The new syntax allows different settings to be provided
per NIC.

-smb (removed in 3.1)
'''''''''''''''''''''''''

The -smb /some/dir argument is replaced by either
-netdev user,id=x,smb=/some/dir (for pluggable NICs, accompanied with
-device ...,netdev=x), or -nic user,smb=/some/dir (for on-board NICs).
The new syntax allows different settings to be provided per NIC.

-tftp (removed in 3.1)
''''''''''''''''''''''''''

The -tftp /some/dir argument is replaced by either
-netdev user,id=x,tftp=/some/dir (for pluggable NICs, accompanied with
-device ...,netdev=x), or -nic user,tftp=/some/dir (for embedded NICs).
The new syntax allows different settings to be provided per NIC.

-localtime (removed in 3.1)
'''''''''''''''''''''''''''''''

Replaced by -rtc base=localtime.

-nodefconfig (removed in 3.1)
'''''''''''''''''''''''''''''''''

Use -no-user-config instead.

-rtc-td-hack (removed in 3.1)
'''''''''''''''''''''''''''''''''

Use -rtc driftfix=slew instead.

-startdate (removed in 3.1)
'''''''''''''''''''''''''''''''

Replaced by -rtc base=date.

-vnc ...,tls=..., -vnc ...,x509=... & -vnc ...,x509verify=... (removed in 3.1)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The "tls-creds" option should be used instead to point to a "tls-creds-x509"
object created using "-object".

-mem-path fallback to RAM (removed in 5.0)
''''''''''''''''''''''''''''''''''''''''''''''

If guest RAM allocation from file pointed by mem-path failed,
QEMU was falling back to allocating from RAM, which might have resulted
in unpredictable behavior since the backing file specified by the user
as ignored. Currently, users are responsible for making sure the backing storage
specified with -mem-path can actually provide the guest RAM configured with
-m and QEMU fails to start up if RAM allocation is unsuccessful.

-net ...,name=... (removed in 5.1)
''''''''''''''''''''''''''''''''''''''

The name parameter of the -net option was a synonym
for the id parameter, which should now be used instead.

-numa node,mem=... (removed in 5.1)
'''''''''''''''''''''''''''''''''''''''

The parameter mem of -numa node was used to assign a part of guest RAM
to a NUMA node. But when using it, it's impossible to manage a specified RAM
chunk on the host side (like bind it to a host node, setting bind policy, ...),
so the guest ends up with the fake NUMA configuration with suboptiomal
performance.
However since 2014 there is an alternative way to assign RAM to a NUMA node
using parameter memdev, which does the same as mem and adds
means to actually manage node RAM on the host side. Use parameter memdev
with memory-backend-ram backend as replacement for parameter mem
to achieve the same fake NUMA effect or a properly configured
memory-backend-file backend to actually benefit from NUMA configuration.
New machine versions (since 5.1) will not accept the option but it will still
work with old machine types. User can check the QAPI schema to see if the legacy
option is supported by looking at MachineInfo::numa-mem-supported property.

-numa node (without memory specified) (removed in 5.2)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Splitting RAM by default between NUMA nodes had the same issues as mem
parameter with the difference that the role of the user plays QEMU using
implicit generic or board specific splitting rule.
Use memdev with memory-backend-ram backend or mem (if
it's supported by used machine type) to define mapping explicitly instead.
Users of existing VMs, wishing to preserve the same RAM distribution, should
configure it explicitly using -numa node,memdev options. Current RAM
distribution can be retrieved using HMP command info numa and if separate
memory devices (pc|nv-dimm) are present use info memory-device and subtract
device memory from output of info numa.

-smp (invalid topologies) (removed in 5.2)
''''''''''''''''''''''''''''''''''''''''''''''

CPU topology properties should describe whole machine topology including
possible CPUs.

However, historically it was possible to start QEMU with an incorrect topology
where n <= sockets * cores * threads < maxcpus,
which could lead to an incorrect topology enumeration by the guest.
Support for invalid topologies is removed, the user must ensure
topologies described with -smp include all possible cpus, i.e.
sockets * cores * threads = maxcpus.

-machine enforce-config-section=on|off (removed in 5.2)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The enforce-config-section property was replaced by the
-global migration.send-configuration={on|off} option.

-no-kvm (removed in 5.2)
''''''''''''''''''''''''''''

The -no-kvm argument was a synonym for setting -machine accel=tcg.

-realtime (removed in 6.0)
''''''''''''''''''''''''''''''

The -realtime mlock=on|off argument has been replaced by the
-overcommit mem-lock=on|off argument.

-show-cursor option (removed in 6.0)
''''''''''''''''''''''''''''''''''''''''

Use -display sdl,show-cursor=on, -display gtk,show-cursor=on
or -display default,show-cursor=on instead.

-tb-size option (removed in 6.0)
''''''''''''''''''''''''''''''''''''

QEMU 5.0 introduced an alternative syntax to specify the size of the translation
block cache, -accel tcg,tb-size=.

-usbdevice audio (removed in 6.0)
'''''''''''''''''''''''''''''''''''''

This option lacked the possibility to specify an audio backend device.
Use -device usb-audio now instead (and specify a corresponding USB
host controller or -usb if necessary).

-vnc acl (removed in 6.0)
'''''''''''''''''''''''''''''

The acl option to the -vnc argument has been replaced
by the tls-authz and sasl-authz options.

-mon ...,control=readline,pretty=on|off (removed in 6.0)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The pretty=on|off switch has no effect for HMP monitors and
its use is rejected.

-drive file=json:{...{'driver':'file'}} (removed in 6.0)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The 'file' driver for drives is no longer appropriate for character or host
devices and will only accept regular files (S_IFREG). The correct driver
for these file types is 'host_cdrom' or 'host_device' as appropriate.

Floppy controllers' drive properties (removed in 6.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''

Use -device floppy,... instead. When configuring onboard floppy
controllers
::

-global isa-fdc.driveA=...
-global sysbus-fdc.driveA=...
-global SUNW,fdtwo.drive=...

become
::

-device floppy,unit=0,drive=...

and
::

-global isa-fdc.driveB=...
-global sysbus-fdc.driveB=...

become
::

-device floppy,unit=1,drive=...

When plugging in a floppy controller
::

-device isa-fdc,...,driveA=...

becomes
::

-device isa-fdc,...
-device floppy,unit=0,drive=...

and
::

-device isa-fdc,...,driveB=...

becomes
::

-device isa-fdc,...
-device floppy,unit=1,drive=...

-drive with bogus interface type (removed in 6.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''

Drives with interface types other than if=none are for onboard
devices. Drives the board doesn't pick up can no longer be used with
-device. Use if=none instead.

-usbdevice ccid (removed in 6.0)
'''''''''''''''''''''''''''''''''''''

This option was undocumented and not used in the field.
Use -device usb-ccid instead.

RISC-V firmware not booted by default (removed in 5.1)
''''''''''''''''''''''''''''''''''''''''''''''''''''''

QEMU 5.1 changes the default behaviour from -bios none to -bios default
for the RISC-V virt machine and sifive_u machine.

-no-quit (removed in 7.0)
'''''''''''''''''''''''''''''

The -no-quit was a synonym for -display ...,window-close=off which
should be used instead.

--enable-fips (removed in 7.1)
''''''''''''''''''''''''''''''''''

This option restricted usage of certain cryptographic algorithms when
the host is operating in FIPS mode.

If FIPS compliance is required, QEMU should be built with the libgcrypt
or gnutls library enabled as a cryptography provider.

Neither the nettle library, or the built-in cryptography provider are
supported on FIPS enabled hosts.

-writeconfig (removed in 7.1)
'''''''''''''''''''''''''''''''''

The -writeconfig option was not able to serialize the entire contents
of the QEMU command line. It is thus considered a failed experiment
and removed without a replacement.

loaded property of secret and secret_keyring objects (removed in 7.1)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The loaded=on option in the command line or QMP object-add either had
no effect (if loaded was the last option) or caused options to be
effectively ignored as if they were not given. The property is therefore
useless and should simply be removed.

opened property of rng-* objects (removed in 7.1)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The opened=on option in the command line or QMP object-add either had
no effect (if opened was the last option) or caused errors. The property
is therefore useless and should simply be removed.

-display sdl,window_close=... (removed in 7.1)
''''''''''''''''''''''''''''''''''''''''''''''''''

Use -display sdl,window-close=... instead (i.e. with a minus instead of
an underscore between "window" and "close").

-alt-grab and -display sdl,alt_grab=on (removed in 7.1)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Use -display sdl,grab-mod=lshift-lctrl-lalt instead.

-ctrl-grab and -display sdl,ctrl_grab=on (removed in 7.1)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Use -display sdl,grab-mod=rctrl instead.

-sdl (removed in 7.1)
'''''''''''''''''''''''''

Use -display sdl instead.

-curses (removed in 7.1)
''''''''''''''''''''''''''''

Use -display curses instead.

Creating sound card devices using -soundhw (removed in 7.1)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Sound card devices should be created using -device or -audio.
The exception is pcspk which can be activated using -machine pcspk-audiodev=<name>.

-watchdog (since 7.2)
'''''''''''''''''''''''''

Use -device instead.

QEMU Machine Protocol (QMP) commands Copy Chapter

block-dirty-bitmap-add "autoload" parameter (removed in 4.2)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The "autoload" parameter has been ignored since 2.12.0. All bitmaps
are automatically loaded from qcow2 images.

cpu-add (removed in 5.2)
''''''''''''''''''''''''''''

Use device_add for hotplugging vCPUs instead of cpu-add. See
documentation of query-hotpluggable-cpus for additional details.

change (removed in 6.0)
'''''''''''''''''''''''''''

Use blockdev-change-medium or change-vnc-password or
display-update instead.

query-events (removed in 6.0)
'''''''''''''''''''''''''''''''''

The query-events command has been superseded by the more powerful
and accurate query-qmp-schema command.

migrate_set_cache_size and query-migrate-cache-size (removed in 6.0)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Use migrate_set_parameter and info migrate_parameters instead.

migrate_set_downtime and migrate_set_speed (removed in 6.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Use migrate_set_parameter instead.

query-cpus (removed in 6.0)
'''''''''''''''''''''''''''''''

The query-cpus command is replaced by the query-cpus-fast command.

query-cpus-fast arch output member (removed in 6.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The arch output member of the query-cpus-fast command is
replaced by the target output member.

chardev client socket with wait option (removed in 6.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Character devices creating sockets in client mode should not specify
the 'wait' field, which is only applicable to sockets in server mode

query-named-block-nodes result encryption_key_missing (removed in 6.0)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Removed with no replacement.

query-block result inserted.encryption_key_missing (removed in 6.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Removed with no replacement.

query-named-block-nodes and query-block result dirty-bitmaps[i].status (removed in 6.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The status field of the BlockDirtyInfo structure, returned by
these commands is removed. Two new boolean fields, recording and
busy effectively replace it.

query-block result field dirty-bitmaps (removed in 6.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The dirty-bitmaps field of the BlockInfo structure, returned by
the query-block command is itself now removed. The dirty-bitmaps
field of the BlockDeviceInfo struct should be used instead, which is the
type of the inserted field in query-block replies, as well as the
type of array items in query-named-block-nodes.

object-add option props (removed in 6.0)
''''''''''''''''''''''''''''''''''''''''''''''''

Specify the properties for the object as top-level arguments instead.

Human Monitor Protocol (HMP) commands Copy Chapter

usb_add and usb_remove (removed in 2.12)
''''''''''''''''''''''''''''''''''''''''''''''''

Replaced by device_add and device_del (use device_add help for a
list of available devices).

host_net_add and host_net_remove (removed in 2.12)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Replaced by netdev_add and netdev_del.

The hub_id parameter of hostfwd_add / hostfwd_remove (removed in 5.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The [hub_id name] parameter tuple of the 'hostfwd_add' and
'hostfwd_remove' HMP commands has been replaced by netdev_id.

cpu-add (removed in 5.2)
''''''''''''''''''''''''''''

Use device_add for hotplugging vCPUs instead of cpu-add. See
documentation of query-hotpluggable-cpus for additional details.

change vnc TARGET (removed in 6.0)
''''''''''''''''''''''''''''''''''''''

No replacement. The change vnc password and change DEVICE MEDIUM
commands are not affected.

acl_show, acl_reset, acl_policy, acl_add, acl_remove (removed in 6.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The acl_show, acl_reset, acl_policy, acl_add, and
acl_remove commands were removed with no replacement. Authorization
for VNC should be performed using the pluggable QAuthZ objects.

migrate-set-cache-size and info migrate-cache-size (removed in 6.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Use migrate-set-parameters and info migrate-parameters instead.

migrate_set_downtime and migrate_set_speed (removed in 6.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Use migrate-set-parameters instead.

info cpustats (removed in 6.1)
''''''''''''''''''''''''''''''''''

This command didn't produce any output already. Removed with no replacement.

Guest Emulator ISAs Copy Chapter

RISC-V ISA privilege specification version 1.09.1 (removed in 5.1)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The RISC-V ISA privilege specification version 1.09.1 has been removed.
QEMU supports both the newer version 1.10.0 and the ratified version 1.11.0, these
should be used instead of the 1.09.1 version.

System emulator CPUS Copy Chapter

KVM guest support on 32-bit Arm hosts (removed in 5.2)
''''''''''''''''''''''''''''''''''''''''''''''''''''''

The Linux kernel has dropped support for allowing 32-bit Arm systems
to host KVM guests as of the 5.7 kernel. Accordingly, QEMU is deprecating
its support for this configuration and will remove it in a future version.
Running 32-bit guests on a 64-bit Arm host remains supported.

RISC-V ISA Specific CPUs (removed in 5.1)
'''''''''''''''''''''''''''''''''''''''''

The RISC-V cpus with the ISA version in the CPU name have been removed. The
four CPUs are: rv32gcsu-v1.9.1, rv32gcsu-v1.10.0, rv64gcsu-v1.9.1 and
rv64gcsu-v1.10.0. Instead the version can be specified via the CPU priv_spec
option when using the rv32 or rv64 CPUs.

RISC-V no MMU CPUs (removed in 5.1)
'''''''''''''''''''''''''''''''''''

The RISC-V no MMU cpus have been removed. The two CPUs: rv32imacu-nommu and
rv64imacu-nommu can no longer be used. Instead the MMU status can be specified
via the CPU mmu option when using the rv32 or rv64 CPUs.

compat property of server class POWER CPUs (removed in 6.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The max-cpu-compat property of the pseries machine type should be used
instead.

moxie CPU (removed in 6.1)
''''''''''''''''''''''''''''''

Nobody was using this CPU emulation in QEMU, and there were no test images
available to make sure that the code is still working, so it has been removed
without replacement.

lm32 CPUs (removed in 6.1)
''''''''''''''''''''''''''''''

The only public user of this architecture was the milkymist project,
which has been dead for years; there was never an upstream Linux
port. Removed without replacement.

unicore32 CPUs (removed in 6.1)
'''''''''''''''''''''''''''''''''''

Support for this CPU was removed from the upstream Linux kernel, and
there is no available upstream toolchain to build binaries for it.
Removed without replacement.

x86 Icelake-Client CPU (removed in 7.1)
'''''''''''''''''''''''''''''''''''''''''''

There isn't ever Icelake Client CPU, it is some wrong and imaginary one.
Use Icelake-Server instead.

System emulator machines Copy Chapter

s390-virtio (removed in 2.6)
''''''''''''''''''''''''''''''''

Use the s390-ccw-virtio machine instead.

The m68k dummy machine (removed in 2.9)
'''''''''''''''''''''''''''''''''''''''''''

Use the none machine with the loader device instead.

xlnx-ep108 (removed in 3.0)
'''''''''''''''''''''''''''''''

The EP108 was an early access development board that is no longer used.
Use the xlnx-zcu102 machine instead.

spike_v1.9.1 and spike_v1.10 (removed in 5.1)
'''''''''''''''''''''''''''''''''''''''''''''''''''''

The version specific Spike machines have been removed in favour of the
generic spike machine. If you need to specify an older version of the RISC-V
spec you can use the -cpu rv64gcsu,priv_spec=v1.10.0 command line argument.

mips r4k platform (removed in 5.2)
''''''''''''''''''''''''''''''''''''''

This machine type was very old and unmaintained. Users should use the malta
machine type instead.

mips fulong2e machine alias (removed in 6.0)
''''''''''''''''''''''''''''''''''''''''''''''''

This machine has been renamed fuloong2e.

pc-0.10 up to pc-1.3 (removed in 4.0 up to 6.0)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''

These machine types were very old and likely could not be used for live
migration from old QEMU versions anymore. Use a newer machine type instead.

Raspberry Pi raspi2 and raspi3 machines (removed in 6.2)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The Raspberry Pi machines come in various models (A, A+, B, B+). To be able
to distinguish which model QEMU is implementing, the raspi2 and raspi3
machines have been renamed raspi2b and raspi3b.

Aspeed swift-bmc machine (removed in 7.0)
'''''''''''''''''''''''''''''''''''''''''''''

This machine was removed because it was unused. Alternative AST2500 based
OpenPOWER machines are witherspoon-bmc and romulus-bmc.

ppc taihu machine (removed in 7.2)
'''''''''''''''''''''''''''''''''''''''''''''

This machine was removed because it was partially emulated and 405
machines are very similar. Use the ref405ep machine instead.

linux-user mode CPUs Copy Chapter

tilegx CPUs (removed in 6.0)
''''''''''''''''''''''''''''''''

The tilegx guest CPU support has been removed without replacement. It was
only implemented in linux-user mode, but support for this CPU was removed from
the upstream Linux kernel in 2018, and it has also been dropped from glibc, so
there is no new Linux development taking place with this architecture. For
running the old binaries, you can use older versions of QEMU.

ppc64abi32 CPUs (removed in 7.0)
''''''''''''''''''''''''''''''''''''

The ppc64abi32 architecture has a number of issues which regularly
tripped up the CI testing and was suspected to be quite broken. For that
reason the maintainers strongly suspected no one actually used it.

TCG introspection features Copy Chapter

TCG trace-events (since 6.2)
''''''''''''''''''''''''''''

The ability to add new TCG trace points had bit rotted and as the
feature can be replicated with TCG plugins it was removed. If
any user is currently using this feature and needs help with
converting to using TCG plugins they should contact the qemu-devel
mailing list.

System emulator devices Copy Chapter

spapr-pci-vfio-host-bridge (removed in 2.12)
'''''''''''''''''''''''''''''''''''''''''''''''''

The spapr-pci-vfio-host-bridge device type has been replaced by the
spapr-pci-host-bridge device type.

ivshmem (removed in 4.0)
''''''''''''''''''''''''''''

Replaced by either the ivshmem-plain or ivshmem-doorbell.

ide-drive (removed in 6.0)
''''''''''''''''''''''''''''''

The 'ide-drive' device has been removed. Users should use 'ide-hd' or
'ide-cd' as appropriate to get an IDE hard disk or CD-ROM as needed.

scsi-disk (removed in 6.0)
''''''''''''''''''''''''''''''

The 'scsi-disk' device has been removed. Users should use 'scsi-hd' or
'scsi-cd' as appropriate to get a SCSI hard disk or CD-ROM as needed.

Related binaries Copy Chapter

qemu-nbd --partition (removed in 5.0)
'''''''''''''''''''''''''''''''''''''''''

The qemu-nbd --partition $digit code (also spelled -P)
could only handle MBR partitions, and never correctly handled logical
partitions beyond partition 5. Exporting a partition can still be
done by utilizing the --image-opts option with a raw blockdev
using the offset and size parameters layered on top of
any other existing blockdev. For example, if partition 1 is 100MiB
long starting at 1MiB, the old command::

qemu-nbd -t -P 1 -f qcow2 file.qcow2

can be rewritten as::

qemu-nbd -t --image-opts driver=raw,offset=1M,size=100M,file.driver=qcow2,file.file.driver=file,file.file.filename=file.qcow2

qemu-img convert -n -o (removed in 5.1)
'''''''''''''''''''''''''''''''''''''''''''

All options specified in -o are image creation options, so
they are now rejected when used with -n to skip image creation.

qemu-img create -b bad file $size (removed in 5.1)
''''''''''''''''''''''''''''''''''''''''''''''''''''''

When creating an image with a backing file that could not be opened,
qemu-img create used to issue a warning about the failure but
proceed with the image creation if an explicit size was provided.
However, as the -u option exists for this purpose, it is safer to
enforce that any failure to open the backing image (including if the
backing file is missing or an incorrect format was specified) is an
error when -u is not used.

qemu-img amend to adjust backing file (removed in 6.1)
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The use of qemu-img amend to modify the name or format of a qcow2
backing image was never fully documented or tested, and interferes
with other amend operations that need access to the original backing
image (such as deciding whether a v3 zero cluster may be left
unallocated when converting to a v2 image). Any changes to the
backing chain should be performed with qemu-img rebase -u either
before or after the remaining changes being performed by amend, as
appropriate.

qemu-img backing file without format (removed in 6.1)
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The use of qemu-img create, qemu-img rebase, or qemu-img convert to create or modify an image that depends on a backing file
now requires that an explicit backing format be provided. This is
for safety: if QEMU probes a different format than what you thought,
the data presented to the guest will be corrupt; similarly, presenting
a raw image to a guest allows a potential security exploit if a future
probe sees a non-raw image based on guest writes.

To avoid creating unsafe backing chains, you must pass -o backing_fmt= (or the shorthand -F during create) to specify the
intended backing format. You may use qemu-img rebase -u to
retroactively add a backing format to an existing image. However, be
aware that there are already potential security risks to blindly using
qemu-img info to probe the format of an untrusted backing image,
when deciding what format to add into an existing image.

Block devices Copy Chapter

VXHS backend (removed in 5.1)
'''''''''''''''''''''''''''''

The VXHS code did not compile since v2.12.0. It was removed in 5.1.

sheepdog driver (removed in 6.0)
''''''''''''''''''''''''''''''''''''

The corresponding upstream server project is no longer maintained.
Users are recommended to switch to an alternative distributed block
device driver such as RBD.

Devel Atomics (`docs/devel/atomics.rst`) Copy Section

========================= Atomic operations in QEMU

CPUs perform independent memory operations effectively in random order.
but this can be a problem for CPU-CPU interaction (including interactions
between QEMU and the guest). Multi-threaded programs use various tools
to instruct the compiler and the CPU to restrict the order to something
that is consistent with the expectations of the programmer.

The most basic tool is locking. Mutexes, condition variables and
semaphores are used in QEMU, and should be the default approach to
synchronization. Anything else is considerably harder, but it's
also justified more often than one would like;
the most performance-critical parts of QEMU in particular require
a very low level approach to concurrency, involving memory barriers
and atomic operations. The semantics of concurrent memory accesses are governed
by the C11 memory model.

QEMU provides a header, qemu/atomic.h, which wraps C11 atomics to
provide better portability and a less verbose syntax. qemu/atomic.h
provides macros that fall in three camps:

• compiler barriers: barrier();

• weak atomic access and manual memory barriers: qatomic_read(),
  qatomic_set(), smp_rmb(), smp_wmb(), smp_mb(),
  smp_mb_acquire(), smp_mb_release(), smp_read_barrier_depends();

• sequentially consistent atomic access: everything else.

In general, use of qemu/atomic.h should be wrapped with more easily
used data structures (e.g. the lock-free singly-linked list operations
QSLIST_INSERT_HEAD_ATOMIC and QSLIST_MOVE_ATOMIC) or synchronization
primitives (such as RCU, QemuEvent or QemuLockCnt). Bare use of
atomic operations and memory barriers should be limited to inter-thread
checking of flags and documented thoroughly.

Compiler memory barrier

barrier() prevents the compiler from moving the memory accesses on
either side of it to the other side. The compiler barrier has no direct
effect on the CPU, which may then reorder things however it wishes.

barrier() is mostly used within qemu/atomic.h itself. On some
architectures, CPU guarantees are strong enough that blocking compiler
optimizations already ensures the correct order of execution. In this
case, qemu/atomic.h will reduce stronger memory barriers to simple
compiler barriers.

Still, barrier() can be useful when writing code that can be interrupted
by signal handlers.

Sequentially consistent atomic access

Most of the operations in the qemu/atomic.h header ensure sequential
consistency, where "the result of any execution is the same as if the
operations of all the processors were executed in some sequential order,
and the operations of each individual processor appear in this sequence
in the order specified by its program".

qemu/atomic.h provides the following set of atomic read-modify-write
operations::

void qatomic_inc(ptr)
void qatomic_dec(ptr)
void qatomic_add(ptr, val)
void qatomic_sub(ptr, val)
void qatomic_and(ptr, val)
void qatomic_or(ptr, val)

typeof(*ptr) qatomic_fetch_inc(ptr)
typeof(*ptr) qatomic_fetch_dec(ptr)
typeof(*ptr) qatomic_fetch_add(ptr, val)
typeof(*ptr) qatomic_fetch_sub(ptr, val)
typeof(*ptr) qatomic_fetch_and(ptr, val)
typeof(*ptr) qatomic_fetch_or(ptr, val)
typeof(*ptr) qatomic_fetch_xor(ptr, val)
typeof(*ptr) qatomic_fetch_inc_nonzero(ptr)
typeof(*ptr) qatomic_xchg(ptr, val)
typeof(*ptr) qatomic_cmpxchg(ptr, old, new)

all of which return the old value of *ptr. These operations are
polymorphic; they operate on any type that is as wide as a pointer or
smaller.

Similar operations return the new value of *ptr::

typeof(*ptr) qatomic_inc_fetch(ptr)
typeof(*ptr) qatomic_dec_fetch(ptr)
typeof(*ptr) qatomic_add_fetch(ptr, val)
typeof(*ptr) qatomic_sub_fetch(ptr, val)
typeof(*ptr) qatomic_and_fetch(ptr, val)
typeof(*ptr) qatomic_or_fetch(ptr, val)
typeof(*ptr) qatomic_xor_fetch(ptr, val)

qemu/atomic.h also provides loads and stores that cannot be reordered
with each other::

typeof(*ptr) qatomic_mb_read(ptr)
void         qatomic_mb_set(ptr, val)

However these do not provide sequential consistency and, in particular,
they do not participate in the total ordering enforced by
sequentially-consistent operations. For this reason they are deprecated.
They should instead be replaced with any of the following (ordered from
easiest to hardest):

• accesses inside a mutex or spinlock

• lightweight synchronization primitives such as QemuEvent

• RCU operations (qatomic_rcu_read, qatomic_rcu_set) when publishing
  or accessing a new version of a data structure

• other atomic accesses: qatomic_read and qatomic_load_acquire for
  loads, qatomic_set and qatomic_store_release for stores, smp_mb
  to forbid reordering subsequent loads before a store.

Weak atomic access and manual memory barriers

Compared to sequentially consistent atomic access, programming with
weaker consistency models can be considerably more complicated.
The only guarantees that you can rely upon in this case are:

• atomic accesses will not cause data races (and hence undefined behavior);
  ordinary accesses instead cause data races if they are concurrent with
  other accesses of which at least one is a write. In order to ensure this,
  the compiler will not optimize accesses out of existence, create unsolicited
  accesses, or perform other similar optimzations.

• acquire operations will appear to happen, with respect to the other
  components of the system, before all the LOAD or STORE operations
  specified afterwards.

• release operations will appear to happen, with respect to the other
  components of the system, after all the LOAD or STORE operations
  specified before.

• release operations will synchronize with acquire operations;
  see :ref:acqrel for a detailed explanation.

When using this model, variables are accessed with:

• qatomic_read() and qatomic_set(); these prevent the compiler from
  optimizing accesses out of existence and creating unsolicited
  accesses, but do not otherwise impose any ordering on loads and
  stores: both the compiler and the processor are free to reorder
  them.

• qatomic_load_acquire(), which guarantees the LOAD to appear to
  happen, with respect to the other components of the system,
  before all the LOAD or STORE operations specified afterwards.
  Operations coming before qatomic_load_acquire() can still be
  reordered after it.

• qatomic_store_release(), which guarantees the STORE to appear to
  happen, with respect to the other components of the system,
  after all the LOAD or STORE operations specified before.
  Operations coming after qatomic_store_release() can still be
  reordered before it.

Restrictions to the ordering of accesses can also be specified
using the memory barrier macros: smp_rmb(), smp_wmb(), smp_mb(),
smp_mb_acquire(), smp_mb_release(), smp_read_barrier_depends().

Memory barriers control the order of references to shared memory.
They come in six kinds:

• smp_rmb() guarantees that all the LOAD operations specified before
  the barrier will appear to happen before all the LOAD operations
  specified after the barrier with respect to the other components of
  the system.

  In other words, smp_rmb() puts a partial ordering on loads, but is not
  required to have any effect on stores.

• smp_wmb() guarantees that all the STORE operations specified before
  the barrier will appear to happen before all the STORE operations
  specified after the barrier with respect to the other components of
  the system.

  In other words, smp_wmb() puts a partial ordering on stores, but is not
  required to have any effect on loads.

• smp_mb_acquire() guarantees that all the LOAD operations specified before
  the barrier will appear to happen before all the LOAD or STORE operations
  specified after the barrier with respect to the other components of
  the system.

• smp_mb_release() guarantees that all the STORE operations specified after
  the barrier will appear to happen after all the LOAD or STORE operations
  specified before the barrier with respect to the other components of
  the system.

• smp_mb() guarantees that all the LOAD and STORE operations specified
  before the barrier will appear to happen before all the LOAD and
  STORE operations specified after the barrier with respect to the other
  components of the system.

  smp_mb() puts a partial ordering on both loads and stores. It is
  stronger than both a read and a write memory barrier; it implies both
  smp_mb_acquire() and smp_mb_release(), but it also prevents STOREs
  coming before the barrier from overtaking LOADs coming after the
  barrier and vice versa.

• smp_read_barrier_depends() is a weaker kind of read barrier. On
  most processors, whenever two loads are performed such that the
  second depends on the result of the first (e.g., the first load
  retrieves the address to which the second load will be directed),
  the processor will guarantee that the first LOAD will appear to happen
  before the second with respect to the other components of the system.
  However, this is not always true---for example, it was not true on
  Alpha processors. Whenever this kind of access happens to shared
  memory (that is not protected by a lock), a read barrier is needed,
  and smp_read_barrier_depends() can be used instead of smp_rmb().

  Note that the first load really has to have a data dependency and not
  a control dependency. If the address for the second load is dependent
  on the first load, but the dependency is through a conditional rather
  than actually loading the address itself, then it's a control
  dependency and a full read barrier or better is required.

Memory barriers and qatomic_load_acquire/qatomic_store_release are
mostly used when a data structure has one thread that is always a writer
and one thread that is always a reader:

+----------------------------------+----------------------------------+
| thread 1                         | thread 2                         |
+==================================+==================================+
| ::                               | ::                               |
|                                  |                                  |
|   qatomic_store_release(&a, x);  |   y = qatomic_load_acquire(&b);  |
|   qatomic_store_release(&b, y);  |   x = qatomic_load_acquire(&a);  |
+----------------------------------+----------------------------------+

In this case, correctness is easy to check for using the "pairing"
trick that is explained below.

Sometimes, a thread is accessing many variables that are otherwise
unrelated to each other (for example because, apart from the current
thread, exactly one other thread will read or write each of these
variables). In this case, it is possible to "hoist" the barriers
outside a loop. For example:

+------------------------------------------+----------------------------------+
| before                                   | after                            |
+==========================================+==================================+
| ::                                       | ::                               |
|                                          |                                  |
|   n = 0;                                 |   n = 0;                         |
|   for (i = 0; i < 10; i++)               |   for (i = 0; i < 10; i++)       |
|     n += qatomic_load_acquire(&a[i]);    |     n += qatomic_read(&a[i]);    |
|                                          |   smp_mb_acquire();              |
+------------------------------------------+----------------------------------+
| ::                                       | ::                               |
|                                          |                                  |
|                                          |   smp_mb_release();              |
|   for (i = 0; i < 10; i++)               |   for (i = 0; i < 10; i++)       |
|     qatomic_store_release(&a[i], false); |     qatomic_set(&a[i], false);   |
+------------------------------------------+----------------------------------+

Splitting a loop can also be useful to reduce the number of barriers:

+------------------------------------------+----------------------------------+
| before                                   | after                            |
+==========================================+==================================+
| ::                                       | ::                               |
|                                          |                                  |
|   n = 0;                                 |     smp_mb_release();            |
|   for (i = 0; i < 10; i++) {             |     for (i = 0; i < 10; i++)     |
|     qatomic_store_release(&a[i], false); |       qatomic_set(&a[i], false); |
|     smp_mb();                            |     smb_mb();                    |
|     n += qatomic_read(&b[i]);            |     n = 0;                       |
|   }                                      |     for (i = 0; i < 10; i++)     |
|                                          |       n += qatomic_read(&b[i]);  |
+------------------------------------------+----------------------------------+

In this case, a smp_mb_release() is also replaced with a (possibly cheaper, and clearer
as well) smp_wmb():

+------------------------------------------+----------------------------------+
| before                                   | after                            |
+==========================================+==================================+
| ::                                       | ::                               |
|                                          |                                  |
|                                          |     smp_mb_release();            |
|   for (i = 0; i < 10; i++) {             |     for (i = 0; i < 10; i++)     |
|     qatomic_store_release(&a[i], false); |       qatomic_set(&a[i], false); |
|     qatomic_store_release(&b[i], false); |     smb_wmb();                   |
|   }                                      |     for (i = 0; i < 10; i++)     |
|                                          |       qatomic_set(&b[i], false); |
+------------------------------------------+----------------------------------+

.. _acqrel:

Acquire/release pairing and the *synchronizes-with* relation Copy Chapter

Atomic operations other than qatomic_set() and qatomic_read() have
either acquire or release semantics [#rmw]_. This has two effects:

.. [#rmw] Read-modify-write operations can have both---acquire applies to the
read part, and release to the write.

• within a thread, they are ordered either before subsequent operations
  (for acquire) or after previous operations (for release).

• if a release operation in one thread synchronizes with an acquire operation
  in another thread, the ordering constraints propagates from the first to the
  second thread. That is, everything before the release operation in the
  first thread is guaranteed to happen before everything after the
  acquire operation in the second thread.

The concept of acquire and release semantics is not exclusive to atomic
operations; almost all higher-level synchronization primitives also have
acquire or release semantics. For example:

• pthread_mutex_lock has acquire semantics, pthread_mutex_unlock has
  release semantics and synchronizes with a pthread_mutex_lock for the
  same mutex.

• pthread_cond_signal and pthread_cond_broadcast have release semantics;
  pthread_cond_wait has both release semantics (synchronizing with
  pthread_mutex_lock) and acquire semantics (synchronizing with
  pthread_mutex_unlock and signaling of the condition variable).

• pthread_create has release semantics and synchronizes with the start
  of the new thread; pthread_join has acquire semantics and synchronizes
  with the exiting of the thread.

• qemu_event_set has release semantics, qemu_event_wait has
  acquire semantics.

For example, in the following example there are no atomic accesses, but still
thread 2 is relying on the synchronizes-with relation between pthread_exit
(release) and pthread_join (acquire):

+----------------------+-------------------------------+
  | thread 1             | thread 2                      |
  +======================+===============================+
  | ::                   | ::                            |
  |                      |                               |
  |   *a = 1;            |                               |
  |   pthread_exit(a);   |   pthread_join(thread1, &a);  |
  |                      |   x = *a;                     |
  +----------------------+-------------------------------+

Synchronization between threads basically descends from this pairing of
a release operation and an acquire operation. Therefore, atomic operations
other than qatomic_set() and qatomic_read() will almost always be
paired with another operation of the opposite kind: an acquire operation
will pair with a release operation and vice versa. This rule of thumb is
extremely useful; in the case of QEMU, however, note that the other
operation may actually be in a driver that runs in the guest!

smp_read_barrier_depends(), smp_rmb(), smp_mb_acquire(),
qatomic_load_acquire() and qatomic_rcu_read() all count
as acquire operations. smp_wmb(), smp_mb_release(),
qatomic_store_release() and qatomic_rcu_set() all count as release
operations. smp_mb() counts as both acquire and release, therefore
it can pair with any other atomic operation. Here is an example:

+----------------------+------------------------------+
  | thread 1             | thread 2                     |
  +======================+==============================+
  | ::                   | ::                           |
  |                      |                              |
  |   qatomic_set(&a, 1);|                              |
  |   smp_wmb();         |                              |
  |   qatomic_set(&b, 2);|   x = qatomic_read(&b);      |
  |                      |   smp_rmb();                 |
  |                      |   y = qatomic_read(&a);      |
  +----------------------+------------------------------+

Note that a load-store pair only counts if the two operations access the
same variable: that is, a store-release on a variable x synchronizes
with a load-acquire on a variable x, while a release barrier
synchronizes with any acquire operation. The following example shows
correct synchronization:

+--------------------------------+--------------------------------+
  | thread 1                       | thread 2                       |
  +================================+================================+
  | ::                             | ::                             |
  |                                |                                |
  |   qatomic_set(&a, 1);          |                                |
  |   qatomic_store_release(&b, 2);|   x = qatomic_load_acquire(&b);|
  |                                |   y = qatomic_read(&a);        |
  +--------------------------------+--------------------------------+

Acquire and release semantics of higher-level primitives can also be
relied upon for the purpose of establishing the synchronizes with
relation.

Note that the "writing" thread is accessing the variables in the
opposite order as the "reading" thread. This is expected: stores
before a release operation will normally match the loads after
the acquire operation, and vice versa. In fact, this happened already
in the pthread_exit/pthread_join example above.

Finally, this more complex example has more than two accesses and data
dependency barriers. It also does not use atomic accesses whenever there
cannot be a data race:

+----------------------+------------------------------+
  | thread 1             | thread 2                     |
  +======================+==============================+
  | ::                   | ::                           |
  |                      |                              |
  |   b[2] = 1;          |                              |
  |   smp_wmb();         |                              |
  |   x->i = 2;          |                              |
  |   smp_wmb();         |                              |
  |   qatomic_set(&a, x);|  x = qatomic_read(&a);       |
  |                      |  smp_read_barrier_depends(); |
  |                      |  y = x->i;                   |
  |                      |  smp_read_barrier_depends(); |
  |                      |  z = b[y];                   |
  +----------------------+------------------------------+

Comparison with Linux kernel primitives

Here is a list of differences between Linux kernel atomic operations
and memory barriers, and the equivalents in QEMU:

• atomic operations in Linux are always on a 32-bit int type and
  use a boxed atomic_t type; atomic operations in QEMU are polymorphic
  and use normal C types.

• Originally, atomic_read and atomic_set in Linux gave no guarantee
  at all. Linux 4.1 updated them to implement volatile
  semantics via ACCESS_ONCE (or the more recent READ/WRITE_ONCE).

  QEMU's qatomic_read and qatomic_set implement C11 atomic relaxed
  semantics if the compiler supports it, and volatile semantics otherwise.
  Both semantics prevent the compiler from doing certain transformations;
  the difference is that atomic accesses are guaranteed to be atomic,
  while volatile accesses aren't. Thus, in the volatile case we just cross
  our fingers hoping that the compiler will generate atomic accesses,
  since we assume the variables passed are machine-word sized and
  properly aligned.

  No barriers are implied by qatomic_read and qatomic_set in either
  Linux or QEMU.

• atomic read-modify-write operations in Linux are of three kinds:

  text Copy

  ===================== =========================================
     ``atomic_OP``         returns void
     ``atomic_OP_return``  returns new value of the variable
     ``atomic_fetch_OP``   returns the old value of the variable
     ``atomic_cmpxchg``    returns the old value of the variable
     ===================== =========================================

  In QEMU, the second kind is named atomic_OP_fetch.

• different atomic read-modify-write operations in Linux imply
  a different set of memory barriers; in QEMU, all of them enforce
  sequential consistency.

• in QEMU, qatomic_read() and qatomic_set() do not participate in
  the total ordering enforced by sequentially-consistent operations.
  This is because QEMU uses the C11 memory model. The following example
  is correct in Linux but not in QEMU:

  text Copy

  +----------------------------------+--------------------------------+
  | Linux (correct)                  | QEMU (incorrect)               |
  +==================================+================================+
  | ::                               | ::                             |
  |                                  |                                |
  |   a = atomic_fetch_add(&x, 2);   |   a = qatomic_fetch_add(&x, 2);|
  |   b = READ_ONCE(&y);             |   b = qatomic_read(&y);        |
  +----------------------------------+--------------------------------+

  because the read of y can be moved (by either the processor or the
  compiler) before the write of x.

  Fixing this requires an smp_mb() memory barrier between the write
  of x and the read of y. In the common case where only one thread
  writes x, it is also possible to write it like this:

  text Copy

  +--------------------------------+
  | QEMU (correct)                 |
  +================================+
  | ::                             |
  |                                |
  |   a = qatomic_read(&x);        |
  |   qatomic_set(&x, a + 2);      |
  |   smp_mb();                    |
  |   b = qatomic_read(&y);        |
  +--------------------------------+

Sources

• Documentation/memory-barriers.txt from the Linux kernel

Devel Bitops (`docs/devel/bitops.rst`) Copy Section

================== Bitwise operations

The header qemu/bitops.h provides utility functions for
performing bitwise operations.

.. kernel-doc:: include/qemu/bitops.h

Devel Blkdebug (`docs/devel/blkdebug.txt`) Copy Section

Block I/O error injection using blkdebug Copy Chapter

Copyright (C) 2014-2015 Red Hat Inc

This work is licensed under the terms of the GNU GPL, version 2 or later. See
the COPYING file in the top-level directory.

The blkdebug block driver is a rule-based error injection engine. It can be
used to exercise error code paths in block drivers including ENOSPC (out of
space) and EIO.

This document gives an overview of the features available in blkdebug.

Background Copy Chapter

Block drivers have many error code paths that handle I/O errors. Image formats
are especially complex since metadata I/O errors during cluster allocation or
while updating tables happen halfway through request processing and require
discipline to keep image files consistent.

Error injection allows test cases to trigger I/O errors at specific points.
This way, all error paths can be tested to make sure they are correct.

Rules Copy Chapter

The blkdebug block driver takes a list of "rules" that tell the error injection
engine when to fail an I/O request.

Each I/O request is evaluated against the rules. If a rule matches the request
then its "action" is executed.

Rules can be placed in a configuration file; the configuration file
follows the same .ini-like format used by QEMU's -readconfig option, and
each section of the file represents a rule.

The following configuration file defines a single rule:

$ cat blkdebug.conf
[inject-error]
event = "read_aio"
errno = "28"

This rule fails all aio read requests with ENOSPC (28). Note that the errno
value depends on the host. On Linux, see
/usr/include/asm-generic/errno-base.h for errno values.

Invoke QEMU as follows:

$ qemu-system-x86_64
-drive if=none,cache=none,file=blkdebug:blkdebug.conf:test.img,id=drive0
-device virtio-blk-pci,drive=drive0,id=virtio-blk-pci0

Rules support the following attributes:

event - which type of operation to match (e.g. read_aio, write_aio,
flush_to_os, flush_to_disk). See the "Events" section for
information on events.

state - (optional) the engine must be in this state number in order for this
rule to match. See the "State transitions" section for information
on states.

errno - the numeric errno value to return when a request matches this rule.
The errno values depend on the host since the numeric values are not
standardized in the POSIX specification.

sector - (optional) a sector number that the request must overlap in order to
match this rule

once - (optional, default "off") only execute this action on the first
matching request

immediately - (optional, default "off") return a NULL BlockAIOCB
pointer and fail without an errno instead. This
exercises the code path where BlockAIOCB fails and the
caller's BlockCompletionFunc is not invoked.

Events Copy Chapter

Block drivers provide information about the type of I/O request they are about
to make so rules can match specific types of requests. For example, the qcow2
block driver tells blkdebug when it accesses the L1 table so rules can match
only L1 table accesses and not other metadata or guest data requests.

The core events are:

read_aio - guest data read

write_aio - guest data write

flush_to_os - write out unwritten block driver state (e.g. cached metadata)

flush_to_disk - flush the host block device's disk cache

See qapi/block-core.json:BlkdebugEvent for the full list of events.
You may need to grep block driver source code to understand the
meaning of specific events.

State transitions Copy Chapter

There are cases where more power is needed to match a particular I/O request in
a longer sequence of requests. For example:

write_aio
flush_to_disk
write_aio

How do we match the 2nd write_aio but not the first? This is where state
transitions come in.

The error injection engine has an integer called the "state" that always starts
initialized to 1. The state integer is internal to blkdebug and cannot be
observed from outside but rules can interact with it for powerful matching
behavior.

Rules can be conditional on the current state and they can transition to a new
state.

When a rule's "state" attribute is non-zero then the current state must equal
the attribute in order for the rule to match.

For example, to match the 2nd write_aio:

[set-state]
event = "write_aio"
state = "1"
new_state = "2"

[inject-error]
event = "write_aio"
state = "2"
errno = "5"

The first write_aio request matches the set-state rule and transitions from
state 1 to state 2. Once state 2 has been entered, the set-state rule no
longer matches since it requires state 1. But the inject-error rule now
matches the next write_aio request and injects EIO (5).

State transition rules support the following attributes:

event - which type of operation to match (e.g. read_aio, write_aio,
flush_to_os, flush_to_disk). See the "Events" section for
information on events.

state - (optional) the engine must be in this state number in order for this
rule to match

new_state - transition to this state number

Suspend and resume Copy Chapter

Exercising code paths in block drivers may require specific ordering amongst
concurrent requests. The "breakpoint" feature allows requests to be halted on
a blkdebug event and resumed later. This makes it possible to achieve
deterministic ordering when multiple requests are in flight.

Breakpoints on blkdebug events are associated with a user-defined "tag" string.
This tag serves as an identifier by which the request can be resumed at a later
point.

See the qemu-io(1) break, resume, remove_break, and wait_break commands for
details.

Devel Blkverify (`docs/devel/blkverify.txt`) Copy Section

= Block driver correctness testing with blkverify =

== Introduction ==

This document describes how to use the blkverify protocol to test that a block
driver is operating correctly.

It is difficult to test and debug block drivers against real guests. Often
processes inside the guest will crash because corrupt sectors were read as part
of the executable. Other times obscure errors are raised by a program inside
the guest. These issues are extremely hard to trace back to bugs in the block
driver.

Blkverify solves this problem by catching data corruption inside QEMU the first
time bad data is read and reporting the disk sector that is corrupted.

== How it works ==

The blkverify protocol has two child block devices, the "test" device and the
"raw" device. Read/write operations are mirrored to both devices so their
state should always be in sync.

The "raw" device is a raw image, a flat file, that has identical starting
contents to the "test" image. The idea is that the "raw" device will handle
read/write operations correctly and not corrupt data. It can be used as a
reference for comparison against the "test" device.

After a mirrored read operation completes, blkverify will compare the data and
raise an error if it is not identical. This makes it possible to catch the
first instance where corrupt data is read.

== Example ==

Imagine raw.img has 0xcd repeated throughout its first sector:

$ ./qemu-io -c 'read -v 0 512' raw.img
00000000:  cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd  ................
00000010:  cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd  ................
[...]
000001e0:  cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd  ................
000001f0:  cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd  ................
read 512/512 bytes at offset 0
512.000000 bytes, 1 ops; 0.0000 sec (97.656 MiB/sec and 200000.0000 ops/sec)

And test.img is corrupt, its first sector is zeroed when it shouldn't be:

$ ./qemu-io -c 'read -v 0 512' test.img
00000000:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00000010:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[...]
000001e0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000001f0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
read 512/512 bytes at offset 0
512.000000 bytes, 1 ops; 0.0000 sec (81.380 MiB/sec and 166666.6667 ops/sec)

This error is caught by blkverify:

$ ./qemu-io -c 'read 0 512' blkverify:a.img:b.img
blkverify: read sector_num=0 nb_sectors=4 contents mismatch in sector 0

A more realistic scenario is verifying the installation of a guest OS:

$ ./qemu-img create raw.img 16G
$ ./qemu-img create -f qcow2 test.qcow2 16G
$ ./qemu-system-x86_64 -cdrom debian.iso \
      -drive file=blkverify:raw.img:test.qcow2

If the installation is aborted when blkverify detects corruption, use qemu-io
to explore the contents of the disk image at the sector in question.

Devel Block Coroutine Wrapper (`docs/devel/block-coroutine-wrapper.rst`) Copy Section

======================= block-coroutine-wrapper

A lot of functions in QEMU block layer (see block/*) can only be
called in coroutine context. Such functions are normally marked by the
coroutine_fn specifier. Still, sometimes we need to call them from
non-coroutine context; for this we need to start a coroutine, run the
needed function from it and wait for the coroutine to finish in a
BDRV_POLL_WHILE() loop. To run a coroutine we need a function with one
void* argument. So for each coroutine_fn function which needs a
non-coroutine interface, we should define a structure to pack the
parameters, define a separate function to unpack the parameters and
call the original function and finally define a new interface function
with same list of arguments as original one, which will pack the
parameters into a struct, create a coroutine, run it and wait in
BDRV_POLL_WHILE() loop. It's boring to create such wrappers by hand,
so we have a script to generate them.

Usage

Assume we have defined the coroutine_fn function
bdrv_co_foo(<some args>) and need a non-coroutine interface for it,
called bdrv_foo(<same args>). In this case the script can help. To
trigger the generation:

1. You need bdrv_foo declaration somewhere (for example, in
   block/coroutines.h) with the generated_co_wrapper mark,
   like this:

.. code-block:: c

int generated_co_wrapper bdrv_foo(<some args>);

2. You need to feed this declaration to block-coroutine-wrapper script.
   For this, add the .h (or .c) file with the declaration to the
   input: files(...) list of block_gen_c target declaration in
   block/meson.build

You are done. During the build, coroutine wrappers will be generated in
/block/block-gen.c.

Links

1. The script location is scripts/block-coroutine-wrapper.py.

2. Generic place for private generated_co_wrapper declarations is
   block/coroutines.h, for public declarations:
   include/block/block.h

3. The core API of generated coroutine wrappers is placed in
   (not generated) block/block-gen.h

Devel Build System (`docs/devel/build-system.rst`) Copy Section

================================== The QEMU build system architecture

This document aims to help developers understand the architecture of the
QEMU build system. As with projects using GNU autotools, the QEMU build
system has two stages, first the developer runs the "configure" script
to determine the local build environment characteristics, then they run
"make" to build the project. There is about where the similarities with
GNU autotools end, so try to forget what you know about them.

Stage 1: configure

The QEMU configure script is written directly in shell, and should be
compatible with any POSIX shell, hence it uses #!/bin/sh. An important
implication of this is that it is important to avoid using bash-isms on
development platforms where bash is the primary host.

In contrast to autoconf scripts, QEMU's configure is expected to be
silent while it is checking for features. It will only display output
when an error occurs, or to show the final feature enablement summary
on completion.

Because QEMU uses the Meson build system under the hood, only VPATH
builds are supported. There are two general ways to invoke configure &
perform a build:

• VPATH, build artifacts outside of QEMU source tree entirely::

  cd ../
  mkdir build
  cd build
  ../qemu/configure
  make

• VPATH, build artifacts in a subdir of QEMU source tree::

  mkdir build
  cd build
  ../configure
  make

The configure script automatically recognizes
command line options for which a same-named Meson option exists;
dashes in the command line are replaced with underscores.

Many checks on the compilation environment are still found in configure
rather than meson.build, but new checks should be added directly to
meson.build.

Patches are also welcome to move existing checks from the configure
phase to meson.build. When doing so, ensure that meson.build does
not use anymore the keys that you have removed from config-host.mak.
Typically these will be replaced in meson.build by boolean variables,
get_option('optname') invocations, or dep.found() expressions.
In general, the remaining checks have little or no interdependencies,
so they can be moved one by one.

Helper functions Copy Chapter

The configure script provides a variety of helper functions to assist
developers in checking for system features:

do_cc $ARGS...
Attempt to run the system C compiler passing it $ARGS...

do_cxx $ARGS...
Attempt to run the system C++ compiler passing it $ARGS...

compile_object $CFLAGS
Attempt to compile a test program with the system C compiler using
$CFLAGS. The test program must have been previously written to a file
called $TMPC. The replacement in Meson is the compiler object cc,
which has methods such as cc.compiles(),
cc.check_header(), cc.has_function().

compile_prog $CFLAGS $LDFLAGS
Attempt to compile a test program with the system C compiler using
$CFLAGS and link it with the system linker using $LDFLAGS. The test
program must have been previously written to a file called $TMPC.
The replacement in Meson is cc.find_library() and cc.links().

has $COMMAND
Determine if $COMMAND exists in the current environment, either as a
shell builtin, or executable binary, returning 0 on success. The
replacement in Meson is find_program().

check_define $NAME
Determine if the macro $NAME is defined by the system C compiler

check_include $NAME
Determine if the include $NAME file is available to the system C
compiler. The replacement in Meson is cc.has_header().

write_c_skeleton
Write a minimal C program main() function to the temporary file
indicated by $TMPC

error_exit $MESSAGE $MORE...
Print $MESSAGE to stderr, followed by $MORE... and then exit from the
configure script with non-zero status

query_pkg_config $ARGS...
Run pkg-config passing it $ARGS. If QEMU is doing a static build,
then --static will be automatically added to $ARGS

Stage 2: Meson

The Meson build system is currently used to describe the build
process for:

1. executables, which include:

   • Tools - qemu-img, qemu-nbd, qga (guest agent), etc

   • System emulators - qemu-system-$ARCH

   • Userspace emulators - qemu-$ARCH

   • Unit tests

2. documentation

3. ROMs, which can be either installed as binary blobs or compiled

4. other data files, such as icons or desktop files

All executables are built by default, except for some contrib/
binaries that are known to fail to build on some platforms (for example
32-bit or big-endian platforms). Tests are also built by default,
though that might change in the future.

The source code is highly modularized, split across many files to
facilitate building of all of these components with as little duplicated
compilation as possible. Using the Meson "sourceset" functionality,
meson.build files group the source files in rules that are
enabled according to the available system libraries and to various
configuration symbols. Sourcesets belong to one of four groups:

Subsystem sourcesets:
Various subsystems that are common to both tools and emulators have
their own sourceset, for example block_ss for the block device subsystem,
chardev_ss for the character device subsystem, etc. These sourcesets
are then turned into static libraries as follows::

libchardev = static_library('chardev', chardev_ss.sources(),
                            name_suffix: 'fa',
                            build_by_default: false)

chardev = declare_dependency(link_whole: libchardev)

As of Meson 0.55.1, the special .fa suffix should be used for everything
that is used with link_whole, to ensure that the link flags are placed
correctly in the command line.

Target-independent emulator sourcesets:
Various general purpose helper code is compiled only once and
the .o files are linked into all output binaries that need it.
This includes error handling infrastructure, standard data structures,
platform portability wrapper functions, etc.

Target-independent code lives in the common_ss, softmmu_ss and
user_ss sourcesets. common_ss is linked into all emulators,
softmmu_ss only in system emulators, user_ss only in user-mode
emulators.

Target-independent sourcesets must exercise particular care when using
if_false rules. The if_false rule will be used correctly when linking
emulator binaries; however, when compiling target-independent files
into .o files, Meson may need to pick both the if_true and
if_false sides to cater for targets that want either side. To
achieve that, you can add a special rule using the CONFIG_ALL
symbol::

# Some targets have CONFIG_ACPI, some don't, so this is not enough
softmmu_ss.add(when: 'CONFIG_ACPI', if_true: files('acpi.c'),
                                    if_false: files('acpi-stub.c'))

# This is required as well:
softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('acpi-stub.c'))

Target-dependent emulator sourcesets:
In the target-dependent set lives CPU emulation, some device emulation and
much glue code. This sometimes also has to be compiled multiple times,
once for each target being built. Target-dependent files are included
in the specific_ss sourceset.

Each emulator also includes sources for files in the hw/ and target/
subdirectories. The subdirectory used for each emulator comes
from the target's definition of TARGET_BASE_ARCH or (if missing)
TARGET_ARCH, as found in default-configs/targets/*.mak.

Each subdirectory in hw/ adds one sourceset to the hw_arch dictionary,
for example::

arm_ss = ss.source_set()
arm_ss.add(files('boot.c'), fdt)
...
hw_arch += {'arm': arm_ss}

The sourceset is only used for system emulators.

Each subdirectory in target/ instead should add one sourceset to each
of the target_arch and target_softmmu_arch, which are used respectively
for all emulators and for system emulators only. For example::

arm_ss = ss.source_set()
arm_softmmu_ss = ss.source_set()
...
target_arch += {'arm': arm_ss}
target_softmmu_arch += {'arm': arm_softmmu_ss}

Module sourcesets:
There are two dictionaries for modules: modules is used for
target-independent modules and target_modules is used for
target-dependent modules. When modules are disabled the module
source sets are added to softmmu_ss and the target_modules
source sets are added to specific_ss.

Both dictionaries are nested. One dictionary is created per
subdirectory, and these per-subdirectory dictionaries are added to
the toplevel dictionaries. For example::

hw_display_modules = {}
qxl_ss = ss.source_set()
...
hw_display_modules += { 'qxl': qxl_ss }
modules += { 'hw-display': hw_display_modules }

Utility sourcesets:
All binaries link with a static library libqemuutil.a. This library
is built from several sourcesets; most of them however host generated
code, and the only two of general interest are util_ss and stub_ss.

The separation between these two is purely for documentation purposes.
util_ss contains generic utility files. Even though this code is only
linked in some binaries, sometimes it requires hooks only in some of
these and depend on other functions that are not fully implemented by
all QEMU binaries. stub_ss links dummy stubs that will only be linked
into the binary if the real implementation is not present. In a way,
the stubs can be thought of as a portable implementation of the weak
symbols concept.

The following files concur in the definition of which files are linked
into each emulator:

default-configs/devices/*.mak
The files under default-configs/devices/ control the boards and devices
that are built into each QEMU system emulation targets. They merely contain
a list of config variable definitions such as::

include arm-softmmu.mak
CONFIG_XLNX_ZYNQMP_ARM=y
CONFIG_XLNX_VERSAL=y

*/Kconfig
These files are processed together with default-configs/devices/*.mak and
describe the dependencies between various features, subsystems and
device models. They are described in :ref:kconfig

default-configs/targets/*.mak
These files mostly define symbols that appear in the *-config-target.h
file for each emulator [#cfgtarget]_. However, the TARGET_ARCH
and TARGET_BASE_ARCH will also be used to select the hw/ and
target/ subdirectories that are compiled into each target.

.. [#cfgtarget] This header is included by qemu/osdep.h when
compiling files from the target-specific sourcesets.

These files rarely need changing unless you are adding a completely
new target, or enabling new devices or hardware for a particular
system/userspace emulation target

Adding checks Copy Chapter

New checks should be added to Meson. Compiler checks can be as simple as
the following::

config_host_data.set('HAVE_BTRFS_H', cc.has_header('linux/btrfs.h'))

A more complex task such as adding a new dependency usually
comprises the following tasks:

• Add a Meson build option to meson_options.txt.

• Add code to perform the actual feature check.

• Add code to include the feature status in config-host.h

• Add code to print out the feature status in the configure summary
  upon completion.

Taking the probe for SDL2_Image as an example, we have the following
in meson_options.txt::

option('sdl_image', type : 'feature', value : 'auto',
Unless the option was given a non-auto value (on the configure
command line), the detection code must be performed only if the
dependency will be used::

sdl_image = not_found
if not get_option('sdl_image').auto() or have_system
sdl_image = dependency('SDL2_image', required: get_option('sdl_image'),
method: 'pkg-config',
static: enable_static)
endif

This avoids warnings on static builds of user-mode emulators, for example.
Most of the libraries used by system-mode emulators are not available for
static linking.

The other supporting code is generally simple::

Create config-host.h (if applicable)

config_host_data.set('CONFIG_SDL_IMAGE', sdl_image.found())

Summary

summary_info += {'SDL image support': sdl_image.found()}

For the configure script to parse the new option, the
scripts/meson-buildoptions.sh file must be up-to-date; make update-buildoptions (or just make) will take care of updating it.

Support scripts Copy Chapter

Meson has a special convention for invoking Python scripts: if their
first line is #! /usr/bin/env python3 and the file is not executable,
find_program() arranges to invoke the script under the same Python
interpreter that was used to invoke Meson. This is the most common
and preferred way to invoke support scripts from Meson build files,
because it automatically uses the value of configure's --python= option.

In case the script is not written in Python, use a #! /usr/bin/env ...
line and make the script executable.

Scripts written in Python, where it is desirable to make the script
executable (for example for test scripts that developers may want to
invoke from the command line, such as tests/qapi-schema/test-qapi.py),
should be invoked through the python variable in meson.build. For
example::

test('QAPI schema regression tests', python,
args: files('test-qapi.py'),
env: test_env, suite: ['qapi-schema', 'qapi-frontend'])

This is needed to obey the --python= option passed to the configure
script, which may point to something other than the first python3
binary on the path.

Stage 3: makefiles

The use of GNU make is required with the QEMU build system.

The output of Meson is a build.ninja file, which is used with the Ninja
build system. QEMU uses a different approach, where Makefile rules are
synthesized from the build.ninja file. The main Makefile includes these
rules and wraps them so that e.g. submodules are built before QEMU.
The resulting build system is largely non-recursive in nature, in
contrast to common practices seen with automake.

Tests are also ran by the Makefile with the traditional make check
phony target, while benchmarks are run with make bench. Meson test
suites such as unit can be ran with make check-unit too. It is also
possible to run tests defined in meson.build with meson test.

Useful make targets Copy Chapter

help
Print a help message for the most common build targets.

print-VAR
Print the value of the variable VAR. Useful for debugging the build
system.

Important files for the build system

Statically defined files Copy Chapter

The following key files are statically defined in the source tree, with
the rules needed to build QEMU. Their behaviour is influenced by a
number of dynamically created files listed later.

Makefile
The main entry point used when invoking make to build all the components
of QEMU. The default 'all' target will naturally result in the build of
every component. Makefile takes care of recursively building submodules
directly via a non-recursive set of rules.

*/meson.build
The meson.build file in the root directory is the main entry point for the
Meson build system, and it coordinates the configuration and build of all
executables. Build rules for various subdirectories are included in
other meson.build files spread throughout the QEMU source tree.

tests/Makefile.include
Rules for external test harnesses. These include the TCG tests,
qemu-iotests and the Avocado-based integration tests.

tests/docker/Makefile.include
Rules for Docker tests. Like tests/Makefile, this file is included
directly by the top level Makefile, anything defined in this file will
influence the entire build system.

tests/vm/Makefile.include
Rules for VM-based tests. Like tests/Makefile, this file is included
directly by the top level Makefile, anything defined in this file will
influence the entire build system.

Dynamically created files Copy Chapter

The following files are generated dynamically by configure in order to
control the behaviour of the statically defined makefiles. This avoids
the need for QEMU makefiles to go through any pre-processing as seen
with autotools, where Makefile.am generates Makefile.in which generates
Makefile.

Built by configure:

config-host.mak
When configure has determined the characteristics of the build host it
will write a long list of variables to config-host.mak file. This
provides the various install directories, compiler / linker flags and a
variety of CONFIG_* variables related to optionally enabled features.
This is imported by the top level Makefile and meson.build in order to
tailor the build output.

config-host.mak is also used as a dependency checking mechanism. If make
sees that the modification timestamp on configure is newer than that on
config-host.mak, then configure will be re-run.

The variables defined here are those which are applicable to all QEMU
build outputs. Variables which are potentially different for each
emulator target are defined by the next file...

Built by Meson:

${TARGET-NAME}-config-devices.mak
TARGET-NAME is again the name of a system or userspace emulator. The
config-devices.mak file is automatically generated by make using the
scripts/make_device_config.sh program, feeding it the
default-configs/$TARGET-NAME file as input.

config-host.h, $TARGET_NAME-config-target.h, $TARGET_NAME-config-devices.h
These files are used by source code to determine what features are
enabled. They are generated from the contents of the corresponding
*.mak files using Meson's configure_file() function.

build.ninja
The build rules.

Built by Makefile:

Makefile.ninja
A Makefile include that bridges to ninja for the actual build. The
Makefile is mostly a list of targets that Meson included in build.ninja.

Makefile.mtest
The Makefile definitions that let "make check" run tests defined in
meson.build. The rules are produced from Meson's JSON description of
tests (obtained with "meson introspect --tests") through the script
scripts/mtest2make.py.

Devel Ci (`docs/devel/ci.rst`) Copy Section

.. _ci:

== CI

Most of QEMU's CI is run on GitLab's infrastructure although a number
of other CI services are used for specialised purposes. The most up to
date information about them and their status can be found on the
project wiki testing page <https://wiki.qemu.org/Testing/CI>_.

.. include:: ci-definitions.rst.inc
.. include:: ci-jobs.rst.inc
.. include:: ci-runners.rst.inc

Devel Clocks (`docs/devel/clocks.rst`) Copy Section

Modelling a clock tree in QEMU

What are clocks? Copy Chapter

Clocks are QOM objects developed for the purpose of modelling the
distribution of clocks in QEMU.

They allow us to model the clock distribution of a platform and detect
configuration errors in the clock tree such as badly configured PLL, clock
source selection or disabled clock.

The object is Clock and its QOM name is clock (in C code, the macro
TYPE_CLOCK).

Clocks are typically used with devices where they are used to model inputs
and outputs. They are created in a similar way to GPIOs. Inputs and outputs
of different devices can be connected together.

In these cases a Clock object is a child of a Device object, but this
is not a requirement. Clocks can be independent of devices. For
example it is possible to create a clock outside of any device to
model the main clock source of a machine.

Here is an example of clocks::

+---------+      +----------------------+   +--------------+
| Clock 1 |      |       Device B       |   |   Device C   |
|         |      | +-------+  +-------+ |   | +-------+    |
|         |>>-+-->>|Clock 2|  |Clock 3|>>--->>|Clock 6|    |
+---------+   |  | | (in)  |  | (out) | |   | | (in)  |    |
              |  | +-------+  +-------+ |   | +-------+    |
              |  |            +-------+ |   +--------------+
              |  |            |Clock 4|>>
              |  |            | (out) | |   +--------------+
              |  |            +-------+ |   |   Device D   |
              |  |            +-------+ |   | +-------+    |
              |  |            |Clock 5|>>--->>|Clock 7|    |
              |  |            | (out) | |   | | (in)  |    |
              |  |            +-------+ |   | +-------+    |
              |  +----------------------+   |              |
              |                             | +-------+    |
              +----------------------------->>|Clock 8|    |
                                            | | (in)  |    |
                                            | +-------+    |
                                            +--------------+

Clocks are defined in the include/hw/clock.h header and device
related functions are defined in the include/hw/qdev-clock.h
header.

The clock state Copy Chapter

The state of a clock is its period; it is stored as an integer
representing it in units of 2 :sup:-32 ns. The special value of 0 is used to
represent the clock being inactive or gated. The clocks do not model
the signal itself (pin toggling) or other properties such as the duty
cycle.

All clocks contain this state: outputs as well as inputs. This allows
the current period of a clock to be fetched at any time. When a clock
is updated, the value is immediately propagated to all connected
clocks in the tree.

To ease interaction with clocks, helpers with a unit suffix are defined for
every clock state setter or getter. The suffixes are:

• _ns for handling periods in nanoseconds
• _hz for handling frequencies in hertz

The 0 period value is converted to 0 in hertz and vice versa. 0 always means
that the clock is disabled.

Adding a new clock Copy Chapter

Adding clocks to a device must be done during the init method of the Device
instance.

To add an input clock to a device, the function qdev_init_clock_in()
must be used. It takes the name, a callback, an opaque parameter
for the callback and a mask of events when the callback should be
called (this will be explained in a following section).
Output is simpler; only the name is required. Typically::

qdev_init_clock_in(DEVICE(dev), "clk_in", clk_in_callback, dev, ClockUpdate);
qdev_init_clock_out(DEVICE(dev), "clk_out");

Both functions return the created Clock pointer, which should be saved in the
device's state structure for further use.

These objects will be automatically deleted by the QOM reference mechanism.

Note that it is possible to create a static array describing clock inputs and
outputs. The function qdev_init_clocks() must be called with the array as
parameter to initialize the clocks: it has the same behaviour as calling the
qdev_init_clock_in/out() for each clock in the array. To ease the array
construction, some macros are defined in include/hw/qdev-clock.h.
As an example, the following creates 2 clocks to a device: one input and one
output.

.. code-block:: c

/* device structure containing pointers to the clock objects */
typedef struct MyDeviceState {
    DeviceState parent_obj;
    Clock *clk_in;
    Clock *clk_out;
} MyDeviceState;

/*
 * callback for the input clock (see "Callback on input clock
 * change" section below for more information).
 */
static void clk_in_callback(void *opaque, ClockEvent event);

/*
 * static array describing clocks:
 * + a clock input named "clk_in", whose pointer is stored in
 *   the clk_in field of a MyDeviceState structure with callback
 *   clk_in_callback.
 * + a clock output named "clk_out" whose pointer is stored in
 *   the clk_out field of a MyDeviceState structure.
 */
static const ClockPortInitArray mydev_clocks = {
    QDEV_CLOCK_IN(MyDeviceState, clk_in, clk_in_callback, ClockUpdate),
    QDEV_CLOCK_OUT(MyDeviceState, clk_out),
    QDEV_CLOCK_END
};

/* device initialization function */
static void mydev_init(Object *obj)
{
    /* cast to MyDeviceState */
    MyDeviceState *mydev = MYDEVICE(obj);
    /* create and fill the pointer fields in the MyDeviceState */
    qdev_init_clocks(mydev, mydev_clocks);
    [...]
}

An alternative way to create a clock is to simply call
object_new(TYPE_CLOCK). In that case the clock will neither be an
input nor an output of a device. After the whole QOM hierarchy of the
clock has been set clock_setup_canonical_path() should be called.

At creation, the period of the clock is 0: the clock is disabled. You can
change it using clock_set_ns() or clock_set_hz().

Note that if you are creating a clock with a fixed period which will never
change (for example the main clock source of a board), then you'll have
nothing else to do. This value will be propagated to other clocks when
connecting the clocks together and devices will fetch the right value during
the first reset.

Clock callbacks Copy Chapter

You can give a clock a callback function in several ways:

• by passing it as an argument to qdev_init_clock_in()
• as an argument to the QDEV_CLOCK_IN() macro initializing an
  array to be passed to qdev_init_clocks()
• by directly calling the clock_set_callback() function

The callback function must be of this type:

.. code-block:: c

typedef void ClockCallback(void *opaque, ClockEvent event);

The opaque argument is the pointer passed to qdev_init_clock_in()
or clock_set_callback(); for qdev_init_clocks() it is the
dev device pointer.

The event argument specifies why the callback has been called.
When you register the callback you specify a mask of ClockEvent values
that you are interested in. The callback will only be called for those
events.

The events currently supported are:

• ClockPreUpdate : called when the input clock's period is about to
  update. This is useful if the device needs to do some action for
  which it needs to know the old value of the clock period. During
  this callback, Clock API functions like clock_get() or
  clock_ticks_to_ns() will use the old period.
• ClockUpdate : called after the input clock's period has changed.
  During this callback, Clock API functions like clock_ticks_to_ns()
  will use the new period.

Note that a clock only has one callback: it is not possible to register
different functions for different events. You must register a single
callback which listens for all of the events you are interested in,
and use the event argument to identify which event has happened.

Retrieving clocks from a device Copy Chapter

qdev_get_clock_in() and dev_get_clock_out() are available to
get the clock inputs or outputs of a device. For example:

.. code-block:: c

Clock *clk = qdev_get_clock_in(DEVICE(mydev), "clk_in");

or:

.. code-block:: c

Clock *clk = qdev_get_clock_out(DEVICE(mydev), "clk_out");

Connecting two clocks together Copy Chapter

To connect two clocks together, use the clock_set_source() function.
Given two clocks clk1, and clk2, clock_set_source(clk2, clk1);
configures clk2 to follow the clk1 period changes. Every time clk1
is updated, clk2 will be updated too.

When connecting clock between devices, prefer using the
qdev_connect_clock_in() function to set the source of an input
device clock. For example, to connect the input clock clk2 of
devB to the output clock clk1 of devA, do:

.. code-block:: c

qdev_connect_clock_in(devB, "clk2", qdev_get_clock_out(devA, "clk1"))

We used qdev_get_clock_out() above, but any clock can drive an
input clock, even another input clock. The following diagram shows
some examples of connections. Note also that a clock can drive several
other clocks.

::

+------------+ +--------------------------------------------------+
| Device A | | Device B |
| | | +---------------------+ |
| | | | Device C | |
| +-------+ | | +-------+ | +-------+ +-------+ | +-------+ |
| |Clock 1|>>-->>|Clock 2|>>+-->>|Clock 3| |Clock 5|>>>>|Clock 6|>>
| | (out) | | | | (in) | | | | (in) | | (out) | | | (out) | |
| +-------+ | | +-------+ | | +-------+ +-------+ | +-------+ |
+------------+ | | +---------------------+ |
| | |
| | +--------------+ |
| | | Device D | |
| | | +-------+ | |
| +-->>|Clock 4| | |
| | | (in) | | |
| | +-------+ | |
| +--------------+ |
+--------------------------------------------------+

In the above example, when Clock 1 is updated by Device A, three
clocks get the new clock period value: Clock 2, Clock 3 and Clock 4.

It is not possible to disconnect a clock or to change the clock connection
after it is connected.

Clock multiplier and divider settings Copy Chapter

By default, when clocks are connected together, the child
clocks run with the same period as their source (parent) clock.
The Clock API supports a built-in period multiplier/divider
mechanism so you can configure a clock to make its children
run at a different period from its own. If you call the
clock_set_mul_div() function you can specify the clock's
multiplier and divider values. The children of that clock
will all run with a period of parent_period * multiplier / divider.
For instance, if the clock has a frequency of 8MHz and you set its
multiplier to 2 and its divider to 3, the child clocks will run
at 12MHz.

You can change the multiplier and divider of a clock at runtime,
so you can use this to model clock controller devices which
have guest-programmable frequency multipliers or dividers.

Note that clock_set_mul_div() does not automatically call
clock_propagate(). If you make a runtime change to the
multiplier or divider you must call clock_propagate() yourself.

Unconnected input clocks Copy Chapter

A newly created input clock is disabled (period of 0). This means the
clock will be considered as disabled until the period is updated. If
the clock remains unconnected it will always keep its initial value
of 0. If this is not the desired behaviour, clock_set(),
clock_set_ns() or clock_set_hz() should be called on the Clock
object during device instance init. For example:

.. code-block:: c

clk = qdev_init_clock_in(DEVICE(dev), "clk-in", clk_in_callback,
                         dev, ClockUpdate);
/* set initial value to 10ns / 100MHz */
clock_set_ns(clk, 10);

To enforce that the clock is wired up by the board code, you can
call clock_has_source() in your device's realize method:

.. code-block:: c

if (!clock_has_source(s->clk)) {
error_setg(errp, "MyDevice: clk input must be connected");
return;
}

Note that this only checks that the clock has been wired up; it is
still possible that the output clock connected to it is disabled
or has not yet been configured, in which case the period will be
zero. You should use the clock callback to find out when the clock
period changes.

Fetching clock frequency/period Copy Chapter

To get the current state of a clock, use the functions clock_get()
or clock_get_hz().

clock_get() returns the period of the clock in its fully precise
internal representation, as an unsigned 64-bit integer in units of
2^-32 nanoseconds. (For many purposes clock_ticks_to_ns() will
be more convenient; see the section below on expiry deadlines.)

clock_get_hz() returns the frequency of the clock, rounded to the
next lowest integer. This implies some inaccuracy due to the rounding,
so be cautious about using it in calculations.

It is also possible to register a callback on clock frequency changes.
Here is an example, which assumes that clock_callback has been
specified as the callback for the ClockUpdate event:

.. code-block:: c

void clock_callback(void *opaque, ClockEvent event) {
    MyDeviceState *s = (MyDeviceState *) opaque;
    /*
     * 'opaque' is the argument passed to qdev_init_clock_in();
     * usually this will be the device state pointer.
     */

    /* do something with the new period */
    fprintf(stdout, "device new period is %" PRIu64 "* 2^-32 ns\n",
                    clock_get(dev->my_clk_input));
}

If you are only interested in the frequency for displaying it to
humans (for instance in debugging), use clock_display_freq(),
which returns a prettified string-representation, e.g. "33.3 MHz".
The caller must free the string with g_free() after use.

Calculating expiry deadlines Copy Chapter

A commonly required operation for a clock is to calculate how long
it will take for the clock to tick N times; this can then be used
to set a timer expiry deadline. Use the function clock_ticks_to_ns(),
which takes an unsigned 64-bit count of ticks and returns the length
of time in nanoseconds required for the clock to tick that many times.

It is important not to try to calculate expiry deadlines using a
shortcut like multiplying a "period of clock in nanoseconds" value
by the tick count, because clocks can have periods which are not a
whole number of nanoseconds, and the accumulated error in the
multiplication can be significant.

For a clock with a very long period and a large number of ticks,
the result of this function could in theory be too large to fit in
a 64-bit value. To avoid overflow in this case, clock_ticks_to_ns()
saturates the result to INT64_MAX (because this is the largest valid
input to the QEMUTimer APIs). Since INT64_MAX nanoseconds is almost
300 years, anything with an expiry later than that is in the "will
never happen" category. Callers of clock_ticks_to_ns() should
therefore generally not special-case the possibility of a saturated
result but just allow the timer to be set to that far-future value.
(If you are performing further calculations on the returned value
rather than simply passing it to a QEMUTimer function like
timer_mod_ns() then you should be careful to avoid overflow
in those calculations, of course.)

Obtaining tick counts Copy Chapter

For calculations where you need to know the number of ticks in
a given duration, use clock_ns_to_ticks(). This function handles
possible non-whole-number-of-nanoseconds periods and avoids
potential rounding errors. It will return '0' if the clock is stopped
(i.e. it has period zero). If the inputs imply a tick count that
overflows a 64-bit value (a very long duration for a clock with a
very short period) the output value is truncated, so effectively
the 64-bit output wraps around.

Changing a clock period Copy Chapter

A device can change its outputs using the clock_update(),
clock_update_ns() or clock_update_hz() function. It will trigger
updates on every connected input.

For example, let's say that we have an output clock clkout and we
have a pointer to it in the device state because we did the following
in init phase:

.. code-block:: c

dev->clkout = qdev_init_clock_out(DEVICE(dev), "clkout");

Then at any time (apart from the cases listed below), it is possible to
change the clock value by doing:

.. code-block:: c

clock_update_hz(dev->clkout, 1000 * 1000 * 1000); /* 1GHz */

Because updating a clock may trigger any side effects through
connected clocks and their callbacks, this operation must be done
while holding the qemu io lock.

For the same reason, one can update clocks only when it is allowed to have
side effects on other objects. In consequence, it is forbidden:

• during migration,
• and in the enter phase of reset.

Note that calling clock_update[_ns|_hz]() is equivalent to calling
clock_set[_ns|_hz]() (with the same arguments) then
clock_propagate() on the clock. Thus, setting the clock value can
be separated from triggering the side-effects. This is often required
to factorize code to handle reset and migration in devices.

Aliasing clocks Copy Chapter

Sometimes, one needs to forward, or inherit, a clock from another
device. Typically, when doing device composition, a device might
expose a sub-device's clock without interfering with it. The function
qdev_alias_clock() can be used to achieve this behaviour. Note
that it is possible to expose the clock under a different name.
qdev_alias_clock() works for both input and output clocks.

For example, if device B is a child of device A,
device_a_instance_init() may do something like this:

.. code-block:: c

void device_a_instance_init(Object *obj)
{
    AState *A = DEVICE_A(obj);
    BState *B;
    /* create object B as child of A */
    [...]
    qdev_alias_clock(B, "clk", A, "b_clk");
    /*
     * Now A has a clock "b_clk" which is an alias to
     * the clock "clk" of its child B.
     */
}

This function does not return any clock object. The new clock has the
same direction (input or output) as the original one. This function
only adds a link to the existing clock. In the above example, object B
remains the only object allowed to use the clock and device A must not
try to change the clock period or set a callback to the clock. This
diagram describes the example with an input clock::

+--------------------------+
|        Device A          |
|         +--------------+ |
|         |   Device B   | |
|         | +-------+    | |
>>"b_clk">>>| "clk" |    | |
|  (in)   | |  (in) |    | |
|         | +-------+    | |
|         +--------------+ |
+--------------------------+

Migration Copy Chapter

Clock state is not migrated automatically. Every device must handle its
clock migration. Alias clocks must not be migrated.

To ensure clock states are restored correctly during migration, there
are two solutions.

Clock states can be migrated by adding an entry into the device
vmstate description. You should use the VMSTATE_CLOCK macro for this.
This is typically used to migrate an input clock state. For example:

.. code-block:: c

MyDeviceState {
    DeviceState parent_obj;
    [...] /* some fields */
    Clock *clk;
};

VMStateDescription my_device_vmstate = {
    .name = "my_device",
    .fields = (VMStateField[]) {
        [...], /* other migrated fields */
        VMSTATE_CLOCK(clk, MyDeviceState),
        VMSTATE_END_OF_LIST()
    }
};

The second solution is to restore the clock state using information already
at our disposal. This can be used to restore output clock states using the
device state. The functions clock_set[_ns|_hz]() can be used during the
post_load() migration callback.

When adding clock support to an existing device, if you care about
migration compatibility you will need to be careful, as simply adding
a VMSTATE_CLOCK() line will break compatibility. Instead, you can
put the VMSTATE_CLOCK() line into a vmstate subsection with a
suitable needed function, and use clock_set() in a
pre_load() function to set the default value that will be used if
the source virtual machine in the migration does not send the clock
state.

Care should be taken not to use clock_update[_ns|_hz]() or
clock_propagate() during the whole migration procedure because it
will trigger side effects to other devices in an unknown state.

Devel Code Of Conduct (`docs/devel/code-of-conduct.rst`) Copy Section

Code of Conduct

The QEMU community is made up of a mixture of professionals and
volunteers from all over the world. Diversity is one of our strengths,
but it can also lead to communication issues and unhappiness.
To that end, we have a few ground rules that we ask people to adhere to.

• Be welcoming. We are committed to making participation in this project
  a harassment-free experience for everyone, regardless of level of
  experience, gender, gender identity and expression, sexual orientation,
  disability, personal appearance, body size, race, ethnicity, age, religion,
  or nationality.

• Be respectful. Not all of us will agree all the time. Disagreements, both
  social and technical, happen all the time and the QEMU community is no
  exception. When we disagree, we try to understand why. It is important that
  we resolve disagreements and differing views constructively. Members of the
  QEMU community should be respectful when dealing with other contributors as
  well as with people outside the QEMU community and with users of QEMU.

Harassment and other exclusionary behavior are not acceptable. A community
where people feel uncomfortable or threatened is neither welcoming nor
respectful. Examples of unacceptable behavior by participants include:

• The use of sexualized language or imagery

• Personal attacks

• Trolling or insulting/derogatory comments

• Public or private harassment

• Publishing other's private information, such as physical or electronic
  addresses, without explicit permission

This isn't an exhaustive list of things that you can't do. Rather, take
it in the spirit in which it's intended: a guide to make it easier to
be excellent to each other.

This code of conduct applies to all spaces managed by the QEMU project.
This includes IRC, the mailing lists, the issue tracker, community
events, and any other forums created by the project team which the
community uses for communication. This code of conduct also applies
outside these spaces, when an individual acts as a representative or a
member of the project or its community.

By adopting this code of conduct, project maintainers commit themselves
to fairly and consistently applying these principles to every aspect of
managing this project. If you believe someone is violating the code of
conduct, please read the :ref:conflict-resolution document for
information about how to proceed.

Sources Copy Chapter

This document is based on the Fedora Code of Conduct <http://web.archive.org/web/20210429132536/https://docs.fedoraproject.org/en-US/project/code-of-conduct/>__
(as of April 2021) and the Contributor Covenant version 1.3.0 <https://www.contributor-covenant.org/version/1/3/0/code-of-conduct/>__.

Devel Conflict Resolution (`docs/devel/conflict-resolution.rst`) Copy Section

.. _conflict-resolution:

Conflict Resolution Policy

Conflicts in the community can take many forms, from someone having a
bad day and using harsh and hurtful language on the mailing list to more
serious code of conduct violations (including sexist/racist statements
or threats of violence), and everything in between.

For the vast majority of issues, we aim to empower individuals to first
resolve conflicts themselves, asking for help when needed, and only
after that fails to escalate further. This approach gives people more
control over the outcome of their dispute.

How we resolve conflicts Copy Chapter

If you are experiencing conflict, please consider first addressing the
perceived conflict directly with other involved parties, preferably through
a real-time medium such as IRC. You could also try to get a third-party (e.g.
a mutual friend, and/or someone with background on the issue, but not
involved in the conflict) to intercede or mediate.

If this fails or if you do not feel comfortable proceeding this way, or
if the problem requires immediate escalation, report the issue to the QEMU
leadership committee by sending an email to [email protected], providing
references to the misconduct.
For very urgent topics, you can also inform one or more members through IRC.
The up-to-date list of members is available on the QEMU wiki <https://wiki.qemu.org/Conservancy>__.

Your report will be treated confidentially by the leadership committee and
not be published without your agreement. The QEMU leadership committee will
then do its best to review the incident in a timely manner, and will either
seek further information, or will make a determination on next steps.

Remedies Copy Chapter

Escalating an issue to the QEMU leadership committee may result in actions
impacting one or more involved parties. In the event the leadership
committee has to intervene, here are some of the ways they might respond:

1. Take no action. For example, if the leadership committee determines
   the complaint has not been substantiated or is being made in bad faith,
   or if it is deemed to be outside its purview.

2. A private reprimand, explaining the consequences of continued behavior,
   to one or more involved individuals.

3. A private reprimand and request for a private or public apology

4. A public reprimand and request for a public apology

5. A public reprimand plus a mandatory cooling off period. The cooling
   off period may require, for example, one or more of the following:
   abstaining from maintainer duties; not interacting with people involved,
   including unsolicited interaction with those enforcing the guidelines
   and interaction on social media; being denied participation to in-person
   events. The cooling off period is voluntary but may escalate to a
   temporary ban in order to enforce it.

6. A temporary or permanent ban from some or all current and future QEMU
   spaces (mailing lists, IRC, wiki, etc.), possibly including in-person
   events.

In the event of severe harassment, the leadership committee may advise that
the matter be escalated to the relevant local law enforcement agency. It
is however not the role of the leadership committee to initiate contact
with law enforcement on behalf of any of the community members involved
in an incident.

Sources Copy Chapter

This document was developed based on the Drupal Conflict Resolution Policy and Process <https://www.drupal.org/conflict-resolution>__
and the Mozilla Consequence Ladder <https://github.com/mozilla/diversity/blob/master/code-of-conduct-enforcement/consequence-ladder.md>__

Devel Control Flow Integrity (`docs/devel/control-flow-integrity.rst`) Copy Section

============================ Control-Flow Integrity (CFI)

This document describes the current control-flow integrity (CFI) mechanism in
QEMU. How it can be enabled, its benefits and deficiencies, and how it affects
new and existing code in QEMU

Basics Copy Chapter

CFI is a hardening technique that focusing on guaranteeing that indirect
function calls have not been altered by an attacker.
The type used in QEMU is a forward-edge control-flow integrity that ensures
function calls performed through function pointers, always call a "compatible"
function. A compatible function is a function with the same signature of the
function pointer declared in the source code.

This type of CFI is entirely compiler-based and relies on the compiler knowing
the signature of every function and every function pointer used in the code.
As of now, the only compiler that provides support for CFI is Clang.

CFI is best used on production binaries, to protect against unknown attack
vectors.

In case of a CFI violation (i.e. call to a non-compatible function) QEMU will
terminate abruptly, to stop the possible attack.

Building with CFI Copy Chapter

NOTE: CFI requires the use of link-time optimization. Therefore, when CFI is
selected, LTO will be automatically enabled.

To build with CFI, the minimum requirement is Clang 6+. If you
are planning to also enable fuzzing, then Clang 11+ is needed (more on this
later).

Given the use of LTO, a version of AR that supports LLVM IR is required.
The easies way of doing this is by selecting the AR provided by LLVM::

AR=llvm-ar-9 CC=clang-9 CXX=clang++-9 /path/to/configure --enable-cfi

CFI is enabled on every binary produced.

If desired, an additional flag to increase the verbosity of the output in case
of a CFI violation is offered (--enable-debug-cfi).

Using QEMU built with CFI Copy Chapter

A binary with CFI will work exactly like a standard binary. In case of a CFI
violation, the binary will terminate with an illegal instruction signal.

Incompatible code with CFI Copy Chapter

As mentioned above, CFI is entirely compiler-based and therefore relies on
compile-time knowledge of the code. This means that, while generally supported
for most code, some specific use pattern can break CFI compatibility, and
create false-positives. The two main patterns that can cause issues are:

• Just-in-time compiled code: since such code is created at runtime, the jump
  to the buffer containing JIT code will fail.

• Libraries loaded dynamically, e.g. with dlopen/dlsym, since the library was
  not known at compile time.

Current areas of QEMU that are not entirely compatible with CFI are:

1. TCG, since the idea of TCG is to pre-compile groups of instructions at
   runtime to speed-up interpretation, quite similarly to a JIT compiler

2. TCI, where the interpreter has to interpret the generic call operation

3. Plugins, since a plugin is implemented as an external library

4. Modules, since they are implemented as an external library

5. Directly calling signal handlers from the QEMU source code, since the
   signal handler may have been provided by an external library or even plugged
   at runtime.

Disabling CFI for a specific function Copy Chapter

If you are working on function that is performing a call using an
incompatible way, as described before, you can selectively disable CFI checks
for such function by using the decorator QEMU_DISABLE_CFI at function
definition, and add an explanation on why the function is not compatible
with CFI. An example of the use of QEMU_DISABLE_CFI is provided here::

/*
 * Disable CFI checks.
 * TCG creates binary blobs at runtime, with the transformed code.
 * A TB is a blob of binary code, created at runtime and called with an
 * indirect function call. Since such function did not exist at compile time,
 * the CFI runtime has no way to verify its signature and would fail.
 * TCG is not considered a security-sensitive part of QEMU so this does not
 * affect the impact of CFI in environment with high security requirements
 */
QEMU_DISABLE_CFI
static inline tcg_target_ulong cpu_tb_exec(CPUState *cpu, TranslationBlock *itb)

NOTE: CFI needs to be disabled at the caller function, (i.e. a compatible
cfi function that calls a non-compatible one), since the check is performed
when the function call is performed.

CFI and fuzzing Copy Chapter

There is generally no advantage of using CFI and fuzzing together, because
they target different environments (production for CFI, debug for fuzzing).

CFI could be used in conjunction with fuzzing to identify a broader set of
bugs that may not end immediately in a segmentation fault or triggering
an assertion. However, other sanitizers such as address and ub sanitizers
can identify such bugs in a more precise way than CFI.

There is, however, an interesting use case in using CFI in conjunction with
fuzzing, that is to make sure that CFI is not triggering any false positive
in remote-but-possible parts of the code.

CFI can be enabled with fuzzing, but with some caveats:

1. Fuzzing relies on the linker performing function wrapping at link-time.
   The standard BFD linker does not support function wrapping when LTO is
   also enabled. The workaround is to use LLVM's lld linker.
2. Fuzzing also relies on a custom linker script, which is only supported by
   lld with version 11+.

In other words, to compile with fuzzing and CFI, clang 11+ is required, and
lld needs to be used as a linker::

AR=llvm-ar-11 CC=clang-11 CXX=clang++-11 /path/to/configure --enable-cfi
-enable-fuzzing --extra-ldflags="-fuse-ld=lld"

and then, compile the fuzzers as usual.

Devel Decodetree (`docs/devel/decodetree.rst`) Copy Section

======================== Decodetree Specification

A decodetree is built from instruction patterns. A pattern may
represent a single architectural instruction or a group of same, depending
on what is convenient for further processing.

Each pattern has both fixedbits and fixedmask, the combination of which
describes the condition under which the pattern is matched::

(insn & fixedmask) == fixedbits

Each pattern may have fields, which are extracted from the insn and
passed along to the translator. Examples of such are registers,
immediates, and sub-opcodes.

In support of patterns, one may declare fields, argument sets, and
formats, each of which may be re-used to simplify further definitions.

Fields

Syntax::

field_def := '%' identifier ( unnamed_field )* ( !function=identifier )?
unnamed_field := number ':' ( 's' ) number

For unnamed_field, the first number is the least-significant bit position
of the field and the second number is the length of the field. If the 's' is
present, the field is considered signed. If multiple unnamed_fields are
present, they are concatenated. In this way one can define disjoint fields.

If !function is specified, the concatenated result is passed through the
named function, taking and returning an integral value.

One may use !function with zero unnamed_fields. This case is called
a parameter, and the named function is only passed the DisasContext
and returns an integral value extracted from there.

A field with no unnamed_fields and no !function is in error.

Field examples:

+---------------------------+---------------------------------------------+
| Input | Generated code |
+===========================+=============================================+
| %disp 0:s16 | sextract(i, 0, 16) |
+---------------------------+---------------------------------------------+
| %imm9 16:6 10:3 | extract(i, 16, 6) << 3 | extract(i, 10, 3) |
+---------------------------+---------------------------------------------+
| %disp12 0:s1 1:1 2:10 | sextract(i, 0, 1) << 11 | |
| | extract(i, 1, 1) << 10 | |
| | extract(i, 2, 10) |
+---------------------------+---------------------------------------------+
| %shimm8 5:s8 13:1 | expand_shimm8(sextract(i, 5, 8) << 1 | |
| !function=expand_shimm8 | extract(i, 13, 1)) |
+---------------------------+---------------------------------------------+

Argument Sets

Syntax::

args_def := '&' identifier ( args_elt )+ ( !extern )?
args_elt := identifier (':' identifier)?

Each args_elt defines an argument within the argument set.
If the form of the args_elt contains a colon, the first
identifier is the argument name and the second identifier is
the argument type. If the colon is missing, the argument
type will be int.

Each argument set will be rendered as a C structure "arg_$name"
with each of the fields being one of the member arguments.

If !extern is specified, the backing structure is assumed
to have been already declared, typically via a second decoder.

Argument sets are useful when one wants to define helper functions
for the translator functions that can perform operations on a common
set of arguments. This can ensure, for instance, that the AND
pattern and the OR pattern put their operands into the same named
structure, so that a common gen_logic_insn may be able to handle
the operations common between the two.

Argument set examples::

&reg3 ra rb rc
&loadstore reg base offset
&longldst reg base offset:int64_t

Formats